Most experienced researchers and consultants can put together an effective survey instrument, but even then, small issues can creep into the questionnaire that can compromise quality, introduce bias or lead to misleading results. While many of the common mistakes listed here may seem obvious, it is always good practice to review a questionnaire with these potential issues in mind.

Here are some of the common questionnaire mistakes market researchers should avoid:

Mistake #1: Leading the Respondent

This issue results when an adjective is used that can cause bias. While often unintentional, it is sometimes seen in political polling when trying to provoke a certain reaction to an issue. When it occurs in market research surveys, it tends to inflate ratings in aspects like customer satisfaction or product research.

Examples:

#1: Our dedicated, top-rated customer service representatives do their best to serve your needs. How satisfied are you with your experience today? [SATISFACTION SCALE]

Issue: The preamble to the actual question "signals" to the respondent that the representatives are highly rated and may lead to a respondent to discount a less than satisfactory experience. Eliminate any language that may add to a respondent's knowledge (good or bad) about a product or service.

#2: How likely are you to purchase the newly designed, state-of-the-art cell phone? [PURCHASE LIKELIHOOD SCALE]

Issue: The adjectives "newly designed" and "state-of-the-art" adds and/or highlights information about the product; use this type of language in product/concept descriptions, not in the question itself.

Mistake #2: Overlapping, Incomplete or Unclear Response Choices

Response options should almost always be mutually exclusive and definitive. Any ambiguity will confuse and/or frustrate respondents.

Examples:

#1: How long have you been in your current occupation?

• 1 - 2 years
• 2 - 5 years
• 5 - 10 years

Issue: The options do not take into account under 1 year or over 10 years. And the overlap of time would lead to those in the occupation for exactly 2 or 5 years to potentially answer differently, resulting in incorrect results. Add more granularity to choices or use number fill-ins when appropriate.

#2: What type of vehicle do you own?

• 2-door
• 4-door
• Van
• Hatchback
• Truck
• SUV
• Sedan

Issue: A respondent's vehicle would likely fall across multiple categories from the options provided, and some vehicle types are not included. Analyzing the results from this question would not provide a clear or accurate picture of the marketplace. Be sure lists are exhaustive and cover all possibilities.

Mistake #3: Vague Questions

Having respondents guess what response the study is looking for is a wasted opportunity. This most often occurs in open-ends when looking for something specific, but the respondent takes the question in a different direction. Also, be specific with timeframes.

Examples:

#1: How can we improve our food menu?

Issue: "Improve" could be taken many ways, from the type of food offered to the size of the type used on the menu. Be specific and clear when asking for input to gather more relevant data.

#2: How likely will you be to purchase [PRODUCT] in the near future?

Issue: The researcher likely assumes a certain timeframe for the phrase "in the near future," however it could be interpreted many different ways as respondent buying and consumption cycles can vary greatly. Use specific timeframes relevant to the category.

Mistake #4: Ignoring Cultural Differences and Sensitivities

As mentioned in our post 3 Things to Consider When Asking Sensitive Questions respondents in different regions can be sensitive about certain topics related to income, sexual orientation, religion, personal care, and consumption/use of certain products and services.

Examples:

#1: What is your annual household income?

• Below $45,000
• $45,000 or more

Issue: While simplifying income levels can help encourage respondents to answer unless it is a screener question, income, and other potentially sensitive questions should include "prefer not to answer" as a choice.

#2: Which of the following beverages have you consumed in the last 6 months?

• Beer
• Coffee
• Liquor
• Softdrink
• Tea
• Water
• Wine
• Other

Issue: In the majority of Muslim countries, including questions about alcohol consumption may offend certain respondents. Especially in global studies, be sure to utilize local resources to ensure the questionnaire is culturally appropriate.

Mistake #5: Combining Questions/Attributes

As mentioned previously, it is important that the respondent clearly understands the question they are answering or the attribute they are rating. Make sure each question focuses on one attribute/behavior.

Example:

#1: Which delivery service do you consider the fastest and most economical?

Issue: While the overall goal may be to determine the fastest and most economical service, breaking these features into two questions will provide clarity to both the respondent and provide more detail for the researcher during analysis.

Mistake #6: Too many open-ended questions

Open ends can be revealing, but often they are used when response options have not been well-thought-out or as a substitute for qualitative examination. Too many open-ends results in faster respondent fatigue, which means higher drop-out rates and/or less consideration when answering. Both issues compromise data quality. If real unstructured data is the goal, consider a different methodology such as in-person qualitative or an online community.

Mistake #7: Not Considering the Respondent Experience

As Jun Uematsu, Chief Operating Officer at dataSpring, mentions in the blog Understanding Questionnaire Design in Japan, market research participation rates are declining significantly, in part due to poor respondent experience: long surveys, confusing questions, multiple attribute matrix, and multiple open-ends try the patience of even the most dedicated respondent.

As Jun recommends, "Now the recommended maximum length of interview (LOI) is 10 minutes, which is keeping with today's busy consumer. In addition, limit matrix/grid questions and minimize open-ends, which causes drop-out and irregular answers. Also avoid unnecessary screening questions, which can be tedious for the respondent and also contribute to higher drop-outs."

In Closing

While not an exhaustive list, avoiding these seven questionnaire mistakes will improve all aspects of your study, from sampling/screening to respondent experience to data quality.

For more information on how to design your questionnaires properly, check out our market research survey essentials, or download our online survey terms basics brochure.