The posts here represent the opinions of CMB employees and guests—not necessarily the company as a whole. 

Subscribe to Email Updates


see all

Question Abuse: The Lost Art of Market Research Battery Development

Posted by Cathy Harrison on Thu, Feb 04, 2010

We all agree that lengthy questionnaires contribute to respondent fatigue, which negatively impacts data quality.  One of the worst contributors to this problem are long batteries of attitudinal statements to be rated on a 10- or 11-point scale.  

This is especially prevalent in trackers, where batteries often grow exponentially each year, like a snowball rolling downhill.  Every stakeholder feels they must add to the list to ‘improve' it  in an attempt to measure some shade of gray of what is already there.  Respondents wonder, "Didn't I just answer that?" or worse, have their eyes glaze over and think about what they should eat for dinner.

The answer is not to eliminate these batteries altogether, as they can serve as powerful input to insightful advanced analytics, but to streamline and strengthen them.  Putting a priority on tightening your attitudinal battery means you (and your clients) will be rewarded with a more succinct questionnaire that better measures the most important factors involved in customer decision making.  You will also save valuable questionnaire time and keep respondents engaged.

One way to accomplish these improvements is through pretesting - a sorely neglected task due to time and budgetary constraints (it always come down to that!). 

Through my experience on the client and supplier sides, I have come up with six additional steps to help build compact, highly effective question batteries.  While it's difficult to implement them all every time, even adding one or two elements to your research plan can greatly improve your research findings.

1.      Freshen your thinking with an ethnography - Remove yourself from assuming why behaviors are happening.  Use exploratory research to collect attitudes and behaviors independent from one another as input to creating variables for quantitative testing. 

2.      Create variables based on these observations and gather both variables from past research that had a significant impact on results and relevant variables from syndicated research you (or your agency) have access to.

3.      Qualitatively test all the variables for clarity and validity.  Do respondents understand the statement?  Are you measuring what you think you are trying to measure?

4.      Run short quantitative surveys to test the variables.  Conduct both Factor Analysis (which statements are measuring the same underlying factor?) and Latent Class Analysis (which statements best differentiate?) on the results.

5.      Evaluate and prioritize the statements based on how strongly they load on and are exclusive to the factor.  Give priority to variables from syndicated research in order to maximize the actionability of your battery.

6.      Pare down your statements to the best variables for measuring all or the most important factors by eliminating redundant or poorly performing variables.


Using Segmentation to Understand the Small Business

Contrary to some people's beliefs, not all small businesses are the same regarding health insurance needs, buying processes, and hot buttons.

Watch the full Webinar.


Posted by Cathy Harrison.  Cathy is a client services executive at CMB, loves social media, music, and kick-butt research.  You can follow Cathy on Twitter at @virtualMR

Topics: research design