Jeffrey Henning (@jhenning), Kathryn Korostoff (@ResearchRocks) and I (@VirtualMR) are participating in the complimentary AMA MRC Virtual Event: Unveiling Marketing Research's Future Online on June 23 (hope you can join us!). Our session, "Tweet Off! Three MR Tweeps Bicker, Badger & Bust out of 140 Characters", will involve debates on the prickliest topics in market research.
As a lead-in to the discussion, I am serving as the verdict judge for an early debate between Kathryn and Jeffrey on the subject of self-reported respondent data.
Kathryn's point: "Self-reported information is not perfect. But it is less perfect in some cases than in others."
Jeffrey's counterpoint: "Respondents, as a group, have sufficient ability to self-report to provide valuable data for market researchers."
No one would argue that self-reported data is perfect and any knowledgeable researcher would agree that some self-reported measures are better than others.
In her argument, Kathryn makes several good points about the well-known biases of self-reported data, such as over-reporting, impacts to post-behaviors, and social desirability (supported by research including Social Desirability Bias and the Validity of Indirect Questioning by Robert J. Fisher © 1993 Journal of Consumer Research Inc.)
Still, there are biases in every methodology. And as researchers, one of our roles is to identify, control, and consider these biases in designing and evaluating market research studies.
For example, behavioral/observational research beyond descriptive observational variables (just reporting the behavior that is observed) requires researchers to make inferences and evaluations, which introduces biases. Or in the case of using sales data, customer databases or scanner data are robust but often do not include all distribution channels.
So, instead of discarding some ‘less-than-perfect' self-reported data, there are several ways researchers can address the shortcomings of self-reported data to yield useful conclusions:
1) Review the self-reported results with a critical eye as part of a comprehensive, multi-mode research plan
2) Calibrate purchase intentions based on historical data vs. actual behavior (for example, 75% of top 3 box purchase intent actually purchase)
3) Assign relative value to Inflated self-reported expenditures (high, medium, low) for use in a segmentation or product development/launch study
My Verdict: Respondents, as a group, have a sufficient ability to self-report for market researchers to draw useful conclusions - as long as actual magnitudes and values are interpreted relatively rather than literally.