Voices: the CMB Blog

As a market researcher, a big part of my job consists of designing questionnaires and figuring out what is the best way to ask a particular question. This is not always easy. The discussions of whether or not to use an open ended question, a scale, a yes/no, or a multiple response can take a long time, and elicit a lot of debate… we’re research nerds, after all!

One trick that I’ve learned over time is, whenever I get stuck with these types of issues, I just make up hypothetical results for the different types of questions I am considering, and then I assess whether or not the data would answer my client’s business objective.

Hard to visualize? Let me give you a fun example: A couple of months ago, @CrumbsBakeShop opened a few blocks away from our office. After repeated visits, I decided I needed to keep track of the different ones I tried so I’d be able to recommend to friends and coworkers which ones to try first (talk about brand advocacy!).  Before I knew it, the Crumbs Cupcake Rating Spreadsheet was born.  

Here’s how it went down:  

1.     The Design

Because my mantra is simplicity, my first reaction was to create a simple “Diego approved” ranking: A thumbs up or thumbs down whether or not I liked a cupcake:

Attempt #1: Good/Bad rating:

 

 

 

 

But thinking of the end results, I could picture a chart where all of the cupcakes are “Diego approved” (these are really good cupcakes!!!!)

 

 

 

 

So that would not be helpful to achieve my objective of being able to help others pick the best cupcake(s) of the bunch.  

Then I went on with attempt #2: Rankings.

 

 

 

 

This method also had (fatal) flaws. First, rankings do not allow for ties (what if two cupcakes  are equally kickass?) and more importantly, rankings do not show the relative distance between places. So let’s say Vanilla is first and Pistachio is second… how much MORE delicious is vanilla?

 

Finally, there were also limitations with data collection: I would not be tasting the 30+ cupcakes all on the same day (I am not THAT big of a pig), so it would be difficult to place a cupcake in a ranking system after testing 20+  others (e.g., Is this “Artie Lange” better or worse than #19? Or is it an 18^th place?).

Attempt #3 led me to a 0-10 rating scale:

 

 

 

Which seemed pretty reasonable… until I visualized the results:

 

 

 

 

 

 

Again, all of my cupcakes would be in the 8-10 scale, since crumbs is so much better than any other cupcake bakery in town. So… how to make it more useful? Change the frame of reference! 

 

 

 

Sweet! A scale that should work:

 

 

 

 

2.     Data collection

Now came the fun part… cupcake tasting! As I went on my quest to taste every crumbs cupcake, I realized I needed to solve (new) data collection issues:

 

1)      Taste bias: As much as I consider myself a cupcake expert (over 10 years of enthusiastic experience), I realized my opinion is not the ONLY one that matters. So I enrolled three other friends (#team crumbs: @jenisgolden, @skearney21, @caitdailey), so we could all provide rankings and “correct” for outliers.

• However, to avoid sampling bias (e.g., the oven was not working well that day) we ALL have to try the same cupcake on the same day (we split it in 4). This also has the added benefit that we can taste 2 different flavors per day (and aren’t we all looking for shorter data collection times?)

2)     Time of day bias: We decided to try the cupcakes at the same time of the day (late afternoon) to avoid misrepresentation (e.g., a blueberry cupcake would be much better rated in the morning than in the afternoon).

3)     Group think bias: Nobody is supposed to share their opinion until everyone has tasted and made up their minds about the cupcake (although I’ll admit I am known for premature disapproval gestures…) 

3.     Data Reporting

For analyzing and sharing the results, my first thought was to show rating averages:  

 

 

 

 

 

 

 

 

 

But I quickly realized this was not enough: Averages hide the data distribution. So for example: Is “Blackout” a 0 because 2 people LOVED it (gave 5s) and 2 people hated it (gave -5s?)… Or was it just an average cupcake for everyone (all 0s) […the latter is actually true]. So, to complement my average rankings, we reported average AND top 3 scores.

 

Ta-da!  

As a final touch, team crumbs members were encouraged to add a comment to explain their rankings, which adds a qualitative measure to our system. Here are some of our favorite quotes:

“As a chocolate lover, there wasn't much more to ask from a cupcake. The vanilla custard in the filling and cheesecake frosting helped avoid a chocolate overload.” –@skearney21 on ‘blackout’.  

“Elvis would be proud.” –@dejota3 on ‘Elvis’.

“The combo of coconut and pineapple goes great in an adult beverage and holds true with cupcakes as well.” @caitdailey on ‘Pineapple Coconut’  

“It's a vanilla cupcake with a bad looking, jolly rancher tasting frosting.” --@dejota3 on ‘Watermelon’

“This was very costco muffin-ish. That being said I did enjoy the coffee cake-esque topping.” -@jenisgolden on ‘Peach Cobbler’  

“Liked the consistency of the blueberry but could have used a little more. I think Diego's cutting ruined it for me…” @caitdailey on ‘Blueberry swirl’   

So what do you think? Do you see any more flaws with my system? (despite the fact that apparently I have too much time in my hands and an unhealthy obsession with cupcakes). Let me know!  

Happy tasting!  

@dejota3 is Diego Jimenez, Diego is a senior project manager for the tech and telecom practice and still has a long way to go before he tries all of the cupcakes.  

@andresita is a creative genius and designed the illustrations for this blog post. She thinks she would be a terrible #teamcrumbs judge due to her obsession with—and bias towards—chocolate.

           

The recent NY Times’ article about people's exhaustion with customer satisfaction surveys made the CMB email rounds in record time. The gist of the article is that people are tired of being asked to fill out customer satisfaction surveys before, after, and during every minor transaction they have with a company. They especially don’t like being pressured by a pleading cashier to “please help me out” and take a survey.

As a company with specialties in customer experience and loyalty—analysis that is dependent on customer feedback—the piece hit a nerve. And the takeaways from the article are worth discussing in a bit more detail:

Surveys that are too long exhaust and annoy people

On the surface there’s not much news here, asking screen upon screen of questions is going to negatively impact more than just response rates, it makes people cranky. The article mentions an excellently titled blog post Infant Who Begins Babies “R” Us Customer Satisfaction Survey Dies of Old Age from a woman who wrote about her displeasure at being asked to complete a questionnaire of more than 45 questions. And it certainly matches up with our findings that ideal survey length is 3 to 6 minutes—any shorter and you may not be giving the respondent enough time to answer, too much longer and you risk making them feel like you don’t value their time.

To incent or not to incent...

that is the question. The answer is a bit more complicated than the Times article suggests. The article quotes Vanderbilt University Management Professor Robert Oliver: “The frequent requests to fill out these surveys, especially with no incentives, have been so annoying that people just stop doing it.” And while it’s true that a subset of survey takers are motivated primarily by monetary incentives or free gifts, the top motivation for more people is a desire to improve the company, its products or services. Saying thank you and letting people know that the collected information will improve service is an effective incentive in itself.

Just because you can, doesn’t mean you should

As the article says, the availability of online surveys, free apps, and mobile platforms, means anyone can administer a survey pretty much anywhere. This is problematic, not only because good questionnaire design and data analysis requires specialized skills, but because a constant barrage of questions can feel like harassment.

I love surveys and I will take one anytime I can, but even I don’t like the feeling of being pressured by cashiers, or forced to answer questions before I complete a transaction. There is a difference between asking people for their time to help improve the company and browbeating them into providing data.  A moratorium on forcing customers to answer questions before their card goes through at the register would be an excellent start.

Make a plan before you survey

One of the problems we see, that the Times article doesn’t cover, is the distressing fact that two-thirds of companies report problems with managers not knowing what to do with the data.  Many companies are collecting data with no idea how to apply it or improve the customer experience. Without a focused plan for addressing and applying customer feedback, companies are just going through the motions, and that's a loss for customers and companies alike.

 

Learn more about our strategic approach to customer satisfaction surveys and best practices in our free download: Putting the Customer First in Customer Satisfaction Surveys

 

 

 

 

Posted by Megan McManaman. Megan is CMB's Content Marketing Manager. She is having a lot of fun with the 1940 Census data and recently realized her dream of taking the American Community Survey.

Last week, 14 CMBers headed down to sunny Orlando for The Market Research Event (TMRE). Back in the office, I asked my fellow attendees which parts of the conference stood out to them. As expected, Jim’s presentation with GE Care Credit, and Rich’s session with Aflac were fan favorites (and these were votes from people other than Jim and Rich!). But over and over again the same talk was highlighted—Sheena Iyengar’s keynote on the nature of choice based on her 2010 book The Art of Choosing. Dr. Iyengar’s presentation was my favorite as well; so much so, that I bought the book, guarded it from jealous colleagues, and read it in two days.

Iyengar, a psychology professor at Columbia, explores the nature of choice through a social-psychological lens, and by conducting rigorous and thoughtful academic research— refreshing in a time when fluffy pop-sociology books rule the book lists. She contends choice is at once  central to self-identity, and capable of overwhelming us when they we are unable to effectively process them.

Though we live in an age where abundance is worshipped, and Target and Costco thrive, those of us who’ve found ourselves staring open mouthed at the cereal aisle for too long understand the paralysis that can accompany too much choice. In particular, it can be debilitating to be faced with an array of choices we are not readily able to distinguish from one another. Iyengar, who is blind, told a great story about asking two women in a nail salon to help her choose a color by describing the difference between the virtually indistinguishable Adore-a-ball and Ballet Slippers light pink nail polish. Her point is we’re often drawn to the places that offer us the greatest choice  (would you go to a salon that only offered 5 colors?), but if we’re not experts, we don’t know how to choose without a way to articulate what makes one product different from another.

Iyengar’s famous ”jam experiment” (you may have heard about it, even if you don’t know her name) illustrates this point nicely. In the mid-90’s as a Stanford grad student she set up two tables with samples of jam at a California supermarket. One table featured just six choices, while the other offered nearly 30. She found that shoppers who visited the table with fewer samples were significantly more likely to buy the jam, as compared with those who’d sample the greater selection. What was more striking was that those who were offered fewer choices also reported greater satisfaction with their purchase as well.

What does this mean for market researchers, for companies, and for those of us standing around perplexed in the grocery store? Iyengar does propose “7” as the optimal number of choices a person can handle, but concedes choosing is more complicated than a “magical” number — it’s informed by context and culture. Perfect number or not, she argues that limiting choices to what can truly be differentiated will make for more satisfied and effective choosers. Iyengar makes a good case that focusing on improving the process of choosing is far more effective tool than just increasing the number of options.

Posted by Megan McManaman. Megan is CMB's Content Marketing Manager and would happily go to a nail salon with just 5 nail polish colors. Follow her on Twitter at @Megz79.

“You would make great rail meat.”

If you’d asked me this April, I would have guessed that “rail meat” had something to do with a train and maybe an unlucky cow. But as sailors know rail meat  is “a sailing term used for people who use their weight to stop a boat from capsizing while racing.”

This summer a friend introduced me to sailing and gave me the opportunity to be rail meat in a of couple races.  I learned that sailing is both chaotic and loud, and it was during the endless shouting of commands (where the only word I really understood was “duck!”) that I came to realize that sailing is all about communication.  The teams that exceled were the ones that found a way to clearly communicate with each other.  In business as in sailing, breakdowns in communication can spell disaster.  In the world of market research, how we communicate our questions to respondents can have a real impact on the results we get.

Here are a few tips to avoid capsizing a project.

Know your audience.  I remember the very first time I was allowed to steer the boat during a fun outing and was told to “tack.”  My random wheel spinning elicited the question “what are you doing?” to which I honestly responded “I have no idea!”  When writing questionnaires, it is important to write for your respondents.  If they are not industry insiders, then don’t use those terms.  Remember, you will always get data back, but it may not be an answer to the question you posed unless it's clearly worded.

Keep it simple.  Use the simplest terminology possible when you are writing the questionnaire.  Then have someone outside your core project team read it to see how they interpret questions.  You might be amazed at the kinds of words people interpret differently.  For instance, if I ask you to define “marketing” I bet I would get many different interpretations back even though most readers of this blog are marketers.

Stay Organized:  A poorly organized and worded questionnaire can be frustrating at any length. Asking respondents to complete questions organized by topic is logical. Begin the questionnaire with simple questions, and move to the more complex. This increases the comfort of the respondent and the likelihood they’ll stay with you until the last question.

Like staying upright on a sailboat, developing questionnaires can be hard work but committing yourself to communicating clearly in your questionnaire will help keep your project sailing smoothly.

Kirsten Rasmuson is a Senior Project Manager at CMB. Next summer she looks forward to a promotion from rail meat to crew member.  

Last week, my family and I enjoyed a trip to Orlando.  And with two girls (age 4 and 5), of course we visited some theme parks. While there I had the good fortune of being asked to complete a customer feedback interview at not one, but TWO, of the parks. Good fortune? To be stopped at an amusement park on a hot day with tired kids? Definitly. As someone who focuses on customer feedback research, I look forward to every opportunity to learn more about how people experience the process and  how companies apply the results to improve performance.

And just as I hoped, it was very enlightening. I could write a hundred blogs on the interview experience, but the one thing I want to focus on here is the timing of the interviews. Timing is a frequently discussed and debated topic among market researchers, and I want to add a more personal twist. As I mentioned above, the folks at each park intercepted me on premises, but at Park 1, the interviewer asked me a couple qualifying questions and collected my email address: I received the email about a week later, and completed the lengthy questionnaire online. 

At Park 2, the staff member asked me to complete an online interview at a computer in an office on-site.  So, this amusement park was getting my “immediate” reactions to the questions.  Which was better?  Well, it depends.  Really, the two experiences made me think of some great research, books, and ideas occurring in the field of human emotions and behavioral economics.  Daniel Kahneman is always a great resource in this area, and a popular TED video describes the two instances very well.

 “Using examples from vacations to colonoscopies, Nobel laureate and founder of behavioral economics Daniel Kahneman reveals how our "experiencing selves" and our "remembering selves" perceive happiness differently. This new insight has profound implications for economics, public policy -- and our own self-awareness]

David McRaney gives a nice summary of the video’s theme on his blog

“The psychologist Daniel Kahneman has much to say on this topic.He says the self which makes decisions in your life is usually the remembering one. It drags your current (experiencing – sic) self around in pursuit of new memories, anticipating them based on old memories.

The current self has little control over your future. It can only control a few actions like moving your hand away from a hot stove or putting one foot in front of the other. Occasionally, it prompts you to eat cheeseburgers, or watch a horror movie, or play a video game.

The current self is happy experiencing things. It likes to be in the flow.

It is the remembering self which has made all the big decisions. It is happy when you can sit back and reflect on your life up to this point and feel content. It is happy when you tell people stories about the things you have seen and done.”

Kahneman’s delineation between the “Experiencing Self” and the “Remembering Self” really resonated in the two customer feedback studies I described.  To put it in terms of Kahneman’s theory: at Park 1 (off-site survey), when I was asked a few preliminary questions and later sent an email invitation, I evaluated the visit from my “Remembering Self.”  At Park 2 (on-site interview), when I was asked to evaluate the visit while still experiencing my park visit, I evaluated the visit from my “Experiencing Self.”

This has big implications for the data and information the parks will gain from the feedback.  The evaluation from my Remembering Self is closer to my decision frame of mind; it gives a better read on the aspects of the visit that lead to my choice to select/return to the park.  For the evaluation of Park 1, I had already viewed pictures of my girls enjoying themselves and begun concluding whether I would want to return again in the future. 

Of course, I could not recall many specific feelings or problems during the visit, yet the questionnaire (one week later) asked about a wide range of things, from cleanliness to security (which presents a big disconnect between the Experiencing and Remembering selves, as feelings of security/fear in-the-moment quickly dissipate).  Sure, we had a chatty restaurant server looking to up-sell us on every dish – a big annoyance as I work hard to remember every moment of the visit – but if Park 1 (off-site survey) is looking for problems to fix, it will not find them (beyond the glaring items). 

Therefore, we shouldn’t dismiss the timing of the interview at Park 2 (on-site interview).  In fact, Kahneman’s example of pain experienced during a colonoscopy is not that much different from what I experienced at that park.  For instance, a long wait for lunch at a restaurant was quite frustrating, especially with two hungry children, and I was very open about the frustration when completing the questionnaire onsite.  Park 2 would not have received such open comments if I hadn’t given them “in the moment.” 

On the other hand, I was also less glowing in my overall satisfaction ratings, saying I was less likely to return.  I was hot, tired, and worried about my kids melting down. The interesting thing about it is this: I would be more likely to return to Park 2, where the interview was on-site.  Now that my Remembering Self has reflected on the experiences – and had the “fog of battle” clear from my head – I realize that my family gained a lot more cherished memories from Park 2, and I would be far more likely to return compared to the other park.

Therefore, if the purpose of the interview is to understand the experiences, memories, and drivers of choice, it’s critical to time the interview for my “Remembering Self” to respond.  If the purpose is to find specific points of pain or joy (regardless of their role on choice), then it’s critical to time the interview for my “Experiencing Self” to respond.

Posted by Jeff McKenna who will be chairing the Action Planning track and leading discussions around the getting the most out of your voice of the customer program at the Total Customer Experience conference October 3-5. 

Are you planning on going to Total Customer Experience? CMB is an event sponsor. Feel free to use the code: TCEL11CMB when you register for a discounted price. We hope to see you there.