As a market researcher, a big part of my job consists of designing questionnaires and figuring out what is the best way to ask a particular question. This is not always easy. The discussions of whether or not to use an open ended question, a scale, a yes/no, or a multiple response can take a long time, and elicit a lot of debate… we’re research nerds, after all!
One trick that I’ve learned over time is, whenever I get stuck with these types of issues, I just make up hypothetical results for the different types of questions I am considering, and then I assess whether or not the data would answer my client’s business objective.
Hard to visualize? Let me give you a fun example: A couple of months ago, @CrumbsBakeShop opened a few blocks away from our office. After repeated visits, I decided I needed to keep track of the different ones I tried so I’d be able to recommend to friends and coworkers which ones to try first (talk about brand advocacy!). Before I knew it, the Crumbs Cupcake Rating Spreadsheet was born.
Here’s how it went down:
1. The Design
Because my mantra is simplicity, my first reaction was to create a simple “Diego approved” ranking: A thumbs up or thumbs down whether or not I liked a cupcake:
Attempt #1: Good/Bad rating:
But thinking of the end results, I could picture a chart where all of the cupcakes are “Diego approved” (these are really good cupcakes!!!!)
So that would not be helpful to achieve my objective of being able to help others pick the best cupcake(s) of the bunch.
Then I went on with attempt #2: Rankings.
This method also had (fatal) flaws. First, rankings do not allow for ties (what if two cupcakes are equally kickass?) and more importantly, rankings do not show the relative distance between places. So let’s say Vanilla is first and Pistachio is second… how much MORE delicious is vanilla?
Finally, there were also limitations with data collection: I would not be tasting the 30+ cupcakes all on the same day (I am not THAT big of a pig), so it would be difficult to place a cupcake in a ranking system after testing 20+ others (e.g., Is this “Artie Lange” better or worse than #19? Or is it an 18th place?).
Attempt #3 led me to a 0-10 rating scale:
Which seemed pretty reasonable… until I visualized the results:
Again, all of my cupcakes would be in the 8-10 scale, since crumbs is so much better than any other cupcake bakery in town. So… how to make it more useful? Change the frame of reference!
Sweet! A scale that should work:
2. Data collection
Now came the fun part… cupcake tasting! As I went on my quest to taste every crumbs cupcake, I realized I needed to solve (new) data collection issues:
1) Taste bias: As much as I consider myself a cupcake expert (over 10 years of enthusiastic experience), I realized my opinion is not the ONLY one that matters. So I enrolled three other friends (#team crumbs: @jenisgolden, @skearney21, @caitdailey), so we could all provide rankings and “correct” for outliers.
-
However, to avoid sampling bias (e.g., the oven was not working well that day) we ALL have to try the same cupcake on the same day (we split it in 4). This also has the added benefit that we can taste 2 different flavors per day (and aren’t we all looking for shorter data collection times?)
2) Time of day bias: We decided to try the cupcakes at the same time of the day (late afternoon) to avoid misrepresentation (e.g., a blueberry cupcake would be much better rated in the morning than in the afternoon).
3) Group think bias: Nobody is supposed to share their opinion until everyone has tasted and made up their minds about the cupcake (although I’ll admit I am known for premature disapproval gestures…)
3. Data Reporting
For analyzing and sharing the results, my first thought was to show rating averages:
But I quickly realized this was not enough: Averages hide the data distribution. So for example: Is “Blackout” a 0 because 2 people LOVED it (gave 5s) and 2 people hated it (gave -5s?)… Or was it just an average cupcake for everyone (all 0s) […the latter is actually true]. So, to complement my average rankings, we reported average AND top 3 scores.
Ta-da!
As a final touch, team crumbs members were encouraged to add a comment to explain their rankings, which adds a qualitative measure to our system. Here are some of our favorite quotes:
“As a chocolate lover, there wasn't much more to ask from a cupcake. The vanilla custard in the filling and cheesecake frosting helped avoid a chocolate overload.” –@skearney21 on ‘blackout’.
“Elvis would be proud.” –@dejota3 on ‘Elvis’.
“The combo of coconut and pineapple goes great in an adult beverage and holds true with cupcakes as well.” @caitdailey on ‘Pineapple Coconut’
“It's a vanilla cupcake with a bad looking, jolly rancher tasting frosting.” --@dejota3 on ‘Watermelon’
“This was very costco muffin-ish. That being said I did enjoy the coffee cake-esque topping.” -@jenisgolden on ‘Peach Cobbler’
“Liked the consistency of the blueberry but could have used a little more. I think Diego's cutting ruined it for me…” @caitdailey on ‘Blueberry swirl’
So what do you think? Do you see any more flaws with my system? (despite the fact that apparently I have too much time in my hands and an unhealthy obsession with cupcakes). Let me know!
Happy tasting!
@dejota3 is Diego Jimenez, Diego is a senior project manager for the tech and telecom practice and still has a long way to go before he tries all of the cupcakes.
@andresita is a creative genius and designed the illustrations for this blog post. She thinks she would be a terrible #teamcrumbs judge due to her obsession with—and bias towards—chocolate.