WELCOME TO OUR BLOG!

The posts here represent the opinions of CMB employees and guests—not necessarily the company as a whole. 

Subscribe to Email Updates

Liz White

Recent Posts

MR Insights from the 2016 Election: A Love Letter to FiveThirtyEight

Posted by Liz White

Thu, Nov 03, 2016

social-icons-01.png

Methodology matters.  Perhaps this much is obvious, but as the demand for market researchers to deliver better insights yesterday increases, dialing up the pressure to limit planning time, it’s worth re-emphasizing the impact of research approach on outcomes.  Over the past year, I’ve come across numerous reminders of this while following this election cycle and the excellent coverage over at Nate Silver’s FiveThirtyEight.com.  I’m not particularly politically engaged, and as the long, painful campaign has worn on I’ve become even less so; but, I keep coming back to FiveThirtyEight, not because of the politics, but because so much of the commentary is relevant to market research.  I rarely ever visit the site (particularly the ‘chats’) without coming across an idea that inspires me or makes me more thoughtful about the research I’m doing day-to-day, and generally speaking that idea centers on methodology.  Here are a few examples:

Probabilistic Screening

Prrobabilistic screening example.png

In my day-to-day work, I would guesstimate that 90-95% of the studies I see are intended to capture a more specific population of interest than the general population, making the screening criteria used to identify members of that population absolutely vital. In general, these criteria consist of a series of questions (e.g., are you a decision maker, do you meet certain age and income qualifications, have you purchased something in X category before, would you consider using brand Y), with only those with the right pattern or patterns of responses getting through.

But what if there were a better way to do this? Reading the above on FiveThirtyEight got me thinking about the kinds of studies in which using a probabilistic screener (and weighting the data accordingly) might actually be better than what we do now. These would be studies where the following is true:

  1. Our population of interest might or might not engage in the behavior of interest
  2. We have some kind of prior data on the behavior of interest tied to individual characteristics

“Yeah right,” you might say, “like we ever have robust enough data available on the exact behavior we’re interested in.” Well, this might be a perfect opportunity for incorporating the (to all appearances) ever-increasing amounts of passive customer data that are available into our surveys. It’s inspiring, at any rate, to think about how a more nuanced screener might make our research more predictive.

Social Desirability Bias & More Creative Questioning

Prrobabilistic screening example.png

Prrobabilistic screening example.png

Social desirability is very much a market research-101 topic, but that doesn’t mean it’s something that’s either been definitively solved for or that the same solution would work in every case. The issue comes up a lot, not only in the context of respondent attitudes, but even more commonly when asking about demographics like income or age. There are lots of available solutions, some of which involve manipulating the data to ‘normalize’ it in some way, and some of which involve creative questioning like the example shown above. I think the right takeaways from the above are:

  • Coming up with creative variations on your typical questions might help avoid respondent bias, and even has the potential to make questions more engaging for respondents
  • It’s important to think critically about whether or not creative questioning will resonate appropriately with your respondents
Plus, brainstorming alternatives is fun! For example:
  • Is someone you respect voting for Donald Trump?
  • Do the blogs you prefer to read tend to favor Trump or Clinton?
  • What media outlets do you visit to get your political news?

The Vital Importance of Context

Prrobabilistic screening example.png

At the heart of FiveThirtyEight’s commentary here is a reminder of the vital importance of context. It’s all very well to push respondents through a series of scales and return means or top box frequencies; but depending on the situation, that may tell only a small part of the story. What does an average rating of ‘6.5’ really tell you? In the end, without proper context, this kind of result has very little inherent meaning.

So how do we establish context? Some options (all of which rely on prior planning) include:

  • Indexing (against past performance or competitors)
  • Trade-off techniques (MaxDiff, DCM)
  • Predictive modeling against an outcome variable

Wrapping this up, there are two takeaways that I’d like to leave you with:

  • First, methodology matters. It’s worthwhile to spend the time to be thoughtful and creative in your market research approach.
  • Second, if you aren’t already, head over to FiveThirtyEight and read their entire backlog of 2016 election coverage. The site is an incredible reservoir of market research insight, and I can say with 95% confidence that you’ll be happy you checked it out.

 Liz White is a member of CMB’s Advanced Analytics team, and checks FiveThirtyEight.com five times a day (plus or minus two times).

Topics: methodology, Market research

Happy WoW-loween: World of Warcraft Gets Player Delight Right

Posted by Liz White

Thu, Nov 05, 2015

 world of warcraft, segmentation, customer experience

We all have our own way of celebrating the fall season. For some, it’s apple-picking, leaf-peeping, or downing mug after mug of Pumpkin Spice Lattes. For me, the defining event of the fall happens not in Boston, but in Azeroth at the World of Warcraft’s (WoW) annual celebration of Hallow’s End. Held every year, this two-week, in-game holiday is both a great example of effective seasonal marketing and a demonstration of Blizzard Entertainment’s nuanced understanding of its customer base. Not to mention, it’s just plain fun.

Hallow’s End was introduced to WoW in 2005, and in the past ten years, it’s grown dramatically in scope and popularity. Although Blizzard hosts other in-game seasonal celebrations (Pilgrim’s Bounty, Feast of the Winter Veil, and Brewfest are just a few), Hallow’s End seems to attract more notice both in and out of the game than any of the others. 

world of warcraft, segmentation, customer experience

Why all the excitement?  The success of Hallow’s End is due in large part to the fact that it offers something for every kind of player. Who are they, and what do they get out of Hallow’s End? Here’s a sampling:

  • Mount Collectors: Hallow’s End heralds the return of the Headless Horseman, a formidable raid boss with a sweet ride. The Horseman’s steed, an undead horse with glowing green eyes and hooves, is one of the most coveted mounts in WoW, and it’s only available for players to win during this event. Those who grab one will gleefully parade their prize for the rest of the year, and those who don’t are doomed to count down the days until the Horseman’s return. 

world of warcraft, segmentation, customer experience

  • World Travelers: For many (myself included) the most compelling feature of WoW is the massive scale and breathtaking beauty of the game’s world. Sadly, high-level adventurers have little incentive to explore low-level areas. During Hallow’s End, however, Candy Buckets appear in inns throughout Azeroth, offering in-game currency and achievements for players who seek them out. The Candy Bucket hunt is a great excuse to revisit old haunts and to seek out some new ones. 
  • Pet Battlers: Pet Battling is relatively new to WoW, but it’s become quite popular. Pets are small creatures or constructs that a player accumulates over time (ranging from the common Brown Rabbit to the exotic Anubisath Idol). Like WoW characters, pets can be leveled to acquire new abilities and then pitted against one another in gruesome fights to the death. Hallow’s End provides the opportunity for players to add seven new pets to their arsenal, including several creepy crawlies as well as a feline familiar who wears a witches’ hat and rides on a broom. Deadly and adorable! 
  • Duelists & Jokesters: In addition to its various quests and collectables, Hallow’s End creates a communal space for players, who gather to celebrate in front of the flaming Wickerman (see him below in one of my own screenshots!). It’s unusual to have so many players assembled at once, and this combined with the holiday mood tends to lead to player dueling. For those who love to duel, Hallow’s End is a perfect opportunity. WoW also encourages player-on-player action during the holiday by offering holiday themed wands that can be used to transform other players into bats, ghosts, skeletons, and even (gasp!) humans.

world of warcraft, segmentation, customer experience

And that’s not all! In addition to the above, other Hallow’s End offerings include raid-quality equipment (for dungeon delvers), garrison decorations (for garrison builders), and experience bonuses (for those leveling up).  Regardless of why and how you play, the holiday has something for you.

While World of Warcraft has had its ups and downs, it’s indisputably one of the most well-known and well-loved games. One reason is that Blizzard not only allows, but promotes and celebrates, a wide range of play styles during Hallow’s End and beyond. Ask yourself, does your business offer products or services intended for a broad customer base? Do you understand who they are, what they like, and what makes them different from one another? CMB can help! Contact us to talk segmentation and product development, and we’ll help you add firepower to your own arsenal. 

Happy Hallow’s End!

Liz White (BadDecision) is a level 100 Blood Elf Warrior, who loves blacksmithing, long flights over Azeroth, and running advanced analytics for CMB. Give her a shout either IRL or in-game, and she’ll be happy to help you optimize your build.

world of warcraft, segmentation, customer experience

Topics: customer experience and loyalty, market strategy and segmentation, digital media and entertainment research

Conjoint Analysis: 3 Common Pitfalls and How to Avoid Them

Posted by Liz White

Thu, Jan 08, 2015

conjoint analysis, cmbIf you work in marketing or market research, chances are you’re becoming more and more familiar with conjoint analysis: a powerful research technique used to predict customer decision-making relative to a product or service. We love conjoint analysis at CMB, and it’s easy to see why. When conducted well, a conjoint study provides results that make researchers, marketers, and executives happy. These results:

  • Are statistically robust
  • Are flexible and realistic
  • Describe complex decision-making
  • Are easy to explain and understand

If you need a quick introduction or a refresher on conjoint analysis, I recommend Sawtooth Software’s excellent video, which can be found here.For these reasons conjoint analysis is one of the premiere tools in our analytical toolkit. However, as with any analytical approach, conjoint analysis should be applied thoughtfully to realize maximum benefits. Below, I describe three of the most common pitfalls related to conjoint analysis and tips on how to avoid them.

Pitfall #1: Rushing the Design

This is the most common pitfall, but it’s also be the easiest one to avoid. As anyone who has conducted a conjoint study knows, coming up with the right design takes time. When planning the schedule for a conjoint analysis study, make sure to leave time for the following steps:

  • Identify your business objective, and work to identify the research questions (and conjoint design) that will best address that objective.
  • Brainstorm a full list of product features that you’d like to test. Collaborate with coworkers from various areas of your organization—including marketing, sales, pricing, and engineering as well as the final decision-makers—to make sure your list is comprehensive and up-to-date.
    • You may also want to plan for qualitative research (e.g., focus groups) at this stage, particularly if you’re looking to test new products or product features. Qualitative research can prioritize what features to test and help to translate “product-speak” into language that customers find clear and meaningful.
    • If you’re looking to model customer choices among a set of competitive products, collect information about your competitors’ products and pricing.
    • Once all the information above is collected, budget time to translate your list of product features into a conjoint design. While conjoint analysis can handle complex product configurations, there’s often work to be done to ensure the final design (a) captures the features you want to measure, (b) will return statistically meaningful results, and (c) won’t be overly long or confusing for respondents.
    • Finally, budget time to review the final design. Have you captured everything you needed to capture?  Will this make sense to your customers and/or prospective customers? If not, you may need to go back and update the design. Make sure you’ve budgeted for this as well.

Pitfall #2: Overusing Prohibitions

Most conjoint studies typically involve a conversation about prohibitions—rules about what features can be shown under certain circumstances. For example:

Say Brand X’s products currently come in red, blue, and black colors while Brand Y’s products are only available in blue and black. When creating a conjoint design around these products, you might create a rule that if the brand is X, the product could be any of the three colors, but if the brand is Y, the product cannot be red.

While it’s tempting to add prohibitions to your design to make the options shown to respondents more closely resemble the options available in the market, overusing prohibitions can have two big negative effects:

  1. Loss of precision when estimating the value of different features for respondents.
  2. Loss of flexibility for market simulations.

The first of these effects can typically be identified in the design phase and fixed by reducing the number of prohibitions included in a model. The second is potentially more damaging as it usually becomes an issue after the research has already been conducted. For example:

We’ve conducted the research above for Brand Y, including the prohibition that if the brand is Y, the product cannot be red. Looking at the results, it becomes clear that Brand X’s red product is much preferred over their blue and black products. The VP of Brand Y would like to know what the impact of offering a Brand Y product in red would be.  Unfortunately, because we did not test a red Brand Y product, we are unable to use our conjoint data to answer the VP’s question.

In general, it is best to be extremely conservative about using prohibitions—use them sparingly and avoid them where possible. 

Pitfall #3: Not Taking Advantage of the Simulator

While the first two pitfalls are focused on conjoint design, the final pitfall is about the application of conjoint results. Once the data from the conjoint analysis has been analyzed, it can be used to stimulate virtually any combination of the features tested and predict the impact that different combinations will have on customer decision-making. . .which is just one of the reasons conjoint analysis is such a valuable tool. All of that predictive power can be distilled into a conjoint simulator that anyone—from researchers to marketers to C-suite executives—can use and interpret.

At CMB, the clients I’ve seen benefit most from conjoint analysis are the clients that take full advantage of the simulators we deliver, rather than simply relying on the scenarios created for reporting. Once you receive a conjoint simulator, I recommend the following:

  1. Distribute copies of the simulator to all key stakeholders.
  2. Have the simulator available when presenting the results of your study, and budget time in the meeting to run “what-if” scenarios then and there. This can allow you to leverage the knowledge in the room in real time, potentially leading to practical and informed conclusions.
  3. Continue to use your simulator to support decision-making even after the study is complete, using new information to inform the simulations you run. A well-designed conjoint study will continue to have value long after your project closes.

Liz is a member of the Analytics Team at CMB, and she can’t wait to hear your research questions!

Topics: advanced analytics, research design