Voices: the CMB Blog

Dear Dr. Jay,

We’ve been testing new concepts for years. The magic score to move forward in the new product development process is a 40% top 2 box score to purchase intent on a 5 point scale. How do I know if 40% is still a good benchmark? Are there any other measures that might be useful in predicting success?

-Normatively Challenged

Dear Norm,

I have some good news—you may have a big data mining challenge. Situations like yours are why I always ask our clients two questions: (1) what do you already know about this problem, and (2) what information do you have in-house that might shed some light on a solution? You say you’ve been testing concepts for years.  Do you have a database of concepts already set up? If not, can you easily get access to your concept scores?

Look back on all of the concepts you have ever tested, and try to understand what makes for a successful idea. In addition to all the traditional concept test measures like purchase intent, believability, and uniqueness, you can also append marketing spend, distribution measures, and perhaps even social media trend data. You might even want to include economic condition information like the rate of inflation, the prime rate of interest, and the average DOW stock index. While many of these appended variables might be outside of your control, they may serve to help you understand what might happen if you launch a new product under various market conditions.

Take heart Norm, you are most definitely not alone. In fact, I recently attended a presentation on Big Data hosted by the Association of Management Consulting Firms. There, Steve Sashihara, CEO of Princeton Consultants, suggested there are four key stages for integrating big data into practice. The first stage is to monitor the market. At CMB, we typically rely on dashboards to show what is happening. The second stage is to analyze the data. Are you improving, getting worse, or just holding your own? However, only going this far with the data doesn’t really provide any insight into what to do. To take it to the next level, you need enter the third stage: building predictive models that forecast what might happen if you make changes to any of the factors that impact the results. The true value to your organization is really in the fourth stage of the process—recommending action. The tools that build models have become increasingly powerful in the past few years. The computing power now permits you to model millions of combinations to determine the optimal outcomes from all possible executions.

In my experience, there are usually many attributes that can be improved to optimize your key performance measure. In modeling, you’re looking for the attributes with the largest impact and the cost associated with implementing those changes to your offer. It’s possible that the second best improvement plan might only cost a small percentage of the best option. If you’re in the business of providing cellular device coverage, why build more towers if fixing your customer service would improve your retention almost as much?

Got a burning research question? You can send your questions to DearDrJay@cmbinfo.com or submit anonymously here.

Dr. Jay Weiner is CMB’s senior methodologist and VP of Advanced Analytics. Jay earned his Ph.D. in Marketing/Research from the University of Texas at Arlington and regularly publishes and presents on topics, including conjoint, choice, and pricing.

I once again had the pleasure of attending the CASRO Digital Research Conference this year. It’s the one of the best conferences available to data collection geeks like me, and this year’s presentations did not disappoint. Here are a few key takeaways from this year’s conference.

1. The South shuts down when it snows. After having a great weekend in Nashville after the conference, my flight was cancelled on Monday due to about an inch of snow and a little ice. Needless to say, I was happy to return to Boston and its nine feet of snow.

2. “Big data” is an antiquated term. Over the past few years, big data has been the big buzz in the industry. Much like we said goodbye to traditional “market research,” we can now say adios to “big data.” Good riddance. The term was vague at best. However, that doesn’t mean that the concept is going away. It’s simply being replaced by new, more meaningful terminology like “integrated data” and “multi-sourced data.” But one thing isn’t changing. . .

3. Researchers still don’t know what to do with all that data. What can I say about multi-sourced data that I haven’t already said many times over the past couple years? Clients still want it, and researchers still want to oblige. But this fact remains: adequate tools still do not exist to deliver meaningful integrated data in most cases. We have a long way to go before most researchers will be able to leverage all of this data to its full potential in a meaningful way for our clients.

4. There’s a lot more to mobile research than how a questionnaire looks on a screen. For the past three or four years, it seems like every year is going to be “the year of mobile” at these types of conferences. Because of this, I always attend the mobile-related sessions skeptically. When we talk about mobile, more often than not, the main concern is how the questionnaire will look on a mobile device. But mobile research is much more than that. One of the best things I heard at the conference this year was that researchers should “follow the humans.” This is true on so many levels. Of course, a person can respond to a questionnaire invitation on his/her mobile device, but so much of a person’s daily life, including behaviors and attitudes, is shaped by mobile. Welcome to the world of the ultra-informed consumer. I can confidently say that 2015 is most definitely the year of mobile! (I do, however, reserve the right to say the same thing again next year.)

5. Researchers need to think like humans. It’s easy to get caught up in percentages in our world, and researchers sometimes lose sight of the human aspect of our industry. We like to think that millionaire CEOs are constantly checking their emails on their desktop computers, waiting for their next “opportunity” to take a 45-minute online questionnaire for a twenty-five cent reward. I attended sessions at the conference about gamification, how to make questionnaires more user-friendly, and also how to make questionnaires more kid-friendly by adding voice-to-text and text-to-voice options. All of these things have the potential to ease the burden on research participants, and as an industry, this must happen. We have a long way to go, but. . .

6. Now is the time to play catch-up with the rest of the world. Last year, I ended my recap by saying that change is happening faster than ever. I still think that’s true about the world we live in. With all of the technological advances and new opportunities provided to us, it’s an exciting time to be alive. However, I’m not sure I can honestly say that change is happening faster than ever when it comes to the world of research. I’ve been a part of this industry for a very fulfilling seventeen years, and sometimes my pride in the industry clouds my thinking. Let’s face the facts. The truth is that, as an industry, we are lagging far behind as the world speeds by. Research techniques and tools are evolving at a very slow pace, and I don’t see this changing in the near future. (In our defense, this is true for many industries and not only market research.) I still believe that those of us who are working to leverage the changing world we live in will be much better equipped for success than those who sit idly and watch the world fly.

I’m still confident that my industry is primed and ready for significant and meaningful change—even if we sometimes take the path of a tortoise. As a weekend pitmaster, I know that low and slow is sometimes the best approach. The end result is what really counts.

Jared is CMB’s Field Services Director, and has been in market research industry for seventeen years. When he isn’t enjoying the exciting world of data collection, he can be found competing at barbecue contests as the pitmaster of the cooking team Insane Swine BBQ.

Last week, we spent a few days networking with and learning from some of the industry’s best and brightest at The Quirk's Event. At the end of the day, a few key ideas stuck out to us, and we wanted to share them with you. 1. Insights need to be actionable: This point may seem obvious, but multiple presenters at the conference hammered in on this point. Corporate researchers are shifting from a primarily separate entity to a more consultative role within the organization, so they need to deliver insights that best answer business decisions (vs. passing along a 200 slide data-dump). This mindset should flow through the entire lifespan of a project—starting at the beginning by crafting a questionnaire that truly speaks to the business decisions that need to be made (and cuts out all the fluff that may be “nice to have” but is not actionable) all the way to thoughtful analysis and reporting. Taking this approach will help ensure final deliverables aren’t left collecting dust and are instead used to lead engagement across the organization. 

2. Allocate time and resources to socializing these insights throughout the organization: All too often, insightful findings are left sitting on a shelf when they have potential to be useful across an organization. Several presenters shared creative approaches to socializing the data so that it lives long after the project ended. From transforming a conference room with life-size cut-outs of key customer segments to creating an app employees can use to access data points quickly and on-the-go, researchers and their partners are getting creative within how they share the findings. The most effective researchers think about research results as a product to be marketed to their stakeholders.
 
3. Leverage customer data to help validate primary research: Most organizations have a plethora of data to work with, ranging from internal customer databases to secondary sources to primary research. These various sources can be leveraged to paint a full picture of the consumer (and help to validate findings). Etsy (a peer-to-peer e-commerce site) talked about comparing data collected from its customer database to its own primary research to see if what buyers and sellers said they did on the site aligned with what they actually did. For Etsy, past self-reported behaviors (e.g., number of purchases, number of times someone “favorites” a shop, etc.) aligned strongly with its internal database, but future behavior (e.g., likelihood to buy from Etsy in the future) did not. Future behaviors might not be something we can easily predict by asking directly in a survey, but that data could be helpful as another way to identify customer loyalty or advocacy. A note of caution: if you plan on doing this data comparison, make sure the wording in your questionnaire aligns with what you plan on matching in your existing database. This ensures you’re getting an apples to apples comparison.
 
4. Be cautious when comparing cross-country data: A multi-country study is typically going to ask for a “global overview” or cross-country comparison, but this can lead to inaccurate recommendations. Most are aware of cultural biases such as extreme response (e.g., Brazilian respondents often rate higher on rating scales while Japanese respondents tend to rate lower) or acquiescence (e.g., China often has the propensity to want to please the interviewer), and these biases should be kept in the back of your mind when delving into the final data. Comparing scaled data directly between countries with very different rating tendencies could lead to to falsely thinking one country is underperforming. A better indication of performance would be to provide an in-country comparison to competitors or looking at in-country trending data.
 
5. Remember your results are only as useful as your design is solid: A large number of stakeholders invested in a study’s outcome can lead to a project designed by committee since each stakeholder will inevitably have different needs, perspectives, and even vocabularies. A presenter shared an example from a study that asked recent mothers, “How long was your baby in the hospital?” Some respondents thought the question referred to the baby’s length, so they answered in inches. Others thought the question referred to the baby’s duration in the hospital, so they answered in days. Therein lies the problem.  Throughout the process, it’s our job to ensure that all of the feedback and input from multiple stakeholders adheres to the fundamentals of good questionnaire design: clarity, answerable, ease, and lack of bias.

Have you been to any great conferences lately and have insights to share? Tell us in the comments!

Jen is a Project Manager on the Tech practice who always has the intention to make a purchase on Etsy but never actually pulls the trigger.  

Ashley is a Project Manager on the FIH/RTE practice who has pulled the trigger on several Etsy items (as evidenced in multiple “vintage” tchotchkes and half-complete craft projects around her home).

Goodby, Silverstein & Partners’ Kelli Robertson talked with CMB’s Research Director, Tara Lasker, about a recent messaging study they partnered on for Cisco. This study aimed to determine the best way to communicate Cisco’s role in the “Internet of Everything.” 

TARA: There’s been a lot of buzz lately about using data to support strategic thinking. Can you talk a little bit about how you strike that balance between the two in your role?

KELLI: Well, I don’t think data just supports thinking—I think it also generates it. There’s nothing more exciting than a table full of data and going through that data to find ideas and the story. I think that’s one of the things we did with this study. I think you always have to start with hypotheses and use the data gathered to prove or disprove them, which is what we did. You also have to be open to the data giving you new ideas. For us, data isn’t just about validating—it’s about learning.

It’s also important to realize that data helps bring consensus. Marketing is hard today because everything is so uncertain, and I think it’s easy for clients to dismiss things you learn from eight or even thirty qualitative interviews. It’s a lot harder to dismiss data. So if you can combine the data with the new ideas, you’re more likely to create consensus and generate buy-in from the people you’re working with.

TARA:  That’s definitely true, and we see that throughout many of our client engagements. Moving on to our study, can you talk about how GSP and CMB partnered to help solve some of the challenges that Cisco faced?

KELLI: The first thing that CMB did really well was to quickly grasp the topic. This includes how technology influences business, the somewhat complicated concept of the “Internet of Everything,” and all of the product and technology solutions that create the “Internet of Everything.” There wasn’t a lot of explaining that I had to do because CMB just jumped in. I think that’s a testament to all of your experience with clients in the technology industry. You also recognized that the “Internet of Everything” might be a complicated concept for respondents to grasp, so you helped us craft a few different ways to talk about it in the survey, which allowed us to better measure true awareness and understanding.

Here’s another example. This was a global study, and CMB had a lot of recommendations including using max diff scaling to prioritize messages and alleviate any global scale bias. These recommendations allowed us to overcome a challenge that I wouldn’t have even known about if it hadn’t been for you. You also recommended that we test a few diagnostics within the top scoring messages. That helped us gain a better understanding of why messages were compelling instead of just showing us which ones were at the top of the list. Those diagnostics helped us feel confident in the messages that stood out.

TARA: We did a lot of secondary research on our end and asked colleagues at CMB with the most tech experience about the “Internet of Everything.” We tried to think from a respondent’s perspective when answering the questions to make sure that we were getting the most useful data we could possibly get and to ensure the respondents were reacting the way we wanted without misunderstanding.

KELLI: I think that background research you’re referring to was what allowed you to help us so much. I live in the “Internet of Everything” world. I have for the past two years. You allowed us to go deep into the “Internet of Everything,” but kept in mind the fact that people won’t view it with the same amount of understanding that we do. That helped us ask questions in a more broad sense and allowed us to have good juxtapositions regarding innovation, business, and technology.

TARA: Exactly. We also looked at the different roles within an organization and how they saw it. For example, the C-suite and technical decision makers understood and liked the more detailed messaging while business managers liked the broader, softer messaging. Speaking of, can you talk about what impact this research has had on Cisco’s brand messaging strategy? What’s happened since we’ve presented the results?

KELLI: Well, as you know, Cisco keeps coming back to get more data, and the study is really being adopted. It helped us form the messaging strategy for Cisco moving forward. For example, it helped us craft the right language to explain how Cisco is making the “Internet of Everything” possible. There’s been this question in the marketplace: what does Cisco do to make the “Internet of Everything” happen? The study helped us answer that question and address the skepticism our audience has had in the most compelling way.

The study also helped us define a sweet spot within our target audience. Prior to this, we talked broadly about C-suite executives, business decision-makers, and technical decision-makers. We summarize our audience as C-suite executives, but the study uncovered a very clear mindset that matched Cisco’s aspirations. Now we’re able to use that data to talk about our audience psychographically. We’ve found an attitudinal sweet spot because of the confidence in the data. Without the study, we could guess that C-suite executives and business decision-makers felt a certain way, but the data is invaluable in changing the way we think about who we reach out to, how we influence them, and the attitude Cisco needs to have. That’s been really invaluable, and it influences a lot of our decisions in tone and placement media.

The study also helped validate some of the Cisco product solutions that we should prioritize in our messaging. In the past, Cisco was primarily a networking company. Now, Cisco is offering a suite of product solutions way beyond networking. This study helped us uncover which of those product solutions triggered the most thoughts of innovation in our audience’s mind, which helped us prioritize where we should focus our product efforts.

TARA: Let’s talk a little more about the buy-in. This is the second time we’ve worked together on a project like this, and we’ve always had a great partnership. You understand your client and the questions they need answered, and we work through the research design and analysis. Ultimately, the goal is to get buy-in and adoption. So, can you talk about the adoption throughout Cisco?

KELLI: We’ve presented this countless times at Cisco, and we’re still getting requests to present it. We also just presented all of the work to the global regions in Cisco to help inform their work. They use a lot of the work we do, but they also do a lot of work on their own, so I’m sharing it with them so that they can use it to help inform what they do. Certain people within the organization are even using the data in their day-to-day work, which is amazing.

One of the things I’ve been most excited about is that we’re working with the thought leadership team at Cisco, who help set the agenda and public relations initiatives around key themes and topics. They’ve spent a lot of time pouring through the results, and they ended up coming back with a huge list of questions that are going to drive their thinking for the next year. So it’s helping set thought leadership, which is great.

One of the biggest things we tested is Cisco’s mission statement—“Changing the way we work, live, play, learn.” That is a statement that has always been on paper, and it has always been referred to as Cisco’s mission statement. The data we got back showed how compelling this statement was to our audience. It came back as one of the top messages if not the top message. I think that’s been giving Cisco a lot of confidence that they need to do more with their mission statement and that it needs to become not just words on paper, but something that drives all action within Cisco. I think this study is going to breathe new life into this big, bold mission statement and give them the courage to use it more overtly to make bolder decisions. There’s a difference between having a mission statement and being on a mission, and I feel like this data gave them the confidence to be a company on a mission—on a mission to change the way we work, live, play, learn.

TARA: Over the years, you’ve been one of my favorite clients for several reasons—one of them being that you really approach the relationship like a true partnership. We really work together. We get to a place where you know the client, challenges, political environment, and research questions that need to be answered. CMB brings research expertise, which allows us to design the study in a way that is going to answer your questions, so you don’t have to worry about the technicalities. I feel like both times we’ve partnered, we’ve ended up in a good, clear place at the end because of the way we work together throughout the process.

KELLI: I agree, and I will say that who we chose wasn’t necessarily my decision. I worked with the head of our research group. When we were going through RFPs, it became clear that few research companies are so thorough. There’s just this reality that not a lot of other research companies are as strategic, bring the breadth of experience, dive in, and ask questions of other experts in the organization the way you do….and these were things we noticed from the first RFP. There’s just something special you have bottled over there.

TARA: Thanks, Kelli! Hopefully we’ll get the chance to work together again in the future.

Tara Lasker is a Research Director at CMB and Kelli is a Group Brand Strategy Director at GSP. They both enjoy good beer, good music, commiserating over the trials and tribulations of motherhood, and telling a great story with primary research data.  

Data has existed in the modern business world for a long time (think manila folders in file cabinets in every office on every floor). Digitized data has been around for a while now, too (think virtual folders in hard drives connected to seemingly bottomless computer networks). So why, in just the past few years, have all of us become so excited about and actually engaged in data? We even decided to give it a new name—“big” data. Where did all this excitement come from? Why is it happening? If you asked Tom Breur, Cengage Learning’s VP of Analytics who spoke about big data at NEMRA’s Spring into Action event earlier this year, he would tell you that it’s because there has been a recent surge in data volume (mostly thanks to the emergence of machine-generated data and machine-to-machine communication). This surge led to an ever-expanding data surplus—a surplus that would not have had a home if it weren’t for subsequent innovations in the type of software that manages huge amounts of data and the innovations that led to much more efficient data warehousing capabilities.Initially, large companies were the only ones who had any sort of big data capability (credit scores and fraud protection are two early examples), and until recently these companies were the only ones to leverage those capabilities to play the big data game when it came to predicting their customers’ behavior. But in their July-August issue, Inc. Magazine featured an article detailing how smaller companies are now allowed to play as well, thanks to decreasing technology costs and increasing user-friendliness of big data software.

All of this begs the question: will companies, big and small, no longer need market researchers? After all, big data solutions allow companies to learn about their customers and make more informed business decisions, and let’s not forget that the newest big data solutions are so user-friendly that companies can do all the consumer insights themselves. However, I don’t think market researchers will be replaced anytime soon. Big data may be able to tell you the “what,” but it can’t tell you the “why.”

Enter the story of the widely-covered 2013 Google Flu Trends “Epidemic.” By running algorithms based on flu-related Google searches and searchers’ locations, Google Flu Trends had been historically accurate in predicting how much of the U.S. population had the flu. However, in 2013, it inaccurately predicted the number. In fact, it predicted twice the number reported by the Centers for Disease Control and Prevention! How did this happen? The widespread media coverage of the severe flu season in the U.S. spread like a virus throughout social media, which led to an increase in flu-related Google searches. Many of these searches were from people who thought they might have the flu—“I’m sniffling! I’m sneezing!”—but didn’t. Since Google Flu Trends didn’t consider the context and wasn’t able to ask Googlers why they were Googling flu-like symptoms, it thought 11% of the U.S. population had the flu when the actual number was closer to 6%.

Mark Hansen of Columbia University summed it up best when he said, “Data is not a magic force in society; it’s an extension of us.” Can you believe it? Big data is actually quite human. It tells a story about people because it comes from people, and it’s simply a new medium through which people are telling stories about themselves. It’s like collaborative storytelling. Remember those stories that your teachers would have you start and then make other kids add to? It’s similar, but with a simple twist: big data is collaborative non-fiction. But the authors are still people, which brings it back to market researchers. As market researchers, we not only ask people questions about how they feel or what they do, but we also ask why. We’re able to apply the context that, as evidenced by the Google Flu Trends Epidemic, big data is not able to accomplish alone.

Even though we’re not being replaced, we still have to adapt. For example, there is a great opportunity in synthesizing what we do with the data our research partners have in-house. By combining our knowledge of the “why” with a research partner’s “what,” we can identify the error in what would have otherwise been our research partner’s version of the Google Flu Trends Epidemic if they had not been appropriately focused on why the data looked the way it did. For a company attempting to adjust its product offerings, this could be the difference between abandoning its most loyal customers and maintaining those loyal customers by keeping them happy, all while successfully gaining new customers in the process.

The number of success stories that result from combining the best of both worlds—the what and the why—seems to be ever-expanding. Here at CMB, we have had the pleasure of co-authoring a few of those success stories. For market research, big data is a good thing and worth adapting for. Company by company, the market research industry should adapt in order to set itself up not only for survival, but also for leadership in the next century of consumer insights so we can continue to play the role of co-author in a story that has only just begun.

Join us at The Market Research Event in October! Use the code CMB2014 and receive 25% off your registration.