Data has existed in the modern business world for a long time (think manila folders in file cabinets in every office on every floor). Digitized data has been around for a while now, too (think virtual folders in hard drives connected to seemingly bottomless computer networks). So why, in just the past few years, have all of us become so excited about and actually engaged in data? We even decided to give it a new name—“big” data. Where did all this excitement come from? Why is it happening? If you asked Tom Breur, Cengage Learning’s VP of Analytics who spoke about big data at NEMRA’s Spring into Action event earlier this year, he would tell you that it’s because there has been a recent surge in data volume (mostly thanks to the emergence of machine-generated data and machine-to-machine communication). This surge led to an ever-expanding data surplus—a surplus that would not have had a home if it weren’t for subsequent innovations in the type of software that manages huge amounts of data and the innovations that led to much more efficient data warehousing capabilities.Initially, large companies were the only ones who had any sort of big data capability (credit scores and fraud protection are two early examples), and until recently these companies were the only ones to leverage those capabilities to play the big data game when it came to predicting their customers’ behavior. But in their July-August issue, Inc. Magazine featured an article detailing how smaller companies are now allowed to play as well, thanks to decreasing technology costs and increasing user-friendliness of big data software.

All of this begs the question: will companies, big and small, no longer need market researchers? After all, big data solutions allow companies to learn about their customers and make more informed business decisions, and let’s not forget that the newest big data solutions are so user-friendly that companies can do all the consumer insights themselves. However, I don’t think market researchers will be replaced anytime soon. Big data may be able to tell you the “what,” but it can’t tell you the “why.”

Enter the story of the widely-covered 2013 Google Flu Trends “Epidemic.” By running algorithms based on flu-related Google searches and searchers’ locations, Google Flu Trends had been historically accurate in predicting how much of the U.S. population had the flu. However, in 2013, it inaccurately predicted the number. In fact, it predicted twice the number reported by the Centers for Disease Control and Prevention! How did this happen? The widespread media coverage of the severe flu season in the U.S. spread like a virus throughout social media, which led to an increase in flu-related Google searches. Many of these searches were from people who thought they might have the flu—“I’m sniffling! I’m sneezing!”—but didn’t. Since Google Flu Trends didn’t consider the context and wasn’t able to ask Googlers why they were Googling flu-like symptoms, it thought 11% of the U.S. population had the flu when the actual number was closer to 6%.

Mark Hansen of Columbia University summed it up best when he said, “Data is not a magic force in society; it’s an extension of us.” Can you believe it? Big data is actually quite human. It tells a story about people because it comes from people, and it’s simply a new medium through which people are telling stories about themselves. It’s like collaborative storytelling. Remember those stories that your teachers would have you start and then make other kids add to? It’s similar, but with a simple twist: big data is collaborative non-fiction. But the authors are still people, which brings it back to market researchers. As market researchers, we not only ask people questions about how they feel or what they do, but we also ask why. We’re able to apply the context that, as evidenced by the Google Flu Trends Epidemic, big data is not able to accomplish alone.

Even though we’re not being replaced, we still have to adapt. For example, there is a great opportunity in synthesizing what we do with the data our research partners have in-house. By combining our knowledge of the “why” with a research partner’s “what,” we can identify the error in what would have otherwise been our research partner’s version of the Google Flu Trends Epidemic if they had not been appropriately focused on why the data looked the way it did. For a company attempting to adjust its product offerings, this could be the difference between abandoning its most loyal customers and maintaining those loyal customers by keeping them happy, all while successfully gaining new customers in the process.

The number of success stories that result from combining the best of both worlds—the what and the why—seems to be ever-expanding. Here at CMB, we have had the pleasure of co-authoring a few of those success stories. For market research, big data is a good thing and worth adapting for. Company by company, the market research industry should adapt in order to set itself up not only for survival, but also for leadership in the next century of consumer insights so we can continue to play the role of co-author in a story that has only just begun.

Join us at The Market Research Event in October! Use the code CMB2014 and receive 25% off your registration.