Voices: the CMB Blog

Hi Dr. Jay –

The city of Boston is trying develop one key measure to help officials track and report how well the city is doing. We’d like to do that in house. How would we go about it?

-Olivia

Hi Olivia,

This is the perfect tie in for big data and the key performance index (KPI). Senior management doesn’t really have time to pour through tables of numbers to see how things are going. What they want is a nice barometer that can be used to summarize overall performance. So, how might one take data from each business unit and aggregate them into a composite score?

We begin the process by understanding all the measures we have. Once we have assembled all of the potential inputs to our key measure, we need to develop a weighting system to aggregate them into one measure. This is often the challenge when working with internal data. We need some key business metric to use as the dependent variable, and these data are often missing in the database.

For example, I might have sales by product by customer and maybe even total revenue. Companies often assume that the top revenue clients are the bread and butter for the company. But what if your number one account uses way more corporate resources than any other account? If you’re one of the lucky service companies, you probably charge hours to specific accounts and can easily determine the total cost of servicing each client. If you sell a tangible product, that may be more challenging. Instead of sales by product or total revenue, your business decision metric should be the total cost of doing business with the client or the net profit for each client. It’s unlikely that you capture this data, so let’s figure out how to compute it. Gross profit is easy (net sales – cost of goods sold), but what about other costs like sales calls, customer service calls, and product returns? Look at other internal databases and pull information on how many times your sales reps visited in person or called over the phone, and get an average cost for each of these activities. Then, you can subtract those costs from the gross profit number. Okay, that was an easy one.

Let’s look at the city of Boston case for a little more challenging exercise. What types of information is the city using? According to the article you referenced, the city hopes to “corral their data on issues like crime, housing for veterans and Wi-Fi availability and turn them into a single numerical score intended to reflect the city’s overall performance.” So, how do you do that? Let’s consider that some of these things have both income and expense implications. For example, as crime rates go up, the attractiveness of the city drops and it loses residents (income and property tax revenues drop). Adding to the lost revenue, the city has the added cost of providing public safety services. If you add up the net gains/losses from each measure, you would have a possible weighting matrix to aggregate all of the measures into a single score. This allows the mayor to quickly assess changes in how well the city is doing on an ongoing basis. The weights can be used by the resource planners to assess where future investments will offer the greatest pay back.

 Dr. Jay is fascinated by all things data. Your data, our data, he doesn’t care what the source. The more data, the happier he is.

Every year, Survey Magazine names 10 “Data Dominators,” who are conquering data in different ways at their companies. This year, our very own Talia Fein was chosen. She discusses her passion for data in Survey Magazine’s August issue, and we’ve reposted the article below.

When I first came to CMB, a research and strategy company in Boston, I was fresh out of undergrad and an SPSS virgin. In fact, I remember there being an SPSS test that all new hires were supposed to take, but I couldn’t take it because I didn’t even know how to open a data file. Fast forward a few months, and I had quickly been converted to an SPSS specialist, a numbers nerd, or—perhaps more appropriately—a data dominator.  I was a stickler for process and precision in all data matters, and I took great pride in ensuring that all data and analyses were perfect and pristine. To put it bluntly, I was a total nerd.

I recently returned to CMB after a four-year hiatus. When I left CMB, I quickly became the survey and data expert among my new colleagues and the point person for all SPSS and data questions. But it wasn’t just my data skills that were being put to use. To me, data management is also about the process and the organization of data. In my subsequent roles, I found myself looking to improve the data processes and streamline the systems used for survey data. I brought new software programs to my companies and taught my teams how to manage data effectively and efficiently.

When I think about the future of the research industry, I imagine survey research as being the foundation of a house.  Survey data and data management are the building blocks of what we do. When we do them excellently, we are a well-oiled machine. But a well-oiled machine doesn’t sell products or help our clients drive growth. We need to have the foundation in place in order to extend beyond it and to prepare ourselves for the next big thing that comes along. And that next big thing, in my mind, is big data technology. There is a lot of data out there, and a lot of ways of managing and analyzing it, and we need to be ready for that.  We need to expand our ideas about where our data is coming from and what we can do with it. It is our job to connect these data sources and to find greater meaning than we were previously able to. It is this non-traditional use of data and analytics that is the future of our industry, and we have to be nimble and creative in order to best serve our clients’ ever-evolving needs.

One recent example of this is CMB’s 2015 Mobile Wallet study, which leveraged multiple data sources and—in the process—revealed which were good for what types of questions. In the case of this research, we analyzed mobile behavioral data, including mobile app and mobile web usage, along with survey-based data to get a full picture of consumers’ behaviors, experiences, and attitudes toward mobile wallets. We also came away with new Best Practices for how best to manage passive mobile behavioral data, as it presents new challenges that are unique from managing survey data. Our clients are making big bets on new technology, and they need the comprehensive insights that come from integrating multiple sources. We specifically sampled different sources because we know that—in practice—many of our clients are being handed multiple data sets from multiple data sources. In order to best serve these clients, we need to be able to leverage all the data sources that are at our and their disposal so that we can glean the best insights and make the best recommendations.

Talia Fein is a Project & Data Manager at Chadwick Martin Bailey (CMB), a market research consulting firm in Boston. She’s responsible for the design and execution of market research studies for Fortune 500 companies as well as the data processing and analysis through all phases of the research. Her portfolio includes clients such as Dell, Intel, and Comcast, and her work includes customer segmentation, loyalty, brand tracking, new product development, and win-loss research.

Dear Dr. Jay,

How can I explain the value of data integration to my CMO and other non-research folks?

- Jeff B. 

Hi Jeff,

Years ago, at a former employer that will remain unnamed, we used to entertain ourselves by playing Buzzword Bingo in meetings. We’d create Bingo cards with 30 or so words that management like to use (“actionable,” for instance). You’d be surprised how fast you could fill a card. If you have attended a conference in the past few years, you know we as market researchers have plenty of new words to play with. Think: big data, integrated data, passive data collection, etc. What do all these new buzzwords really mean to the research community? It boils down to this: we potentially have more data to analyze, and the data might come from multiple sources.

If you only collect primary survey data, then you typically only worry about sample reliability, measurement error, construct validity, and non-response bias. However, with multiple sources of data, we need to worry about all of that plus level of aggregation, impact of missing data, and the accuracy of the data. When we typically get a database of information to append to survey data, we often don’t question the contents of that file. . . but maybe we should.

A client recently sent me a file with more than 100,000 records (ding ding, “big data”). Included in the file were survey data from a number of ad hoc studies conducted over the past two years as well as customer behavioral data (ding ding, “passive data”). And, it was all in one file (ding ding, “integrated data”). BINGO!

I was excited to get this file for a couple of reasons. One, I love to play with really big data sets, and two, I was able to start playing right away. Most of the time, clients send me a bunch of files, and I have to do the integration/merging myself. Because this file was already integrated, I didn’t need to worry about having unique and matching record identifiers in each file.

Why would a client have already integrated these data? Well, if you can add variables to your database and append attitudinal measures, you can improve the value of the modeling you can do. For example, let’s say that I have a Dunkin’ Donuts (DD) rewards card, and every weekday, I stop by a DD close to my office and pick up a large coffee and an apple fritter. I’ve been doing this for quite some time, so the database modelers feel fairly confident that they can compute my lifetime value from this pattern of transactions. However, if the coffee was cold, the fritter was stale, and the server was rude during my most recent transaction, I might decide that McDonald’s coffee is a suitable substitute and stop visiting my local DD store in favor of McDonald’s. How many days without a transaction will it take the DD algorithm to decide that my lifetime value is now $0.00? If we had the ability to append customer experience survey data to the transaction database, maybe the model could be improved to more quickly adapt. Maybe even after 5 days without a purchase, it might send a coupon in an attempt to lure me back, but I digress.

Earlier, I suggested that maybe we should question the contents of the database. When the client sent me the file of 100,000 records, I’m pretty sure that was most (if not all) of the records that had both survey and behavioral measures. Considering the client has millions of account holders, that’s actually a sparse amount of data. Here’s another thing to consider: how well do the two data sources line up in time? Even if 100% of my customer records included overall satisfaction with my company, these data may not be as useful as you might think. For example, overall satisfaction in 2010 and behavior in 2015 may not produce a good model. What if some of the behavioral measures were missing values? If a customer recently signed up for an account, then his/her 90-day behavioral data elements won’t get populated for some time. This means that I would need to either remove these respondents from my file or build unique models for new customers.

The good news is that there is almost always some value to be gained in doing these sorts of analysis. As long as we’re cognizant of the quality of our data, we should be safe in applying the insights.

Got a burning market research question?

OR 

Dr. Jay Weiner is CMB’s senior methodologist and VP of Advanced Analytics. Jay earned his Ph.D. in Marketing/Research from the University of Texas at Arlington and regularly publishes and presents on topics, including conjoint, choice, and pricing.