Originally posted on Research Access

Earlier this month, Chris Neal of Chadwick Martin Bailey shared with members of the New England chapter of the Marketing Research Association tips for running mobile diary studies, based on lessons learned from a recent project.For the Council for Research Excellence (CRE), CMB studied mobile video usage to understand:

• How much time is spent on mobile devices watching TV (professionally produced TV shows)?

• Does this cannibalize TV set viewing?

• What motivates consumers to watch on mobile?

• How can mobile TV viewing be accurately tracked?

The research included a quantitative phase with two online surveys and mobile journaling, followed by a series of home ethnographies. The quant work included a screening survey, the mobile diary, and a final online survey.

• The screening survey was Census balanced to estimate market size, with three groups recruited for comparison: those without mobile devices (smartphones or tablets), those with mobile devices who don’t watch TV on them, and those with mobile devices that they watch TV on. The total number of respondents was 5,886.

• The mobile diary activity asked respondents to complete their journal 4 times a day for 7 days.

• A final attitudinal survey was used to better understand motivations and behaviors associated with decisions about TV watching.

Along the way, CMB learned some valuable best practices for mobile diary studies, including tips for recruiting, incentives, design and analysis. The 10 key lessons learned:

1. Mobile panels don’t work for low incidence – Take care when using mobile panels – given the small size of many mobile panels, you may have better luck recruiting through traditional online panels, as CMB did. For this study, it was because of the comparatively low incidence of actual mobile TV watching.

2. Overrecruit – You will lose many recruits to the journaling exercise when it comes time to downloading the mobile diary application. As a general rule, over-recruit by 100% – get twice the promises of participation that you need. Most dropout occurs after the screening and before the participant has recorded a single mobile diary entry. For many members of online survey panels, journaling is a new experience. The second biggest point of dropout was after recording 1 or 2 diary entries.

3. Keep it short – To minimize this dropout, you have to keep the diary experience as short as possible: no more than 3 to 5 minutes long. The more times you ask participants to complete a diary each day, the greater the dropout rate.

4. Think small screen – Make sure the survey is designed to provide a good experience on small screens – avoid grids and sum-allocation questions and limit open-ended prompts and use of images. Use vertical scales instead of horizontal scales. “Be wary of shiny new survey objects for smartphone survey-takers,” said Chris. Smartphone users had 5 times the dropout rate of tablet or laptop users in this study. Enable people to log on to their journal from whatever device they were using at the time, including their computer.

5. Beware battery hogs – When evaluating smartphone apps, be wary of those that drain battery life by constantly logging GPS location. Check the app store reviews of the application.

6. Keep consistent – Keep the diary questionnaire the same for every time block, to get respondents into the habit of answering it.

7. Experiment with incentives to maximize participation – Tier incentives to motivate people to stick with the study and complete all time blocks. To earn the incentive for the CMB study, Chris said that respondents had to participate at least once a day for all 7 days, with additional incentives for every journal log entered (participants were reminded this didn’t have to involve actual TV watching, just filling out the log). In the end, 90% of journaling occasions were filled out.

8. Remind via SMS and email – In-app notifications are not enough to prompt participation. Use email and text messages for each time block as well. Most respondents logged on within 2 hours of receiving a reminder.

9. Use online surveys for detailed questions – Use the post-journaling survey to capture greater detail and to work around the limits of mobile surveys. You can then use these results to “slice and dice” the journal responses.

10. Weight by occasions – Remember to weight the data file to total occasions not total respondents. For missing data, leave it missing. Develop a plan detailing which occasion-based data you’re going to analyze and what respondent-level analysis you are going to do. You may need to create a separate occasion-level data file and a separate respondent-level data file.

Properly done, mobile diary studies provide an amazing depth of data. For this project, CMB captured almost 400,000 viewing occasions (mobile and non-mobile TV watching), for over 5 million occasion-based records!

Interested in the actual survey results? CRE has published the results presentation, “TV Untethered: Following the Mobile Path of TV Content” [PDF].

Jeffrey Henning, PRC is president of Researchscape International, a market research firm providing custom surveys to small businesses. He is a Director at Large on the MRA Board of Directors; in 2012, he was the inaugural winner of the MRA’s Impact award. You can follow him on Twitter @jhenning.