Widening Relative Error in Television Diary Samples [REPORT]

Local broadcast and cable ratings increasingly are falling outside the traditional 10% benchmark error range that has served as the mainstay used in advertising negotiations. This is according to new study findings which show random error in Nielsen diaries continues to grow.

Increased movement from phone-based sampling to address-based sampling has improved television audience-sample frames.  However, as evidenced by differential response rates in sampled homes, non-response bias cannot be completely resolved by weighting measures and remains a persistent challenge.

This is among key conclusions of a newly completed two-year study by the Council for Research Excellence (CRE), a diverse group of senior-level research professionals from throughout the media and advertising industries dedicated to advancing the knowledge and practice of audience measurement methodology.  The study was conducted in three markets employing various audience-measurement methods. 

The objectives were to evaluate sample quality in address-based vs. phone-based audience samples; to better understand response bias in address-based samples; and ultimately to help improve diary-based TV-audience measurement and in turn improve ratings quality.  

“Nielsen’s address based sampling has reduced response bias but there remains a significant difference between metered and diary samples,” said Ceril Shagrin, executive vice president of Univision Communications, who serves as chair of the CRE as well as its Sample Quality Committee.  “An improved sample frame improves representation of younger, cell phone-only homes — which are growing rapidly — but more needs to be done. Measures such as enlarging the sample size or weighting the results don’t adequately resolve the issue; you can weight demographically, but we’ve learned more clearly that you can’t sufficiently weight for non-response.”

The three television markets included in the study were Dallas-Fort Worth, the fifth-largest TV market and a Nielsen local people-meter (LPM) market; Albuquerque-Santa Fe, the 47th-largest TV market and a standard Nielsen-meter market; and Paducah, Kentucky-Cape Girardeau, Missouri-Harrisburg, Illinois — Nielsen market # 81 and a diary market.  The markets were selected due to varying measurement methodology; varying market size; and divergent characteristics, such as geographical coverage, ethnic make-up, number of over-the-air households and penetration levels of electronic devices.

Respondents in the study sample tended to be older.  In Dallas, for example, 54.7% of sample participants (or “in-tabs”) were over 50 compared to just 28.2% for heads of households within the 35-54-year-old age range and just 17.2% of those in the under-35 age range.  Non-participants (“non-intabs”) outnumbered participants in the 35-to-54 and under-35 categories.  The patterns were similar in the smaller markets.

In addition to the differences between in-tab and non-intab homes, the study uncovered substantial information about household consumer-electronics ownership, for example:

• 38.2% of all the households surveyed did not have a landline phone and had at least one cell phone;
• The portion of adult heads-of-households aged 25-to-29 who use only a cell phone has grown from 46% in June 2009 to 62% in December 2012;
• A 27-household sample of non-TV homes found 63% had a computer with high-speed Internet and 41% use the Internet on a cell phone; 38% viewed TV on a desktop PC, laptop or tablet; and of the 61% who still watched television on a TV set, nearly half watched in someone else’s home, followed distantly by bars and restaurants.

This marks the second major CRE study of response bias.  Data analysis from a 2009 study, “Measuring the Unmeasured Viewer,” revealed more about unmeasured viewers than any prior effort. 

Data collected from the new study, conducted with the assistance of RTI International, is being compared with Nielsen LPM, metered and diary market data in an effort to determine, among other details, the impact on response bias of weighting, geographic and demographic variables, and ownership of a traditional TV set as well as of a land-line phone or cell phone-only.  

The study, involving the CRE’s Sample Quality, Local Measurement, and Media-related Universe Estimates Committees, also marks the CRE’s second examination of ownership of media-related devices in Dallas-Fort Worth.  Data analysis from a 2009 Universe Estimates Committee study of the market revealed significant differences from the Nielsen sample in ownership of HD sets and DVD players.  

“While these latest findings may be intuitive, even unsurprising, it’s important that we now have hard information,” Shagrin said.  “Both of our studies, for example, found that samples were missing young people and Hispanics; our new study shows the differentials are not as wide, but they still constitute a significant gap.  We needed to understand whether deeper information about media-related equipment ownership can be obtained from diary samples.   And, even though we had only a very small sample, we now have a better idea of how much ‘TV program’ viewing is done – and on what devices — in what are currently defined as ‘non-TV homes.’  This helped confirm for us that people who had not previously been measured view TV differently – and that it’s important that this viewing be measured.  We need to reach harder-to-reach households.”