The answers to these questions have a common origin that takes us back 80 years to 1936. Commercial consumer and social research was a new business then – AC Nielsen had been founded 13 years previously (in 1936 it was about the same age as Facebook is now) and Gallup inc. was just one year old. As it turned out, events in 1936 would soon make Gallup front-page news.
Like 2016, 1936 was a general election year in the US and there were opinion polls. One organization that considered itself expert in this field was The Literary Digest, a magazine that had been in the polling business since 1916.
The Literary Digest was on a streak – it had correctly predicted the election of Presidents Wilson (1916), Harding (1920), Coolidge (1924), Hoover (1928) and Roosevelt (1932). These results were based on a simple technique – they mailed out millions of postcards and counted the results from the returns. Big Data early 20th century style! Confident in this method as a result of their impressive track record, they repeated it in 1936. They predicted the winner to be Alf Landon, the Republican Party candidate. Many readers will probably now be thinking “who?”
Meanwhile, Dr. Gallup was approaching the problem in a different way. Instead of obtaining millions of returns, his organization selected and interviewed a smaller sample of 50,000 voters, designed to represent US voters of all types. The Gallup Poll predicted that FDR would be re-elected for a second term.
Gallup was right and The Literary Digest was wrong…badly wrong, as Roosevelt won 46 of the then 48 states. The magazine never really recovered – advertisers pulled out and it closed in 1938. This cover from November 1936 says it all.
But why did it get it so wrong? The problem was fairly obvious in hindsight. Although it had millions of returns, The Literary Digest polled mailing lists comprising its own readers, registered automobile owners and telephone users. This was the Great Depression and anyone who could afford a magazine subscription, a car or even a telephone was likely to be wealthier than average and therefore more likely to vote Republican.
In short, The Literary Digest method was biased, proving that bigger isn’t always better. The Gallup Poll was designed to equally represent all types of voters and because of this, even though the sample size was much smaller, the results were much closer to the truth. Of course, with smaller sample sizes there is the possibility of sampling error (results varying by chance due to who is selected in the poll). In this case, the statistical margin of error on the 54% claiming to be FDR supporters was only around 1%.
The success of this design set the course for social and consumer research thereafter: representative samples, properly selected and projected to the population became the accepted way to provide intelligence about consumers. In the last decade, this approach has been challenged by the advent of big data sources and the lessons of 1936 have become relevant again.
And so, turning to TV audience measurement, we see a parallel with this discussion. Nielsen measures national linear TV audiences using a sample, a panel that is recruited to represent all US TV households and continuously updated to maintain its relevance. The current sample size is 35,000 homes containing about 100,000 persons. For more detail on the various elements of Nielsen’s TV measurement in the US see here.
In recent years, other insights into TV usage have become available via set-top box (STB) data and Smart TVs (the “Big Data solutions”). These insights are based on millions of boxes returning data. ComScore/Rentrak and TRA have emerged with audience measurement products using these data. And the owners of the data are creating advertising solutions that activate these data. So with millions of boxes and TVs providing data, why is Nielsen still the currency for linear TV?
There are a number of reasons, and the lesson of 1936 is an overarching reason: bigger isn’t always better. Some of the differences between the Nielsen approach and the Big Data solutions illustrate why:
The comments above are not intended to disparage Big Data solutions but to illustrate why Nielsen remains the main currency for linear TV. Additionally, Nielsen has the incumbent advantage and is also MRC accredited, giving confidence to the users of the data.
It should be noted that for smaller audiences (eg niche networks and local TV audiences), Big Data solutions can provide more stable and usable insights than Nielsen panel data, where sample size limitations can result in sparse reported ratings. The industry is indeed using these data for those applications. The best measurement of TV audiences would be an integration of all available and relevant data – a hybrid of panel data to provide total coverage and persons’ audiences and Big Data to provide more granular detail than the economics of panels allow.
So as we approach November, can we trust political polls? The answer is a cautious “yes, but be aware of their limitations”. There are many reasons why polls may not predict the actual result of an election:
Keep these limitations in mind when you hear poll results. And remember, politicians only talk about the polls that show them in a good light, while news shows and networks look for fluctuations in polls so that they can weave a story, and improve their audiences, as measured by Nielsen.