Poynter has just launched a new course on polls and surveys in partnership with three international opinion research organizations. Thanks to the support of the American Association for Public Opinion Research (AAPOR), the World Association for Social, Opinion and Market Research (ESOMAR) and the World Association for Public Opinion Research (WAPOR), this course is free of charge on Poynter’s online training platform, News University. Today’s Coffee Break Course is courtesy of this new teaching.
Many polls ask the same question of different samples at different times to measure change. Just as the margin of sampling error has to be taken into account when determining a difference in public opinion, it also matters when writing about a change in public opinion over time.
Determining whether a new poll shows a “real” or statistically significant change from past polls often involves complex calculations, but some simple rules apply for most comparisons:
- If a difference between two polls is smaller than either poll’s margin of sampling error, it is probably not significant and should not be classified as a “change” without asking researchers to test it or investigating further on your own.
- If a difference between two polls is greater than 1.5 times the margin of sampling error for both, it is almost always significant and can be confidently classified as a “change.”
When these two rules don’t apply, you should feel comfortable asking survey researchers who conducted the poll whether a difference in poll numbers is statistically significant — most can calculate this easily.
You can also check out this PDF from International Communications Research (ICR) that approximates whether a difference between polls is statistically significant under a host of scenarios. One trend you’ll notice is when the poll result is close to 50 percent (rather than 90 or 10), the difference between two polls must be larger to be statistically significant. Margin of sampling error is largest — and poll differences must be higher to be statistically significant — when the poll result is close to 50 percent.
Taken from Understanding and Interpreting Polls (International), a self-directed course at Poynter NewsU, developed in partnership with the American Association for Public Opinion Research (AAPOR), the World Association for Social, Opinion and Market Research (ESOMAR) and the World Association for Public Opinion Research (WAPOR).
Have you missed a Coffee Break Course? Here’s our complete lineup.
