Tracking polls are very useful enterprises. I know. I used them a lot over the years, both for media, political and for commercial clients. At the same time, I am deeply suspicious of most tracking polls that are done for public consumption today because they are flat out NOT as accurate as full scale, single time telephone polling.
They have variability problems based on the way they are conducted, which makes their excessive interpretation by the media highly suspect. Moreover, the supposed opinion changes they report over interpreted by reporters who know very little about the ways the polls are conducted.
Take this story on Sunday, August 3rd, by the Associated Press – a story which relies on tracking polls and largely ignores the more accurate regular polls by Pew and CNN, reported only two days earlier and conducted I am sure, properly. The conclusions of the story are simply not correct, and terribly and I believe intentionally misleading: http://news.yahoo.com/s/ap/mccain_obama_poll. AP should be embarrassed by this story, but they will not be. Very unprofessional!
Problems With All Telephone Polling:
A short explanation of the difference between these types of polls is useful at this point. All telephone polling is today very difficult to conduct accurately. This is a simple truth that the best professional pollsters are loath to talk about candidly because such polls are their bread and butter during elections. There are marked problems with the use of the telephone for opinion polling that did not exist nearly as much twenty or thirty years ago. Here are some of them:
- The combination of “refusal rates” and “unable to reach” rates today are so high with telephone polling that the overall completion rates (percentage of people called who are actually interviewed) is generally below 20 percent , and sometimes 10 percent, if these rates are calculated fairly and reported accurately, which is often not the case. This inability to reach a high proportion of the targeted sample introduces a whole range of biases that are difficult to explain and overcome.
- The "completion rates" are also much lower today in part to avoid telemarketing; and a growing number of people are on “do not call lists” or use caller identification to avoid such calls.
- The percentage of people without land line phones and/or who use predominately their cell phones has risen sharply in recent years, making these individuals much more difficult to reach at all.
- The pressure from “paying clients” to be “cost effective” puts more pressure on the polling organizations to cheat on the standards of complying with the rigorous rules of random selection and repeated callbacks in polling.
This situation exists with standard one time telephone polling, but it is magnified with telephone tracking polls. With a telephone tracking poll, the polling organization completes approximately 400 to 600 new interviews every day, substituting the new day of polling for the equivalent number of respondents who are dropped out of the poll after three days, making the reported result a running average of the three completed days of polling. The additional problems that affect the accuracy of these types of polls are:
- In a normal telephone poll, the entire poll lasts over three days, with the second and third day used to attempt to complete those people who were unreachable on the first day of polling. These “callbacks” are critical to complete as much of the original random sample as possible. This rigorousness is almost never accomplished as well with the telephone tracking polls, a fact which polling organizations are also loath to discuss.
- Some groups of voters are much more difficult reach with weekend calling, particularly Friday night, Saturday night, and Sunday during the day) because they are away from the phone or replying entirely on their cell phones. In my experience, this is truer of African Americans, Latinos, urban voters, and younger voters, whose lifestyles do not match the more sedate (and available) lifestyles of older, more settled voters, particularly those who live in smaller, more rural communities that are more likely to be Republican.
- Given fewer callbacks and weekend variability, most tracking studies show sharp variability that is due to simple sampling variability, response bias and differential completion rates between Republicans and Democrats, and the variation has little to do with anything happening in the society as a whole.
Democrats are normally more affected by some of these issues than Republicans, because the Republican base of voters is made up more of older, more settled, and more easily reached voters. The Obama coalition, however, is even more subject to polling inaccuracy because of harder to reach respondents – particularly African Americans, Latinos, and voters under 35 who are generally much harder to reach, particularly on weekends and with cell phone use.
The problems are even more acute with the tracking polls, where these “harder to reach” issues are both more pronounced and less professionally executed, particularly on the weekends. The only way to deal with these problems is to “weight” the data to some predetermined “expected” proportions by various demographics, but that substitutes the subjective values and estimations of pollsters who have their own partisan biases substituting for scientific rigor.
The August Closure in Tracking Poll Results:
The problems are acutely present in the most recent polls. Both Gallup and Rasmussen released tracking polls on Saturday, August 2nd, (Friday night polling included) that show McCain pulling even with Obama in the Rasmussen Poll and up one point in the Gallup Poll. However, Pew (not tracking) released its national poll on Friday, with no weekend polling, that has Obama up by 5 points; and CNN (not tracking) released its poll on Thursday (again, no weekend polling) that has Obama up by 7 points. In my opinion, the latter are fair reflections of reality and the former have little to do with reality. But the media shouted up the former because they show “movement” and “closeness”, both of which are more interesting to their audiences.
These factors are among the “dirty little secrets” both of the media and the polling communities, neither of which can be held accountable for the errors that occur in the polling up to the final days before the actual election. The media polls are almost entirely over interpreted, without any accounting for normal or statistical variability that is due to the improper polling methods that are employed by their suppliers. The members of the media get excited about “changes” in the reported polling numbers that are not changes at all – creating so-called “interpretations” for polling movement that in reality did not happen. This is precisely what is happening with the AP story, cited earlier.
Some of these problems will disappear during the last poll conducted by every major polling organization. The reason is simple. The final polls are all analyzed and scrutinized by the media and by the associations of professional polling organizations and accuracy will matter because the final polls will be compared with the final results. When I was running Harris Interactive (Harris Poll) in 2000, both our telephone poll and our Internet based polls were the most accurate of any national polling organizations, a fact confirmed by the National Council of Public Polls.
In the meantime, you simply should not believe the reported ups and the downs that are reported in these tracking polls. Much of this is nothing more than statistical noise that is excessively interpreted by reporters who do not really understand the mechanics of what they are given.
Obama Continues to Have a Small Lead:
The underlying reality is that Barack Obama continues to enjoy a modest, but statistically significant lead over John McCain, with a higher percentage of voters having made up their minds than is usual at this time in a campaign. He can clearly still lose this election, but the underlying dynamics strongly favor him at this point. Because of the long primary campaign, he is better known to more Americans than would normally be the case and he is much less vulnerable to some last minute attack campaign such as the one that upended John Kerry in 2004.
Unfortunately, neither the polling organizations nor the media itself have any real serious interest in the accuracy of what they are reporting until the final poll before the election. The media always prefers closer races because it raises viewer interest and readership. Few things are more boring than Ronald Reagan’s overwhelming victory in 1984. Therefore, you can be absolutely certain that the collected media will trumpet any “closing of the race” to the hilltops, whether that closing is real or a fiction of the polling itself. Then, when the faulty tracking polls have moved back to show an Obama lead, they will find some new interpretation for this that is equally faulty.
The real interests of the polling organization is getting paid for lots of polling, and in the polls conducted just prior to the actual voting, where they will use better polling methods, regardless of the expense, because they can be held accountable for their final results.
Estimating Black Turnout:
The most significant single polling problem in this election will be to make a reliable estimate of Black turnout on the actual day of the elections. Blacks are routinely undercounted in all polls (because they are systematically harder to reach on the phone), contrasted with their final percentage in the election, and it is essential to “weight” the Black polling numbers to make them conform to the likely percentage of Black voters in the actual election.
The weighting is a judgment call based both on historical experience and to answers to a small number of questions on the survey. In the past, Blacks have made up about the 13 percent of the voters in Presidential elections -- the same percentage that they are in the electorate as a whole. However, there is good reason to believe that this number will jump in November to 15 percent or higher, based on the actual voting that will occur. Given their overwhelming loyalty to Barack Obama, a one percent under estimation of Black turnout will cause Obama’s polling numbers to be one point lower with John McCain’s one point higher, or a net reduction of two points in the margin. This will make the race appear closer than it is.
What to Believe with Polls:
Your best course of action between now and the actual election is to rely more on the summary of all national polls rather than any single poll, and to ignore the tracking polls all together. The summary of a number of polls will show more stability, less variability, and is more easily interpreted over a week or a month time period, when some of the sources of simple variability, and polling organization biases, cancel each other out. If you do that today, you will believe that Obama is ahead by five to seven points, rather than tied as shown in these silly tracking polls.
Whatever the polls say, the vote for the Presidency will be closer than the vote for Congress, where Democrats are very likely to deliver a crushing and historic defeat to the Republicans in both Houses. It is a defeat that George Bush and his colleagues have richly earned with eight years of truly horrible governance. George Bush is the “Herbert Hoover” of this generation, but that is an insult to Hoover who was in fact much brighter, more accomplished and more competent than the current occupant of the White House.
Barack Obama faces a racial problem with some percentage of the electorate, who will find excuses why they cannot vote for a person of color, but he will make up some of these losses with new voters, high Black turnout, and a clearly more energized Democratic base of voters in the electorate. If pollsters forecast that Black turnout will increase by 10 to 15 percent, relative to White voting, making Blacks 15 percent of the electorate or more due to the Obama candidacy; that situation could change the results by four points from what a normal election would produce and make the final polls look very poor indeed. Moreover, a surge in Black voters will deliver hundreds of close state and national races into the hands of the Democrats, magnifying the size and depth of the Democratic victory.
In any case, you should treat the convergence of the two candidates in the tracking polls with a healthy skepticism. I do not believe it and I do not think you should either. Equally, you should be entirely skeptical of the “talking heads” on television, who provide instant explanations for variations in polling results that simply are not real or meaningful.
At the same time, these tracking polls are very helpful for Senator Obama who gets much more mileage out of the false expectation of a close election than does Senator McCain. Every time the vote supposedly gets close, Obama’s financial contributions go up; his organization gets more active and motivated; and the polls change no one’s actual opinion of whom they will support in November. He won’t ever thank the lousy tracking polls publically, but he should.
Just my opinion,
Gordon Black
No comments:
Post a Comment