Reporters and Obama's Lead in the Polls

Why Are Reporters Surprised By Obama’s Narrow Lead?
http://news.yahoo.com/s/politico/20080806/pl_politico/12334

Senator Barack Obama is the first American of African descent to have a realistic opportunity to become the President of the United States. Given two hundred years or more of slavery and another hundred years or so of an American Apartheid, this fact alone is remarkable, both to Americans and to the rest of the world where all kinds of slavery and Apartheid still exist in large doses. Racism, after all, has not been abolished in American hearts and race still matters to too many people, although voters have grown much more wary about sharing their true feelings to pollsters or to each other on this sensitive subject.

Given this reality, the amazement of the situation is that he is leading at all and not that he cannot win decisively or in a landslide. There was never any way that an African American candidate was ever going to win a landslide vote the first time around, no matter how miserable the failure of the Republican Administration. If he were running directly against George Bush, he might well be a few more points ahead, but he is not running against George Bush and Senator John McCain is by far the best Republican candidate for distancing the Republican choice from the current Administration.

There is some, not easily estimated, percentage of the American electorate who will not vote for Barack Obama because of who he is – both racially and otherwise. It is no longer socially acceptable to be a racist, at least not in polite society, so people keep their private views of the subject to themselves. Many of them will not tell pollsters over the phone how they really feel, that is for certain, but they can freely express their feelings in the choice of their candidate – in this case, Senator John McCain. For a portion of these voters, particularly working class white males, their feelings are so strong that it will override their own economic self interest, although they probably use cognitive dissonance to resolve the contradiction.

Therefore, and it is pretty simply math, Senator Barack Obama has to overcome this bloc of antagonists, some of whom traditional vote Democratic, with a bloc of protagonists who represent a new constituency for the Democrats. There is no other way for him to win, albeit narrowly and probably not by more than five points at the most.

The Power of the New Constituency:

Obama is going to need every vote he can get from those under 35, who have much more tolerant views of racial issues, other African-Americans, Latinos, and moderate and educated Independents and even Republicans. To do precisely this, he has built and mobilized the single most effective campaign organization in American political history. Three of these groups are made up of people who are less inclined to participate in elections than the voters, who are supporting McCain – older, more settled, more involved in their communities, better educated, etc. Obama has to turn these voters out in numbers not seen in any recent elections, a formidable challenge.

Ironically, the most difficult and inaccessible of these groups are the African Americans and Latinos who reside in the large urban ghettos (and some rural areas of the South) that still are part of the landscape of American politics. These neighborhoods are mostly represented by old line Black and Latino office holders, whose organizations look a lot like old time political machines, although less effective. An army of middle class Blacks and Whites is not likely to be effective in some of these areas, and it is simply not clear whether a dramatic increase in turnout is even in the interest of the office holders who represent these voters, whether they are African American or not.

At the same time, a substantial victory among these groups, particularly the White voters under 35, is a very substantial threat to the Republican Coalition for years to come. Moreover, the successful mobilization of these voters, in large numbers, will add margins for Democrats at every level of office. McCain simply has no ability to capture a majority among the younger voters, other than the Evangelicals, and a substantial loss among the youth vote hangs on for a long time in politics (see earlier Blogs).

Challenging for the Center is More Reliable:

An equally important strategy for Senator Obama is to go after the Center in American politics, no matter how much that enrages some of his more liberal and left wing supporters. A centrist strategy is always the way to win in American politics, although Democrats seem to have had a harder time doing this than Republicans. When, instead, a Democratic Presidential candidate does as George McGovern did in 1972, it invariably proves disastrous.

By temperament, Obama is very well designed to do precisely this. He is measured, calm, reasonable, sensible, and balanced – traits that have come through rather clearly over the course of the primary season. McCain, by contrast, is a well known hot head, given to great fits of temper on occasion and even intemperate remarks. Today, unlike a year ago, Obama is also “tested” by the most intense primary campaign in American history, a fact that will now work to his advantage.

On policy issues, Obama is catching some real breaks that are helping him to reposition to the Center. On foreign policy, the Center and events have simply moved to him – a reasonably prudent withdrawal from Iraq (sixteen months sounds fine) and a shift to stabilize with more military support the youthful government of Afghanistan (which everyone now seems to applaud). Both McCain and Bush have been forced to embrace withdrawal, but not on a time table to anyone’s confidence or liking.

The rapidly rising prices of oil and gas, coupled with enormous oil company profits that are undermining the credibility of the oil lobby, have shifted the focus on energy to the national security issue that it is. The country ready for and is up to a new challenge, and shifting our economy from its fossil fuel dependence simply no longer sounds radical at all, but very sensible, in spite of the propaganda to the contrary from the oil and coal lobbies.

This large an undertaking will require a multiplicity of policy changes, and brokering the deal is likely to be the first major challenge facing the new Administration. Obama is in an ideal position to broker this deal, particularly because he has taken so little money from any of these powerful lobbies and because he has large constituencies in the Democratic Party (and among Republican moderates) who will support him on a brokered deal. The single interest lobbies – oil, coal, ethanol, wind, solar, nuclear, etc. – are all going to have to play a changing role in this plan going forward (without any veto), but there must be an honest broker to make it happen, and that certainly is not John McCain nor could it ever have been George Bush.

On the central issue of the economy, however, the situation is much more difficult. In truth, neither McCain nor Obama are talking realistically about the economy, and they are smart enough to know the game they are both playing. McCain promotes the complete fiction that we can solve our problems through the elimination of so-called “wasteful spending,” and he would keep all of the tax cuts in place. Obama promotes the complete fiction that we can raise all the money we are now spending by taxing just the rich and that is equally preposterous. There just flat are not enough rich people to close the budget gap, even if we taxed them at 100 percent. Moreover, the more they tax capital and dividends, the greater will be the decline of investment capital in the small businesses that produce most of the new jobs in the United States. The policies of the Democrats are almost invariably designed to placate organized unions, particularly the public employees, and they seem to be consistently unsympathetic to strategies that genuinely create new jobs in America, not matter what they claim to the contrary.

Virtually everything both candidates saying on the economy is posturing for the purpose of winning votes, with little semblance of reality. As a former businessman, it is obvious that neither of them ever met a payroll, never raised and spent someone else’s risk capital, nor understands the simple fact that the engine of job creation in America is small business, which converts invested risk capital directly into new jobs.

The real danger with both candidates is that they will do something dramatic, precipitous, and dangerous in economic policy changes that will throw the economy into reverse, creating a more dramatic version of the problems they already have. Good “politics” frequently is not good “economics.”

This Election Will Stay Close Until the End:

Baring a major stumble by one of the candidates, this election is likely to stay relatively close all the way to the end. The voters have had more exposure to both candidates than is usually the case before September, and voters are more fixed in their preferences than you would normally expect at this point. Despite the silly, poorly executed tracking polls, Obama seems to maintain a 5 to 7 point lead over McCain, but he has not been able to expand on that even with very favorable publicity during the summer.

Although both candidates can say that race does not matter, it still does, and there is a constituency of unknown size, particularly among working class White males, that simply will not vote for Barack Obama in 2008. At the same time, it is very difficult for McCain to overcome the huge deficiencies of the Bush Administration. There is an even larger constituency who feel that the past eight years are among the worse governed years in American history, that that constituency is unlikely to embrace John McCain, even if uncomfortable with Barack Obama

Personally, I expect the campaign to get pretty ugly before it is over, partly because it will remain so close. There is a generation of Republican political consultants, trained in the Karl Rove School of political viciousness and character assassination, who work with a variety of interests in the Republican Party. I expect that we will hear from them loudly and clearly before this is over.

This is usually the tactic of October, when it is hard for the opposing candidate to respond with the amount of time left in the race. The media loves this really nasty campaigning, no matter what it pretends to the contrary, and they becomes the hand maiden to the distortions, repeating them endlessly in the hopes of making things interesting, while Senator John McCain will distance himself from the attacks. It worked with John Kerry in 2004, and it has worked elsewhere. The theory is that if you throw enough filth, some of it will stick, particularly if it is close to the election leaving little room for a response. The Republicans are very, very good at it, even if John McCain disavows it.

Just my opinion,

Gordon Black

Obama Is Ahead -- Five to Seven Points

A Short Note on Reading Polls and Media Interpretation:

I wrote you last week that you should just ignore the tracking polls, which suffer from poorer execution than the one-time national polls that appear every week or so. Over the past two weeks, the onetime polls – CBS, CNN, Time, Pew, etc. – show Barack Obama with a very stable and consistent 5 to 7 point lead nationally over John McCain.

The tracking polls by contrast have consistently had the race between no points and a couple of points up for Obama over exactly the same time period, and the tracking polls show so-called “movement” that is not really movement at all, but simply shifts that are occurring because of the way the polls are executed.

The tracking polls receive more media attention because the so-called “closeness” and “movement” is interesting, even if it is not real at all. The so-called “movement” gives the talking heads something to theorize about – endless speculation that cannot be confirmed or denied by anyone, but sounds at least plausible. The talking heads, particularly those who wear Republican clothing, prefer the tracking polls because a close race is more newsworthy than a five to seven point stable lead.

My reading of this election at this point in August, weeks before the conventions, is as follows:

1. Senator Barack Obama enjoys a small, but quite stable lead over Senator John McCain in the range of 5 to 7 points, but his support still hovers below 50 percent with a significant undecided factor that certainly could decide the final outcome.
2. The so-called “movement” – and all interpretations of that movement – is sheer nonsense based on poorly executed tracking polls that have a harder time reaching people who are more likely to support Obama – the young under 35, African Americans, and Latinos.
3. There is a consistent “media bias” to over-interpret polling that shows the race close as opposed to polls that show the race favoring Obama. This is not a bias toward McCain per se, but a bias in favor of a race that is close because it is more news worthy and interesting.
4. Many of the so-called “explanations” for changes in the polling numbers are without any real foundation at all – sheer speculation lacking in substances and based on changes that are themselves suspect.
5. The regular, one time polls, with one notable exception from Gallup, are all fairly close and well within the normal tolerances of repeated polling. When a single poll comes out that is substantially at variance with most of the rest, you should just disregard it because consistently applied commonly used techniques produce results that are usually very consistent, within one to three points at most.

This race will continue to be close, and it is never going to be a blow out for Obama, despite the overwhelming antagonism toward the Bush Administration. I will send a longer blog on this subject tomorrow.

Just my opinion,

Gordon Black

Do Not Believe the Tracking Polls

Do Not Believe the Tracking Polls:

Tracking polls are very useful enterprises. I know. I used them a lot over the years, both for media, political and for commercial clients. At the same time, I am deeply suspicious of most tracking polls that are done for public consumption today because they are flat out NOT as accurate as full scale, single time telephone polling.

They have variability problems based on the way they are conducted, which makes their excessive interpretation by the media highly suspect. Moreover, the supposed opinion changes they report over interpreted by reporters who know very little about the ways the polls are conducted.

Take this story on Sunday, August 3rd, by the Associated Press – a story which relies on tracking polls and largely ignores the more accurate regular polls by Pew and CNN, reported only two days earlier and conducted I am sure, properly. The conclusions of the story are simply not correct, and terribly and I believe intentionally misleading: http://news.yahoo.com/s/ap/mccain_obama_poll. AP should be embarrassed by this story, but they will not be. Very unprofessional!

Problems With All Telephone Polling:

A short explanation of the difference between these types of polls is useful at this point. All telephone polling is today very difficult to conduct accurately. This is a simple truth that the best professional pollsters are loath to talk about candidly because such polls are their bread and butter during elections. There are marked problems with the use of the telephone for opinion polling that did not exist nearly as much twenty or thirty years ago. Here are some of them:

1. The combination of “refusal rates” and “unable to reach” rates today are so high with telephone polling that the overall completion rates (percentage of people called who are actually interviewed) is generally below 20 percent , and sometimes 10 percent, if these rates are calculated fairly and reported accurately, which is often not the case. This inability to reach a high proportion of the targeted sample introduces a whole range of biases that are difficult to explain and overcome.
2. The "completion rates" are also much lower today in part to avoid telemarketing; and a growing number of people are on “do not call lists” or use caller identification to avoid such calls.
3. The percentage of people without land line phones and/or who use predominately their cell phones has risen sharply in recent years, making these individuals much more difficult to reach at all.
4. The pressure from “paying clients” to be “cost effective” puts more pressure on the polling organizations to cheat on the standards of complying with the rigorous rules of random selection and repeated callbacks in polling.

This situation exists with standard one time telephone polling, but it is magnified with telephone tracking polls. With a telephone tracking poll, the polling organization completes approximately 400 to 600 new interviews every day, substituting the new day of polling for the equivalent number of respondents who are dropped out of the poll after three days, making the reported result a running average of the three completed days of polling. The additional problems that affect the accuracy of these types of polls are:

1. In a normal telephone poll, the entire poll lasts over three days, with the second and third day used to attempt to complete those people who were unreachable on the first day of polling. These “callbacks” are critical to complete as much of the original random sample as possible. This rigorousness is almost never accomplished as well with the telephone tracking polls, a fact which polling organizations are also loath to discuss.
2. Some groups of voters are much more difficult reach with weekend calling, particularly Friday night, Saturday night, and Sunday during the day) because they are away from the phone or replying entirely on their cell phones. In my experience, this is truer of African Americans, Latinos, urban voters, and younger voters, whose lifestyles do not match the more sedate (and available) lifestyles of older, more settled voters, particularly those who live in smaller, more rural communities that are more likely to be Republican.
3. Given fewer callbacks and weekend variability, most tracking studies show sharp variability that is due to simple sampling variability, response bias and differential completion rates between Republicans and Democrats, and the variation has little to do with anything happening in the society as a whole.

Democrats are normally more affected by some of these issues than Republicans, because the Republican base of voters is made up more of older, more settled, and more easily reached voters. The Obama coalition, however, is even more subject to polling inaccuracy because of harder to reach respondents – particularly African Americans, Latinos, and voters under 35 who are generally much harder to reach, particularly on weekends and with cell phone use.

The problems are even more acute with the tracking polls, where these “harder to reach” issues are both more pronounced and less professionally executed, particularly on the weekends. The only way to deal with these problems is to “weight” the data to some predetermined “expected” proportions by various demographics, but that substitutes the subjective values and estimations of pollsters who have their own partisan biases substituting for scientific rigor.

The August Closure in Tracking Poll Results:

The problems are acutely present in the most recent polls. Both Gallup and Rasmussen released tracking polls on Saturday, August 2nd, (Friday night polling included) that show McCain pulling even with Obama in the Rasmussen Poll and up one point in the Gallup Poll. However, Pew (not tracking) released its national poll on Friday, with no weekend polling, that has Obama up by 5 points; and CNN (not tracking) released its poll on Thursday (again, no weekend polling) that has Obama up by 7 points. In my opinion, the latter are fair reflections of reality and the former have little to do with reality. But the media shouted up the former because they show “movement” and “closeness”, both of which are more interesting to their audiences.

These factors are among the “dirty little secrets” both of the media and the polling communities, neither of which can be held accountable for the errors that occur in the polling up to the final days before the actual election. The media polls are almost entirely over interpreted, without any accounting for normal or statistical variability that is due to the improper polling methods that are employed by their suppliers. The members of the media get excited about “changes” in the reported polling numbers that are not changes at all – creating so-called “interpretations” for polling movement that in reality did not happen. This is precisely what is happening with the AP story, cited earlier.

Some of these problems will disappear during the last poll conducted by every major polling organization. The reason is simple. The final polls are all analyzed and scrutinized by the media and by the associations of professional polling organizations and accuracy will matter because the final polls will be compared with the final results. When I was running Harris Interactive (Harris Poll) in 2000, both our telephone poll and our Internet based polls were the most accurate of any national polling organizations, a fact confirmed by the National Council of Public Polls.
In the meantime, you simply should not believe the reported ups and the downs that are reported in these tracking polls. Much of this is nothing more than statistical noise that is excessively interpreted by reporters who do not really understand the mechanics of what they are given.

Obama Continues to Have a Small Lead:

The underlying reality is that Barack Obama continues to enjoy a modest, but statistically significant lead over John McCain, with a higher percentage of voters having made up their minds than is usual at this time in a campaign. He can clearly still lose this election, but the underlying dynamics strongly favor him at this point. Because of the long primary campaign, he is better known to more Americans than would normally be the case and he is much less vulnerable to some last minute attack campaign such as the one that upended John Kerry in 2004.

Unfortunately, neither the polling organizations nor the media itself have any real serious interest in the accuracy of what they are reporting until the final poll before the election. The media always prefers closer races because it raises viewer interest and readership. Few things are more boring than Ronald Reagan’s overwhelming victory in 1984. Therefore, you can be absolutely certain that the collected media will trumpet any “closing of the race” to the hilltops, whether that closing is real or a fiction of the polling itself. Then, when the faulty tracking polls have moved back to show an Obama lead, they will find some new interpretation for this that is equally faulty.

The real interests of the polling organization is getting paid for lots of polling, and in the polls conducted just prior to the actual voting, where they will use better polling methods, regardless of the expense, because they can be held accountable for their final results.

Estimating Black Turnout:

The most significant single polling problem in this election will be to make a reliable estimate of Black turnout on the actual day of the elections. Blacks are routinely undercounted in all polls (because they are systematically harder to reach on the phone), contrasted with their final percentage in the election, and it is essential to “weight” the Black polling numbers to make them conform to the likely percentage of Black voters in the actual election.

The weighting is a judgment call based both on historical experience and to answers to a small number of questions on the survey. In the past, Blacks have made up about the 13 percent of the voters in Presidential elections -- the same percentage that they are in the electorate as a whole. However, there is good reason to believe that this number will jump in November to 15 percent or higher, based on the actual voting that will occur. Given their overwhelming loyalty to Barack Obama, a one percent under estimation of Black turnout will cause Obama’s polling numbers to be one point lower with John McCain’s one point higher, or a net reduction of two points in the margin. This will make the race appear closer than it is.

What to Believe with Polls:

Your best course of action between now and the actual election is to rely more on the summary of all national polls rather than any single poll, and to ignore the tracking polls all together. The summary of a number of polls will show more stability, less variability, and is more easily interpreted over a week or a month time period, when some of the sources of simple variability, and polling organization biases, cancel each other out. If you do that today, you will believe that Obama is ahead by five to seven points, rather than tied as shown in these silly tracking polls.

Whatever the polls say, the vote for the Presidency will be closer than the vote for Congress, where Democrats are very likely to deliver a crushing and historic defeat to the Republicans in both Houses. It is a defeat that George Bush and his colleagues have richly earned with eight years of truly horrible governance. George Bush is the “Herbert Hoover” of this generation, but that is an insult to Hoover who was in fact much brighter, more accomplished and more competent than the current occupant of the White House.

Barack Obama faces a racial problem with some percentage of the electorate, who will find excuses why they cannot vote for a person of color, but he will make up some of these losses with new voters, high Black turnout, and a clearly more energized Democratic base of voters in the electorate. If pollsters forecast that Black turnout will increase by 10 to 15 percent, relative to White voting, making Blacks 15 percent of the electorate or more due to the Obama candidacy; that situation could change the results by four points from what a normal election would produce and make the final polls look very poor indeed. Moreover, a surge in Black voters will deliver hundreds of close state and national races into the hands of the Democrats, magnifying the size and depth of the Democratic victory.

In any case, you should treat the convergence of the two candidates in the tracking polls with a healthy skepticism. I do not believe it and I do not think you should either. Equally, you should be entirely skeptical of the “talking heads” on television, who provide instant explanations for variations in polling results that simply are not real or meaningful.

At the same time, these tracking polls are very helpful for Senator Obama who gets much more mileage out of the false expectation of a close election than does Senator McCain. Every time the vote supposedly gets close, Obama’s financial contributions go up; his organization gets more active and motivated; and the polls change no one’s actual opinion of whom they will support in November. He won’t ever thank the lousy tracking polls publically, but he should.

Just my opinion,

Gordon Black