By the Numbers

"Science consists in discovering the frame and operations of Nature, and reducing them, as far as may be, to general rules or laws — establishing these rules by observations and experiments, and thence deducing the causes and effects of things." – Isaac Newton


Earlier this week, Politico.com ran a story about media outlets “preparing for the possibility that their Election Day surveys could be skewed because of overstated support for Barack Obama, largely because of the enthusiasm of his supporters.”

Exit polling is a notoriously inexact science. After all, early exit polls in 2004 suggested John Kerry would be elected president. This year, multiple variables, including the zeal of Obama’s supporters, the candidate’s racial background, widespread early voting, etc. is making those whose job it is to track voter sentiment very nervous.

In theory, exit polls should perfectly match election results. But this is not always the case. After all, preliminary exit polls during the Democratic primary overestimated Obama’s strength in 18 of 20 states by an average error of seven percentage points. (Some have attributed this discrepancy to the Bradley Effect). This overestimation was most shockingly seen in New Hampshire. Experts credited this phenomenon to the youth and enthusiasm of Obama’s supporters. Eager to participate in polls, their numbers were falsely inflated. Older voters, more likely to be Republicans, often refused the request to participate.

So what are we to make of the current predictions, which show Obama ahead by a comfortable margin of error?

Back in early June of this year, just days before Hillary Clinton suspended her presidential campaign, my friend Neil deGrasse Tyson, an astrophysicist, author, host of “Nova scienceNOW” and director of the Hayden Planetarium in New York City, wrote an Op-Ed in the New York Times titled “Vote by Numbers.”

In it, Tyson stated that if the election were to be held that day, Barack Obama would lose to John McCain (and McCain would lose to Hillary Clinton). His conclusion was based on “a new method of analysis on the statistics of polls that has been accepted for publication in the journal Mathematical and Computer Modeling.”

The authors of this new way of examining statistical data, J. Richard Gott III of Princeton, and Wes Colley of the University of Alabama in Huntsville, are not political scientists.

They are astrophysicists.

For Tyson, this made complete sense. After all, “one of the tasks of scientists is to clarify the apparent complexity of the universe by using the language of mathematics.”

Gott and Colley discovered that “in swing states, the median result of all the polls conducted in the weeks prior to an election is an especially effective predictor of which candidate will win that election — even in states where the polls consistently fall within the margin of error.” (For a far more detailed description of the rules that went into creating the formula, CLICK HERE).

In a paragraph that was actually cut for space, Tyson wrote, “Normally when you think of an average, you invoke the elementary school definition of the word. But you later learn in statistics that there are three kinds of averages – the mean, the median, and the mode. And in the field of mathematical physics, the concept of average is further generalized into higher dimensions. They look for the median count of wins and losses in a set of polls for a state, which is the same thing as asking, “Who won the most polls?” Turns out, median statistics are especially effective in swing states, where the margin of error in a single poll is typically greater than the spread in votes. As such, it can be a far more accurate and robust predictor of who will actually win an election than most people’s politically informed commentary.

“If you ask three people what time it is, and one person says its five to ten, the second says its ten o’clock, and the third says its four thirty. You do not take the average, and declare the time of day to be early afternoon. You grab the time in the middle (the median) and proceed on the more likely assumption that it’s about ten o’clock.”

So accurate was this new method, in fact, that when the 2004 presidential election between John Kerry and George W. Bush was crunched, Gott and Colley correctly predicted the winner in 49 states (they missed only Hawaii with its four electoral votes)! This at a time when almost all political analysts said the race was either too close to call or leaning toward Kerry. The median method correctly predicted Bush as the victor, even months in advance.

Using Gott and Colley’s rules, Tyson applied the median method to the 2008 presidential race and discovered that without a doubt, Obama would lose to McCain and McCain to Clinton were the election held that day. Tyson was quick to stress that the analysis did not predict what would happen in November. Instead, unlike many fickle polls, the Gott and Colley method “describes the present better than any other known method does.”

Hundreds of people wrote the New York Times to take issue with Tyson’s reasoning. He was bombarded with commenters who, while they were quick to point out their admiration of his work, took great issue with his reasoning. Many assumed Tyson wrote the piece as a last ditch effort to help Hillary Clinton. Obama supporters especially thumbed their noses at the results. Tyson defended the political neutrality of the piece’s tone and content. In responding to some of his detractors, Tyson pointed out that while math can be abused, at its best math is politically neutral. There is no such thing as red math or blue math. Just accurate math and inaccurate math.

Over the months, Tyson has forwarded me updates. It has been fascinating — and not a little disheartening — to watch the numbers. While a vast battery of polls was showing McCain down and out and Obama surging, the median method disputed them all, showing McCain still held a comfortable lead. As much as I wanted to dispute the method’s findings, Tyson kept reminding me that people may lie, but math does not. Obama’s popularity, in this case, was irrelevant.

“Our system of electoral politics requires you to not only be popular, but popular in all the right places. While human behavior can be, and often is fickle, median human behavior is not. Otherwise, there would be no such thing as an advertising industry. Accuracy matters.”

Though the analysis admitted some gains (“Colorado is a runaway for Obama”) it also spelled out disastrous defeats (“Florida is hopelessly out of reach”). For every Nevada and Virginia it gave to Obama, the median method indicated that other states were drifting further and further away.

Going into September, there were only glimmers of hope.

“As of this morning, the median analysis show **no close states** (no swing states), in spite of the rants of analysts; Obama is beating McCain 273 to 265.”

It was not nearly enough. In effect, the best Obama could hope for was a draw. Tyson and the study’s authors indicated that Obama would need to win by far more than eight electoral votes to guarantee a victory. It would require a miracle, a sort of October surprise to fundamentally shake everything up.

Then came the financial crisis.

As the gravity of the situation bit into Americans’ realities, things began to shift on the median method map.

Obama’s totals surged to 291. Florida, once “hopeless,” now suddenly looked within striking distance.

The following is from an e-mail I received from Tyson just a few days ago. It contained the word “bloodbath.”

As of October 19, median analysis of polls for the previous 30 days give the following Electoral College count:

Obama 367
McCain 171

Obama has been ahead of McCain since August 22nd, when Nevada put him in the lead. He took Florida on October 14th, which has given him a significant lead since then. The enhanced Florida campaigning in that state has worked. The overall trend looks very bad for McCain and could lead to a landslide win for Obama, giving him the mandate he will require to fulfill the many diverse promises he has made to his electorate.

The current Blue/Red state division in the median analysis can be seen in the USA map below (see the image opening this blog post), with states scaled by area to the precise number of electoral votes they represent. This “Electoral Area Map” developed by my Princeton Astrophysics colleague Richard Gott, gives an immediate sense of who is winning and who is losing.

A reminder that states which many people still claim as up for grabs are not so in the median analysis. All states given to Obama (including Ohio and Pennsylvania) are, at the moment, rather secure.

It is important to remember that this method, like all polls, is nothing more than a snapshot of a day. However, this method has proven itself to be more accurate than anything else out there. If the election were held today, Obama wins easily.

Gott and Colley’s method is a robust engine, deciphering those things people normally think are difficult-to-predict variables. In June, when Tyson first publicized the findings in The New York Times, the data pointed, with mathematical certainty, to deficiencies Obama would have to surmount should he wish to overtake his contender. It appears he has done just that. It behooves campaign strategists to pay close attention to what the median method shows. Unlike those of his supporters who were in denial of the results, Obama’s campaign paid attention, and targeted the important states with their many electoral votes where he was behind. The unbiased, impartial data is also strategically useful information for the McCain campaign, showing exactly where the Republicans must make up ground. However, with only 11 days left before Election Day, McCain’s chances to turn things around seem insurmountable if not utterly impossible.

UPDATE: Politico now reports that there are no more swing states. Every one has now gone for Obama and is outside the margin of error.