It is said that on returning from a day at the races, a certain Lord Falmouth was asked by a friend how he had fared. “I’m quits on the day”, came the triumphant reply. “You mean by that,” asked the friend, “that you are glad when you are quits?” When the said Lord replied that indeed he was, his companion suggested that there was a far easier way of breaking even, and without the trouble or annoyance. “By not betting at all!” The noble lord said that he had never looked at it like that and, according to legend, gave up betting from that very moment.
While this may well serve as a very instructive tale for many, a certain Edward O. Thorpe, writing in 1962, took a rather different view. He had devised a strategy, based on probability theory, for consistently beating the house at Blackjack (or ‘21’). In his book, ‘Beat the Dealer: A Winning Strategy for the Game of Twenty – One’, Thorp presents the system. On the inside cover of the dust jacket he claims that “the player can gain and keep a decided advantage over the house by relying on the strategy”.
The basic rules of blackjack are simple. To win a round, the player has to draw cards to beat the dealer’s total and not exceed a total of 21.
Because players have choices to make, most obviously as to whether to take another card or not, there is an optimal strategy for playing the game. The precise strategy depends on the house rules, but generally speaking it pays, for example, to hit (take another card) when the total of your cards is 14 and the dealer’s face-up card is 7 or higher. If the dealer’s face-up card is a 6 or lower, on the other hand, you should stand (decline another card). This is known as ‘basic strategy.’
While basic strategy will reduce the house edge, it is generally not enough to turn the edge in the player’s favour. That requires exploitation of the additional factor inherent in the tradition that the used cards are put to one side and not shuffled back into the deck.
This means that by counting which cards have been removed from the deck, we can re-evaluate the probabilities of particular cards or card sizes being dealt moving forward. For example, a disproportionate number of high cards in the deck is good for the player, not least because in those situations where the rules dictate that the house is obliged to take a card, a plethora of remaining high cards increases the dealer’s probability of going bust (exceeding a total of 21).
Thorp’s genius was in devising a method of reducing this strategy to a few simple rules which could be understood, memorized and made operational by the average player in real time. As the book blurb puts it, “The presentation of the system lends itself readily to the rapid play normally encountered in the casinos.”
Since the publication of the book, the strategy has been amended and improved, but Ed Thorp’s original insights stand. The problem simply changed to one familiar to many successful horse players, i.e. how to get your money on before being closed down. That’s another story, for another day. As is the time I met up with the great man himself, and the time, years later, when I shared a pint with some of the legendary MIT blackjack team.
Arbitrage is the practice of making a risk-free profit by taking advantage of a price differential between two or more markets. For example, if it is possible to BUY the number of corners in a soccer match at 10 with one odds-maker (so that a profit is obtained for every corner in excess of ten) and to SELL the number of corners at 11 with another odds-maker, a profit will accrue whatever the actual number of corner kicks taken. Such examples of free money are in short supply, however, and a more realistic trading approach may be to employ what I first proposed and termed in 2000 as a ‘quasi-arbitrage’ or ‘Quarb’ strategy. The assumption underpinning this Quarb strategy is straightforward. It is that the average market opinion (where there is a range of opinions) is a better indicator of the truth than the outlier (or ‘maverick’) opinion. Take that number of corner kicks as an example. If there are four market-makers, say, three of which offer clients the chance to BUY at 10 and SELL at 11 and the fourth which allows you to BUY at 11 and SELL at 12, the average market price is the sum of the mid-points (10.5+10.5+10.5+11.5) divided by four, i.e. 10.75. If we take this to be the best estimate of the actual expected value of the outcome, a SELL at the ‘maverick’ price of 11 is a value bet. In other words, the ‘Quarb’ strategy advocates the aggregation and averaging of the range of forecasts, implied in the odds on offer from professional odds-makers, to identify the best possible forecast of the actual outcome.
PollyVote (www.pollyvote.com) is an election forecasting site which follows this principle of combining forecasts. By aggregating the vote-share snapshot contained in traditional opinion polls with a panel of American politics experts, a prediction market and a range of quantitative forecasting models, Polly provides a daily updated forecast of the share of the two-party vote that the Democratic and Republican Presidential candidates will obtain on polling day. For their polling input they use a variation of the RealClearPolitics average and for their prediction market they use the Iowa Electronic Markets, a research and teaching-oriented marketplace which allows small bets on a range of contracts including the Presidential vote-share. The panel is made up of respected political scientists and pundits and the quantitative models (based on an analysis of the likely impact of variables ranging from unemployment and inflation to incumbency and wars) draw on an array of different methodologies. Each of these forecasts is assigned an equal 25% weight and in 2004 generated an almost spot-on forecast of the actual respective vote shares.
Is the RealClearPolitics average the best baseline to use for aggregating the polls? Is the Iowa market the best choice of prediction market? Is the chosen panel and the way its views are aggregated the best way to assimilate the combined knowledge of the political intelligentsia? Is the range of forecasting models appropriate? Does the averaging of the four methodologies perform better than any one alternative forecasting methodology?
This debate will continue, and has been informed by how well Polly has performed over the past three elections in forecasting the vote shares of the candidates. The answer is very well indeed. This is a parrot which looks very much alive!
Now that the dust has slowly started to settle on the 2012 U.S. election campaign season, we are left with a picture of what happened, who won and who lost. Aside from the politicians who came in on the wrong side of the vote, the biggest losers of this election cycle are the prognosticators of the right, who were not only almost universally wrong about the outcome, but in many cases wrong by several orders of magnitude. An independent ranking of the best and worst forecasters, based on the number of key states predicted correctly, awards the ‘broken forecast’ prize to the man who called Pennsylvania, Minnesota, Wisconsin, Iowa, New Hampshire, Ohio, Colorado, Virginia and Florida for Mr. Romney, wrong in each and every case. Step forward Fox News pundit, Dick Morris. Close runners up for most woeful political prescience include another Fox News pundit and writer for the Washington Examiner, Michael Barone, as well as Steve Forbes, of Forbes Magazine and George Will of the Washington Post.
Among the pollsters, my ‘getting it hopelessly wrong’ award goes jointly to Gallup and Rasmussen Reports, for a faintly ludicrous set of polls, skewed (presumably by accident) in favour of the Republicans, in the days and weeks running up to election day. This mirrors the huge skew to the Republicans recorded by Rasmussen in 2010, and further back, dating to his calling of the election for Mr. Bush by nine percent in 2000 (Bush lost the popular vote).
The ‘hopelessly wrong’ award for state voting goes to Susquehanna Polling of Pennsylvania, who got their polls woefully skewed (again presumably by accident) to the benefit of Mr. Romney, while the broad consensus of the out-of-state polls simultaneously got the result almost spot on. Another loser was RealClearPolitics, who managed to eliminate from consideration some of the most accurate pollsters while sticking with many of the worst.
On the plus side, it was a very good night for five sages who called the state tally perfectly, these being Nate Silver of FiveThirtyEight at the New York Times, Markos Moulitsas and Daily Kos Elections, Simon Jackman of The Huffington Post, Josh Putnam of Davidson College and Drew Linzer of Emory University. More broadly speaking, it was a great night for serious statistical analysis, for the ‘quants’ as they are sometimes labelled (including Sam Wang, of the Princeton Election Consortium).
The unsung hero of this election, however, is not a particular pundit, or statistician, or polling organization. It is not an expert at all. The unsung hero is the anonymous crowd. The reason is that while an expert may know more than anyone else in the room, he is unlikely to know more than the room as a whole, to be wiser or cleverer than the crowd. It is an idea picked up and developed in a fascinating recent paper by David Rothschild and Justin Wolfers, of the Wharton School, University of Pennsylvania, who argue that better forecasts can be made by asking people who they think will win than by asking their personal voting intention.
This is the idea that is driving the growing belief in the power of markets like the betting exchange, Betfair, to unveil the future. Despite the swings and roundabouts in the polls and among the pundits, these markets never wavered in their belief that the president would be re-elected. It all comes down to the idea that those who know the most will tend to bet the most, and accurate forecasts can thus be obtained by following the money. This is the basis of the science of ‘prediction markets’, which are essentially speculative markets created or employed for the purpose of making predictions.
In the weeks to come, data will be crunched to see exactly how well these markets did perform in Election 2012. It will be a while until we know the details, but already I am confident enough to make at least one prediction — they will have performed very well indeed.
Follow Leighton Vaughan Williams on Twitter: www.twitter.com/leightonvw
As the first election returns started to filter through on Tuesday evening, it soon became apparent that President Barack Obama was very likely to be re-elected to a second term in office. A comparison of early precinct returns with those registered in 2008 soon confirmed the evidence of the exit polls. Mitt Romney simply had far too few votes to seriously challenge the President where it mattered.
Up to this point, the balance of opinion among so-called political experts was pretty evenly divided between those thinking, like Fox News pundits Dick Morris and Michael Barone, that Romney would win by a landslide, and statistical analysts like Nate Silver, of FiveThirtyEight, Sam Wang of the Princeton Election Consortium and Drew Linzer of Votamatic, who believed the opposite.
Meanwhile, a glance at the betting/prediction markets showed what they had always shown — that the president would be re-elected by a handsome margin, and that nothing that Mr. Romney or his team could do was likely to prevent that happening. As the first voting stations opened, the bookmakers, the spread betting operators and the betting exchange, Betfair, all converged at a probability of re-election of about 80 percent. Intrade had it a bit lower, but still made President Obama the very clear favorite.
So what about the opinion polls? There was little consensus, and some of them, like so-called scientific pollsters Rasmussen and Gallup, had Mitt Romney up in their final published poll, as they had in most of their polls in the days and weeks before. More strangely, some commentators were apparently still taking these two operators seriously.
The national polling average on the RealClearPolitics website, which included Rasmussen and Gallup, also put Romney up for much of the lead-up to the election, although it showed a late shift to put Obama up by 0.7 percent on election day. RealClearPolitics publish a selection of the latest polls and take an average of these to come up with this overall figure.
But is this what the opinion polls were actually showing? The answer depends on whether you look at all the published polls, as broadly available on Pollster, or whether you exclude the polls you don’t favor, a methodology favored by RealClearPolitics. Not only do they not include a number of well respected polls in their overall average, they do not recognize them at all! Post-election analysis has in fact shown that the pollsters they exclude performed signficantly better than those (like Rasmussen and Gallup) they choose for inclusion.
Taking an average of all the national polls, good and bad, had it in the President’s favor pretty much all along.
More important than the national picture are the state findings. An analysis of all the state polls showed that President Obama was rather more comfortably ahead, including in the key swing states of Ohio, Virginia, Wisconsin, Colorado, Iowa, Nevada and New Hampshire.
Combining the national and state polls, and awarding them a similar weighting, revealed Mr. Obama to be polling about 2.5 percent ahead of Mr. Romney — a bit more in the state polls and a bit less in the national polls.
Depending on the statistical analysis used to convert polling advantage into a probability of winning, the polling evidence on the day gave the President somewhere above a 90 percent chance of winning.
In this battle between the Fox News political ‘experts’ and the sophisticated polling analysts, the betting markets sided heavily with the analysts. To be fair, the raw betting data indicated a slightly more conservative evaluation of Mr. Obama’s chances than did the models in the days leading up to the election. But this is for good reason. Why so?
First, the markets allow for the uncertainties a poll-based model can’t allow for, such as some late surprise or revelation. The point here is that potential volatility is the friend of the underdog, and while polls are snapshots of opinion, the betting markets are all about forecasting the eventual outcome.
Second, there is a well-established favorite-longshot bias which occurs in most betting markets, whereby the markets tend to slightly under-estimate the true likelihood of the favorite winning, and vice-versa. Prediction market analysts compensate for this to provide an even better forecast.
The markets have been successfully predicting elections since at least 1868, and since the advent of low-margin betting exchanges like Betfair have never been so sophisticated or so accurate. These exchanges differ from traditional betting markets by eliminating the odds-setting bookmaker, instead providing the technology to match up the best offers to back and lay an outcome on offer from all the clients of the exchange. They offer what is essentially a conversation in which the loudest voices, mediated by the focus of money, are generally the best informed.
There were big losers on election night, Rasmussen and Gallup polling, along with RealClearPolitics and Fox News being among them.
Mr. Obama was, of course, a big winner on election night. But there were others. Statistical modelling was one such winner, and the other big winners were the prediction markets, which had it right all along.
Follow the money! It really does work every time.
The conventional wisdom for the last several days is that the betting markets are pointing to the re-election of the President while the polls apparently have it too close to call, or have Mitt Romney as the narrow favourite.
So what’s the evidence?
Certainly, a glance at the betting markets reveals a pretty strong consensus that President Obama has a strong edge with the bookmakers, the spread betting operators and the betting exchange, Betfair.com, each of these currently converging at a probability of re-election of about 75%. Intrade.com has it a bit lower, but still make President Obama the very clear favourite.
So what about the opinion polls? Here the idea that Mr. Romney is ahead is in great part derived from an inspection of the headline national poll findings published on the RealClearPolitics.com (RCP) web-site. RCP publish a selection of the latest polls and take an average of this selection to come up with their overall figure. This average had Romney leading by up to a point or so over the last few days, and currently has it about tied.
But is this what the opinion polls are actually showing? The answer depends on whether you look at all the published polls, as broadly available on Pollster, or whether you exclude the polls you don’t favour, a methodology favoured by RCP. Not only do they not include a number of these polls in their overall average, they do not recognize them at all!
Taking average of all the national polls still has it close, but close in the President’s favour.
More important than the national picture, as Al Gore found to his cost, are the state findings. An analysis of all the state polls shows that President Obama is rather more comfortably ahead, including in the key swing states of Ohio, Virginia, Wisconsin, Colorado, Iowa, Nevada and New Hampshire.
Combining the national and state polls, and awarding them a similar weighting, reveals Mr. Obama to be polling close to 2.5% ahead of Mr. Romney – more in the state polls and less in the national polls.
Depending on the statistical analysis used to convert polling advantage into probability of winning, current polling evidence would seem to give the President somewhere above an 8 in 10 chance of winning (based on national and state polls) to rather more than a 90% chance (based on the state polls alone). FiveThirtyEight.com and the Princeton Election Consortium (http://election.princeton.edu/) represent good examples of this sort of analysis.
So the betting markets, contrary to conventional wisdom, are in fact more conservative in their evaluation of Mr. Obama’s chances than are the polls.
There are a number of possibilities, but I will highlight three of them.
First is the well-established favourite-longshot bias which occurs in most betting markets, whereby the markets tend to under-estimate the true likelihood of the favourite winning, and vice-versa.
Secondly, the markets may be allowing for the uncertainties a model can’t allow for them, such as some late surprise. The point is that volatility is the friend of the underdog, and the enemy of the favourite.
Thirdly, the markets may be wary that one of the outlier polling organisations might just possibly be right, and the overwhelming polling consensus might just possibly be wrong. This is what I call the ‘Rasmussen effect’, after the polling company that seems to have leaned Republican by a few points in its national and state polling for some time now (http://www.huffingtonpost.com/alan-abramowitz/the-rasmussen-difference_b_2030330.html?utm_hp_ref=@pollster).This effect was particularly prominent in the 2010 elections (http://fivethirtyeight.blogs.nytimes.com/2010/11/04/rasmussen-polls-were-biased-and-inaccurate-quinnipiac-surveyusa-performed-strongly/), and this is the pollster that had Mr. Bush leading by nine per cent on election day, 2000, as evidenced by the National Council on Public Polling (http://www.ncpp.org/?q=node/20). That was about ten points out!
If Rasmussen is right, though, the race is very tight in the key states and Obama would be much less of a favourite than the polling consensus suggests.
So Rasmussen must be right and the other polls wrong, and Obama must lose Ohio, which Rasmussen is calling a pure toss-up, and he must not win elsewhere to compensate. All other routes are regarded as far more unlikely by the markets, though by no means impossible.
So what is the most likely outcome of this election? However you cut the data, the answer is the same. It’s a win for the President.
Polls don’t decide elections, however, and neither do the betting markets. People do, and that’s why anything can happen, and sometimes does!
When even a CNN poll, composed of a sample of debate watchers they freely admit to be eight per cent more Republican than the country as a whole, hands the debate to President Obama by seven percentage points, you know that Governor Romney has taken something of a beating. The win was also recorded by CNN’s focus group of uncommitted voters in Ohio, and was confirmed by the consensus of other instant polling, notably from Survey USA, CBS News, Battleground, Google Consumer Surveys and Public Policy Polling (PPP). The most interesting of these polls is perhaps the PPP survey of independents watching the debate in the swing state of Colorado. By 58% to 36%, these voters named Obama the winner.
Those who prefer to track the debate through the online betting markets, such as Betfair and Intrade, saw things no differently, as a flood of money came for the President through the course of the debate. Betfair traders piled their money into backing the re-election of the President, while Intrade ran a market exclusively on who would win the debate, deferring to the CNN instant poll as the final arbiter. The latter market started out marginally favouring Mr. Obama but soon attracted a growing tide of support for the incumbent, until by the end of the debate the odds implied a probability of an Obama win of more than 90%. Brave traders indeed, trusting to a CNN poll which had already heavily over-sampled Republicans in the Vice-Presidential debate. Either they were pretty sure that the President had won so handsomely that he could overcome a potential repeat skew, or they were unaware of it. Whatever the reasoning, the market got it right.
According to this evidence, Mitt Romney not only lost the second Presidential debate, he was bludgeoned out of it.
But will it make a difference to the election outcome?
For the answer to this we can turn to those all-important fluctuating price lines which represent the probability that one candidate or other will take the White House.
Interestingly, it was such a price line that signalled, during the first of the UK Prime Ministerial debates in 2010, the impending disappearance of the Conservative Party’s hopes of winning an overall majority in that year’s General Election. David Cameron’s performance in that debate mirrored perceptions of Barack Obama’s performance in the first of this year’s Presidential debates. Cameron’s performance in the second debate was substantially better, mirroring again Obama’s performance in his second debate.
Does this tell us something? The Conservative leader’s first debate flop almost certainly cost his party an overall majority in the House of Commons. His second debate performance helped him to Downing Street at the head of a coalition government. So are there any parallels we can draw for the US election?
In one sense, the parallels are limited because the US Presidential election is winner-takes-all. But there is another sense in which a parallel does exist. If we consider control of the Senate and House of Representatives as being in some degree linked to the performance and popularity of the President, what is sometimes called the ‘coattail’ effect, we might just have witnessed a performance outstanding enough in the context of the two debates to hand the White House and Senate to the Democrats, but perhaps not enough for a full sweep of Congress.
Is this the way it will eventually turn out?
We shall see!