Forecasting Elections and Other Things – Where did it all go wrong?

There are a number of ways that have been used over the years to forecast the outcome of elections. These include betting markets, opinion polls, expert analysis, crystal balls, tea leaves, Tarot cards and astrology! Let’s start by looking at the historical performance of betting markets in forecasting elections.

The recorded history of election betting markets can be traced as far back as 1868 for US presidential elections and 1503 for papal conclaves. In both years, the betting favourite won (Ulysses S. Grant, 1868 elected President; 1503 Cardinal Francesco Piccolomini elected Pope Pius III). From 1868 up to 2016, no clear favourite for the White House had lost the presidential election other than in 1948, when longshot Harry Truman defeated his Republican rival, Thomas Dewey. The record of the betting markets in predicting the outcome of papal conclaves since 1503 is less complete, however, and a little more chequered. The potential of the betting markets and prediction markets (markets created to provide forecasts) to assimilate collective knowledge and wisdom has increased in recent years as the volume of money wagered and number of market participants has soared. Betting exchanges (where people offer and take bets directly, person-to-person) now see tens of millions of pounds trading on a single election. An argument made for the value of betting markets in predicting the probable outcome of elections is that the collective wisdom of many people is greater than that of the few. We might also expect that those who know more, and are better able to process the available information, would on average tend to bet more. Moreover, the lower the transactions costs of betting and the lower the cost of accessing and processing information, the more efficient we might expect betting markets to become in translating information into forecasts. In fact, the betting public have not paid tax on their bets in the UK since 2001, and margins have fallen significantly since the advent of person-to-person betting exchanges which cut out the middleman bookmaker. Information costs have also plummeted as we have witnessed the development of the Internet and search engines. Modern betting markets might be expected for these reasons to provide better forecasts than ever.

There is indeed plenty of solid anecdotal evidence about the accuracy of betting markets, especially compared to the opinion polls. The 1985 by-election for the vacant parliamentary seat of Brecon and Radnor in Wales offers a classic example. Mori, the polling organisation, had the Labour candidate on the eve of poll leading by a massive 18%, while Ladbrokes, the bookmaker, simultaneously quoted the Liberal Alliance candidate as odds-on 4/7 favourite. When the result was declared, there were  two winners – the Liberal candidate and the bookmaker.

In the 2000 US presidential election, IG Index, the spread betting company, offered a spread on the day of 265 to 275 electoral college votes about both Bush and Gore. Meanwhile, Rasmussen, the polling company, had Bush leading Gore by 9% in the popular vote. In the event, the electoral college (courtesy of a controversial US Supreme Court judgment) split 271 to 266 in favour of Bush, both within the quoted spreads. Gore also won the popular vote, putting the pollster out by almost 10 percentage points.

In the 2004 US presidential election, the polls were mixed. Fox had Kerry up by 2 per cent, for example, while GW/Battleground had Bush up 4. There was no consensus nationally, much less state by state. Meanwhile, the favourite on the Intrade prediction market for each state won every single one of those states.

In 2005, I was asked on to a BBC World Service live radio debate in the immediate run-up to the UK general election, where I swapped forecasts with Sir Robert Worcester, Head of the Mori polling organisation. I predicted a Labour majority of about 60, as I had done a few days earlier in the Economist magazine and on BBC Radio 4 Today, based on the betting at the time. Mori had Labour on a projected majority of over 100 based on their polling. The majority was 66.

In the 2008 US presidential election, the Betfair exchange market’s state-by-state predictions called 49 out of 50 states correctly. Only Indiana was called wrong.  While the betting markets always had Obama as firm favourite, the polls had shown different candidates winning at different times in the run-up to the election. On polling day, Obama was as short as 1 to 20 to win on the betting exchanges, but some polls still had it well within the margin of error. He won by 7.2%. By 365 Electoral Votes to 173.

In the 2012 US presidential election, the RealClearPolitics average of national polls on election day showed Obama and Romney essentially tied. Gallup and Rasmussen had Romney leading, others had Obama narrowly ahead. To be precise, the average of all polls had Obama up 0.7%. Obama won by 4% and by 332 electoral votes to 206.

In the week running up to polling day in the 2014 Scottish referendum, polls had No to independence with leads of between 1% (Panelbase and TNS BMRB) to, at the very top end, Survation (7%), and YES to independence with leads of between 2% (YouGov) and 7% (ICM/Sunday Telegraph). The final polls had No to independence between 2% and 5% ahead. The actual result was No by 10.6%. The result had been reflected in the betting markets throughout, with No to independence always a short odds-on favourite. To give an example of the general bookmaker prices, one client of William Hill staked a total of £900,000 to win £193,000 which works out at an average price of about 1 to 5.

In the 2015 Irish referendum on same-sex marriage, the final polls broke down as a vote share of 70% for Yes to 30% for No. In the spread betting markets, the vote share was being quoted with mid-points of 60% Yes  and 40% No. The final result was 62% Yes, 38% No, almost exactly in line with the betting markets.

In the Israeli election of 2015, the final polls showed Netanyahu’s Likud party trailing the main opposition party by 4% (Cannel 2, Channel 10, Jerusalem Post, by 2% (Teleseker/Walla) and by 3% (Channel 1). Meanwhile, Israel’s Channel 2 television news on election day featured the betting odds on the online prediction market service, Predictwise. PredictWise had Netanyahu as 80% favourite. The next day, Netanyahu declared that he won “against the odds.” In fact, he did not. He won against the polls.

In the 2015 UK general election, the polling averages throughout the campaign had the Conservatives and Labour neck and neck, within a percentage point or so of each other. Meanwhile, the betting odds always had Tory most seats at very short odds-on. To compare at a point in time, three days before polling, the polling average had it tied. Simultaneously, Conservatives most seats was trading on the markets as short as 1 to 6.

If this anecdotal evidence is correct, it is natural to ask why the betting markets outperform the opinion polls in terms of forecasting accuracy. One obvious reason is that there is an asymmetry. People who bet in significant sums on an election outcome will usually have access to the polling evidence, while opinion polls do not take account of information contained in the betting odds (though the opinions expressed might, if voters are influenced by the betting odds). Sophisticated political bettors also take account of how good different pollsters are, what tends to happen to those who are undecided when they actually vote, differential turnout of voters, what might drive the agenda between the dates of the polling surveys and election day itself, and so on. All of this can in principle be captured in the markets.

Pollsters, except perhaps with their final polls (and sometimes even then) tend to claim that they are not producing a forecast, but a snapshot of opinion. This is the classic ‘snapshot defence’ wheeled out by the pollsters when things go badly wrong. In contrast, the betting markets are generating odds about the final result, so can’t use this questionable defence. In any case, polls are used by those trading the markets to improve their forecasts, so they are (or should be) a valuable input. But they are only one input. Those betting in the markets have access to much other information as well including, for example, informed political analysis, statistical modelling, focus groups and on-the-ground information including local canvass returns.

Does Big Data back up the anecdotal evidence? To test the reliability of the anecdotal evidence pointing to the superior forecasting performance of the betting markets over the polls, we collected vast data sets for a paper published in the Journal of Forecasting (‘Forecasting Elections’, 2016, by Vaughan Williams and Reade) of every matched contract placed on two leading betting exchanges and from a dedicated prediction market for US elections, since 2000. This was collected over 900 days before the 2008 election alone, and to indicate the size, a single data set was made up of 411,858 observations from one exchange alone for that year. Data was derived notably from presidential elections at national and state level, Senate elections, House elections, and elections for Governor and Mayor. Democrat and Republican selection primaries were also included. Information was collected on the polling company, the length of time over which the poll was conducted, and the type of poll. The betting was compared over the entire period with the opinion polls published over that period, and also with expert opinion and a statistical model. In this paper, as well as in Vaughan Williams and Reade – ‘Polls and Probabilities: Prediction Markets and Opinion Polls’, we specifically assessed opinion polls, prediction and betting markets, expert opinion and statistical modelling over this vast data set of elections in order to determine which performed better in term of forecasting outcomes. We considered accuracy, bias and precision over different time horizons before an election.

A very simple measure of accuracy is the percentage of correct forecasts, i.e. how often a forecast correctly predicts the election outcome. We also identified the precision of the forecasts, which relates to the spread of the forecasts. A related but distinctly different concept to accuracy is unbiasedness. An unbiased probability forecast is also, on average, equal to the probability that the candidate wins the election. Forecasts that are accurate can also be biased, provided the bias is in the correct direction. If polls are consistently upward biased for candidates that eventually win, then despite being biased they will be vey accurate in predicting the outcome, whereas polls that are consistently downward biased for candidates that eventually win will be very inaccurate as well as biased.

We considered accuracy, precision and bias over different time horizons before an election. We found that the betting/prediction market forecasts provided the most accurate and precise forecasts and were similar in terms of bias to opinion polls. We found that betting/prediction market forecasts also tended to improve as the elections approached, while we found evidence of opinion polls tending to perform worse.

In summary, we concluded that betting and prediction markets provide the most accurate and precise forecasts. We noted that forecast horizon matters: whereas betting/prediction market forecasts tend to improve nearer an election, opinion polls tend to perform worse, while expert opinion performs consistently throughout, though not as well as betting markets. There was also a systematic small bias against favourites, so that most likely outcome is actually usually a little more likely than suggested in the odds. Finally, if the polls and betting markets say different things, it is normally advisable to look to the betting markets.

So let’s turn again to why might we expect the betting markets to beat the polls. Most fundamentally, opinion polls, like all market research, provide a valuable source of information, but they are only one source of information, and some polls have historically been more accurate than others. Traders in the markets consider such things as what tends to happen to ‘undecideds’. Is there a late swing to incumbents or ‘status quo’? What is the likely impact of late endorsements by press or potential late announcements? Late on-the-day ‘tabloid press effect’, esp. on emotions. Influences undecideds, drives turnout to chosen editorial line. What is the likely turnout? What is the impact of differential turnout. Finally, sophisticated bettors take account of the relative accuracy of different polls and look behind the headline results to the detailed breakdown and the methodology used the poll. Betting markets should aggregate all the available information and analysis.

Moreover, people who know the most, and are best able to process the information, will tend to bet the most, but people who know only a little tend to bet only a little. The more money involved, or the greater the incentives, the more efficient and accurate will the market tend to be. It really is in this sense a case of “follow the money”.

Sometimes it is even possible to follow the money all the way to the future. To capture tomorrow’s news today. A classic example is the ‘Will Saddam Hussein be captured or neutralised by the end of the month’ Intrade exchange market? Early on 13 December, 2003, the market moved from 20 (per cent chance) to 100. The capture was announced early on 14 December, 2003, and officially took place at 20:30 hours Iraqi time, several hours after the Intrade market moved to 100. I call these, with due deference to Star Trek,  ‘Warp speed markets’.

But we need to be cautious. With rare exceptions, betting markets don’t tell us what the future will be. They tell us at best what the probable future will be. They are, in general, not a crystal ball. And we need to be very aware of this. Even so, the overwhelming consensus of evidence prior to the 2015 UK General Election pointed to the success of political betting markets in predicting the outcome of elections.

And then the tide turned.

The 2016 EU referendum in the UK (Brexit), the 2016 US presidential election (Trump) and the 2017 UK General Election (No overall majority) produced results that were a shock to the great majority of pollsters as well as to the betting markets. The turning of the tide could be traced, however, to the Conservative overall majority in 2015, which came as a shock to the markets and pollsters alike. After broadly 150 years of unparalleled success for the betting markets, questions were being asked. The polls were equally unsuccessful, as were most expert analysts and statistical models.

The Meltdown could be summarised in two short words. Brexit and Trump. Both broadly unforeseen by the pollsters, pundits, political scientists or prediction markets. But two big events in need of a big explanation. So where did it all go wrong?  There are various theories to explain why the markets broke down in these recent big votes.

Theory 1: The simple laws of probability. An 80% favourite can be expected to lose one time in five, if the odds are correct. In the long run, according to this explanation, things should balance out. It’s like there are five parallel universes. The UK on four of the parallel universes votes to Remain in the EU, but not in the fifth.Hillary Clinton wins in four of the parallel universes but not in the fifth. In other words, it’s just chance, no more strange than a racehorse starting at 4/1 winning the race. But for that to be a convincing explanation, it would need to assume that 2015 election, Brexit, Trump and 2017 election were totally correlated. Even if there is some correlation of outcome, the markets were aware of each of the predictive failures in the previous votes and still favoured the losing outcome by a factor of 4 or 5 to 1. That means we can multiply the probabilities. 1/5×1/5×1/5×1/5 = 1/625.   1/6×1/6×1/6×1/6 = 1/1296. Either way, its starting to look unlikely.

Theory 2: A second theory to explain recent surprise results is that something fundamental has changed in the way that information contained in political betting markets is perceived and processed. One interpretation is that the hitherto widespread success of the betting markets in forecasting election outcomes, and the publicity that was given to this, turned them into an accepted measure of the state of a race, creating a perception which was difficult to shift in response to new information. This is a form of ‘anchoring’. To this extent, market prices to some extent led opinion rather than simply reflecting it. From this perspective, the prices in the markets became a yardstick of the true probabilities and thus somewhat inflexible in response to the weight of new information.This leads to the herding hypothesis. Because the prediction markets had by 2015 become so firmly entrenched in conventional wisdom as an accurate forecasting tool, people herded around the forecasts, propelling the implied probabilities of existing forecasts upwards. So a 55% probability of victory, for example, became transformed into something much higher. In consequence, a prediction market implied probability of 70%, say, might be properly adjusted to a true probability of, say, 55%. In principle, it is possible to de-bias (or de-herd) each prediction market probability into a more accurate adjusted probability. We also need to look at the idea of self-reinforcing feedback loops. City traders look to the betting exchanges and the fixed-odds and spread bookmakers’ odds for evidence of what is the true state of play in each race. That influences the futures markets, which in turn influences perceptions among bettors. A sort of prediction market loop, in which expectations become self-reinforcing. This is a form of ‘groupthink’ in which those trading the futures and prediction markets are taking the position they are simply because others are doing so. This is further reinforced by the key arbitrating divide which more than anything acts as a distinguishing marker between Brexit supporters and Remain supporters, between Trump voters and Hillary Clinton voters – educational level. More than any other factor, it is the ‘University education’ marker that identifies the Remain voter, the Clinton voter. Also, the vast majority of City traders as well as betting exchange traders are University-educated, and tend to mix with similar, which may have reinforced the perception that Trump and Brexit were losing tickets. Indeed, more than ever before, as the volume of information increases, and people’s ability to sort between and navigate and share these information sources increases, there is a growing disjoint between the information being seen and processed by different population silos. This is making it increasingly difficult for those inhabiting these different information universes to make any sense of what is driving the preferences of those in alternative information universes, and therefore engaging with them and forming accurate expectations of their likely voting behaviour and likelihood of voting. The divide is increasingly linked to age and educational profile, reducing the diversity of opinion which is conventionally critical in driving the crowd wisdom aspect of prediction markets. It also helps explain the broad cluelessness of the political and political commentating classes in understanding and forecasting these event outcomes. Of course, the pollsters, pundits, political scientists and politicians were broadly speaking just as clueless. So why?

Theory 3: Conventional patterns of voting broke down in 2015 and subsequently, primarily due to unprecedented differential voter turnout patterns across key demographics, which were not correctly modelled in most of the polling and which were missed by political pundits, political scientists, politicians and those trading the betting markets. In particular, there was unprecedented turnout in favour of Brexit and Trump by demographics that usually voted in relatively low numbers, notably the more educationally disadvantaged sections of society. And this may be linked to a breakdown of the conventional political wisdom. This wisdom holds that campaigns don’t matter, that swings of support between parties are broadly similar across the country, that elections can only be won from the centre, and that the so-called ‘Overton window’ must be observed. This idea, conceived by political scientist Joseph Overton, is that for any political issue there’s a range of socially acceptable and broadly tolerated positions (the ‘Overton window’) that’s narrower than the range of possible positions. It’s an idea which in a Brexit/Trump age seems to have gone very much out of the window.

Theory 4: Manipulation. Robin Hanson and Ryan Oprea co-authored a paper titled, ‘A Manipulator Can Aid Prediction Market Accuracy‘, in a special issue of Economica in 2009 which I co-edited. Manipulation can actually improve prediction markets, they argue, for the simple reason that manipulation offers informed investors a proverbial ‘free lunch.’ In a stock market, a manipulator sells and buys based on reasons other than expectations and so offers other investors a greater than normal return. The more manipulation, therefore, the greater the expected profit from betting. For this reason, investors should soon move to take advantage of any price discrepancies thus created within and between markets, as well as to take advantage of any perceived mispricing relative to fundamentals. Thus the expected value of the trading is a loss for the manipulator and a profit for the investors who exploit the mispricing. Manipulation creates liquidity, which draws in informed investors and provides the incentive to acquire and process further information, which makes the market ever more efficient.

Theory 5: Fake News. There are other theories, which may be linked to the demographic turnout theory, including notably the impact of misinformation (fake news stories), of hacked campaign email accounts, and direct manipulation of social media accounts. In fact, we know when it all started to go wrong. That was 7th May, 2015, when the Conservatives won an unforeseen overall majority in the General Election. That result led to Brexit. That in turn arguably helped propel Trump to power. And it led to the shock 2017 UK election result. Common to all these unexpected outcomes is the existence of a post-truth misinformation age of ‘fake news’ and the potential to exploit our exposure to social media platforms by those with the money, power and motivation to do so. The weaponisation of fake news might explain the breakdown in the forecasting power of the betting markets and pollsters, commencing in 2015, as well as the breakdown of the traditional forecasting methodologies in predicting Brexit and Trump. This has in large part been driven by the power of fake news distribution and the targeting of such via social media platforms, to alter traditional demographic turnout patterns. This is by boosting turnout among certain demographics and suppressing it among others. The weaponisation of fake news by the tabloid press is of course nothing new but it has become increasingly virulent and sophisticated and its online presence amplifies its reach and influence. The weaponisation of fake news by the tabloid press can also help explain on-the-day shifts in turnout patterns.

What it does not explain is some very odd happenings in recent times. Besides Brexit and Trump, Leicester City became 5.000/1 winners of the English Premier League. The makers and cast of La La Land had accepted the Oscar for Best Picture before it was snatched away in front of billions to be handed to Moonlight. This only echoed the exact same thing happening to Miss Venezuela when her Miss Universe crown was snatched away after her ceremonial walk to be awarded to Miss Philippines.  And did the Atlanta Falcons really lose the SuperBowl after building an unassailable lead? And did the BBC Sports Personality of the Year Award go to someone whose chance of winning was so small he didn’t even turn up to the ceremony, while the 1/10 favourite was beaten by a little-known motorcyclist and didn’t even make the podium.  Which leads us to Theory 6.

Theory 6: We live in a simulation. In the words of a New Yorker columnist in February 2017: “Whether we are at the mercy of an omniscient adolescent prankster or suddenly the subjects of a more harrowing experiment than any we have been subject to before … we can now expect nothing remotely normal to take place for a long time to come. They’re fiddling with our knobs, and nobody knows the end.”

So maybe the aliens are in control in which case all bets are off. Or have we simply been buffeted as never before by media manipulation and fake news? Or is it something else? Whatever the truth, we seem to be at the cusp of a new age. We know not yet which way that will lead us. Hopefully, the choice is still in our hands.