It was a puzzle first posed by the gifted Swiss mathematician, Nicolas Bernoulli, in a letter to Pierre Raymond de Montmort, on Sept. 9, 1713, and published in the Commentaries of the Imperial Academy of Science of St. Petersburg. Mercifully it is simple to state. Less mercifully, it is a nightmare to solve. To state the paradox, imagine tossing a coin until it lands heads-up, and suppose that the payoff grows exponentially according to the number of tosses you make. If the coin lands heads-up on the first toss, then the payoff is £2; if it lands heads-up on the second toss, the payoff is £4; if it takes three tosses, the payoff is £8; and so forth, ad infinitum. Now the odds of the game ending on the first toss is ½; of it ending on the second toss is (1/2)^2 = ¼; on the third, (1/2)^3 = 1/8, etc., so your expected win from playing the game = (1/2 x £2) + (1/4 x £4 + 1/8 x £8) + …, i.e. £1 + £1 + £1 … = infinity. It follows that you should be willing to pay any finite amount for the privilege of playing this game. Yet it seems irrational to pay very much at all. So what is the solution? There have been very many attempts at a solution over the years, some more satisfying than others, but none totally so. For the best attempt to date, I think we should go back to 1923, and the classic Moritz explanation, offered by R.E. Moritz, writing in the American Mathematical Monthly. “The mathematical expectation of one chance out of a thousand to secure a billion dollars is a million dollars, but this does not mean that anyone in his senses would pay a million dollars for a single chance of winning a billion dollars”. Of the more recent attempts, I like best that offered by Benjamin Hayden and Michael Platt, in 2009. “Subjects … evaluate [the gamble] … as if they were taking the median rather than the mean of the payoff distribution … [so] this classic paradox has a straightforward explanation rooted in the use of a statistical heuristic.” Surprisingly, though, there is still no real consensus on the solution to this puzzler of more than three hundred years vintage. If and when we do finally solve it, we will have made a giant step toward establishing a more complete and precise understanding of the meaning of rationality and the working of the human economic mind. Care to try? References Moritz, R. E. (1923). Some curious fallacies in the study of probability. The American Mathematical Monthly, 30, 58–65. Hayden B. and Platt, M. (2009). The mean, the median, and the St. Petersburg Paradox. Judgment and Decision Making, 4, no. 4, June, 256-272. Link: http://journal.sjdm.org/9226/jdm9226.html
Now that Jeb Bush has officially announced he is seeking the nomination of the Republican party as its candidate for president of the United States, it seems a good time to ask how likely it is that he will actually become the 45th US President. How can we best answer this? A big clue can be found in a famous study of the history of political betting markets in the US, which shows that of the US presidential elections between 1868 and 1940, in only one year, 1916, did the candidate favoured in the betting end up losing, when Woodrow Wilson came from behind to upset Republican Charles E. Hughes in a very contest. Even then, they were tied in the betting by the close of polling. The power of the betting markets to assimilate the collective knowledge and wisdom of those willing to back their judgement with money has only increased in recent years as the volume of money wagered has risen dramatically, the betting exchanges alone seeing tens of millions of pounds trading on a single election. In 2004, a leading betting exchange actually hit the jackpot when its market favourite won every single state in that year’s election. The power of the markets has been repeated in every presidential election since. For example, in 2008, the polls had shown both John McCain and Barack Obama leading at different times during the campaign, while the betting markets always had Obama as firm favourite. Indeed, on polling day, he was as short as 20 to 1 on to win with the betting exchanges, while some polling still had it well within the margin of error. In the event, Obama won by a clear 7.2%. By 365 Electoral College Votes to 173. In 2012, Barack Obama led Mitt Romney by just 0.7% in the national polling average on election day, with major pollsters Gallup and Rasmussen showing Romney ahead. British bookmakers were quoting the president at 5 to 1 on (£5 to win £1). Indeed, Forbes reflected the view of most informed observers, declaring that: With one day to go before the election, we’re becoming super-saturated with poll data predicting a squeaker in the race for president. Meanwhile, bookmakers and gamblers are increasingly certain Obama will hang on to the White House. He went on to win by 3.9% and by 332 Electoral College Votes to 206. What is happening here is that the market is tapping into the collective wisdom of myriad minds who feed in the best information and analysis they can because their own financial rewards depend directly upon this. As such, it is a case of “follow the money” because those who know the most, and are best able to process the available information, tend to bet the most. Moreover, the lower the transaction costs (in the UK the betting public do not pay tax on their bets) and information costs (in never more plentiful supply due to the Internet) the more efficient we might expect betting markets to become in translating information today into forecasts of tomorrow. For these reasons, modern betting markets are likely to provide better forecasts than they have done ever before. In this sense, the market is like one mind that combines the collective wisdom of everybody. So what does this brave new world of prediction markets tell us about the likely Republican nominee in 2016? Last time, they were telling us all along that it would be Mitt Romney. This time the high-tech crystal ball offered by the markets is seeing not one face, but three, and yes – Jeb Bush is one of them. But two other faces loom large alongside the latest incarnation of the Bush dynasty. One is Florida senator, Marco Rubio, and the other is the governor of Wisconsin, Scott Walker. According to the current odds, at least, it is very likely that one of these men will be the Republican nominee. According to the betting, Bush will struggle to win the influential Iowa caucus, which marks the start of the presidential election season. The arch conservative voters there are expected to go for the man from Wisconsin. New Hampshire, the first primary proper, is likely to be closer. Essentially, though, this will be a contest between the deep pockets and connections of the Bush machine, the deep appeal of Scott Walker tot the “severely conservative” (a phrase famously coined by Mitt Romney), and the appeal of Marco Rubio to those looking for solid conservative credentials matched with relative youth and charisma. By the time the race is run, the betting markets currently indicate that Bush is the name that is most likely (though by no means sure) to emerge. Rubio is likely to push him hardest – and it could be close. At current odds, though, Bush does have the best chance of all the candidates in the field of denying the Democrats, and presumably Hillary Clinton the White House. But whoever is nominated by the Republican Party, it is the Democrats who are still firm favourites to retain the keys to Washington DC’s most prestigious address. Note A version of this blog, with links to sources, first appeared in The Conversation UK on June 15, 2015. https://theconversation.com/jeb-bush-dives-into-the-presidential-race-ask-the-betting-markets-how-hell-do-43300
Why did the Conservatives win an overall (albeit narrow) majority in the 2015 UK Election, and almost a hundred more seats than Labour? Numerous hypotheses have been put forward, often centred around the ideas of leadership, economic competence and attracting the ‘aspirational’ voters of ‘middle England.’ If this analysis is correct, it tells Labour something very important about the ground their next leader will need to fight on, and indeed who that leader should be. But is this the whole picture?
This is where the opinion polls can in fact tell us something important. There has certainly been much discussion since the election results were declared about weaknesses in their survey design, but in itself that does not seem sufficient to explain the huge disparity in what actually happened at the polling stations (Tories ahead of Labour by 6.5%) and what happened in the polls (essentially tied). I argue here that a big part of the reason for this disparity is what I term the ‘lethargic Labour‘ effect, i.e. the differential tendency of Labour supporters to stay at home compared to Tory supporters. ‘Lethargic’ is a term I choose carefully for its association with apathy and general passivity, and it is a factor which I believe has huge implications for political strategy in the years ahead.
To understand this, it is instructive to look to the exit poll, which was conducted at polling stations with people who had actually voted. This was much more accurate than the other polls, including those conducted during Election Day over the telephone or online, and showed a much lower Labour share of the vote. A dominant explanation for this disparity is that there was a significant difference in the number of those who declared they had voted Labour or that they would vote Labour and those who actually did vote.
This ‘lethargic Labour’ effect is quite different to the so-called ‘shy Tory’ effect which was advanced as part of the explanation for the polling meltdown of 1992, when the Conservatives in that year’s General Election were similarly under-estimated in the opinion polls. This ‘shy Tory’ effect is the idea that Tories in particular were shy of revealing their voting intention to pollsters. Yet in 2015 we would expect, if this were a real effect, to have seen it displayed in under-performance by the Tories in telephone polls compared to the relatively more anonymous setting of online polls. There is no such evidence, if anything the reverse being the case for much of the polling cycle.
I am not proposing that the idea of ‘lethargic Labour’ supporters offers the whole explanation for the Tory victory. There is also a historically well-established late swing to incumbents, which cannot be blamed on the raw polls, but is sometimes built into poll-based forecasting models which can account for some of the differential, and there is additionally late tactical switching to consider, where an elector, when face to face with an actual ballot paper, casts a vote to hinder a least preferred candidate.
Interestingly, the betting markets significantly out-performed the polls and also sophisticated modelling based on those polls which allowed for late swing, but they beat the latter somewhat less comprehensively, at least at constituency level. At national aggregated level, the betting markets beat both very convincingly, though the swing-adjusted polls performed rather better than the published polls.
So what does this tell us? It suggests that there was indeed a late swing to the Tories, as well as probably a late tactical swing, both of which were picked up in the betting markets in advance of the actual poll. But the scale of the victory (at least compared to general expectations) was not fully anticipated by any established forecasting methodology. This suggests that there was an extra variable, which was not properly factored in by any forecasting methodology. This extra variable, I suggest, is the ‘lethargic Labour’ supporters, who existed in far greater numbers than was generally supposed.
To the extent that this explanation of the Tory majority prevails, it has profound implications for the strategy of the Labour Party over the next few years in seeking to win office.
It tells us that if Labour are to win the next election, a strategy will have to be devised which motivates their own supporters to actually turn out and vote. In other words, a strategy must be devised which attracts these ‘lapsed Labour’ voters, as I term them, into active Labour voters, which inspires the faithful to get out of their armchairs and into the polling pews. If they can’t construct an effective strategy to do that, it doesn’t really matter how effective their leader is, how economically competent they are seen to be, how well they appeal to the ‘aspirational’ voter. It is very unlikely that Labour will be able to win.
In summary, the Labour Party will need to motivate their more ‘lethargic’ supporters to actually show that support in the ballot box, will need to convert their supporters from being ‘lapsed’ voters into actual voters. If they can do that, the result of the next election is wide open.
If the opinion polls had proved accurate, we would have been woken up on the morning of May 8, 2015, to a House of Commons in which the Labour Party had quite a few more seats than the Conservatives, and by the end of the day the country would have had a new Prime Minister called Ed Miliband. This didn’t happen. Instead the Conservative Party was returned with almost 100 more seats than Labour and a narrow majority in the Commons. So what went wrong? Why did the polls get it so wrong?
This is not a new question. The polls were woefully inaccurate in the 1992 General Election, predicting a Labour victory, only for John Major’s Conservatives to win by a clear seven percentage points. While they had performed a bit better since, history repeated itself this year.
So what is the problem and can it be fixed? A big issue, I believe, is the methodology used. Pollsters simply do not make any effort to duplicate the real polling experience. Even as Election Day approaches, they very rarely identify to those whom they survey who the candidates are, instead simply prompting party labels. This tends to miss a lot of late tactical vote switching. Moreover, the filter they use to determine who will actually vote as opposed to say they will vote is clearly faulty, which can be seen if we compare the actual voter turnout figures with those projected in the polling numbers. Almost invariably, they over-estimate how many of those who say they will vote do actually vote. Finally, the raw polls do not make allowance for what we can learn from past experience as to what happens when people actually make the cross on the ballot paper compared to their stated intention. We know that there tends to be a late swing to the incumbents in the privacy of the polling booth. For this reason, it is wise to adjust the raw polls for this late swing.
Of all these factors, which was the main cause of the polling meltdown? For the answer, I think we need just look to the exit poll, which was conducted at polling stations with people who had actually voted. This exit poll, as in 2010, was quite accurate, while similar exit-style polls conducted during polling day over the telephone or online with those who declared they had voted or were going to vote failed pretty much as spectacularly as the other final polls. The explanation for this difference can, I believe, be traced to the significant difference in the number of those who declare they have voted or that they will vote and those who actually do vote. If this difference works particularly to the detriment of one party compared to another, then that party will under-perform in the actual vote tally relative to the voting intentions declared on the telephone or online. In this case, it seems a very reasonable hypothesis that rather more of those who declared they were voting Labour failed to actually turn up at the polling station than was the case with declared Conservatives. Add to that late tactical switching and the well-established late swing in the polling booth to incumbents and we have, I believe, a large part of the answer.
Interestingly, those who invested their own money in forecasting the outcome performed a lot better in predicting what would happen than did the pollsters. The betting markets had the Conservatives well ahead in the number of seats they would win right through the campaign and were unmoved in this belief throughout. Polls went up, polls went down, but the betting markets had made their mind up. The Tories, they were convinced, were going to win significantly more seats than Labour.
I have interrogated huge data sets of polls and betting markets over many, many elections stretching back years and this is part of a well-established pattern. Basically, when the polls tell you one thing, and the betting markets tell you another, follow the money. Even if the markets do not get it spot on every time, they will usually get it a lot closer than the polls.
So what can we learn going forward? If we want to predict the outcome of the next election, the first thing we need to do is to accept the weaknesses in the current methodologies of the pollsters, and seek to correct them, even if it proves a bit more costly. With a limited budget, it is better to produce fewer polls of higher quality than a plethora of polls of lower quality. Then adjust for known biases. Or else, just look at what the betting is saying. It’s been getting it right since 1868, before polls were even invented, and continues to do a pretty good job.
Updated June 1, 2015. This was the most polled election in British history, and most projections based on the polls suggested that Labour would finish with most seats in the House of Commons. But there is another way to predict elections, by looking at the bets made by people gambling on them. The betting markets were always saying that the Conservatives would win a lot more seats than Labour at the election of May, 2015. So where should we be looking for our best estimate of what is actually going to happen in an eelction, to the polls or to the markets? It’s a question that we have been considering actively in the UK for nearly 30 years. We can trace the question to July 4, 1985, for that is the day that the political betting markets finally came of age in this country. A by-election was taking place in the constituency of Brecon and Radnor, a semi-rural corner of Wales, and at the time the key players, according to both the betting markets and the opinion polls, were the Labour and Liberal candidates. Ladbrokes were making the Liberal the odds-on favourite. But on the very morning of the election a poll by MORI gave the Labour candidate a commanding 18% lead. Meanwhile, down at your local office of Ladbrokes the Liberal stubbornly persisted as the solid odds-on favourite. So we had the bookmaker saying black and the pollster white, or more strictly yellow and red. And who won? It turned out to be the Liberal, and of course anyone who ignored the pollster and followed the money. Since then, the betting markets have called it correctly in every single UK general election. While this may be a surprise to many, it will be much less so to those who had followed the history of political betting markets in the US, which correctly predicted (according to a famous study) almost every single US Presidential election between 1868 and 1940. In only one year, 1916, did the candidate favoured in the betting the month before the election end up losing, and that in a very tight race. The power of the betting markets to assimilate the collective knowledge and wisdom of those willing to back their judgement with money has only increased in recent years as the volume of money wagered has risen dramatically, the betting exchanges alone seeing tens of millions of pounds trading on a single election. Indeed, in 2004 one betting exchange actually hit the jackpot when their market favourite won every single state in that year’s election. This is like a tipster calling the winner of 50 football matches in a row simply by naming the favourite. The power of the markets has been repeated in every Presidential election since. Two weeks before the 2005 UK general election, buoyed already by the track record of the markets in forecasting UK elections, and that 2004 prediction miracle, I was sufficiently confident, when asked by The Economist, to call the winner and the seat majority in the 2005 UK General Election over two weeks out. My prediction of a 60-seat majority for the Labour Party, repeated in an interview on the BBC Today programme, was challenged in a BBC World Service debate by a leading pollster, who wanted to bet me that his figure of a Labour majority of over 100 was a better estimate. I declined the bet and saved him some money. The Labour majority was 66 seats. The assumption here is that the collective wisdom of many people is greater than the conclusions of a few. Those myriad people feed in the best information and analysis they can because their own financial rewards depend directly upon this. And it really is a case of ‘follow the money’ because those who know the most, and are best able to process the available information, tend to bet the most. Moreover, the lower the transaction costs (the betting public do not pay tax on their bets in the UK) and information costs (in never more plentiful supply due to the Internet) the more efficient we might expect betting markets to become in translating information today into forecasts of tomorrow. For these reasons, modern betting markets are likely to provide better forecasts than they have done ever before. This is not to say that betting markets are always exactly right, and pollsters are always hopelessly wrong, but when they diverge the overwhelming weight of evidence suggests that it is to the betting markets that we should turn. The same happened in 2010, where the betting markets for weeks were strongly predicting a hung parliament, while the polls swung from at one point having the Liberal Democrats in the lead to in another case putting the Tories a whole twelve points up on election day.
Indeed, in an event as big and recent as the 2012 US presidential election, no less a name than Gallup called it for Mitt Romney, and the national polling average had the candidates essentially tied. Meanwhile, Barack Obama was very short odds-on to win.
Last year’s Scottish referendum is another example. While the polls had it very tight, and with more than one poll calling it for independence, the betting markets were always pointing solidly to a No. The mismatch between the polls and the result echoed the 1995 Quebec separation referendum in Canada. There the final polling showed ‘YES to separation’ with a 6% lead. In the event, ‘No to separation’ won by 1%. We happen to know that one very large trader in the Scottish referendum markets had this, among other things, very much in mind, when he placed his £900,000 to win a net £193,000. The point is that people who bet in significant sums on an election outcome will usually have access to all the polling evidence and their actions take into account past experience of how good different pollsters are, what tends to happen to those who are undecided when they actually vote, and even sophisticated consideration of what might drive the agenda between the dates of the latest polling surveys and polling day itself. All of this is captured in the markets but not in the polls. To some this is magic. For example, I was at a conference in the US when an American delegate, totally dumbfounded that we are allowed to bet on this type of thing in the UK, and that we would anyway be mad enough to do so, asked me who would win the Greek election. I only needed to spend 30 seconds on my iPhone to tell her that Syriza were sure things. She couldn’t believe a market could tell me that. The same thing happened when I announced two weeks before the recent election in Israel that Netanyahu was already past the post. The polls pointed the other way, but the global prediction markets had Benjamin Netanyahu as a clear and strong favourite to win. On declaring victory, he declared that he had won against the odds. Not so. He had in fact won against the polls. To be fair to the opinion polls, they were onside in the Greek election, as they were in the French and Australian elections. That is good. The real question, though, is whom to believe when they diverge. In those cases, there is very solid evidence, derived from the interrogation of huge data sets of polls and betting trades going back many years, much of which I have undertaken myself, that the markets prevail. To be still fairer to the pollsters, they are not claiming to be producing a forecast. They are measuring a snapshot of opinion, though we are have to be careful about his ‘snapshot defence’, as I term it, as sometimes this can be used as cover for a poor methodology. In any case, those inhabiting the betting markets are certainly trying to produce a forecast, so we would to that extent hope that they would be better at it. Moreover, the polls are used by those trading the markets to improve their forecasts, so they are potentially a valuable input. But they are only one input. Those betting in the markets have access to so much other information as well, including informed political analysis, statistical modelling, canvass returns, and so on. I say that the polls are potentially a valuable input. The most recent election in the UK, on May 7, 2015, demonstrated that this is certainly not always the case. In that election, the polls, even on the day, showed it neck and neck in vote share, consistent with the Labour Party winning significantly more seats. The betting markets, meanwhile, always had the Conservatives well ahead in the seats tally, and were unmoved even by last minute polls indicating a late swing to Labour. Take a comparison of the polls and betting markets three days before polling. The polls were indicating most seats for Labour while the betting markets had the Conservatives as short as 6 to 1 on favourites to win most seats, i.e. £6 to win £1. That’s confidence. In other words, those who put their money on the line were pretty sure what was going to happen all along. Since then we have had the ‘gay marriage’ referedum in Ireland. The polls were pointing to a decisive 70%-30% win for YES. The betting markets had the line at 60-40. In the event, YES won by 62-38. In conclusion, there is a growing belief that betting markets will become more than just a major part of our future. Properly used they will, more importantly, be able to tell us what that future is likely to be. We seem therefore to have created, almost by accident, a ‘high-tech’ crystal ball that taps into the accumulated expertise of mankind and makes it available to all. In this sense, the market is like one mind that combines the collective wisdom of everybody. In this brave new world of prediction markets, it seems only sensible to make the most of it.
Follow on Twitter: @leightonvw
If you add up 1 and 2, what do you get? The answer is 3. Ok. Let’s go one step further. What if you add up 1 and 2 and 3? What do you get now? Now the answer is 6. Now 1 plus 2 plus 3 plus 4. That sums to 10. Now what if I do this for ever, in other words add up all the natural numbers right to infinity? What do I get? Most people say it is infinity. Mathematicians often say that there is no sum because technically you can’t sum a ‘divergent series’, as opposed to a series which converges to a number (such as 1+1/2+1/4+1/8+…, which converges to 2).
But let’s be ambitious and see where we get.
Let’s start simple and add up the following series:
1-1+1-1+1-1+1-1+1-1+… to infinity.
What is this?
If you stop at an odd step in the series, such as the first or third or fifth step, the series sums to 1. But if you stop at an even step, say the second or fourth or sixth, the series sums to 0. Both are equally likely, so it is intuitively obvious that we can take the average of 1 and 0, which is 0.5 as the solution of this equation.
For those who aren’t convinced by the obvious, however, we can show it a little more rigorously like this:
Let S = 1-1+1-1+1-1 …….
So, 1-S = 1 – (1-1+1-1+1-1 …) = 1-1+1-1-1-1…
So, 1-S = S
Therefore, S = ½.
We can also show it by the method of averaging partial sums, which I’ve added in an appendix to this post, as the third method.
So there are three different ways to demonstrate that the series: 1-1+1+1-1+1-… equals 0.5.
Now that we have established this, the task of calculating the solution to:
1+2+3+4+5… becomes quite straightforward.
So we have established that 1-1+1-1+1-1+… = ½
We’ll call this series S1.
But what if we want to calculate S2, which is the series 1-2+3-4 + ….?
The way to do this is to add it to itself, to get 2.S2
1-2+3-4 +5- … + (1-2+3-4+5…)
The easiest way to do this is to move the second series one step along, which is fine as it is an infinite series. The start with 1 and now add up each pair of the remaining terms.
So we get:
1 + (-2+1) + (3-2) + (-4+3) + (5-4) + … = 1-1+1-1+1 ………….
But we have seen this series before. It is S1, and is equal to ½.
So, 2.S2 = ½
Therefore, S2 = ¼
Now what we are trying to sum is 1+2+3+4+5+6+…….
Let us call this S.
So, S – S2 = 1+2+3+4+5+6+… – (1-2+3-4+5-6…)
So, S-S2 = 0+4+0+8+0+12+…
This series is identical to: 4+8+12+16+20+24+…
This is 4 x (1+2+3+4+5+6+…)
In other words, S-S2 = 4S
We know already that S2 = ¼
So, 3S = -1/4
And that is the proof that the sum of all the natural numbers up to infinity equals -1/12.
Who said it’s infinity? Who said you can’t sum divergent series? It’s got a solution and it’s the only meaningful one. Add up all the positive integers up to infinity and you get a negative number, -1/12.
There is no mathematical sleight of hand here. It is a properly derived solution, and we know from everything we understand about the laws of modern physics that it works in explaining the real world.
My next question is to ask you whether infinity is odd or even. What happens if I press the number 1 after 1 minute, then zero after a further 30 seconds, then 1 again after a further 15 seconds, then zero after a further 7.5 seconds and so on. What number am I pressing at the precise end of two minutes? Am I pressing 1 or zero, or both simultaneously, or neither. Imagine this was a magic light bulb that never blew. At the end of precisely 2 minutes, would it be on or off or both on and off? Or would it be -1/12 on and +1/12 off, or vice-versa.
Makes you think!
Thanks, by the way, to the excellent people on the Numberphile channel on YouTube, who inspired my interest in this.
In the series, 1-1+1-1+1-1 …
The first term = 1.
The sum of the first and second terms = 1-1 = 0
The sum of the first, second and third terms = 1-1+1=1
The sum of the first, second, third and four terms = 1-1+1-1 =0
And so on.
So the series of these sums (known as partial sums) = 1,0,1,0 …
Now, averaging these partial sums gives the following series:
1 divided by 1, 1+0 divided by 2, 1+0+1 divided by 3, 1+0+1+0 divided by 4, etc.
This works out as:
1, ½, 2/3, ½, 3/5, ½, 4/7 …
If we continue this series, we end up with ½, as the odd numbered terms get ever smaller, and eventually vanishingly so, leaving us just with 1/2.
This method of averaging partial sums to derive the total sum is well-established and can be used, for example, to calculate the sum of 1+1/2+1/4+1/8+ …
This can be shown to converge on 2 using the same method.