A version of this post was published in the Daily Telegraph on March 9, 2015.
This is likely to be the most polled election in British history, and most projections based on the latest polls are suggesting that Labour would probably finish with most seats in the House of Commons. But there is another way to predict elections, by looking at the bets made by people gambling on them. The betting markets are currently suggesting that David Cameron will still be in No. 10 after the election in May. So where should we be looking for our best estimate of what is actually going to happen, to the polls or to the markets? It’s a question that we have for been considering in the UK for nearly 30 years.
We can trace the question to July 4, 1985, for that is the day that the political betting markets finally came of age in this country. A by-election was taking place in a semi-rural corner of Wales, and at the time the key players, according to both the betting markets and the opinion polls, were the Labour and Liberal candidates. Ladbrokes were making the Liberal the odds-on favourite. But on the very morning of the election a poll by MORI gave the Labour candidate a commanding 18% lead. Meanwhile, down at your local office of Ladbrokes the Liberal stubbornly persisted as the solid odds-on favourite. So we had the bookmaker saying black and the pollster white, or more strictly yellow and red. And who won? It turned out to be the Liberal, and of course anyone who ignored the pollster and followed the money.
While this was a surprise to many, it was much less so to those who had followed the history of political betting markets in the US, which correctly predicted (according to a famous study) almost every single US Presidential election between 1868 and 1940. In only one year, 1916, did the candidate favoured in the betting the month before the election, Charles E. Hughes, end up losing, and that in a very tight race.
The power of the betting markets to assimilate the collective knowledge and wisdom of those willing to back their judgement with money has only increased in recent years as the volume of money wagered has risen dramatically, the betting exchanges alone seeing tens of millions of pounds trading on a single election. Indeed, in 2004 one betting exchange actually hit the jackpot when their market favourite won every single state in that year’s election. This is like a racing tipster calling the winner of 50 races in a row simply by naming the race favourite. The power of the markets has been repeated in every Presidential election since.
Two weeks before the 2005 UK general election, buoyed already by that 2004 prediction miracle, I was sufficiently confident, when asked by The Economist, to call the winner and the seat majority in the 2005 UK General Election over two weeks out. My prediction of a 60-seat majority for the Labour Party, repeated in an interview on the BBC Today programme, was challenged in a BBC World Service debate by a leading pollster, who wanted to bet me that his figure of a Labour majority of over 100 was a better estimate. I declined the bet and saved him some money. The Labour majority was a little over 60 seats.
The assumption here is that the collective wisdom of many people is greater than the conclusions of a few. Those myriad people feed in the best information and analysis they can because their own financial rewards depend directly upon this. And it really is a case of ‘follow the money’ because those who know the most, and are best able to process the available information, tend to bet the most.
Moreover, the lower the transaction costs (the betting public do not pay tax on their bets in the UK) and information costs (in never more plentiful supply due to the Internet) the more efficient we might expect betting markets to become in translating information today into forecasts of tomorrow. For these reasons, modern betting markets are likely to provide better forecasts than they have done ever before.
This is not to say that betting markets are always right, and pollsters are always wrong. For example, the betting markets over-estimated the number of seats the Liberal Democrats would win in 2010, though they were spot on in forecasting that no party would win an overall majority. Again, on election night, the exit poll in 2010 was very close to the actual result, while the betting markets took fright at a couple of early declarations. Similarly, statistician and polling analyst Nate Silver was very accurate (at least on the day) in his 2012 US election forecasts, by weighting all the polls by their past accuracy and compiling a composite forecast. Even so, individual pollsters were all over the place. No less a name than Gallup called it for Mitt Romney.
On the whole, though, when the betting markets say one thing and the polls say another, the evidence suggests that it is a good idea to go with the markets. Last year’s Scottish referendum is another example. While the polls had it very tight, and with more than one poll calling it for independence, the betting markets were always pointing to a No.
The mismatch between the polls and the result echoed the 1995 Quebec separation referendum in Canada. There the final polling showed ‘YES to separation’ with a 6% lead. In the event, ‘No to separation’ won by 1%. We happen to know that one very large trader in the Scottish referendum markets had this, among other things, very much in mind.
People who bet in significant sums on an election outcome will usually have access to all the polling evidence and their actions take into account past experience of how good different pollsters are, what tends to happen to those who are undecided when they actually vote, and even sophisticated consideration of what might drive the agenda between the dates of the latest polling surveys and polling day itself. All of this is captured in the markets but not in the polls.
To some this is magic. For example, I was recently at a conference in the US when an American delegate, totally dumbfounded that we are allowed to bet on this type of thing in the UK, and that we would anyway be mad enough to do so, asked me who would win the Greek election. I only needed to spend 30 seconds on my iPhone to tell her that Syriza were as good as past the post. She couldn’t believe a market could tell me that.
To be fair to the opinion polls, they were onside in the Greek election, as they were in the French and Australian elections. That is good. The real question, though, is whom to believe when they diverge. In those cases, there is very solid evidence, derived from the interrogation of huge data sets of polls and betting trades, going back many years, that overall the markets prevail.
To be still fairer to the pollsters, they are not claiming to be producing a forecast. They are measuring a snapshot of opinion. Those inhabiting the betting markets are genuinely trying to produce a forecast, so we would hope that they would be better at it. Moreover, the polls are used by those trading the markets to improve their forecasts, so they are a valuable input. But they are only one input. Those betting in the markets have access to so much other information as well, including informed political analysis, statistical modelling, canvass returns, and so on.
In conclusion, there is a growing belief that betting markets will become more than just a major part of our future. Properly used they will, more importantly, be able to tell us what that future is likely to be! We seem therefore to have created, almost by accident, a ‘high-tech’ crystal ball that taps into the accumulated expertise of mankind and makes it available to all. In this brave new world of prediction markets, it seems only sensible to make the most of it.
So let’s do so. Who will win the election in May? According to the current betting markets, no party will win an overall majority. Less clear is which party will win most seats, though the Conservatives currently have the edge, with Mr. Cameron favourite to remain as PM. Bottom line from the markets, though, is that this election really is too close to call, and all realistic options are still very much in play. That may change. If and when it does, the markets will be the first to tell us.
Follow on Twitter: @leightonvw
If you add up 1 and 2, what do you get? The answer is 3. Ok. Let’s go one step further. What if you add up 1 and 2 and 3? What do you get now? Now the answer is 6. Now 1 plus 2 plus 3 plus 4. That sums to 10. Now what if I do this for ever, in other words add up all the natural numbers right to infinity? What do I get? Most people say it is infinity. Mathematicians often say that there is no sum because technically you can’t sum a ‘divergent series’, as opposed to a series which converges to a number (such as 1+1/2+1/4+1/8+…, which converges to 2).
But let’s be ambitious and see where we get.
Let’s start simple and add up the following series:
1-1+1-1+1-1+1-1+1-1+… to infinity.
What is this?
If you stop at an odd step in the series, such as the first or third or fifth step, the series sums to 1. But if you stop at an even step, say the second or fourth or sixth, the series sums to 0. Both are equally likely, so it is intuitively obvious that we can take the average of 1 and 0, which is 0.5 as the solution of this equation.
For those who aren’t convinced by the obvious, however, we can show it a little more rigorously like this:
Let S = 1-1+1-1+1-1 …….
So, 1-S = 1 – (1-1+1-1+1-1 …) = 1-1+1-1-1-1…
So, 1-S = S
Therefore, S = ½.
We can also show it by the method of averaging partial sums, which I’ve added in an appendix to this post, as the third method.
So there are three different ways to demonstrate that the series: 1-1+1+1-1+1-… equals 0.5.
Now that we have established this, the task of calculating the solution to:
1+2+3+4+5… becomes quite straightforward.
So we have established that 1-1+1-1+1-1+… = ½
We’ll call this series S1.
But what if we want to calculate S2, which is the series 1-2+3-4 + ….?
The way to do this is to add it to itself, to get 2.S2
1-2+3-4 +5- … + (1-2+3-4+5…)
The easiest way to do this is to move the second series one step along, which is fine as it is an infinite series. The start with 1 and now add up each pair of the remaining terms.
So we get:
1 + (-2+1) + (3-2) + (-4+3) + (5-4) + … = 1-1+1-1+1 ………….
But we have seen this series before. It is S1, and is equal to ½.
So, 2.S2 = ½
Therefore, S2 = ¼
Now what we are trying to sum is 1+2+3+4+5+6+…….
Let us call this S.
So, S – S2 = 1+2+3+4+5+6+… – (1-2+3-4+5-6…)
So, S-S2 = 0+4+0+8+0+12+…
This series is identical to: 4+8+12+16+20+24+…
This is 4 x (1+2+3+4+5+6+…)
In other words, S-S2 = 4S
We know already that S2 = ¼
So, 3S = -1/4
And that is the proof that the sum of all the natural numbers up to infinity equals -1/12.
Who said it’s infinity? Who said you can’t sum divergent series? It’s got a solution and it’s the only meaningful one. Add up all the positive integers up to infinity and you get a negative number, -1/12.
There is no mathematical sleight of hand here. It is a properly derived solution, and we know from everything we understand about the laws of modern physics that it works in explaining the real world.
My next question is to ask you whether infinity is odd or even. What happens if I press the number 1 after 1 minute, then zero after a further 30 seconds, then 1 again after a further 15 seconds, then zero after a further 7.5 seconds and so on. What number am I pressing at the precise end of two minutes? Am I pressing 1 or zero, or both simultaneously, or neither. Imagine this was a magic light bulb that never blew. At the end of precisely 2 minutes, would it be on or off or both on and off? Or would it be -1/12 on and +1/12 off, or vice-versa.
Makes you think!
Thanks, by the way, to the excellent people on the Numberphile channel on YouTube, who inspired my interest in this.
In the series, 1-1+1-1+1-1 …
The first term = 1.
The sum of the first and second terms = 1-1 = 0
The sum of the first, second and third terms = 1-1+1=1
The sum of the first, second, third and four terms = 1-1+1-1 =0
And so on.
So the series of these sums (known as partial sums) = 1,0,1,0 …
Now, averaging these partial sums gives the following series:
1 divided by 1, 1+0 divided by 2, 1+0+1 divided by 3, 1+0+1+0 divided by 4, etc.
This works out as:
1, ½, 2/3, ½, 3/5, ½, 4/7 …
If we continue this series, we end up with ½, as the odd numbered terms get ever smaller, and eventually vanishingly so, leaving us just with 1/2.
This method of averaging partial sums to derive the total sum is well-established and can be used, for example, to calculate the sum of 1+1/2+1/4+1/8+ …
This can be shown to converge on 2 using the same method.
Follow on Twitter: @leightonvw
When two parties to a discussion differ, it is useful, in seeking to resolve the ‘argument’, to determine from where the differences arise, and whether these differences are resolvable in principle. The reason for the difference might be that the parties to the difference have access to different evidence, or else interpret that evidence differently. Another possibility is that one of the parties is applying a different or better process of non-evidence based reasoning to the other. Finally, the differences might arise from each party adopting a different axiomatic starting point. So, for example, if two parties differ in a discussion on euthanasia or abortion, or even the minimum wage, with one party strongly in favour of one side of the issue and the other strongly opposed, it is critical to establish whether this difference is evidence-based, reason-based, or derived from axiomatic differences. We are assuming here that the stance adopted by each party on an issue is genuinely held, and is not part of a strategy designed to advance some other objective or interest. The first thing is to establish whether any amount of evidence could in principle change the mind of an advocate of a position. If not, that leads us to ask where the viewpoint comes from. Is it purely reason-based, in which case (in the sense I use the term) it should in principle be either provable or demonstrably obvious to any rational person who holds a different view, without the need to appeal to evidence. Or else, is the viewpoint held axiomatically, so it is not refutable by appeal to reason or evidence? If the different viewpoints are held axiomatically by the parties to the difference, there the discussion should fall silent. If the differences are not held axiomatically, both parties should in principle be able to converge on agreement. So the question reduces to establishing whether the differences arise from divergences in reasoning, which should be resolvable in principle, or else by differences in access to evidence or proper evaluation of the evidence. Again, the latter should be resolvable in principle. In some cases, a viewpoint is held almost but not completely axiomatically. It is therefore in principle open to an appeal to evidence and/or reason. The bar may be set so high, though, that the viewpoint is in practice axiomatically held. If only one side to the difference holds a view axiomatically, or as close as to make it indistinguishable in practical terms, then the views could in principle converge by appeal to reason and evidence, but only converge to one side, i.e. the side which is holding the view axiomatically. This leads to a situation in which it is in the interest of a party seeking to change the view of the other party to conceal that their viewpoint is held axiomatically, and to represent it as reason-based or evidence-based, but only where the other party is not known to also hold their divergent view axiomatically. This leads to a game-theoretic framework where the optimal strategy, in a case where both parties know that the other party holds a view axiomatically, is to depart the game. In all other cases, the optimal strategy depends on how much each party knows about the drivers of the viewpoint of the other party, and the estimated marginal costs and benefits of continuing the game in an uncertain environment. It is critical in attempting to resolve such differences of viewpoint to determine whence they arise, therefore, in order to determine the next step. If they are irresolvable in principle, it is important to establish that at the outset. If they are resolvable in principle, setting this framework out at the beginning will help identify the cause of the differences, and thus help to resolve them. What applies to two parties is generalizable to any number, though the game-theoretic framework in any particular state of the game may be more complex. In all cases, transparency in establishing whether each party’s viewpoint is axiomatically held, reason-based or evidence-based, is the welfare-superior environment, and should be aimed for by an independent facilitator at the start of the game. Addressing differences in this way helps also to distinguish whether views are being proposed out of conviction, or whether they are being advanced out of self-interest or as part of a strategy designed to achieve some other objective or interest.
Follow on Twitter: @leightonvw
December 21st, 2014 was the shortest day of the year, at least in the UK, located in the Northern hemisphere of our planet.
So does that mean that the mornings should start to get lighter after that day (earlier sunrise), as well as the evenings (later sunset). Not so, and there’s a simple reason for that. The length of a solar day, i.e. the period of time between the solar noon (the time when the sun is at its highest elevation in the sky) on one day and the next, is not 24 hours in December, but about 30 seconds longer than that.
For this reason, the days get progressively about 30 seconds longer throughout December, so that by the end of the month a standard 24-hour clock is lagging roughly 15 minutes behind real solar time.
Let’s say just for a moment that the hours of sunlight (the time difference between sunrise and sunset) stayed constant through December. This means that a 24-hour clock which timed sunset at 3.50pm one day would be 30 seconds slow by 3.50pm the next day. The solar day would be 30 seconds longer than this, so the sun would not set the next day till 3.50pm and 30 seconds. After ten days the sun would not set till 3.55pm according to the 24-hour clock. So the sunset would actually get later through all of December. For the same reason, the sunrise would get later through the whole of December.
In fact, the sunset doesn’t get progressively later through all of December because the hours of sunlight shorten for about the first three weeks. The effect of this is that the sun would set earlier and rise later.
These two things (the shortening hours of sunlight and the extended solar day) work in the opposite direction. The overall effect is that the sun starts to set later from a week or so before the shortest day, but doesn’t start to rise earlier till about a week or so after the shortest day.
So the old adage that that the evenings will start to draw out after the end of the third week of December or so, and the mornings will get lighter, is false. The evenings have already been drawing out for several days before the shortest day, and the mornings will continue to grow darker for several days more.
There’s one other curious thing. The solar noon coincides with noon on our 24-hour clocks just four times a year. One of those days is Christmas Day! So set your clock to noon on December 25th, look up to the sky and you will see the sun at its highest point. Just perfect!
Bayes’ theorem concerns how we formulate beliefs about the world when we encounter new data or information. The original presentation of Rev. Thomas Bayes’ work, ‘An Essay toward Solving a Problem in the Doctrine of Chances’, was given in 1763, after Bayes’ death, to the Royal Society, by Mr. Richard Price. In framing Bayes’ work, Price gave the example of a person who emerges into the world and sees the sun rise for the first time. At first, he does not know whether this is typical or unusual, or even a one-off event. However, each day that he sees the sun rise again, his confidence increases that it is a permanent feature of nature. Gradually, through a process of statistical inference, the probability he assigns to his prediction that the sun will rise again tomorrow approaches 100 per cent. The Bayesian viewpoint is that we learn about the universe and everything in it through approximation, getting closer and closer to the truth as we gather more evidence. The Bayesian view of the world thus sees rationality probabilistically.
As such, Bayes’ perspective on cause and effect can be contrasted with that of David Hume, the logic of whose argument on this issue is contained in ‘An Enquiry Concerning Human Understanding’. According to Hume, we cannot justify our assumptions about the future based on past experience unless there is a law that the future will always resemble the past. No such law exists. Therefore, we have no fundamentally rational support for believing in causation. Bayes instead applies and formalizes the laws of probability to the science of reason, to the issue of cause and effect.
I propose that we apply the same Bayesian perspective to Immanuel Kant’s duty-based ‘Categorical Imperative.’ This can be summarised in the form: ‘Act only according to that maxim which you could simultaneously will to be a universal law.’ On this basis, to lie or to break a promise doesn’t work as a practical imperative, because if everyone lied or broke their promises, then the very concept of telling the truth or keeping one’s promises would be turned on its head. A society that worked according to the universal principle of lying or promise-breaking would be unworkable. Kant thus argues that we have a perfect duty not to lie or break our promises, or indeed do anything else that we could not justify being turned into a universal law.
The problem with this approach in many eyes is that it is too restrictive. If a crazed gunman demands that you reveal which way his potential victim has fled, you must not lie to save him because this could not be universalisable as a rule of behaviour.
I propose that the application of a justification argument can solve the problem. This argument from justification is that you have no duty to respond to any request which is posed without reasonable appeal to duty. So, in this example, the gunman has no reasonable appeal to duty from you, so you can make an exception to the general rule.
Why is this consistent with the practical implications of Kant’s ‘universal law’ maxim? It’s an issue of probability. In the great majority of situations, you have no defence based on the argument from justification for lying or breaking a promise. So the universal expectation is that truth-telling and promise-keeping is overwhelmingly probable. The more often this turns out to be true in practice, the closer this approach converges on Kant’s absolute imperative by a process of simple Bayesian updating.
In a world in which ethics is indeed based on duty, it is this broader conception of duty which, I propose, should inform our actions.
The Abilene Paradox is a classic management parable. Does it sound familiar in your family or workplace? If so, it may be time to do something about it. THE ABILENE PARADOX On a hot afternoon visiting in Coleman, Texas, the family is comfortably playing dominoes on a porch, until the father-in-law suggests that they take a trip to Abilene [53 miles north] for dinner. The wife says, “Sounds like a great idea.” The husband, despite having reservations because the drive is long and hot, thinks that his preferences must be out-of-step with the group and says, “Sounds good to me. I just hope your mother wants to go.” The mother-in-law then says, “Of course I want to go. I haven’t been to Abilene in a long time.” The drive is hot, dusty, and long. When they arrive at the cafeteria, the food is as bad as the drive. They arrive back home four hours later, exhausted. One of them dishonestly says, “It was a great trip, wasn’t it?” The mother-in-law says that, actually, she would rather have stayed home, but went along since the other three were so enthusiastic. The husband says, “I wasn’t delighted to be doing what we were doing. I only went to satisfy the rest of you.” The wife says, “I just went along to keep you happy. I would have had to be crazy to want to go out in the heat like that.” The father-in-law then says that he only suggested it because he thought the others might be bored. The group sits back, perplexed that they together decided to take a trip which none of them wanted. They each would have preferred to sit comfortably, but did not admit to it when they still had time to enjoy the afternoon. =============================================== Originally stated by George Washington University Professor, Jerry B. Harvey.