In particular, an analysis of the favourite in each of the seats traded on the Betfair market gave the Tories 366 seats and Labour 208. The Predictwise betting aggregation site gave the Conservatives an 81% chance of securing an overall majority of seats, in line with the large sums of money trading on the Betfair exchange.
The PredictIt prediction market, meanwhile, estimated just a 15% chance that the Tories would secure 329 or fewer seats in the House of Commons (with 326 technically required for a majority), while the Oddschecker odds comparison site rated a “hung parliament” result an 11/2 chance (an implied probability of 15.4%). Only the Almanis crowd forecasting platform expressed any real doubt, putting the chance of a Conservative overall majority at a relatively paltry 62%.
In reality, the Conservative party lost more than a dozen seats net, ending up with 318 – eight short of a majority. Labour secured 262 seats, the Scottish National party 35, and the Liberal Democrats 12. Their projected vote shares are 42.4%, 40%, 3% and 7.9% respectively.
So did the opinion polls do any better than the betting markets? With the odd exception, no.
In their final published polls, ICM put the Tories on 46%, up 12% on Labour. ComRes predicted the Tories would score 44% with a 10-point lead. BMG Research was even further out, putting the Conservatives on 46% and a full 13% clear of Labour. YouGov put the Tories seven points clear of Labour (though their constituency-level model did a lot better), as did Opinium; Ipsos MORI and Panelbase had them eight points clear on 44%.
Other polls were at least in the ballpark. Kantar Public put the Tories 5% ahead of Labour, and SurveyMonkey (for the Sun) called the gap at 4%. Survation, the firm closest to the final result in their unpublished 2015 poll, this time put the Conservatives on 42% and Labour on 40%, very close to the actual result. Qriously (for Wired)was the only pollster to put Labour ahead, by three points.
According to the 2017 UK Parliamentary Election Forecast polling model, the Conservatives were heading for 366 seats, Labour 207, and the Liberal Democrats seven. Allowing for statistical uncertainty, the projection was of an “almost certain” overall majority for the Conservatives. The probability of a hung parliament was put at just 3%. All misses – though that doesn’t necessarily reflect on the model, which after all can only be as good as the polls fed into it.
Many others were wrong, too. The 2017 General Election Combined Forecast, which aggregates betting markets and polling models, forecast a Conservative majority of 66 seats. Other “expert” forecasts came from Britain Elects (Tories 356 seats, Labour 219 seats), Ashcroft (363, 217), Electoral Calculus (358, 218), Matt Singh (374, 207), Nigel Marriott (375, 202), Election Data (387, 186), Michael Thrasher (349, 215), Iain Dale (392, 163) and Andreas Murr and his colleagues (361, 236).
So what went wrong?
In the wake of the 2015 election, the Brexit referendum and Donald Trump’s victory, forecasters are getting used to fielding that question. But the answer isn’t that difficult: the problem is in quantifying the key factor in the common forecasting meltdown in advance. That factor is turnout, and notably relative turnout by different demographics.
In the Brexit referendum and 2016 US presidential election, turnout by poorer and less educated voters, especially outside urban areas, hit unprecedentedly high levels, as people who had never voted before (and may never vote again) came out in droves. In both cases, forecasters’ pre-vote turnout models had predicted that these voters wouldn’t show up in nearly the numbers they did.
In the 2017 election, it was turnout among the young in particular that rocketed. This time the factor was widely expected to matter, and indeed get-out-the-vote campaigns aimed at the young were based on it. But most polling models failed to properly account for it, and that meant their predictions were wrong.
Polling is a moving target, and the spoils go to those who are most adept at taking and changing aim. So will the lesson be learned for next time? Possibly. But next time, under-25s might not turn out in anything like the same numbers – or a different demographic altogether might surprise everyone. We might not have long to wait to find out.
This leads us to the classic ‘Prisoner’s Dilemma’ problem. In this scenario, two prisoners, linked to the same crime, are offered a discount on their prison terms for confessing if the other prisoner continues to deny it, in which case the other prisoner will receive a much stiffer sentence. However, they will both be better off if both deny the crime than if both confess to it. The problem each faces is that they can’t communicate and strike an enforceable deal. The box diagram below shows an example of the Prisoner’s Dilemma in action.
Prisoner 2 Confesses | Prisoner 2 Denies | |
Prisoner 1 Confesses | 2 years each | Freedom for P1; 8 years for P2 |
Prisoner 1 Denies | 8 years for P1; Freedom for P2 | 1 year each |
The Nash Equilibrium is for both to confess, in which case they will both receive 2 years. But this is not the outcome they would have chosen if they could have agreed in advance to a mutually enforceable deal. In that case they would have chosen a scenario where both denied the crime and received 1 year each.
So a Nash equilibrium is a stable state that involves interacting participants in which none can gain by a change of strategy as long as the other participants remain unchanged. It is not necessarily the best outcome for the parties involved, but it is the outcome we would most likely predict.
The Prisoner’s Dilemma is a one-stage game, however. What happens in games with more than one round, where players can learn from the previous moves of the other players?
Take the case of a 2-round game. The payoff from the game will equal the sum of payoffs from both moves.
The game starts with two players, each of whom is given £100 to place into a pot. They can then secretly choose to honour the deal or to cheat on the deal, by means of giving an envelope to the host containing the card ‘Honour’ or ‘Cheat’. If they both choose to ‘Honour’ the deal, an additional £100 is added to the pot, yielding each an additional £50. So they end up with £150 each. But if one honours the deal and the other cheats on the deal, the ‘Cheat’ wins the original pot (£200) and the ‘Honour’ player loses all the money in that round. A third outcome is that both players choose to ‘Cheat’, in which case each keeps the original £100. So in this round, the dominant strategy for each player (assuming no further rounds) is to ‘Cheat’, as this yields a higher payoff if the opponent ‘Honours’ the deal (£200 instead of £150) and a higher payoff if the opponent ‘Cheats’ (£100 instead of zero). The negotiated, mutually enforceable outcome, on the other hand, would be to agree to both ‘Honour’ the deal and go away with £150.
But how does this change in a 2-round game.
Actually, it makes no difference. In this scenario, the next round is the final round, in which you may as well ‘Cheat’ as there are no future rounds to realise the benefit of any goodwill realised from honouring the deal. Your opponent knows this, so you can assume your opponent who wishes to maximise his total payoff, will be hostile on the second move. He will assume the same about you.
Since you will both ‘Cheat’ on the second and final move, why be friendly on the first move?
So the dominant strategy is to ‘Cheat’ on the first round.
What if there are three rounds? The same applies. You know that your opponent will ‘Cheat’ on the final round and therefore the penultimate round as well. So your dominant strategy is to ‘Cheat’ on the first round, the second round and the final round. The same goes for your opponent. And so on. In any finite, pre-determined number of rounds, the dominant strategy in any round is to ‘Cheat.’
But what if the game involves an indeterminate number of moves? Suppose that after each move, you roll two dice. If you get a double-six, the game ends. Any other combination of numbers, play another round. Keep playing until you get a double-six. Your score for the game is the sum of your payoffs.
This sort of game in fact mirrors many real-world situations. In real life, you often don’t know when the game will end.
What is the best strategy in repeated play? For the game outlined above, we shall denote ‘Honour the deal’ as a ‘Friendly’ move and ‘Cheat’ as a hostile move. But the notion of a Friendly or Hostile approach can adopt other guises in different games.
There are seven proposed strategies here.
Which of these is the dominant strategy in this game of iterated play? Actually, there is no dominant strategy in an iterated game, but which strategy actually wins if every strategy plays every other strategy.
‘Always Hostile’ does best against ‘Always Friendly’ because every time you are Friendly against an ‘Always Hostile’, you are punished with the ‘sucker’ payoff.
‘Always Friendly’ does best against Retaliation, because the extra payoff you get from a Hostile move is eventually negated by the Retaliation.
Thus even the choice of whether to be Friendly or Hostile on the first move depends on the opponent’s strategy.
For every two distinct strategies, A and B, there is a strategy C against which A does better than B, and a strategy D against which B does better than A.
So which strategy wins when every strategy plays every other strategy in a tournament? This has been computer simulated many times. And the winner is Tit for Tat.
It’s true that Tit for Tat can never get a higher score than a particular opponent, but it wins tournaments where each strategy plays every other strategy. In particular, it does well against Friendly strategies, while it is not exploited by Hostile strategies. So you can trust Tit for Tat. It won’t take advantage of another strategy. Tit for Tat and its opponents both do best when both are Friendly. Look at this way. There are two reasons for a player to be unilaterally hostile, i.e. to take advantage of an opponent or to avoid being taken advantage of by an opponent. Tit for Tat eliminates the reasons for being Hostile.
What accounts for Tit for Tat’s success, therefore, is its combination of being nice, retaliatory, forgiving and clear.
In other words, success in an evolutionary ‘game’ is correlated with the following characteristics:
Be willing to be nice: cooperate, never be the first to defect.
Don’t be played for a sucker: return defection for defection, cooperation for cooperation.
Don’t be envious: focus on how well you are doing, as opposed to ensuring you are doing better than everyone else.
Be forgiving if someone is willing to change their ways and co-operate with you. Don’t bear grudges for old actions.
Don’t be too clever or too tricky. Clarity is essential for others to cooperate with you.
As Robert Axelrod, who pioneered this area of game theory in his book, ‘The Evolution of Cooperation’: Tit for Tat’s “niceness prevents it from getting into unnecessary trouble. Its retaliation discourages the other side from persisting whenever defection is tried. Its forgiveness helps restore mutual cooperation. And its clarity makes it intelligible to the other player, thereby eliciting long-term cooperation.”
How about the bigger picture? Can Tit for Tat perhaps teach us a lesson in how to play the game of life? Yes, in my view it probably can.
Further Reading and Links
Axelrod, Robert (1984), The Evolution of Cooperation, Basic Books
Axelrod, Robert (2006), The Evolution of Cooperation (Revised ed.), Perseus Books Group
Axelrod, R. and Hamilton, W.D. (1981), The Evolution of Cooperation, Science, 211, 1390-96. http://www-personal.umich.edu/~axe/research/Axelrod%20and%20Hamilton%20EC%201981.pdf
https://en.wikipedia.org/wiki/The_Evolution_of_Cooperation
Messi knows from the team statistician that if he aims straight and the goalkeeper stands still, his chance of scoring is just 30%. But if he aims straight and Navas dives to one corner, his chance of converting the penalty rises to 90%.
On the other hand, if Messi aims at a corner and the goalkeeper stands still, his chance of scoring is a solid 80%, while it falls to 50% if the goalkeeper dives to a corner.
We are here simplifying the choices to two distinct options, for the sake of simplicity and clarity.
Navas also knows from his team statistician that if he dives to one corner and Messi aims straight, his chance of saving is just 10%. But if he stands still and Messi aims at one corner, his chance of saving the penalty rises to 50%.
On the other hand, if Navas stands still and Messi aims at a corner, his chance of making the save is just 20%, while it rises to 70% if Messi aims straight.
So this is the payoff matrix, so to speak, facing Messi as he weighs up his decision.
Goalkeeper – Stands Still | Goalkeeper – dive to one corner | |
Lionel Messi – Aims straight | 30% | 90% |
Lionel Messi – Aims at corner | 80% | 50% |
So what should he do? Aim straight or to a corner. And what should Navas do? Stand still or dive?
Here is the payoff matrix facing Navas.
Messi – Aims straight | Messi – Aims at a corner | |
Navas – Stands still | 70% | 20% |
Navas – Dives to one corner | 10% | 50% |
Game theory can help here.
Neither player has what is called a dominant strategy in game-theoretic terms, i.e. a strategy that is better than the other, no matter what the opponent does. The optimal strategy will depend on what the opponent’s strategy is.
In such a situation, game theory indicates that both players should mix their strategies, in Messi’s case aiming for the corner two-thirds of the time, while the goalkeeper should dive 5/9 of the time.
These figures are derived by finding the ratio where the chance of scoring (or saving) is the same, whichever of the two tactics the other player uses.
The Proof
Suppose the goalkeeper opts to stand still, then Messi’s chance (if he aims for the corner 2/3 of the time) = 1/3 x 30% + 2/3 x 80% = 10% + 53.3% = 63.3%
If the goalkeeper opts to dive, Messi’s chance = 1/3 x 90% + 2/3 x 50% = 30% + 33.3% = 63.3%
Adopting this mixed strategy (aim for the corner 2/3 of the time and shoot straight 1/3 of the time), the chance of scoring is therefore the same. This is the ideal mixed strategy, according to standard game theory.
From the point of view of Navas, on the other hand, if Messi aims straight, his chance of saving the penalty kick (if he dives 5/9 of the time) = 5/9 x 10% + 4/9 x 70% = 5.6% + 31.1% = 36.7%
If Messi opts to aim for the corner, Navas’ chance = 5/9 x 50% + 4/9 x 20% = 27.8% + 8.9% = 36.7%
Adopting this mixed strategy (dive for the corner 5/9 of the time and stand still 4/9 of the time), the chance of scoring is therefore the same. This is the ideal mixed strategy, according to standard game theory.
The chances of Messi scoring and Navas making the save in each case add up to 100%, which cross-checks the calculations.
Of course, if the striker or the goalkeeper gives away real new information about what he will do, then each of them can adjust tactics and increase their chance of scoring or saving.
To properly operationalise a mixed strategy requires one extra element, and that is the ability to truly randomise the choices, so that Messi actually does have exactly a 2/3 chance of aiming for the corner, and Navas actually does have a 5/9 chance of diving for the corner. There are different ways of achieving this. One method of achieving a 2/3 ratio is to roll a die and go for the corner if it comes up 1, 2, 3 or 4, and aim straight if it comes up 5 or 6. Or perhaps not! But you get the idea.
For the record, Messi aimed at the left corner, Navas guessed correctly and got an outstretched hand to it, pushing it back into play. Leo stepped forward deftly to score the rebound. Cristiano Ronaldo equalised from the spot eight minutes later. And that’s how it ended at the Bernabeu. Real Madrid 1 Barcelona 1. Honours even in El Clasico.
Appendix
Messi’s strategy
x = probability that Messi should aim at corner
y = probability that Messi should aim straight
So,
80x + 30y (if Navas stands still) = 50x + 90y (if Navas dives)
x + y = 1
So,
30x = 60y
30x = 60 (1-x)
90x = 60
x = 2/3
y=1/3
Navas’ strategy
x = probability that Navas should dive to corner
y = probability that Navas should stand still
So,
10x + 70y (if Messi aims straight) = 50x + 20y (if Messi aims at corner)
x+y = 1
So,
10x + 70y = 50x + 20y
40x = 50y
40x = 50(1-x)
90x = 50
x = 5/9
y = 4/9
“If the Super Bowl is won by a team from the old National Football League (now the NFC, or National Football Conference),” they wrote, “then the stock market is very likely to finish the year higher than it began. On the other hand, if the game is won by a team from the old American Football League (now the AFC, or American Football Conference), the market will finish lower than it began.”
It is important to note, though, that some AFC teams count as NFL wins because they originated in the old NFL, i.e. Pittsburgh Steelers, Baltimore Ravens (formerly Cleveland Browns, Baltimore/Indianapolis Colts).
Over the 22-year history of the Super Bowl to the date of submission of their study in 1988, they documented a 91% accuracy rate for their predictor.
What happened in 1989? The NFC team, San Francisco 49ers, beat the AFC’s Cincinnati Bengals– the stock market rose 27%.
Further confirmation of an idea first proposed by New York Times sportswriter Leonard Koppett, published as ‘The Super Bowl Predictor’ by investment advisor Robert H. Stovall in the January 1988 issue of ‘Financial World.’
So what happened in 1990? Well, the NFC’s San Francisco 49ers won a second consecutive victory, beating the AFC’s Denver Broncos, by 55 points to 10. But the stock market fell in 1990, by 4.3%.
But then the Super Bowl Predictor returned to form, correctly predicting the direction of the stock market in 1991, 1992, 1993, 1994, 1995, 1996, 1997. Since the launch of the Super Bowl that made for 28 correct predictions out of 31 (a success rate of 90.3%).
Since then, the Super Bowl Predictor has had a much more chequered record. predicted correctly only about half the time since 1997. In 2009, Robert Stovell, a strategist for Wood Asset Management in Sarasota, Florida, and an early champion of the Stock Market Indicator wrote: “Nothing seems to be working anymore {in the stock market]”. Used to be, I was only happy when it was over 90% (accurate), and when it was still above 80% I was pleased. But certainly 79% is still far above a failing grade.” (quoted on January 12, 2009, in MarketBeat (WSJ.com’s ‘inside look at the markets’).
Since then, the Predictor has called it right five times (2010, 2011, 2012, 2014 and 2015, and wrong twice (2013 and 2016). As of January, 2017, the indicator been right a total of 40 times out of 50, as measured by the S&P 500 index. This year the AFC’s New England Patriots stormed from 25 points behind at one point in the game to beat the Atlanta Falcons by 34 points to 28 in overtime. That should presage a bad year for the stock markets in 2017. We shall see.
So is the Super Bowl Indicator a real forecasting tool, or is it simply descriptive of wat has happened rather than containing any predictive value?
You decide!
Further Reading
Krueger, T.M. and Kennedy, W.F. (1990), An Examination of the Super Bowl Stock Market Predictor, Journal of Finance, 1990, 45 (2), 691-697.
Prices or odds may change when new information is released, but this new information is unpredictable.
So the best estimate of the price or odds likely to prevail at any point in the future is the price now. This is dismal, if true, because it would mean that it is not possible to beat the market, except by chance. But this can’t be true, or else it creates a paradox. If the market was always efficient, traders would have no economic incentive to acquire information, since information acquisition and processing is not a costless activity and would add nothing to what can be obtained by simply looking at current market prices.
It has been mathematically proved (Grossman and Stiglitz, 1980, American Economic Review) that when information is not costless to obtain or process, asset prices can never fully reflect all the information available to traders. So in the real world, markets are not completely efficient. They cannot be. This result is a relief, at least in principle, to those seeking to beat the market.
The equilibrium proposed by Grossman and Stiglitz is one in which some profits are available to some investors.
Essentially, rational, ‘informed’ traders will seek to acquire and process new information whenever the benefits of doing so are greater then the costs. Up to the point, economists say, where the marginal costs equal the marginal benefits of obtaining and processing information.
But to the extent that trading is a zero-sum game, or worse, as most betting markets are, winners need losers.
So who are the winners and who are the losers?
To take the example of a poker game, good players need weak players in the game. In the financial literature these ‘weak players’ are known as ‘noise (or ‘uninformed’) traders.’ Noise makes trading in financial markets possible, and thus allows us to observe prices for financial assets. But noise also causes markets to be somewhat inefficient.
Imagine a world with no noise traders, no information costs, no trading costs – an efficient market, a market in which it would be irrational to place any trades, a market without traders, a strange kind of world. So the Efficient Market hypothesis in its strictest form cannot be true. So it is possible in principle to beat the market. How might we do this? By a ‘technical’ strategy which uses information contained in past and present prices or odds. Or by a ‘fundamental’ strategy which uses information about real variables, such as form. Or by some combination of these. Those practical matters can be examined at another time. Here we are looking exclusively at whether markets are inefficient as a matter of principle, in a world of positive information costs, and we have concluded that the answer is Yes.
There is another systematic reason why markets might be inefficient, and that is the existence of asymmetric information. A notable case of this is called ‘adverse selection’, which refers to a situation in which the buyer or seller of a product knows something about the product quality or condition that the other party does not know, allowing them to have a better estimate of what the true cost of the product should be. This can lead to the breakdown of a market in which it exists. George Akerlof’s seminal article (‘The Market for Lemons’), published in 1970 in the Quarterly Journal of Economics, which examined the problem of adverse selection on the market for used cars has important implications for any market characterised by adverse selection.
Here is the problem. If Mr. Smith wants to SELL me his horse, do I really WANT to buy it? It’s a question as old as markets and horses have existed, but it was for many, many years, one of the unspoken questions of economics. So how do we solve this paradox? For most of the history of economics, the answer was quite simple. Simply assume perfect markets and perfect information, so the horse buyer would know everything about the horse, and so would the seller, and in those cases where the horse is worth more to the buyer than the seller, both can strike a mutually beneficial deal. There’s a term for this: ‘gains from trade’.
In the real world, the person selling the horse is likely to know rather more about it than the potential purchaser. This is called ‘asymmetric information’, and the buyer is facing what is called an ‘adverse selection’ problem, as he has adverse information relative to the seller. Akerlof had become intrigued by the way in which economists were limited by their assumption of well-functioning markets characterised by perfect information. For example, the conventional wisdom was that unemployment was simply caused by money wages adjusting too slowly to changes in the supply and demand for labour. This was the so-called ‘neo-classical synthesis’ and it assumed classic markets, albeit they could be a bit slow to work.
At the same time, economists had come to doubt that changes in the availability of capital and labour could in themselves explain economic growth. The role of education was called upon as a sort of magic bullet to explain why an economy grew as fast as it did. But how can we distinguish the impact on productivity of the education itself from the extent to which education simply helped grade people? The idea here is that more able people will tend on average to seek out more education. So how far does education contribute to growth, and how far is it simply a signal and a screen for employers? In the real world, of course, these signals could be useful because employers are like the horse buyers – they know less about the potential employees than the employees know about themselves, the classic adverse selection problem.
Akerlof turned to the used car market for the answer, not least because at the time a major factor in the business cycle was the big fluctuation in sales of new cars. Just like in the market for horses, the first thing a potential used car buyer is likely to ask is “Why should I WANT to buy that used car if he wants so much to SELL it to me”. The suspicion is that the car is what Americans call a ‘lemon’, a sub-standard pick of the crop. Owners of better quality used cars, called ‘plums’, are much less likely to want to sell.
Now let’s say that you’re willing to spend £10,000 on a plum but only £5,000 on a lemon. In such a case, the best price you’d be willing to pay is about £7,500, and only then if you thought there was an equal chance of a lemon and a plum. At this price, though, sellers of the plums will tend to back out, but sellers of the troublesome lemons will be very happy to accept your offer.
But as a buyer you know this, so will not be willing to pay £7,500 for what is very likely to be a lemon. The prices that will be offered in this scenario may well spiral down to £5,000 and only the worst used cars will be bought and sold. The bad lemons have effectively driven out the good plums, and buyers will start buying new cars instead of plums. Just as with horses, asymmetric and imperfect information in the used car market has the potential, therefore, to severely compromise its effective operation.
We can assume that the demand for used cars depends most strongly on two variables – the price of the car and the average quality of used cars traded. Both the supply of used cars and the average quality will depend upon the price. In equilibrium, the supply equals the demand for the given average quality. As the price declines, normally the quality ill also fall. And it’s quite possible that no cars will be traded at any price.
This same idea shows the problem of medical insurance. In a free market for medical insurance, people above a certain age, for example, will have great difficulty in buying medical insurance. So why doesn’t the price rise to match the risk? The answer is that as the price level rises the people who insure themselves will be those who are increasingly certain that they will need the insurance. In consequence, the average medical condition of insurance applicants deteriorates as the price level rises – such that no insurance sales [for these age groups] may take place at any price. This is strictly analogous to the car case, where the average quality of used cars supplied fell with a corresponding fall in the price level. The principle of ‘adverse selection’ is potentially present in all lines of insurance. Adverse selection can arise whenever those seeking insurance have freedom to buy or not to buy, to choose the insurance plan, and to continue or discontinue as a policy holder.
There are ways to counteract the effects of quality uncertainty, such as guarantees on consumer durables. Brand names perform a complementary function. Brand names not only indicate quality but also give the consumer a means of retaliation if the quality does not meet expectations. Chains – such as hotel chains or restaurant chains – are similar to brand names. Licensing practices also reduce quality uncertainty. And education and labour markets themselves have their own ‘brand names.’
So one of the big problems that confront markets is the fact that some of the participants often don’t know certain things that others in the market do know. This includes the market for most consumer durables, virtually all jobs markets, many financial markets, etc. In these cases, one of the roles of economics is to ask what system of incentives is most likely to address this problem of imperfect and asymmetric information. In economics, signalling is the idea that one party (termed the agent) credibly conveys some information about itself to another party (the principal). Signals should be distinguished from what have been called ‘indices’ ( a term coined by Robert Jervis in his 1968 PhD thesis). Indices are attributes over which one has no control. Think of these as generally unalterable attributes of something or someone. Signals are things that are visible and that are in part designed to communicate. In a sense, they are alterable attributes. So employees send a signal about their ability level to the employer by acquiring certain education credentials. The informational value of the credential comes from the fact that the employer assumes it is positively correlated with having greater ability.
Education credentials can be used as a signal to the firm, indicating a certain level of ability that the individual may possess; thereby narrowing the informational gap. In a seminal article on signalling, published in 1973 by Michael Spence, he proposes the key assumption that good-type employees pay less for one unit of education than bad-type employees. In Spence’s model it is optimal for the higher ability person to obtain the credential (the observable signal) but not for the lower ability individual. The premise for the model is that a person of high ability has a lower cost for obtaining a given level of education than does a person of lower ability. Cost can be in terms of tuition costs, or intangible costs, such as stress and time and effort in obtaining the qualification. Thus, if both individuals act rationally it is optimal for the higher ability person to obtain the qualification but not for the lower ability person so long as the employers respond to the signal correctly. This will result in the workers self-sorting into the two groups. For this to work, it must be excessively costly, or impossible, to project a false image. The basic argument follows from the intuition that a behaviour that costs nothing can be equally well taken by anyone and so provides no information. It follows that perceivers should focus on behaviour which is costly to undertake. Signalling is an action by a party with good information that is confined to situations of asymmetric information.
The concept of screening should be distinguished from signalling, the latter implying that the informed agent moves first. When there is asymmetric information in the market, screening can involve incentives that encourage the better informed to self-select or self-reveal.
Joseph Stiglitz pioneered the theory of screening, examining how a less informed party can induce the other party to reveal their information. They can provide a menu of choices in such a way that the optimal choice of the other party depends on their private information. For example, a theme park might offer a menu of gold and silver tickets, where the more expensive gold ticket allows the customer to avoid the queue at rides. This will induce the customers to self-sort and reveal genuine information as to the value they place on their time and their desire to avoid the queues.
So can markets be efficient? In the strictest sense, the answer is No. But there are ways in which they can be made more efficient than they would be in their natural state.
Further Reading and Links
Akerlof, G. (1970), The Market for Lemons: Quality, Uncertainty and the Market Mechanism. Quarterly Journal of Economics. 84:488-500.
Grossman, S.J. and Stiglitz, J. (1980), The Impossibility of Informationally Efficient Markets, American Economic Review, June, 393-408.
Jervis, R. Signaling and Perception, in Kristen Monroe, ed., Political Psychology (Earlbaum, 2002).
Spence, M. (1973). “Job Market Signaling”. Quarterly Journal of Economics (The Quarterly Journal of Economics, 87 (3): 355–374.
Joseph E. Stiglitz, 1975. “The Theory of ‘Screening’, Education, and the Distribution of Income,” American Economic Review, 65(3), pp. 283–300.
Joseph E. Stiglitz, 1981. “Information and the Change in the Paradigm of Economics”, Nobel Prize Lecture, December 8.
A. Michael Spence, 1981. “Signaling in Retrospect and the Informational Structure of Markets”, Nobel Prize Lecture, December 8.
George A. Akerlof, “Behavioral Macroeconomics and Macroeconomic Behavior”, Nobel Prize Lecture, December 8.
In a two-horse race, if both horses have an equal chance of winning (objectively), and both are offered at evens, then the expected profit of the market-maker (and of the bettor) is zero, ignoring operating, information and transactions costs.
In a two-horse race, if both are offered at evens (regardless of the respective probabilities of victory of the two horses), then it would require a stake of £x (split equally between the two horses) to be sure of being returned that £x (a net profit of zero) whichever horse wins. In this circumstance, the over-round of the bookmaker is said to be 100%, i.e. a notional profit margin of zero.
In practice, even if the notional profit margin is zero, the bookmaker is at a disadvantage if the horses are not equally matched, as a sophisticated bettor can take advantage by staking more than half on the horse with the greater chance of winning.
More generally, the over-round does not yield an accurate indicator of the bookmaker’s profit margin if bettors do not stake across all options in such a way as to ensure that their total stake of £x yields a certain return of £x, factored by the over-round.
For example, if the over-round is 120%, the notional margin to the bookmaker is 20%, and put simply bettors would have to stake £120 to ensure a return of £100. Say, for instance, that both horses in a 2-horse race are being offered at 4 to 6. Then the bettor would need to stake £60 on each (£120 in total) to be guaranteed a return of £100 (£40 plus the £60 stake returned) whichever horse won. In such circumstances, the bookmaker is guaranteed at 20% profit, regardless of the outcome.
If one horse is offered at 4 to 6 and the other at 6 to 4, the bettor can guarantee a zero profit (and loss) by staking £60 at 4 to 6 and £40 at 6 to 4. That way, a £100 return is guaranteed for a total stake of £100, regardless of the outcome. Again, if the horse offered at 4 to 6 is actually a 4 to 7 chance, and bettors stake exclusively on this horse, their expected return is positive (although there is now a risk of losing the entire stake), and the expected return of the bookmaker is negative (though the actual return may be positive).
To summarize, the notional margin, as implied in the over-round, formally equates to the actual margin only if bettors stake proportionately more on the outcome offered at shorter odds.
Creating an over-round
Take as an example the following odds offered about a binary proposition to players, where the odds-maker believes that the objective probability of X winning is 1 in 5 (0.2) and of Y winning is 4 in 5 (0.8).
Assuming an over-round of 100% (i.e. margin of zero), the odds-setter (taken here to be a bookmaker) would set the following odds:
Odds about X = 5.0 (4 to 1): Odds about Y = 1.25 (1 to 4).
Assume now that the odds-maker wishes to create an over-round of 108%.
In each case the odds offered should be cut, by 8 per cent in each case. So 8% of 5.0 = 0.4. Deducting 0.4 from 5.0 gives 4.6. 8% of 1.25 = 0.1. Deducting 0.1 from 1.25 gives 1.15.
So in the particular example, the odds offered would be as follows:
Odds about X = 4.6; Odds about Y = 1.15.
Assuming an equal amount bet (say £1,000) bet on both sides of the proposition (i.e. a total of £2,000, consisting of perhaps 200 people betting £10 each), the profit (loss) to the bookmaker would vary depending on the outcome.
If horse X wins, the bookmaker will pay out:
4.6x £1,000 = £4,600
Total amount staked (on X and Y) = £2,000.
Net profit to bookmaker if horse X wins = £2,000 – £4,600 = – £2,600
So if horse X wins, bookmaker loses £2,600.
If horse Y wins, the bookmaker will pay out:
1.15 x £1,000 = £1,150
Total amount staked (on X and Y) = £2,000
Net profit to bookmaker if horse Y wins = £2,000 – £1,150 = £850
Expected value of profit = expected value of profit from X + expected value of profit from Y = (-£2,600) x 0.2 + (£850) x 0.8 = -£520 + £680 = £160.
This is assuming that the implied probabilities in the odds are the correct probabilities, i.e. odds of 4/1 = probability of 1/5 (0.2); odds of 1/4 = probability of 4/5 (0.8).
Note also that £160 = 8% of total stake on X and Y (£2,000).
This all assumes, as observed, that the objective probabilities are correctly observed and that the amount staked on both sides of the proposition are equal.
Even if we assume that the objective probabilities are correctly observed then there is still substantial volatility of outcome (i.e. risk) for the bookmaker. If the objective probability is incorrectly observed, however, the outcome for the bookmaker may be worse, i.e. a systematic loss.
For example, assume the probability of horse X winning is actually 25%; assume probability of horse Y winning is 75%.
At the given odds levels, and assuming equal stakes across both propositions, we derive the following.
As above, if horse X wins, the bookmaker will pay out, as before:
4.6 x £1,000 = £4,600
Total amount staked (on X and Y) = £2,000.
Net profit to bookmaker if horse X wins = £2,000 – £4,600 = – £2,600
So if horse X wins, bookmaker loses £2,600.
If horse Y wins, the bookmaker will pay out, as before:
1.15 x £1,000 = £1,150
Total amount staked (on X and Y) = £2,000
Net profit to bookmaker if horse Y wins = £2,000 – £1,150 = £850
Expected value of profit = expected value of profit from X + expected value of profit from Y = (-£2,600) x 0.25 + (£850) x 0.75 = -£650 + £637.50 = -£12.50, i.e. a loss of £12.50.
Insofar as the objective probability of horse X winning is greater than 20%, the expected profit to the bookmaker will decline. At 24.65%, the profit (rounded to the nearest pound) can be shown to be equal to zero, and above that to turn negative.
Assume objective probability of horse X winning = 0.2465; objective probability of horse Y winning = 0.753.
Then, expected value of profit = expected value of profit from X + expected value of profit from Y = (-£2,600) x 0.2465 + (£850) x 0.7535 = -£640 + £640 = 0
To the extent that the objective probabilities are inaccurately estimated, therefore there is significant potential from the bookmaker’s point of view for a negative expected (as well as actual) profit.
Using the probabilities from the original example, the staking pattern from the bettor’s point of view that will lead to a unique expected loss (8% in this case) across both betting propositions is to bet more on the favourite and less on the longshot, in this case £1,600 and £400 respectively.
This leads to the following outcomes:
Profit to a £400 bet on horse X (if it wins) at 4.60 = £1,840
Profit to a £1,600 on horse Y (if it wins) at 1.15 = £1,840
Guaranteed profit by staking these sums on each horse from the bettor’s point of view = – £160, i.e. a net loss of 8% of total stake.
Insofar as bettors can be induced to bet in these proportions, the operator is guaranteed a profit regardless of the outcome. If the average bet size is the same for bets made on either side, then we need four times as many bettors on the favourite as the longshot to achieve this. Otherwise, the same outcome can be achieved if those who are backing the favourite bet four times as much in total as those backing the longshot.
Another way to manage risk in the face of unbalanced staking patterns is to move the odds so as to limit the maximum loss.
In order to reduce the maximum downside (i.e. when X wins) the bookmaker may move the odds in such a way as to attract money on one horse and away from the other horse. To do this, the odds about one horse may be lengthened and those about the other horse shortened before a negative downside is occurred to ether outcome. While such a strategy may reduce the exposure of the operator, the price may be paid in reduced profits.
Ultimately, line management from the operator’s point of view is about balancing risk and return, while maintaining an edge in favour of the ‘house’. From the bettor’s point of view, it is about exploiting opportunities which might arise where one (or more) of the odds making up that over-round are mispriced in the bettor’s favour, a possibility which can arise even when the over-round favours the ‘house.’
In the former, it has been traced, according to contemporaries, to the election of George Washington and has existed in organized markets since the 1860s.
The first recorded example of betting on a papal election, however, can be traced much further back, to the papal conclave of September, 1503, at which time it was considered already an old practice.
The brokers in the Roman banking houses (sensali) who made books and offered odds on who would be elected, made Cardinal Francesco Piccolomini the 100/30 favourite, ahead of Cardinals Guiliano della Rovere (100/15) and Georges d’Amboise (the favourite if judged by the vocal support of the street crowds) at 100/13.
Although Piccolomini is thought to have trailed in the first round of voting with 4 votes to 13 for d’Amboise and 15 for della Rovere, Piccolomini apparently benefited from a switch of votes from d’Amboise to himself in subsequent voting, and duly became Pope Pius III.
The bookmakers were proved right.
The next conclave for which we have the betting odds is that of December, 1521, in which odds were offered on no fewer than twenty cardinals.
Giulio de’Medici, the cousin of Leo X, was the betting favourite, at 100 to 25 (4/1), followed closely by Cardinal Alessandro Farnese at 100/20 (5/1), whose odds shortened to 100 to 40 (5/2) after a Roman mob plundered his house.
Though Farnese at one point came close to being elected Pope, he could not reach the required two-thirds of the vote, and ultimately the cardinals looked outside of the conclave, electing Adrian of Utrecht as Pope Adrian VI.
In the papal election of 1549-50, Cardinal Gianmaria del Monte (who was eventually elected Julius III) had opened in the betting as the 5/1 (against) favourite, but within three days Cardinal Reginald Pole had been established at odds of 4/1. On December 5, as balloting began, Pole was clear favourite at 100/95.
On that day, he received 26 of the 28 votes that would have given him the two-thirds majority required to elect him Pontiff. Although on the point of being made Pope by acclamation, Pole insisted on waiting until he won the formal two-thirds majority.
By the time that four additional French cardinals, opposed to Pole, arrived December 11, however, he was trading at 5/2, and a month later he was being offered at odds of 100/16. His chance had gone.
In the papal conclave of April, 1555, Gian Pietro Carafa stood a good chance of being elected pope, ranking among the top three papabile in the first ballot of the conclave. It is reported that brokers intentionally “spread the rumour that Naples [i.e. Carafa] had died”, in order to attract money on the other candidates. Carafa went on to be elected Pope.
The first 1590 conclave, in September, is the earliest in which reports of insider trading emerged, when two of the key influencers of votes in the conclave, Cardinals Montalto and Sforza secretly agreed to join forces in support of Niccolo Sfondrato.
It is reported that both made fortunes betting on him, at odds of 10/1 the day before he was elected as Pope Urban VII.
As the conclave opened, he had been trading at 100/11, compared to Giambattista Castagna, who was offered at 100/22.
During the second conclave of 1590, Cardinal Gabriele Paleotti at one point increased to an implied probability of 70 per cent in the betting: “Wednesday at the twenty-second hour rumour began to hold Paleotti as pope, and it went on increasing so that at the end of the morning, he had risen to 70 in the wagering.” The odds were not reflected in the outcome. Giovanni Battista Castagna was elected Pope Urban VII.
In 1603, despite a papal bull ‘Cogit Nos’, by Pope Gregory XIV, issued on March 21, 1591, which imposed a penalty of excommunication for wagering on papal or cardinal elections, or length of the papal reign, 21 cardinals were quoted odds of winning by the bookmakers.
The favourite was Cesare Baronius, at 10/1. The closest he came to election, however, was gaining the support of 32 cardinal electors, nine short of the required tally. Ultimately, Alessandro de’Medici became Pope Leo XI.
This ban on papal betting was abrogated in 1918 by Pope Benedict XV’s reforms.
In relation to the papal conclave of 1878, a New York Times correspondent wrote that: “The death and advents of the Popes has always given rise to an excessive amount of gambling in the lottery, and today the people of Italy are in a state of excitement that is indescribable.” There is no available known record, however, of the odds offered on that election. Similarly, the papal conclaves of 1903 and 1922 also attracted a great deal of wagering interest, which was reported widely in the international press, though no known record remains of the odds offered.
Bookmaker odds in Milan are available, however, for the 1958 conclave, which show Cardinal Angelo Roncalli the 2/1 favourite, followed by Cardinals Agagianian and Ottaviani at 3/1, then Stefan Wyszynski and Giuseppe Siri at 4/ 1. The odds were justified when Cardinal Roncalli was elected Pope John XXIII.
For the first conclave of 1978, bookmakers in London were offering odds of 5/2 about Cardinal Sergio Pignedoli, 7/2 about Sebastian Baggio and Ugo Poletti and 4/1 about Carlo Benelli. The best odds about a non-Italian were 8/1 about Johannes Willebrands. Of these only Pignedoli showed any strength in the voting, unconfirmed reports of the voting indicating that he obtained about 18 votes in the first ballot, compared to about 23 for Albino Luciani and 25 for Giuseppe Siri. Ultimately, Cardinal Luciani was elected Pope John Paul I.
For the second conclave of 1978, following the death of Pope John Paul I, the Associated Press noted that:
“Once again, there is no odds-on favourite to be elected as the new pope of the Roman Catholic Church … mentioned most often are Corradi Ursi, Salvatore Pappalardo, Ugo Poletti, Giuseppe Siri, Giovanni Colombo, Giovanni Benelli and Antonio Poma… Non-Italian front-runners include Argentinian Eduardo Pironio, 57, and Dutchman Johannes Willebrands, 68.”
Cardinal Carol Wojtyla, archbishop of Krakow, was elected Pope John Paul II, after the eighth ballot.
In 2005, Cardinal Joseph Ratzinger opened in the betting at 12/1 with one major bookmaker.
At that point, another leading bookmaker made Cardinal Arinze favourite, with Archbishop Tettamanzi, Cardinal Ratzinger and Cardinal Hummes as the next in the betting.
After three ballots, Ratzinger was favourite on two out of the three online betting boards monitored by CNN, his shortest odds being 5/2. He was at that point in the conclave being offered at between 9/2 favourite and 11/2 second favourite.
By the last day of the conclave, Cardinal Ratzinger had shortened to a clear 3/1 favourite, closely followed by Carlo Martini at 100/30 and Jean-Marie Lustiger at 7/2.
By that point, Francis Arinze had dropped back to 8/1, the same price as Claudio Hummes (who was now in the top six in all three lists). He had opened at 12/1. At the same time, Jorge Bergoglio was trading at 12/1 and Angelo Scola at 25/1.
According to a newspaper report, “among those speculating about who the next pope will be, the big money – literally is on Joseph Ratzinger, who delivered a stirring homily at the late Pope’s funeral … As of yesterday, most gambling sites gave Ratzinger … the best odds, with a host of second-tier candidates not far behind.”
Side bets were available on the name of the next pope.
Benedict was the 3 to 1 favourite. John Paul was offered at 7 to 2. Pius at 6 to 1. Peter at 8 to 1. John at 10 to 1.
Joseph Ratzinger was elected Benedict XVI.
The first show of odds following the 2005 conclave for the successor to Benedict was: Angelo Scola 6-1; Christoph Schonborn 7-1; Oscar Maradiga 7-1; Jorge Bergoglio 9-1; Francis Arinze 10-1; Dionigi Tettamanzi 25-1.
In 2013, a survey of the so-called experts made Angelo Scola favourite, although the expert assessment and the betting odds diverged to some degree after that. A survey of Vatican watchers by YouTrend.It listed Timothy Dolan of the United States as the second most likely pope, followed by Cardinals Marc Ouellet, Odilo Scherer and Thomas O’Malley. Luis Tagle of the Phillipines was sixth was ranked sixth. Some of the bookmakers’ favourites, notably Cardinals Turkson and Bertone, did not appear on this experts’ list.
The implied win probabilities in the Oddschecker display of best bookmaker odds on March 3^{rd} were as follows: Scola, 23%; Turkson, 22%; Bertone, 16%; Ouellet, 12%; Bagnasco, 10%; Ravasi, 8%; Sandri, 7%; Erdo, 7%; Scherer, 6%; Schonborn, 6%; Maradiaga, 5%; Arinze, 5%; O’Malley, 4%; Tagle, 4%; Bergoglio, 4%; Dolan, 3%; Hummes, 3%; Grocholewski, 3%; Dziwisz, 3%; Carrera, 2%; Piacenza, 2%; Marini, 2%; Rylko, 2%; Sarah, 2%; Martino 2%. Note that the probabilities add up to more than 100 due to rounding and the in-built margin in the bookmakers’ odds.
A Washington Post analysis, published on March 11th, calculated the implied probabilities of the ‘frontrunners’ based on betting sites including the betting exchange, Betfair.
The results were: Scola, 19.9%; Scherer, 11.9%, Turkson, 9.7%; Bertone, 8.3%; Ouellet, 5%; Erdo, 4.9%; O’Malley, 3.8%; Schonborn, 3.7%; Ravasi, 3.4%; Tagle, 2.6%; Sandri, 2.5%; Dolan, 2.3%; Bagnasco, 2.3%.
On the morning of the final ballot, on March 13th, 2013, the Guardian newspaper Liveblog reported that: “Ladbrokes has Scola at 9/4, Scherer at 3/1 and Turkson at 6/1. Paddy Power has Scola at 11/4, Scherer at 7/2 and Turkson at 9/2.”
A post by Vatican Insider journalist Andrea Tornielli was also published ahead of the final ballot, stating that “The first casting of ballots, which will serve as a primary, will see votes merge towards the Archbishop of Milan, Angelo Scola, as well as the Canadian Marc Ouellet and the Brazilian Odilo Pedro Scherer. Some votes might also go to the Argentinian Jorge Mario Bergoglio and to other cardinals mentioned during the past few hours, such as the Sinhalese Malcolm Ranjith, the American Timothy Dolan and others. It remains to be seen if, among these nominations, there will be one able to garner at least two-thirds of the votes.”
Despite this level of detail, the same article declared that “From the moment cardinal electors entered the Santa Marta residence, they have not had any contact with the outside world and have to use protected paths that are constantly under surveillance, to get about. Every space they enter is monitored and blocked off from all forms of communication… All those who have to access the Holy See during the Conclave are bound to the strictest confidentiality.”
Then came the three strikes of the clock.
The first strike of the clock was a post by Vatican Insider journalist Giacomo Galeazzi, time-stamped on Vatican Insider Twitter at 8.24am that morning. It noted that there were only five candidates left in the running: Scola, Scherer, Bergoglio, Ouellet, Dolan.
The second strike of the clock was a link to a post by Vatican Insider journalist Giacomo Galeazzi, time-stamped on Vatican Insider Twitter at 11.12am: “After the first negative scrutinies, lunch breaks and dinners in Santa Marta House, the cardinals’ residence during the conclave, become opportunities for informal discussions on disregarding candidates with weaker consensuses, to the advantage of the papabile who have obtained more votes so far (Scola, Bergoglio, Ouellet).”
So, by 11.12 am, according to Galeazzi, it was effectively down to three – Cardinals Scola, Bergoglio and Ouellet.
The third strike of the clock came at 11.57am, when the Guardian Liveblog reported that: “La Stampa’s Vatican Insider claims that most of the votes have been going to Cardinals Scola, Bergoglio and Ouellet. This morning it was claiming most of them were going to Scola, Scherer, Bergoglio, Ouellet and Dolan. But it’s hard to know where they can be getting this information from.”
So what was actually going on while the clock was striking once, twice, thrice? A post-election report, published in La Repubblica, claims that Scola received approximately 35 votes in the first vote, to 20 for Bergoglio and 15 for Ouellet. National Catholic Reporter also reports that there was some support for Scherer: “After two rounds of voting Wednesday morning, it had become clear that neither Scola nor Scherer were likely to cross the finish line and gain the 77 votes needed for election … The fourth ballot, the first of Wednesday afternoon, saw Bergoglio separate himself from the pack.”
So it appears that Galeazzi’s tweeted reports conformed broadly to what we now understand to have been the case. Somehow it seems he knew!!!
But the markets failed to respond except for a flicker towards Bergoglio on the exchanges after the Guardian Liveblog posted the niche Galeazzi tweets to their wider audience.
So, either the new information was not (for good or bad reason) sufficiently believed. Or it was for the most part overlooked by those trading on the exchanges. Or the market was not sufficiently liquid to make it possible to earn a significant return, so most sophisticated traders did not bother to participate.
Whatever the reason, the betting markets did not perform as well as might have been expected in responding to new public information, which subsequently turned out to be accurate, unless the reports were accurate by sheer chance and deserved to be disbelieved. After all, it was ‘Vatican Insider’ itself that declared how “All those who have to access the Holy See during the Conclave are bound to the strictest confidentiality.”
This cannot be explained either in terms of the fog of conflicting signals as there were no other credible sources issuing conflicting information.
So the ‘Galeazzi anomaly’, as I term it, turns into a mystery, partly because he seemed to know what he shouldn’t have known, but also because hardly anyone seemed to believe him. Giacomo Galeazzi shouted wolf, and there was a wolf! It is a lesson that some, in an efficient market, will now have learned.
Further Reading.
Vaughan Williams, L. and Paton, D., (2015), Forecasting the Outcome of Closed-Door Decisions: Evidence from 500 Years of Betting on Papal Conclaves, Journal of Forecasting, 34 (5), August, 391-404.
Assume, for example, that Mr. Miller and Mr. Stiller both start with £1,000.
Now Mr. Miller places a level £10 stake on 100 horses quoted at 2 to 1
Mr. Stiller places a level £10 stake on 100 horses quoted at 20 to 1.
Who is likely to end up with more money at the end?
My Ladbrokes Flat Season Pocket Companion for 1990 provides a nicely laid out piece of evidence here for British flat horse racing between 1985 and 1989. The table conveniently presented in the Companion shows that not one out of 35 favourites sent off at 1/8 or shorter (as short as 1/25) lost between 1985 and 1989. This means a return of between 4% and 12.5% in a couple of minutes, which is an astronomical rate of interest. The point being made is that broadly speaking the shorter the odds, the better the return. The group of ‘white hot’ favourites (odds between 1/5 and 1/25) won 88 out of 96 races for a 6.5% profit. The following table looks at other odds groupings.
Odds Wins Runs Profit %
1/5-1/2 249 344 +£1.80 +0.52
4/7-5/4 881 1780 -£82.60 -4.64
6/4 -3/1 2187 7774 -£629 -8.09
7/2-6/1 3464 21681 -£2237 -10.32
8/1-20/1 2566 53741 -£19823 -36.89
25/1-100/1 441 43426 -£29424 -67.76
An interesting argument advanced by the Strathclyde-based statistician Dr. Robert Henery in 1985 is that the favourite-longshot bias is a consequence of bettors discounting a fixed fraction of their losses, i.e. they underweight their losses compared to their gains.
This argument also explains an observed link between the sum of bookmakers’ prices and the number of runners in a race. The prices being summed here are simply the odds. If, for example, odds of 3/1 (against) are offered about each of the five horses in a race, the implied probability of winning for each horse is ¼ and the sum of prices is 5/4.
In this context, an ‘over-round’ is defined as the excess of the sum of prices over 1, in this case ¼.
The rationale behind Henery’s hypothesis is that bettors will tend to explain away and therefore discount losses as atypical, or unrelated to the judgment of the bettor.
This is consistent with contemporaneous work on the psychology of gambling, such as Gilovich in 1983 and Gilovich and Douglas in 1986.
These studies demonstrate how gamblers tend to discount their losses, often as ‘near wins’ or the outcome of ‘fluke’ events, while bolstering their wins.
Let’s look more closely at how the Henery odds transformation works.
If the true probability of a horse losing a race is q, then the true odds against winning are q/(1-q).
For example, if the true probability of a horse losing a race (q) is ¾, the chance that it will win the race is ¼, i.e. 1- ¾. The odds against it winning are: q/(1-q) = 3/4/(1-3/4) = 3/4/(1/4) = 3/1.
Henery now applies a transformation whereby the bettor will assess the chance of losing not as q, but as Q which is equal to fq, where f is the fixed fraction of losses undiscounted by the bettor.
If, for example, f = ¾, and the true chance of a horse losing is ½ (q=1/2), then the bettor will rate subjectively the chance of the horse losing as Q = fq.
So Q = ½. ¾ = 3/8, i.e. a subjective chance of winning of 5/8.
So the perceived (subjective) odds of winning associated with true (objective odds) of losing of 50% (Evens, i.e. q=1/2) is 3/5 (60%), i.e. odds-on.
This is derived as follows:
Q/(1-Q) = fq/(1-fq) = 3/8/(1-3/8) = 3/8/(5/8) = 3/5
If the true probability of a horse losing a race is 80%, so that the true odds against winning are 4/1 (q = 0.8), then the bettor will assess the chance of losing not as q, but as Q which is equal to fq, where f is the fixed fraction of losses undiscounted by the bettor.
If, for example, f = ¾, and the true chance of a horse losing is 4/5 (q=0.2), then the bettor will rate subjectively the chance of the horse losing as Q = fq.
So Q = 3/4. 4/5 = 12/20, i.e. a subjective chance of winning of 8/20 (2/5).
So the perceived (subjective) odds of winning associated with true (objective odds) of losing of 20% (4 to 1, i.e. q=0.8) is 6/4 (40%).
This is derived as follows:
Q/(1-Q) = fq/(1-fq) = 12/20 / (1-12/20) = 12/8 = 6/4
To take this to the limit, if the true probability of a horse losing a race is 100%, so that the true odds against winning are ∞ to 1 against (q = 1), then the bettor will again assess the chance of losing not as q, but as Q which is equal to fq, where f is the fixed fraction of losses undiscounted by the bettor.
If, for example, f = ¾, and the true chance of a horse losing is 100% (q=1), then the bettor will rate subjectively the chance of the horse losing as Q = fq.
So Q = 3/4. 1 = 3/4, i.e. a subjective chance of winning of 1/4.
So the perceived (subjective) odds of winning associated with true (objective odds) of losing of 100% (∞ to 1, i.e. q=1) is 3/1 (25%).
This is derived as follows:
Q/(1-Q) = fq/(1-fq) = 3/4 / (1/4) = 3/1
Similarly, if the true probability of a horse losing a race is 0%, so that the true odds against winning are 0 to 1 against (q = 0), then the bettor will assess the chance of losing not as q, but as Q which is equal to fq, where f is the fixed fraction of losses undiscounted by the bettor.
If, for example, f = ¾, and the true chance of a horse losing is 0% (q=0), then the bettor will rate subjectively the chance of the horse losing as Q = fq.
So Q = 3/4. 0 = 0, i.e. a subjective chance of winning of 1.
So the perceived (subjective) odds associated of winning with true (objective odds) of losing of 0% (0 to 1, i.e. q=0) is also 0/1.
This is derived as follows:
Q/(1-Q) = fq/(1-fq) = 0 / 1 = 0/1
This can all be summarised in a table.
Objective odds (against) Subjective odds (against) | |
Evens 3/5 | |
4/1 6/4 | |
Infinity to 1 3/1 | |
0/1 0/1 |
We can now use these stylised examples to establish the bias.
In particular, the implication of the Henery odds transformation is that, for a given f of ¾, 3/5 is perceived as fair odds for a horse with a 1 in 2 chance of winning.
In fact, £100 wagered at 3/5 yields £160 (3/5 x £100, plus stake returned) half of the time (true odds = evens), i.e. an expected return of £80.
£100 wagered at 6/4 yields £250 (6/4 x £100, plus the stake back) one fifth of the time (true odds = 4/1), i.e. an expected return of £50.
£100 wagered at 3/1 yields £0 (3/1 x £100, plus the stake back) none of the time (true odds = Infinity to 1), i.e. an expected return of £0.
It can be shown that the higher the odds the lower is the expected rate of return on the stake, although the relationship between the subjective and objective probabilities remains at a fixed fraction throughout.
Now on to the over-round.
The same simple assumption about bettors’ behaviour can explain the observed relationship between the over-round (sum of win probabilities minus 1) and the number of runners in a race, n.
If each horse is priced according to its true win probability, then over-round = 0. So in a six horse race, where each has a 1 in 6 chance, each would be priced at 5 to 1, so none of the lose probability is shaded by the bookmaker. Here the sum of probabilities = (6 x 1/6) – 1 = 0.
If only a fixed fraction of losses, f, is counted by bettors, the subjective probability of losing on any horse is f(qi), where qi is the objective probability of losing for horse i, and the odds will reflect this bias, i.e. they will be shorter than the true probabilities would imply. The subjective win probabilities in this case are now 1-f(qi), and the sum of these minus 1 gives the over-round.
Where there is no discounting of the odds, the over-round (OR) = 0, i.e. n times correct odds minus 1. Assume now that f = ¾, i.e. ¾ of losses are counted by the bettor.
If there is discounting, then the odds will reflect this, and the more runners the bigger will be the over-round.
So in a race with 5 runners, q is 4/5, but fq = 3/4 x 4/5 = 12/20, so subjective win probability = 1-fq = 8/20, not 1/5. So OR = (5 x 8/20) – 1 = 1.
With 6 runners, fq = ¾ x 5/6 = 15/24, so subjective win probability = 1 – fq = 9/24. OR = (6x 9/24) – 1 = (54/24) -1 = 1_{1/4. }
With 7 runners, fq = ¾ x 6/7 = 18/28, so subjective win probability = 1-fq = 10/28. OR = (7 x 10/28) – 1 = 42/28 = 1_{1/2}
If there is no discounting, then the subjective win probability equals the actual win probability, so an example in a 5-horse is that each has a win probability of 1/5. Here, OR = (5×1/5) – 1 = 0. In a 6-horse race, with no discounting, subjective probability = 1/6. OR = (6 x 1/6) – 1 = 0.
Hence, the over-round is linearly related to the number of runners, assuming that bettors discount a fixed fraction of losses (the ‘Henery Hypothesis’).
If the Henery Hypothesis is correct as a way of explaining the favourite-longshot bias, the bias can be explained as the natural outcome of bettors’ pre-existing perceptions and preferences.
This is quite consistent with a market efficiently processing the information available to it.
Are there other explanations for the favourite-longshot bias, and the observed link between over-round and runners, which do not rely on the Henery Hypothesis? Any coherent theory of the favourite-longshot bias should be able to explain both observed regularities. That is a topic for another time.
For convenience, assume that all dates in the calendar are equally likely as birthdays, and ignore the Leap Year special of February 29^{th}
The first thing to look at is the likelihood that two randomly chosen people would share the same birthday.
Let’s call them Fred and Felicity. Say Felicity’s birthday is May 1^{st}. What is the chance that Fred shares this birthday with Felicity? Well there are 365 days in the year, and only one of these is May 1^{st} and we are assuming that all dates in the calendar are equally likely as birthdays.
So, the probability that Fred’s birthday is May 1^{st} is 1/365, and the chance he shares a birthday with Felicity is 1/365.
So what is the probability that Fred’s birthday is not May 1^{st? }It is 364/365. This is the probability that Fred doesn’t share a birthday with Felicity.
More generally, for any randomly chosen group of two people, the probability that the second person has a different birthday to the first is 364/365.
With 3 people, the chance that all three are different is the chance that the first two are different (364/365) multiplied by the chance that the third birthday is different (363/365).
So, the probability that 3 people have different birthdays = 364/365 x 363/365
This can be written as (364)_{2 }/ 365^{2}
Similarly, probability that 5 people have different birthdays = (364)_{4} / 365^{4}
= 364x363x362x361/365^{4}
So far, the chance of no matches is very high. But by the tenth person the probability of no matches is:
(364/365)*(363/365)(362/365)*(361/365)(360/365)*(359/365)(358/365)*(357/365) (356/365) = 0.8831
More generally, for n people, probability they all have different birthdays =
(364)_{n-1 } / 365^{n-1}
For 23 people, probability of all different birthdays = (364)_{22 }/ 365^{2} = 0.4927
For 22 people, probability of all different birthdays = (364)_{21 }/ 365^{2} = 0.5243
So, in a group of 23 people, there is a (1-0.4927) = 0.5073 chance of that at least two of the group share a birthday.
So how large should a randomly chosen group of people be, to make it more likely than not that at least two of them share a birthday? The answer is 23.
The intuition behind this is quite straightforward if we recognise just how many pairs of people there are in a group of 23 people, any pair of which could share a birthday.
In a group of 23 people, there are, according to the standard formula, ^{23}C_{2 }pairs of people (called 23 Choose 2) pairs of people.
Generally, the number of ways k things can be chosen from n is:
^{n} C _{k} = n! / (n-k)! k!
Thus, ^{23}C_{2 }= 23! / 21! 2! = 23 x 22 / 2 = 253
So, in a group of 23 people, there are 253 pairs of people to choose from.
_{ }Therefore, a group of 23 people generates 253 chances, each of size 1/365, of having at least two people in the group sharing the same birthday.
These chances have some overlap: if A and B have a common birthday, and A and C have a common birthday, then inevitably so do B and C. So the probability of at least two people sharing a birthday in a group of 23 is less than 253/365 (69.3%). It is, as shown previously, 50.73%.
To conclude, the next time you see two football teams line up, include the referee. It is now more likely than not that two of those on the pitch share the same birthday. Strange, but true!