Editor's note: The following column is the third part of an ongoing series of articles by Aaron Brown examining the claims made in The Physics of Wall Street: A Brief History of Predicting the Unpredictable
, a new book by James Owen Weatherall. Click here to read Part 1. Click here to read part 2.
Several weeks ago, I started an account of James Weatherall’s new book, The Physics of Wall Street
, which I claim is one of the most arrogant books in the world, and therefore an invaluable teaching aid. This week I want to talk about how Weatherall misunderstands the important research that began with a study of roulette wheels.
Weatherall appears to have skimmed mathematician Ed Thorp’s classic work Beat the Dealer
. He refers to Ed as a “dilettante” for working with Claude Shannon to beat roulette before moving on to finance. (However, when physicist Doyne Farmer did the same thing years later, beating roulette is referred to as a suitable topic for a physics dissertation; there are different rules for physicists and mortals.) In fact, the attack on roulette was a key moment in the development of financial theory.
The traditional way to beat roulette was to use a random walk model, the only kind of model Weatherall recognizes in finance. You assume each number comes up at random, and independently, but according to a non-uniform probability distribution. You watch a wheel for an extended period, recording the results, and look for numbers that come up often enough to overcome the house edge and be profitable bets. However, many people claimed that method was impossible in large casinos because the probability distribution of numbers was too close to uniform.
Ed’s insight was that you could make money either way. If the wheels were sufficiently non-uniform, you could do it the traditional way. But if they were so precisely engineered that outcomes were uniform, they would be predictable if you had even a tiny amount of information about the spin. The effort to ensure uniform randomness creates predictability. This duality is at the heart of modern finance.
In the roulette case, it was possible to measure the speed of the ball and the speed of the wheel before you had to place your bets. The precision of the wheels meant that it was possible to compute with considerable accuracy the number that would be under the ball at the time it left the rim of the wheel. From that point on, the process is chaotic, so it’s not practical to predict the final number exactly. But the distribution of final numbers, conditional on knowing the number under the ball when it starts its descent, is highly non-uniform and profitable betting strategies are possible.
We have to tease out a distinction between concepts that are often covered by the word “random”: unpredictable and uniform. One person says the roulette wheel is random; because you know nothing about the next spin different from any other spin, the wheel is unpredictable. Another person says the wheel is random because each number comes up with exactly the same long-term frequency, the wheel is uniform.
It’s easy to select numbers with uniform distribution -- just call them out in order, going back to the beginning when you reach the end. It’s also pretty easy to select numbers at random; you don’t need precise engineering, just a system chaotic enough to make computing the outcome impractical using the information available to players. But casinos have to make devices that are both uniform and random. That’s not only tricky, but Ed and many followers proved it is beyond the ability of casinos, and no one has yet demonstrated convincingly that it’s possible in practice at all. A roulette wheel attempts to solve the problem with a precisely engineered wheel to ensure uniformity. Chaos is used to simulate randomness in two ways. First, a human spins the wheel and the ball in different directions and there are so many revolutions of each that small changes in initial conditions—that is, slightly different speeds of the wheel or the ball or a slightly different time gap between the time the wheel and ball are spun—make a large difference to the final outcome. Differences in initial conditions that are too small to detect by eye make a difference of more than a full revolution of the wheel by the time the ball finally drops, so the process is essentially random—unless you have electronic help. There are people who claim to be able to make useful predictions without devices, but I have never seen this demonstrated convincingly and do not believe it personally.
Chaos reenters the process once the ball descends from the lip. The ball skips and spins erratically before it finally settles down into a slot. While the outcome of this process cannot be predicted even with lots of information and computing power—meaning it is random in practice even to someone with electronic aids—it is nowhere near uniform. Ed and Shannon bypassed the first chaotic input by measuring the system after the spin began, used the uniformity of the next phase to their advantage, and exploited the non-uniformity of the final phase to make bets that were profitable on average.
In a sloppily built wheel—say, one that is not quite circular, or is slightly tilted, or wobbles a bit, or is made of materials with varying elasticity, hardness, and friction—the initial measurement would be useless. However, such a wheel would be likely to favor some numbers over others. In a perfectly built wheel, or say, a computer simulation of an ideal wheel, it would be possible to compute the final number given precise measurement of the initial conditions. In any wheel in between that anyone has ever built, some combination of those approaches results in significant advantage to the bettor.
People have attacked every form of casino-generated randomness using the same conceptual approach. Where things are non-uniform, exploit it by betting the patterns. Where things are uniform, exploit the precision required to ensure uniformity by building useful inferences from observations casinos think are too small to matter. Shuffles, dice, wheels, drawing of balls or slips of paper and electronic random number generators have all been beaten, at one time or another.
This brings us to one of the reasons I reacted so strongly to Weatherall’s claim that physicists were responsible for reengineering the global financial system. It’s not that I think people from other fields made larger contributions. That happens to be true, but it’s a sterile debate. After all, “physics” can cover the entire range of quantitative science, and there is no dispute that it was quants who did the job. Everyone involved was at least pretty good at physics, and some were fully-qualified professional physicists, and others were considerably better at physics than most fully-qualified professionals.
The problem is naming any field suggests that the change in the financial system was an intellectual achievement, brought about by people importing ideas. The real common denominator, apart from quantitative ability, was an engineer’s hubris: a combination of insistence on clear objective validation, a desire to push all limits, and an unlimited willingness to bet on oneself. In Ray Bradbury’s words, we were people who believed in jumping off cliffs and building our wings on the way down. That may sound reckless, but it’s quite different. A reckless person jumps off a cliff, then thinks about what to do next. We thought things through carefully, prepared as much as possible, but then we jumped. We paid a price for that—failures, crashes, humiliations, ridicule—including ridicule because people said the problems we attacked were unimportant or disreputable, or that they were impossible, or that we were pathetic nerds without the expert training and machismo to succeed in real risk taking—but in the end, we learned to fly. It’s irritating to have a guy come by after the fact, point to all the people flying around, and say because some of the people flying have PhDs in physics , and because the wings exploit Bernoulli’s principle, and because Daniel Bernoulli was kind of a physicist (you could equally well say mathematician or probabilist), that physicists taught people how to fly.
Okay, people make fun of me for pushing such a heroic version of the development of finance. I am no hero. A slightly less arrogant metaphor is advances in civil engineering. As Henry Petroski describes in his classic To Engineer Is Human: The Role of Failure in Successful Design
, the first implementation of a new idea (such as a skyscraper or suspension bridge) does not fail. The best engineers use the best materials and include huge safety margins in all calculations. But once something has been done, new engineers push the envelope to make things bigger, faster, and cheaper and to satisfy more demanding operational and aesthetic constraints. Eventually someone goes too far and there is a disaster. The accumulation of these experiences defines the new art.
Finance went through precisely this process, many times over. Weatherall is like a guy who points to a modern suspension bridge, in all its refined elegance purchased at great cost, and claims it demonstrates the principle of catenary shapes first discussed by Fausto Veranzio, who was kind of a physicist, and Robert Hooke, who was a physicist. Weatherall views the bridge as a physical principle, with additions to account for the physical properties of the materials and the demands of the project. I view the bridge as the original implementation of a modern suspension bridge with all the non-essentials carved away by tragic experience. He could write a history of the suspension bridge and leave out all the engineers and real bridges and aspects that proved to be non-essential, covering only some theoreticians--who he would label “physicists”—who drew catenaries on paper and mused about the implications (some of his musers were also important engineers, people like Ed Thorp, Fischer Black and Emanual Derman were among the great innovators in finance, but in his book they get credit only for their musings).
Our modern financial system is not an intellectual achievement that belongs to any field; it is an engineering feat that belongs to everyone who participated. It did not spring from a theory in any academic field that led to changes and a natural evolution, it is a system consciously designed by working engineers. Weatherall treats some of the practical problems in beating roulette—broken wires, suspicious pit bosses, burns from the equipment—as amusing sidelights. The ability to surmount real problems was at least as important to the process as the intellectual inspiration behind it.
Next week, we’ll see how this attack on casinos transformed finance.