Editor's note: The following column is the last part of an ongoing series of articles by Aaron Brown examining the claims made in The Physics of Wall Street: A Brief History of Predicting the Unpredictable, a new book by James Owen Weatherall. Click here to read Part 1.
I called James Weatherall’s The Physics of Wall Street
the most arrogant book in the world. I’ve used it as a foundation to discuss the common errors many people with quantitative training make when first thinking about finance. I conclude by considering Weatherall’s main point, that the cure for financial problems is for everyone to think like a physicist.
Weatherall tells us that when he began his book, he spoke to his PhD advisor who told him it was “impossible to do science on Wall Street” because “investment banks and hedge funds are usually very secretive.” Weatherall contrasts this to the openness and speed with which physicists submit their insights to peer-reviewed journals. This is the first sense in which he would like people in finance to think like physicists.
It’s pretty arrogant to dismiss and entire field as unscientific on the basis of a casual opinion from an outsider. In fact, my experience is almost the complete opposite. I know of many cases of academics who are slow to reveal insights, waiting until they have developed them enough to claim substantial credit. Hoarding data and results is not uncommon. Most important communication is outside the peer-review process; by the time a finding appears in a major journal it is generally long since known among the researchers who care. While I think the peer review provides an important audit function in physics, it is more relevant to grants and promotions and credit than to the dissemination of new ideas—and electronic communication has made this even more true. Peer review is part of the administrative machinery of government-funded projects—academic, military, or bureaucratic—it has little to do with science. The great advances in physics have been the results of informal network collaboration based on personal reputations, not the quality control from anonymous referees. The ponderous rules have a cost, driving some smart people away (often into more anarchic fields like finance) and consuming time and energy that could be spent on science.
Ironically, the few firms in finance that correspond to Weatherall’s advisor’s view are the ones celebrated by Weatherall for their scientific foundations. Renaissance Technologies and D.E. Shaw are two of the most secretive firms in the world. Generally in finance, you need to communicate your ideas to be successful. You need people to trade with you, and often to finance you or to invest in you or to provide other services. Few people succeed by keeping some formula secret, success generally comes from creating a mutually profitable cycle, the bigger the better. You see a lot more effort on Wall Street devoted to communication than to protecting secrets.
Another reason for the openness is success in finance is objective, few people are interested in arguing about credit for ideas, and no one takes such people seriously. You go into finance to participate in an adventure, and of course to get rich, not to attach your name to a theorem or to get tenure or to win awards from outside bodies. If your idea fails, it fails—however friendly you are with a journal editor. If your idea succeeds, it succeeds; you don’t need good relations with granting agencies.
I admit it is a common belief that finance is secretive. I think the reason is that it’s so simple. People read the books by or about successful practitioners, or speak to them in person, and do not believe that it can be so simple to get so rich. Actually, it’s more that they don’t want to believe it. They are not interested personally in getting rich that way, but it’s distressing to know that others do it. It’s more comforting to believe that the practitioners are crooks, or that they are liars, hiding the secret of their success.
An important detail is that simple does not mean easy. It does not take a genius IQ or formal study to get rich on Wall Street, in fact those things may be more likely to slow you down than to help you. You do not need a stunning insight, or even a minor new wrinkle on an old idea. Success in trading does take skills that few people possess. In the areas I am most familiar with it takes extraordinary attention to data quality, skeptical objectivity to an uncommon degree, and rigid risk discipline. My rule of thumb is that 80% of the people who try trading don’t bother to learn how to do it, 80% of the people who learn how to do it can’t do it or don’t want to do it, and 80% of the people who learn how and can do it fail due to inability to deal properly with risk. That means one person out of 125 succeeds in trading, yet none of the three prerequisites is a particularly unusual accomplishment.
There is room for intelligence and creativity in finance, but it’s not directly related to making money. At least for people of a certain type, the financial system is by far the easiest place to change the world. Many refugees from other fields, including physics, find it liberating to work in a field so open to new ideas, so objective in evaluating them, and so influential. Another nice feature is that if you screw up, it’s only money, nobody dies. It can be a lot of money, and it can cause a lot of hardship, but unlike life and death, you can make up for it tomorrow. This is only half of what Weatherall means by thinking like a physicist. He also means that physicists develop models using simplifying assumptions, then use them to predict. They are aware of the shortcomings of the models and they use deviations from model predictions to come up with improved models. In finance he claims people instead take models developed by physicists and implement them without being aware of their defects. They ignore the deviations from models until they cause a disaster. They are stuck on first and second generation models, while physics has moved on to third and fourth generation.
It is easy to come up with examples that fit Weatherall’s paradigm in physics as well as other fields. Isaac Newton assumed an inverse square law for gravitation and considered two incompressible perfect spheres with nothing else in the universe. This allowed him to compute the orbits of the planets to tolerable precision. Deviations from these predictions taught us new things: unexpected changes in the orbit of Uranus led to the discovery of Neptune, the anomalous precession of the perihelion of Mercury’s orbit became one of the major proofs of general relativity. We now have far more sophisticated models of gravity than Newton’s, and are working on refining them further.
But you can also tell this story another way. Newtonian orbits gave rise to the idea of a clockwork universe, which turned out to be deeply misleading. In the actual solar system, tidal forces and the influence of other planets on orbits mean that orbits are constantly changing. The solar system does appear to be reasonably stable, but for an entirely different reason than the one suggested by the simplified two-body problem. The adoption of important ideas was delayed due to over-rigid Newtonian thinking.
Obviously Newtonian gravity was a tremendous intellectual advance, and probably a necessary step to further progress in physics. So it is good that some people think this way. But it is also good that some people thought other ways and were willing to come out of left field with inconsistent ideas rather than working on refining existing models. Intellectual history is filled with advances that came about because someone insisted stubbornly that inconvenient observations were correct, and other advances that came about because someone had the courage and insight to ignore observations.
Weatherall looks at finance and sees bad physics. He mistakes a vigorous intellectual diversity for stupidity. He credits all the parts he likes to physicists, independent of the historical evidence. Everything else is bad physics. He blames the failures on bad physics, but failure is the way applied fields progress. As civil engineering professor Henry Petrosky wrote, no one wants to learn from failure, but we don’t learn enough from success to advance the state of the art.
I will close by returning a favor to Weatherall and discussing the economics of physics, which I think explains a lot about his attitude toward finance. Why does someone get a PhD in physics? Economists are apt to answer the question in terms of signaling. An educational credential takes investment. While some of that effort may pay off directly in learning skills, the investment seems disproportionate to the skill gain. That is, you could learn all the physics you learn in a PhD program faster and with less effort if you didn’t have to worry about courses, exams, papers, qualifying exams, dissertations, teaching, departmental politics, interdepartmental politics, and other things. That’s especially true when you add in all the effort of the people who do not complete the degree, which in an economic sense form part of the cost. Some people disagree and claim PhD programs teach intangible skills essential for a life of productive research and there is no way to do that more efficiently, but I know of no evidence to back that up, and a lot of evidence against it (none specifically about physics PhDs, but plenty about other educational credentials).
The wasted effort is a costly signal. A credential has a value, and it will look like a good deal to people who find the cost low, but a bad deal to people who find the cost high. Getting a physics PhD proves you are the kind of person who finds the effort less objectionable (or more pleasurable) than most people. Employers observe the credential and can trust it, because the kind of people who hate the kind of stuff you have to do to get a physics PhD, won’t get one, even if the career value is high.
We can rule out about 95% of the population because they don’t have the quantitative aptitude to get B grades in the basic physics courses PhD programs require, at least not with reasonable effort. The remaining 5% could get PhDs in other quantitative fields, or do something that does not require a doctorate. With the exception of a few pure theoretical physicists (who probably choose the field for non-economic reasons anyway) almost anything you study in physics you can study in another field as well. The main thing that distinguishes physics is the demand by government. Roughly a third of physicists in the US work directly for the government in the defense department, NASA, national laboratories and other places. Another third are university faculty or researchers, virtually all supported directly or indirectly by the government. Of the remaining third of physicists, who work in private industry, about half work in government-related fields like defense contracting or health care. These are current statistics; government employment, especially military, was higher in the past, when many current physicists chose their field. Other developed countries have a broadly similar picture. Of course governments hire other science PhDs as well, but far more physicists. Moreover, physics jobs and research are more likely to involve gigantic expensive projects than other sciences. This leads us to expect physics PhD programs will be least costly for people who thrive in government employment, who can be trusted with expensive and dangerous equipment, who are comfortable networking within large organizations. People who can’t keep secrets, who ask “what’s the worst that can happen?” before throwing a random switch, who need unfettered intellectual freedom and cannot be polite to people they don’t like, should find physics PhD programs intolerable.
This puts Weatherall’s one-size-fits-all thinking in a different light. Observation, model, refinement, new model makes a lot of sense in a bureaucratic setting, or for a large, expensive project. It’s stable and low risk compared to other methods of investigation like “try it and see” or “let’s throw away the textbook and start from scratch” or that perennial favorite of fiction, “that’s so crazy it just might work.” What Weatherall really wants is a financial system run by certified salaried experts rather than the messy and wildly successful system we have staffed by annoyingly rich individualists whose only credentials are hubris and competitive success.
Okay, that’s what a lot of people want. It makes him one more deluded high-IQ guy anxious to take a small step on the road to serfdom—nothing unusual there. What makes him arrogant is that he resolutely refuses to see that there is beautiful and successful theory as well as effective practice, and that it is done by serious thinkers with or without credentials. He squeezed finance into a narrow mold, which is not even all of physics, and finds the result pathetic. Then he writes a book without any effort to test his conclusions or discuss things with experienced practitioners, and asks the government to pay him to figure out the answers, which he’s sure he can do better than the people who have worked and won their way to positions of influence today.
I hope this series has been useful. You don’t have to read Weatherall’s book to make sense of it, the same errors he makes are made by many other people. I thank Weatherall for expressing them so clearly because it makes them excellent springboards to the truth.
Links to previous stories in this series: Part 1, Part 2, Part 3, Part 4, Part 5, Part 6, Part 7, Part 8, Part 9, Part 10, Part 11, Part 12, Part 13, Part 14, Part 15, Part 16, Part 17.