I have seen in Richard Dawkins’ work many references to the difficulty people have, when looking at an animal, in accepting that it is not the product of a top-down design, but the result of a random process — more exactly the upper bound of a random process, in which (roughly, and only roughly) the most successful mutations tend to make it. Yet my problem is that when those who accept the evolutionary argument look at a computer, at a laser beam, at a successful drug, at a surgical technique, at the spread of a language, at the growth of a city, or at an commercial enterprise, they tend to fall for the belief that its discovery or establishment partook of some grand design. And, in hindsight, some “explanation” will be given as to why it happened: there was a plot — it could not have been an accident.

Alas, we are victims of the narrative fallacy — even in scientific research (but while we learned how to manage it in religion, and to some degree in finance, we do not seem to be aware of its prevalence in research). The pattern-seeking, causality producing machine in us blinds us with illusions of order in spite of our horrifying past forecast errors. I hold that not only discoveries are also largely the result of a random process, but that their randomness is even less tractable than, and not as simple as, biological evolution. While nature might produce milder form of stochasticity, the environment for manmade discoveries is governed by a far, far more severe, wilder form of processes, those called “fat tailed”.

Against what one might expect, this makes me extremely optimistic about the future in several selective research-oriented domains, those in which there is an asymmetry in outcomes favoring the positive over the negative — like evolution. These domains thrive on randomness. The higher the uncertainty in such environments, the rosier the future — since we only select what works and discard the rest. With unplanned discoveries, you pick what’s best; as with a financial option, you do not have any obligation to take what you do not like. Rigorous reasoning applies less to the planning than to the selection of what works. I also call these discoveries positive “Black Swans”: you can’t predict them but you know where they can come from and you know how they will affect you. My optimism in these domains comes from both the continuous increase in the rate of trial and error and the increase in uncertainty and general unpredictability. I have seen in Richard Dawkins’ work many references to the difficulty people have, when looking at an animal, in accepting that it is not the product of a top-down design, but the result of a random process — more exactly the upper bound of a random process, in which (roughly, and only roughly) the most successful mutations tend to make it. Yet my problem is that when those who accept the evolutionary argument look at a computer, at a laser beam, at a successful drug, at a surgical technique, at the spread of a language, at the growth of a city, or at an commercial enterprise, they tend to fall for the belief that its discovery or establishment partook of some grand design. And, in hindsight, some “explanation” will be given as to why it happened: there was a plot — it could not have been an accident.

Alas, we are victims of the narrative fallacy — even in scientific research (but while we learned how to manage it in religion, and to some degree in finance, we do not seem to be aware of its prevalence in research). The pattern-seeking, causality producing machine in us blinds us with illusions of order in spite of our horrifying past forecast errors. I hold that not only discoveries are also largely the result of a random process, but that their randomness is even less tractable than, and not as simple as, biological evolution. While nature might produce milder form of stochasticity, the environment for manmade discoveries is governed by a far, far more severe, wilder form of processes, those called “fat tailed”.

Against what one might expect, this makes me extremely optimistic about the future in several selective research-oriented domains, those in which there is an asymmetry in outcomes favoring the positive over the negative — like evolution. These domains thrive on randomness. The higher the uncertainty in such environments, the rosier the future — since we only select what works and discard the rest. With unplanned discoveries, you pick what’s best; as with a financial option, you do not have any obligation to take what you do not like. Rigorous reasoning applies less to the planning than to the selection of what works. I also call these discoveries positive “Black Swans”: you can’t predict them but you know where they can come from and you know how they will affect you. My optimism in these domains comes from both the continuous increase in the rate of trial and error and the increase in uncertainty and general unpredictability.

Taleb’s optimism about the value we can derive from uncertain environments has parallels with the use of real options to value and analyze the payoffs from different potential courses of action. We know from the Black-Scholes model in financial theory that the value of a financial option (e.g. a call or a put on a stock) increases with the volatility of the underlying equity. Real options are essentially investments which give one the opportunity, but not the necessity, of pursuing further courses of action down the road. Similarly to a financial option, a real option increases in value the more uncertain, or random, an environment we are operating in. It has become cliche at this point to describe the current global business environment as increasingly rapid and complex, but what we can learn from these generalizations is that the value of “keeping your options open” is ever increasing.

Real options sit at a rich crossroads between financial theory and decision theory; expect more discussion of real options here soon!

Read more: Nassim Taleb: The Birth of Stochastic Science

Mauboussin then makes the distinction between endogenous risk, which emerges from a complex system itself, and exogenous risk, which is forced on a system from outside. System dynamics that emerge endogenously, without being precipitated by any outside event, are behind many of the interesting phenomena we see in the markets and society at large today. While endogenous risk is just beginning to be understood, it’s a critical area of research. We’ve looked at some examples of endogenous system dynamics earlier this week with Exploiting the Herd, A Case Study and Exploiting the Herd: Case Study Two. Mauboussin presents some of the common frameworks used to understand endogenous risk:

The first framework is the wisdom of crowds, which writer Jim Surowiecki laid out well a couple of years ago in his book of the same title. The basic idea is simple and somewhat counterintuitive: if you get a diverse group of people together to solve a problem, the group’s answer will typically be better than that of any individual, even an expert. The wisdom of crowds is a more common way of describing a type of complex adaptive system–the heart of the Santa Fe Institute’s work–and is an apt description of the stock market.

The key is that the crowd is only wise under certain conditions. You need agent diversity, an aggregation mechanism, and some sort of incentives. When one or more of these conditions is violated, all bets are off. In human systems, diversity is the most likely condition to be violated. When you take away diversity, the complex system can become fragile and in some cases will lead to large-scale changes. Booms and crashes are good examples of diversity breakdowns in markets. Fads and fashions also illustrate the concept. And that leads to the second framework: diffusion theory.

Technologies, ideas, and illnesses tend to diffuse following an S-curve pattern. So, for example, a new technology will start with only a few adopters, and will grow at a relatively slow rate early on. The rate then accelerates, and the technology takes off. This field has been studied in detail, and is of prime interest to epidemiologists and technologists, just to name two groups. The key point is the growth rate is not stable: it’s low to start, rises, and then slows down again. Also important is that most technologies or ideas don’t diffuse–they simply sputter out.

The final framework is network theory, or how the individual nodes in a network are connected. Network theory bears on a wide variety of phenomena, including your network of friends, transmitters on the power grid, or the spread of disease. In recent years, scientists have made major advances in understanding the nature of networks. We now know that the structure of the network is important in understanding how things get transmitted over the network.

There are two features of these frameworks worth emphasizing. First, they are non-linear. For example, in the case of the wisdom of crowds you can reduce diversity, reduce diversity, and nothing happens. Then you reduce it a bit more and the system reacts violently–the proverbial straw that broke the camel’s back. Many of you know this idea as the tipping point.

That leads to the second feature: lack of proportionality. The size of the perturbation and the outcome are not always linked. Sometimes small perturbations lead to large outcomes, and vice versa. When you combine a lack of linearity with a lack of proportionality, it’s not hard to see that predictions are difficult and cause and effect thinking is often futile.

Read more: Mauboussin on Strategy: Interdisciplinary Perspectives on Risk

Previously:

• Michael Mauboussin: How Do You Compare?
• More Than You Know
• Mauboussin on Discounted Cash Flow Models
• Mauboussin on Strategy: Size Matters
• All posts tagged Michael Mauboussin

Taleb’s most recent notebook entry on the logic of prediction errors has some interesting observations about our perception of “conditional expectations” and randomness:

One main life expectancy is from Mediocristan, i.e. is subjected to mild randomness. In a developed country a newborn female is expected to die at around 79, according the insurance tables. When she reaches her 79th birthday, her life expectancy, assuming that she is in typical health, is another 10 years. At the age of 90, she will have another 4.7 years to go. At the age of 100, 2 ½ years. At the age of 119 , if she lives miraculously that long, she will have about nine months left. As she lives beyond the expected date of death, the number of additional years to go decreases. This is the major property of random variables related to the bell-curve. The odds of a large number is small, so the conditional expectation of additional days drops.

With scalable variables, the ones from Extremistan that we encounter in real life, you will witness the exact opposite effect. Say a project is expected to terminate in 79 days, the same expectation in days as the newborn female has in years. But the errors are scalable, i.e. power-law distributed. On the 79th days, if the project is not completed, it will be expected to take another 25 days to completion. But on the 90th day, if the project is not completed, it will have about 58 days to go. On the 100th, it will have 89 days to go. On the 119th , it will have an extra 149 days. On day 600, if the project is not finished, you will be expected to need to wait an extra 1590 days. As you see the longer you go, the longer you are expected to wait.

This subtle, but extremely consequential property of scalable randomness is unusually counterintuitive. I believe that this is the core reason for our missing in our forecasts as we do not take into account the logic of the large deviations from the norm. The distribution is Mandelbrotian.

This idea can illustrate many phenomena; it applies to the completion date of your next opera house, the time a refugee is expected to wait until he can finally return home, or the day when the next war will end.

Researchers sometimes argue that statisticians have little to contribute when few realizations of the process being estimated are observed. We show that this argument is incorrect even in the extreme situation of estimating the probabilities of events so rare that they have never occurred. We show how statistical forecasting models allow us to use empirical data to improve inferences about the probabilities of these events. Our application is estimating the probability that your vote will be decisive in a U.S. Presidential election, a problem that has been studied by political scientists for more than two decades…

Read more: Estimating the Probability of Events That Have Never Occurred: When Is Your Vote Decisive?

Nassim Nicholas Taleb
Dean’s Professor, Sciences of Uncertainty, UMASS at Amherst

Mild vs. Wild Randomness
The talk is on the nontrivial difference between Mild (Gaussian) and Wild randomness (non-Gaussian), its consequences for knowledge, prediction, and social fairness, and how it renders much of the statistical tools ineffectual. Related papers can be found on Taleb’s website: http://www.fooledbyrandomness.com/. Of particular relevance are /epistemologyfattails.pdf, /knolwedge.pdf, and /amherstclass/blackswanexcerpts.pdf (the last of which requires a username and password which will be included in the email announcement).

Anyone want to share access to that last file, which I assume from the filename is excerpts from Taleb’s next book, The Black Swan? Comments are open!

Previously: Columbia Collective Dynamics Group

SM: Explain what high-frequency information is and why it’s bad for you.

NT: Here’s a simple example. Psychologists ran experiments to see how people absorb information. In one experiment they found people who work with racehorses, and asked them to name up to 50 pieces of information they would need [to determine if it was going to be a winner]. They ranked them by order of importance. They took the 10 most important ones out of 50 and looked at the prediction of accuracy to determine if a horse will win a race. Then they took the 20 most important pieces, then the 30. In the end, you had no gain in predictive power beyond the first 10 pieces of information, but a huge gain of overconfidence…

I showed in the book that if something has a 93% probability to happen over a year, in one second the chances of it happening are 50-50. If you sample it at a smaller frequency, there’s more information and less noise. You would think your brain can handle cleaning up the noise, but it can’t. We have cognitive impediments that make us very vulnerable to information overload. You get like a pot that you fill up with water. Too much information is bad for your ability to see the meaning. This is the second effect; I call it a scaling problem.

People ask, Isn’t information good for you? Information is good for you, but try to filter it. Your brain can’t handle filtering itself. For example, you will feel good if your portfolio performs well two hours in a row. It’s insignificant; it does not reflect your abilities or the abilities of your manager. You feel good without knowing why. Likewise, having the opposite will put you in a bad mood, without the faintest idea why. Basically, noise can control you because you don’t use your cognitive skills when looking at data at a high frequency.

Read the full interview: Smart Money — Chance Encounters with Nassim Taleb