I have seen in Richard Dawkins’ work many references to the difficulty people have, when looking at an animal, in accepting that it is not the product of a top-down design, but the result of a random process — more exactly the upper bound of a random process, in which (roughly, and only roughly) the most successful mutations tend to make it. Yet my problem is that when those who accept the evolutionary argument look at a computer, at a laser beam, at a successful drug, at a surgical technique, at the spread of a language, at the growth of a city, or at an commercial enterprise, they tend to fall for the belief that its discovery or establishment partook of some grand design. And, in hindsight, some “explanation” will be given as to why it happened: there was a plot — it could not have been an accident.

Alas, we are victims of the narrative fallacy — even in scientific research (but while we learned how to manage it in religion, and to some degree in finance, we do not seem to be aware of its prevalence in research). The pattern-seeking, causality producing machine in us blinds us with illusions of order in spite of our horrifying past forecast errors. I hold that not only discoveries are also largely the result of a random process, but that their randomness is even less tractable than, and not as simple as, biological evolution. While nature might produce milder form of stochasticity, the environment for manmade discoveries is governed by a far, far more severe, wilder form of processes, those called “fat tailed”.

Against what one might expect, this makes me extremely optimistic about the future in several selective research-oriented domains, those in which there is an asymmetry in outcomes favoring the positive over the negative — like evolution. These domains thrive on randomness. The higher the uncertainty in such environments, the rosier the future — since we only select what works and discard the rest. With unplanned discoveries, you pick what’s best; as with a financial option, you do not have any obligation to take what you do not like. Rigorous reasoning applies less to the planning than to the selection of what works. I also call these discoveries positive “Black Swans”: you can’t predict them but you know where they can come from and you know how they will affect you. My optimism in these domains comes from both the continuous increase in the rate of trial and error and the increase in uncertainty and general unpredictability. I have seen in Richard Dawkins’ work many references to the difficulty people have, when looking at an animal, in accepting that it is not the product of a top-down design, but the result of a random process — more exactly the upper bound of a random process, in which (roughly, and only roughly) the most successful mutations tend to make it. Yet my problem is that when those who accept the evolutionary argument look at a computer, at a laser beam, at a successful drug, at a surgical technique, at the spread of a language, at the growth of a city, or at an commercial enterprise, they tend to fall for the belief that its discovery or establishment partook of some grand design. And, in hindsight, some “explanation” will be given as to why it happened: there was a plot — it could not have been an accident.

Alas, we are victims of the narrative fallacy — even in scientific research (but while we learned how to manage it in religion, and to some degree in finance, we do not seem to be aware of its prevalence in research). The pattern-seeking, causality producing machine in us blinds us with illusions of order in spite of our horrifying past forecast errors. I hold that not only discoveries are also largely the result of a random process, but that their randomness is even less tractable than, and not as simple as, biological evolution. While nature might produce milder form of stochasticity, the environment for manmade discoveries is governed by a far, far more severe, wilder form of processes, those called “fat tailed”.

Against what one might expect, this makes me extremely optimistic about the future in several selective research-oriented domains, those in which there is an asymmetry in outcomes favoring the positive over the negative — like evolution. These domains thrive on randomness. The higher the uncertainty in such environments, the rosier the future — since we only select what works and discard the rest. With unplanned discoveries, you pick what’s best; as with a financial option, you do not have any obligation to take what you do not like. Rigorous reasoning applies less to the planning than to the selection of what works. I also call these discoveries positive “Black Swans”: you can’t predict them but you know where they can come from and you know how they will affect you. My optimism in these domains comes from both the continuous increase in the rate of trial and error and the increase in uncertainty and general unpredictability.

Taleb’s optimism about the value we can derive from uncertain environments has parallels with the use of real options to value and analyze the payoffs from different potential courses of action. We know from the Black-Scholes model in financial theory that the value of a financial option (e.g. a call or a put on a stock) increases with the volatility of the underlying equity. Real options are essentially investments which give one the opportunity, but not the necessity, of pursuing further courses of action down the road. Similarly to a financial option, a real option increases in value the more uncertain, or random, an environment we are operating in. It has become cliche at this point to describe the current global business environment as increasingly rapid and complex, but what we can learn from these generalizations is that the value of “keeping your options open” is ever increasing.

Real options sit at a rich crossroads between financial theory and decision theory; expect more discussion of real options here soon!

Read more: Nassim Taleb: The Birth of Stochastic Science

According to a news report in the June 15, 2006 Nature, it has been established mathematically that soccer goals are contagious, statistically speaking: scoring one goal increases the probability that your team will score more. Michael Hopkin, who write the piece, calls this “one of soccer’s classic clichés,” and attributes the result to Martin Weigel (Herriot-Watt University, Edinburgh) and his colleagues Elmar Bittner, Andreas Nussbaumer and Wolfhard Janke, all at Leipzig University. The four have posted a preprint on arXiv.org with the title “Football fever: goal distributions and non-Gaussian statistics.” As they put it: “modifying the Bernoulli random process underlying the Poissonian model to include a simple component of self-affirmation seems to describe the data surprisingly well and allows to understand the observed deviation from Gaussian statistics.” They analyzed “historical football score data from many leagues in Europe as well as from international tournaments, including data from all past tournaments of the ‘FIFA World Cup’ series” and concluded: “The best fits are found for models where each extra goal encourages a team even more than the previous one: a true sign of football fever.”

Read more:

• Math at the World Cup
• Football fever: goal distributions and non-Guassian statistics

Taleb’s most recent notebook entry on the logic of prediction errors has some interesting observations about our perception of “conditional expectations” and randomness:

One main life expectancy is from Mediocristan, i.e. is subjected to mild randomness. In a developed country a newborn female is expected to die at around 79, according the insurance tables. When she reaches her 79th birthday, her life expectancy, assuming that she is in typical health, is another 10 years. At the age of 90, she will have another 4.7 years to go. At the age of 100, 2 ½ years. At the age of 119 , if she lives miraculously that long, she will have about nine months left. As she lives beyond the expected date of death, the number of additional years to go decreases. This is the major property of random variables related to the bell-curve. The odds of a large number is small, so the conditional expectation of additional days drops.

With scalable variables, the ones from Extremistan that we encounter in real life, you will witness the exact opposite effect. Say a project is expected to terminate in 79 days, the same expectation in days as the newborn female has in years. But the errors are scalable, i.e. power-law distributed. On the 79th days, if the project is not completed, it will be expected to take another 25 days to completion. But on the 90th day, if the project is not completed, it will have about 58 days to go. On the 100th, it will have 89 days to go. On the 119th , it will have an extra 149 days. On day 600, if the project is not finished, you will be expected to need to wait an extra 1590 days. As you see the longer you go, the longer you are expected to wait.

This subtle, but extremely consequential property of scalable randomness is unusually counterintuitive. I believe that this is the core reason for our missing in our forecasts as we do not take into account the logic of the large deviations from the norm. The distribution is Mandelbrotian.

This idea can illustrate many phenomena; it applies to the completion date of your next opera house, the time a refugee is expected to wait until he can finally return home, or the day when the next war will end.

Nassim Nicholas Taleb
Dean’s Professor, Sciences of Uncertainty, UMASS at Amherst

Mild vs. Wild Randomness
The talk is on the nontrivial difference between Mild (Gaussian) and Wild randomness (non-Gaussian), its consequences for knowledge, prediction, and social fairness, and how it renders much of the statistical tools ineffectual. Related papers can be found on Taleb’s website: http://www.fooledbyrandomness.com/. Of particular relevance are /epistemologyfattails.pdf, /knolwedge.pdf, and /amherstclass/blackswanexcerpts.pdf (the last of which requires a username and password which will be included in the email announcement).

Anyone want to share access to that last file, which I assume from the filename is excerpts from Taleb’s next book, The Black Swan? Comments are open!

Previously: Columbia Collective Dynamics Group

Is it possible for the human eye to distinguish between realised stock price processes and computer-generated random walks? And is it possible to predict stock prices based on historical price and trading volume time series? These are the two crucial questions we address with this online survey.

Take the survey now!

Via Marginal Revolution