The prophetic fallacy

This book isn’t just about prediction, or even the limits of knowledge. It is about the ascent of man. According to Nate Silver, the American electoral analyst, the digital age and its explosion of knowledge constitute a great turning point in human history. Never before have we had so much evidence on which to base our predictions of the future. Yes, there have been setbacks along the way, but we should feel optimistic about the direction of travel.

Yet there is a paradox about the evolution of prediction. Innovations in technology and information make it possible to analyse data far more quickly and extensively. The availability of so much raw data, however, means that much of the analysis is woeful. There are many more bullets; but also many more inept marksmen. T. S. Eliot’s question has never been more apt: ‘Where is the wisdom we have lost in knowledge? Where is the knowledge we have lost in information?’

The Signal and the Noise acknowledges this, but Silver remains an optimist, hoping to build on the tradition of Enlightenment self-confidence. A similar case for optimism was memorably expressed 15 years ago by Peter L. Bernstein, whose Against the Gods argued that modern finance constituted the front line of the great march of knowledge over uncertainty. The revolutionary idea that defines the boundary between modern time and the past is the mastery of risk, argued Bernstein. ‘Until human beings discovered a way across that boundary, the future was the murky domain of oracles and soothsayers who held a monopoly over knowledge of future events.’ Unlike Bernstein, Silver has to temper his optimism to account for a massive financial crisis — which the combined predictions of regulators, ratings agencies and experts failed to spot.