Ridley: The forecasters paradox | Edit ShortsCars

There is a paradox at the heart of innovation that I have yet to resolve. Looking backwards, innovation looks inevitable and therefore predictable: once you invent the drone, warfare changes; once you invent the internet, social media arrives; once you invent electricity, the light bulb is unavoidable. Looking forwards, however, nobody sees these things coming.

Search the writings of defence experts from even 10 years ago, and you will find few people realising just how much drones were going to change the nature of war. Comb through the work of internet pioneers in the late 1980s, and you will encounter almost nobody foreseeing the profitability of search engines, let alone social media. Read the musings of DNA experts in the 1970s, and none of them expected forensic applications of DNA fingerprinting to transform criminal justice.

Yet almost every great invention is plagued with disputes between rivals who claim they invented it independently: from the telegraph to CRISPR gene editing. Twenty-one different people have good claims to have invented light bulbs largely independently: Frederick de Moleyns, Heinrich Goebel, Joseph Swan, Alexander Lodygin, Henry Woodward, Matthew Evans, Hiram Maxim, Thomas Edison and more. If any of these had taken a different turn, we would still have light bulbs. If the Wright brothers had died in a glider accident, it would have delayed powered flight only a little. If Larry Page had never met Sergey Brin, we would still have search engines, though not the word “Google.”

Such simultaneous invention implies predictability. Yet where are the predictions? A colleague wrote a lengthy report on the future of telecoms in 1990 for The Economist and left out mobile phones. I was part of the House of Lords Artificial Intelligence committee that wrote what some said was a rather good report on AI in 2018, and yet you will not find the phrase “large language model” in our report at all—just four years before ChatGPT changed the world.

This unpredictability causes some very clever people to say some very stupid things when lured into speaking about the future. Talk of nuclear energy was “moonshine,” said Ernest Rutherford, the man who split the atom, in 1933. “There is no reason for any individual to have a computer in his home,” said Ken Olsen, the founder and chairman of the most successful computer company of the time, Digital Equipment Corporation, in 1977. “By 2005 or so it will become clear that the Internet’s impact on the economy has been no greater than the fax machine’s,” wrote the Nobel Prize-winning economist Paul Krugman in 1998. “There’s no chance that the iPhone is going to get any significant market share. No chance,” said Steve Ballmer of Microsoft in 2007.

This is why I insist there is no such thing as an expert on the future and “modellers” who pretend otherwise are often little better than those who practised “haruspicy”—interpreting the entrails of chickens. Complex dynamic systems like the economy, the weather or an epidemic just are not susceptible to anything resembling foresight, however sophisticated the mathematics that go into the models. In his 20-year study of expert predictions in economics, Philip Tetlock showed that the average expert was no better at foretelling what will happen in the future than “a dart-throwing chimpanzee.”

“In one of history’s great ironies,” wrote Tetlock in his book Superforecasting, “scientists today know vastly more than their colleagues a century ago, and possess vastly more data-crunching power, but they are much less confident in the prospects for perfect predictability.”

Sure, there are some who Nostradamus-like make gnomic remarks that strike lucky from time to time. And perhaps we should make an exception for Arthur C. Clarke, who saw communication satellites coming. But Clarke hit the nail on the head when he said, in a television film, “only if what I tell you appears absolutely unbelievable have we any chance of visualising the future as it really will happen.”

Anyway, as I say, I still find it hard to reconcile the unpredictability of the future with the inevitability of innovations that follow certain inventions. One way to resolve this paradox, following Friedrich Hayek, is to remember that knowledge is dispersed throughout society—held as much between and among our heads as inside them. The wisdom of crowds decides when and how to apply a new technology and is far wiser than me, you or Ernest Rutherford.

Another solution is to invoke what I call Amara’s hype cycle. Roy Amara was a computer scientist in Silicon Valley in the 1960s. He was almost certainly not the first person who said this, but he was one of the clearest: we underestimate the impact of a new technology in the long run, but we overestimate it in the short run. That is why we are often disappointed by the early impacts of an invention (Krugman and the internet, for instance) but taken by surprise later. To put it in more technical terms, the adoption and development of an innovation is non-linear. It starts slowly, then takes off suddenly. That is inherently hard to forecast.

More generally, I think chaos theory is helpful here. An innovation that eventually disrupts the world is like the apocryphal butterfly wing flap in the Amazon that starts a hurricane in the Atlantic. This “butterfly effect,” coined by the meteorologist Edward Lorenz in 1961, suggests that small events can trigger huge and unpredictable consequences. Good luck catching a real butterfly in the act of starting a hurricane.

I read every essay on the future impact of artificial intelligence with this unpredictability in mind. Nobody, however clever, can yet foresee what the consequences of generative AI will be. My generally optimistic take could be as wrong as other people’s doom-mongering. It could be huge or a nothingburger. We just do not know, and it’s no good hoping we will know. We will find out as it happens.