On the evening of September 29 1913, Rudolf Diesel died in mysterious circumstances, apparently jumping from a ferry crossing the North Sea. 

On his way to open a factory in Britain, the German engineer left behind him a hat and neatly folded overcoat on the deck of the SS Dresden, large debts and a remarkable industrial legacy that was to transform the global economy. Almost every cargo ship, tractor and bulldozer in the world runs on diesel today.

Shortly before his death, Diesel wrote a letter bemoaning his fate as someone who had defied convention and tried to do things differently. Introducing an invention, he explained, “is a time fraught with combating stupidity and jealousy, inertia and venom, furtive resistance and an open conflict of interests, an appalling time spent battling with people, a martyrdom to be overcome, even if the invention is a success”.

Diesel was, according to Matt Ridley, an “innovation hero” who not only helped to produce more efficient engines but also envisaged radically new, if less successful, ways of organising worker-run factories. But his frustrating experience was an extreme form of a challenge confronting all those who seek to innovate. 

While innovation is almost universally hailed as a desirable objective, why is it so hard to achieve in practice? Why is it, in Diesel’s words, so often “a martyrdom” rather than an inspiration to be encouraged?

The first Diesel engine in the test facility of the Maschine Factory Augsburg, July 1893 © Ullstein bild/Getty Images

In truth, Ridley never fully answers the questions he raises, partly, no doubt, because it is such a complex subject. Nevertheless, his book How Innovation Works is an entertaining attempt to explore what innovation is, how it works and why it is resisted. 

Firmly in the neo-optimist school of commentators, alongside the likes of Steven Pinker and the late Hans Rosling, Ridley argues that innovation is the unacknowledged “infinite improbability drive” (of the science-fiction writer Douglas Adams’ imagining) that powers the modern economy.

“Pause in awe at what innovation does,” Ridley enthuses. In 1900 an American man would — on average — start work at 14, spend 25 per cent of his life in back-breaking toil and die aged 47. Today, his modern equivalent spends half his time in education or retirement, works 10 per cent of his hours on earth and dies aged 80. That change is almost entirely due to the wondrous power of innovation.

Packed with insightful examples, Ridley’s book explores the stories behind innovations as varied as fertiliser, vaccinations, corrugated iron, shipping containers and sliced bread, and the development of fracking, nuclear power, search engines and gene editing. Although these stories are often told through the lives of individuals, Ridley insists that we should think of innovation in a less personalised way, stressing that it is almost always a collective enterprise, a team sport.

Google co-founders Sergey Brin and Larry Page © Getty Images

Even if Google’s co-founders Larry Page and Sergey Brin had never been born, we would still have powerful search engines, Ridley contends. Innovation, he writes, is typically incremental, gradual, serendipitous and inexorable. It is not so much about one eureka moment as thousands of trial-and-error experiments. It is far more about natural selection than intelligent design.

Ridley usefully emphasises the distinction between invention and innovation. Johannes Gutenberg invented the printing press but Martin Luther was the innovator in how to use it. It was Luther who helped turn an obscure pastime for the ecclesiastical elite into a mass-market publishing phenomenon. “Like Jeff Bezos at Amazon or Mark Zuckerberg at Facebook, he had realised the potential of a new technology on a huge scale,” writes Ridley.

The book is also engaging on how new innovations often spark resistance. Potatoes were for a while rejected by English Protestants as agents of Roman Catholic influence. More recently, the EU banned genetically modified foods, caving in, as Ridley sees it, to public fears of “Frankenfoods”.

Where he is less sure-footed, and at times contradictory, is on how to tackle our current “innovation famine”. His account is too simplistic, drawing cartoon heroes and villains. Short-sighted managers, fee-hungry patent lawyers and fearful bureaucrats are fair game for criticism. But Ridley glosses over arguments for the precautionary principle of regulation, which suggests innovations should be subject to scrutiny before mass adoption, even though recent experiences with social media and artificial intelligence suggest that it might be a good idea. 

Ridley preaches that the main ingredient in the secret sauce of innovation is freedom. Freedom to express counter-intuitive opinions and experiment are vital but so is freedom from government interference and regulation. “As always, regulation favours incumbents,” he asserts, failing to acknowledge that much anti-trust regulation is explicitly designed to cut incumbents down to size and promote insurgents. 

Similarly, while trumpeting the virtues of freedom, he lavishes praise on China’s innovation machine, although he does footnote that the country’s political authoritarianism may one day catch up with it.

As a confirmed Brexiter, Ridley is naturally hostile to the EU, ignoring the scale advantages of the single market. Modern industrial strategy, of the kind advocated by Mariana Mazzucato in her book The Entrepreneurial State, is dismissed as the economic equivalent of creationism.

Ridley’s book is a powerful corrective to the growing chorus of economists who demand that the state should become increasingly interventionist in response to the current crisis. He is right that we should not allow state or corporate managerialism to crowd out the space for individual initiative, experimental enterprise and permissionless innovation to flourish. Schumpeter’s “perennial gale of creative destruction” should not be replaced by the “gentle breezes of rent-seeking”. 

Some of the counterarguments to Ridley’s case are implicitly made by two other books on innovation and the internet, also written by journalists. Like Ridley’s book, they are both readable but frustratingly incomplete, long on anecdote and short on analysis.

In Smoke & Mirrors, Gemma Milne wanders through nine areas of science and technology with an expert eye, puncturing all the balloons of hyped expectations. Many of the hottest areas of tech investment, including vertical farming, fusion energy, quantum computing, brain-computer interfaces and artificial intelligence, are all subject to her healthy and informed scepticism. 

All too often investment in technology reflects the faddish wish list of the rich and powerful rather than the objective real-world needs of society. Hype often determines those fads, conceals complexity and reinforces existing power structures.

Ridley would surely counter that the experts in many of these fields, who are so dismissive of media and investor hype, have themselves a terrible record of predicting the course of innovation. The amateur innovator, able to spot the mass applications of new technologies, can sometimes see further than the original inventor.

Ironically, Milne overstates her case and hypes her own argument. “There are many stories of lying, deception and misuse of power, but I’d argue that hype can be even more damaging than deliberate, sometimes criminal, actions,” she writes. Really? On the evidence presented in this book, most readers would very probably return a Scottish verdict of not proven.

In The System, James Ball takes a similarly critical look at who runs the internet. Its earliest champions believed the internet would be the “thing of revolution”, encouraging a new wave of radical innovation in almost every field. Instead it has been captured by the powers that be and turned into an instrument of control, Ball argues. “The internet, left unchecked, is a monopoly-making machine, an engine designed to concentrate power, attention and more in the hands of those who already have it,” he writes.

His book is a sprightly history of the internet seen from the perspective of its inventors, investors, custodians, rule-makers and rebels. Its initial growth was glacial before becoming too fast to see. Today, about half the world’s population has access to the internet using a total of 25bn-30bn connected devices. As Ball puts it, the internet has evolved from being a closed community of US government employees and academic researchers, to a fringe activity of nerds, to the bright shining hope of capitalism, to critical global infrastructure.

Crowds at the Consumer Electronics Show in Las Vegas in January © New York Times/Redux/eyevine

That astonishing growth would conform to Ridley’s notions of innovation owing more to serendipity and experimentation rather than top-down design and control. “Short-term solutions tend to stay with us for a very long time. And long-term solutions tend to never happen,” according to an engineer involved in developing the internet’s protocols. 

However, as Ball explains, the lack of infrastructural and governance bedrock is worrisome now that the internet has become the cornerstone of the global economy. Some form of concerted intervention is also required to prevent the Big Tech companies from buying up all those digital start-ups that might one day supplant them and killing the innovation cycle. 

Ball recommends that we should pay far more attention to how the internet works and not allow ourselves to be “bamboozled into inaction” as we were with the finance industry before the 2008 crash. We should also treat the Big Tech companies as regular corporations that should be subject to regulation just as previous oligopolists were. 

The overall lesson from these books is that innovation is an extremely powerful force for positive change that should be more explicitly acknowledged and encouraged. Governments should listen more to the insurgents and resist being captured by the incumbents.

The Bodley Head/FT Essay Prize

Now in its eighth year, the FT and The Bodley Head, one of Britain’s leading publishers of non-fiction, team up to find the best young essay-writing talent from around the world. The competition has been the springboard for many writers; entries can be submitted at ft.com/bodley2020. It is open to anyone between 18 and 35 years of age.

But governments, regulators and civil society organisations do have a legitimate and essential role in deciding how some innovations are released into the wild and how they are deployed.

In the mid-2010s, the Silicon Valley investor Peter Thiel argued that we lived in a world in which “bits were unregulated and atoms were regulated”. That explained the phenomenal growth of the computer software industry and the sluggishness of pharmaceutical drug development, he argued.

But real life is more complicated than theory. No one would swallow an unregulated drug. And, as the US Congress has recently made clear, politicians are increasingly moving into the business of regulating bits, whether the tech industry likes it or not.

The challenge will be how best to protect vital societal interests without stifling permission less innovation. It will be one of the most difficult and demanding challenges of our times.

How Innovation Works: And Why It Flourishes in Freedom, by Matt Ridley, Fourth Estate RRP£20, 406 pages

Smoke & Mirrors: How Hype Obscures the Future and How to See Past It, by Gemma Milne, Robinson RRP£14.99, 322 pages

The System: Who Owns the Internet, and How It Owns Us, by James Ball, Bloomsbury RRP£20, 256 pages

John Thornhill is the FT’s innovation editor

Join our online book group on Facebook at FT Books Café

Get alerts on Non-Fiction when a new story is published

Copyright The Financial Times Limited 2021. All rights reserved.
Reuse this content (opens in new window)

Follow the topics in this article