Numbers Can Be Dangerous
We live in a world in which we fetishise numerical evidence at the exclusion of all other evidence.
It's odd, as we only do this in work.
When I ask my wife what the weather's like, I don't disbelieve her assertion that it's sunny until she can tell me the precise quantification of Watts per square metre of solar radiation.
And yet - in work - we tend not to believe something is true unless we can quantify it.
There are two dangers to this:
1) We ignore useful non-numerical evidence
2) We unquestioningly accept numerical evidence
Why Numbers are Dangerous
Let's start with (2); unquestioningly accepting numerical evidence.
This is particularly dangerous because it's often difficult to spot when the numbers are wrong.
If I tell you that it's a good day for a picnic because there's an expectation of 100 watts of solar radiation per square metre today, you might get the prosecco out. You might not immediately spot that this is a tenth of the norm and, actually, indicates a very bad day for a picnic.
Whereas if I tell you it's a good day for a picnic because it's raining, you can spot that I've said something stupid.
Numbers feel authoritative even when they are wrong and that makes them dangerous.
This isn't a theoretical risk.
The Excel error that killed people
Think back to the financial crisis in 2008.
Going into it, central bankers around the world agreed that the right response to the risk of a financial meltdown was to stimulate demand. That kind of Keynesian demand management rescued the global economy from the Great Depression in the late 1930s and there was both practical and theoretical evidence that it worked.
But things changed when Carmen Reinhart and Kenneth Rogoff published a paper showing that economic growth slowed once countries' debt to GDP ratio went over 90%. Given most major economies were close to this threshold, this had a powerful impact on treasuries and finance ministries around the world. It's why we had austerity.
The model on which Reinhart and Rogoff's paper was published turned out to contain a basic Excel error. When corrected, the fall off in growth turned out not to exist at all.
My disabled daughter has suffered badly in the last decade as a result of both her school and child mental health services having inadequate budgets. My wife is chair of governors at my younger daughter's school. She's currently leading a restructure to cut costs because there's still not enough money. The teachers are on strike as a result. My daughter is not in school.
We've decimated our public services because governments all around the world trusted numbers that had been generated via an Excel formula error.
Given that UK life expectancy stopped increasing from 2014 (for the first sustained period since before World War II), we can assume some people'd be alive today were it not for that Excel mistake.
The pointlessly rebuilt roundabout
I was talking with a transport modeller recently who told me of a recent appraisal they were involved with for a roundabout rebuild. It would cost many millions but there was a very strong BCR.
The scheme was about to be nodded through when my colleague decided to take a closer look. What did he find? That 90% of the benefit came from journey time savings on a single movement in one direction between two arms of the roundabout. Most of the work contributed no benefit at all, despite the cost and embedded carbon.
And it's not at all certain that these micro-journey time savings actually materialise or add significant societal benefit. The strong BCR had made a questionable scheme look unquestionable.
The benefits of Autonomous Vehicles
In February, the Government published a report highlighting that Autonomous Vehicles will add £42 billion to the UK economy by 2035 and create 38,000 skilled jobs. It was published in the context of the launch of Robotaxi services in London this year. The £42 billion figure was quoted on the BBC and multiple other media.
This sounded a huge number so I posted a question on LinkedIn asking what on earth the rationale for such a number could be.
AV expert Tym Syrytczyk very kindly took up the challenge and did a whole bunch of digging.
He discovered that the original numbers had been calculated in 2019, long before any of the current robotaxi propositions existed, and was a figure for the value of an AV manufacturing sector in the UK. i.e. nothing to do with the actual announcement being made by the Government.
Tym, as an advocate, states in his Substack outlining the error that the true figure for the value of robotaxis is likely to be far higher than £42 billion. I think it will be lower. But what's unambiguous is that the number quoted was both out of date and misused.
What do all three examples have in common? That credible-sounding numbers convinced highly capable professionals that something untrue was, in fact, true.
As a result, we're giving excessive support to a potentially harmful technology, rebuilding roads that shouldn't be touched and crashing the global economy.
Why Numbers Aren’t Everything
But it's not just that numbers risk doing harm; they also crowd out other sources of evidence.
What I find fascinating is that, when given permission to use other evidence, people welcome it. But there's a professional tyranny that prevents it.
I've published two reports this year.
Both, even if I do say so myself, have been very well-received.
The first was my Mini Switzerland report, published jointly with Hope Valley Climate Action and worked up with a fantastic team of dedicated volunteers - funded by the Foundation for Integrated Transport. This was so successful that it’s now being implemented. Yet the report is largely a number-free description of the concept.
That's not, by the way, because we're not planning on publishing numerical evidence. Both Wiktor Woszczek and Omer Bor, two fantastic Mini Switzerland volunteers, are hard at work on the financial model and the economic case.
It’s just that we decided to press ‘publish’ before the numbers were ready and - hey - it turned out that the descriptive evidence was compelling.
The second report was my Trams and Towns report, published jointly with Create Streets and the Campaign for Better Transport - funded by the RAC Foundation. This was even less numerical as it was explictly written as a description of stakeholders’ experiences.
This isn't to denigrate numerical evidence: it's frequently crucial.
But the evidence in these two reports is real too.
The Tyranny of Numbers
The chokehold of numbers on our decision-making largely results - in my opinion - from Excel. Once it became easier for everyone to generate relatively complex numbers, there was an expectation that everyone would. Eventually any evidence that hadn't been generated by a spreadsheet became invalidated - and any evidence that had been was trusted uncritically.
The solution is a different habit of mind: one that asks, of every number, how it was generated and whether that process was sound - and seeks out evidence, not numbers. I wrote a few weeks ago about how we should approach every business case with the mindset of a detective.
When it comes to numbers, I propose two simple rules of thumb to remember:
1) You don’t need to know the precipitation rate to know that it’s raining
2) A number is just a number. Its meaning derives from how it was generated. If you don’t understand how it was generated, you don’t understand its meaning.
👋 I'm 𝗧𝗵𝗼𝗺𝗮𝘀. I help public and private sector organisations drive 𝗶𝗻𝗻𝗼𝘃𝗮𝘁𝗶𝗼𝗻, deliver 𝗰𝗵𝗮𝗻𝗴𝗲, and achieve 𝗳𝗮𝘀𝘁𝗲𝗿 results through better ways to make decisions.
🚀 I do this through 𝘀𝗽𝗲𝗮𝗸𝗶𝗻𝗴, 𝗺𝗲𝗻𝘁𝗼𝗿𝗶𝗻𝗴, and 𝗰𝗼𝗻𝘀𝘂𝗹𝘁𝗶𝗻𝗴. Let’s talk!
