“The Outcome Detectives” - Or… A Blog about Business Cases
A blog Post about why Evidence-Led does not mean Model-Led
A New Netflix Drama Pitch: The Outcome Detectives
Holly is a hard-nailed transport policy officer. She’s lived through some tough times and has the scars to prove it. With her trusty sidekick Glen, she’s out to discover which interventions really result in which outcomes.
It’s tough work. The city is a complex place, but the clues are all around her. By interviewing people in the community, collecting the evidence, analysing the data and using the power of her phenomenal mind, she can figure out whether or not a bus lane should be made 24/7.
Join us in a 20-part series as we track Holly (veteran of the Ofo wars) as she uses her unrivalled detective skills to piece together the benefits for her city of a low traffic neighbourhood, better pavements and a new station.
But it ain’t gonna be easy. Though Holly’s been through it all and you’d think she’s earned her spurs, she’s got a rival in the force: dodgy copper Carl. He doesn’t get out there and walk the beat. He stays in the station and relies on his network of modellers to tell him the answers; most of whom don’t even work for the force.
Over the years, Carl’s convicted dozens of innocent interventions of crimes they did not commit. That modal filter: convicted by model of increasing traffic on the main road and condemned. That tramway extension: convicted by model of offering a low BCR and banged up for life.
Can Holly prove that pounding the streets gets better outcomes and prevent Carl from convicting any more innocent transport schemes for crimes they did not commit?
OK, look, I think you can sense where I’m going with this.
Because it’s a Netflix drama pitch, I’ve had to ham up the tension a bit. If Adolescence had been pitched as a playground tiff, it wouldn’t have been made. So I’m not saying Carl’s wrong for using the model. Models are useful (though don’t forget the first part of that sentence!). But I am saying he’s wrong for only using the model.
What A Detective Does
There’s a reason detective shows are so popular: being a detective is a seriously difficult job, and the stakes are high. Get it wrong in one direction and you risk a court convicting an innocent person. Get it wrong the other way, and you risk a criminal staying free to reoffend.
As a result, the absolute focus has to be on finding the truth.
In that context, evidence can be whatever the evidence is.
Obviously, some things routinely make good evidence.
That’s why someone will always dust for fingerprints.
(Indeed, they really will always dust for fingerprints. When Sn-ap’s office was burgled and all our computers stolen, a chap came round to dust for fingerprints. He cheerfully told me that no-one would ever look at the results as no detective had been allocated to the case. He was right: no-one ever did).
It’s why so much of the process is formulaic - to reach the answer to this kind of question, predictable steps and processes make it easier.
But it’s also why the law lists types of evidence that are inadmissible, and not the other way round. You can’t use an improperly-obtained confession or hearsay or the records of a private conversation between the defendant and their lawyer.
But anything else can be evidence. Evidence is not a narrow term.
Guiltbot
There is currently a backlog of 80,000 court cases. The system is overwhelmed. I personally have been called to give evidence in a Crown Court trial in June for a violent crime I witnessed in 2022. That’s how bad things are right now.
Wouldn’t it be so much easier if you could just put a series of indicators into a programme and it told you the answer?
So let’s imagine Guiltbot. How might it work?
We’d put in the DNA match (Y/N), the fingerprints match (Y/N), whether there’s a confession (Y/N), the distance the defendant was from the scene at the moment the crime took place (km), plus an upload of the phone cell-site data and any tracking from Oyster, smartcards or ANPR.
Based on that lot, Guiltbot will tell us whether they did it.
Do you know something? Guiltbot would sometimes get it right. Maybe often.
Because Guiltbot would have been built by the world’s leading experts, based on rigorous analysis and detailed research, the experts who created it will be defensive when it’s criticised because their research is good.
The problem is, campaigners arguing to go back to proper evidence collection aren’t criticising Guiltbot, they’re criticising the use of Guiltbot. It’s about the inevitable limitations of a model.
Tools like the Transport Appraisal Guidance (TAG), upon which many transport models are built, are like Guiltbot: outstanding bits of thoroughly researched modelling to help make a decision.
But they’re only one lens.
Moreover, TAG has its origins in the road-building boom of the 1970s to 1990s. So its data is about the priority then, which was reducing journey time through increasing capacity.
Similarly, let’s imagine that Guiltbot dates from the same era.
In 1993, vehicle-related crime peaked at 430,000 recorded crimes per year. Since then, it’s collapsed.
If we were using Guiltbot in courts today, we’d be using a programme that has a series of assumptions and norms baked into it relating to how likely it is that someone stole a car: but today’s crimes are more likely to be cyber or street theft or shoplifting. Valid tool for the wrong task.
Detective work
Detective work is important because we might take away someone’s liberty.
Transport planning is less important.
But both are exceptionally difficult and involve evidence.
A Business Case is an incredibly difficult thing to create.
Given it’s a case for change, it not only involves describing the situation as exists today (which is often harder than it sounds; lots of things are true today - which you choose to talk about makes a big difference to whether or not the case is credible) but, even harder, it involves describing the future you’re hoping to create.
Now that’s really difficult, for the simple reason that the future hasn’t happened yet.
Not only that, but we’re describing the future in the real world, which is the most complex things any of us interacts with.
A future describing what will happen in the real world when an intervention that doesn’t currently exist comes into existence is, in many ways, more difficult than figuring out whether a criminal did it.
At least the detective only has to work with the present and the past: a transport planner is working with the present and the future.
It’s no wonder that, often, busy people don’t really think of it like that.
They think of a Business Case not as a highly challenging and complex evidence-gathering exercise but as a process to go through. Part of the process is to run the model, and input the answer.
But that’s not gathering evidence - it’s running a model. Running a model might generate evidence but it isn’t all the evidence in itself.
What is a Business Case?
Here’s how HM Treasury describe a business case:
WHAT IS A BUSINESS CASE? The business case is a management tool and is developed over time as a living document as the proposal develops. The Business Case keeps together and summarises the results of all the necessary research and analysis needed to support decision making in a transparent way. In its final form it becomes the key document of record for the proposal, also summarising objectives, the key features of implementation management and arrangements for post implementation evaluation. Business cases can cover a wide range of types and levels of spending. Each case will be developed to reflect the type of proposal being considered. The effort departments expend on developing the proposal should be proportionate to the likely costs and benefits.
The key point is that it collates “all of the necessary research and analysis”.
The amount of research and analysis that is necessary can vary enormously.
At one extreme, you have an intervention that is almost free to implement and almost free to reverse.
In that case there is no research and analysis necessary: the easiest way to find out what will happen is to do it. If you don’t like the result, you can undo it. No research will be more accurate than discovery.
At the other extreme, you have highly complex interventions that require big investments are almost impossible to reverse. In this case, the evidence needs to be laser focused on whether the intervention achieves the outcomes the sponsoring authority are looking to achieve. If a model based around vehicle journey times isn’t the best tool to get that evidence, then it’s not the best tool for this business case.
The Five Cases
The HM Treasury Five Cases model makes more provision for this than is often realised. For those not familiar, the Five Cases are the standard five sections of a Treasury business case, used across the public sector. Here they are:
They are a decision framework, not a modelling framework. Within that framework, the Economic Case is the one that often gets lost in models.
Because it talks about the ‘best option’, it’s the point at which an appraisal is required; to judge different options against each other.
At that point, people reach for the models.
But the Economic Case is (2), and there’s a reason the Strategic Case is (1).
The Strategic Case is where Detective Holly needs to assemble the evidence that this intervention is the right thing to do in the first place.
What problem are we trying to solve?
What outcomes are we seeking?
How do we know those outcomes matter?
How do we know this intervention will deliver those outcomes?
There is no rule about what that evidence must look like; it is not defined as a model print-out.
It is a clear articulation of outcomes, supported by the best available evidence for whether this intervention can achieve them.
That evidence might include examples from similar places… or interviews with residents… or observed behavioural change… or multiple different types of modelling… or comparative case studies… or (yes!) professional judgement.
The Five Cases model never says “run the model and accept the output”.
What it does say is: assemble all the necessary research and analysis.
Norms Aren’t Rules
In my spoof film trailer, I invented Carl, the copper who convicts transport schemes for crimes they haven’t committed because he stays in the station and doesn’t collect the evidence.
Carl is, truthfully, many of us.
The sector has got used to staying in the office and running the model.
That’s a norm of behaviour; it’s not a rule.
Sometimes, both sides have adopted the same norm.
The person in the local or strategic authority seeking funding expects simply to send a model result, and the person at DfT expects to receive it.
That still doesn’t make it right.
If we’re going to get better transport outcomes, we need to be more Holly.
Remember, Business Cases are investigations, not processes.
Take Action!
Business Cases are investigations, not processes. But if you want a process for your investigation, here it is:
Define outcomes first
Seek multiple forms of evidence
Treat modelling as one input, not the verdict
Accept uncertainty
