My Response to DfT’s transport Appraisal, Modelling, and Evaluation Strategy (AMES) Consultation

In case you were wondering, this is what I said…

What do you think are strengths in current practice in transport appraisal, modelling and evaluation?

There are two key strengths in current practice:

Structured discipline and transparency

The UK’s appraisal and modelling frameworks provide a consistent, auditable structure for decision-making. This is vital when spending public money, particularly for large, long-term investments. The discipline of setting out costs, impacts and assumptions can bring rigour to the process, especially when decision-makers engage critically with the evidence.

Analytical capability and data availability

The UK benefits from a strong analytical community, with access to rich datasets (e.g. census, ticketing, mobile data) and mature tools for modelling demand. When used well, this enables a nuanced understanding of transport behaviour and the likely impacts of interventions. In the right hands, these tools support insight, not just numbers.

However, these strengths are often undermined by how the tools are used. Frequently, business cases are optimised to achieve a positive Cost Benefit Analysis based on a narrow definition of success (e.g. reduced travel times), even when the true objective of the scheme is something different.

What do you think are weaknesses or gaps in current practice in transport appraisal, modelling and evaluation?

The biggest weakness is that the process often overreaches its purpose. Appraisal is meant to inform judgement but, in practice, it frequently substitutes for it.

Three issues stand out:

Too much focus on forecasts not outcomes

There is too much weight placed on modelling precise long-term forecasts, and not enough attention paid to defining the strategic outcomes we actually want. Predicting the future is impossible. Any appraisal that depends on an accurate prediction of the future risks arbitrary conclusions. This can be seen from the way Cost Benefit Analyses can oscillate between positive and negative depending on changes in assumptions that could both be correct. Instead of trying to predict the future numerically, the central focus of appraisal should be about ensuring clarity on the purpose a scheme is designed to serve and whether the scheme actually serves that purpose.

Overreliance on benefit-cost ratios

BCRs are often treated as definitive answers, despite being sensitive to small changes and deeply reliant on assumptions. This creates a false sense of objectivity. It also crowds out innovation: the more novel an idea, the harder it is to model confidently, so the more likely it is to be rejected. This leads to risk-averse decision-making and incrementalism.

BCRs were created to enable a robust comparison between apples and pears. The trouble is apples are not pears, and no amount of modelling will make them directly comparable. Instead of trying to create a single numerical comparison (which is impossible), the question should be: "Will an apple or a pear best help me achieve my goals?". This is about judgement informed by evidence. It does not need a single number, however satisfying that feels.

A prescriptive view of ‘good’ outcomes

The appraisal framework bakes in certain assumptions about what is desirable: for example, that reduced journey time is always a benefit. But for many schemes, longer journeys may support better outcomes: safer streets, more active travel, stronger local town centres, better integration between modes. The role of appraisal should be to test whether a scheme achieves its stated aims, not to dictate what those aims should be.

Overall, the current system values what is measurable, rather than testing whether a scheme achieves the outcome the promoter seeks to achieve based on the evidence that the promoter can assemble.

What do you think are the top priorities for developing transport appraisal, modelling and evaluation?

Please mention up to three priorities.

1.

Shift from numerical precision to evidential sufficiency

Appraisal should move away from seeking a single number (e.g. a BCR) and towards building a reasoned, evidenced case that a proposal will support a strategic outcome: more like a court of law than a calculator. The test is not: “What number did the model produce?” but: “Do we have enough evidence to believe this will work?” This evidence does not need to be numerical.

2.

Align appraisal to strategic goals, not just generalised assumptions

The framework should start from the stated purpose of the intervention, not from a default set of "good" outcomes. For example, slower journey times may support place-making or transport integration. The appraisal process should test how well a scheme supports its intended goals, not penalise it for defying outdated assumptions.

3.

Support innovation by accepting uncertainty, not penalising it

The current system favours familiar, easy-to-model schemes. We need tools and methods that allow for prudent experimentation including scenario testing, qualitative assessment and clear articulation of “leap of faith” assumptions. Novel ideas shouldn’t be discarded just because they can’t be modelled.