Te Rā Whakamana: Evidence-based according to whom?

09/12/2025

An interlude on Christian Gade’s quiet clarification of what we actually claim when we invoke evidence. This piece sits outside the implementation series I have been building, but Gade’s work is too useful to put aside. Gade’s recent contribution to Evidence & Policy asks a question so fundamental that I am surprised it needed to be asked: when is an individual or organisation justified in claiming that a policy is evidence-based?

The buzzword “evidence-based” has become one of those phrases that does more work as a rhetorical move than as a description of epistemic practice. We invoke it to close down debate, to signal seriousness, to distinguish ourselves from those benighted souls who proceed on mere ideology or intuition. Gade’s intervention is to slow us down and ask what, precisely, we are claiming when we reach for that authority.

The Three Conditions

Gade’s argument is deceptively simple. A claim that a policy is evidence-based is justified if, and only if, three conditions are met.

First, the claimant must possess comparative evidence: evidence about the effects of the proposed policy relative to at least one alternative. This rules out the common move of pointing to evidence that a policy produces some effect and declaring victory. Evidence-based claims are inherently claims about relative performance. “This works” is not an evidence-based claim. “This works better than that, on this dimension” might be.

Second, the policy must be supported by this comparative evidence according to at least one of the claimant’s preferences. This is where Gade’s argument turns quietly radical. Whether evidence “supports” a policy is not a purely empirical question for it depends on what you are trying to achieve, on which dimensions of effect you care about, on what you are willing to trade off against what.

Third, the claimant must be able to provide a sound account of this support, explaining both the evidence and the preferences that together ground the claim. “Just because” or “the research shows” without further elaboration does not meet the threshold.

The Preference Problem

It is the second condition that deserves the most careful attention. Gade illustrates it with a justice policy example. Imagine you possess evidence that Policy A reduces recidivism by 60 per cent compared to Policy B, whilst Policy B achieves 20 per cent higher victim satisfaction and 30 per cent lower cost. If you hold preferences across all three dimensions, you are justified in claiming both policies are evidence-based. Policy A is evidence-based with respect to recidivism; Policy B is evidence-based with respect to victim satisfaction and cost.

The implication is striking: being justified in calling a policy evidence-based does not mean it ought to be implemented. You might legitimately hold that Policy B is evidence-based whilst simultaneously believing Policy A should be adopted, because you weigh recidivism reduction more heavily than cost or satisfaction. Evidence-based is not a trump card. It is a relational claim, always tethered to preferences that must themselves be defended on other grounds.

What This Means Here

I want to dwell on something Gade addresses directly: the question of whose evidence counts. He notes that some might argue for raising the threshold by requiring that only scientific evidence, perhaps only certain types of scientific evidence, should ground evidence-based claims. He declines this move. Such an approach, he observes, would exclude specific epistemic communities, including those adhering to indigenous knowledge, as well as those who are innovating and learning, from the realm of evidence-based practice altogether.

This dynamic is not new. I have written elsewhere about WAI 4, the Kaituna River claim, where the Ministry of Works spent twenty years insisting that piping Rotorua’s treated sewage into an ancestral river was the technically sound, evidence-based solution. Ministry engineers had flow rates, nutrient loads, and cost-benefit analyses. What they lacked was any framework for weighing spiritual contamination, the severance of whakapapa relationships, or the loss of traditional gathering sites. When Ngāti Pikiao witnesses brought mātauranga Māori to the Tribunal — Emily Schuster’s knowledge of which plants required what water conditions, Mata Morehu’s understanding of the river’s spiritual ecology, they were not offering alternative opinions. They were providing evidence, read through different preferences about what dimensions of harm mattered. The Ministry’s claim to technical neutrality collapsed once its preference set was made visible: it preferred certain forms of knowledge and specific outcome dimensions whilst systematically excluding others. When the Tribunal required consideration of the whole picture, the culturally appropriate solution proved cheaper and more effective across every metric. The “evidence-based” pipeline was revealed as expensive, outdated, and destructive, not because the Ministry lacked evidence, but because it acted like an epistocracy that rendered local evidence invisible.

Or consider the Regulatory Standards Bill, which proceeded through the House as though the economics and formulae of cost-benefit analysis were the self-evident centre of regulatory practice. They are not, at least, not for everyone. The vast literature on regulation reminds us that regulatory work is first and foremost about behaviour and change management: about shaping actions, compliance, building social licenses, managing risk, and negotiating the boundaries of acceptable practice. Economics is a relatively recent and contested addition to the regulatory toolkit, not its foundation. Yet the debate was conducted as if only economic epistemics mattered, as if the preference for the market lens over other dimensions of regulatory quality was not a preference at all, but simply the way serious people think about regulation. And that is just not true. Gade’s framework exposes this move for what it is: a failure to make visible the normative commitments shaping which evidence gets counted and how.

Or consider a related debate: the Social Investment discourse. Proponents claimed, with some justification, that targeting resources toward those at the highest risk of poor outcomes was evidence-based: predictive analytics could identify who would likely cost the system the most, and early intervention could reduce those costs. Critics countered that this framing embodied preferences for fiscal efficiency, for individual-level risk factors over structural determinants, and for quantifiable outcomes over relational goods, all of which were contestable and had not been made explicit. Both sides accused the other of ignoring the evidence. Gade’s framework suggests both were making claims that could, in principle, be justified, but only if they were transparent about the normative work their preferences were doing.

The Obligation to Give Account

For those of us interested in policy advisory systems, Gade’s third condition carries particular weight. The requirement is not merely to possess evidence and hold preferences, but to be able to explain both, and to show how they connect to ground the claim being made.

This is an argument for reasoning made visible. It connects to questions I have been exploring in other work about what advisors owe decision-makers: not just recommendations, but the premises that support them, including the normative premises that shape how evidence has been interpreted.

In a system where ministers are accountable for decisions but depend on others for analysis and advice, the capacity to provide such accounts is not merely an epistemic nicety. It is a condition of legitimate advice.

When an official tells a minister that a policy is evidence-based, Gade’s framework suggests they should be prepared to answer three questions: Compared to what alternatives? According to which preferences? And can you show your working? If they cannot, the claim is not justified: not because the policy is wrong, but because the epistemic warrant has not been established. And, if the same is true for an official, then ought that not be the same test for a think-tank, consultancy, lobbyist or an academic?

A Modest Clarification

Gade’s article is short: ten pages, including references. It makes no grand theoretical claims, cites the Oxford English Dictionary for its key definitions, and proceeds with almost excessive care.

Yet it performs a valuable discipline. It forces precision about what we are actually asserting when we reach for “evidence-based” as a source of authority.

So the next time you hear a policy described as evidence-based, or you hear another commentator ask why the evidence was not followed, and find yourself reaching for a definition, maybe pause to ask the three questions. The answers may prove illuminating. They may also prove uncomfortable, which is usually a sign that something worth examining has surfaced.

Reference

Gade, C. (2024). When is it justified to claim that a practice or policy is evidence-based? Reflections on evidence and preferences. Evidence & Policy, 20(2), 244–253. https://doi.org/10.1332/174426421X16905606522863