Te Rā Whakamana: Evidence-based according to whom?

09/12/2025

I use Christian Gade’s three conditions to ask what we are claiming when we call a policy evidence-based, and whose preferences and knowledge sit behind that claim.

An interlude on Christian Gade’s quiet clarification of what we claim when we invoke evidence. This piece sits outside the implementation series I have been building, but Gade’s work is too useful to set aside. His recent contribution to Evidence & Policy asks a question so fundamental that I am surprised it needed asking: when is an individual or organisation justified in claiming that a policy is evidence-based?

“Evidence-based” has become a phrase that does more work as a rhetorical move than as a description of epistemic practice. We use it to close down debate, to signal seriousness, and to distinguish ourselves from those who supposedly proceed on ideology or intuition alone. Gade (2024) slows us down by asking what, precisely, we are claiming when we reach for that authority.

The three conditions

Gade’s argument is deceptively simple: a claim that a policy is evidence-based is justified if, and only if, three conditions are met.

The first is that the claimant must have comparative evidence, that is, evidence about the effects of the proposed policy relative to at least one alternative. This rules out the common move of pointing to evidence that a policy produces some effect and declaring the matter settled, because evidence-based claims are by their nature claims about relative performance. A claim that something works better than an alternative on a stated dimension might qualify; a claim that it simply works does not.

The second is that the policy must be supported by this comparative evidence according to at least one of the claimant’s preferences. This is where Gade’s argument becomes quietly radical, because whether evidence supports a policy is not a purely empirical question; it depends on what you are trying to achieve, which dimensions of effect you care about, and what you are willing to trade off against what.

The third is that the claimant must be able to give a sound account of this support, explaining both the evidence and the preferences that together ground the claim, and an appeal to “the research” without further elaboration falls short of that threshold.

The preference problem

The second condition deserves the most careful attention, and Gade illustrates it with an example from justice policy. Imagine evidence that Policy A reduces reoffending by 60 per cent compared with Policy B, while Policy B delivers 20 per cent higher victim satisfaction at 30 per cent lower cost. If you hold preferences across all three dimensions, you are justified in calling both policies evidence-based: Policy A with respect to reoffending, and Policy B with respect to victim satisfaction and cost.

The implication is striking, because being justified in calling a policy evidence-based does not mean it ought to be implemented. You might reasonably hold that Policy B is evidence-based while believing that Policy A should be adopted, because you weigh reductions in reoffending more heavily than cost or satisfaction. On this account evidence-based is a relational claim, always tied to preferences that must themselves be defended on other grounds, rather than a trump card.

What this means here

I want to dwell on something Gade addresses directly, which is whose evidence counts. He notes that some might raise the threshold by requiring that only scientific evidence, or only certain kinds of scientific evidence, can ground an evidence-based claim, and he declines that move because it would exclude whole epistemic communities, including those holding Indigenous knowledge and those who are innovating and learning, from evidence-based practice altogether.

This dynamic is not new. I have written elsewhere about Wai 4, the Kaituna River claim, in which the Ministry of Works insisted for years that piping Rotorua’s treated sewage into an ancestral river was the technically sound, evidence-based solution. Its engineers had flow rates, nutrient loads, and cost-benefit analyses, but no framework for weighing spiritual contamination, the severance of whakapapa relationships, or the loss of traditional gathering sites. When Ngāti Pikiao witnesses brought mātauranga Māori to the Tribunal, including Emily Schuster’s knowledge of the water conditions particular plants require and Mata Morehu’s understanding of the river’s spiritual ecology, they were providing evidence read through different preferences about which dimensions of harm mattered, rather than giving alternative opinions. The Ministry’s claim to technical neutrality did not survive once its preference set was made visible: it favoured certain forms of knowledge and certain outcome dimensions while excluding others. When the Tribunal required the whole picture to be considered, the culturally appropriate solution proved cheaper and more effective on every measure, and the evidence-based pipeline was revealed as expensive, outdated, and destructive, because the Ministry had acted as an epistocracy that rendered local evidence invisible rather than because it lacked evidence.

The Regulatory Standards Bill is a more recent example. It proceeded through the House as though the economics of cost-benefit analysis were the self-evident centre of regulatory practice, which for many people they are not. The wider literature on regulation reminds us that regulatory work is first and foremost about behaviour and change management: shaping actions and compliance, building social licence, managing risk, and negotiating the boundaries of acceptable practice. Economics is a relatively recent and contested addition to the regulatory toolkit rather than its foundation, yet the debate was conducted as if only economic reasoning mattered, and as if the preference for a market lens over other dimensions of regulatory quality were simply the way serious people think about regulation. In my view that does not hold, and Gade’s framework shows the move for what it is: a failure to make visible the normative commitments that determine which evidence counts and how.

The debate over social investment is a related case. Its proponents claimed, with some justification, that directing resources to those at highest risk of poor outcomes was evidence-based, since predictive analytics could identify who was likely to cost the system most and early intervention could reduce those costs. Critics countered that the framing embodied contestable preferences that had not been made explicit: for fiscal efficiency, for individual risk factors over structural determinants, and for quantifiable outcomes over relational goods. Each side accused the other of ignoring the evidence, and Gade’s framework suggests that both were making claims that could in principle be justified, but only if they were open about the normative work their preferences were doing.

The obligation to give an account

For those of us interested in policy advisory systems, Gade’s third condition carries particular weight, because it requires us to explain both evidence and preferences, and to show how they connect to ground the claim, rather than simply to hold them.

I read this as an argument for reasoning made visible. It connects to questions I have been exploring elsewhere about what advisers owe decision-makers: the premises that support their recommendations, including the normative premises that shape how evidence has been interpreted, as well as the recommendations themselves.

In a system where ministers are accountable for decisions but depend on others for analysis and advice, the capacity to give such an account is a condition of legitimate advice rather than an epistemic nicety.

When an official tells a minister that a policy is evidence-based, Gade’s framework suggests they should be ready to answer three questions: compared with what alternatives, according to which preferences, and can they show their working? If they cannot, the claim is not justified, not because the policy is wrong but because its epistemic warrant has not been established. If that test applies to officials, I think it ought to apply equally to think tanks, consultancies, lobbyists, and academics.

A modest clarification

Gade’s article is short, ten pages including references. It makes no grand theoretical claims, relies on the Oxford English Dictionary for its key definitions, and proceeds with almost excessive care, yet it imposes a valuable discipline by requiring precision about what we are asserting when we reach for evidence as a source of authority.

The next time you hear a policy described as evidence-based, or hear a commentator ask why the evidence was not followed, it may be worth pausing to ask the three questions. The answers may prove illuminating, and they may also prove uncomfortable, which in my experience tends to be a sign that something worth examining has surfaced.

References

Gade, C. (2024). When is it justified to claim that a practice or policy is evidence-based? Reflections on evidence and preferences. Evidence & Policy, 20(2), 244–253. https://doi.org/10.1332/174426421X16905606522863