Loose Threads: The Evidence Ecosystem and the Implementation Gap

23/12/2025

A reading from Aotearoa of what the UK’s Academy of Social Sciences thinks is wrong with evidence in government, and what their diagnosis reveals about the limits of technocratic thinking.

Today’s Loose Threads sits alongside the implementation series I have been building. It is about a report from the United Kingdom’s Academy of Social Sciences that landed this week, and it is too instructive to set aside. Not because we should adopt its recommendations, we should not, at least not without considerable care, but because it illustrates with unusual clarity a way of thinking about evidence and public policy that travels easily across the Anglosphere and arrives here dressed as common sense, only to fall over at implementation.

The report, Enhancing the Evidence Ecosystem, offers 12 recommendations to strengthen social science evidence for public policymaking. The recommendations are what you would expect from a learned academy addressing government: clearer institutional leadership, greater transparency about the evidence underpinning decisions, better data infrastructure, enhanced competency frameworks for the policy profession, and more systematic use of advisory committees. Infrastructure, expertise, process, institutional architecture. The solutions are technocratic through and through.

I want to dwell on what this framing assumes, and what it leaves unexamined.

We have a long habit of borrowing from our Anglo cousins. The Westminster system, Northcote-Trevelyan, the linear policy cycle, capability frameworks, Gateway Reviews: the genealogy of some of our public administration thinking runs through London and occasionally detours through Canberra and Ottawa. Some of this inheritance has been valuable, but we borrow too often and too uncritically, and we are not always attentive to what does not travel well.

The AcSS report warrants caution on at least three grounds.

First, the report remains captive to what I have been calling first-generation implementation thinking. The assumption, characteristic of the 1970s literature, is that policy failure is primarily a problem of design and transmission: if we can improve the quality, breadth, and timeliness of evidence flowing to decision-makers, better decisions will follow. Hence, the emphasis on infrastructure, on competency frameworks, and on ensuring Chief Scientific Advisers and Chief Analysts have properly delineated roles. All upstream. All focused on improving inputs.

What this framing obscures is the downstream question. What happens when evidence meets the reality of delivery? The report has almost nothing to say about implementation capability, about the state’s capacity to translate policy intention into operational reality, about the interpretive hinge that connects advice to action through translation, trust-building, negotiation, and sense-making. The entire architecture presupposes that getting better evidence into government is the hard problem. Whether the government retains any capacity to act on that evidence is treated as someone else’s concern.

Fifty years of implementation research has moved well beyond this. The second and third generations recognised that policy is not what is written in law but what is enacted in the daily micro-decisions of frontline workers exercising discretion under impossible conditions; that implementation is not a technical problem to solve but a political and social process to navigate; that the gap between intention and outcome is not a bug to fix but a permanent feature of complex systems. The AcSS report, for all its sophistication about disciplinary breadth, has not absorbed these lessons.

The language gives it away. “Levers.” “Infrastructure.” “Competency frameworks.” “Evidence pipelines.” This is the vocabulary of the machine: improve the inputs, tighten the specifications, and the outputs will follow. Every minister who has watched a carefully designed policy collapse on contact with reality knows this is not how it works. Every frontline worker who has struggled to implement an impossible policy knows why.

Second, the report sidesteps the preference problem that sits at the heart of any claim about evidence.

I explored this recently through Christian Gade’s work. Gade’s argument is deceptively simple: calling a policy “evidence-based” is only justified when you can show comparative evidence (this works better than that), identify the preferences according to which the evidence supports the policy, and give an account of how those connect. Evidence that a policy reduces recidivism does not, by itself, make it evidence-based. You must also show that recidivism reduction is the dimension you care about, relative to cost, or victim satisfaction, or whatever other effects the evidence might reveal.

The implication is that whether evidence “supports” a policy is not a purely empirical question. It depends on what you are trying to achieve, on which dimensions of effect you weigh, and on what you are willing to trade off against what. Those preferences are not delivered by the evidence itself. They are normative commitments; said more precisely, they are political commitments that must be defended on political grounds.

The AcSS report calls for broader disciplinary representation: sociologists alongside economists, qualitative alongside quantitative. This is presented as a solution to evidence gaps. But disciplinary diversity does not resolve the preference problem, let alone the implementation one. It multiplies the contestants. More voices in the evidence ecosystem amplify the cacophony; it does not adjudicate between competing claims or tell us whose preferences should prevail.

The report’s implicit answer is that these are matters for experts to sort out among themselves, with ministers making final calls on the basis of expert advice. This is the epistocratic settlement: democracy sets the broad direction, but the real work of policy is technical, and technical questions are best answered by those with technical knowledge. The episocracy: not the community, or the whānau, let alone the small business struggling with the weight of administrative burden.

It is worth remembering where this settlement comes from. The Northcote-Trevelyan Report of 1854 is the foundational document of the modern Westminster civil service: recruitment by competitive examination, promotion by merit rather than patronage, and a permanent administrative class insulated from democratic turnover. It was a reform born of Victorian confidence that governing required a particular kind of educated expertise, best cultivated at Oxford and Cambridge and selected through classical examination. The epistocratic premise is baked in from the beginning. The question of whose knowledge counts was answered before it was asked: the knowledge that counts is the knowledge possessed by those who pass the examinations set by those already inside.

The AcSS recommendations work entirely within this 170-year-old settlement. They do not question it; they seek to optimise it. Broaden the disciplines, improve the infrastructure, train the officials, but do not disturb the fundamental assumption that policy is ultimately a matter for state-employed experts advising ministers, with democratic participation confined to the ballot box and the occasional select committee inquiry. This is not a neutral or inevitable way of organising the relationship between knowledge and power. It is a very particular constitutional choice, made in Victorian England, that has travelled to every corner of the Anglosphere as though it were simply the way serious countries do things. Frankly, it also ignores the emergence of policy advisory systems, most of which are now made up of non-state expertise, experience, and advisory capacity.

Third, and most importantly for those of us working here, the report says nothing about whose knowledge counts as evidence in the first place.

The recommendations assume that “social science” is a relatively stable category. Broaden the disciplines, improve the infrastructure, train the civil servants to be better consumers of research, and the evidence ecosystem will flourish. But this framing takes for granted a particular epistemic hierarchy: academic research at the top, policy analysis in the middle, and other forms of knowledge, regional, local, experiential, and indigenous, as raw material to be processed or obstacles to be managed.

In Aotearoa, this hierarchy is not merely contestable; it is constitutionally contested. Te Tiriti creates obligations that cannot be discharged by improving the supply of academic social science to ministers. Mātauranga Māori is not a data source awaiting integration into the evidence ecosystem. It is a knowledge tradition with its own epistemology, its own standards of validity, its own relationship to whenua, to wai, to community. The question of how that tradition relates to the policy advisory system cannot be answered by adding more Māori researchers to advisory committees, though that might be part of it. It requires a different conversation altogether: about rangatiratanga, about partnership, about what it means for the Crown to act in good faith.

The AcSS report, naturally enough, has nothing to say about this. It is a British document addressing British problems. But when we reach for solutions from London, we need to notice what travels and what does not. Technocratic approaches to the evidence-policy gap assume a settled relationship between expertise and democracy, and a settled account of whose knowledge counts. Neither assumption holds here. Perhaps neither should hold anywhere.

So what, then, is the alternative?

I would suggest it begins with recognising where the state’s distinctive contribution actually lies. The state is no longer the best producer of evidence; that capability has largely migrated to universities, consultancies, think tanks, and the ecosystem of contracted expertise that works in and around the policy advisory systems. The state is not even the best synthesiser of evidence into advice; others can do that too. The structuring of analysis for decision-making is no longer the preserve of the state. What the state can do, and what only the state can do, is advise on implementation. Translate policy intention into operational reality at scale. It holds the statutory authority, the delivery, performance assurance apparatus, the investment systems, and the accountability relationships that enable action.

Implementation is where the preference problem meets reality. You can debate endlessly whether a policy is “evidence-based” according to this or that set of preferences. But implementation forces choices. It requires deciding which trade-offs to accept, which dimensions of effect to prioritise, and which communities’ knowledge to take seriously. It is the site where contestation must be resolved, however provisionally, into action.

I would offer that our difficulty is that implementation capability has been neglected in the same decades that saw analytical capability migrate outward. The focus has been on strategy, on innovation, on the front end of the policy cycle. The back end, the painstaking, unglamorous work of making things actually happen, has been treated as somebody else’s problem. And so we get reports like this one: sophisticated about how to get better evidence to ministers, silent about whether the system retains any capacity to act on it.

But let me be generous. The Academy of Social Sciences has done useful work in mapping the evidence ecosystem and identifying pressure points within it. For readers in the United Kingdom, the recommendations may be worth debating on their merits. For those of us reading from here, at the top of the world, as part of the Asia Pacific, the report is most valuable as an illustration of a particular style of thinking: one that treats the evidence-policy relationship as fundamentally a technical problem amenable to a technocratic solution.

It is not. The questions that matter most, whose knowledge counts, whose preferences prevail, how evidence relates to democratic deliberation, how the Crown honours its obligations under Te Tiriti, are ultimately political. They cannot be resolved by improving infrastructure or training public servants to be better consumers of research. They require democratic contestation, which is slower, messier, and less satisfying than the clean lines of an evidence ecosystem diagram.

Evidence-informed policy is a worthy aspiration. But evidence without democratic legitimacy is just technocracy. And technocracy, however well-intentioned, is not the same as good government. It may prove illuminating to ask, next time someone invokes the authority of evidence, whose evidence and according to whose preferences. The answers may prove uncomfortable. That is usually a sign that something worth examining has surfaced.

The AcSS report “Enhancing the Evidence Ecosystem: Improving Outcomes for Government and Citizens” is available at www.acss.org.uk