Loose Threads: Productivity, Robotdebt and the LLM Hype

28/08/2025

Today’s post is part of my Loose Threads series: a place where I pick up strands in the public conversation that have been left dangling, half-woven, or quietly dropped. Today, I offer a view balanced by two data points: the Robodebt Royal Commission in Australia and a recent project I was involved in, where a county council explored the use of AI and large language models in consultation processes. Ultimately, I aim to demonstrate why the promise that AI and LLMs will deliver “productivity” in the public sector runs the risk of creating the same mistake that Robodebt exposed: collapsing complexity into certainty and mistaking efficiency for effective governance. It’s by no means certain, but we should be on guard.

I’ve been re-reading the Robodebt Royal Commission and thinking about change management. At the same time, I’ve just completed a local authority project where the county council was attempting to integrate AI and LLMs into their engagement and participatory systems. On the surface, these seem entirely different: one a cautionary tale from Australia, the other a hopeful experiment from the United Kingdom. Yet the more I sit with them, the more they feel like two sides of the same story: both attempts to manage the unmanageable complexity of public life through technological solutions.

So, for this post, my working hypothesis is this: both Robodebt and the current enthusiasm for AI and LLMs in government spring from the same fundamental misunderstanding about how public administration actually works. To test this out, we will examine four interconnected issues that both technologies share. First, the seductive dream of reducing human complexity to manageable algorithms. Second, the inability of AI and LLMs to grasp the specific contexts in which policy actually applies. Third, the false certainty they both project when making consequential decisions. And finally, how they systematically exclude the democratic voice that legitimises state action.

Robodebt was built on a seductive premise: take human lives, with all their irregularity, and collapse them into equations. The messy reality of part-time work, insecure contracts, and seasonal income got flattened into averages that looked tidy once processed by the machine. Someone juggling whether to pay electricity, rent, or get a new tyre so the car passes its warrant so the car can be used to get to a zero-hour contract: you know the specific, irreducible complexity that comes with living life as most of us live it, has simply disappeared into the algorithm. The harm, as we now know, was massive.

The local authority project I worked on wasn’t as blunt in its approach. Officers wanted to automate reports, summarise consultations, and streamline inputs from public engagement processes. Still, the same fundamental temptation remained: to take the jagged and contested edges of democratic processes, smooth them out, and call the process a success. However, public-facing and participatory democracy occurs precisely in those jagged and sharp-elbowed spaces where different interests collide, and the public interest must be calculated politically.

This connects directly to the second problem: context matters, and it’s often intractable. Public policy doesn’t land on statistical averages; it lands in particular places, within specific community relationships, sometimes via whakapapa, whenua, and whānau, but always in the specific and complex lives of our neighbours and the streets we live in.

Large language models don’t know those contexts. They’re designed to generalise, to produce outputs that sound plausible across many different settings. That’s perfectly fine for drafting a standard memo, but it becomes dangerous when what’s actually needed is deep attention to difference: the difference between someone who can pay rent this week and someone who cannot. Robodebt demonstrated what happens when systems systematically ignore such differences: they create debts out of thin air and, perhaps more destructively, grind down the trust that holds democratic institutions together.

The third problem flows from this disconnect between algorithmic confidence and contextual reality. Robodebt enforced debts that didn’t exist as though they were incontrovertible facts. The machine produced a number; officials acted as if it must be true. This same dynamic appears with LLMs, which carry an identical risk: they remain confident even when fundamentally wrong, offering no pause, no hesitation, no acknowledgment of what they cannot know.

In government settings, that artificial confidence can easily masquerade as legitimate authority. Yet governing requires something quite different: the capacity to hold uncertainty, to admit what remains inexplicable, and to let doubt sharpen rather than paralyse judgement. Not all public outcomes prove controllable, regardless of how much data you deploy. Change programmes consistently forget this fundamental limitation. Robodebt was designed to deliver efficiency, but ultimately cost far more, both financially and socially. That paradox sits at the heart of governance: you set out toward one destination and frequently arrive at its opposite.

LLMs assume continuity, making predictions based on established patterns. But governing occurs in the space where intention and consequence refuse to align, where paradox represents the norm, and where rapid learning becomes essential for survival.

This brings us to the fourth, final and perhaps most crucial problem: the systematic exclusion of democratic voice. We now expect, quite reasonably, to have a voice in decisions that affect us: that means being heard in consultations, being able to challenge government priorities, and, for some, directly helping shape regulations, rules, and budgets. Robodebt excluded the voice by design, reducing citizens to data points in an algorithmic process.

LLMs achieve a similar effect, albeit more subtly. My experience suggests they tend to close dialogue into outputs. They might offer answers, but they don’t necessarily build the relationships that sustain democratic engagement. The productivity they promise is essentially about generating paperwork faster.

I propose an alternative hypothesis: that genuine productivity in the public sector originates from trust built through conversation, shared context, and sustained dialogue about both the nature of problems and the structure of potential solutions. Productivity also comes from strategic clarity from the authorising environment. This kind of productivity cannot be measured simply by the speed of output generation.

Once trust disappears, the efficiency of your system becomes irrelevant. Robodebt left us with this stark lesson. The Royal Commission stated it bluntly: automation without accountability corrodes the state itself. LLMs don’t escape this fundamental truth. They potentially repackage the same dream of technological control in a more sophisticated form, but the underlying danger remains unchanged. If we allow them to operate without proper accountability mechanisms, we risk ending up back in the same wreckage.

If there is a constructive way ahead, it lies in combining the best of our technological tools with the best of our collective judgement. LLMs can certainly help clear administrative undergrowth: summarising lengthy documents, helping to draft policy options, freeing up precious hours in advisory systems. But they cannot determine what matters most, or how to balance fairness against cost, or how to rebuild trust when it begins to fray. And while some AI can produce an algorithm that targets risk, the real work remains that of actual people, grounded in relationships, technically competent, and usually with some local or sector accountability, who are also able to provide sustained attention to context.

The lesson from Robodebt is not to reject technology wholesale, but to weave it carefully into public life: transparent about its limitations, accountable in its application, and always secondary to the slow, complex craft of governing our shared democratic spaces. Hold to that principle, and technology might actually support rather than corrode the kind of productivity that genuinely matters.

PS: Waitangi Tribunal Thursdays restarts next week.

References

Royal Commission into the Robodebt Scheme. (2023). Report of the Royal Commission into the Robodebt Scheme (Final report, Vols. 1–3). Commonwealth of Australia. https://robo-debt.royalcommission.gov.au

Additional Reading

If you’re looking for an entry point into AI and LLMs discourse, and you’d prefer to avoid the hype and noise, and you don’t want to dive into peer-reviewed journals, here are a few more accessible articles and ideas to get you started. Start anywhere.

Aaron Tay, “The AI-Powered Library Search That Refused to Search.” 2025 A real-world example of AI systems failing in practice

Abeba Birhane, “AI for Good [Appearance?]” 2025 A valuable critique of the AI and LLM industry’s superficial and performatative diversity efforts

Adi Robertson, “Chatbots Aren’t Telling You Their Secrets.” 2025 Answers the question of why AI opacity prevents accountability, and why that matters

Aiha Nguyen and Alexandra Mateescu, “Generative AI and Labour: Power, Hype, and Value at Work.” 2024 Useful labour analysis of AI deployment

Charlie Warzel, “AI is a Mass-Delusion Event.” 2025 Essential context on the hype driving decisions

Cooper Lund, “The Incuriosity Engine.” 2025 AI’s impact on curiosity and learning: especially useful for policy teams and educators

Edward Zitron, “OpenAI is a Systemic Risk to the Tech Industry.” 2025 Explores the potential economy-wide risks that go unreported and undiscussed

Ezekiel Dixon-Román et al, “The Racialising Forces Of/In AI Technologies.” 2019 My go-to: a foundational analysis of how AI systems embed racial bias: cause they do, sorry and not sorry

Gary Marcus, “Why DO Large Language Models Hallucinate?” 2025 Technical explanation of AI’s fundamental unreliability and unpredictability

Jasmine Lianalyn Rocha, “AI in the Workplace: The Dangers of Generative AI in Employment Decisions.” 2024 How AI systems make consequential decisions about people’s livelihoods: an important link to the Robodebt case

Jeremy Hsu, “AI Doesn’t Know ‘No’ – and that’s a Huge Problem for Medical Bots.” 2025 Why AI can’t and shouldn’t handle health decision-making

Josh Freeman, “If Memory is the Residue of Thought, What are we Learning from AI?” 2025 Early reader for those interested in how AI undermines human cognitive development

Katharine Miller, “Privacy in an AI Era: How Do We Protect our Personal Information?” 2025 A primer on AI’s privacy threats

Krishani Dhanji, “Use of AI Could Worsen Racism, Sexism in Australia, Human Rights Commissioner Warns.” 2025 Direct warning about AI repeating Australia’s algorithmic discrimination

Kyle Orland, “LLMs ‘Simulated Reasoning’ Abilities are a ‘Brittle Mirage’ Researchers Find.” 2025 Research showing AI reasoning is not what it appears

Matthew Cheney, “There is (Still) No Ethical Use of AI.” 2025 Argument for fundamental reconsideration of AI deployment

Mercy Mutemi, “African Workers Are Taking on Meta And the World Should Pay Attention.” 2025 Current labour exploitation in AI training

Michael Townsen Hicks et al, “ChatGPT is Bullshit.” 2024 Academic analysis of why LLMs can’t be trusted for factual decisions

New York Bar Association, Presidential Task Force on Artificial Intelligence and Digital Technologies, “The Impact of the Use of AI on People with Disabilities.” 2025 How AI systems exclude and harm vulnerable populations

Reese Rogers, “OpenAI Designed ChatGPT-5 to be Safer: It Still Outputs Gay Slurs.” 2025 Evidence that bias persists despite safety efforts

Salvador Santino F. Regilme, “Artificial Intelligence Colonialism: Environmental Damage, Labour Exploitation, and Human Rights Crises in the Global South.” 2024 Comprehensive analysis of AI’s colonial impacts

Siôn Geschwindt, “ChatGPT Advises Women to Ask for Lower Salaries, Study Finds.” 2025 A concrete example of AI perpetuating discrimination

Stephanie Kirmir, “Disability, Accessibility, and AI.” 2024 How AI systems exclude whaikaha

United Nations Advisory Body on Artificial Intelligence, “Final Report: Governing AI for Humanity.” 2024 International consensus on the need for AI governance