We shape our tools and thereafter they shape us.
John M. Culkin, 1967
In my previous article, I wrote about what it means to put a boundary around an agentic AI system, covering agents, permissions, networks and guardrails. Everything I went through there was technical. But an agentic AI system in use also includes the people who work with it, and the context in which they work. You can’t leave them out of the picture and expect any meaningful governance to hold.
Earlier this month, Satya Nadella, CEO of Microsoft argued that we need to start thinking of advanced AI models as insider risks1. “We need to start with an engineering approach to containment and governance.” he said, “ We need to surround non-deterministic models with strong, deterministic system design, human controls, and reliable operating procedures, and establish industry standards where existing ones are insufficient.” I agree. That’s a start, but his focus goes on to be overwhelmingly technical: permissions, containment, verification. He does acknowledge human controls and operating procedures but without examining their real role, or how the technology might change the people expected to exercise them. What I noted most is that his article places humans outside the agentic system.
And that is a reflection of the wider tendency in agentic AI safety. It’s predominantly about testing the model, hardening its harness, observing and responding to it’s actions. Which assumes we’ve dealt with the role of people within the system. Yet a model can stay within every technical boundary and still be part of a system that causes harm. That harm emerges from the interaction between the technical components and people using them. Those people operate inside the agentic system.
To see why, it’s probably best to start with something simpler than an autonomous agent.
When a prediction becomes a decision
There’s a tool used by UnitedHealth called nH Predict. It estimates how long patients on Medicare Advantage plans in the US might need rehabilitation in a nursing facility after they’re discharged from hospital. The tool is a statistical regression based on patients with similar past cases and it estimates their likely length of stay. Good old-fashioned AI then - not an LLM or agent in sight.
In May 2022, an elderly man called Gene Lokken, fell badly and broke his leg and ankle. After treatment he was sent into a nursing facility to recover, and after two months, UnitedHealth stopped paying for his care, even though his clinicians said he needed to remain. His family appealed unsuccessfully and so they had to pay for his continued stay until his death the following year. They’re now suing, alleging that UnitedHealth allowed nH Predict’s estimates to override clinical judgement. UnitedHealth disputes that, saying the tool was no more than a guide to care planning, not a basis for coverage decisions.2
The case raises this important question, one central to agentic AI adoption: what happens when a prediction, intended to only inform a decision, acquires the authority to determine it?
The tool changed the people
You see, according to internal documents and former employees’ accounts, naviHealth, the company behind nH Predict, had set targets for case managers to keep patients’ stays close to the tool’s estimates. The tolerance was just 1%, with staff describing pressure to follow the numbers or face consequences all the way to dismissal for departing from them.3
So put yourself in that case managers shoes. You’re handling dozens of cases, with that kind of pressure on you. Following the estimate is easy. Questioning it would take time, evidence and a willingness to challenge the target against which your work is judged, and quite possibly the security of your job. Gradually, to you that number stops being one input among several and it starts to become the default. From that point, everything you look at, everything you question or justify all shifts around it. You’re still a case manager who still probably cares about patients. But the conditions within which you exercise judgement have changed. The tool and the number it creates changes you.
And patients of course are part of that system too. So a letter telling someone their covered care is ending carries authority. That changes what they believe is possible and it can affect whether they appeal, pay for care or leave the facility. In this case, very few patients challenged the decision, but when they did 90% of the decisions were reversed. That doesn’t necessarily tell you the model was wrong, more likely it tells you that the human workflow beyond the model was seriously flawed, and that’s what created the harm.
… and people people changed the tool
The relationship runs both ways. A prediction of typical length of stay is a planning aid, not a determination of an individual patient’s medical need. But then management driven by different incentives can turn what was only a prediction into a performance expectation. Over time, routine approvals give it authority and then the steady stream of coverage letters can turn it into financial consequences and norms.
None of that requires any piece of software to change. What changes is the role people give its output, and the work organised around it. That’s precisely what is allege happened with nH Predict. UnitedHealth says it didn’t. The case is playing out still, but regardless of how it’s resolved, I think it’s the mechanism that matters: people can change what a technology does without changing the technology itself.
People are part of the system
It’s tempting to call this a failure of individual judgement, or a lack of oversight. Case managers should have pushed back. Managers should have known better. So the fix is documented policy, more training and a culture that encourages people to speak up. Just smear a little old-fashioned governance over it.
But that leaves the conditions that shaped those decisions untouched. As a case manager, you might be told to exercise independent judgement, even be trained to do it, but if you’re held to a 1% target, then the target is going to win.
Nancy Leveson, in Engineering a Safer World4, describes a better way to think about this. Leveson treats people as components of the system, not safeguards or independent overseers keeping watch outside it. She models how their decisions depend on what they can comprehend, their biases, the information they can see, the pressures they face and the authority they can exercise without repercussion. She concludes that if you want to change how people behave, then you have to examine and change those conditions in which they act.
“Operator behavior is a product of the environment in which it occurs.”
Nancy Leveson, Engineering a Safer World
This only gets more important as we move on from tools like nH Predict to agentic systems. Human oversight is the default answer to almost every difficult question about agent risk. A person will review the decision. They can step in and override. But there’s a big difference between having a human in the loop and establishing that meaningful oversight exists. The reviewer really has to have the evidence, time, competence and authority to challenge an agent. And they have to work within an organization that allows them to do that review.
And because it wasn’t hard enough, there’s another unavoidable complication. Even if we do manage to design effective oversight at the outset, we can’t assume it will stay effective. Automation changes how people work, to what they pay attention to and what they practise. So over time, it erodes the very judgment and capabilities we’re relying on when something goes wrong.
People are part of the system, and the system changes them.
Calibration is built and lost in use
The tragedy of Air France 447 shows how that happens.
In June 2009, ice crystals formed in the pitot tubespeed sensors of an Airbus A330 over the Atlantic. The autopilot disconnected. The aircraft was stilly fully operational, but the speed sensors were off. Sadly the crew, accustomed to uneventful autopilot flight and confused by conflicting readings, failed to recognize and recover from a stall. All 228 people on board died. [5]5
It was reported as pilot error, but the investigation tells a bit more complicated story, involving gaps in training, confusing indications, poor crew coordination and limited experience of manual flying at high altitude. The safety of the aircraft depended on people taking over when automation failed, but the wider system hadn’t adequately prepared them for that moment.
Lisanne Bainbridge described this irony of automation in 1983: “as automated control becomes more capable, the human contribution at the moments when intervention is needed can become more crucial.” But automation removes the routine practice through which people develop the skills to intervene.6
I’ve previously written about this as the problem of cognitive calibration.7 The question isn’t really about whether people can or should trust AI-driven automation or distrust it. It’s really about whether their trust matches what the system can do, in those particular circumstances in which they’re using it. And whether or not they they could even recognize when their mental model is no longer calibrated.
And calibration changes with experience. Every time you see an accurate, confident output, you feel that checking it next time is a little less necessary. People stop noticing things they once closely examined and they lose practice making judgements independently. A model-only evaluation can tell us how a model performs under test. It can’t tell us what its users will have stopped doing after months of mostly reliable operation. Just think about how you use AI models right now - do you hold the same scrutiny and attentive review as you once did?
In that sense, a very accurate model or a very helpful agent may create a more subtle oversight problem than an obviously unreliable or unhelpful one.
Why this all gets harder with agentic AI
So let me be clear that nH Predict wasn’t an autonomous agent. It didn’t plan, call tools or take some sequence of actions on its own. People were involved throughout that whole process. And yet, that wider system could still produce harm.
But just imagine an agentic version of this. Say an agent pulls together the patient’s assessment, and then it applies the policy. It comes to a draft or even final decision and it drafts the letter. Now if they’re still employed, maybe the person who once did that work now becomes a supervisor to the agent.
But supervising review is a very different job to performing review. Now they’re going to see and review a lot fewer cases in detail, because instead they’re looking at aggregates and metrics. Even if they do review, their comprehension of the case is now coming increasingly from the tool they’re expected to challenge. And the more work the agent performs, the fewer opportunities people will have to maintain independent judgement about it.
Agents bring in speed and scale. A case manager who defers too readily is only going to affect their own caseload. But a supervisor who routinely approves an agent’s work would probably affect hundreds or thousands of decisions. We’ve all witnessed approval steps that degrade into nothing more than rubber-stamping, and there’s no reason I can see that agentic AI will reduce that without a better approach. Having a human approval step doesn’t tell you that meaningful oversight exists. Designed and operated poorly, it might actually make things worse.
So if human behavior is part of the system, and the harm sits in the interactions between people and tools, then the conditions shaping that behaviour and those interactions have to be a central part of our governance approach. We have to design human controls with the same seriousness and deliberate attention as we might apply to technical ones.
We have to design governance around the full scope of the system, treating us users as a part of that system, not just the tech.
The system changes as we use it
I don’t think the first question for agentic AI governance then can be whether the model is safe, or even the model operating within a harness is well contained. It should be what the whole system will do in use, how that use will change the people within it, and how their changing behaviour will alter the system in return.
That’s not to dismiss the value of evaluations, access controls or containment. Those are still essential technical controls. But the nH Predict allegations and the Air France 447 disaster show how automation can undermine the human capabilities on which safe operation depends.
You can’t govern an agentic enterprise system as a technical problem with a technical solution. People are part of the system, so are the conditions under which they exercise judgment. And both change in response to the system and how it operates. That requires an adaptive approach that deals with both the technical and the human behaviors together.
In my next article, I’ll turn to safety as an emergent property. How it’s something that exists only at the level of the whole system. We’ll look at a real case of an agentic system in which every single part was nominally safe, and yet together they still produced a dangerous outcome.
Thank you for reading - your thoughts and feedback always welcome.
References
Epigraph. John M. Culkin, “A Schoolman’s Guide to Marshall McLuhan,” Saturday Review, 18 March 1967, pp. 51–53, 70.
Satya Nadella, “Models as Insider Risks in the Super Intelligence Era” October 2026 https://snscratchpad.com/posts/models-as-insider-risks/
Estate of Gene B. Lokken et al. v. UnitedHealth Group, Inc. et al., No. 0:23-cv-03514 (D. Minn.), complaint filed 14 November 2023. Complaint; Casey Ross and Bob Herman, “UnitedHealth faces class action lawsuit over algorithmic care denials in Medicare Advantage plans,” STAT, 14 November 2023.
Casey Ross and Bob Herman, “UnitedHealth pushed employees to follow an algorithm to cut off Medicare patients’ rehab care,” STAT, 14 November 2023. Republished in the Boston Globe.
Bureau d’Enquêtes et d’Analyses, Final Report: Air France 447, July 2012. Official investigation.
Lisanne Bainbridge, “Ironies of Automation,” Automatica 19, no. 6 (1983): 775–779. doi:10.1016/0005-1098(83)90046-8.
James Kavanagh and Alberto Chierici, “Cognitive Calibration,” Doing AI Governance, AI Career Pro. governance.aicareer.pro.
A team that starts exercising those override rights could initially look less efficient: more exceptions, longer reviews, fewer cases cleared. Management would have to recognise that as a possible sign of oversight working.