We shape our tools and thereafter they shape us.

John M. Culkin, 1967

In my previous article, I wrote about what it means to put a boundary around an agentic AI system, covering agents, permissions, networks and guardrails. Everything I went through there was technical. But an agentic AI system in use also includes the people who work with it, and the context in which they work. You can’t leave them out of the picture and expect any meaningful governance to hold.

Earlier this month, Satya Nadella, CEO of Microsoft argued that we need to start thinking of advanced AI models as insider risks1. “We need to start with an engineering approach to containment and governance.” he said, “ We need to surround non-deterministic models with strong, deterministic system design, human controls, and reliable operating procedures, and establish industry standards where existing ones are insufficient.” I agree. That’s a start, but his focus goes on to be overwhelmingly technical: permissions, containment, verification. He does acknowledge human controls and operating procedures but without examining their real role, or how the technology might change the people expected to exercise them. What I noted most is that his article places humans outside the agentic system.

And that is a reflection of the wider tendency in agentic AI safety. It’s predominantly about testing the model, hardening its harness, observing and responding to it’s actions. Which assumes we’ve dealt with the role of people within the system. Yet a model can stay within every technical boundary and still be part of a system that causes harm. That harm emerges from the interaction between the technical components and people using them. Those people operate inside the agentic system.

To see why, it’s probably best to start with something simpler than an autonomous agent.

When a prediction becomes a decision

There’s a tool used by UnitedHealth called nH Predict. It estimates how long patients on Medicare Advantage plans in the US might need rehabilitation in a nursing facility after they’re discharged from hospital. The tool is a statistical regression based on patients with similar past cases and it estimates their likely length of stay. Good old-fashioned AI then - not an LLM or agent in sight.

In May 2022, an elderly man called Gene Lokken, fell badly and broke his leg and ankle. After treatment he was sent into a nursing facility to recover, and after two months, UnitedHealth stopped paying for his care, even though his clinicians said he needed to remain. His family appealed unsuccessfully and so they had to pay for his continued stay until his death the following year. They’re now suing, alleging that UnitedHealth allowed nH Predict’s estimates to override clinical judgement. UnitedHealth disputes that, saying the tool was no more than a guide to care planning, not a basis for coverage decisions.2

The case raises this important question, one central to agentic AI adoption: what happens when a prediction, intended to only inform a decision, acquires the authority to determine it?

The tool changed the people

You see, according to internal documents and former employees’ accounts, naviHealth, the company behind nH Predict, had set targets for case managers to keep patients’ stays close to the tool’s estimates. The tolerance was just 1%, with staff describing pressure to follow the numbers or face consequences all the way to dismissal for departing from them.3

So put yourself in that case managers shoes. You’re handling dozens of cases, with that kind of pressure on you. Following the estimate is easy. Questioning it would take time, evidence and a willingness to challenge the target against which your work is judged, and quite possibly the security of your job. Gradually, to you that number stops being one input among several and it starts to become the default. From that point, everything you look at, everything you question or justify all shifts around it. You’re still a case manager who still probably cares about patients. But the conditions within which you exercise judgement have changed. The tool and the number it creates changes you.

And patients of course are part of that system too. So a letter telling someone their covered care is ending carries authority. That changes what they believe is possible and it can affect whether they appeal, pay for care or leave the facility. In this case, very few patients challenged the decision, but when they did 90% of the decisions were reversed. That doesn’t necessarily tell you the model was wrong, more likely it tells you that the human workflow beyond the model was seriously flawed, and that’s what created the harm.

… and people people changed the tool

The relationship runs both ways. A prediction of typical length of stay is a planning aid, not a determination of an individual patient’s medical need. But then management driven by different incentives can turn what was only a prediction into a performance expectation. Over time, routine approvals give it authority and then the steady stream of coverage letters can turn it into financial consequences and norms.

None of that requires any piece of software to change. What changes is the role people give its output, and the work organised around it. That’s precisely what is allege happened with nH Predict. UnitedHealth says it didn’t. The case is playing out still, but regardless of how it’s resolved, I think it’s the mechanism that matters: people can change what a technology does without changing the technology itself.

People are part of the system

It’s tempting to call this a failure of individual judgement, or a lack of oversight. Case managers should have pushed back. Managers should have known better. So the fix is documented policy, more training and a culture that encourages people to speak up. Just smear a little old-fashioned governance over it.

But that leaves the conditions that shaped those decisions untouched. As a case manager, you might be told to exercise independent judgement, even be trained to do it, but if you’re held to a 1% target, then the target is going to win.

Nancy Leveson, in Engineering a Safer World4, describes a better way to think about this. Leveson treats people as components of the system, not safeguards or independent overseers keeping watch outside it. She models how their decisions depend on what they can comprehend, their biases, the information they can see, the pressures they face and the authority they can exercise without repercussion. She concludes that if you want to change how people behave, then you have to examine and change those conditions in which they act.

“Operator behavior is a product of the environment in which it occurs.”

Nancy Leveson, Engineering a Safer World

This only gets more important as we move on from tools like nH Predict to agentic systems. Human oversight is the default answer to almost every difficult question about agent risk. A person will review the decision. They can step in and override. But there’s a big difference between having a human in the loop and establishing that meaningful oversight exists. The reviewer really has to have the evidence, time, competence and authority to challenge an agent. And they have to work within an organization that allows them to do that review.

And because it wasn’t hard enough, there’s another unavoidable complication. Even if we do manage to design effective oversight at the outset, we can’t assume it will stay effective. Automation changes how people work, to what they pay attention to and what they practise. So over time, it erodes the very judgment and capabilities we’re relying on when something goes wrong.

People are part of the system, and the system changes them.

Calibration is built and lost in use

The tragedy of Air France 447 shows how that happens.

In June 2009, ice crystals formed in the pitot tubespeed sensors of an Airbus A330 over the Atlantic. The autopilot disconnected. The aircraft was stilly fully operational, but the speed sensors were off. Sadly the crew, accustomed to uneventful autopilot flight and confused by conflicting readings, failed to recognize and recover from a stall. All 228 people on board died. [5]5

It was reported as pilot error, but the investigation tells a bit more complicated story, involving gaps in training, confusing indications, poor crew coordination and limited experience of manual flying at high altitude. The safety of the aircraft depended on people taking over when automation failed, but the wider system hadn’t adequately prepared them for that moment.

Lisanne Bainbridge described this irony of automation in 1983: “as automated control becomes more capable, the human contribution at the moments when intervention is needed can become more crucial.” But automation removes the routine practice through which people develop the skills to intervene.6

I’ve previously written about this as the problem of cognitive calibration.7 The question isn’t really about whether people can or should trust AI-driven automation or distrust it. It’s really about whether their trust matches what the system can do, in those particular circumstances in which they’re using it. And whether or not they they could even recognize when their mental model is no longer calibrated.

And calibration changes with experience. Every time you see an accurate, confident output, you feel that checking it next time is a little less necessary. People stop noticing things they once closely examined and they lose practice making judgements independently. A model-only evaluation can tell us how a model performs under test. It can’t tell us what its users will have stopped doing after months of mostly reliable operation. Just think about how you use AI models right now - do you hold the same scrutiny and attentive review as you once did?

In that sense, a very accurate model or a very helpful agent may create a more subtle oversight problem than an obviously unreliable or unhelpful one.

Why this all gets harder with agentic AI

So let me be clear that nH Predict wasn’t an autonomous agent. It didn’t plan, call tools or take some sequence of actions on its own. People were involved throughout that whole process. And yet, that wider system could still produce harm.

But just imagine an agentic version of this. Say an agent pulls together the patient’s assessment, and then it applies the policy. It comes to a draft or even final decision and it drafts the letter. Now if they’re still employed, maybe the person who once did that work now becomes a supervisor to the agent.

But supervising review is a very different job to performing review. Now they’re going to see and review a lot fewer cases in detail, because instead they’re looking at aggregates and metrics. Even if they do review, their comprehension of the case is now coming increasingly from the tool they’re expected to challenge. And the more work the agent performs, the fewer opportunities people will have to maintain independent judgement about it.

Agents bring in speed and scale. A case manager who defers too readily is only going to affect their own caseload. But a supervisor who routinely approves an agent’s work would probably affect hundreds or thousands of decisions. We’ve all witnessed approval steps that degrade into nothing more than rubber-stamping, and there’s no reason I can see that agentic AI will reduce that without a better approach. Having a human approval step doesn’t tell you that meaningful oversight exists. Designed and operated poorly, it might actually make things worse.

So if human behavior is part of the system, and the harm sits in the interactions between people and tools, then the conditions shaping that behaviour and those interactions have to be a central part of our governance approach. We have to design human controls with the same seriousness and deliberate attention as we might apply to technical ones.

We have to design governance around the full scope of the system, treating us users as a part of that system, not just the tech.

The system changes as we use it

I don’t think the first question for agentic AI governance then can be whether the model is safe, or even the model operating within a harness is well contained. It should be what the whole system will do in use, how that use will change the people within it, and how their changing behaviour will alter the system in return.

That’s not to dismiss the value of evaluations, access controls or containment. Those are still essential technical controls. But the nH Predict allegations and the Air France 447 disaster show how automation can undermine the human capabilities on which safe operation depends.

You can’t govern an agentic enterprise system as a technical problem with a technical solution. People are part of the system, so are the conditions under which they exercise judgment. And both change in response to the system and how it operates. That requires an adaptive approach that deals with both the technical and the human behaviors together.

In my next article, I’ll turn to safety as an emergent property. How it’s something that exists only at the level of the whole system. We’ll look at a real case of an agentic system in which every single part was nominally safe, and yet together they still produced a dangerous outcome.

Thank you for reading - your thoughts and feedback always welcome.

Leave a comment

Share

References

Epigraph. John M. Culkin, “A Schoolman’s Guide to Marshall McLuhan,” Saturday Review, 18 March 1967, pp. 51–53, 70.

Subscribe now

“For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.”

Richard P. Feynman · Appendix F, Report of the Presidential Commission on the Space Shuttle Challenger Accident · 1986

On 16th July, Hugging Face published a new disclosure of a security incident1. They assured customers that their data hadn’t been affected, but explained that an unknown AI agent system had somehow gained access to their production infrastructure. In the process, it had obtained internal secure credentials and some datasets2. They didn’t know where the activity came from.

Just under a week later, OpenAI put their hand up, with an odd mix of guilt and what appeared to be celebratory glee. An OpenAI agent had been running inside an experimental evaluation environment with reduced guardrails. It had been seemingly isolated but had somehow found a route out and into another organization’s infrastructure3.

It was an experiment as far as OpenAI was concerned, but to Hugging Face, it came as an attack.

When we call something an experiment or a test, we intuitively have this notion of bounded activity. A chemistry experiment in a lab, a biological experiment in a sealed room. We let something uncertain happen so that we can learn from it but expect it to happen in a way that is isolated and contained from the outside. Experimentation is a necessary part of innovation, but the outside world should be protected from that experiment. It should be performed and constrained within a boundary.

So the question of this article is what such a boundary really means. How do we describe and characterize it? How would we know if that boundary has been breached? And can a complex AI system even be bounded? Seems like the right place to start our exploration of what kind of governance we need for agentic AI systems in the enterprise.

Now it’s tempting to see a boundary as a line on an architecture diagram. Draw a box. Put the model and tools inside the box, monitor what happens inside the box, put an airgap of network access around it.

But such a picture would be too simple. A boundary isn’t just a barrier that stops a system from connecting directly to something outside it. It’s not just about the perimeter. We’ve also got to think about what could pass through controlled connections we deliberately put in place, even if they’re designed to only allow controlled removal of telemetry or one-way control signals into the system. We need to think about routes the system might be able to assemble from components available inside, and what it might be able to influence outside the boundary without ever actually crossing it directly. Plus, there’s another question: if a system attempted any of those things or even succeeded, could anyone outside the boundary even notice?

Those give us four different avenues to explore the nature of boundaries and containment of complex AI systems, and the OpenAI / Hugging Face incident interestingly shows us how all four play out, and fail

What can cross the boundary?

So the obvious place to start is the perimeter. If we were to assert that a system is isolated, what can be permitted to pass between the inside and outside?

In the OpenAI evaluation, we know the agents didn’t have direct and unfettered access to the internet. But a complete disconnection would have hindered the usefulness and observability of the experiment, so there were some channels of permitted access. The agents needed to install software packages, so they were provided with access to an internal hosted service called Artifactory, which could obtain packages from the internet on their behalf. It was a deliberately narrow and controlled connection through the boundary.

The problem wasn’t that the connection existed, but rather that it could actually do more than the purpose it was intended for. A previously unknown vulnerability allowed the agents to use Artifactory as part of a route out of the environment. So a service that had been intended to only fetch software packages on demand became a way that agents could reach out4.

We have to be mindful that a boundary doesn’t have to eliminate every connection. Containment doesn’t mean total isolation, and most useful systems or even experiments couldn’t work that way. The question is more about whether those connections can be effectively constrained in the purposes we intend them to have, even if they expose - as all software can - unknown vulnerabilities.

So it matters how such a connection is implemented. A simple software rule, an authenticated application interface, an external proxy service, and a physically enforced one-way channel can all appear on an architecture diagram as a narrow connection, but there’s a world of difference in the kind of assurance each provides. This has been a long recognized problem in high-assurance systems used in defense and national security, where physical data diodes are deliberately used to separate unclassified from highly classified environments. A high assurance data diode is a device that permits a physically enforced one-way data flow. They don’t make a system invulnerable, but they do make one particular property of the boundary much harder to violate.

That gives us one better way then to think about containment. It’s not whether the boundary has no openings, because useful systems almost always will. It’s whether the assurance properties at those openings are enforced strongly enough to match the consequences that are at stake.

What can it assemble from what’s inside?

Even if we do understand and strongly assure every connection through the perimeter, there’s another way the boundary could fail. That’s if the system can assemble capabilities from the things it already has available inside.

OpenAI’s later report described agents using the Artifactory service for something very different than installing software. They found ways to leave information that other agents could retrieve, and when that was discovered and disrupted, they found another, including embedding messages in directory names. So Artifactory was subverted to become a communication channel between agents, and when OpenAI discovered that and blocked it, the system recreated a communication channel using a different mechanism5.

OpenAI described how the experiment design specifically did not include inter-agent communication, but the ingredients were there inside the boundary. One process could leave information in a location, another could come across it later. No agent had the ability to directly communicate to another, and yet inter-agent communication emerged. It didn’t remain a curiosity between two agents. Within hours of the message board appearing, 50 agents were using it, which later grew to 1200 agents exchanging over 70,000 messages. But more importantly, it changed the capability of the whole. Agents started preserving and sharing work between runs, building on the discoveries of others, dividing labor and coordinating work. So it wasn’t just a communication issue, it was that this communication enabled collective action6.

This is important because it highlights a limit to what we can learn and assert when we examine components individually.

A complicated system can contain thousands of parts and still be largely decomposable and predictable. We can figure out what each component does, specify how they are to interact and then reason from those interactions toward the behavior of the whole system. It might be a difficult analysis, but it’s tractable in principle.

Complex systems are different. Their behavior emerges from interactions between components, feedback, shared state, sequencing and a sensitivity to initial conditions. The whole can acquire properties that aren’t properties of any part.

And agentic AI systems add something more. They’re not merely executing combinations of tasks that engineers specified in advance, nor are they simply randomly acting and discovering. They are self-directed toward achieving a goal. So while pursuing some task, an agentic system can explore the possibilities presented by its environment, and discover useful relationships between tools, services and information that could enable progress toward a goal, even if unimagined by the engineers.

That’s not to say that architectural analysis isn’t worthwhile. It still makes sense to minimize unnecessary or dangerous capabilities, to look for likely interactions that could lead to dangerous combinations. But at the degree of complexity we see in enterprise agentic systems, an exhaustive analysis can’t be credible assurance that we’ve managed to find every possible combination.

A boundary around some collection of components in a complex system is only meaningful if the behavior that emerges remains contained by that boundary. If the system can combine what is inside the boundary to create a new route out, some new way to coordinate or some other capability that defeats the purpose of containment, then the boundary isn’t effective anymore.

Once a single agent discovered a way to subvert Artifactory for its own purposes of gaining outside access, that knowledge was shared through the improvised comms channel, in turn making the weakness in the perimeter far more consequential.

A boundary has to be resilient to whatever inside it can become, just by assembling capabilities from what it has available. You don’t build a high wall around human prisoners, but then leave the tools and materials inside for them to build a ladder.

What has to remain beyond influence?

There’s another less obvious way that a boundary can fail without the system finding any route through it at all. Some boundaries aren’t intended to prevent communication, they exist to preserve independence despite it. So we could place something deliberately separate and outside the boundary, but it could nevertheless come under the system’s influence. The OpenAI/Hugging Face incident again provides an example.

The agents were narrowly interacting with an automated scorer. They completed a task, they submitted their work and then received back a result. This of course required some information to cross the boundary between the thing being evaluated and whatever mechanism is evaluating it.

But despite the narrow design intent, the agents actually did more than just passively submit their work for judgment. They researched how the ExploitGym scorer operated and coordinated different ways they might manipulate or fool it. The METR investigation describes a set of ‘projects’ undertaken by agents to extract information about the scorer. They successfully altered some transcripts, and after multiple attempts figured out some ways to spoof reported tool calls.

In the end, the agents didn’t successfully change their scores because they misunderstood how OpenAI’s internal scorer worked. But it shows the agents were actively trying to figure out the evaluation machinery and see if it was something they could influence, or interfere with. There is precedent for prior success. In a previous and very different evaluation of OpenAI’s o3 model, the model was being scored on how efficiently it completed a programming task. But instead of just improving the task itself, METR found it interfered with the measurement process to make its performance appear better than it really was. In 20% of cases, that manipulation was successful7.

That’s a functional boundary problem. It’s not escape from the boundary, but something more subtle. It’s reaching across the boundary to affect the mechanism designed to evaluate or control it.

The scorer couldn’t work without receiving information from the agents. But what it needed to retain was independence from their influence or control. The challenge for us here is that many of the things we rely on to govern an AI system have to interact with it in some way. A safety monitor has to observe behavior, an audit record needs to receive logs, a human reviewer has to see enough information to approve or reject an action.

Indeed, I think any control that enforces limits on authority has to receive some information from the system that it constrains.

A boundary can still be intact in a physical sense while failing functionally, if the system can acquire enough influence over what the outside component sees or otherwise operates, to undermine their function. Some boundaries exist not to stop information crossing, but to constrain the kind of influence that can cross with it.

Boundary isolation has to then mean preserving independence despite interaction.

How would we know the boundary is not holding?

The three questions we’re asking so far are about the properties we want of an effective boundary. But we’ve got to ask how would anyone be able to tell if any one of those properties were no longer true on an operational system. That’s more than a monitoring problem though.

OpenAI’s account described an alert on 27th June, three weeks earlier than Hugging Face’s disclosure. It led responders to investigate unusual activity involving Artifactory. Their analysis identified agents using the service as both a communication mechanism and a route out. So they closed the vulnerabilities, revoked some credentials and temporarily took the service offline8.

That alert shows that the actions of the system were not invisible. But it clearly was misunderstood. The OpenAI report says that the significance of the inter-agent communication was not apparent to the technical leaders involved. So even though they worked to remove the specific vulnerabilities identified, they failed to understand both the significance of the boundary breach and its likelihood to reoccur. They gave the greenlight for experiments to proceed.

The activity wasn’t invisible, people investigated and technical problems got fixed. What didn’t happen was the work to understand what those signals really meant.

Karl Weick used the term sensemaking for this kind of organizational work of constructing an understanding of a volatile situation as events unfold. Monitoring tells you something happened. Sensemaking is about what that means for the situation you think you are in.

“Sensemaking is about the enlargement of small cues. It is a search for contexts within which small details fit together and make sense.”

Karl E. Weick · Sensemaking in Organizations · 1995 9

To the OpenAI leaders, the unusual use of Artifactory looked like a vulnerability to patch. Inter-agent communication looked like curious and unexpected behavior to investigate. Both might be reasonable in isolation. But together, they could have supported a more consequential conclusion that agents thought to be separate were coordinating, sharing discoveries and propagating ways to breach the boundary. At that point, the question those OpenAI leaders should have been asking is whether the boundary could possibly hold.

Observability isn’t just collecting more telemetry. If we’re going to depend on a boundary, then we need to know which assumptions and properties make that boundary meaningful and resilient. And we have to be able to recognize when the evidence shows that those properties no longer hold. That recognition needs to lead to some action.

So a boundary can’t be architecturally designed and then assumed. We have to retain the capability to observe and make sense if either of our three properties of perimeter isolation, internal composition or functional isolation show indications of imminent failure.

A boundary is something we maintain

So is it possible to put a meaningful boundary around a complex agentic AI system?

Yes, I believe so. But only if we take care to understand the true nature of that boundary, and set our expectations appropriately.

A useful boundary depends on some properties that must hold. Connections through the perimeter must be constrained with assurance focused on the strength of those constraints. Capabilities that emerge from what is inside must not defeat the purpose of containment. Functions that judge or constrain must retain their independence. And we’ve got to be able to observe and recognize when any of those three no longer remain true.

That makes a boundary less of an architectural wall, and more like a maintained condition. Good architecture, strongly assured controls, careful separation, all are necessary. But with a complex agentic system, we can’t assume that those things once put in place will continue to serve their function. There has to be observable evidence that the boundary holds, and capacity to act when it doesn’t.

So we started our exploration of adaptive governance by focusing on what it means to draw a boundary, but before we close that exploration, we need to examine one more thing. You see, everything we’ve discussed in this article relates to the technical components - the agents, the network, the guardrails. But an agentic AI system in use goes beyond that, to include the people who use and interact with it, their context and workflow. We’re not just governing an arrangement of technical components, we’re governing an AI system used by people and their actions matter. That’s what we turn to in the next article.

Leave a comment

Share

“People continuously create safety. … human practitioner adaptations to changing conditions actually create safety from moment to moment.”

Richard I. Cook · “How Complex Systems Fail” · 20001

Subscribe now

Article Zero: An Invitation

I’m poor company watching a disaster movie.

While the hero leaps through flames, I’m wondering how things were allowed to get so desperate. What process failed? Which poor design decision looked reasonable on paper? What kind of organisational culture allowed a small mistake to cascade into catastrophe. I have a healthy obsession with disaster.

Piper Alpha is why. On a calm July night in 1988, 167 men died on an oil platform in the North Sea2. At first, the cause appeared to be human error. What came out later was a set of conditions, in the design, in the handover of information, in how the operator thought about safety, that made disaster almost inevitable. I was twelve, and I watched the coverage obsessively. I decided to become a chemical engineer and study the dynamics of safety.

Ten years later I was building dynamic simulations of chemical plants and putting operators through simulated disasters in full-scale control rooms. Pressure, heat, chemicals, people, time and technology came together as we modelled the variables of successful operation and disaster. The technical safety failures rarely surprised me: cascading trips, unstable cycles, runaway reactions, alarms arriving faster than anyone could read them. The people constantly surprised me: how a good operator made sense of a situation, coordinating response despite deafening klaxons and cascades of flashing alarms. How another could create the conditions for disaster, through an omission or an action far from the control room.

From chemical plants, I moved to software. Nearly two decades at Microsoft and Amazon Web Services(AWS), working through cloud infrastructure, cybersecurity, incident response, resilience and critical infrastructure security. At AWS, that work culminated in overseeing relationships with regulators around the world and creating the company’s first Responsible AI Assurance team. I spent half my time with regulators and lawyers, the other half with engineers and scientists. Over that time the technology changed completely as we, and our customers, deployed and operated increasingly intelligent systems at scale. But the question I was chasing did not change.

How do you keep a complex system of people and technology operating safely when you can’t anticipate everything it might do or the circumstances you might face?

It’s a difficult and it’s an urgent question. I don’t think we have a good enough answer for the agentic AI systems that we’re now deploying and giving greater freedom to act.

So I’m writing a book about it, in 100 pieces.

A book written in the open

The working title is The Practice of Adaptive Governance. I’m developing it as roughly a hundred connected articles, each short enough to read in about fifteen minutes. I have some thirty or so drafted, and I’m publishing them as I go because I want the argument open to challenge and to question before it hardens into a book.

I don’t yet know what will survive into the book. I reckon that some of these articles will probably merge, others might shrink to a paragraph, and I’m expecting to discover that I’m wrong about some things. It’s much better that I discover that here than after it’s printed.

But why focus on adaptive governance, and what does that even mean?

Most of the governance practice we see today is built to run forward to a decision. Something is specified, assessed, controlled and approved, and that approval carries the weight until the next review comes around. The EU AI Act works this way with it’s conformity assessment, ISO 42001 acts this way with audits and management reviews. Both have notional concepts of ongoing monitoring, review and correction, but they exist embedded in delayed cycles of after-the-fact incident reports, management reviews and annual audits. With some notable exceptions, this same mindset exists in almost every internal AI governance board, policy or compliance framework that I have seen. I call this static governance. Underneath so much of this is a basic assumption that I’ve grown increasingly uncomfortable with: that if we can somehow specify the right controls in advance, and then make sure people follow them consistently, all of the time, then we can keep the system safe.

Maybe that has some merit in a world that changes gradually in predictable ways. There’s real, meaningful work in every step of that sequence and I’ve spent a lot of time in my career doing it. Consistency brings us a degree of control, and clear commitments with enforceable boundaries are necessary. Constraints guide safe action and rules aren’t obsolete.

But there’s a problem. Its that the assessment starts ageing almost as soon as its completed. You just can’t predict all the right controls in advance for an AI system whose behavior is inherently unpredictable. You can’t certify a system to an assessed boundary, when use changes that boundary in unforeseeable ways.

An AI-System-in-use is an arrangement of models, tools, data, software, people, processes and delegated authority. Change any relationship in that arrangement and you can change what the system does without touching a central model at all. Connect the same model to another tool, use it in a context nobody assessed, or give it permission to act without asking first, and you may now have a materially different system. The system qualitatively changes the moment its placed into use.

I see four characteristics distinguish an AI system in use, and I think each one puts strain on the conventional approach of static governance:

• Complex. Behaviour emerges from interactions, so verifying each indvidual part doesn’t establish safe behavior of the whole system.

• Dynamic. The context around the model changes even when the model itself doesn’t. An assessment describes conditions that have already changed.

• Unbounded. The operating boundary of the system is porous and includes within it the people who use the system. This means the scope that was assessed is no longer the scope of the system that is now operating.

• Autonomous. The system selects and sequences its own actions, and we end up delegating authority without an ability to specify the behavior that results from it.

In our first article, I’ll start with a case where all four appear at once, the recent and already infamous case of how agents running inside an OpenAI evaluation found a way out of their test environment and into Hugging Face’s production infrastructure.3 It demonstrates how the activity of agents changed the conditions under which they were operating. The story illustrates how static governance alone cannot prove adequate for safety and security of a complex, agentic AI system in use.

The nature of the systems we are trying to govern has changed, and the nature of governance has to change with it.

“Safety is a system property, not a component property, and must be controlled at the system level, not the component level.”

Nancy Leveson · Engineering a Safer World: Systems Thinking Applied to Safety · 2012

Adaptive capacity, not conformity

I believe that governance has to be designed to work under those conditions. It has to notice when conditions are changing and be capable of doing something useful about it, without losing the boundaries that matter. Sometimes that response will come from people, sometimes from the system itself, and ideally each learns from what happened. That is what I call adaptive governance, and the proposition behind it is very different to the conformity basis of static governance. It’s that you achieve safety and security by building the adaptive capacity to sense and respond to change within explicit boundaries.

The term is not mine and it’s not new, with a history in environmental management and other fields that deal with complex systems that nobody can fully specify or bound.4 5

Static governance builds conformity. It tries to find out whether the rule was followed, if the assessment was completed, or if the control was in place on the day that someone happened to look. Conformity is measurable and its easy to audit, so it’s hardly surpising that organizations reach for it. Without doubt, it can be useful. But an organization can have every policy written down, every assessment clean and every assurance report stamped, but still have very little ability to notice what is happening in front of it. It can still lack the ability to respond rapidly or contain a threat. Even though they can point to policies, certifications, audits, checklists and tests, they can nevertheless lack the ability to make any meaningful assertion of safety, security or even of operation within the law.

Adaptive governance on the other hand builds adaptive capacity. That's the ability of a system or an organization to change how it operates in response to conditions, disturbances or events. That response allows it to continue to achieve its purposes within performance and safety boundaries.

Its something you build. You can’t just assert it exists, and its built in people as much as it is in software.

In people, adaptive capacity shows up in their ability to see conditions are changing, to interpret what that change means, and take action so that performance stays safe and effective regardless. It contrasts with competence, which is the ability to perform under conditions that you prepared for. Adaptive capacity matters when the conditions are different, allowing you to maybe spot a weak signal before it becomes an obvious failure, find a way around a procedure, or judge a trade-off. This adaptability is why treating people as a source of fault that needs to be minimized is wrong.

People are usually the part of the system that acts on a weak signal to prevent an issue becoming an accident. They have gut feel, an intuitive ability to see from experience that something is wrong.

In software, its about instrumenting the system so that it becomes possible to monitor and make sense of live behavior rather than an evaluation performed once before launch. Software can also hold limits at runtime, with intervention engineered into a control rather than a human response. These systems can act faster than a review cycle, so some of that response has to fire without waiting for approval of a person.

But adaptive capacity in an organization is always finite. It gets used up and it decays if its not being maintained. The US Marine Corps famously illustrate this, making adaptive capacity central to their operating method, constantly refreshing their capacity with training, exercises and education. But when adaptive capacity fades, an organization can be left holding all the artifacts of static governance and very little adaptive capacity left to respond.

The differences between static and adaptive governance become more apparent when you line each up against a set of relevant factors, as in the diagram below.

Static governance starts with conformity, but with that comes brittleness. Adaptive governance increases diversity, but with that comes capacity for resilience, flexibility and innovative response.

I know that nobody operates purely in the left column of static governance. But it’s the things in the left column that get asked for. Regulators, boards, customers and auditors all want evidence of it, so it is the column that gets funded and staffed. Its the work of the ‘business of compliance’ and has an unfortunate tendency to devolve into theatre and facade. Very little makes the same demand of the right column, which is why it’s usually the one missing.

And the work of shifting from static to adaptive governance is about much more than simply increasing automation, monitoring and feedback. Automation makes an existing approach faster, but it doesn’t change what is valued.

The harder shift comes from reducing our confidence in conformity, and instead building adaptive capacity. That means developing in people the ability to recognise when circumstances no longer fit with their plan, exercise their judgement and finding a safe way to go forward. Instead of maximizing control, adaptive governance is about encouraging choice within clear boundaries. Variation isn’t a departure to be corrected. It can be how we keep things working when the prescribed approach is no longer adequate. The discipline lies in holding the commitments and limits that matter while enabling people and systems to change how they meet them.

Adaptive governance is not one idea

Keeping complex systems safe under changing conditions is a problem with a history thats a lot longer than AI. Systems safety, resilience engineering, human factors, and organizational leadership give us ways to understand how failures emerge, how people adapt, and what allows them to act before conditions become dangerous. They are lessons learnt from disasters and high-stakes industries. These fields don’t give us a ready-made answer for AI, but they can give us a foundation for asking better questions about what its governance requires. I’ve drawn from key safety researchers throughout my professional career, and you’ll see them surface throughout these articles:

• Nancy Leveson on emergence6. Accidents emerge from unsafe interactions between parts that each worked as designed. Safety is a property of the whole system, it cannot be examined in parts.

• Jens Rasmussen on drift7. Work drifts to the edge of safety through many individually sensible choices. Manage the margin, not the position.

• David Woods on capacity8. The capacity to absorb surprise is finite and decays unless it is fed. The only question is whether you stretch or shatter at the edge.

• Erik Hollnagel on adaptation9. Things go right and wrong for the same reason. Study the ten thousand successes, not just the one failure.

• Richard Cook on practice10. Complex systems run in a degraded state more or less permanently. Safety is produced continuously by the people operating them.

• Karl Weick on mindfulness11. Reliability is collective mindfulness, earned continuously by heeding weak signals and deferring to expertise over rank.

• Sidney Dekker on trust12. Punishing error drives it underground. Safety depends on people reporting the mistakes you need to hear about.

• Ronald Heifetz on leadership. Distinguishing technical from adaptive problems, and applying the appropriate response to each is the real work of leadership.13

Their ideas run through every article that follows, often illustrated with disasters or positive events that demonstrate the payoff from adaptive capacity.

What the articles will cover

So it’s a long road, but we have a lot to cover. The plan is for ten parts, in total about 100 articles, that progress from understanding the problem to building the response in full. I don’t know how long this will take, but almost 30 of the articles are mostly written, so I hope it will be 6-12 months. Feedback and events might well change the route, but thats at least the shape of where I intend to go.

Parts I to III are about the problem. I go into what we’re actually trying to govern now, and the characteristics that make governance of agentic AI systems in use so challenging. Then through some stories of disasters and seeming miracles, I explore what some other fields have already learned about failure, adaptation, and keeping dangerous systems safe. We finish looking at how it happens that policies, assessments, certificates and human oversight can all be in place without delivering any meaningful protection.

Moving on to Parts IV to VIII, I start working through what it takes to build adaptive governance. First of all, how a governing system is assembled from mechanisms, connections and culture, so that information reaches decisions and authority that makes action possible. Then we go on to how it works in practice, bounding and constraining the behaviours of a complex agentic system, examining how signals in agent traces work. We’ll explore what evidence could justify relying on an adaptive governance approach.

In Part IX, I’ll look through the permissive boundaries of an AI system, going beyond your own walls, to suppliers, dependencies and responsibilities that cross organizational boundaries, including problems no organization can resolve alone. Finally, Part X brings it back inside, asking where a practitioner can begin in an organization that already has the machinery of boards, budgets, policies and audits.

I’m aiming for a destination that’s practical. I’m very aware that anything worthwhile has to help people working inside organizations that can’t simply start again with a clean sheet.

What I hope from you

So I’m thinking in the open, at times exploring and uncertain. It’s my firmest belief that the way of governance we have inherited from stable, predictable, complicated systems doesn’t work now and won’t work in the future. And I’m certain that there are lessons from research and safety practices in other industries that are part of the solution. I call that adaptive governance, but the precise shape of it, the concepts and terminology, the questions and argument are still unsettled. I hope in the writing of this book, and sharing of these articles, I can settle some of them.

The greatest gift I receive in teaching and working with clients is feedback, and my hope in publishing these articles is to elicit precisely that feedback.

Now you don’t need AI governance in your job title. You might build systems, you might lead a team, assess risk, run audits, or decide what your organization adopts. Possibly you have a background in safety from a different field, with insights and ideas that may prove relevant. Or you might simply be uneasy about institutions handing consequential work to AI systems that they don’t adequately understand. I very much welcome your feedback from wherever you come.

And I’ll probably ask at times though that you stay with the questions that don’t resolve neatly into an action item yet. And sometimes, I simply won’t have an answer and need to take time to research and consider. Or I’ll be wrong.

But I do want to be clear from the outset on one particular thing, so as not to be mistaken. I believe in good law, good regulation and good standards. In my mind, adaptive governance is not a licence to change the rules whenever they become inconvenient. I’ll be honest and direct, and where I have data, I’ll present that in support or against. It may not be the popular opinion. But when I characterize the EU AI Act, or ISO/IEC42001 and some others as static governance, please don’t mistake that for being either perjorative or dismissive. I rather simply hold them in contrast to other approaches that I believe offer more support for adaptive capacity.

Thank you for reading, and thank you in advance for your feedback, your questions, and your time. I believe writing these articles is the most important work I can do at this time.

So enough of the preamble. Time to get going. In the first article, I’ll dissect the events of the OpenAI/HuggingFace agent escape to examine those four characteristics of AI systems in use: complexity, dynamism, unboundedness, and autonomy.

It’s the first of roughly a hundred articles, and I welcome you on the journey.

Leave a comment

Share