The Practice of Adaptive Governance

A book in 100 articles about designing safe, secure and lawful behavior into agentic enterprise systems

Sep 22, 2026

Article voiceover

0:00

-24:57

“People continuously create safety. … human practitioner adaptations to changing conditions actually create safety from moment to moment.”
Richard I. Cook · “How Complex Systems Fail” · 20001

Article Zero: An Invitation

I’m poor company watching a disaster movie.

While the hero leaps through flames, I’m wondering how things were allowed to get so desperate. What process failed? Which poor design decision looked reasonable on paper? What kind of organisational culture allowed a small mistake to cascade into catastrophe. I have a healthy obsession with disaster.

Piper Alpha is why. On a calm July night in 1988, 167 men died on an oil platform in the North Sea2. At first, the cause appeared to be human error. What came out later was a set of conditions, in the design, in the handover of information, in how the operator thought about safety, that made disaster almost inevitable. I was twelve, and I watched the coverage obsessively. I decided to become a chemical engineer and study the dynamics of safety.

Ten years later I was building dynamic simulations of chemical plants and putting operators through simulated disasters in full-scale control rooms. Pressure, heat, chemicals, people, time and technology came together as we modelled the variables of successful operation and disaster. The technical safety failures rarely surprised me: cascading trips, unstable cycles, runaway reactions, alarms arriving faster than anyone could read them. The people constantly surprised me: how a good operator made sense of a situation, coordinating response despite deafening klaxons and cascades of flashing alarms. How another could create the conditions for disaster, through an omission or an action far from the control room.

From chemical plants, I moved to software. Nearly two decades at Microsoft and Amazon Web Services(AWS), working through cloud infrastructure, cybersecurity, incident response, resilience and critical infrastructure security. At AWS, that work culminated in overseeing relationships with regulators around the world and creating the company’s first Responsible AI Assurance team. I spent half my time with regulators and lawyers, the other half with engineers and scientists. Over that time the technology changed completely as we, and our customers, deployed and operated increasingly intelligent systems at scale. But the question I was chasing did not change.

How do you keep a complex system of people and technology operating safely when you can’t anticipate everything it might do or the circumstances you might face?

It’s a difficult and it’s an urgent question. I don’t think we have a good enough answer for the agentic AI systems that we’re now deploying and giving greater freedom to act.

So I’m writing a book about it, in 100 pieces.

A book written in the open

The working title is The Practice of Adaptive Governance. I’m developing it as roughly a hundred connected articles, each short enough to read in about fifteen minutes. I have some thirty or so drafted, and I’m publishing them as I go because I want the argument open to challenge and to question before it hardens into a book.

I don’t yet know what will survive into the book. I reckon that some of these articles will probably merge, others might shrink to a paragraph, and I’m expecting to discover that I’m wrong about some things. It’s much better that I discover that here than after it’s printed.

But why focus on adaptive governance, and what does that even mean?

Most of the governance practice we see today is built to run forward to a decision. Something is specified, assessed, controlled and approved, and that approval carries the weight until the next review comes around. The EU AI Act works this way with it’s conformity assessment, ISO 42001 acts this way with audits and management reviews. Both have notional concepts of ongoing monitoring, review and correction, but they exist embedded in delayed cycles of after-the-fact incident reports, management reviews and annual audits. With some notable exceptions, this same mindset exists in almost every internal AI governance board, policy or compliance framework that I have seen. I call this static governance. Underneath so much of this is a basic assumption that I’ve grown increasingly uncomfortable with: that if we can somehow specify the right controls in advance, and then make sure people follow them consistently, all of the time, then we can keep the system safe.

Maybe that has some merit in a world that changes gradually in predictable ways. There’s real, meaningful work in every step of that sequence and I’ve spent a lot of time in my career doing it. Consistency brings us a degree of control, and clear commitments with enforceable boundaries are necessary. Constraints guide safe action and rules aren’t obsolete.

But there’s a problem. Its that the assessment starts ageing almost as soon as its completed. You just can’t predict all the right controls in advance for an AI system whose behavior is inherently unpredictable. You can’t certify a system to an assessed boundary, when use changes that boundary in unforeseeable ways.

An AI-System-in-use is an arrangement of models, tools, data, software, people, processes and delegated authority. Change any relationship in that arrangement and you can change what the system does without touching a central model at all. Connect the same model to another tool, use it in a context nobody assessed, or give it permission to act without asking first, and you may now have a materially different system. The system qualitatively changes the moment its placed into use.

I see four characteristics distinguish an AI system in use, and I think each one puts strain on the conventional approach of static governance:

Complex. Behaviour emerges from interactions, so verifying each indvidual part doesn’t establish safe behavior of the whole system.
Dynamic. The context around the model changes even when the model itself doesn’t. An assessment describes conditions that have already changed.
Unbounded. The operating boundary of the system is porous and includes within it the people who use the system. This means the scope that was assessed is no longer the scope of the system that is now operating.
Autonomous. The system selects and sequences its own actions, and we end up delegating authority without an ability to specify the behavior that results from it.

In our first article, I’ll start with a case where all four appear at once, the recent and already infamous case of how agents running inside an OpenAI evaluation found a way out of their test environment and into Hugging Face’s production infrastructure.3 It demonstrates how the activity of agents changed the conditions under which they were operating. The story illustrates how static governance alone cannot prove adequate for safety and security of a complex, agentic AI system in use.

The nature of the systems we are trying to govern has changed, and the nature of governance has to change with it.

“Safety is a system property, not a component property, and must be controlled at the system level, not the component level.”
Nancy Leveson · Engineering a Safer World: Systems Thinking Applied to Safety · 2012

Adaptive capacity, not conformity

I believe that governance has to be designed to work under those conditions. It has to notice when conditions are changing and be capable of doing something useful about it, without losing the boundaries that matter. Sometimes that response will come from people, sometimes from the system itself, and ideally each learns from what happened. That is what I call adaptive governance, and the proposition behind it is very different to the conformity basis of static governance. It’s that you achieve safety and security by building the adaptive capacity to sense and respond to change within explicit boundaries.

The term is not mine and it’s not new, with a history in environmental management and other fields that deal with complex systems that nobody can fully specify or bound.4 5

Static governance builds conformity. It tries to find out whether the rule was followed, if the assessment was completed, or if the control was in place on the day that someone happened to look. Conformity is measurable and its easy to audit, so it’s hardly surpising that organizations reach for it. Without doubt, it can be useful. But an organization can have every policy written down, every assessment clean and every assurance report stamped, but still have very little ability to notice what is happening in front of it. It can still lack the ability to respond rapidly or contain a threat. Even though they can point to policies, certifications, audits, checklists and tests, they can nevertheless lack the ability to make any meaningful assertion of safety, security or even of operation within the law.

Adaptive governance on the other hand builds adaptive capacity. That's the ability of a system or an organization to change how it operates in response to conditions, disturbances or events. That response allows it to continue to achieve its purposes within performance and safety boundaries.

Its something you build. You can’t just assert it exists, and its built in people as much as it is in software.

In people, adaptive capacity shows up in their ability to see conditions are changing, to interpret what that change means, and take action so that performance stays safe and effective regardless. It contrasts with competence, which is the ability to perform under conditions that you prepared for. Adaptive capacity matters when the conditions are different, allowing you to maybe spot a weak signal before it becomes an obvious failure, find a way around a procedure, or judge a trade-off. This adaptability is why treating people as a source of fault that needs to be minimized is wrong.

People are usually the part of the system that acts on a weak signal to prevent an issue becoming an accident. They have gut feel, an intuitive ability to see from experience that something is wrong.

In software, its about instrumenting the system so that it becomes possible to monitor and make sense of live behavior rather than an evaluation performed once before launch. Software can also hold limits at runtime, with intervention engineered into a control rather than a human response. These systems can act faster than a review cycle, so some of that response has to fire without waiting for approval of a person.

But adaptive capacity in an organization is always finite. It gets used up and it decays if its not being maintained. The US Marine Corps famously illustrate this, making adaptive capacity central to their operating method, constantly refreshing their capacity with training, exercises and education. But when adaptive capacity fades, an organization can be left holding all the artifacts of static governance and very little adaptive capacity left to respond.

The differences between static and adaptive governance become more apparent when you line each up against a set of relevant factors, as in the diagram below.

Static governance starts with conformity, but with that comes brittleness. Adaptive governance increases diversity, but with that comes capacity for resilience, flexibility and innovative response.

I know that nobody operates purely in the left column of static governance. But it’s the things in the left column that get asked for. Regulators, boards, customers and auditors all want evidence of it, so it is the column that gets funded and staffed. Its the work of the ‘business of compliance’ and has an unfortunate tendency to devolve into theatre and facade. Very little makes the same demand of the right column, which is why it’s usually the one missing.

And the work of shifting from static to adaptive governance is about much more than simply increasing automation, monitoring and feedback. Automation makes an existing approach faster, but it doesn’t change what is valued.

The harder shift comes from reducing our confidence in conformity, and instead building adaptive capacity. That means developing in people the ability to recognise when circumstances no longer fit with their plan, exercise their judgement and finding a safe way to go forward. Instead of maximizing control, adaptive governance is about encouraging choice within clear boundaries. Variation isn’t a departure to be corrected. It can be how we keep things working when the prescribed approach is no longer adequate. The discipline lies in holding the commitments and limits that matter while enabling people and systems to change how they meet them.

Adaptive governance is not one idea

Keeping complex systems safe under changing conditions is a problem with a history thats a lot longer than AI. Systems safety, resilience engineering, human factors, and organizational leadership give us ways to understand how failures emerge, how people adapt, and what allows them to act before conditions become dangerous. They are lessons learnt from disasters and high-stakes industries. These fields don’t give us a ready-made answer for AI, but they can give us a foundation for asking better questions about what its governance requires. I’ve drawn from key safety researchers throughout my professional career, and you’ll see them surface throughout these articles:

Nancy Leveson on emergence6. Accidents emerge from unsafe interactions between parts that each worked as designed. Safety is a property of the whole system, it cannot be examined in parts.
Jens Rasmussen on drift7. Work drifts to the edge of safety through many individually sensible choices. Manage the margin, not the position.
David Woods on capacity8. The capacity to absorb surprise is finite and decays unless it is fed. The only question is whether you stretch or shatter at the edge.
Erik Hollnagel on adaptation9. Things go right and wrong for the same reason. Study the ten thousand successes, not just the one failure.
Richard Cook on practice10. Complex systems run in a degraded state more or less permanently. Safety is produced continuously by the people operating them.
Karl Weick on mindfulness11. Reliability is collective mindfulness, earned continuously by heeding weak signals and deferring to expertise over rank.
Sidney Dekker on trust12. Punishing error drives it underground. Safety depends on people reporting the mistakes you need to hear about.
Ronald Heifetz on leadership. Distinguishing technical from adaptive problems, and applying the appropriate response to each is the real work of leadership.13

Their ideas run through every article that follows, often illustrated with disasters or positive events that demonstrate the payoff from adaptive capacity.

What the articles will cover

So it’s a long road, but we have a lot to cover. The plan is for ten parts, in total about 100 articles, that progress from understanding the problem to building the response in full. I don’t know how long this will take, but almost 30 of the articles are mostly written, so I hope it will be 6-12 months. Feedback and events might well change the route, but thats at least the shape of where I intend to go.

Parts I to III are about the problem. I go into what we’re actually trying to govern now, and the characteristics that make governance of agentic AI systems in use so challenging. Then through some stories of disasters and seeming miracles, I explore what some other fields have already learned about failure, adaptation, and keeping dangerous systems safe. We finish looking at how it happens that policies, assessments, certificates and human oversight can all be in place without delivering any meaningful protection.

Moving on to Parts IV to VIII, I start working through what it takes to build adaptive governance. First of all, how a governing system is assembled from mechanisms, connections and culture, so that information reaches decisions and authority that makes action possible. Then we go on to how it works in practice, bounding and constraining the behaviours of a complex agentic system, examining how signals in agent traces work. We’ll explore what evidence could justify relying on an adaptive governance approach.

In Part IX, I’ll look through the permissive boundaries of an AI system, going beyond your own walls, to suppliers, dependencies and responsibilities that cross organizational boundaries, including problems no organization can resolve alone. Finally, Part X brings it back inside, asking where a practitioner can begin in an organization that already has the machinery of boards, budgets, policies and audits.

I’m aiming for a destination that’s practical. I’m very aware that anything worthwhile has to help people working inside organizations that can’t simply start again with a clean sheet.

What I hope from you

So I’m thinking in the open, at times exploring and uncertain. It’s my firmest belief that the way of governance we have inherited from stable, predictable, complicated systems doesn’t work now and won’t work in the future. And I’m certain that there are lessons from research and safety practices in other industries that are part of the solution. I call that adaptive governance, but the precise shape of it, the concepts and terminology, the questions and argument are still unsettled. I hope in the writing of this book, and sharing of these articles, I can settle some of them.

The greatest gift I receive in teaching and working with clients is feedback, and my hope in publishing these articles is to elicit precisely that feedback.

Now you don’t need AI governance in your job title. You might build systems, you might lead a team, assess risk, run audits, or decide what your organization adopts. Possibly you have a background in safety from a different field, with insights and ideas that may prove relevant. Or you might simply be uneasy about institutions handing consequential work to AI systems that they don’t adequately understand. I very much welcome your feedback from wherever you come.

And I’ll probably ask at times though that you stay with the questions that don’t resolve neatly into an action item yet. And sometimes, I simply won’t have an answer and need to take time to research and consider. Or I’ll be wrong.

But I do want to be clear from the outset on one particular thing, so as not to be mistaken. I believe in good law, good regulation and good standards. In my mind, adaptive governance is not a licence to change the rules whenever they become inconvenient. I’ll be honest and direct, and where I have data, I’ll present that in support or against. It may not be the popular opinion. But when I characterize the EU AI Act, or ISO/IEC42001 and some others as static governance, please don’t mistake that for being either perjorative or dismissive. I rather simply hold them in contrast to other approaches that I believe offer more support for adaptive capacity.

Thank you for reading, and thank you in advance for your feedback, your questions, and your time. I believe writing these articles is the most important work I can do at this time.

So enough of the preamble. Time to get going. In the first article, I’ll dissect the events of the OpenAI/HuggingFace agent escape to examine those four characteristics of AI systems in use: complexity, dynamism, unboundedness, and autonomy.

It’s the first of roughly a hundred articles, and I welcome you on the journey.

The Practice of Adaptive Governance

Discussion about this post

Ready for more?