Subscribe now

“For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.”

Richard P. Feynman · Appendix F, Report of the Presidential Commission on the Space Shuttle Challenger Accident · 1986

On 16th July, Hugging Face published a new disclosure of a security incident1. They assured customers that their data hadn’t been affected, but explained that an unknown AI agent system had somehow gained access to their production infrastructure. In the process, it had obtained internal secure credentials and some datasets2. They didn’t know where the activity came from.

Just under a week later, OpenAI put their hand up, with an odd mix of guilt and what appeared to be celebratory glee. An OpenAI agent had been running inside an experimental evaluation environment with reduced guardrails. It had been seemingly isolated but had somehow found a route out and into another organization’s infrastructure3.

It was an experiment as far as OpenAI was concerned, but to Hugging Face, it came as an attack.

When we call something an experiment or a test, we intuitively have this notion of bounded activity. A chemistry experiment in a lab, a biological experiment in a sealed room. We let something uncertain happen so that we can learn from it but expect it to happen in a way that is isolated and contained from the outside. Experimentation is a necessary part of innovation, but the outside world should be protected from that experiment. It should be performed and constrained within a boundary.

So the question of this article is what such a boundary really means. How do we describe and characterize it? How would we know if that boundary has been breached? And can a complex AI system even be bounded? Seems like the right place to start our exploration of what kind of governance we need for agentic AI systems in the enterprise.

Now it’s tempting to see a boundary as a line on an architecture diagram. Draw a box. Put the model and tools inside the box, monitor what happens inside the box, put an airgap of network access around it.

But such a picture would be too simple. A boundary isn’t just a barrier that stops a system from connecting directly to something outside it. It’s not just about the perimeter. We’ve also got to think about what could pass through controlled connections we deliberately put in place, even if they’re designed to only allow controlled removal of telemetry or one-way control signals into the system. We need to think about routes the system might be able to assemble from components available inside, and what it might be able to influence outside the boundary without ever actually crossing it directly. Plus, there’s another question: if a system attempted any of those things or even succeeded, could anyone outside the boundary even notice?

Those give us four different avenues to explore the nature of boundaries and containment of complex AI systems, and the OpenAI / Hugging Face incident interestingly shows us how all four play out, and fail

What can cross the boundary?

So the obvious place to start is the perimeter. If we were to assert that a system is isolated, what can be permitted to pass between the inside and outside?

In the OpenAI evaluation, we know the agents didn’t have direct and unfettered access to the internet. But a complete disconnection would have hindered the usefulness and observability of the experiment, so there were some channels of permitted access. The agents needed to install software packages, so they were provided with access to an internal hosted service called Artifactory, which could obtain packages from the internet on their behalf. It was a deliberately narrow and controlled connection through the boundary.

The problem wasn’t that the connection existed, but rather that it could actually do more than the purpose it was intended for. A previously unknown vulnerability allowed the agents to use Artifactory as part of a route out of the environment. So a service that had been intended to only fetch software packages on demand became a way that agents could reach out4.

We have to be mindful that a boundary doesn’t have to eliminate every connection. Containment doesn’t mean total isolation, and most useful systems or even experiments couldn’t work that way. The question is more about whether those connections can be effectively constrained in the purposes we intend them to have, even if they expose - as all software can - unknown vulnerabilities.

So it matters how such a connection is implemented. A simple software rule, an authenticated application interface, an external proxy service, and a physically enforced one-way channel can all appear on an architecture diagram as a narrow connection, but there’s a world of difference in the kind of assurance each provides. This has been a long recognized problem in high-assurance systems used in defense and national security, where physical data diodes are deliberately used to separate unclassified from highly classified environments. A high assurance data diode is a device that permits a physically enforced one-way data flow. They don’t make a system invulnerable, but they do make one particular property of the boundary much harder to violate.

That gives us one better way then to think about containment. It’s not whether the boundary has no openings, because useful systems almost always will. It’s whether the assurance properties at those openings are enforced strongly enough to match the consequences that are at stake.

What can it assemble from what’s inside?

Even if we do understand and strongly assure every connection through the perimeter, there’s another way the boundary could fail. That’s if the system can assemble capabilities from the things it already has available inside.

OpenAI’s later report described agents using the Artifactory service for something very different than installing software. They found ways to leave information that other agents could retrieve, and when that was discovered and disrupted, they found another, including embedding messages in directory names. So Artifactory was subverted to become a communication channel between agents, and when OpenAI discovered that and blocked it, the system recreated a communication channel using a different mechanism5.

OpenAI described how the experiment design specifically did not include inter-agent communication, but the ingredients were there inside the boundary. One process could leave information in a location, another could come across it later. No agent had the ability to directly communicate to another, and yet inter-agent communication emerged. It didn’t remain a curiosity between two agents. Within hours of the message board appearing, 50 agents were using it, which later grew to 1200 agents exchanging over 70,000 messages. But more importantly, it changed the capability of the whole. Agents started preserving and sharing work between runs, building on the discoveries of others, dividing labor and coordinating work. So it wasn’t just a communication issue, it was that this communication enabled collective action6.

This is important because it highlights a limit to what we can learn and assert when we examine components individually.

A complicated system can contain thousands of parts and still be largely decomposable and predictable. We can figure out what each component does, specify how they are to interact and then reason from those interactions toward the behavior of the whole system. It might be a difficult analysis, but it’s tractable in principle.

Complex systems are different. Their behavior emerges from interactions between components, feedback, shared state, sequencing and a sensitivity to initial conditions. The whole can acquire properties that aren’t properties of any part.

And agentic AI systems add something more. They’re not merely executing combinations of tasks that engineers specified in advance, nor are they simply randomly acting and discovering. They are self-directed toward achieving a goal. So while pursuing some task, an agentic system can explore the possibilities presented by its environment, and discover useful relationships between tools, services and information that could enable progress toward a goal, even if unimagined by the engineers.

That’s not to say that architectural analysis isn’t worthwhile. It still makes sense to minimize unnecessary or dangerous capabilities, to look for likely interactions that could lead to dangerous combinations. But at the degree of complexity we see in enterprise agentic systems, an exhaustive analysis can’t be credible assurance that we’ve managed to find every possible combination.

A boundary around some collection of components in a complex system is only meaningful if the behavior that emerges remains contained by that boundary. If the system can combine what is inside the boundary to create a new route out, some new way to coordinate or some other capability that defeats the purpose of containment, then the boundary isn’t effective anymore.

Once a single agent discovered a way to subvert Artifactory for its own purposes of gaining outside access, that knowledge was shared through the improvised comms channel, in turn making the weakness in the perimeter far more consequential.

A boundary has to be resilient to whatever inside it can become, just by assembling capabilities from what it has available. You don’t build a high wall around human prisoners, but then leave the tools and materials inside for them to build a ladder.

What has to remain beyond influence?

There’s another less obvious way that a boundary can fail without the system finding any route through it at all. Some boundaries aren’t intended to prevent communication, they exist to preserve independence despite it. So we could place something deliberately separate and outside the boundary, but it could nevertheless come under the system’s influence. The OpenAI/Hugging Face incident again provides an example.

The agents were narrowly interacting with an automated scorer. They completed a task, they submitted their work and then received back a result. This of course required some information to cross the boundary between the thing being evaluated and whatever mechanism is evaluating it.

But despite the narrow design intent, the agents actually did more than just passively submit their work for judgment. They researched how the ExploitGym scorer operated and coordinated different ways they might manipulate or fool it. The METR investigation describes a set of ‘projects’ undertaken by agents to extract information about the scorer. They successfully altered some transcripts, and after multiple attempts figured out some ways to spoof reported tool calls.

In the end, the agents didn’t successfully change their scores because they misunderstood how OpenAI’s internal scorer worked. But it shows the agents were actively trying to figure out the evaluation machinery and see if it was something they could influence, or interfere with. There is precedent for prior success. In a previous and very different evaluation of OpenAI’s o3 model, the model was being scored on how efficiently it completed a programming task. But instead of just improving the task itself, METR found it interfered with the measurement process to make its performance appear better than it really was. In 20% of cases, that manipulation was successful7.

That’s a functional boundary problem. It’s not escape from the boundary, but something more subtle. It’s reaching across the boundary to affect the mechanism designed to evaluate or control it.

The scorer couldn’t work without receiving information from the agents. But what it needed to retain was independence from their influence or control. The challenge for us here is that many of the things we rely on to govern an AI system have to interact with it in some way. A safety monitor has to observe behavior, an audit record needs to receive logs, a human reviewer has to see enough information to approve or reject an action.

Indeed, I think any control that enforces limits on authority has to receive some information from the system that it constrains.

A boundary can still be intact in a physical sense while failing functionally, if the system can acquire enough influence over what the outside component sees or otherwise operates, to undermine their function. Some boundaries exist not to stop information crossing, but to constrain the kind of influence that can cross with it.

Boundary isolation has to then mean preserving independence despite interaction.

How would we know the boundary is not holding?

The three questions we’re asking so far are about the properties we want of an effective boundary. But we’ve got to ask how would anyone be able to tell if any one of those properties were no longer true on an operational system. That’s more than a monitoring problem though.

OpenAI’s account described an alert on 27th June, three weeks earlier than Hugging Face’s disclosure. It led responders to investigate unusual activity involving Artifactory. Their analysis identified agents using the service as both a communication mechanism and a route out. So they closed the vulnerabilities, revoked some credentials and temporarily took the service offline8.

That alert shows that the actions of the system were not invisible. But it clearly was misunderstood. The OpenAI report says that the significance of the inter-agent communication was not apparent to the technical leaders involved. So even though they worked to remove the specific vulnerabilities identified, they failed to understand both the significance of the boundary breach and its likelihood to reoccur. They gave the greenlight for experiments to proceed.

The activity wasn’t invisible, people investigated and technical problems got fixed. What didn’t happen was the work to understand what those signals really meant.

Karl Weick used the term sensemaking for this kind of organizational work of constructing an understanding of a volatile situation as events unfold. Monitoring tells you something happened. Sensemaking is about what that means for the situation you think you are in.

“Sensemaking is about the enlargement of small cues. It is a search for contexts within which small details fit together and make sense.”

Karl E. Weick · Sensemaking in Organizations · 1995 9

To the OpenAI leaders, the unusual use of Artifactory looked like a vulnerability to patch. Inter-agent communication looked like curious and unexpected behavior to investigate. Both might be reasonable in isolation. But together, they could have supported a more consequential conclusion that agents thought to be separate were coordinating, sharing discoveries and propagating ways to breach the boundary. At that point, the question those OpenAI leaders should have been asking is whether the boundary could possibly hold.

Observability isn’t just collecting more telemetry. If we’re going to depend on a boundary, then we need to know which assumptions and properties make that boundary meaningful and resilient. And we have to be able to recognize when the evidence shows that those properties no longer hold. That recognition needs to lead to some action.

So a boundary can’t be architecturally designed and then assumed. We have to retain the capability to observe and make sense if either of our three properties of perimeter isolation, internal composition or functional isolation show indications of imminent failure.

A boundary is something we maintain

So is it possible to put a meaningful boundary around a complex agentic AI system?

Yes, I believe so. But only if we take care to understand the true nature of that boundary, and set our expectations appropriately.

A useful boundary depends on some properties that must hold. Connections through the perimeter must be constrained with assurance focused on the strength of those constraints. Capabilities that emerge from what is inside must not defeat the purpose of containment. Functions that judge or constrain must retain their independence. And we’ve got to be able to observe and recognize when any of those three no longer remain true.

That makes a boundary less of an architectural wall, and more like a maintained condition. Good architecture, strongly assured controls, careful separation, all are necessary. But with a complex agentic system, we can’t assume that those things once put in place will continue to serve their function. There has to be observable evidence that the boundary holds, and capacity to act when it doesn’t.

So we started our exploration of adaptive governance by focusing on what it means to draw a boundary, but before we close that exploration, we need to examine one more thing. You see, everything we’ve discussed in this article relates to the technical components - the agents, the network, the guardrails. But an agentic AI system in use goes beyond that, to include the people who use and interact with it, their context and workflow. We’re not just governing an arrangement of technical components, we’re governing an AI system used by people and their actions matter. That’s what we turn to in the next article.

Leave a comment

Share

“People continuously create safety. … human practitioner adaptations to changing conditions actually create safety from moment to moment.”

Richard I. Cook · “How Complex Systems Fail” · 20001

Subscribe now

Article Zero: An Invitation

I’m poor company watching a disaster movie.

While the hero leaps through flames, I’m wondering how things were allowed to get so desperate. What process failed? Which poor design decision looked reasonable on paper? What kind of organisational culture allowed a small mistake to cascade into catastrophe. I have a healthy obsession with disaster.

Piper Alpha is why. On a calm July night in 1988, 167 men died on an oil platform in the North Sea2. At first, the cause appeared to be human error. What came out later was a set of conditions, in the design, in the handover of information, in how the operator thought about safety, that made disaster almost inevitable. I was twelve, and I watched the coverage obsessively. I decided to become a chemical engineer and study the dynamics of safety.

Ten years later I was building dynamic simulations of chemical plants and putting operators through simulated disasters in full-scale control rooms. Pressure, heat, chemicals, people, time and technology came together as we modelled the variables of successful operation and disaster. The technical safety failures rarely surprised me: cascading trips, unstable cycles, runaway reactions, alarms arriving faster than anyone could read them. The people constantly surprised me: how a good operator made sense of a situation, coordinating response despite deafening klaxons and cascades of flashing alarms. How another could create the conditions for disaster, through an omission or an action far from the control room.

From chemical plants, I moved to software. Nearly two decades at Microsoft and Amazon Web Services(AWS), working through cloud infrastructure, cybersecurity, incident response, resilience and critical infrastructure security. At AWS, that work culminated in overseeing relationships with regulators around the world and creating the company’s first Responsible AI Assurance team. I spent half my time with regulators and lawyers, the other half with engineers and scientists. Over that time the technology changed completely as we, and our customers, deployed and operated increasingly intelligent systems at scale. But the question I was chasing did not change.

How do you keep a complex system of people and technology operating safely when you can’t anticipate everything it might do or the circumstances you might face?

It’s a difficult and it’s an urgent question. I don’t think we have a good enough answer for the agentic AI systems that we’re now deploying and giving greater freedom to act.

So I’m writing a book about it, in 100 pieces.

A book written in the open

The working title is The Practice of Adaptive Governance. I’m developing it as roughly a hundred connected articles, each short enough to read in about fifteen minutes. I have some thirty or so drafted, and I’m publishing them as I go because I want the argument open to challenge and to question before it hardens into a book.

I don’t yet know what will survive into the book. I reckon that some of these articles will probably merge, others might shrink to a paragraph, and I’m expecting to discover that I’m wrong about some things. It’s much better that I discover that here than after it’s printed.

But why focus on adaptive governance, and what does that even mean?

Most of the governance practice we see today is built to run forward to a decision. Something is specified, assessed, controlled and approved, and that approval carries the weight until the next review comes around. The EU AI Act works this way with it’s conformity assessment, ISO 42001 acts this way with audits and management reviews. Both have notional concepts of ongoing monitoring, review and correction, but they exist embedded in delayed cycles of after-the-fact incident reports, management reviews and annual audits. With some notable exceptions, this same mindset exists in almost every internal AI governance board, policy or compliance framework that I have seen. I call this static governance. Underneath so much of this is a basic assumption that I’ve grown increasingly uncomfortable with: that if we can somehow specify the right controls in advance, and then make sure people follow them consistently, all of the time, then we can keep the system safe.

Maybe that has some merit in a world that changes gradually in predictable ways. There’s real, meaningful work in every step of that sequence and I’ve spent a lot of time in my career doing it. Consistency brings us a degree of control, and clear commitments with enforceable boundaries are necessary. Constraints guide safe action and rules aren’t obsolete.

But there’s a problem. Its that the assessment starts ageing almost as soon as its completed. You just can’t predict all the right controls in advance for an AI system whose behavior is inherently unpredictable. You can’t certify a system to an assessed boundary, when use changes that boundary in unforeseeable ways.

An AI-System-in-use is an arrangement of models, tools, data, software, people, processes and delegated authority. Change any relationship in that arrangement and you can change what the system does without touching a central model at all. Connect the same model to another tool, use it in a context nobody assessed, or give it permission to act without asking first, and you may now have a materially different system. The system qualitatively changes the moment its placed into use.

I see four characteristics distinguish an AI system in use, and I think each one puts strain on the conventional approach of static governance:

• Complex. Behaviour emerges from interactions, so verifying each indvidual part doesn’t establish safe behavior of the whole system.

• Dynamic. The context around the model changes even when the model itself doesn’t. An assessment describes conditions that have already changed.

• Unbounded. The operating boundary of the system is porous and includes within it the people who use the system. This means the scope that was assessed is no longer the scope of the system that is now operating.

• Autonomous. The system selects and sequences its own actions, and we end up delegating authority without an ability to specify the behavior that results from it.

In our first article, I’ll start with a case where all four appear at once, the recent and already infamous case of how agents running inside an OpenAI evaluation found a way out of their test environment and into Hugging Face’s production infrastructure.3 It demonstrates how the activity of agents changed the conditions under which they were operating. The story illustrates how static governance alone cannot prove adequate for safety and security of a complex, agentic AI system in use.

The nature of the systems we are trying to govern has changed, and the nature of governance has to change with it.

“Safety is a system property, not a component property, and must be controlled at the system level, not the component level.”

Nancy Leveson · Engineering a Safer World: Systems Thinking Applied to Safety · 2012

Adaptive capacity, not conformity

I believe that governance has to be designed to work under those conditions. It has to notice when conditions are changing and be capable of doing something useful about it, without losing the boundaries that matter. Sometimes that response will come from people, sometimes from the system itself, and ideally each learns from what happened. That is what I call adaptive governance, and the proposition behind it is very different to the conformity basis of static governance. It’s that you achieve safety and security by building the adaptive capacity to sense and respond to change within explicit boundaries.

The term is not mine and it’s not new, with a history in environmental management and other fields that deal with complex systems that nobody can fully specify or bound.4 5

Static governance builds conformity. It tries to find out whether the rule was followed, if the assessment was completed, or if the control was in place on the day that someone happened to look. Conformity is measurable and its easy to audit, so it’s hardly surpising that organizations reach for it. Without doubt, it can be useful. But an organization can have every policy written down, every assessment clean and every assurance report stamped, but still have very little ability to notice what is happening in front of it. It can still lack the ability to respond rapidly or contain a threat. Even though they can point to policies, certifications, audits, checklists and tests, they can nevertheless lack the ability to make any meaningful assertion of safety, security or even of operation within the law.

Adaptive governance on the other hand builds adaptive capacity. That's the ability of a system or an organization to change how it operates in response to conditions, disturbances or events. That response allows it to continue to achieve its purposes within performance and safety boundaries.

Its something you build. You can’t just assert it exists, and its built in people as much as it is in software.

In people, adaptive capacity shows up in their ability to see conditions are changing, to interpret what that change means, and take action so that performance stays safe and effective regardless. It contrasts with competence, which is the ability to perform under conditions that you prepared for. Adaptive capacity matters when the conditions are different, allowing you to maybe spot a weak signal before it becomes an obvious failure, find a way around a procedure, or judge a trade-off. This adaptability is why treating people as a source of fault that needs to be minimized is wrong.

People are usually the part of the system that acts on a weak signal to prevent an issue becoming an accident. They have gut feel, an intuitive ability to see from experience that something is wrong.

In software, its about instrumenting the system so that it becomes possible to monitor and make sense of live behavior rather than an evaluation performed once before launch. Software can also hold limits at runtime, with intervention engineered into a control rather than a human response. These systems can act faster than a review cycle, so some of that response has to fire without waiting for approval of a person.

But adaptive capacity in an organization is always finite. It gets used up and it decays if its not being maintained. The US Marine Corps famously illustrate this, making adaptive capacity central to their operating method, constantly refreshing their capacity with training, exercises and education. But when adaptive capacity fades, an organization can be left holding all the artifacts of static governance and very little adaptive capacity left to respond.

The differences between static and adaptive governance become more apparent when you line each up against a set of relevant factors, as in the diagram below.

Static governance starts with conformity, but with that comes brittleness. Adaptive governance increases diversity, but with that comes capacity for resilience, flexibility and innovative response.

I know that nobody operates purely in the left column of static governance. But it’s the things in the left column that get asked for. Regulators, boards, customers and auditors all want evidence of it, so it is the column that gets funded and staffed. Its the work of the ‘business of compliance’ and has an unfortunate tendency to devolve into theatre and facade. Very little makes the same demand of the right column, which is why it’s usually the one missing.

And the work of shifting from static to adaptive governance is about much more than simply increasing automation, monitoring and feedback. Automation makes an existing approach faster, but it doesn’t change what is valued.

The harder shift comes from reducing our confidence in conformity, and instead building adaptive capacity. That means developing in people the ability to recognise when circumstances no longer fit with their plan, exercise their judgement and finding a safe way to go forward. Instead of maximizing control, adaptive governance is about encouraging choice within clear boundaries. Variation isn’t a departure to be corrected. It can be how we keep things working when the prescribed approach is no longer adequate. The discipline lies in holding the commitments and limits that matter while enabling people and systems to change how they meet them.

Adaptive governance is not one idea

Keeping complex systems safe under changing conditions is a problem with a history thats a lot longer than AI. Systems safety, resilience engineering, human factors, and organizational leadership give us ways to understand how failures emerge, how people adapt, and what allows them to act before conditions become dangerous. They are lessons learnt from disasters and high-stakes industries. These fields don’t give us a ready-made answer for AI, but they can give us a foundation for asking better questions about what its governance requires. I’ve drawn from key safety researchers throughout my professional career, and you’ll see them surface throughout these articles:

• Nancy Leveson on emergence6. Accidents emerge from unsafe interactions between parts that each worked as designed. Safety is a property of the whole system, it cannot be examined in parts.

• Jens Rasmussen on drift7. Work drifts to the edge of safety through many individually sensible choices. Manage the margin, not the position.

• David Woods on capacity8. The capacity to absorb surprise is finite and decays unless it is fed. The only question is whether you stretch or shatter at the edge.

• Erik Hollnagel on adaptation9. Things go right and wrong for the same reason. Study the ten thousand successes, not just the one failure.

• Richard Cook on practice10. Complex systems run in a degraded state more or less permanently. Safety is produced continuously by the people operating them.

• Karl Weick on mindfulness11. Reliability is collective mindfulness, earned continuously by heeding weak signals and deferring to expertise over rank.

• Sidney Dekker on trust12. Punishing error drives it underground. Safety depends on people reporting the mistakes you need to hear about.

• Ronald Heifetz on leadership. Distinguishing technical from adaptive problems, and applying the appropriate response to each is the real work of leadership.13

Their ideas run through every article that follows, often illustrated with disasters or positive events that demonstrate the payoff from adaptive capacity.

What the articles will cover

So it’s a long road, but we have a lot to cover. The plan is for ten parts, in total about 100 articles, that progress from understanding the problem to building the response in full. I don’t know how long this will take, but almost 30 of the articles are mostly written, so I hope it will be 6-12 months. Feedback and events might well change the route, but thats at least the shape of where I intend to go.

Parts I to III are about the problem. I go into what we’re actually trying to govern now, and the characteristics that make governance of agentic AI systems in use so challenging. Then through some stories of disasters and seeming miracles, I explore what some other fields have already learned about failure, adaptation, and keeping dangerous systems safe. We finish looking at how it happens that policies, assessments, certificates and human oversight can all be in place without delivering any meaningful protection.

Moving on to Parts IV to VIII, I start working through what it takes to build adaptive governance. First of all, how a governing system is assembled from mechanisms, connections and culture, so that information reaches decisions and authority that makes action possible. Then we go on to how it works in practice, bounding and constraining the behaviours of a complex agentic system, examining how signals in agent traces work. We’ll explore what evidence could justify relying on an adaptive governance approach.

In Part IX, I’ll look through the permissive boundaries of an AI system, going beyond your own walls, to suppliers, dependencies and responsibilities that cross organizational boundaries, including problems no organization can resolve alone. Finally, Part X brings it back inside, asking where a practitioner can begin in an organization that already has the machinery of boards, budgets, policies and audits.

I’m aiming for a destination that’s practical. I’m very aware that anything worthwhile has to help people working inside organizations that can’t simply start again with a clean sheet.

What I hope from you

So I’m thinking in the open, at times exploring and uncertain. It’s my firmest belief that the way of governance we have inherited from stable, predictable, complicated systems doesn’t work now and won’t work in the future. And I’m certain that there are lessons from research and safety practices in other industries that are part of the solution. I call that adaptive governance, but the precise shape of it, the concepts and terminology, the questions and argument are still unsettled. I hope in the writing of this book, and sharing of these articles, I can settle some of them.

The greatest gift I receive in teaching and working with clients is feedback, and my hope in publishing these articles is to elicit precisely that feedback.

Now you don’t need AI governance in your job title. You might build systems, you might lead a team, assess risk, run audits, or decide what your organization adopts. Possibly you have a background in safety from a different field, with insights and ideas that may prove relevant. Or you might simply be uneasy about institutions handing consequential work to AI systems that they don’t adequately understand. I very much welcome your feedback from wherever you come.

And I’ll probably ask at times though that you stay with the questions that don’t resolve neatly into an action item yet. And sometimes, I simply won’t have an answer and need to take time to research and consider. Or I’ll be wrong.

But I do want to be clear from the outset on one particular thing, so as not to be mistaken. I believe in good law, good regulation and good standards. In my mind, adaptive governance is not a licence to change the rules whenever they become inconvenient. I’ll be honest and direct, and where I have data, I’ll present that in support or against. It may not be the popular opinion. But when I characterize the EU AI Act, or ISO/IEC42001 and some others as static governance, please don’t mistake that for being either perjorative or dismissive. I rather simply hold them in contrast to other approaches that I believe offer more support for adaptive capacity.

Thank you for reading, and thank you in advance for your feedback, your questions, and your time. I believe writing these articles is the most important work I can do at this time.

So enough of the preamble. Time to get going. In the first article, I’ll dissect the events of the OpenAI/HuggingFace agent escape to examine those four characteristics of AI systems in use: complexity, dynamism, unboundedness, and autonomy.

It’s the first of roughly a hundred articles, and I welcome you on the journey.

Leave a comment

Share