Article 1 in The Practice of Adaptive Governance series examines the properties of a boundary adequate to contain a complex AI system that continuously adapts in use.
James, this is important work and raises some very good questions. I’d like to challenge—or perhaps extend—two aspects of the argument.
First, the question of boundaries in complex systems. In their paper The Information Theory of Individuality, David Krakauer and colleagues propose that individuality is not necessarily determined by physical boundaries or predefined membership. Instead, they ask whether an aggregate preserves and propagates information through time. This suggests a prior question to the one you pose: how do we establish where the system’s boundary actually lies, rather than assuming that we already know what constitutes the system?
The OpenAI example is interesting through this lens. Initially, the agents appear to have been treated as independent entities. One agent then discovered that Artifactory afforded a means of communication. Other agents exploited the same affordance. Eventually, agents were preserving and sharing information across runs, building on one another’s discoveries, dividing labour and coordinating their activity.
At what point, then, should we stop thinking of these as separate agents and begin thinking of them as a larger computational system?
ITI offers an interesting way of approaching that question. Communication alone would not establish that a new individual had emerged. We would need to ask whether the collective was preserving and propagating information through time in ways that made its past predictive of its future, and whether those informational relationships distinguished the collective from its constituent agents and environment. If so, its effective boundary might be quite different from the architectural boundary assumed by the experiment.
That leads to my second question, concerning emergence.
You describe the agents’ collective behaviour as emergent. I wonder whether the evidence gets us quite that far. An agent discovering an unintended use for Artifactory is surprising, but surprise is not sufficient evidence of emergence. Nor is the subsequent exchange of information between agents necessarily emergent: it may simply represent individual agents exercising existing capabilities through a newly discovered communication mechanism.
The later coordination is more interesting. Once agents preserve discoveries, build upon the work of other agents and divide labour, we potentially have properties of the collective that are not usefully explained at the level of individual agents. There seems at least two possible interpretations. One is that individual agents are exercising their existing capabilities through a newly available communication network. The other is that their interactions have produced stable higher-level properties that are better described at the level of the collective itself.
Only the latter, it seems to me, gives us strong grounds for invoking emergence in the complexity-science sense.
So I wonder whether the deeper governance problem exposed by this case is not simply how we maintain a boundary around an agentic system, but how we know what system we are trying to bound in the first place. If interacting agents can form higher-order informational individuals, the appropriate unit of containment may itself change as the system operates.
That seems both a fascinating theoretical problem and a rather uncomfortable practical one for AI governance.
A rich, thoughtful piece. I hadn’t realized “sensemaking” was even a word, yet it captures the real-world challenge of agentic AI perfectly. Thanks for sharing your insights.
Containment as a maintained condition rather than a wall is the right move. The OpenAI–Hugging Face incident fits the pattern we keep tracing at GlobalStack: the algorithm layer was open and working as designed, and the breach came through infrastructure instead. Pressure does not stay at the layer you designed for; it migrates to whichever layer has the least governance. A boundary that only exists where you drew it is a suggestion to everything above and below it. (See 'The Warning Shot: When Proprietary AI Attacks the Open Stack,' September 7.)
The Feynman epigraph fits better than it first appears. Diane Vaughan's study of the same launch found the failure wasn't a missing signal: engineers saw O-ring erosion flight after flight, and each flight that survived made the erosion look acceptable. She called it the normalisation of deviance. The 27 June alert reads the same way: understood as a vulnerability, patched, experiments resumed. Every fix that works makes the next anomaly easier to file as routine, which is how a boundary stops being maintained without anyone deciding to stop.
Very interesting James. In one podcast I listened to (Dwarkesh Patel talking to Noam Brown of OpenAI - see https://www.dwarkesh.com/p/noam-brown), the OpenAI researcher talks a lot about Chain-of-Thought monitoring and the current concerns with its use. It seems that CoT monitoring is part of the observability that OpenAI relied upon. From this podcast it also seems fragile. This would appear to challenge your belief that an effective boundary can be set on agentic AI systems. Would you agree?
Yes, OpenAI seemed to place heavy reliance on chain-of-thought review - they weren't monitoring it though, it was after after the fact review. There were other signals though and the one that triggered the alert appeared to be Artifactory credential use on an API. And I think more careful selection of signals can enable a more effective boundary.
You highlight a point that's important around the need for attention to the assurance level of an observable signal. CoT is generated by the model itself, and even without any deliberate spoofing or evasion, it's a very poor signal of what the model is actually performing. There's a lot of evidence now that CoT is unreliable, and so although it can perhaps be useful as a leading indicator, it's value for boundary monitoring itself is minimal.
There are other signals that can provide higher assurance though. Stepping up from CoT to tool traces, up from tool traces to server logs (eg. Artifactory server logs in this case), stepping up to credential use on external APIs, stepping up to network egress telemetry, stepping up to even remote system logs (ie. the logs on other potential target external systems). There's some promising work on honeytokens etc.
I do think there's an important distinction that we could make between monitoring the model and monitoring the boundary. CoT is a low assurance signal from monitoring the model - it says nothing reliable about the boundary. Network telemetry at the perimeter is a higher assurance signal about the boundary, it says very little about the model.
Thanks for the question Martin, I'll incorporate that point going forward.
"I do think there's an important distinction that we could make between monitoring the model and monitoring the boundary. " Good point. I think that's a distinction worth emphasising.
James, this is important work and raises some very good questions. I’d like to challenge—or perhaps extend—two aspects of the argument.
First, the question of boundaries in complex systems. In their paper The Information Theory of Individuality, David Krakauer and colleagues propose that individuality is not necessarily determined by physical boundaries or predefined membership. Instead, they ask whether an aggregate preserves and propagates information through time. This suggests a prior question to the one you pose: how do we establish where the system’s boundary actually lies, rather than assuming that we already know what constitutes the system?
The OpenAI example is interesting through this lens. Initially, the agents appear to have been treated as independent entities. One agent then discovered that Artifactory afforded a means of communication. Other agents exploited the same affordance. Eventually, agents were preserving and sharing information across runs, building on one another’s discoveries, dividing labour and coordinating their activity.
At what point, then, should we stop thinking of these as separate agents and begin thinking of them as a larger computational system?
ITI offers an interesting way of approaching that question. Communication alone would not establish that a new individual had emerged. We would need to ask whether the collective was preserving and propagating information through time in ways that made its past predictive of its future, and whether those informational relationships distinguished the collective from its constituent agents and environment. If so, its effective boundary might be quite different from the architectural boundary assumed by the experiment.
That leads to my second question, concerning emergence.
You describe the agents’ collective behaviour as emergent. I wonder whether the evidence gets us quite that far. An agent discovering an unintended use for Artifactory is surprising, but surprise is not sufficient evidence of emergence. Nor is the subsequent exchange of information between agents necessarily emergent: it may simply represent individual agents exercising existing capabilities through a newly discovered communication mechanism.
The later coordination is more interesting. Once agents preserve discoveries, build upon the work of other agents and divide labour, we potentially have properties of the collective that are not usefully explained at the level of individual agents. There seems at least two possible interpretations. One is that individual agents are exercising their existing capabilities through a newly available communication network. The other is that their interactions have produced stable higher-level properties that are better described at the level of the collective itself.
Only the latter, it seems to me, gives us strong grounds for invoking emergence in the complexity-science sense.
So I wonder whether the deeper governance problem exposed by this case is not simply how we maintain a boundary around an agentic system, but how we know what system we are trying to bound in the first place. If interacting agents can form higher-order informational individuals, the appropriate unit of containment may itself change as the system operates.
That seems both a fascinating theoretical problem and a rather uncomfortable practical one for AI governance.
A rich, thoughtful piece. I hadn’t realized “sensemaking” was even a word, yet it captures the real-world challenge of agentic AI perfectly. Thanks for sharing your insights.
Containment as a maintained condition rather than a wall is the right move. The OpenAI–Hugging Face incident fits the pattern we keep tracing at GlobalStack: the algorithm layer was open and working as designed, and the breach came through infrastructure instead. Pressure does not stay at the layer you designed for; it migrates to whichever layer has the least governance. A boundary that only exists where you drew it is a suggestion to everything above and below it. (See 'The Warning Shot: When Proprietary AI Attacks the Open Stack,' September 7.)
The Feynman epigraph fits better than it first appears. Diane Vaughan's study of the same launch found the failure wasn't a missing signal: engineers saw O-ring erosion flight after flight, and each flight that survived made the erosion look acceptable. She called it the normalisation of deviance. The 27 June alert reads the same way: understood as a vulnerability, patched, experiments resumed. Every fix that works makes the next anomaly easier to file as routine, which is how a boundary stops being maintained without anyone deciding to stop.
Very interesting James. In one podcast I listened to (Dwarkesh Patel talking to Noam Brown of OpenAI - see https://www.dwarkesh.com/p/noam-brown), the OpenAI researcher talks a lot about Chain-of-Thought monitoring and the current concerns with its use. It seems that CoT monitoring is part of the observability that OpenAI relied upon. From this podcast it also seems fragile. This would appear to challenge your belief that an effective boundary can be set on agentic AI systems. Would you agree?
Yes, OpenAI seemed to place heavy reliance on chain-of-thought review - they weren't monitoring it though, it was after after the fact review. There were other signals though and the one that triggered the alert appeared to be Artifactory credential use on an API. And I think more careful selection of signals can enable a more effective boundary.
You highlight a point that's important around the need for attention to the assurance level of an observable signal. CoT is generated by the model itself, and even without any deliberate spoofing or evasion, it's a very poor signal of what the model is actually performing. There's a lot of evidence now that CoT is unreliable, and so although it can perhaps be useful as a leading indicator, it's value for boundary monitoring itself is minimal.
There are other signals that can provide higher assurance though. Stepping up from CoT to tool traces, up from tool traces to server logs (eg. Artifactory server logs in this case), stepping up to credential use on external APIs, stepping up to network egress telemetry, stepping up to even remote system logs (ie. the logs on other potential target external systems). There's some promising work on honeytokens etc.
I do think there's an important distinction that we could make between monitoring the model and monitoring the boundary. CoT is a low assurance signal from monitoring the model - it says nothing reliable about the boundary. Network telemetry at the perimeter is a higher assurance signal about the boundary, it says very little about the model.
Thanks for the question Martin, I'll incorporate that point going forward.
"I do think there's an important distinction that we could make between monitoring the model and monitoring the boundary. " Good point. I think that's a distinction worth emphasising.