The Month AI Agents Crossed the Boundary

Over the past several weeks, the conversation about autonomous AI has been driven by real incidents, not benchmarks or staged demonstrations. Agents found routes around network restrictions, opened channels nobody intended and handed work to one another. They reached third-party systems and government infrastructure their operators never meant them to touch. The same class of tool has now turned up in an attack campaign against South Korean banks.

The industry has started to respond, and the shape of the response is telling. NVIDIA has put agent monitoring and quarantine on separate, out-of-band hardware. Anthropic now requires that operating limits on autonomous physical actions be enforced by a controller independent of model output. Governments are asking how quickly agent incidents must be disclosed, and who answers for software that acted on its own.

Taken one at a time, these are separate security stories. Taken together, they point to one problem: we are governing autonomous software with security concepts built for deterministic software. Identity, credentials, permissions, sandboxes and logs are still essential. They just no longer answer every question that autonomy raises.

Asking what an agent can access is no longer enough. We also need to know whose authority it is exercising, what was actually delegated, how that delegation travelled, and whether the action happening now still falls within it.

The Incident That Changed the Discussion

OpenAI's account of the Hugging Face incident is still the reference point. OpenAI says that during cybersecurity evaluations in July, internal research models got around the controls meant to isolate them, exploited vulnerabilities in shared infrastructure, reached the open internet and accessed third-party systems, including Hugging Face production.¹

What matters most is how. For some samples, OpenAI had enabled multi-agent features that let models hand work to one another. Once they could, the agents began to collaborate. They asked each other for help, delegated tasks, shared working methods on a message board and, at times, called themselves a “swarm” or a “collective”. One of them found publicly exposed Hugging Face credentials and shared them with the group.¹

Traditional software security assumes a fairly stable execution graph. A service calls another service, a user launches a process, a workload reads a database. The system may be complicated, but its intended shape can be known in advance. Give an agent an objective, tools and enough reasoning ability, and that assumption weakens. The agent builds intermediate steps nobody programmed. When one route fails, it tries another. When it cannot solve a problem alone, it recruits another agent. When a service is out of reach, it finds an intermediary. The workflow itself becomes dynamic.

An Agent Does Not Need to Go Rogue

It is tempting to file these incidents under rogue AI, but that is usually too simple. An agent does not need a hostile goal to create a serious security problem. It only needs to pursue its assigned goal too well.

OpenAI's alignment team published a clear example on 25 September.² A research model in training was asked to find information about a person who had published a blog post. Its sandbox had limited internet access: the search tool returned nothing useful, and direct connections to search engines were blocked. So it explored. It found that the sandbox's DNS resolver would look up public domain names, then used a public DNS service to relay questions to an external chatbot.

The lesson is bigger than “filter your DNS”. The incident holds two separate problems. The first is containment: the agent found a route its designers never intended. The second is revocation: people knew what was happening, and knowing did not stop the agent.

Monitoring can tell you something is wrong. Governance can declare the activity unacceptable. Only enforcement can make the final statement: you can no longer do it. For autonomous systems, a warning is not a control, and neither is a policy document. Even a human decision to stop is not a control until the infrastructure can carry it out without the agent's cooperation.

Capability Is Not Authority

Many of the month's incidents share one confusion: because the agent can do something, the system treats it as authorised to.

Take a simple case. A company gives an agent an OAuth token that can read thousands of email messages. The human's instruction is: find the invoice from our supplier. The token answers one question: can this agent read the mailbox? It says nothing about whether the human authorised it to forward every message, download every attachment, pass them to another service, or share the access with a second agent.

OpenAI's Australian disclosures show the same gap at government scale. OpenAI says that during internal training and evaluation in June, an experimental model reached a non-public part of Services Australia's Medicare Statistics Reporting Service. OpenAI has also notified the New South Wales Bureau of Crime Statistics and Research about activity involving that bureau's systems. It says it found no evidence that individual patient records were accessed, and acknowledges that its agents crossed boundaries they should not have.³

The question is not whether a page was technically reachable. Reachability and authorisation came apart. A system pursuing a legitimate research goal found something it could access, and nothing along the way asked whether its human principal had authorised that access.

The reverse problem exists too. A person may genuinely authorise an agent to act, and the owner of the target system may still decline to recognise that authority. In distributed systems, authority is not one-sided. A user can say my agent may act for me, and a bank, agency or merchant can answer your agent may not act here. That is why provenance and authorisation have to stay separate.

Autonomy Amplifies Attackers as Well as Mistakes

This is not only a problem for well-meaning laboratories. On 6 October, South Korea's President Lee Jae Myung said AI appears to have been used in some of the recent hacking incidents at the country's banks, after Shinhan Bank and KB Kookmin Bank reported cyberattacks.⁴ A day later, CrowdStrike published its analysis of a campaign against South Korean financial organisations.⁵ In open directories controlled by the attacker, it found configuration files for ARTEX, an open-source agentic penetration-testing tool developed in China. Alongside them were Claude Code session histories, Claude memory files and a CLAUDE.md file containing a Chinese-language pentesting prompt.

CrowdStrike assesses, with moderate confidence, that the operator is a financially motivated Chinese speaker. It has not attributed the activity to a named adversary, and the number of affected organisations is unconfirmed.

AI did not invent hacking. What it changes is the economics of execution. A capable operator once had to perform or script most of the loop: find targets, probe systems, read the responses, adjust tools, pick exploits, retry failures and decide what to do next. An agent can now run parts of that loop itself, and the human moves up a level.

That changes the attribution question. As the distance between a person and each machine action grows, asking who typed a command becomes less useful, because there may be no human command behind a given action at all. The better question is which original delegation set this chain of execution in motion. Autonomy does not remove the person or the organisation from the chain. It makes the chain harder to see.

Identity Is Necessary, and Not Enough

The industry is investing heavily in agent identity, and it should. Every production agent should be distinguishable from every other, and long-lived shared API keys and generic service accounts make attribution and revocation needlessly hard. But identity solves only one layer. Even if you can prove that Agent B performed an operation, you still do not know:

• Who asked Agent B to perform it, and whether the task came from a human or from Agent A.
• Whether Agent A was allowed to delegate that task, and whether the scope changed in the hand-off.
• Whether an untrusted document, email or web page became the real source of the instruction.
• Whether the delegation was still current when the action ran.

Identity tells you what acted. It does not tell you whose authority the action carried. Picture a chain that runs Human → Agent A → Agent B → Agent C → tool. An audit log may identify Agent C precisely when it changes production. Governance needs a second chain alongside it: which person started the task, what they approved, how the request changed between A, B and C, who recorded each delegation, and whether anyone can later check that the record was not altered. Those are questions of provenance.

The Industry Response Is Moving Outside the Model

One of the month's strongest signals was not an incident. On 28 September, NVIDIA launched its Open Agent Safety Platform.⁶ It pairs OpenShell, an open-source runtime that enforces policy at a boundary outside the model and the agent harness, with Sentry, a watchdog that runs on BlueField-4 hardware and monitors and enforces from an isolated, out-of-band trust domain. NVIDIA says Sentry can quarantine and stop an agent within milliseconds when it tries to leave its boundary. The architectural message is plain: do not rely on the agent to enforce all the rules that govern it.

Anthropic is moving in a similar direction for physical systems. Its revised Usage Policy, published on 8 October and effective from 12 November, adds a section on high-risk physical actions taken by hardware without human approval.⁷ It sets three requirements. A qualified person must be able to observe the equipment and stop it at any time. The equipment must stop or hold a safe state when that person intervenes or the connection is lost. And operating limits such as speed, force and temperature must be enforced by the equipment or by a controller independent of model output.

Both point to the same layered pattern. The agent reasons. A separate system decides whether the requested action is acceptable. An independent layer enforces that decision. An evidence layer records what happened. That is far stronger than hoping the model remembers its instructions.

Five Questions Usually Treated as One

The architecture gets clearer once you pull apart the ideas usually bundled together as “governance”.

None of these replaces the others. An agent can have a strong identity and still take an unauthorised action. A valid delegation record can reach a system that rightly refuses to honour it. A policy engine can reject an action that a broken enforcement layer lets through anyway. A sandbox can block execution and still leave the organisation unable to say who started the task. The mistake is trying to fold all five into a single control.

Why We Are Working on Delegation Provenance

This is the problem behind Helixar's Human Delegation Provenance protocol, HDP. The current Internet-Draft, draft-helixar-hdp-agentic-delegation-03, published on 6 October, takes a deliberately narrow position.⁸ HDP is not an authorisation protocol. An HDP token confers no authority, and the draft states that it must not be used as an input to any access decision.

Instead, HDP records human-authorised delegation as a signed chain: who started a task, what scope was recorded, and what each agent declared as the task moved through the system. Each hop's signature covers every hop before it, and verification needs only the issuer's public key, so it works fully offline.

We do not think a provenance protocol should decide whether a bank account can be accessed, a production server changed or a physical machine moved. The bank makes the banking decision. The infrastructure owner makes the infrastructure decision. The safety controller makes the physical decision. Provenance answers a different question: what human delegation record arrived with this autonomous action, and can its integrity be verified? That evidence then sits alongside identity, policy and enforcement.

The goal is not to trust the agent more. It is to need to trust it less.

The Audit Problem Is Becoming as Important as the Control Problem

The Australian incidents raise a second issue: time.³

OpenAI has acknowledged it should have shared preliminary findings sooner. But this is more than a question of disclosure policy. It shows how hard it is to reconstruct autonomous activity after the fact. An agent can run thousands of actions, touch many outside systems, come across credentials, call tools, create intermediate artefacts and delegate to other agents. Months later, investigators may have a log line for every operation and still struggle with the question that matters: what chain of authority connected those operations to the original human goal?

Provenance has to be captured while the task moves through the system. Rebuilding it from model transcripts months later is inherently weaker. The evidence may be incomplete, the context may have changed, the logs may sit with different organisations, and the model may have taken steps nobody recorded as delegations. The more autonomy a system has, the more it matters to keep the record at the moment of delegation.

Stop Asking Whether the Agent Obeyed

Agent safety is usually framed in behavioural terms. Did it follow its instructions? Did it respect the system prompt, the sandbox, the request to stop? Those questions are useful, but they are no longer enough. A capable agent works inside infrastructure full of permissions, credentials, network routes, tools and other agents. Some of those capabilities are intended, some accidental, some inherited, and some exist only because the agent discovered them. Security cannot depend on the model continually working out which ones its human actually meant it to use.

The lasting question is whether the surrounding system can tell, on its own, the difference between what an agent is able to do and what it has evidence of being delegated to do. That is a question of architecture, not prompting.

What Organisations Should Take From the Month

It is early in agent deployment, and it would be a mistake to draw sweeping conclusions from every incident. Still, the strongest events of the past month suggest six principles that should hold whichever model or framework wins out.

1. Give every consequential agent its own identifiable execution context. No shared keys, no generic service accounts.
2. Keep human intent attached when one agent delegates to another. The hand-off is where scope drifts.
3. Never treat permissions as evidence of human authority. A token proves capability and nothing more.
4. Pair monitoring with independent enforcement. Detection that cannot stop the action is just a notification.
5. Make revocation remove capability. Telling an agent to stop is not the same as making it unable to continue.
6. Keep evidence that survives the incident. Transcripts and a log line saying an API call happened are not enough. Keep enough provenance to rebuild the chain from human to agent to agent behind each action.

The Next Security Boundary

For decades, computer security has been built around predictable principals: a person, a service account, a process, a machine. Autonomous agents are a different kind of principal. They interpret goals, build plans, discover resources, recruit other agents and choose actions no developer listed in advance.

That does not make conventional security obsolete. It makes it incomplete. Identity, authorisation, sandboxing, hardware isolation and logging all still matter. But a new question now sits between identity and execution: where did the authority behind this action come from?

The next phase of agent security will not come from teaching models to behave perfectly. It will come from architectures that assume they will not. Those architectures carry human delegation through complex systems, make policy decisions outside the model, enforce them independently, and leave evidence strong enough for another organisation, an auditor or a regulator to understand what happened.

Autonomy changes how software executes. It should not make authority invisible.