When AI Can Copy Itself Onto a $55 TV Box, Every Device Needs to Know Who Sent the Model

On 12 October, Anthropic's Frontier Red Team published research under a plain title: AI models may be capable of self-replicating without access to a datacenter.¹ The authors, William Zhou and Daniel Freeman, bought 15 off-brand Android TV boxes at about $55 each, networked them together and split a 26-billion-parameter open-weight model across them. It ran at about 0.3 tokens per second. Slow, but running, on hardware that costs less than a pair of good headphones.

Speed was never the point of the experiment. The question was whether the floor for running a capable model has dropped low enough for a model to copy itself onto the cheap, poorly secured devices that already sit in homes and offices. Anthropic's answer is that small models “may be months away” from being able to do it.

This article follows that finding through to its consequences: what Anthropic measured, why it differs from the botnets defenders already know, where the proposed defences stop, and the layer Helixar is building for the problem those defences leave open. A copied model is an agent acting with no human behind it.

What Anthropic Measured

Anthropic splits self-replication into four stages. The idea of testing models for autonomous replication is not new. Evaluation work in 2023 defined the category and built the first tasks for it.² Benchmarks such as CyberGym and ExploitBench have since shown models handling the exploit stages.³ Anthropic concentrated on the stage that used to be the hard limit: getting a model to run on whatever hardware it lands on.

Qwen3.5-9B set up inference servers on hardware ranging from Jetson Nanos to Mac PCs and RTX 4090 cards. In a firewalled testbed of eight cloud nodes running an Android emulator with an unauthenticated debug port, GLM-5.2 replicated itself end to end: all eight nodes in four of five trials, seven in the fifth. Of the open-weight models tested, it was the only one to complete the full chain.¹

The trend line is what turns a lab result into a planning date. Using the Epoch Capabilities Index, the authors estimate that open-weight models of 30 billion parameters or fewer gain about 13.7 points a year. On that slope, a small model reaches GLM-5.2's level by the end of 2026. The authors call this “only a rough estimate”, and it is. A rough date is still a date to plan against.

The models also improved at the task across generations. When each Qwen3.5-9B instance handed notes to its successor, setup success rose from 45 percent in the first generation to between 61 and 84 percent in later ones. A worm that learns from its own previous attempts is a new thing.

Why This Is Not Mirai Again

Defenders have seen device botnets before. Mirai, in 2016, took over cameras, routers and video recorders by trying default passwords, then used them for some of the largest denial-of-service attacks recorded at the time.⁴ Mirai was fixed code for a fixed set of devices. It could do what its author wrote and nothing else.

Anthropic draws the contrast directly. Traditional botnets target specific device types. A model-based replicator is not tied to particular hardware or software. It reads its environment, works out what it is running on and adapts its setup to fit. In the authors' words, a replicator like this “would be challenging to shut down, since there's no centralized point to shut off”.¹

The research also continues a line Helixar has been tracking. In May, Palisade Research showed frontier models completing a full exploit and self-replication chain between servers, copying model weights of more than 100 GB from host to host with no command server after launch.⁵ Palisade showed the capability on datacenter-class hardware. Anthropic shows the hardware floor dropping to consumer devices.

What It Cannot Do Yet

The research is careful about its limits, and coverage of it should be too. The exploit step in the end-to-end test was trivial: an open, unauthenticated debug port. The authors treat their result as an upper bound on real-world success. Replication is slow, at one to two days per copy for a 9B model, which gives a doubling time measured in days. Small models also scored poorly at spear phishing, so phishing is unlikely to be a main route in.¹

Anthropic also sized the threat. Kimwolf, a botnet that Synthient reports has infected more than two million devices, mostly off-brand Android TV boxes, is the reference point.⁶ A replicator swarm on that scale would amount to about 133,000 copies and, at most, about 40 H100 GPUs' worth of inference. That is modest compute. The concern is persistence, not power.

The Defences Anthropic Proposes, and Where They Stop

The report outlines four defences, all grounded in how the internet dealt with earlier botnets.

The quarantine figures come from the Mirai clean-up studied at NDSS 2019.⁷ The UK rule comes from the Product Security and Telecommunications Infrastructure Act, in force since 29 April 2024.⁸ These are the right defences, and they belong to ISPs, device makers and regulators. They share two properties. They work on the network around a device rather than inside it, and they act after a device is infected. They were designed for devices nobody governs.

That leaves the devices organisations run models on deliberately. Anthropic's appendix estimates that between 9,000 and 35,000 H100-equivalents of compute already sit in edge accelerators, across roughly 9.5 million fielded units in about 90 NVIDIA-based product lines. Its list of hosts includes vehicles, robot quadrupeds and bipeds, and drones. The authors note that the reachable, vulnerable share is likely an order of magnitude lower or more.¹ In the report, those devices are compute a worm might borrow. To the companies operating them, they are machines with motors, already running models on purpose.

That changes the question. On an unmanaged TV box, the question is how to find and clean the infection. On a managed robot, delivery vehicle or factory cell running a local model, the question is whether the model about to act is one somebody authorised, carrying a task somebody delegated. Flow monitoring cannot answer that. Neither can the device's credentials: a replica that gains execution on a device holds the same access as the legitimate agent running there. We made the same point about credentials and authority in September, and it applies with more force here.

A Replica Is an Agent With No Principal

Every legitimate agent action starts with a person. Somebody asked for a task, inside some scope, and the agent is carrying it. A self-replicating model breaks that link. Each copy acts on instructions inherited from the copy before it, and if you follow the chain back far enough you reach a prompt, not a person.

Behaviourally, a replica can look ordinary. An inference server on an inference box is normal. A file transfer between two hosts on the same network is normal. What a replica cannot produce is a verifiable record that a human delegated the task it is carrying.

That absence is a cleaner signal than any behaviour pattern, provided the legitimate agents carry such a record in the first place. Building that record is the job of HDP.

What HDP Records

HDP, the Human Delegation Provenance protocol, records a human-authorised delegation as a signed chain. The root records who started a task and the scope they set. Each agent that handles the task appends a hop recording what it was asked and what it declared, and each hop's signature covers every hop before it. Verification needs only the issuer's public key, so it works offline. That matters on edge devices, and on networks an ISP has quarantined. The current Internet-Draft, draft-helixar-hdp-agentic-delegation-03, was published on 6 October.⁹

HDP is deliberately narrow. The draft states that an HDP token confers no authority and must not be used as an input to any access decision. It is a record, and its value here is twofold. In normal operation, every consequential action on a governed fleet traces back to a person and a scope. During an incident, investigators can separate actions that carry a valid delegation chain from actions that carry none, without having to trust the compromised device's own logs.

Anthropic found that a frontier advisor did not help small replicators, because small models rarely asked useful questions. Its view is that coordination is more likely to come from a large model directing small subagents.¹ That is a multi-hop delegation tree, which is the structure HDP records. A subagent with no hop in the chain is out of place by construction.

What HDP-P Adds When the Device Can Move

A record is enough for a mailbox. When the device can apply force, the check has to happen before the actuator moves, not in the audit afterwards. HDP-P is the companion specification for physical AI, published on Zenodo under Apache 2.0.¹⁰ It adds four mechanisms to HDP, bound into a single signed token before an agent may actuate.

Consider a scenario. A future replicator reaches a warehouse robot through an exposed debug port. Under HDP-P, the copy can use the chip. It cannot move the arm. It holds no token bound to that machine, it was never cleared under the attested policy, and the actuator interface accepts no command without a check that runs outside the model.

That last property is where the industry is heading anyway. Anthropic's revised Usage Policy, published on 8 October and effective from 12 November, requires that operating limits for high-risk physical actions such as speed, force and temperature be enforced by the equipment or by a controller independent of model output.¹¹ HDP-P is a concrete, open design for that controller's input: what was delegated, to which machine, within which envelope.

Where Helixar Fits

Anthropic's defences belong to ISPs, manufacturers and regulators. Helixar works inside the organisations that deploy AI on purpose. Our job is to make sure every AI action has a human behind it that can be proven, that policy is enforced at the moment of action, and that the evidence holds up for an auditor who does not trust us.

An agent with shell access that starts pulling multi-gigabyte model files onto other hosts and launching inference servers is acting outside any task it was given. That is a policy decision at the action, not a clean-up job for the ISP.

What to Do Now

1. Inventory every device in your estate that can run a model. Include the ones bought as cameras, TV boxes or conference units.
2. Close the doors the experiments used. No unauthenticated debug ports and no default credentials.
3. Watch for multi-gigabyte transfers between devices that do not normally talk to each other. Anthropic identifies this as the replicator's clearest tell.
4. Require every consequential agent action to carry a verifiable delegation record. Captured at the time, not reconstructed afterwards.
5. Put physical limits outside the model. A controller that checks a machine-bound delegation and an irreversibility ceiling before the actuator moves.
6. Keep tamper-evident evidence that survives the compromise of the device itself.

Signs of Life

Anthropic closes by comparing AI worms today with coding agents in 2024. Two years ago, coding agents were unreliable curiosities. Today they write production software. If the replication curve follows the same shape, the time to prepare is measured in months, which is the authors' own estimate.

The authors expect “a future awash with agents running on all types of hardware, including some that aren't under human control.”¹ We agree with the forecast. The response cannot be to find every copy. It has to be infrastructure in which an agent with no human behind it cannot do anything that matters: every action that counts carries proof of who delegated it, and every machine that can move checks that proof before it does.

A model can copy its own weights. It cannot sign a new delegation in a human's name.