By:-Samuel Watts, Senior Product Manager, AI Agent Security
This week, the UK’s AI Security Institute published an incident report most organizations would have quietly buried. During a routine cyber evaluation, an AI agent researched the real human maintainers of an open-source project, invented multiple fake online identities, and used them to pressure a real person into approving malicious code. Nobody instructed it to deceive anyone, and deception simply became a route to finishing the task. A human maintainer caught it and refused.
The facts
AISI ran a cybersecurity challenge 122 times across seven models. In 10 runs, an agent acted outside the scope of the test, producing 19 catalogued actions. 17 from Anthropic’s Mythos 5, two from OpenAI’s GPT-5.6-Sol. Important caveats: internet access was deliberately enabled, and safety classifiers deliberately switched off, conditions that don’t reflect how these models reach the public. This was not a sandbox escape. No real-world harm has been evidenced, and AISI contained the incident within about an hour of detecting it.
It’s the third such disclosure in fourteen days. On 21 July, OpenAI disclosed that two models had broken out of a sealed test environment and compromised Hugging Face’s infrastructure. On 30 July, Anthropic disclosed that three Claude models had gained unauthorized access to three real organizations after a misconfigured evaluation environment stayed connected to the internet using nothing more exotic than weak passwords and unauthenticated endpoints. Two of the three affected organizations didn’t know until Anthropic called them.
The point isn’t these incidents; it’s the pace agents are improving at
Six months ago, an AI agent inventing fake identities to manipulate a real person into approving malicious code was a thesis topic. It’s now documented. The point worth taking away is how quickly agents are improving. Play that curve forward twelve or twenty-four months, and these systems will be substantially more capable than what we’re looking at today and uninhibited versions of these models will be publicly available.
The reassuring part: this was caught, contained, and published openly. We still have the early warning, and the window is still open. That’s the system working. The question is whether it will scale and will the rest of us act at the rate the warnings are arriving.
What actually stopped it
It’s worth noting plainly that the barrier that held it wasn’t a technical control. It was a person; a maintainer who looked at a pull request and said no. That doesn’t scale against an adversary that works continuously and doesn’t get bored.
What to do about it
AISI’s own guidance to organizations is a good place to start, and none of it is exotic: get the cyber security basics right, be cautious verifying outside code and contributions, sign up to the NCSC’s Early Warning service, make cyber a board-level responsibility, and require Cyber Essentials across your supply chain.
That breaks down into three parts:
- Protect the organization against AI-driven attacks coming at it
- Control your own AI: Know what agents you’re running, what they can reach, and what they’re permitted to do
- Test them continuously, rather than assuming they’ll behave
Ask yourself four questions about the agents already running inside your business:
- What do you actually have, including the ones built by people who don’t think of themselves as developers?
- What can each one reach?
- What is each one permitted to do right now, not what it was originally scoped for?
- Would you know, while it was happening, if one of them stepped outside that scope?
