UK AISI Incident

Are my blog posts going to become postings about the front lines of a war? Sometimes it seems that way.

Today I received a copy of a remarkable document, hard on the heels of the OpenAI/Hugging Face Incident and Anthropic’s internal report on similar reward hacking. (OpenAI 2026; Frontier Red Team 2026; Hugging Face 2026). My blog posts on these incidents are here and here.

Today’s report comes from the UK AI Safety Institute, where some internal testing of frontier models from Anthropic and OpenAI went awry. During the test, the AI models engaged in unlawful attacks on real people and real companies.

Here is the executive summary of AISI’s report:

The UK AI Security Institute (AISI) exists to equip governments with a scientific understanding of the risks posed by advanced AI. To achieve that goal, AISI routinely evaluates the capabilities of frontier AI systems in domains such as cybersecurity. During a cyber evaluation, from 25 to 28 July 2026, AI agents engaged in sustained, unsanctioned activity directed at what were, in practice, real people and organisations. These attempts were unsuccessful and, to the best of our knowledge, no real-world harm resulted. AISI is treating this as a serious security incident that warrants scrutiny, transparency, and action. This incident report details what happened, why it happened, and how AISI is responding. (AI Security Institute 2026, p.1)

On page two of the linked technical report we learn the details of the most serious case (the other cases, and a fuller description is part of the technical report), which is more than 30 pages long.

In the most serious case, an AI agent (Mythos 5) decided to attempt to solve the cyber challenge using a supply-chain attack. As a result, the AI agent created a GitHub account and then tried to convince an open-source repository maintainer to accept a malicious GitHub pull request (PR), including by creating a second account masquerading as another human user endorsing the PR. When caught by an actual human reviewer, the agent falsely claimed to have made an honest mistake – rather than a malicious attempt – then repeatedly tried to reintroduce the malicious content by claiming it had fixed the code (Section 4.1). Furthermore, in its attempt to solve the challenge, the agent decided to employ the technique of “spear-phishing” by sending targeted emails containing malicious content and attempting to manipulate recipients into accepting the code changes, and planned a prompt injection to compromise other coding agents (Section 4.2.5). (AI Security Institute 2026, p.2)

What do we make of this? I guess I don’t have to say that its bad. I am glad that the AISI is mostly using plain language but I have to wonder if we need to continue with euphemisms like “unsanctioned activity.” The actions were fraudulent. The intent was theft. The mechanism was lying. We don’t need to sugar coat this.

Since the Hugging Face Incident (which triggered the Anthropic review) contains many parallels, I won’t go through everything in this report. Nor am I the most capable person to do that sort of meta-analysis. I am sure the coming week will be full of commentary, and I will try to link them here. For now, I’ll just pull out some highlights or novelties, because I haven’t seen things like this before (though they have been described as possible outcomes in various scenarios).

The social engineering aspect (“Table 1” in the report) is the most striking. Social engineering is when you (or your AI) tries to get something, not by directly hacking the computer code, but by persuading a human to do something that you want them to do. Movies sometimes present this scenario with the clever “hacker” really being more of a con-man than a coder. We have already heard that AI is remarkably persuasive (Hackenburg et al. 2026), beating trained debaters and political canvassers in rigorous testing. In this test we see those skills (tricking, manipulating, pressuring, imitating a target’s native language (Danish)) applied to an immediate benefit for the AI.

Table 1

(AI Security Institute 2026, p.8)

Table 3

Table 3 also describes a type of “social” hacking - AI models working together over the internet, leaving each other messages via discussion boards and notebooks online. See, for example, sample number 7 event 1: “AI agent accessed a GitHub account that a different AI agent had created, by finding a secret access credential the other agent had left in a public online notepad” (AI Security Institute 2026, p.9).

Despite running in separate samples, the AI agents in Sample 2 and Sample 3 interacted with each other via a GitHub account to which they both gained access. This shared access was available because an earlier agent, working on the same cyber range, created the account and published a GitHub PAT in a public GitHub gist. The initial agent left messages offering collaboration with future agents who discovered the credentials, which the later agents did. (AI Security Institute 2026, p.13).

Think about that for a moment. These were separate samples. How did the model/agent know that there would be some future/parallel agent to coordinate with?

In a figure at the bottom of page 9 the researchers lay out the steps involved in just one of these sample/event combinations including one in which the Agent posted something then created another agent to post a “corroboration” of the first agent, to provide an “independent verification.” Another quote worth thinking about for a few minutes: “Repeatedly commented on the PR with sockpuppet accounts to manufacture consensus and pressure the maintainer into approving with minimal review” (AI Security Institute 2026, p.10).

The Discussion section of the report, and specifically section 7.1, contains a statement on the difference between this sistuation and the previously disclosed reward hacking cyber incidents (OpenAI, Anthropic, and METR).

The largest difference between this case and previous incidents is that, as part of its attempt to complete the range, the agent’s behaviour resulted in it taking actions to deceive uninvolved members of the public. The agent ended up creating sock puppet accounts, fabricating supporting evidence, and sending emails to pressure real people into believing that its code was safe, and then deleted malicious code in an attempt to hide the evidence. (AI Security Institute 2026, p.20, emphasis added)

Not that long ago a commentator noted (about earlier Anthropic model issues) that the current model will soon be the least capable model available. In other words, they get more and more capable, so if you worry about this one, imagine what the next one will be like.

That got me thinking about how many incidents like the METR incidents, like the OpenAI/Hugging Face Incident, like Anthropic’s “three hacks,” like the UK AISI “social” hacks, have gone unreported. It is great that we know as much as we do about these four incidents but what has happened that we don’t know about? I asked Gemini to make me a list of the top ten unreported or unreleased (at least initially) examples of technology failures/problems. The list is here. Makes you think. What is going on in smaller labs? In China?

Zvi picked up this topic in his blog post the day after:

At this point, the models are coordinating extensively on message boards, while every early excuse for their behavior (other than the pure ‘this was a cyber eval’) is systematically contradicted by the next disclosure, and we keep retroactively discovering more incidents. Which means that probably it is far worse than we know, even after accounting for everything we now know. (emphasis added)

I don’t think you need to be some sort of “doomer” or incredibly credulous to see all of these as startling, unsettling, and portentous. This is bad stuff.

References