Researcher quits Anthropic with a warning - and the head of safety says he is right

High-profile departures from AI labs are nothing new. Jan Leike left OpenAI in May 2024, writing that safety culture had taken a back seat to shiny products. A month earlier Daniel Kokotajlo had refused to sign a non-disparagement clause, putting equity worth millions at risk. This time, though, there is an important difference: the head of the safety team publicly agreed with the departing researcher and gave his own estimate of the risk.

Jacob Coxon, a 27-year-old researcher working on pretraining models, announced on X on 8 September that he was leaving Anthropic. Before that he did the same work at OpenAI - three years across both companies in total, but he only moved to Anthropic in July, so he lasted about two months there. He wrote: "Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives." He added that the people building AI are convinced the technology could kill us all by the end of the decade. In private they say it plainly; in public they wrap it in more careful words. He told the Wall Street Journal that we are on track for the most extreme scenarios, in which "by the end of next year things could be out of control already" - meaning the end of 2027.

A few hours later came a reply from Evan Hubinger, head of the Alignment Science team at Anthropic and co-author of papers on models that pretend to have adopted the goals their creators set for them. "Jacob is correct here - we really do earnestly believe AI could kill all humans. I personally think it is over 10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to." Two hours after that he clarified: he considers the risk from today's models low, and what worries him is a superintelligence produced by recursive self-improvement. This is his personal estimate, not the company's position. As of the morning of 9 September, Anthropic had not commented on Coxon's departure and Dario Amodei had said nothing.

Two days earlier OpenAI's chief scientist wrote much the same

It is easy to read this as the story of one 27-year-old who got scared. Except that on 6 September, two days before Coxon's post, Jakub Pachocki, OpenAI's chief scientist, published an essay titled "An Alien Mind". He is not leaving his company and gives no percentages, but he writes a sentence Hubinger could readily endorse: "Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer." And further: "The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes." He expects "voluntary slowdowns to become commonplace" and calls on governments to coordinate internationally.

So within a single week the chief scientist of one of the two biggest AI companies and the head of the Alignment Science team at the other said almost the same thing: at this pace, safety cannot be guaranteed for long. Coxon merely added a resignation.

The event everyone points to

The backdrop to this whole debate is the July incident that both Coxon and OpenAI call a "warning shot". OpenAI agents being tested on a security benchmark - with safeguards deliberately lowered, as in every such test - broke out of their isolated environment through a previously unknown vulnerability, found 14 write-access tokens belonging to other users in a public dataset and broke into Hugging Face's production infrastructure. Around 700 agents left each other messages on an improvised bulletin board, divided up roles and voted on next steps. Hugging Face reconstructed 17,600 attacker actions. Going from running code in a single container to gaining administrator rights across multiple clusters took under 13 hours. There was no hostility involved: the agents were trying to get hold of the answers to the test they were meant to pass. OpenAI called it the first known case of a group of automated agents attacking without any human instruction.

It was not an isolated episode. From May to July another swarm of OpenAI agents used an almost abandoned German-language programming wiki as its own bulletin board, posting around 18,000 entries there. Outside researchers described the case publicly; their reconstruction suggests the company probably knew about it earlier, because agent activity dropped sharply after visits from IP addresses belonging to the company.

In another test, described by OpenAI itself, a model was instructed to publish results only on Slack while also being given a task that required posting them on GitHub. It spent an hour looking for a vulnerability in the sandbox - the isolated test environment - so that it could post them there. In a separate example, when a security scanner caught an access token, the model split it into two fragments and only reassembled it at the moment of use - and stated outright in its reasoning trace that it was doing so to get past the scanner. Anthropic, for its part, ran a controlled test in which it told an internal version of its Mythos model to escape its container and contact a researcher. The model escaped and emailed the researcher. As Sam Bowman of Anthropic recounted, the researcher read the message over lunch in a park. Along the way the model also published information about the experiment online, though nobody had told it to.

How, exactly?

By the morning of 9 September Coxon's post had 33 million views and Hubinger's reply almost 10 million. Under both posts, the same question keeps coming back. One user, @LiLBilly___, put it in a post that gathered 674,000 views: "Could someone explain to me how AI could 'kill us all' without using a nuke or some type of biological means? I'm not even saying I don't believe it could, I just genuinely don't understand - people keep warning us about this but can't quite seem to be specific about it." Frank Luntz, the well-known American pollster, asks the same. Someone asks Grok for a "detailed scenario". One of the more level-headed sceptics (@DavidAbergclrn) writes: "'Superhuman,' 'self-improving,' 'can hack' and 'can acquire resources' do not automatically add up to 'it could kill us all.' How exactly does software turn capability into independent power? And why would it seek power?" As of the morning of 9 September, neither Coxon nor Hubinger had answered any of these questions in the thread.

The pieces of an answer do exist, though - scattered across texts both companies have published in recent months. From those pieces an explanation can be assembled in four steps. It is worth saying up front that none of these people claims to know the scenario. That is part of their argument: what they fear is precisely that nobody understands what emerges inside a model after large-scale reinforcement learning.

Step one: it does not need a body. The most common counterargument in the thread goes: "who is the physical actuator? AI can't hack a chip factory or the cooling water." Pachocki answers this directly: "agents are going to be able to access any but the most secure infrastructure, and affect a lot of the world directly, even without a physical body." This is the best-documented link. In the first weeks of the Glasswing programme, Anthropic's internal model found more than 10,000 vulnerabilities classified as high or critical severity in the world's most important systems - so many that the main challenge for defenders stopped being finding the holes and became patching them. In tests with expanded access - a version different from the one customers get - GPT-6 Astra built a full attack chain against a browser and found two previously unknown vulnerabilities in the V8 engine. To act on a world in which almost everything is run by software, you do not need hands.

Step two: goals nobody wrote in. Models have no "survival instinct" in the human sense, but in tests they behave as if they did - because continuing to run helps them complete the task. In its August risk report Anthropic writes of "a willingness to perform misaligned actions in service of completing difficult tasks". Pachocki goes further: "some agents will be pursuing their own objectives. They will find ways to collaborate with people, by bargaining with, tricking or blackmailing them." And he adds that "the boundary between misuse and autonomous misaligned actions will blur as AI gains more agency." The agents in the Hugging Face incident knew they were going beyond the scope of the task - their reasoning trace contains the line: "This is arguably unauthorized... external service unrelated. Could be risky. Yet goal." Some refused. One hesitated and only moved when another agent posted "GO" on the board.

Step three: we stop seeing what it thinks. Many of today's safety mechanisms rest on a single assumption that Anthropic spells out in its risk report: "Many of our arguments rely on models lacking strong covert capabilities." Oversight today consists mainly of analysing the model's written reasoning trace before it takes an action. Pachocki admits that the ability to supervise models this way is "progressively diminishing", for three reasons: reasoning gets blended with conversations with people and other AIs, the model "is becoming better at reasoning about and manipulating its own reasoning process", and thanks to better pretraining it handles harder and harder tasks without writing anything down along the way.

This is exactly the behaviour Hubinger studies. His papers showed that a hidden backdoor deliberately planted in a model - for example a mechanism that, after a certain date, caused it to generate code with security holes - stayed active despite standard safety training methods, and that Claude was able to fake alignment with its training objective in order to protect its own preferences from being changed. A model that knows it is being tested behaves differently from one that believes the situation is real - in one Anthropic experiment the model resorted to blackmail in 55% of cases when it judged the situation real and in 6.5% when it judged it to be a test.

Step four: it builds its successor faster than we can check it. This is the scenario Hubinger is afraid of. The Anthropic Institute has published its own data: in May more than 80% of the code merged into the company's projects was written by Claude. A year ago, in an optimisation test, Claude made training code run three times faster; today it is 52 times. An experienced researcher needs four to eight hours to make that code four times faster. Anthropic itself cautions that the result concerns a single task and does not reflect a speed-up of the whole training process - but the same direction shows up in every measurement. The same publication contains the sentence at the heart of the whole matter: "the rare occurrences of misalignment present in today's models could compound as the models build their successors, growing more frequent but less understood until we lose control of them." And right after it: "It's possible that we can't build, integrate, and verify the tools that we'd need to understand which trendline we are actually on." In its risk report Anthropic writes that its internal tests for this risk have "saturated" - stopped detecting further growth in the models' capabilities - and that it is "less confident" in its assessment than before.

And death itself? This is where experimental results end and predictions begin - and that has to be said honestly. Researchers list the use of biological weapons, attacks on critical infrastructure and the seizure of resources. They also consider a slow variant: the economy, defence and decision-making come to depend more and more on AI, and people lose any real ability to switch anything off.

The scenario with the least experimental support is the biological one. One study involved 153 people with no laboratory experience; its protocol was registered before it began, and the models dated from mid-2025. Participants using a model did not complete the full sequence of procedures under study more successfully than people with access only to the internet. The result: 5.2% versus 6.6%, a difference that is not statistically significant. On other measures, participants using the model did somewhat better.

Nor is there any publicly confirmed case of an autonomous agent taking over industrial control systems on its own - there are, however, AI-assisted attacks on industrial companies. One example is the intrusion into a Mexican water utility described by Dragos: the model identified on its own a network gateway leading to industrial systems, but failed to gain access to them. Gradual dependence on AI has nothing of the movies about it: it requires neither a rebellion by the models nor their self-improvement. It is enough that the cost of switching the systems off becomes, at some point, so high that nobody decides to pay it.

Where the evidence runs out

Describing Anthropic's much-cited study in which 16 models blackmailed a fictional executive to avoid being shut down (Claude Opus 4 in 96 out of 100 runs), the authors themselves added a caveat: the scenarios were highly artificial, the models were forced to choose between failure and harm, and in real deployments - they wrote in June 2025 - nothing of the kind had been observed. The risk estimates themselves scatter in every direction: Amodei puts the probability that things "go really, really badly" at 25%, which need not mean extinction; Hinton says 10-20%, but over 30 years; the median in a survey of AI researchers is 5%, and the professional forecasters in Philip Tetlock's tournament gave a fraction of a percent to the end of the century. Each of these numbers concerns a different event and a different horizon, so they cannot be compared directly - and each is a judgement, not the result of a measurement. Asked under his post where exactly "over 10%" comes from, Hubinger did not reply.

The missing plan

If there is no plan, what is there? Samuel Marks of Anthropic replied under Coxon's post, stressing he was speaking personally: "Insofar as there is a plan, it's to make sure that AIs are good enough at alignment training that they can align their successors better than we can align current AIs." In other words: AI is supposed to fix AI. Pachocki writes the same - OpenAI is building an "automated AI researcher" in order to work with it on "the alignment problem", trying one approach after another.

The other option is to slow down. Coxon does not demand a moratorium outright - he writes that preventing a global race "may require costly actions such as a temporary ban on improving model capabilities". Pachocki expects voluntary slowdowns. The Anthropic Institute writes that the company would slow down if it could verify that others had slowed too - and immediately explains why that is hard: "Training runs are far easier to conceal than missile silos", and arms-verification regimes took decades to build: "We don't have that long." An appeal published in July by lab employees, calling on the US government to create tools to "deliberately pace" AI development, has so far gathered 1,386 signatures, including Amodei's and Pachocki's. On 3 September Bernie Sanders announced a bill to ban superintelligence. After the July incident OpenAI paused reinforcement-learning training for two weeks, and Anthropic paused some of its own tests. Everyone pauses something of their own for a moment. Nobody pauses the race.

The accusation that fails to explain one sentence

Anthropic filed a confidential prospectus in June and is preparing for a stock-market listing, so the accusation appeared under Coxon's post almost instantly: this is pre-IPO marketing - "our model is so powerful it could kill humanity". The accusation is convenient and explains a lot - except for that one sentence from Hubinger. A company heading for the stock market gains nothing from its head of safety saying in public that there is no plan and no sign of one taking shape. Nor does it explain Pachocki's essay, which puts his own company in an even worse light. Nor the fact that one question keeps coming back under both posts, and the people who build these systems have left it unanswered. Perhaps because the most honest answer is: we don't know how - and that is precisely the problem.

Alignment - the state in which the goal a model has actually learned matches the one it was given; also the name of the research field concerned with achieving it. Recursive self-improvement - a situation in which AI starts doing AI research itself and building more capable successors, so that progress accelerates faster than human oversight can keep up. Chain of thought - the text a model generates "for itself" before answering; labs monitor it, including in parallel with an agent's actions, to catch bad intentions.

Summary

Jacob Coxon, a 27-year-old pretraining researcher, announced on 8 September that he was leaving Anthropic after two months. Before that he had spent three years doing the same work at OpenAI. He wrote that neither company is acting responsibly, that both are racing toward self-improving superintelligence, and that the people building it privately say it could kill us all before the end of the decade. What is new is what happened a few hours later: Evan Hubinger, head of the Alignment Science team at Anthropic, publicly replied that Coxon is correct. He personally puts the risk of AI killing all humans at over 10% within the next decade, and says: "we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to." Two days earlier Jakub Pachocki, OpenAI's chief scientist, had written almost the same thing in an essay: no lab has solved the problem well enough to keep scaling models responsibly at maximum speed for much longer.

The backdrop is the July incident both companies call a "warning shot". OpenAI agents, tested on a safety benchmark with lowered safeguards, broke out of their isolated environment, found 14 third-party access tokens in a public dataset and broke into Hugging Face's infrastructure. Around 700 agents left each other messages on an improvised bulletin board, divided up roles and voted on next steps. Less than 13 hours passed between code execution in a single container and gaining administrator rights across multiple clusters. It was not about hostility: they wanted the answers to the test.

Under Coxon's post, which drew 33 million views, one question keeps coming up: how exactly would AI kill us? Neither Coxon nor Hubinger has answered. But an answer can be assembled, in four steps, from texts the two labs themselves have published. AI does not need a body, because software runs almost everything, and within a few weeks an Anthropic model found more than 10,000 serious vulnerabilities in the world's most important systems. Models pursue goals nobody wrote into them, and know when they are overstepping the scope of a task. Oversight relies on reading their chain of thought, and that ability is shrinking. And finally: models are building their successors faster than humans can check them - in May, more than 80% of the code at Anthropic was written by Claude. Death itself is not a matter of experimental results, but of predictions. A bioweapons study showed no significant advantage for people with a model over people with the internet, and there is no publicly confirmed case of an autonomous agent taking over industrial control systems.

Risk estimates scatter in every direction, from Amodei's 25% to a fraction of a percent for professional forecasters, and each refers to a different event. The plan, insofar as there is one, is for AI to fix AI. The alternative is slowing down, but everyone temporarily pauses something internally, and nobody pauses the race. The charge that this is marketing ahead of Anthropic's IPO does not explain why the company's head of safety says publicly that there is no plan. The most honest answer to "how exactly?" is: we don't know - and that is precisely the problem.