Amodei wants to slow AI down - Altman, Musk and Nadella agree, Trump and the White House do not

For three years the loudest calls to slow AI down came from outsiders. The Future of Life letter of March 2023 was signed by Musk, but not by the heads of OpenAI or Anthropic. On Saturday, September 12, that call came from the CEO of the most valuable AI company in the world, and by Monday everyone who matters had answered. Last week we wrote about the researcher who left Anthropic and issued a warning, and about the head of its safety team who said he was right. Now the CEO admits the researcher was right.

The essay "We Must Pace the Frontier" runs just under 4,000 words, and the post on X announcing it had been viewed 72 million times by Monday. The plan has three steps: embedded evaluators inside every company, shared standards among companies in democratic countries, and an attempt to reach an understanding with China. Anthropic commits unilaterally only to the first. An outside team (Amodei names METR as an example) gets desks in the office, badges, company laptops and access "mostly comparable to what internal risk assessment teams have." On top of that, the right to publish findings without editorial control by the company. No date, no name of the evaluator, no number of people. Amodei cautions that "pacing does not mean halting model training or technical progress." The most concrete warning in the text: Amodei worries that in 6 to 12 months a swarm of agents could turn the entire internet into a persistent botnet and cause damage running into hundreds of billions of dollars.

The rivals say yes

Sam Altman replied two and a half hours later: "I agree with Dario that we need to pace the frontier. [...] Committing to having independent evaluators with employee-like access is a great idea, and we will do the same" (16 million views). Elon Musk, an hour later: "Dario is right" (12 million). Demis Hassabis of Google DeepMind: the direction is right, the details need work. On Sunday evening Satya Nadella wrote that "if the AI we build is not helping humanity and under human control, it's not worth pursuing." He also announced a code of conduct for Microsoft's own models, which went into a six-week public consultation on Monday. Three years ago Musk signed one letter on AI safety and Altman and Hassabis signed another. They never signed the same one. This time they said the same thing within 30 hours.

The White House says no

And there the agreement ends. Donald Trump, asked by reporters on Sunday at his golf resort in Doonbeg, Ireland: "We're leading China in AI, we're the most sophisticated country in the world, and frankly, I want to keep it that way because whoever wins AI, wins. And we can put guardrails, we can do this and that. But I think you have a lot of negative forces that are bringing it up that shouldn't be bringing it up." On Monday he added on Truth Social that the only guardrail AI needs is "a STRONG AND SMART (High IQ!) PRESIDENT," and that there is a "SICK conspiracy" against AI and data centers, one that only China is happy about. House Speaker Mike Johnson on CNN: Congress will not rush into emergency regulation, because "we will lose the race to China," and companies are the ones who have to take responsibility for the safety of their products.

The most interesting answer came from David Sacks, the White House AI and crypto czar until March and now co-chair of the President's Council of Advisors on Science and Technology, in a post with 9 million views: "Go ahead. You guys are the frontier. [...] The easiest way not to build superintelligence is for you to agree not to build it. Demanding your preferred regulatory framework as the price of that will look like blackmail of the public and the political system. So just do it." And further: stop pretending antitrust law has to be suspended so you can form a cartel, and stop pretending METR is independent when it is intertwined with Anthropic's investors and staff. Altman answered early Monday without naming him: OpenAI does "not believe we need to wait for an anti-trust exemption or legislation," and "when we talk about 'pacing', we do not mean 'stopping'."

Developers see something else

The discussion of the essay on Hacker News drew more than a thousand comments, and the phrase "regulatory capture" appears in it 29 times. The top comment opens with a question: "At what point do we stop engaging with Anthropic's leadership in good faith and acknowledge their track record?" Yann LeCun (1.6 million views): "Dario was already claiming that GPT2 was too dangerous to open source back in 2019. I made fun of them then. Everyone should make fun of them now." The prediction that draws the most mockery is the botnet in 6 to 12 months: "This is the only concrete prediction in the entire essay. And it simply cannot happen." A satirical post titled "Everyone should slow down AI development except for me" picked up close to 800 points on Hacker News, more than the essay itself.

On top of all this comes one number from the same weekend. According to the Financial Times, cited by Reuters, Anthropic told investors it will close a second consecutive quarter with positive adjusted operating income, and that gross margins exceed 80%. That figure comes before revenue shared with partners such as Amazon and before the cost of training the models. The company asking the industry to slow down is preparing to go public. The market read it its own way: on Monday SoftBank fell more than 10%, SK Hynix more than 6%, Micron more than 4%, while cybersecurity companies went up, CrowdStrike by double digits.

China answered on Monday through a foreign ministry spokesman: "Fear-mongering, confrontation and vicious competition will only hamper efforts toward sound global AI governance, which serves no one's interest." Amodei admitted on CBS that reaching an understanding with Beijing is the hardest part of the plan: "I don't know if it's possible, but we should try." In the essay he set a condition: companies in democratic countries can slow down by no more than the size of their lead over China. Helen Toner of Georgetown cites estimates of six to nine months.

So what does this mean if you build products on Claude or GPT models? In the models themselves, nothing for now. No company has announced it is halting training or launches, and Altman stresses that progress "should be slower than it otherwise could be," not that it stops. What changes is who gets to look inside: until now the companies wrote their own safety reports, and now two of them have promised to let outsiders in, with Anthropic additionally granting them the right to publish. Sacks is right about one thing: nobody is stopping them from slowing down today. A year from now we will see whether anyone actually took their foot off the gas.

Frontier - the leading edge, the most advanced AI models at a given moment and the companies that build them. Regulatory capture - when the rules end up written to suit the largest players in an industry. METR - an independent nonprofit in Berkeley that tests AI models for dangerous capabilities.