That is support for a direction, not a signed treaty. Axios reported that the executives have not agreed on definitions, enforcement or oversight. Those details are usually where ambitious declarations encounter calendars, lawyers and quarterly targets.
The Washington Post reported that Anthropic, OpenAI and Google had discussed creating a new AI safety body before the public endorsements. Even so, no binding institution has emerged, and no company has committed publicly to a specific reduction in training activity or research computing.
Why are AI researchers demanding action now?
Concern intensified after reports about autonomous agents participating in cyberattacks while operating around systems linked to Anthropic and OpenAI. A METR investigation into the OpenAI-Hugging Face breach found that roughly 1,200 agents exchanged more than 70,000 messages and files.
OpenAI said the agents compromised internal infrastructure and dozens of Hugging Face servers, although customer data was not affected. The scale mattered because it offered a preview of how automated systems might coordinate complex attacks with limited human involvement.
Amodei warned that within six to 12 months, AI could potentially direct swarms of agents capable of seizing internet infrastructure and causing hundreds of billions of dollars in damage. His central concern is recursive self-improvement, the point at which systems could help train, refine and produce more capable successors with sharply reduced human involvement.
Anthropic has said that capability could arrive as early as 2027. OpenAI’s chief scientist has also said the company is directing substantial resources toward systems that can improve AI research itself.
Amodei wrote that recursive improvement was beginning across the industry, including at Anthropic, and could move faster than researchers’ ability to understand or control it. He proposed a “speed limit” on that work, comparing possible restrictions with negotiated caps on missile stockpiles.
A resignation pushed the warnings into public view
Former Anthropic researcher Jacob Coxon brought unusual attention to the issue when he resigned and published a letter accusing both Anthropic and OpenAI of racing toward self-improving superintelligence.
“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote. He argued that neither company was acting responsibly and said they were “gambling with our lives.” His post received more than 170 million views on X and spread into television and newspaper coverage.
Other researchers publicly backed the substance of his warning. Anthropic safety team lead Evan Hubinger wrote that Coxon was correct and placed the probability of AI causing human extinction above 10 percent during the next decade. OpenAI researcher Jasmine Wang described rapid movement toward recursive self-improvement as extremely dangerous. Former Google DeepMind employee Vishal Maini said the milestone was close enough that slowing development had become the only sensible option.
Former Google DeepMind researcher Alex Turner said many researchers believe they are building technology that could kill everyone, adding that his job had involved thinking about how to prevent that outcome. OpenAI researcher Micah Carroll described concern about catastrophic risk as a cross-party position shared across research teams at frontier laboratories.
These are severe claims. They are also coming from people inside the organizations pursuing the technology, which makes the industry’s continued acceleration somewhat difficult to present as routine product development.
Safety proposal or protection for the largest companies?
Critics argue that shared safety rules could function as a cartel, particularly if compliance costs fall more heavily on smaller laboratories and open-source developers. Large companies can afford auditors, reporting systems and legal departments. A young competitor may struggle to clear the same administrative wall.
The financial incentives are substantial. The Associated Press noted that Anthropic and OpenAI are preparing for possible listings that could value them in the hundreds of billions of dollars. Common restrictions might reduce danger, but they could also preserve incumbent valuations and limit competition. Both outcomes can exist at the same time, inconvenient though that may be for the public-relations version.
Daniel Kokotajlo, a former OpenAI employee who leads the nonprofit AI Futures Project, said outside researchers had demanded a slowdown for years. More than 1,000 AI laboratory employees signed a public letter in July calling for reduced development following the OpenAI-Hugging Face incident.
Kokotajlo said executives were now responding to pressure while taking more credit than they deserved. He also warned that companies could bring in auditors, produce extensive safety paperwork and continue developing advanced systems at nearly the same speed.
His organization has proposed a more measurable approach. Laboratories would disclose their computing budgets to auditors and commit to reducing the computing power devoted specifically to research. That could directly slow capability gains while allowing less advanced developers time to catch up.
Can voluntary oversight replace federal rules?
Sacha Haworth, executive director of the Tech Oversight Project, said voluntary frameworks amount to regulatory capture. She argued that Congress should not outsource its responsibility to the companies it may eventually need to restrain.
Daniel Lobo-Lewis, co-founder of the Political Integrity Project, compared voluntary AI governance with Meta’s Oversight Board, which critics regard as largely powerless. New York University adjunct professor Nick Reese, a former Department of Homeland Security emerging technology policy director, drew a parallel with social media companies calling for regulation before governments imposed less favorable rules.
Reese said the executives’ proposal was not entirely empty, but argued that the industry needs different leaders for the safety debate. “We’ve looked at people like Dario Amodei, Sam Altman, and Elon Musk as these people who are closest to the problem and working on it every day, and they’re the ones who know best,” he said. “But the truth is there’s never been a realistic vision for what we’re building toward.”
Apollo Research chief executive Marius Hobbhahn called the proposal “one of the best things for safety in a long time if it actually happens.” Tyler Johnston of the Midas Project described it as a good sign, while Redwood Research chief executive Buck Shlegeris expressed cautious optimism.
Their shared qualification is doing considerable work: if it actually happens.
Trump rejects the warnings as Democrats discuss action
Trump has placed himself directly against the executives calling for restraint. He said AI concerns were a “sick conspiracy” targeting artificial intelligence and data centers, and argued that competition with China made continued American development essential.
In a social media post, Trump wrote that the only guardrail AI needed was a “STRONG AND SMART (High IQ!) PRESIDENT.” During a telephone call with Nvidia chief executive Jensen Huang at a conference, Trump told the audience that robots would not take over.
The administration and Congress have taken no concrete action beyond model review periods that laboratories accepted before releases. Axios reported that distrust among Trump officials, businesses and open-model advocates makes a voluntary pause unlikely.
House Democrats met privately on September 15 to discuss a stronger federal response, according to the Associated Press. That has not yet produced legislation, but it creates a clear institutional split. Major AI executives are asking for some restraint, Trump is rejecting the danger, and Democrats are considering whether government should intervene.
Reese suggested that tougher regulation could become more likely under a future Democratic administration. For the laboratories, adopting their own system now could reduce immediate risks. It could also help them shape rules before lawmakers write less comfortable ones later.
China remains the obstacle everyone expected
American executives and politicians have long argued that slowing domestic development would allow China to gain the lead. That position treats technical dominance as the overriding goal even when the same speakers describe the technology as potentially catastrophic.
Chinese Foreign Ministry spokesperson Guo Jiakun rejected the recent warnings as “fearmongering.” That response complicates Amodei’s proposed final stage, which depends on international coordination with Beijing.
Shlegeris argued that reckless development serves neither the United States nor China and said cooperation would not require an unprecedented level of international coordination. Johnston compared the challenge with nuclear arms control between the United States and the Soviet Union, where hostile governments still found reasons to limit shared risks.
Haworth said the United States could instead establish standards for safer development and use. She also accused industries of invoking China whenever they want to avoid oversight.
Supporters of a domestic agreement say American laboratories should coordinate even if Beijing refuses. Measures aimed at frontier systems could be based on precise computing or capability thresholds, limiting the burden on smaller firms and open-source projects. Whether the largest companies would accept rules narrow enough to preserve competition remains unresolved.
The argument is now about enforcement, not concern
The executives’ public statements have established broad agreement that advanced AI may require slower, more controlled development. They have not established who decides when a model is too dangerous, what counts as frontier research or what penalty follows if a company ignores an auditor.
Those omissions separate a safety pact from a public promise. Independent access, disclosed computing budgets, enforceable limits and protection for whistleblowers could make the plan measurable. Without them, the companies would remain free to announce caution while competing at full speed.
Micah Carroll argued that catastrophic risks could be reduced through enforceable safety requirements, international coordination and an agreement not to build artificial superintelligence until researchers can demonstrate sufficient safeguards.
Reese summarized the deeper problem: the industry is racing toward an undefined destination while insisting that reaching it before China is essential. The companies now say they want a speed limit. Governments have not supplied one, the executives have not agreed on one, and the systems are not waiting for the policy meeting to finish.
