This week was filled with doomsday concerns about human extinction caused by powerful AI models. Two Anthropic researchers fanned the flames of this controversy. Jacob Coxon quit Anthropic and then went on nearly every news program to discuss why, outlining doomsday scenarios of AI causing human extinction such as by enabling bad actors to produce extinction-level biological warfare agents. To add fuel to the fire, Anthropic’s , Anthropic’s Evan Hubinger, the Lead for Alignment Science, agreed that AI could lead to human extinction within the decade, presumably, by 2030.
Their concerns did not seem outlandish after Anthropic issued its September 2026 report on the serious threats that the company stopped from malevolent uses of its AI model by bad actors.
Today, CEO Dario Amodei has responded to those concerns. He lays out a 3-step plan calling for coordinated “pacing” in developing the most powerful frontier models, not just with U.S. companies, but internationally. Elon Musk expressed his support on X.
Excerpt:
The first step is something Anthropic is unilaterally committing to (and calls on governments to require other frontier companies to match). The second step requires industry-wide coordination.1The third step requires global coordination. The steps do not need to be taken strictly in order, and some of them may be much harder to achieve than others, but I’ve found them to be a useful framework in thinking about what needs to be accomplished. The steps are:
- Embedded Evaluators. Each frontier AI company commits to giving ongoing, employee-like access to a team of embedded third-party evaluators (such as METR), whose role is to verify adherence to safety practices and commitments, report incidents, and help assess the alignment of not just completed AI models but training pipelines and processes. This is the key step for verifiability of any pacing commitments, and has precedent in the banking industry, which sometimes involves regulatory “supervisors” embedded along with employees. Anthropic is unilaterally committing to this step now. We intend this to be part of a broader push to redouble efforts on our safety and alignment work.
- Democratic Coordination. Frontier AI companies within democratic countries coordinate to establish common safety standards as well as limits on the rate of unchecked AI progress. Some forms of coordination that would be impactful for pacing are legally challenging, and will require government support.
- Global Coordination. The US and other democratic governments attempt to coordinate with authoritarian governments, to the extent this is possible, while taking seriously the challenges of verifying compliance.
