‘Time for pause’: Humans may lose control on AI as it starts building itself, warns Anthropic

Anthropic, the AI company behind Claude, has warned that AI could start building itself and create its successor and that “full recursive self-improvement might increase the risks of humans losing control over AI systems.” A full recursive self-improvement refers to a stage where AI is able to improve itself without any human intervention and create a more powerful successor.

The San Francisco-based firm also recommends a global pause on building the most powerful AI systems as the latest models are beginning to show signs they could evade human control.

The company said that as it increasingly delegates AI developments to AI systems, there are signals that it leads to a future where “an AI system (is) capable of fully autonomously designing and developing its own successor.”

Also Read: Delhi hotel owner helped Bangladeshi nationals get fake Indian documents, was arrested, booked last year

‘Average engineer producing eight times more code per day than in 2024’

Anthropic shared that Claude now writes most of the code used to build the company’s AI systems. As a result, engineers are able to get much more work done, with the average engineer producing around eight times more code each day than in 2024. Claude is not just getting better at coding, but it is also improving at research tasks.

Anthropic, an outlier among AI companies, said trends suggest AI systems are going to become much more capable ahead. “These trends have huge implications. AI that can build itself would be a major development in the history of technology—one that could bring enormous good for the world in science, healthcare, and beyond,” it said.

‘All AI companies need to come together to decide on pause’

It argued that all AI companies need to come together to decide to “pause” frontier AI development but warned that if only one company stopped, rivals would simply race ahead. Getting a real pause to work would mean all major AI companies in multiple countries agreeing to stop at the same time, under rules everyone could actually verify, Anthropic added.

“Without a global coordination mechanism, companies and governments will have to make difficult decisions about safety while under competitive and geopolitical pressures,” it said.

Also Read: ‘Futile to try pressuring PM Modi’: Putin says India-US relations do not affect ties with Russia

“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology,” it said.

Anthropic compared the problem to nuclear arms control treaties but said it would be even harder to monitor since AI training is far easier to hide than a missile silo, and the temptation to quietly keep going would be enormous.

Multiple challenges ahead for ‘pause’ proposal

The AI giant’s research arm, Anthropic Institute, plans to study and help build systems that would be necessary to support a slowdown. It also plans to organise discussions involving policymakers, researchers, civil society groups and other AI firms to examine key questions around how to tackle the issue at hand. These questions include how to manage AI-related risks such as recursive self-improvement and how to improve mechanisms for coordination.

Also Read: Former Trump NSA John Bolton to plead guilty over retaining classified documents: Report

The proposal would face an uphill battle in Washington and Silicon Valley, where US officials and tech executives have repeatedly argued that any slowdown in AI development risks handing China a decisive strategic edge in what many see as the defining technology race of the century.

President Donald Trump also signed an executive order this week that allows the government 30 days to conduct a preliminary review of the most powerful US AI models before their release.

The company stressed that such a stage has not yet arrived but argued that governments, regulators and society need to start preparing for it now.