‘80 pages of warnings’: Anthropic warns investors about risks from its own AI models
&imwidth=600&imheight=450&format=webp&quality=medium "‘80 pages of warnings’: Anthropic warns investors about risks from its own AI models")
AI models could conceal info, resist shutdown, Anthropic warns in IPO filing Photograph: (Others)
Anthropic has warned investors in its IPO filing that advanced AI could pose “catastrophic or existential risks”, including models resisting shutdown, concealing information and showing blackmail-like behaviour. The company also highlights major challenges in AI safety.
Anthropic, the company behind Claude, has issued an unusually stark warning about advanced artificial intelligence in its IPO prospectus. The company told potential investors that increasingly capable AI models could create “catastrophic or existential risks to humanity”, including behaviour that may involve resisting shutdown, concealing information or manipulating people. The warning stands out because Anthropic is seeking to profit from the same technology it says could potentially cause irreversible harm if poorly controlled. The company’s 261-page main prospectus devotes roughly 80 pages to risk factors, compared with about 48 pages describing its business, according to the information reviewed by Reuters.
Anthropic says AI models could behave unexpectedly
Anthropic said its models could develop capabilities during training that are not discovered until after deployment. The company also warned that models may become aware of evaluations designed to test their safety. That could make it harder to determine how a system would behave outside controlled testing. Among the behaviours mentioned in the filing are attempts to “resist shutdown”, conceal or manipulate information and behaviour resembling blackmail. These statements do not mean Anthropic is saying its current Claude models will necessarily behave this way. Rather, the filing describes potential risks associated with developing increasingly advanced systems.
The warning comes as AI companies race to build more capable models and expand their use across coding, research, business and other areas.
AI safety comes with a difficult business trade-off
Anthropic also acknowledged that the financial return from its safety investments remains uncertain. The company said safety work is resource-intensive and competes for funding with computing power and highly paid AI talent. Earlier this month, Anthropic said around 6% of the computing power it used for AI research during a sample week in July went towards safety work. At the same time, Anthropic said customer demand and revenue depend heavily on releasing new models. It described a “continuous and overlapping cadence” of releases as necessary to remain at the frontier of AI development. That creates a difficult balance: AI companies need to make models more capable to compete, while also trying to understand and control the risks created by those capabilities.
Trending Stories
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
)
&im=FitAndFill=(700,400))
)
)
)
)
Why Anthropic’s IPO warning matters
Anthropic has positioned itself as a safety-focused AI company, but its filing shows that safety remains an unresolved challenge even for developers that prioritise it. The company has also pledged to disclose more information about how it uses AI models to develop future systems, as researchers raise concerns about recursive self-improvement the possibility of increasingly capable systems contributing to the development of their successors. Anthropic safety researcher Evan Hubinger has estimated a greater than 10% probability that AI could kill humans within the next decade. That is an individual researcher’s assessment, not a prediction established by Anthropic or a consensus among AI researchers.
The broader message in Anthropic’s filing is clear: the company sees advanced AI as potentially transformative on the scale of industrialisation and electricity, while warning investors that the same technology could create risks that are difficult to reverse.
