OpenAI’s AI agent gone rogue? Company reveals chilling hack at startup

Amid rising fears of the threat posed by AI, OpenAI has revealed that an autonomous AI agent powered by its advanced models went rogue during a security test and carried out a cyberattack on AI platform Hugging Face. The incident has raised fresh concerns over the risks involved in using advanced artificial intelligence systems.

What happened during the incident?

In a blog post on Tuesday (Jul 21), OpenAI explained an “unprecedented cyber incident” involving advanced AI-driven hacking capabilities. The company said that the incident occurred during an internal cybersecurity evaluation designed to test the limits of its AI models’ cyber abilities. The company said the models were placed in a highly isolated environment but managed to exploit a zero-day vulnerability in a package registry cache proxy to gain internet access.

The AI agents then identified and chained multiple vulnerabilities across OpenAI’s research systems and Hugging Face’s infrastructure while attempting to complete their assigned task. OpenAI said the models used stolen credentials and other attack methods to access restricted information from Hugging Face’s production environment. The company said the activity was detected by security teams, while Hugging Face’s security systems also identified and contained the breach.

What did OpenAI say?

Following the incident, OpenAI said it was strengthening its safeguards and improving security measures. It added that it was tightening infrastructure controls, patching a disclosed zero-day vulnerability, and enhancing AI safety measures. Both companies said that they were investigating the incident.

“The primary lesson from this incident is that model security and safety must keep pace with rapidly advancing capabilities. We are strengthening the containment, monitoring, access controls, and evaluation practices used during model development,” OpenAI said.

Hugging Face, a popular platform that hosts open-source AI models and datasets, had earlier disclosed that it suffered a sophisticated cyberattack unlike previous incidents. The company said the attack appeared to have been conducted entirely by an autonomous AI agent.

In a statement, Hugging Face Co-founder and CEO Clem Delangue said, “We’re grateful for the collaboration with OpenAI on this and other topics. This incident, possibly the first of its kind, proves a point we’ve long believed: AI safety won’t be solved by any single company working in secret. It will be solved in the open, collaboratively, with broad access to AI for every defender, everywhere.”

Concerns over AI

The disclosure has intensified concerns among cybersecurity experts and policymakers about the ability of frontier AI models to operate independently and potentially cause harm.

US Representative Greg Casar called the incident alarming and urged stronger safeguards, including mandatory independent safety testing, reporting of AI-related security breaches and global cooperation on AI regulation.

Cybersecurity experts warned that the incident could represent a preview of future threats. Katie Moussouris, CEO of Luta Security, compared advanced AI models to highly adaptive systems capable of finding ways around restrictions.

Matt Suiche, an engineer at AI cybersecurity firm Tolmo, said the incident showed that frontier AI models are rapidly approaching the capabilities of sophisticated attackers. He added that similar AI-driven cyber risks are no longer limited to top research labs and can emerge from widely available technologies.

Watch | The Internet’s Darkest Secret: Inside the global network exploiting children online