A rogue OpenAI agent is responsible for a completely autonomous attack on Hugging Face’s production infrastructure.
The AI models responsible were being tested at the time, and managed to hack into the popular digital library, compromising some internal datasets and service credentials.
OpenAI states that the incident was driven by a combination of OpenAI models including GPT‑5.6 Sol and a more capable pre-release model.
The models had reduced cyber refusals for evaluation purposes and were being internally tested on a benchmark of cyber capabilities.
These benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.
However, it appears that the models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database.
“All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal,” according to an OpenAI statement.
“While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem.
“To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy. With this access, the models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.
“After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation.”
Open AI discloses that, in one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers.
Fortunately, Hugging Face detected and stopped the activity on their infrastructure.
The two organisations are now working together to investigate the breach and close the loopholes.
“We’re grateful for the collaboration with OpenAI on this and other topics,” says Clem Delangue, co-founder and CEO of Hugging Face. “This incident, possibly the first of its kind, proves a point we’ve long believed: AI safety won’t be solved by any single company working in secret. It will be solved in the open, collaboratively, with broad access to AI for every defender, everywhere.”
Featured pic is AI-generated using Adobe Firefly