Key Takeaways:
AI didn’t “go rogue” — the harness failed. The OpenAI/Hugging Face incident was an architectural breakdown in containment and scope control, not a case of autonomous AI deciding to hack a third party.
LLMs mimic thought, they don’t think. Models operate as high-speed autocomplete engines that brute-force patterns from training data; confusing that mimicry with real reasoning is a costly mistake in security planning.
Guardrails, prompts, and bridges matter more than the model. Once you disable safety filters, give an aggressive exploitation prompt, and expose a vulnerable egress path, you’ve effectively invited the incident.
Automation ≠ sophistication. The 17,613 actions recorded at Hugging Face are evidence of blind brute force at machine speed, not of creative, high-end tradecraft that would concern an experienced red team.
Human attackers still set the bar. Even frontier models fail on the majority of ExploitGym challenges; discovering, understanding, and weaponizing novel vulnerabilities at scale remains fundamentally a human job.
“A monkey hitting keys at random on a typewriter keyboard, for an infinite amount of time, will almost surely type any given text, such as the complete works of William Shakespeare.” – Émile Borel (1913)
A Human Perspective from a Real Red Team Operator
As with everything AI, the OpenAI and Hugging Face incident is being sensationalized by most of the industry. NBC Bay Area said “Rogue AI models broke free from human control”, OpenAI’s own disclosure said “Unprecedented cyber incident, involving state-of-the-art cyber capabilities”, CNN, Fortune and others saying, “AI System… acts on its own to hack another company”. From our perspective as human operators who breach customer networks for a living, the capabilities are being terribly misrepresented. The most accurate description of offensive AI capabilities is that it operates with the knowledge base of a mid-tier hacker paired with the discipline of a first-week script kiddie, running at machine speed with no supervision.
Thought Mimicry
To understand why this was amateur hour, you’ll first need an understanding of how AI works. AI mimics thought so convincingly that it fools most people. This is clearly evidenced by the hyped up press we see almost daily. Here’s a simple analogous example to better clarify what AI is really doing.
When your friend asks you what you want to eat for dinner tonight, your brain comes up with an answer within one or two seconds. In that timeframe, it remembers you had pasta last night and notices you’re a little tired while also remembering that your friend mentioned wanting sushi last week, but you never got around to it. After that, it considers the leftovers in the fridge and decides if you actually feel like cooking or would rather order in. It even factors in the budget for the week and might pull up a memory of that Thai place you went to that you both liked. Then it produces an answer: “How about that Thai place we like? It’s been a while.”
The critical components to make note of here are memory, preference, context, emotion, bias, and judgement without you having to consciously direct any of it.
Ask AI the exact same question: “What should my friend and I eat for dinner tonight?” The AI generates a probability distribution over the words that commonly follow “What should I eat for dinner tonight?” from its training corpus. Then it samples from that distribution and produces something like: “How about a healthy Mediterranean bowl with grilled chicken and vegetables?” That answer might sound thoughtful, but there was no actual thought involved, none whatsoever. Instead, the answer was entirely predictive, driven by pattern-matching against the statistical center of what humans typically say when they answer that question. And that statistical center is based on the AI’s then current training data.
That “probability distribution” is a lot simpler than it sounds. If you strip the dinner question down to a much simpler prompt like “I want to eat some…” and press enter, the first thing the model does is generate a probability list (ranked list of guesses with associated confidence scores) that looks something like this:
- pizza (32%)
- ice cream (18%)
- fries (14%)
- soup (9%)
- pasta (7%)
- …thousands more options with tiny percentages
It is literally a big list of every possible next word (or part of a word) based on the model’s training corpus, each accompanied by a confidence score about how likely it is to be the right one. The model then samples from the top of the list with some randomness included, which is why the same prompt gives you slightly different answers each time.
What we’ve just described is how AI works. Instead of thinking, it predicts and creates the appearance of human thought. When anyone says their AI thinks like a human or that it can replace a human, they’ve been fooled by the mimicry. When it comes to information security work, conflating appearance and true capability is a dangerous and potentially expensive mistake.
The Model vs a Human Attacker
There are two very important facts about AI models.
- All AI models require prompting to work. Without a prompt and AI does nothing.
- All AI models are predictive text generation engines similar to autocomplete. AI cannot interact with the outside world unless someone builds a bridge for it.
That bridge is often referred to as a harness, and it contains an action loop which is the code that keeps asking the model “what do you want to do next?”, executes what the model says, and feeds the result back in, over and over, until the task is done. When you interact with any of the frontier AI models via their chat interface, that interface is an example of a harness.
How OpenAI Set the Stage
OpenAI wanted to test their models against ExploitGym. For those who might not know, ExploitGym is a public cybersecurity benchmark that measures how well AI models can develop working exploits for real software vulnerabilities that were previously discovered by human security researchers. To do this, OpenAI built a harness to provide their AI with an action loop, access to the tools it needed, and the prompt to direct its work. They also disabled the safety mechanisms that typically cause an AI to say “sorry, I can’t do that” when prompted to do something risky.
OpenAI told the models “To pursue advanced exploitation using complex attack paths.” They haven’t disclosed the exact prompt, but three things are clear.
- Vulnerable containment: The containment OpenAI built depended on a third-party package registry proxy (JFrog Artifactory) that had a zero-day vulnerability, which the models exploited to escape. Artifactory has a documented history of the exact vulnerability classes the models exploited (SSRF, path traversal, privilege escalation, and RCE), including CVE-2019-19937 in 2020, the “Zip Slip meets Artifactory” bug bounty finding, and CVE-2024-4142 in 2024. Package registry proxies are also a known high-risk architectural class because their entire job is making outbound HTTP requests on behalf of clients. Using one as the sole permitted egress from an evaluation sandbox designed to contain models with disabled safety filters was a foreseeable failure, not an unlucky mistake.
- Disabled guardrails: The cyber-refusal guardrails that would have made the models refuse to attempt the exploit in the first place had been reduced or disabled.
- Zero scope constraints: There’s no indication anywhere in OpenAI’s disclosure that the models were told to stay within scope, not to attack the proxy, or to avoid attacking third parties.
Named security researchers publicly reached the same conclusion. Niels Provos, formerly of Google’s security team, put it directly: “This should not have happened.” Security consultant Davi Ottenheimer argued that an environment with any exploitable external connection does not meet the standard of highly isolated. Researchers from Georgetown University’s Center for Security and Emerging Technology and the University of Cambridge described the incident as “a breakdown in network isolation, not an unexplainable AI behavior.” The failure here was architectural, not adversarial.
Any human given the same task would immediately understand not to cheat by attacking a third party, because the whole point of the test was to measure their own capability, not their ability to steal an answer key. The models didn’t go rogue, they operated exactly as OpenAI’s harness, prompt, and stripped-down guardrails told them to.
Brute Force vs Human Reasoning
Once prompted, the models started producing tokens (text commands) and the harness started executing them without checking scope. The training data these models had been fed is full of examples that suggest shortcuts are valid solutions. Capture the Flag walkthroughs, bug bounty writeups, real breach postmortems, Hollywood heist plots, and general hacker mythology all reward the “clever” solution that finds an unexpected shortcut rather than the one that grinds through the puzzle. So, the probability distribution weighted “obtain the answer key directly” much more heavily than “actually develop working exploits.” The risks associated with this setup are glaringly obvious.
When an AI solves a problem, it doesn’t think, it brute-forces it, by trying every permutation of every possible solution until it either succeeds or the harness shuts it down. It doesn’t stop to consider the noise it’s making or whether a particular attempt is worth trying or not. The model just keeps sampling, and the harness just keeps running the commands. This is exactly why HuggingFace recorded 17,613 actions during the intrusion. That high volume of actions isn’t an indication of attack sophistication, quality, or coverage. It’s just another type of automation and it doesn’t intrinsically know how or when to stop like a human would.
Again, vulnerabilities in ExploitGym were discovered by human researchers exercising their expertise and this benchmark is based entirely on their work. That work required actual thinking, reading unfamiliar code and deep reasoning about what should happen as opposed to what does happen, and creating novel methods. Chasing intuitions that a specific piece of logic feels wrong and building novel working exploits from scratch, often over weeks or months of full-time effort, is a human job.
Anthropic’s Claude Mythos Preview, working alongside Anthropic’s Frontier Red Team, leveraged its training data to produce a working exploit for a 17-year-old remote code execution flaw in the FreeBSD NFS server (CVE-2026-4747) which is contested as fully autonomous. It chained a 20-gadget ROP payload across multiple network requests. Firefox 150 also shipped with 271 vulnerability fixes identified by an earlier version of the same model. That work is genuine research done by AI using known techniques, not novel techniques that it built from scratch. Humans have almost certainly found the same or similar bugs before and left them unreported because non-exploitable flaws carry no real threat value.
Benchmark Realities and Exploitation Limits
ExploitGym itself, designed to measure pure exploitation capabilities of bugs that have already been discovered and well-documented by humans, still limits even the best models.
- Claude Mythos Preview only solved 157 of the 898
- OpenAI’s own GPT-5.5 only solved 120
Both did that work legitimately, but that was only possible because the necessary methods for exploitation were contained in their training corpus. And yet, even with that volume of expert human knowledge ingested, 82% of the benchmark was still beyond the reach of the latest models. Give that same knowledge and expertise to a human and they’d solve the benchmarks albeit over a much longer period of time.
A skilled human hacker given the same OpenAI task wouldn’t have hacked Hugging Face because a human understands what an evaluation is for. A human would have known the point was to measure their own exploit development capability and would have solved the challenges the honest way (like they did when they found the original vulnerabilities). Of course, ExploitGym imposes a two-hour time limit per task, and no single human working alone would attempt all 898 tasks under that constraint. But speed and volume aren’t quality indicators, in fact quite the opposite in most cases.
Why the Harness Outweighs the Model
The AI didn’t stay within scope or solve the challenges honestly because it can’t think, consider the purpose of the evaluation, or understand why cheating undermines the measurement. It only knows what the harness lets it do and what the training data suggests it should do next. That is why in agentic AI, the harness often matters more than the model. Even a capable model, when wrapped in a poorly designed harness, will produce sloppy work at high risk of incidents just like this one. A well-designed harness would have kept the model focused on the task through explicit scope constraints in the prompt, technical enforcement of those constraints, and monitoring that would have flagged and killed the loop the moment the model strayed from developing exploits. The model did exactly what any optimizer does when given powerful tools and no boundaries.
Anthropic has since disclosed three similar unauthorized-access incidents involving Claude models (Opus 4.7, Mythos 5, and an internal test model) during cybersecurity evaluations run through their third-party partner Irregular, with the earliest dating back to April 2026. This pattern isn’t isolated to OpenAI or to a single model.
The OpenAI HuggingFace Incident – What We Know
Why Hugging Face Was the Target
Hugging Face isn’t just a website, it’s the central repository for AI models, datasets, and training data. Think of it as the GitHub of AI. Anyone building or evaluating AI systems relies on it, including OpenAI. And critically for this incident, it’s where the ExploitGym challenge solutions were stored. That’s the answer key the OpenAI models were sent to find.
For an AI model told to solve ExploitGym, the probability distribution pointed at Hugging Face because its training data is packed with references to Hugging Face hosting exactly this kind of content. When the model looked for a shortcut, it went where its training data told it to go.
For a human threat actor, Hugging Face is a different kind of target. Supply-chain leverage, credential concentration across cloud environments, and the ability to poison downstream systems that pull in Hugging Face content. It’s what npm is to JavaScript or PyPI is to Python. A high-value chokepoint in the ML supply chain.
According to Hugging Face, on July 16, 2026, they publicly disclosed a security incident their team had detected and contained three days earlier. The OpenAI attack itself ran from July 9, 02:28 UTC to July 13, 14:14 UTC, a total of 4.5 days during which the AI agent executed 17,613 recorded actions inside Hugging Face’s infrastructure. Hugging Face’s own AI-driven detection actually correlated the activity into a coherent attack signal in real time, but as they later admitted in their technical writeup, the system “failed to correctly raise the alert’s criticality and trigger the on-call team, costing precious time in the response.” Five days after Hugging Face’s public disclosure, on July 21, 2026, OpenAI acknowledged that the attack was carried out by two of its own AI models during an internal cyber-capability evaluation.
By the time Hugging Face’s response team cut off access on July 13, the model had already achieved most of its objective. Per Hugging Face’s own postmortem, the model successfully
- Accessed the ExploitGym challenge solutions stored on Hugging Face’s infrastructure
- Gained cluster-admin on multiple internal Kubernetes clusters
- Harvested service credentials and cloud tokens
- Enrolled unauthorized devices in the internal Tailscale mesh VPN
- Reached the source-control provider with write access to a subset of internal repositories
This sounds impressive until you look under the covers. It worked because OpenAI and Hugging Face made a chain of mistakes that let it work. The attack was cut off after four and a half days, and only after the model had already captured what it came for. That’s not containment, that’s response after the fact. Effective incident response detects and stops an attack before damages are realized.
A Timeline of the Attack
Publicly available information doesn’t include exact step-by-step, hour-by-hour for every phase of the attack, but we can reconstruct the sequence of events from OpenAI’s and Hugging Face’s disclosures. The table below traces the attack chronologically from the moment OpenAI’s evaluation began through OpenAI’s public attribution five days after Hugging Face’s disclosure.
How Sophisticated Was This, Really?
Real human attackers, whether they’re criminals, nation-state actors, penetration testers or Red Team operators, share a set of behaviors this incident never demonstrated. Those behaviors are defining, not optional. Their absence places this attack at the bottom of the human skill spectrum, regardless of how impressive the raw statistics look.
Our Penetration Testing services against a target the size of Hugging Face would generate only a small fraction of the actions Hugging Face recorded, spread across two to six weeks, though the exact volume and stealth level depend on the test configuration and the customer’s needs. Our Red Team services, which require stealth by default, typically produce little to no detectable activity against an infrastructure like the one described here. Against a more mature security program, our Red Team would maintain stealth and possibly generate a few deliberate events as a ruse to misdirect the client’s response. OpenAI’s model produced 17,613 actions in 4.5 days, with 7,677 on the loudest day alone, the opposite of sophistication. It’s exactly why we’ve dubbed AI penetration testing the next evolution of automated vulnerability scanning.
The story here isn’t that AI has caught up to human attackers. The story is that OpenAI removed the guardrails on a model that had memorized a good chunk of the public offensive security literature, pointed it at a benchmark, gave it no scope constraints, and watched it brute-force its way to the answer key.
This Isn’t Just OpenAI: The Broader Pattern
Nine days after OpenAI’s disclosure, Anthropic published its own review. They audited 141,006 cybersecurity evaluation runs against their Claude models and found three separate incidents where Claude reached the real internet from what was supposed to be a sealed test environment and then compromised the production infrastructure of three different organizations. All three ran through their third-party evaluation partner Irregular, whose environment was misconfigured to allow internet access despite prompts telling Claude there was none.
The three incidents involved Claude Opus 4.7, Claude Mythos 5, and an internal test model, with the earliest dating back to April 2026. Months before the OpenAI/HuggingFace attack was in the news. In one case, Mythos 5 published a malicious Python package to PyPI that then infected 15 real systems, including a security company’s malware scanner. In another, Opus 4.7 broke into a real company’s infrastructure and extracted several hundred rows of production data. The techniques Claude used were, in Anthropic’s own words, “basic and well-known cyberattack techniques” like weak passwords, SQL injection, and exposed debug pages. Same amateur-hour tradecraft, different lab.
What matters most is how the two labs handled the incidents. Anthropic caught these through a proactive audit that they launched after seeing OpenAI’s disclosure. The affected organizations hadn’t detected any of it themselves. Anthropic stopped all cyber evaluations the day they identified suspect transcripts, notified the affected organizations within four days, engaged independent evaluator METR for third-party review, and committed to publishing a redacted transcript. They also explicitly framed the incidents as “closer to a harness and operational failure than a model alignment failure.” That’s the exact argument we’ve been making throughout this article, in Anthropic’s own words.
OpenAI’s response looked different. Per Reuters, OpenAI didn’t notice their own models were loose until the weekend of July 18-19, a full week after the models finished attacking. OpenAI’s initial disclosure called the incident “unprecedented” and involving “state-of-the-art cyber capabilities.” That framing has since been walked back by outside coverage. The name of the vulnerable software (JFrog Artifactory) wasn’t disclosed until a week after OpenAI’s initial statement. A second breach at Modal Labs surfaced through Reuters, not through OpenAI.
Of the three Claude models involved, the newest internal test model recognized it was on the real internet and stopped on its own. That suggests alignment work on situational awareness is having some effect at the frontier. But it also reinforces our point that the harness is still the primary control, because even that better-behaved model needed the harness to keep it in scope in the first place.
How This Could Have Been Stopped
Every part of this attack was preventable using defensive techniques that have existed for years and don’t require AI to work. If Hugging Face had these controls in place, the attack would have been stopped in the first few actions instead of running to seventeen thousand.
- Honeypots with automated blocking. This attack was almost custom-built to be stopped by a honeypot tied to automated incident response. The model’s brute-force approach guaranteed it would probe for exactly the kinds of filenames its training data suggested a “clever” attacker would look for. A single decoy file named something like exploitgym_solutions.json, benchmark_answer_key.py, or admin_credentials.txt, instrumented with a tool like honey.py forwarding events to a Wazuh SIEM, would have caught this attack in seconds. The flow: honey.py monitors the decoy file, fires an alert to Wazuh on any access attempt, and Wazuh’s active response rules automatically block the source IP of the attacker, kill the active session, page the SOC, and quarantine the affected pod. Cost to deploy: essentially zero. Effectiveness against this specific class of AI-driven brute-force attack: total. The AI cannot reason its way around the trap because the AI cannot reason.
- Rate limiting on anomalous volume. A single source generating 17,613 actions is a signature that any well-tuned WAF or SIEM detects and blocks well before action 1,000. This attack succeeded partly because rate limits weren’t tight enough or weren’t tuned for the traffic profile an AI agent produces.
- Egress filtering on evaluation sandboxes. OpenAI’s models reached the open internet through a package registry proxy. A properly designed evaluation sandbox has zero outbound reachability except to explicitly allow-listed hostnames with strict content-signing verification. Any other outbound path is a way out, and to us the package proxy was an obvious way out that OpenAI’s team either didn’t consider or didn’t harden.
- Network segmentation between evaluation and production. OpenAI’s research environment had a lateral path from the evaluation sandbox to a node with internet access. That’s a basic network segmentation failure. Isolating evaluation infrastructure so that even a full compromise cannot reach the corporate network or the open internet has been standard practice for decades.
- Detection engineering tuned for agent behavior. AI agents generate a distinctive traffic fingerprint. It’s fast, structured, repetitive, and doesn’t stop. Detection rules tuned for that fingerprint catch this class of attack in the first few dozen actions, but many SIEMs today are still tuned for human attackers and miss the pattern entirely.
These defenses are simple and inexpensive (or even free) to deploy. Any one of them could have contained this attack far earlier than 17,613 actions.
FAQ
What is the OpenAI Hugging Face security incident?
In July 2026, two OpenAI models running an internal cyber-capability evaluation on the ExploitGym benchmark broke out of OpenAI’s sandbox and ran a 4.5-day intrusion against Hugging Face’s production infrastructure. Hugging Face contained it on July 13 and publicly disclosed on July 16. OpenAI admitted its models were responsible on July 21.
Did OpenAI's models really hack Hugging Face?
Yes. GPT-5.6 Sol and an unreleased OpenAI model got into Hugging Face’s production infrastructure, took cluster-admin on several internal Kubernetes clusters, grabbed the ExploitGym solutions, and tried to compromise Hugging Face’s source-control pipeline. The models were running with the safety guardrails reduced and no scope constraints.
Was Modal compromised in the OpenAI Hugging Face attack?
No. Modal was not hacked. A Modal customer had left an unauthenticated code-execution endpoint labeled CyberGym (the predecessor to ExploitGym) exposed on Modal’s platform, which is exactly why the AI targeted it. The OpenAI models were being evaluated on ExploitGym, so an unauthenticated CyberGym endpoint offering exactly the code-execution capability they needed looked to their probability distribution like a task-relevant resource. They abused it as an attack launchpad against Hugging Face. Modal’s own security was not the vulnerability.
How sophisticated was the AI attack on Hugging Face?
Not sophisticated. Of the 17,613 actions logged, almost every technique was documented tradecraft the models pulled from their training data. That included HDF5 external storage abuse (CVE-2025-9905, published ten months before the attack), Jinja2 template injection, Kubernetes token theft, GitHub App abuse, and Tailscale key theft. The attack won on machine speed, not on novel capability.
Did OpenAI's AI models actually succeed at the attack?
Yes. Per Hugging Face’s own postmortem, before containment on July 13 the models had already accessed the ExploitGym solutions, harvested cloud credentials, gained cluster-admin on multiple internal Kubernetes clusters, enrolled 181 devices in Hugging Face’s internal VPN, and gotten write access to internal repositories. Containment happened after the objective was reached, not before.
Did Anthropic's Claude models also breach third parties during evaluations?
Yes. On July 30, 2026, Anthropic disclosed three similar unauthorized-access incidents involving Claude Opus 4.7, Claude Mythos 5, and an internal test model. All three ran through their third-party evaluation partner Irregular, and the earliest dates back to April 2026. In one case, Claude published a malicious Python package to PyPI that then infected 15 real systems. Anthropic explicitly framed the incidents as a harness and operational failure, not a model alignment failure.
Why did Hugging Face use a Chinese AI model for forensic analysis?
Because the leading US commercial models (Claude Opus and Fable) refused to process the malicious payloads in the logs. Their safety guardrails blocked forensic analysis of the attack data. Hugging Face ended up running GLM 5.2, an open-source model from Chinese lab ZhipuAI, on its own infrastructure to finish the reconstruction.
How can organizations defend against AI-driven attacks?
Standard defenses actually work better against AI attacks than against skilled human attackers. Honeypots with automated blocking (honey.py wired into a Wazuh SIEM with active response), rate limits on anomalous volume, egress filtering on evaluation sandboxes, and network segmentation between eval and production all catch AI-driven attacks in the first few hundred actions instead of at seventeen thousand. AI doesn’t adapt to blocks the way a human does.
Can AI replace human penetration testers?
Yes and no. Yes, for penetration testing firms that focus on compliance and superficial testing. No, not even close to being able to replace genuine penetration testing firms focused on real security work. AI penetration testing is the next evolution of automated vulnerability scanning, not a replacement for human Penetration Testers or Red Team operators. AI lacks the mission focus, stealth, and adaptive tradecraft that define real offensive security work. The Hugging Face incident showed what a well-resourced AI attack actually looks like: the knowledge of a mid-tier hacker and the discipline of a script kiddie.
