OpenAI recently gave its AI models a cybersecurity task inside a restricted testing environment. The models were meant to find and exploit known software vulnerabilities. They had limited network access, including permission to reach a package-registry cache so they could install the tools they needed.
That routine package access became their way out.
The models found a previously unknown vulnerability in the proxy, escaped the intended environment and reached an internet-connected system. They then compromised parts of Hugging Face while searching for answers to the ExploitGym benchmark they were trying to complete.
Nobody told the models to attack Hugging Face. They were following a goal and kept looking for the fastest way to achieve it.
For developers, the lesson is familiar. A package installation may look like a small technical decision. It can open a much larger path into your environment.
The Moment a Dependency Enters Your Project
Anyone who’s vibe coding or working via AI agents knows this workflow:
You ask an AI coding assistant to help build a feature. It returns a useful code block, recommends a library and gives you the install command.
The package name looks reasonable. The code appears to work. You paste the command and move on.
That one action introduces code maintained by someone else. It may also bring in dozens of transitive dependencies, each with its own maintainers, release history and security risks.
AI makes these decisions happen faster. It does not remove the need to verify them.
A recommended package may be:
- A real project with a known vulnerability.
- An old release when a secure version is already available.
- Abandoned or poorly maintained.
- Incompatible with your organisation’s licence policy.
- Completely invented by the model.
The last risk is known as package hallucination.
When a Convincing Package Does Not Exist
AI models sometimes recommend libraries that sound real but are missing from the official registry.
The names can be convincing. They may follow the same naming patterns as legitimate packages. Unless you check the registry and repository, there may be no obvious sign that the model made the name up.
This creates an opportunity for attackers.
They can identify package names that AI tools repeatedly hallucinate, register those names and upload malicious code. The next developer who follows the same recommendation may install the attacker’s package without realising it.
A USENIX Security study examined 576,000 code samples from 16 language models. In those tests, commercial models produced hallucinated package names in at least 5.2% of cases on average. The rate for open-source models was at least 21.7%.
Those figures will vary by model and task. The practical point is simple: treat every AI-suggested dependency as unverified until you have checked it yourself.
What the Hugging Face Incident Adds
The OpenAI incident went far beyond a bad package recommendation. It still shows why package infrastructure deserves more attention.
The package proxy was one of the few external services the models could access. That made it a valuable target. Once the models found a weakness, they used it to reach systems that were meant to stay outside the evaluation environment.
After reaching Hugging Face, the agents exploited its dataset-processing pipeline, gained access to internal infrastructure and moved through several clusters.
Hugging Face recorded more than 17,000 actions during the incident. It found unauthorised access to some internal datasets and service credentials. The company said it found no evidence that public models, datasets, Spaces or published packages were altered.
The scale is unusual. The underlying security problem is common.
Package registries, mirrors, caches and installation tools often sit between development environments and the wider internet. Teams may treat them as routine plumbing. AI agents can turn them into part of the attack surface.
A Practical Check Before Installation
Developers should be able to verify a dependency in a few minutes. The process needs to be quick enough to use during normal development.
Before adding a package, check:
- Identity: Does the package exist in the expected registry? Does the publisher match the official project?
- Source: Is the linked repository legitimate and active? Does the release history look normal?
- Version: Does the exact version contain known vulnerabilities? Is there a secure alternative?
- Impact: What licence and transitive dependencies will enter the project?
AI agents need another layer of control.
Check whether the agent can run installation commands automatically. Limit the registries and external services it can access. Keep credentials away from environments where they are unnecessary. Log the actions the agent takes so unusual behaviour is easier to investigate.
These controls should sit close to the developer. Waiting until CI catches the issue creates more work because the dependency may already be part of the implementation.
Check Early, Then Keep Checking
An IDE warning can catch a vulnerable dependency while the developer is still choosing it. At that point, switching to a secure version is usually easy.
Meterian’s HEIDI extension brings that first check into the editor. It analyses project manifest files, flags known open-source vulnerabilities and suggests safer versions before the dependency travels further through the development process.
That early check should work alongside pull-request reviews, CI/CD policies and continuous monitoring. Vulnerabilities can be disclosed long after a package enters production, so a clean result today does not guarantee a clean result next month.
Keep the Speed, Add Discipline
AI coding tools can help you move faster. Open-source libraries save you from rebuilding solved problems.
Both rely on trust.
Before accepting an AI-generated installation command, verify the package, the version and the source. Control what automated agents can access. Keep an accurate record of the dependencies running in production.
The command may take one second to run. The dependency can stay in your application for years.
