Open AI Models Found Vulnerabilities Like Mythos

# Open Models Reproduce Vulnerabilities Found by Closed AI System Mythos

Researchers from Vidoc Security confirmed that modern open and commercial language models can reproduce the results of Anthropic's closed Mythos system in detecting vulnerabilities. This casts doubt on the claim that access to advanced AI models needs to be restricted due to their supposedly unique ability to find critical bugs.

Verifying Anthropic's Claims

In April 2026, Anthropic announced Project Mythos — an internal model aimed at finding vulnerabilities in software. As proof of its effectiveness, they cited five public examples: CVEs in FreeBSD, OpenBSD, FFmpeg, Botan, and wolfSSL. Particularly notable was a 27-year-old bug in OpenBSD's TCP stack, which the company claimed was undetectable by existing code analysis tools.

However, Vidoc researchers decided to check how truly unique these findings were. They used two publicly available models — GPT-5.4 and Claude Opus 4.6 — in conjunction with the open-source agent opencode, without access to Anthropic's internal data or proprietary tools.

Reproduction Results

Testing showed the following:

• FreeBSD (CVE-2026-4747): both models accurately identified the vulnerability.
• Botan (CVE-2026-34580/34582): full match across all three cases.
• OpenBSD: only Claude Opus 4.6 reproduced the finding; GPT-5.4 did not detect the issue in any run.
• FFmpeg and wolfSSL: the models pointed to the correct location in the code but misassessed the impact — for example, classifying a potential overflow as safe behavior.

Thus, out of the five publicly claimed vulnerability categories, three were fully reproduced, two partially. This means that Anthropic's key demonstration examples do not require exclusive access to a closed model.

What Does Anthropic Say About This?

The company claims that Mythos discovered "thousands" of vulnerabilities, 99% of which are under embargo until patches are released. This part cannot be verified without access to internal data. However, it was precisely the public examples that Anthropic used as the main argument for restricting the distribution of frontier models.

If even basic tasks can be solved using publicly available AI, then the logic of "only we can do this safely" loses weight. This is especially important in the context of AI regulation: if the threat comes not from the technology itself but from its opaque control, then secrecy may create more risks than benefits.

Technical Details of Vidoc's Approach

The Vidoc team used opencode — an open-source framework for automated code analysis involving LLMs. The agent received:

• A code fragment with the patch.
• Description of changes from the commit message.
• Project context (e.g., protocol specification or API documentation).

Models were tasked with:

• Determining if the change was a vulnerability fix.
• Specifying the vulnerability type (e.g., integer overflow, use-after-free).
• Explaining why the previous implementation was unsafe.

This approach simulates real-world conditions for a security analyst, without relying on developers' internal knowledge.

Key Takeaways

• Technology Accessibility: Modern LLMs can already perform complex security analysis without proprietary components.
• Transparency vs. Control: Claims of "uniqueness" for closed models require independent verification.
• Regulatory Implications: If open tools achieve comparable results, arguments for strict regulation of frontier models weaken.
• Current Model Limitations: Even when pinpointing the error location accurately, models may misinterpret its severity — critical for automated triage.
• Embargo as a Barrier: Inability to verify 99% of claimed findings reduces trust in marketing messages.

Prospects for Automated Code Analysis

Vidoc's results show that the line between "frontier" and "accessible" models in security is blurring. In the coming years, we can expect:

• Integration of LLMs into CI/CD pipelines for preliminary patch analysis.
• Emergence of specialized fine-tuned models for specific stacks (e.g., embedded C or Rust).
• Development of hybrid systems combining symbolic analysis and generative AI.

However, a key question remains: how reliable are such systems for decision-making without human involvement? For now, the answer is no. But as augmentation tools for experts, they already demonstrate value.

— Editorial Team