The System is the Moat: Why Small Models Rival Frontier AI in Cybersecurity

The article argues that advanced AI cybersecurity capabilities are not exclusive to massive frontier models like Anthropic's Mythos. Testing shows that small, open-weights models can effectively detect complex zero-day vulnerabilities and reason about exploits. Consequently, the author suggests that the true value in AI security resides in the orchestration systems and maintainer relationships rather than the models themselves.

Key Points

• Small, open-weights models can identify many of the same high-profile vulnerabilities showcased by frontier models like Mythos at a fraction of the cost.
• AI cybersecurity capability is 'jagged,' meaning performance is inconsistent across different tasks and does not scale smoothly with model size or price.
• The competitive advantage in AI security lies in the 'system'—the orchestration, validation, and human trust—rather than the underlying model.
• Broad-spectrum scanning using many cheap models can be more economically effective for defense than using a single expensive frontier model.
• While frontier models excel at complex, multi-stage exploit construction, core defensive tasks like discovery and patching are already widely accessible.

Sentiment

The community is predominantly skeptical of AISLE's central claim, with most substantive comments arguing the methodology is flawed and the comparison to Mythos is not apples-to-apples. However, there is also notable skepticism toward Anthropic's marketing of Mythos, creating a 'both sides have credibility problems' dynamic. The overall lean is that the article overstates what small models can do while Anthropic overstates what only their model can do.

In Agreement

• The scaffolding and system architecture around the model does most of the heavy lifting — Anthropic's own scaffold was essentially a loop over files, so small models with similar harnesses could achieve comparable results
• AISLE has a real track record of finding CVEs with their system, including 15 in OpenSSL and 180+ across 30+ projects, demonstrating that non-frontier models can find real vulnerabilities in practice
• AI inference costs are dropping rapidly, so today's $20K scan could cost $100 in a few years, making small models even more competitive on a cost basis
• Small models could be run cheaply in parallel with consensus mechanisms or used as a triage layer before expensive frontier model verification
• Anthropic's framing of Mythos as 'too dangerous to release' looks more like enterprise sales marketing than genuine safety concern

Opposed

• AISLE's test is fundamentally flawed: they isolated known vulnerable functions, stripped code down, provided architectural context and vulnerability-type hints — this tests recognition, not discovery
• When tested on patched code where the vulnerability was fixed, half of the small models still reported finding vulnerabilities, revealing unacceptably high false positive rates that would make real-world scanning impractical
• The distinction between 'distinguishing' a vulnerability when presented with it versus autonomously 'finding' it across an entire codebase is critical and the article conflates the two
• AISLE has its own conflict of interest — their business model depends on proving that the moat is in the system rather than the model, making this article essentially marketing for their own services
• No one used small models to find these vulnerabilities before Anthropic did, despite the code being open source for decades — if small models could truly do this, someone would have already done it
• Larger models have better attention mechanisms and can reason across wider contexts, which matters for vulnerabilities that span multiple code segments — small models cap out before reaching the complexity frontier models can handle