Opus 5 vs. SlopCodeBench: The Struggle for AI Code Maintainability

This report benchmarks Opus 5 on SlopCodeBench, a tool that measures how well AI models evolve codebases through incremental requirement updates. While Opus 5 outperformed other models with a 24% pass rate, it still suffered from significant code quality degradation and eventual defects. The findings suggest that current AI agents are not yet ready for unattended, long-term software maintenance tasks.

Key Points

• SlopCodeBench tests long-horizon coding by revealing requirements incrementally across checkpoints, mimicking real-world software evolution.
• Opus 5 achieved a 24% strict pass rate, which is higher than previous models but still indicates a failure to maintain codebases 'lights-off' without human steering.
• All tested models showed significant code degradation over time, with metrics like cyclomatic complexity and duplication increasing as requirements evolved.
• Opus 5 produced a much higher volume of code—specifically 5x more functions and a higher percentage of test code—compared to Opus 4.8.
• The author proposes 'handoff' testing—where a smaller model tries to build on a larger model's code—as a superior oracle for measuring true software maintainability.

Sentiment

Cautiously optimistic about the benchmark's methodology, but critical and skeptical of the current ability of frontier models to maintain long-term code health.

In Agreement

• SlopCodeBench is a superior benchmark because it forces models to grapple with their own previous code, mirroring real-world development.
• Models currently struggle to manage refactoring and complexity while simultaneously implementing new features.
• Current RL pipelines and benchmarks do not sufficiently penalize code complexity, which encourages models to generate 'slop' rather than elegant abstractions.
• Code quality consistently degrades as models iterate, often resulting in 'alien-sounding' names and bizarre code placement.

Opposed

• The jump from a 17% pass rate to 24% is a 41% relative improvement, which is more significant than the article suggests.
• The 'slop' issue may be a harness or prompting problem rather than a model failure; models might just be obeying 'pedantic' instructions.
• Increased function counts are not inherently bad and can actually lead to more testable and readable code if complexity is managed.
• Some users find Opus 5 to be a meaningful improvement in speed and token efficiency for daily work, even if it isn't a 'step function' change.