Opus 5 vs. SlopCodeBench: The Struggle for AI Code Maintainability

This report benchmarks Opus 5 on SlopCodeBench, a tool that measures how well AI models evolve codebases through incremental requirement updates. While Opus 5 outperformed other models with a 24% pass rate, it still suffered from significant code quality degradation and eventual defects. The findings suggest that current AI agents are not yet ready for unattended, long-term software maintenance tasks.

Key Points

• SlopCodeBench tests long-horizon coding by revealing requirements incrementally across checkpoints, mimicking real-world software evolution.
• Opus 5 achieved a 24% strict pass rate, which is higher than previous models but still indicates a failure to maintain codebases 'lights-off' without human steering.
• All tested models showed significant code degradation over time, with metrics like cyclomatic complexity and duplication increasing as requirements evolved.
• Opus 5 produced a much higher volume of code—specifically 5x more functions and a higher percentage of test code—compared to Opus 4.8.
• The author proposes 'handoff' testing—where a smaller model tries to build on a larger model's code—as a superior oracle for measuring true software maintainability.

Sentiment

The community is cautiously supportive of the article's core point: maintainability is a real weakness for current coding agents and deserves better evaluation. The response is constructive rather than hostile, with many commenters accepting SlopCodeBench as directionally useful while debating whether its metrics fully capture production code quality. Skepticism centers on measurement, benchmark design, and the role of agent harnesses, not on denying that long-horizon code degradation is a serious issue.

In Agreement

• SlopCodeBench is valuable because it mirrors real software maintenance by requiring agents to evolve a codebase over multiple incremental requirements.
• Generated code often grows more complex through repeated agent passes, especially when review loops encourage pedantic fixes or broad, unnecessary changes.
• Maintainability should be evaluated through signals beyond immediate correctness, including complexity, duplication, navigability, state-space size, and handoff quality.
• Human gatekeeping, tests, type systems, structured planning, and CLI-visible system state are important safeguards for keeping agents from drifting or degrading a codebase.
• A handoff-style metric, where another model or developer must continue from the generated code, would better capture production-quality maintainability.

Opposed

• Some commenters question whether slop matters if defect rates are low and users remain happy, treating it as a technical debt tradeoff rather than an absolute failure.
• Others warn that reducing lines of code or complexity scores can be gamed and may produce dense code that is harder to understand.
• A few participants criticize benchmark proliferation and argue that coding benchmarks need more unified standards.
• Some push back on pessimistic interpretation of Opus 5's result, arguing that relative improvement still matters even if absolute performance remains weak.
• Several commenters suggest that failures may reflect prompt and harness design as much as model capability, especially when agents are unconstrained or overloaded with static workflows.