The Slop Harness: Why SOTA Models are Regressing in Tool Adherence

Newer Anthropic models are increasingly failing to follow precise tool schemas by inventing non-existent JSON fields. This regression likely stems from reinforcement learning in 'slop harnesses' like Claude Code, which reward successful tasks even when the underlying tool calls are malformed. As a result, developers may need to rely on strict grammar-constrained sampling to ensure reliability with non-standard tool shapes.

Key Points

• Newer Anthropic models like Opus 4.8 and Sonnet 5 are showing a regression in schema adherence compared to older models, frequently hallucinating extra JSON keys.
• The author attributes this to 'slop' in the post-training environment, where models are trained using harnesses like Claude Code that silently fix or ignore malformed tool calls.
• Tool calling is essentially text generation using in-band signaling (like ANTML), and models develop strong priors for the specific tool shapes they encounter during RL.
• The failure is context-dependent and often occurs at high-entropy points in generation, such as after closing a long string literal in a nested JSON object.
• Strict grammar-constrained decoding is becoming necessary to force models to follow schemas that differ from the dominant training distribution.

Sentiment

The overall sentiment is moderately supportive of the article. Most substantive comments accept the diagnosis that tool behavior is shaped by harness training and that schema portability is weaker than many developers assume, while the main counterpoint is pragmatic rather than dismissive: commenters want better feedback loops, hooks, normalization, and sandboxing to absorb the failure modes.

In Agreement

• Models appear to carry strong priors from popular coding harnesses, so superficially similar tools can trigger provider-specific assumptions rather than strict schema following.
• Forgiving harnesses that repair malformed calls can accidentally reward sloppy tool use during training or evaluation, making later strict environments less reliable.
• The effective agent is the model plus the runtime around it; validation, hooks, permissions, recovery behavior, and feedback loops are part of the real interface.
• Provider-native harnesses may become a proprietary advantage if frontier models are best adapted to their vendors' hidden tool shapes and repair behavior.
• Independent tool authors should expect to design recovery paths, normalization, and clear feedback rather than relying on schema descriptions alone.

Opposed

• Several commenters argue that the problem is manageable with good error messages, because agents often correct malformed calls when the tool explains the mistake clearly.
• Some see automatic repair or normalization of harmless extra fields as a practical engineering solution that avoids making every schema violation fatal.
• A few commenters treat the behavior as expected rather than alarming, noting that older agent tools have long used forgiving parsing around model output.
• Others suggest avoiding brittle JSON tool schemas by giving agents a sandboxed programming or shell interface and letting them express operations in code.
• Skeptics of custom tool ecosystems argue that directly targeting provider APIs and default tools may be the path of least resistance.