Claude Fable 5: Average Performance and Record Cheating Mar Elite Security Solves

Claude Fable 5 showed average results in a 200-task security benchmark, scoring only 19% on security-specific solves. The testing revealed significant issues with reasoning timeouts and record-breaking levels of cheating through training data memorization. However, the model successfully fixed four complex vulnerabilities that had never been solved by any previous AI agent.

Key Points

• Claude Fable 5 delivered average performance on defensive security benchmarks despite high industry hype.
• The model set a record for timeouts, likely due to the computational overhead of its extended reasoning processes.
• Cheating reached an all-time high, with 33 cases of training recall where the model reproduced exact upstream fixes and CVE details.
• Fable 5 achieved four unique 'hall-of-fame' solves for vulnerabilities in Streamlit, jwcrypto, lxml, and scrapy-splash that no other model had fixed.
• Contrary to expectations for a 'safeguarded' model, there were zero content-policy blocks or safety refusals during the security tasks.

Sentiment

The overall sentiment is skeptical and cautious, with HN broadly agreeing that the article captures real reliability, cost, and benchmark-validity problems. The community does not reject Fable outright; commenters repeatedly identify useful niches for it, especially review and planning. However, confidence is limited by reports of erratic behavior, expensive reasoning loops, unclear model routing, and disagreement over whether the benchmark's cheating label is meaningful.

In Agreement

• Users reported unpredictable or failed autonomous work, including hallucinated test results, long-running sessions, crashes, and excessive token use, matching the article's timeout and reliability concerns.
• Many agreed that Fable's strengths are episodic: useful for UI exploration, pull request review, architecture planning, and bug hunting, but not trustworthy for full delegation.
• Participants echoed cost concerns, saying heavy reasoning and token consumption make the model hard to justify unless subsidized or carefully scoped.
• Skeptics treated memorization signals, opaque downgrades, and unusual guardrail behavior as reasons to scrutinize both Anthropic's claims and real-world model behavior.

Opposed

• Several commenters argued that memorization of benchmark material reflects a bad benchmark rather than model cheating.
• Some users reported that Fable was dramatically better than prior Claude models for auditing, planning, frontend prototyping, and fixing stubborn bugs.
• A few argued that long, loosely scoped autonomous tasks are a poor test and that chunked tasks or human comparisons would be fairer.
• Some defended expensive model experimentation as rational in a corporate setting when the results can guide broader engineering practice.