Open-Weight GLM 5.2 Beats Claude in Semgrep Cyber Benchmarks

Semgrep's benchmarking of IDOR detection found that the open-weight model GLM 5.2 outperformed frontier models like Claude Code when used without specialized scaffolding. Although Semgrep's proprietary multimodal harness remains the most effective overall, the experiment proves that open-weight AI is now a viable, low-cost option for security research. The results emphasize that the structure surrounding a model is critical, but the raw reasoning power of open models is catching up to industry leaders.

Key Points

• The 'harness' or scaffolding around an LLM, such as endpoint discovery and guided navigation, is the single most important factor in vulnerability detection performance.
• GLM 5.2, an open-weight model from Zhipu AI, unexpectedly outperformed frontier models like Claude Code in a prompt-only IDOR detection benchmark.
• Open-weight models offer significant economic advantages, with GLM 5.2 costing roughly one-sixth of comparable frontier models while allowing for private, on-premises execution.
• Semgrep Multimodal remains the top-performing configuration, validating that combining AI reasoning with rule-based structural analysis yields the best results.
• The rapid improvement of open-weight models suggests security teams should remain flexible and avoid vendor lock-in to leverage the best cost-to-performance ratios.

Sentiment

The overall sentiment is cautiously positive toward the article's core implication that open-weight models are becoming serious competitors, especially on cost and practical usability. Hacker News largely agrees that GLM 5.2 deserves attention and that harness quality is crucial, but it does not fully accept the strongest reading of the headline. The dominant mood is enthusiasm tempered by benchmark skepticism and careful distinction between narrow security detection results and broader claims about frontier-model replacement.

In Agreement

• GLM 5.2 is described by users as a capable, consistent, and affordable programming model for everyday work.
• Open-weight models are seen as close enough for many practical tasks that cost, stability, and control can matter more than chasing the top closed model.
• The article's emphasis on harnesses resonates with commenters who believe context gathering, orchestration, and workflow design are central to model performance.
• Several commenters view the result as evidence that open models are threatening the business case for premium closed-model access.
• Some practitioners report better results by combining a solid model with human review and tailored tooling instead of relying on a frontier model to complete large tasks unattended.

Opposed

• Many commenters argue that the headline overstates the result because the benchmark is narrow and dependent on Semgrep's setup.
• Skeptics question whether the same scaffolding was given across models and whether the comparison fairly separates model capability from harness advantage.
• Several people note that detecting vulnerabilities is not the same as crafting working exploits, so the result does not directly answer Anthropic's Mythos safety claims.
• Commenters warn that benchmarks can be optimized for or memorized and may not match private evaluations or real developer experience.
• Practical objections include missing vision support, provider reliability, local hardware requirements, and trust concerns around hosted non-US models.