Broken Benchmarks: AI Models Near Saturation in Advanced Physics

An expert audit of six major physics benchmarks revealed that frontier AI models are frequently penalized for correct answers due to errors in the evaluation data and reference solutions. After correcting these benchmark flaws, model performance scores saw massive increases, often nearing 90% accuracy. The findings indicate that current physics evaluations are nearly saturated and no longer provide an accurate measure of advanced AI reasoning.

Key Points

• Leading physics benchmarks are frequently flawed, containing incorrect reference answers and ambiguous questions that penalize correct model reasoning.
• Expert re-grading reveals that frontier models like GPT-5.6-Sol perform significantly better than previously reported, with some scores increasing by over 30 percentage points.
• Most audited errors were attributed to benchmarking issues rather than genuine failures in the models' scientific or quantitative reasoning.
• Frontier models are nearing saturation on current closed-ended physics tasks, making these benchmarks less effective for distinguishing between top-tier models.
• There is an urgent need for more demanding, expert-validated, and high-quality evaluations to accurately measure the next generation of AI scientific capabilities.

Sentiment

The community is mixed but constructively engaged. Commenters mostly agree with the article's critique of benchmark quality, while remaining skeptical of strong claims that benchmark saturation demonstrates genuine frontier-level physics understanding.

In Agreement

• Automated benchmark grading can incorrectly reject valid or algebraically equivalent answers, so reported scores may understate actual model performance.
• Flawed benchmark keys and ambiguous grading make saturation claims plausible because some apparent model errors are actually evaluation errors.
• Frontier models can be highly useful expert tools when paired with domain-specific context, harness engineering, post-training, and deterministic tools.
• The results are important for robotics and other embodied-AI domains because better physical reasoning could matter beyond academic benchmark scores.

Opposed

• High benchmark performance may reflect exposure to benchmark tasks or expert-solved benchmark data rather than independent physical reasoning.
• Some benchmark questions are so ambiguous or physically underspecified that success on them may measure textbook pattern matching more than physics competence.
• Current LLMs still make basic mistakes when asked to interpret real physical situations, suggesting benchmark saturation does not prove robust understanding.
• Closed-ended physics benchmarks are a weak proxy for frontier physics research, especially when real work requires formulating the problem and challenging assumptions.