MirrorCode: Benchmarking AI's Ability to Rebuild Entire Software Projects

MirrorCode is a benchmark that tests AI models by requiring them to reimplement entire software programs end-to-end without source code access. It provides the high inference budgets necessary for long-horizon tasks, revealing that models can complete weeks of human-level work in just a few hours. While the most difficult tasks remain unsolved, the results demonstrate a significant leap in autonomous software engineering capabilities.

Key Points

• MirrorCode focuses on long-horizon tasks where AI must reimplement entire software projects from scratch without seeing the original code.
• The benchmark provides significant inference budgets and extended timeframes, allowing models to work for days or weeks on a single task.
• Frontier AI models have successfully completed complex engineering tasks in hours that would typically take human engineers weeks or months.
• The evaluation is designed to be cheat-resistant through sandboxing and the use of held-out, private test sets to verify output accuracy.
• Models demonstrate a surprising ability to perform well even in low-resource programming languages like Ada, though they still perform better in mainstream languages like Go.

Sentiment

The overall sentiment is mixed but constructively skeptical. HN generally accepts MirrorCode as an interesting and useful benchmark, and some commenters see it as evidence of meaningful progress in long-horizon agentic coding. However, the community is more cautious than enthusiastic: most substantive comments stress that current agents still need strong tests, reference behavior, human design judgment, and active refactoring to avoid architectural decay.

In Agreement

• MirrorCode's setup is compelling because long-running agents can use a reference executable and tests to iteratively converge on behavior, which resembles a practical path for AI-assisted reimplementation.
• Several commenters report real progress on large agent-built projects when the work is constrained by strong test suites, clear goals, and human review loops.
• Reimplementing existing software can still be valuable even if it is not novel, because it can reveal useful agent workflows, support porting, and turn known products into reusable components.
• The benchmark highlights that agents are improving at longer-horizon coding tasks beyond isolated functions or bug fixes.

Opposed

• Many commenters argue that autonomous agent projects often devolve into duplicated, patchwork, hard-to-maintain code unless a human continually steers architecture and refactoring.
• Some doubt that cloning existing open-source projects proves genuine software engineering ability, especially when models may have seen similar code during training.
• Commenters distinguish behavior matching from novel design, arguing that internal consistency, product judgment, and long-term architecture remain much harder than passing reference tests.
• Several people question benchmark coverage and methodology, including which tests are used and why the model comparison appears limited.