Scaling Autoresearch: How 16 GPUs Transform AI-Driven Discovery

By giving Claude Code access to 16 GPUs via SkyPilot, researchers accelerated an autonomous neural network training loop by 9x. The agent completed over 900 experiments in 8 hours, discovering that model width was the most critical factor for performance. This parallel approach allowed the agent to move beyond simple trial-and-error to advanced factorial searches and autonomous hardware optimization.

Key Points

• Parallelism increased experiment throughput by 9x, allowing the agent to reach optimal validation loss in 8 hours compared to a projected 72 hours for sequential runs.
• The agent transitioned from simple sequential testing to complex factorial grid searches, enabling it to identify optimal model widths and hyperparameter interactions in single waves.
• The AI agent autonomously developed a hardware-aware strategy, utilizing faster H200 GPUs for high-precision validation while using H100s for broad screening.
• Architecture discovery, specifically scaling model width, provided a more significant performance boost than hyperparameter tuning alone.
• The total cost for the 8-hour session was approximately $300, including both AI API fees and GPU compute costs.

Sentiment

Mildly skeptical. While commenters showed interest in the engineering achievement and the GPU tiering behavior, the prevailing sentiment questioned whether autoresearch represents genuine innovation beyond existing optimization techniques.

In Agreement

• The agent's ability to independently develop a tiered GPU strategy (screening on H100s, validating on H200s) is a genuinely surprising emergent behavior
• SkyPilot is a valuable tool for multi-cloud GPU management and access, making this kind of scaled experimentation practical

Opposed

• The autoresearch trend is essentially reinventing hyperparameter tuning, which has well-established approaches like Bayesian optimization
• Claims of emergent behavior are overstated since resource allocation techniques exist in the model's training data
• AI research agents are fundamentally guided brute-force search rather than genuine discovery tools