Is Pelicanmaxxing Real? Testing AI Models for Benchmark Gaming

The author tested whether AI labs are specifically gaming the 'pelican on a bicycle' benchmark by comparing it to other animal-vehicle combinations. The study found that pelicans and bicycles are actually among the lowest-scoring subjects and show no signs of special optimization. Ultimately, the results suggest that models are improving their general SVG capabilities rather than memorizing specific prompts.

Key Points

• The experiment compared the famous 'pelican on a bicycle' prompt against 47 other animal-vehicle variations across seven top AI models.
• Quantitative data showed that pelicans and bicycles are actually harder for models to draw than other subjects, ranking near the bottom of the quality scale.
• Statistical regression analysis failed to find any significant 'benchmark-specific' boost for any of the tested models.
• Common compositional traits, such as the pelican facing right, were found to be typical for those subjects rather than evidence of memorization.

Sentiment

The overall sentiment is amused, interested, and cautiously supportive of the article. Hacker News generally accepts the article's finding that there is no strong evidence of narrow pelican-bicycle benchmark gaming, but many commenters are skeptical that the benchmark remains a meaningful measure of general model capability. The discussion is constructive rather than hostile, with disagreement centered on benchmark design, SVG usefulness, and how to think about public evaluation prompts.

In Agreement

• The article's broader grid of animal and vehicle prompts is a more rigorous way to test the pelicanmaxxing hypothesis than casual spot checks.
• The results support the idea that labs are not narrowly optimizing for the exact pelican-on-bicycle prompt.
• General improvement at SVG generation is a real capability gain rather than benchmark cheating.
• The original benchmark has become less useful as a test of novelty because the prompt and surrounding meme are now widely known.
• Public benchmarks that gain attention should be treated cautiously because they can become contaminated directly or indirectly.

Opposed

• The benchmark itself is too absurd or impractical to say much about useful model performance.
• The generated SVGs may satisfy rough spatial constraints while still being aesthetically poor and unsuitable for serious art.
• Using an LLM judge to evaluate SVG image quality may introduce unwanted bias into the analysis.
• Testing a familiar prompt family may still be in-distribution, so the experiment may not fully separate memorization from composition ability.
• If labs wanted to optimize for this kind of benchmark, they would likely train on broader augmented examples rather than the exact pelican-bicycle prompt, making narrow tests harder to interpret.