SkillsBench: Validating the Impact of Curated Procedural Knowledge on AI Agents

SkillsBench evaluates how structured procedural knowledge, or 'skills,' impacts the performance of LLM agents across 86 diverse tasks. The study reveals that while human-curated skills significantly boost success rates and help smaller models perform like larger ones, models fail to generate effective skills for themselves. Performance gains are highly variable by domain, with specialized fields like healthcare benefiting the most from these structured interventions.

Key Points

• Introduction of SkillsBench, a standardized benchmark for measuring the efficacy of procedural knowledge packages across 11 domains.
• Human-curated skills significantly improve agent performance, raising average pass rates by 16.2 percentage points, though 16 tasks showed negative results.
• LLMs currently lack the ability to self-generate effective procedural skills, showing no average performance gain when using model-authored instructions.
• Modular design is superior to comprehensive documentation, with 2-3 focused skill modules yielding the best results.
• Agent skills serve as a performance equalizer, enabling smaller LLMs to achieve results comparable to larger, more resource-intensive models.

Sentiment

The community largely agrees with the paper's core finding that curated skills improve agent performance but is sharply critical of the methodology used to test self-generated skills. Practitioners view the self-generated test as unrealistic and disconnected from actual workflows, diminishing the paper's practical relevance on that specific point. The overall tone is constructively critical rather than hostile.

In Agreement

• Curated, human-crafted procedural knowledge meaningfully improves agent performance, especially in domains where models have weak priors from training data
• Skills that encode genuinely novel information—outside training data, context-specific, or alignment guidance—are most effective; self-referential skills add nothing
• Modular, focused skills outperform comprehensive documentation, validating the paper's finding about optimal skill structure
• The domain variability result (healthcare vs software engineering) correctly reflects that skills fill gaps in model knowledge
• The null result for self-generated skills is important evidence that models cannot currently bootstrap their own improvement without external feedback

Opposed

• The self-generated skills methodology is fundamentally flawed—it denies the model web search, codebase exploration, and any feedback loop, making it an unfair and unrealistic test
• Real-world skill generation happens iteratively through human-AI collaboration after problem-solving, not as cold pre-task generation from latent knowledge alone
• The paper misses the most practically important condition: skills built through iterative execution, observation, and refinement with real feedback
• The finding about self-generated skills is unsurprising because nobody actually generates skills the way the paper tests them—it tells the model to generate knowledge it already has
• Academic publishing timelines and the rapid pace of AI tooling evolution mean the paper's conclusions may already be outdated by the time they reach readers