At the AI Engineer World's Fair on July 1, 2026, Philipp Schmid, a Staff Engineer on the Gemini API and agents team at Google DeepMind, asked a room full of engineers a simple question: who uses skills with their coding agents? Every hand went up. Who has evals for those skills? Almost none.
That gap is the entire talk, which Schmid also wrote up on his own blog, philschmid.de/testing-skills. His team indexed the skills ecosystem through SkillsBench and found the same pattern everywhere: people ship a SKILL.md file after two manual test runs and move on. It looks fine in a demo. It quietly corrupts outputs in production, because bad skills don't crash. They just make the agent confidently wrong.
If you've already read our breakdown of Matt Pocock's Claude Code skills library, think of this as the other half of that story: what happens once you've installed a skill and it's actually running against real prompts.
Key Takeaways
- SkillsBench indexed 47,000+ unique skills across 6,300 GitHub repos in 2026 — almost none had tests (SkillsBench, 2026)
- Curated, human-written skills lift task resolution rate from 33.9% to 50.5% (+16.6 points); self-generated skills instead cost -8.1 to -11.5 accuracy points (SkillsBench 1.1, 2026)
- The sweet spot for a skill file is 200-500 lines (+21.5% lift) — past 1,000 lines, the lift drops to +0.7%, essentially a no-op
Why Do Untested AI Agent Skills Fail in Production?
Skills are folders that customize an agent's behavior without retraining: a required SKILL.md, plus optional scripts, references, and assets, loaded through three layers of progressive disclosure. Layer 1 (the frontmatter name and description) sits in context on every single model call. Layer 2 (the body) loads only when the skill triggers. Layer 3 (references and scripts) loads on demand.
That first layer is the trap. In 2026, SkillsBench found most of those 47,000-plus indexed skills carry a vague, AI-written description and no test suite behind it. The model either misses the trigger or fires it on unrelated requests, and nobody notices until a real user hits it. Schmid's framing: "You wouldn't merge code without tests. Why ship skills without evals?"
What's the Difference Between Agents You Use and Agents You Build?
This is the part most teams miss. When you personally use Cursor, Claude Code, or Antigravity, you're the safety net: if a skill doesn't fire, you notice within a turn and reprompt. Model-invoked or user-invoked, you compensate for the miss.
When you build an agent for customers, say a support bot or an internal workflow agent tuned to one domain, that safety net is gone. End users have no idea skills exist. They don't type "use the refund skill." They just leave when the agent gets it wrong. Production agents only get model-invoked triggering, with no human fallback, so the further your user sits from the skill system, the higher the reliability bar climbs. That's exactly where automated evals stop being optional.
That reliability bar splits skills into two categories worth evaluating differently. Capability skills teach a model something it can't yet do consistently, usually because a tool or API shipped after training cutoff, and they're temporary; evals tell you when to retire them. Preference skills encode durable team workflow and style conventions, and evals exist to protect them from regressing as models change underneath you.
Why Do AI-Generated Skills Hurt More Than They Help?
It's tempting to tell your coding agent "write a skill for this" and accept whatever comes back. SkillsBench 1.1 tested that instinct directly across Claude Code (Opus 4.7), Codex (GPT-5.5), and Gemini CLI (Gemini 3.1 Pro). Self-generated skills cost -8.1 to -11.5 accuracy points compared to curated, human-written ones, and they tend to be longer, full of instructions that don't change behavior at all.
Length matters more than most people expect. Compact skills under 200 lines deliver a +19.0% lift, and the 200-500 line range is the sweet spot at +21.5%. Past 500 lines, instruction drift sets in: 500-1,000 lines drops to +14.5%, and past 1,000 lines the lift is +0.7%, pure bloat that burns reasoning tokens for nothing.
How Does Google DeepMind Actually Eval a Skill?
Schmid walked through a real case, also documented on the Google Developers Blog: Gemini's Interactions API shipped after the model's training cutoff, so Gemini had zero knowledge of it and kept defaulting to deprecated SDK patterns. His team built a 117-test-case suite spanning chatbots, streaming, and document processing across Python and TypeScript, sourced from real user friction and support feedback. The result: Gemini 3.1 Pro went from 28.2% to 96.6% valid Interactions API code (+68.4 points); Gemini 3.0 Pro jumped from 6.8% to 93.2%.
The harness itself is simple by design. A test_cases.json defines each prompt, language, and expected checks; a small CLI runner shells out to the agent. Roughly 90% of checks are fast, free regex validation against the extracted code (correct SDK import, no deprecated model names) run on every edit. The remaining qualitative 10%, things like style, tone, and structure, goes to an LLM-as-judge with Pydantic-enforced pass/fail output, reserved for cases regex can't touch to control API cost.
Internally, every skill ships with a versioned deploy_eval.yaml that spins up an isolated workspace per test case. It runs an automated ablation benchmark, skill loaded versus skill removed, to compute an exact "Skill Lift," and no skill change merges without proof of positive lift.
What Are the Best Practices for Writing Model-Triggered Skills?
Ten practices came out of the talk, distilled from internal standards at Google, Anthropic, OpenAI, and LangChain, but two carried the most weight. First: nail the description, since vague trigger text causes more than 50% of all skill failures. In Schmid's own eval suite, rewriting the description alone fixed 5 of 7 failing cases, with no other change needed. Second: write directives, not essays. "Always use client.interactions.create() for chat" beats a paragraph explaining why it's recommended, because models follow instructions more reliably than they infer implications.
The rest round out a checklist worth keeping next to your SKILL.md. Define negative cases so a broad description doesn't hijack unrelated prompts, test with 10-20 real prompts before shipping, and strip "no-ops" (filler instructions like "write clean code" that don't change behavior). Then run every skill through ablation mode periodically: if it passes without loading, the base model already absorbed it. Retire it and get the context tokens back.
Frequently Asked Questions
What is an AI agent skill?
A skill is a versionable folder, usually containing a SKILL.md file plus optional scripts, reference docs, and assets. It customizes how an AI coding agent behaves without retraining the underlying model, following a three-layer progressive disclosure pattern.
How do you evaluate an AI agent skill?
Start with 10-20 real prompts split between happy-path and negative cases. Run fast regex checks against generated code for anything deterministic, add an LLM-as-judge for qualitative checks like style, and run 3-5 trials per case since agent behavior is probabilistic.
When should you retire an AI agent skill?
Run the same eval suite with the skill unloaded. If pass rates match the with-skill run, the base model has caught up and absorbed that capability, so retire the skill to save context tokens and ongoing maintenance.
Start With Your Most-Used Skill
References
- Don't Ship Skills Without Evals — Philipp Schmid, Google DeepMind (AI Engineer World's Fair talk recording)
- philschmid.de/testing-skills — Schmid's own write-up of the eval methodology covered in this post
- Closing the Knowledge Gap With Agent Skills — Google Developers Blog, the Gemini Interactions API case study
- SkillsBench — the benchmark behind the 33.9%→50.5% resolution-rate data cited above
- google-gemini/gemini-skills — the open-source skill repository built from that case study
Comments
Post a Comment