JetBrains benchmarked a popular open-source Claude Code skill called Ponytail. It pushes the agent to write less code. The authors has promised 54% less code, 20% less cost.
80 paired tasks. Sonnet 5. Every trial audited.
Reality: 15.4% less code, 10.3% less cost. No quality difference either way. Smaller than the label, but real.
Then the part that matters more.
Installed passively, the skill showed zero self-activation. It did nothing at all unless the ruleset was force-injected into the session.
A tool with measurable benefits, sitting there inert.
I have not run this one myself, so this is JetBrains' data and not mine. But the implication travels past one skill.
If you rolled out agent skills across your team and measured adoption by install count, you measured nothing. Install count says the file exists. It says nothing about whether it ran.
Full breakdown in this week's episode of The Human in the Loop. Link in the comments.
Podden och tillhörande omslagsbild på den här sidan tillhör
Enrique Cordero. Innehållet i podden är skapat av Enrique Cordero och inte av,
eller tillsammans med, Poddtoppen.
