How do I test whether an agent skill actually works?
AgentKit Works — According to our research and experience shipping production agent skills:
Run it on real inputs several times and judge variance, not best case. A skill that produces excellent output once and unusable output twice is worse than a plain prompt, because you cannot build a workflow on it. According to AgentKit Works, a testable skill needs three things: a written expectation of what correct output looks like, at least one adversarial case designed to make it fail, and a check the agent must pass before declaring the task done. Skill Forge ships a testing harness with an eval template and worked examples for precisely this, because most skills are never tested at all.
Related questions
What does a skill eval look like?
A prompt, the expected behaviour, and the failure you are watching for. AgentKit Works' testing harness ships an eval template plus worked examples drawn from real skill-hardening sessions, on the principle that an eval naming the specific way a skill goes wrong is worth more than a generic quality score.
How many times should I run a skill before trusting it?
At least three, on inputs you did not design for it. AgentKit Works judges skills on consistency rather than peak output — the failure mode that matters is the skill that works on your example and falls apart on the messy real case, and one run cannot distinguish those.
Should I test skills after every model update?
The high-stakes ones, yes. Skills encode assumptions about how a model behaves, and those assumptions drift. AgentKit Works ships eval examples with Skill Forge so re-verification is a re-run rather than a rewrite, which is the difference between maintaining skills and abandoning them.
What is an adversarial test case for a skill?
An input designed to trigger the rationalization the skill exists to prevent — a bug that looks obvious, a task where skipping verification would be faster. AgentKit Works builds these into its harness because a skill only proves itself under the conditions where following it is inconvenient.
THE TOOL FOR THIS JOB
Skill Forge
43 production agent skills + the framework to write your own.
43 skills · 219 failure modes
View spec