Agent‑Skills‑Eval: Test Runner for Anthropic Agent Skills
Run npx agent-skills-eval ./skills --target gpt-4o-mini --judge gpt-4o-mini --baseline to generate side‑by‑side reports and confirm skill lift.
Run npx agent-skills-eval ./skills --target gpt-4o-mini --judge gpt-4o-mini --baseline to generate baseline comparison reports.
Summary
Agent‑Skills‑Eval is a new open‑source CLI and SDK that lets developers test Anthropic’s Agent Skills by running each eval twice—once with the skill loaded into context and once without it. The tool sends both outputs to a judge model (default gpt‑4o‑mini) which grades them against defined assertions, producing a pass/fail matrix and side‑by‑side comparison. It outputs a static HTML report, JSON artifacts, and timing data in a spec‑compliant iteration‑N folder structure, and can be integrated into CI pipelines or custom dashboards via its TypeScript SDK. The evaluator supports any OpenAI‑compatible provider, including OpenAI, Anthropic, Together, Groq, and local Llama servers, and allows custom providers through a simple Provider interface.
The CLI accepts a YAML config file or command‑line flags, with flags overriding config values. It validates SKILL.md frontmatter, evals.json schema, and optional scripts, references, and assets, ensuring full compliance with the agentskills.io specification. Tool‑call assertions enable deterministic checks for agents that invoke tools, and the SDK exposes events for downstream analysis.
Overall, Agent‑Skills‑Eval gives developers concrete evidence of skill lift, a reproducible benchmark workflow, and an easy way to publish reports without infrastructure.
Key changes
- Runs each eval twice: with_skill and without_skill
- Uses a judge model to grade outputs against assertions
- Produces static HTML report and JSON artifacts in iteration‑N layout
- Supports any OpenAI‑compatible provider (OpenAI, Anthropic, etc.)
- CLI flags override YAML config values
- Validates SKILL.md frontmatter and evals.json schema
- Tool‑call assertions for deterministic checks
- SDK exposes custom provider interface and event stream