opensourceprojects.dev

A broadsheet for software that doesn't ask for your email

Check whether your agent skill actually improves task results
GitHub RepoViews22

Project Description

View on GitHub

Does Your Agent Skill Actually Help? SkillEvaluator Puts a Number On It

You've written an agent skill—a folder of instructions and supporting files that tells an AI agent how to do something specific. It feels useful. But have you actually measured whether it improves task results, or does it just make the agent sound more confident? That's the gap SkillEvaluator is built to close. It's an NVIDIA project that checks the quality of your skill and then compares runs with and without it, so you can see the actual effect on outcomes.

What It Does

SkillEvaluator is a command-line tool for evaluating agent skills as defined by the Agent Skills specification. It works in two stages: first, static quality checks that don't need a model, and second, model-backed evaluation that measures whether the skill changes task results.

The tool runs a tiered check system. Tier 1 checks run without an API key at all, and the quality-check command produces a deterministic score with findings—the bundled text-analyzer example scores 86.2/100 and passes a default threshold of 70. It's written in Python (3.12, 3.13, and 3.14 are supported), installs from PyPI, and is part of NVIDIA's broader Verified Skills pipeline.

Why It's Cool

  • It separates cheap checks from expensive ones. You can get a quality score in seconds with no API key, no scanner, no GPU, and no Docker. That's a low-friction way to catch problems before you spend anything on live evaluation.

  • It's honest about what it doesn't know. The README is refreshingly direct here: static quality checks are one thing, but publication evidence stays INCOMPLETE. Security needs scanners, and agent impact needs live evaluation. The tool doesn't pretend a score is the whole story—it tells you which questions it has and hasn't answered.

  • Three entry points, one CLI. You can run it directly, wire it into CI, or hand it to your agent. The CI path is particularly practical: skillevaluator ci init opens a setup PR for GitHub, and there's a GitLab template if that's your platform. Changed skills get checked on every pull request.

  • It can evaluate itself, sort of. There's a SkillEvaluator skill you can add to Claude Code, Codex, or any skill-aware agent with npx skills add NVIDIA/SkillEvaluator, then just ask "Is my skill any good?" It's a nice bit of recursion—the tool for checking skills ships as a skill.

  • There's a paper and real docs. An arXiv paper and NVIDIA documentation back the project, so if you want to understand the methodology rather than just trust the score, the material is there.

How to Try It

The fastest path needs only Git and uv. Clone the repo and run the keyless walkthrough:

git clone --depth 1 https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator

Then run a quality check against the bundled example:

uv run --python 3.13 skillevaluator quality-check \
  src/skillevaluator/tier3/reference_skills/text-analyzer \
  -r cli -r json -o reports/first-quality

You'll get a score, a pass/fail against the default threshold of 70, and a saved JSON report. Swap in your own skill directory when you're ready.

If you'd rather install it standalone:

pip install skillevaluator
skillevaluator validate ./my-skill --tiers 1

For CI, run skillevaluator ci init inside your GitHub repo and it'll open a setup PR. For agent integration, use npx skills add NVIDIA/SkillEvaluator (or copy the folder if you don't have Node.js).

Full details, including the model-backed path with an NVIDIA Build API key, are in the repository and the documentation.

Final Thoughts

SkillEvaluator is aimed at people who are already building agent skills and want evidence rather than vibes—particularly teams that maintain skills across a repo and want regression checks on pull requests. The keyless tier makes it easy to try before committing, and the explicit acknowledgment that static checks aren't the full picture is the kind of honesty that makes a tool trustworthy. If you've been shipping skills on instinct, this gives you a way to find out whether that instinct holds up.

Back to Projects
Project ID: skillevaluatorLast updated: November 16, 2026 at 09:00 AM