# Skill Evaluation

Course: [Agent Skills](https://www.devtutorials.ai/learn/agent-skills) · Section: Skill Quality and Safety
Source: https://www.devtutorials.ai/learn/agent-skills/quality/skill-evaluation

> Testing skills at five levels: structure, activation, output, behavior and human review, with control cases, baselines and regression checks.

## Why Skills Need Tests

**Skill evaluation** = testing a skill the same way every time, so you know a change made it better. In this lesson, you will learn the five kinds of skill tests. You will also learn how to catch a change that breaks something.

A skill is like code.

When you change it, something else may break.

For example:

- You add a word to a description.
- One request now works.
- But two other requests stop working.

You may not notice, because you only tried the request you fixed.

**Simple lesson:** do not trust one quick try. Run the same tests after every change.

### A Real-World Example

A cook changes the soup recipe to add more salt.

The soup tastes better. But the cook never tastes the bread, which uses the same salt jar.

A taste test of every dish catches the problem. Skill tests do the same thing for your skills.

Click the button and open the **Evals** tab. Press **Run evals**.

- **Activation cases:** 16 requests, and the skill each one should fire.
- **Output cases:** a good reply for each sample. It must pass the checks.
- **Control cases:** a bad reply for each sample. It must fail the checks.

One activation case fails on purpose. Leave it for now.

**Simple lesson:** a test set is a fixed list of requests and replies. You run it again and again.

## What You Will Learn

In this lesson, you will learn:

1. The five kinds of skill tests.
2. How to write activation tests.
3. How control cases prove a check works.
4. How a baseline catches a regression.
5. When to run the slow tests.

**About the Evals tab**

- **Run evals** runs the whole DevKit test set.
- **Save as baseline** saves today's result.
- Later runs show **FIXED** or **REGRESSION** next to each case.

## Five Kinds of Tests

Some tests are free and instant. Others cost time and money.

Here are the five kinds, from cheapest to most costly:

| Level | What it checks | Example |
| --- | --- | --- |
| **Structure** | The file itself | Front matter is valid, sections exist |
| **Activation** | Which skill fires | "Create a React component" fires `react-component` |
| **Output** | The shape of the reply | All five release-note headings are there |
| **Behavior** | Real inputs with a real model | 10 real diffs through `pr-review` |
| **Human review** | Judgment and quality | A senior dev reads five security reviews |

Watch one small change go through all five levels:

The cheap activation test caught the problem in a second.

### Write an Output Test

**Output test** = a small file that lists the headings a reply must have, in order.

For `pr-review`, the file `evals/output-test.md` holds four lines:

```md
## Summary
## Findings
## Tests to add
## Verdict
```

After a change, run the test on a saved good reply. If a heading is missing or moved, the test fails.

**Simple lesson:** an output test is a list of headings. It catches a reply that lost its shape.

**Simple lesson:** run the cheap tests on every change. Run the costly ones before a release.

## Activation Tests

**Activation test** = a request, plus the skill that should fire for it.

Write requests the way people really type them.

A good activation test set has three kinds of requests:

- **Typical:** a normal request. Your skill should fire.
- **Near miss:** a similar request. Another skill should fire.
- **None:** a request no skill should answer.

Example for `api-endpoint`:

```text
Add a backend API endpoint to cancel an order   => api-endpoint
Create a React component for the order list     => react-component
What time zone is Tokyo in?                     => none
```

The near miss matters most. It shows your skill does not grab other skills' work.

**Simple lesson:** test what should fire, what should not, and what should fire nothing.

## Control Cases

An output check looks at the reply. For example, it checks the headings.

But how do you know the check works?

**Control case** = a bad reply that the checks must fail.

If a bad reply passes, the check is too weak. It tests nothing.

Watch the checks run on a good reply and a bad reply:

The release-note checks failed the bad reply. The UI script let a bad reply pass.

**Simple lesson:** prove every check can fail. Give it a bad reply on purpose.

**Add one control case per check.**

- Paste a bad reply.
- Run the check.
- It must fail. If it passes, make the check stricter.

## Baselines and Regressions

**Baseline** = a saved test result from a time when things worked.

**Regression** = something that worked before and fails now.

After each change, run the tests again. Then compare with the baseline.

Watch a short description break two requests:

The file was still valid. Only the activation tests showed the damage.

Click the button and open the **Library** tab. Then try this:

1. In **Evals**, press **Run evals**, then **Save as baseline**.
2. In the **Library**, change the `ui-review` description to "Reviews things." Press **Save**.
3. In **Evals**, press **Run evals** again. Two UI cases show **REGRESSION**.
4. Press **Reset** in the Library to bring the DevKit back.

**Simple lesson:** save a baseline first. Never share a change that causes a regression.

## Slow Tests: Behavior and Human Review

**Behavior test** = running real inputs through a real model with the skill.

A model does not give the same reply every time.

So one run is not enough. Run it a few times and count the passes.

Example:

- Release notes: 5 of 5 runs pass. Stable.
- Security review: 4 of 5 runs pass. Watch it.
- React component: 2 of 5 runs pass. Fix the skill.

A low pass rate usually means a step is unclear.

**Small models follow skills less well.**

- A small model in your browser may skip steps.
- Use it to find unclear steps.
- Check final quality with the model your team really uses.

**Human review** = a person reads the replies and judges them.

Code cannot judge everything. A person is needed for:

- How serious a finding really is.
- Whether the advice is useful.
- Whether the tone fits your team.

When a person finds a problem, turn it into a new test case.

**Simple lesson:** count passes over many runs. Let a person judge what code cannot.

## Start Small

You do not need a big test set on day one.

Start with this for each skill:

- 2 activation cases.
- 1 output case.
- 1 control case.

Then add a test every time something goes wrong in real use.

Keep the tests inside the skill folder, next to `SKILL.md`:

```text
pr-review/
  SKILL.md
  evals/
    activation.md
    output-test.md
```

Then the tests move with the skill. Run them again after every change.

In the lab, try it:

1. Open `pr-review` in the **Library** tab.
2. Type `evals/activation.md` in the new file box and press **+ Add file**.
3. Add lines like `please review my pull request => pr-review` and `write release notes => release-note-helper`. Press **Save**.
4. Add `evals/output-test.md` with the four `## ` headings above. Press **Save**.
5. Open the **Evals** tab and press **Run evals**. Your tests show up as **your test** rows.

**Simple lesson:** every real failure becomes a new test.

## TL;DR

1. **Skill evaluation** = running the same tests after every change.
2. **Five levels** = structure, activation, output, behavior, human review.
   - Cheap levels run on every change.
   - Costly levels run before a release.
3. **Activation test** = a request plus the skill that should fire.
   - Include typical, near miss and none.
4. **Control case** = a bad reply that the checks must fail.
5. **Baseline** = a saved good result.
6. **Regression** = something that worked before and fails now.
7. **Pass rate** = how many model runs pass, out of several.
8. **Output test** = the headings a reply must have, in order. Keep it in `evals/` next to `SKILL.md`.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
| --- | --- | --- | --- |
| No tests at all | "I tried it once and it worked." | Run a fixed test set after every change. | Your team finds the break first. |
| Only typical requests | "If it fires, it works." | Add near misses and none cases. | The skill grabs other skills' work. |
| Checks without controls | "The check passed, so the reply is good." | Give every check a bad reply that must fail. | A weak check lets bad replies through. |
| No baseline | "I will remember what passed." | Save a baseline before you change anything. | You cannot tell what broke. |
| Trusting one model run | "It passed once, so it is fine." | Count passes over several runs. | The skill fails half the time for users. |
| Skipping human review | "The checks cover everything." | Let a person judge severity and advice. | Wrong judgment slips through. |

## Check Your Understanding

## Exercises

## What to Remember

- Test skills at five levels: structure, activation, output, behavior and human review.
- Run cheap tests on every change. Run slow tests before a release.
- Activation tests need typical, near miss and none requests.
- Every check needs a control case that must fail.
- Save a baseline. Do not share a change that causes a regression.
- Count model passes over several runs.
- Turn every real failure into a new test.

Next: **Skill Security**. You will check a skill from the internet before you trust it.