Skill Evaluation

By the Syncfusion DevTutorials team · Last updated October 22, 2026

Testing skills at five levels: structure, activation, output, behavior and human review, with control cases, baselines and regression checks.

Why Skills Need Tests

Skill evaluation = testing a skill the same way every time, so you know a change made it better. In this lesson, you will learn the five kinds of skill tests. You will also learn how to catch a change that breaks something.

A skill is like code.

When you change it, something else may break.

For example:

• You add a word to a description.
• One request now works.
• But two other requests stop working.

You may not notice, because you only tried the request you fixed.

Simple lesson: do not trust one quick try. Run the same tests after every change.

A Real-World Example

A cook changes the soup recipe to add more salt.

The soup tastes better. But the cook never tastes the bread, which uses the same salt jar.

A taste test of every dish catches the problem. Skill tests do the same thing for your skills.

Click the button and open the Evals tab. Press Run evals.

• Activation cases: 16 requests, and the skill each one should fire.
• Output cases: a good reply for each sample. It must pass the checks.
• Control cases: a bad reply for each sample. It must fail the checks.

One activation case fails on purpose. Leave it for now.

Simple lesson: a test set is a fixed list of requests and replies. You run it again and again.

What You Will Learn

In this lesson, you will learn:

1. The five kinds of skill tests.
2. How to write activation tests.
3. How control cases prove a check works.
4. How a baseline catches a regression.
5. When to run the slow tests.

Five Kinds of Tests

Some tests are free and instant. Others cost time and money.

Here are the five kinds, from cheapest to most costly:

Watch one small change go through all five levels:

The cheap activation test caught the problem in a second.

Write an Output Test

Output test = a small file that lists the headings a reply must have, in order.

For pr-review, the file evals/output-test.md holds four lines:

## Summary
## Findings
## Tests to add
## Verdict

After a change, run the test on a saved good reply. If a heading is missing or moved, the test fails.

Simple lesson: an output test is a list of headings. It catches a reply that lost its shape.

Simple lesson: run the cheap tests on every change. Run the costly ones before a release.

Activation Tests

Activation test = a request, plus the skill that should fire for it.

Write requests the way people really type them.

A good activation test set has three kinds of requests:

• Typical: a normal request. Your skill should fire.
• Near miss: a similar request. Another skill should fire.
• None: a request no skill should answer.

Example for api-endpoint:

Add a backend API endpoint to cancel an order   => api-endpoint
Create a React component for the order list     => react-component
What time zone is Tokyo in?                     => none

The near miss matters most. It shows your skill does not grab other skills' work.

Simple lesson: test what should fire, what should not, and what should fire nothing.

Control Cases

An output check looks at the reply. For example, it checks the headings.

But how do you know the check works?

Control case = a bad reply that the checks must fail.

If a bad reply passes, the check is too weak. It tests nothing.

Watch the checks run on a good reply and a bad reply:

The release-note checks failed the bad reply. The UI script let a bad reply pass.

Simple lesson: prove every check can fail. Give it a bad reply on purpose.

Baselines and Regressions

Baseline = a saved test result from a time when things worked.

Regression = something that worked before and fails now.

After each change, run the tests again. Then compare with the baseline.

Watch a short description break two requests:

The file was still valid. Only the activation tests showed the damage.

Click the button and open the Library tab. Then try this:

1. In Evals, press Run evals, then Save as baseline.
2. In the Library, change the ui-review description to "Reviews things." Press Save.
3. In Evals, press Run evals again. Two UI cases show REGRESSION.
4. Press Reset in the Library to bring the DevKit back.

Simple lesson: save a baseline first. Never share a change that causes a regression.

Slow Tests: Behavior and Human Review

Behavior test = running real inputs through a real model with the skill.

A model does not give the same reply every time.

So one run is not enough. Run it a few times and count the passes.

Example:

• Release notes: 5 of 5 runs pass. Stable.
• Security review: 4 of 5 runs pass. Watch it.
• React component: 2 of 5 runs pass. Fix the skill.

A low pass rate usually means a step is unclear.

Human review = a person reads the replies and judges them.

Code cannot judge everything. A person is needed for:

• How serious a finding really is.
• Whether the advice is useful.
• Whether the tone fits your team.

When a person finds a problem, turn it into a new test case.

Simple lesson: count passes over many runs. Let a person judge what code cannot.

Start Small

You do not need a big test set on day one.

Start with this for each skill:

• 2 activation cases.
• 1 output case.
• 1 control case.

Then add a test every time something goes wrong in real use.

Keep the tests inside the skill folder, next to SKILL.md:

pr-review/
  SKILL.md
  evals/
    activation.md
    output-test.md

Then the tests move with the skill. Run them again after every change.

In the lab, try it:

1. Open pr-review in the Library tab.
2. Type evals/activation.md in the new file box and press + Add file.
3. Add lines like please review my pull request => pr-review and write release notes => release-note-helper. Press Save.
4. Add evals/output-test.md with the four ## headings above. Press Save.
5. Open the Evals tab and press Run evals. Your tests show up as your test rows.

Simple lesson: every real failure becomes a new test.

TL;DR

1. Skill evaluation = running the same tests after every change.

2. Five levels = structure, activation, output, behavior, human review.

   • Cheap levels run on every change.
   • Costly levels run before a release.

3. Activation test = a request plus the skill that should fire.

   • Include typical, near miss and none.
4. Control case = a bad reply that the checks must fail.
5. Baseline = a saved good result.
6. Regression = something that worked before and fails now.
7. Pass rate = how many model runs pass, out of several.
8. Output test = the headings a reply must have, in order. Keep it in evals/ next to SKILL.md.

Common mistakes

Check Your Understanding

Exercises

What to Remember

• Test skills at five levels: structure, activation, output, behavior and human review.
• Run cheap tests on every change. Run slow tests before a release.
• Activation tests need typical, near miss and none requests.
• Every check needs a control case that must fail.
• Save a baseline. Do not share a change that causes a regression.
• Count model passes over several runs.
• Turn every real failure into a new test.

Next: Skill Security. You will check a skill from the internet before you trust it.