# Testing Agents

Course: [AI Agents](https://www.devtutorials.ai/learn/ai-agents) · Section: Agents for Development Workflows
Source: https://www.devtutorials.ai/learn/ai-agents/dev-workflows/testing-agents

> How a testing agent finds untested code, writes and runs tests, and raises coverage, and how an audit agent scans dependencies, patches them and checks the patch.

Think about a safety inspector checking a new building.

The inspector has a checklist of everything that must be inspected: every door, every fire alarm, and every stair rail. At the end, they can say, "We checked 80 out of 100 items." The remaining 20 items are unknown, and unknown does not mean safe.

A second inspector checks the parts that came from outside suppliers, such as lifts, wiring, and boilers. They compare each model number against a list of recalled products. If a boiler appears on the recall list, it must be replaced and then checked again.

This lesson introduces two agents that perform these kinds of checks for software.

**The main idea:** A **testing agent** finds code that is not covered by tests, writes tests for it, and measures coverage again. A **security audit agent** checks project dependencies for known security issues, updates them, and verifies the fix.

## A Real-World Example

**Testing.** The file `web/status.js` contains 5 small helper functions. Only one of them, `statusLabel`, currently has a test.

That means the project has 20% **function coverage**. The team's goal is 80%.

The testing agent measures the coverage, reads the file, writes tests for 3 additional functions, runs the tests, and measures the coverage again. The result is 80%.

**Audit.** `package.json` includes `lodash` version 4.17.15. This version has a known security issue, CVE-2020-8203, which is fixed in a later version. The audit agent identifies the issue, upgrades `lodash`, and scans the project again to confirm that the problem is resolved.

**In simple words:** Find what is not being tested and add tests. Find known security issues and fix them.

## What You Will Learn

In this lesson, you will learn:

1. How a testing agent generates tests for code that is not yet tested.
2. How tests are run and why new tests must pass before they are considered valid.
3. What coverage measures and what it does not measure.
4. How a dependency scan works.
5. Compliance rules and the patch verification process.

The testing agent has `read_file`, `write_file`, `run_tests`, and `run_coverage`. The coverage tool runs the tests in your browser and records which functions are executed. The audit agent has `read_file`, `write_file`, and `scan_deps`, and it may only modify `package.json`.

## Testing Agents

### Test Generation

**Test generation** means writing new tests. A good testing agent:

1. Finds code that is not covered by tests.
2. Reads the code to understand what each function should do.
3. Writes small tests with clear expected results, including edge cases.

The helper `progressText(done, total)` needs a normal test case, such as verifying that 45 out of 100 produces "45%". It should also include an edge case: what should happen when `total` is 0?

```js runnable title="Tests for an untested helper, including an edge case"
function progressText(done, total) {
  return Math.round((done / total) * 100) + "%";
}

const check = (name, got, wanted) => console.log((got === wanted ? "PASS " : "FAIL ") + name + " (got " + got + ")");
check("45 of 100", progressText(45, 100), "45%");
check("all done", progressText(100, 100), "100%");
check("nothing to do", progressText(0, 0), "0%");
```

Expected output:

```text
PASS 45 of 100 (got 45%)
PASS all done (got 100%)
FAIL nothing to do (got NaN%)
```

The edge case discovered a real bug: 0 divided by 0 produces `NaN`, so the page would display "NaN%". This is exactly why generated tests are valuable. When a new test fails, the testing agent should report the bug, not change the test to match the current behavior.

### Test Execution

New tests must **pass** before they count. A test that fails or does not run successfully does not contribute to coverage. Because of this, the testing agent always runs the tests after writing them and before measuring coverage.

### Coverage Analysis

**Coverage** measures how much of the code is actually executed by the tests. In this course, we use **function coverage**, which is the percentage of functions that are called by at least one test.

```js runnable title="Function coverage, the way run_coverage measures it"
const functions = ["statusLabel", "isFinished", "canRetry", "progressText", "statusColor"];
const before = ["statusLabel"];
const after = ["statusLabel", "isFinished", "canRetry", "progressText"];

const pct = (called) => Math.round((called.length / functions.length) * 100) + "%";
console.log("before: " + pct(before));
console.log("after: " + pct(after));
console.log("still not tested: " + functions.filter((f) => !after.includes(f)).join(", "));
```

Expected output:

```text
before: 20%
after: 80%
still not tested: statusColor
```

The real coverage tool determines whether a function was called by running the tests. It temporarily wraps each function so that every call is recorded during test execution.

```js runnable title="Record which functions the tests call"
const called = {};
function track(name, fn) {
  return (...args) => { called[name] = true; return fn(...args); };
}

const isFinished = track("isFinished", (s) => s === "done" || s === "failed");
const canRetry = track("canRetry", (s) => s === "failed");

isFinished("done");
isFinished("running");
console.log(JSON.stringify(called));
console.log("canRetry called: " + (called.canRetry === true));
```

Expected output:

```text
{"isFinished":true}
canRetry called: false
```

Coverage has an important limitation: a function can be marked as "covered" even if the test checks nothing useful. Reaching 100% coverage means every function was executed by a test. It does not guarantee that every function works correctly.

The agent starts by measuring coverage and finds that `web/status.js` has 20% function coverage. It reads the file and writes tests for `isFinished`, `canRetry`, and `progressText`.

It then runs the tests. All of them pass. After that, it measures coverage again and reaches 80%. In its final report, it also lists the functions that still do not have tests.

## Security Audit Agents

### Dependency Scan

A **dependency** is external code that the project uses and lists in `package.json`. A **dependency scan** compares each dependency name and version against a database of known security problems called **advisories**. Each advisory has an identifier, such as CVE-2020-8203, and information about which version fixes the issue.

```js runnable title="Compare versions against an advisory"
function less(a, b) {
  const x = a.split(".").map(Number), y = b.split(".").map(Number);
  for (let i = 0; i < 3; i++) if (x[i] !== y[i]) return x[i] < y[i];
  return false;
}

const advisory = { pkg: "lodash", fixedIn: "4.17.19", id: "CVE-2020-8203" };
for (const v of ["4.17.15", "4.17.19", "4.17.21"]) {
  console.log("lodash " + v + ": " + (less(v, advisory.fixedIn) ? "VULNERABLE (" + advisory.id + ")" : "ok"));
}
```

Expected output:

```text
lodash 4.17.15: VULNERABLE (CVE-2020-8203)
lodash 4.17.19: ok
lodash 4.17.21: ok
```

Versions must be compared number by number. As plain text, "4.17.9" would sort after "4.17.19", which is incorrect.

### Compliance

**Compliance** means following the rules set by a team, organization, or regulation. For example: "No dependency with a known high-severity security advisory may be deployed."

A security audit agent turns these rules into automated checks that can be run every time.

```js runnable title="A compliance rule as a check"
const scan = [
  { pkg: "lodash", version: "4.17.15", severity: "high" },
  { pkg: "date-fns", version: "2.30.0", severity: null },
  { pkg: "left-pad", version: "1.1.0", severity: "low" },
];
const rule = { blockAt: ["high", "critical"] };

const blockers = scan.filter((d) => rule.blockAt.includes(d.severity));
console.log(blockers.length ? "NOT compliant: " + blockers.map((d) => d.pkg + " " + d.version).join(", ") : "compliant");
console.log("warnings: " + scan.filter((d) => d.severity && !rule.blockAt.includes(d.severity)).map((d) => d.pkg).join(", "));
```

Expected output:

```text
NOT compliant: lodash 4.17.15
warnings: left-pad
```

The rule tells the agent which issues block a release and which are only warnings. This removes guesswork and makes the decision process consistent.

### Patch Check

A **patch check** runs the dependency scan again after an upgrade. It confirms that the new version is actually installed and that the update did not introduce a different security problem.

The scan finds `lodash` version 4.17.15. The agent reads `package.json`, upgrades `lodash` to 4.17.21, and changes nothing else. It then runs the scan again and finds **0 vulnerable dependencies**.

**Easy Way to Remember:** **measure, write, run, measure again** for tests. **Scan, patch, scan again** for dependencies.

Never let a testing agent change a test just because it fails. A newly added failing test may have discovered a real bug, such as the "NaN%" issue above. Report the bug instead.

## Common Mistakes

| Mistake | Wrong Idea | Correct Idea | What Goes Wrong |
|---|---|---|---|
| Coverage as quality | "100% coverage means there are no bugs." | Coverage only means the code was executed by tests. | Weak tests can hide real bugs. |
| Counting tests that never ran | "I wrote the tests, so coverage increased." | Run the tests first. | Coverage appears to improve even though the tests do not pass. |
| Testing only normal cases | "45 of 100 works." | Test edge cases such as 0 of 0 as well. | The page displays "NaN%". |
| Comparing versions as text | "4.17.9 > 4.17.19." | Compare each version number separately. | A vulnerable version may appear safe. |
| Skipping the patch check | "I updated package.json." | Run the scan again after the update. | The security fix may not actually be applied. |

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer.

## What to Remember

- **Measure, write, run, measure again.**
- Coverage shows which code was executed by tests, not whether the code is correct. Always include edge cases in your tests.
- A newly added failing test may have uncovered a real bug. Report the bug instead of changing the test.
- Scan dependencies against known security advisories and compare version numbers as numbers, not as plain text.
- Apply only the required security patch, then run the scan again to verify that the issue has been fixed.

Next: **Agent Teams for Delivery**, where all these agents work on one feature together.