Testing Agents

By the Syncfusion DevTutorials team · Last updated

How a testing agent finds untested code, writes and runs tests, and raises coverage, and how an audit agent scans dependencies, patches them and checks the patch.

Think about a safety inspector checking a new building.

The inspector has a checklist of everything that must be inspected: every door, every fire alarm, and every stair rail. At the end, they can say, "We checked 80 out of 100 items." The remaining 20 items are unknown, and unknown does not mean safe.

A second inspector checks the parts that came from outside suppliers, such as lifts, wiring, and boilers. They compare each model number against a list of recalled products. If a boiler appears on the recall list, it must be replaced and then checked again.

This lesson introduces two agents that perform these kinds of checks for software.

The main idea: A testing agent finds code that is not covered by tests, writes tests for it, and measures coverage again. A security audit agent checks project dependencies for known security issues, updates them, and verifies the fix.

A Real-World Example

Testing. The file web/status.js contains 5 small helper functions. Only one of them, statusLabel, currently has a test.

That means the project has 20% function coverage. The team's goal is 80%.

The testing agent measures the coverage, reads the file, writes tests for 3 additional functions, runs the tests, and measures the coverage again. The result is 80%.

Audit. package.json includes lodash version 4.17.15. This version has a known security issue, CVE-2020-8203, which is fixed in a later version. The audit agent identifies the issue, upgrades lodash, and scans the project again to confirm that the problem is resolved.

In simple words: Find what is not being tested and add tests. Find known security issues and fix them.

What You Will Learn

In this lesson, you will learn:

  1. How a testing agent generates tests for code that is not yet tested.
  2. How tests are run and why new tests must pass before they are considered valid.
  3. What coverage measures and what it does not measure.
  4. How a dependency scan works.
  5. Compliance rules and the patch verification process.

The testing agent has read_file, write_file, run_tests, and run_coverage. The coverage tool runs the tests in your browser and records which functions are executed. The audit agent has read_file, write_file, and scan_deps, and it may only modify package.json.

Top: the testing agent increases coverage from 20% to 80%. Bottom: the audit agent finds a vulnerable dependency, patches it, and scans again.

Testing Agents

Test Generation

Test generation means writing new tests. A good testing agent:

  1. Finds code that is not covered by tests.
  2. Reads the code to understand what each function should do.
  3. Writes small tests with clear expected results, including edge cases.

The helper progressText(done, total) needs a normal test case, such as verifying that 45 out of 100 produces "45%". It should also include an edge case: what should happen when total is 0?

Tests for an untested helper, including an edge case

function progressText(done, total) {
  return Math.round((done / total) * 100) + "%";
}

const check = (name, got, wanted) => console.log((got === wanted ? "PASS " : "FAIL ") + name + " (got " + got + ")");
check("45 of 100", progressText(45, 100), "45%");
check("all done", progressText(100, 100), "100%");
check("nothing to do", progressText(0, 0), "0%");

Expected output:

PASS 45 of 100 (got 45%)
PASS all done (got 100%)
FAIL nothing to do (got NaN%)

The edge case discovered a real bug: 0 divided by 0 produces NaN, so the page would display "NaN%". This is exactly why generated tests are valuable. When a new test fails, the testing agent should report the bug, not change the test to match the current behavior.

Test Execution

New tests must pass before they count. A test that fails or does not run successfully does not contribute to coverage. Because of this, the testing agent always runs the tests after writing them and before measuring coverage.

Coverage Analysis

Coverage measures how much of the code is actually executed by the tests. In this course, we use function coverage, which is the percentage of functions that are called by at least one test.

Function coverage, the way run_coverage measures it

const functions = ["statusLabel", "isFinished", "canRetry", "progressText", "statusColor"];
const before = ["statusLabel"];
const after = ["statusLabel", "isFinished", "canRetry", "progressText"];

const pct = (called) => Math.round((called.length / functions.length) * 100) + "%";
console.log("before: " + pct(before));
console.log("after: " + pct(after));
console.log("still not tested: " + functions.filter((f) => !after.includes(f)).join(", "));

Expected output:

before: 20%
after: 80%
still not tested: statusColor

The real coverage tool determines whether a function was called by running the tests. It temporarily wraps each function so that every call is recorded during test execution.

Record which functions the tests call

const called = {};
function track(name, fn) {
  return (...args) => { called[name] = true; return fn(...args); };
}

const isFinished = track("isFinished", (s) => s === "done" || s === "failed");
const canRetry = track("canRetry", (s) => s === "failed");

isFinished("done");
isFinished("running");
console.log(JSON.stringify(called));
console.log("canRetry called: " + (called.canRetry === true));

Expected output:

{"isFinished":true}
canRetry called: false

Coverage has an important limitation: a function can be marked as "covered" even if the test checks nothing useful. Reaching 100% coverage means every function was executed by a test. It does not guarantee that every function works correctly.

Testing Agent: Reach 80% Coverage

The agent starts by measuring coverage and finds that web/status.js has 20% function coverage. It reads the file and writes tests for isFinished, canRetry, and progressText.

It then runs the tests. All of them pass. After that, it measures coverage again and reaches 80%. In its final report, it also lists the functions that still do not have tests.

Security Audit Agents

Dependency Scan

A dependency is external code that the project uses and lists in package.json. A dependency scan compares each dependency name and version against a database of known security problems called advisories. Each advisory has an identifier, such as CVE-2020-8203, and information about which version fixes the issue.

Compare versions against an advisory

function less(a, b) {
  const x = a.split(".").map(Number), y = b.split(".").map(Number);
  for (let i = 0; i < 3; i++) if (x[i] !== y[i]) return x[i] < y[i];
  return false;
}

const advisory = { pkg: "lodash", fixedIn: "4.17.19", id: "CVE-2020-8203" };
for (const v of ["4.17.15", "4.17.19", "4.17.21"]) {
  console.log("lodash " + v + ": " + (less(v, advisory.fixedIn) ? "VULNERABLE (" + advisory.id + ")" : "ok"));
}

Expected output:

lodash 4.17.15: VULNERABLE (CVE-2020-8203)
lodash 4.17.19: ok
lodash 4.17.21: ok

Versions must be compared number by number. As plain text, "4.17.9" would sort after "4.17.19", which is incorrect.

Compliance

Compliance means following the rules set by a team, organization, or regulation. For example: "No dependency with a known high-severity security advisory may be deployed."

A security audit agent turns these rules into automated checks that can be run every time.

A compliance rule as a check

const scan = [
  { pkg: "lodash", version: "4.17.15", severity: "high" },
  { pkg: "date-fns", version: "2.30.0", severity: null },
  { pkg: "left-pad", version: "1.1.0", severity: "low" },
];
const rule = { blockAt: ["high", "critical"] };

const blockers = scan.filter((d) => rule.blockAt.includes(d.severity));
console.log(blockers.length ? "NOT compliant: " + blockers.map((d) => d.pkg + " " + d.version).join(", ") : "compliant");
console.log("warnings: " + scan.filter((d) => d.severity && !rule.blockAt.includes(d.severity)).map((d) => d.pkg).join(", "));

Expected output:

NOT compliant: lodash 4.17.15
warnings: left-pad

The rule tells the agent which issues block a release and which are only warnings. This removes guesswork and makes the decision process consistent.

Patch Check

A patch check runs the dependency scan again after an upgrade. It confirms that the new version is actually installed and that the update did not introduce a different security problem.

Audit Agent: Dependency Scan

The scan finds lodash version 4.17.15. The agent reads package.json, upgrades lodash to 4.17.21, and changes nothing else. It then runs the scan again and finds 0 vulnerable dependencies.

Easy Way to Remember: measure, write, run, measure again for tests. Scan, patch, scan again for dependencies.

Never let a testing agent change a test just because it fails. A newly added failing test may have discovered a real bug, such as the "NaN%" issue above. Report the bug instead.

Common Mistakes

MistakeWrong IdeaCorrect IdeaWhat Goes Wrong
Coverage as quality"100% coverage means there are no bugs."Coverage only means the code was executed by tests.Weak tests can hide real bugs.
Counting tests that never ran"I wrote the tests, so coverage increased."Run the tests first.Coverage appears to improve even though the tests do not pass.
Testing only normal cases"45 of 100 works."Test edge cases such as 0 of 0 as well.The page displays "NaN%".
Comparing versions as text"4.17.9 > 4.17.19."Compare each version number separately.A vulnerable version may appear safe.
Skipping the patch check"I updated package.json."Run the scan again after the update.The security fix may not actually be applied.

Exercises

Each exercise ends with a value on the last line. Press Check to compare it with the expected answer.

What to Remember

  • Measure, write, run, measure again.
  • Coverage shows which code was executed by tests, not whether the code is correct. Always include edge cases in your tests.
  • A newly added failing test may have uncovered a real bug. Report the bug instead of changing the test.
  • Scan dependencies against known security advisories and compare version numbers as numbers, not as plain text.
  • Apply only the required security patch, then run the scan again to verify that the issue has been fixed.

Next: Agent Teams for Delivery, where all these agents work on one feature together.