Verification as a System in Testing & Verification

Verification as a System is the second chapter of the free Testing & Verification course. It moves the checks from your eyes into gates that run on every change, and covers old code and security.

What each lesson covers:

  • Verification Gates for Agents: how agents fake green, a test-diff guard, the definition of done as code, and where gates live in Claude Code, Codex, Cursor and CI.
  • Characterization Tests: pin what old code does today, so an AI refactor cannot quietly change it.
  • Security and Dependency Checks: catch injected input, hardcoded secrets and made-up package names in AI changes.
  • Flaky Tests and AI: rerun a test on many seeded runs, find the real bug behind a flake, pass the clock in and reset shared state.
  • Reviewing an AI Pull Request: take one AI pull request through every gate in order and write a review with evidence.
  • Capstone: Your Verification Pack: judge ten AI changes with your gates, cite the evidence, and export a pack built for your own project.

Every lab runs real tests in your browser. Several exercises ask you to write the gate or the scanner yourself, and hidden tests act like an agent trying to get past it.

Lessons in this section

  1. Verification Gates for Agents

    Turn rules into gates. See how an agent fakes green, build a test-diff guard that blocks skipped, loosened and deleted tests, write the definition of done as code with clear reasons, and wire the gates into Claude Code, Codex, Cursor and CI.

    • Explain how agents fake green and why rules alone do not stop it
    • Build a test-diff guard that blocks shortcuts and allows real work
    • Write a definition of done as code with reasons
    • Place rules and gates in Claude Code, Codex, Cursor and CI
  2. Characterization Tests

    Before an AI refactor, pin what old code does today. Build a golden table of edge cases, compare old and new code on hundreds of random orders, keep quirks on purpose, and follow a safe order of steps for every refactor.

    • Tell a characterization test from a spec test
    • Pin current behaviour with a golden table of edge cases
    • Compare old and new code on many random inputs
    • Pin and document quirks instead of fixing them mid-refactor
  3. Security and Dependency Checks

    Catch security bugs in AI changes with tests and scanners: HTML and SQL injection, secrets typed into code, and made-up or look-alike package names (slopsquatting and typosquatting). Then add them to your gates.

    • Test that user text cannot inject into HTML or SQL
    • Escape HTML correctly and use parameterized queries
    • Build a secret scanner that allows safe lines
    • Check package names for made-up and look-alike packages
  4. Flaky Tests and AI

    A test that is green on one run and red on another is a bug report, not bad luck. Run one AI test 300 times and find the runs that fail, fix the code instead of retrying, pass the clock in as an input, and stop tests from sharing state.

    • Explain why one green run is not evidence
    • Rerun a check on many seeded runs and list the failing runs
    • Fix the cause of a flake instead of retrying
    • Pass time and randomness into code so tests control them
  5. Reviewing an AI Pull Request

    Take one real AI pull request through every gate in order: read the claim, run the test-diff guard, write tests from the ticket and see them fail first, fix and check with planted bugs and mutation, then let the definition of done give the verdict and write a review with evidence.

    • Review an AI pull request in a fixed order of gates
    • Spot a weakened assertion with the test-diff guard
    • Judge the PR with your own tests from the ticket, not its tests
    • Use planted bugs and mutation to check your tests
  6. Capstone: Your Verification Pack

    Use every gate at once. Run your guard, fail-first, mutation, secret and escaping checks on ten seeded AI changes, accept or reject each with cited evidence, verify a new function to an 80% mutation score, and build a verification pack from your own project's answers.

    • Run the gates from the course on real AI changes
    • Accept or reject ten AI changes with evidence, without false alarms
    • Cite the gate that catches each rejected change
    • Verify a new function to an 80% mutation score