Verifying AI Changes in Testing & Verification

Verifying AI Changes is the first chapter of the free Testing & Verification course. It teaches you to treat every AI change as a claim, and to prove it or catch it with real evidence.

What each lesson covers:

  • Done Is a Claim: why "done" and green tests are not proof, four ways a test passes on wrong code, the key question "would it fail if the code were wrong?", and the evidence ladder.
  • Reading an AI Diff: match every claim in the agent's summary to the diff, spot weakened, deleted and skipped tests, and catch scope creep.
  • Spec First: Tests Before Code: turn a ticket into acceptance criteria and tests before the agent writes code, and see each new test fail first.
  • Reviewing AI Code: a short checklist for made-up APIs, off-by-one mistakes, missing edge cases and scope creep.
  • Auditing AI-Written Tests: why 100% coverage can hide weak tests, mutation testing, over-mocking and saved answers with the bug inside.
  • Property-Based Testing: rules that must hold for every input, hundreds of random inputs from fast-check, and shrinking to the smallest failing case.
  • Integration and Contract Tests: catch an over-mocked test that passes while the real API changes, and decide what to mock and what to run for real.
  • User-Flow Checks: flow tests that act like a shopper, happy and unhappy paths, and your first verification pack.

Every test box runs real Vitest-style tests in your browser, in the Verify Lab. Each lesson ends with exercises graded by running your tests against the correct code and against hidden buggy versions.

Lessons in this section

  1. Done Is a Claim

    Why an AI agent's "done" is a claim, not proof. Four ways a test stays green on wrong code, the one question that tells strong tests from weak ones, and the evidence ladder from weakest to strongest proof.

    • Tell a claim from evidence
    • Spot four ways a test passes on wrong code
    • Ask whether a test would fail if the code were wrong, and check it by planting bugs
    • Order the eight rungs of the evidence ladder
  2. Reading an AI Diff

    Read an AI change line by line. Match every claim in the agent's summary to the diff, spot weakened, deleted and skipped tests, catch scope creep, and turn each unbacked claim into a test.

    • Read added and removed lines in a diff, file by file
    • Match each summary claim to a line in the diff
    • Spot weakened, deleted and skipped tests
    • Spot scope creep
  3. Spec First: Tests Before Code

    Turn a ticket into acceptance criteria and tests before the agent writes code. See every new test fail on the old code first, then pass, and catch edge cases hidden in words like "or more".

    • Turn a ticket into testable acceptance criteria
    • Write one test per criterion with hand-worked values
    • Run tests on the old code and confirm they fail first
    • Use the red, green loop with an AI agent
  4. Reviewing AI Code

    Review AI changes with a short checklist. Catch made-up APIs, off-by-one mistakes, missing edge cases, regular expressions that look right, and scope creep, without raising false alarms.

    • Catch made-up methods and packages with the type check
    • Spot off-by-one mistakes in loops, counts and limits
    • Test the edge cases: empty, zero, boundary, rounding, text, dates
    • Find regular expressions that match too much
  5. Auditing AI-Written Tests

    Measure AI-written tests for real. See why 100% coverage can hide weak tests, run mutation testing and watch mutants survive, then raise the score with exact checks, remove over-mocks and spot saved answers with the bug inside.

    • Explain the gap between coverage and mutation score
    • Run mutation testing and read killed and surviving mutants
    • Raise the mutation score with exact checks at the edges
    • Spot and remove over-mocking
  6. Property-Based Testing

    Write rules that must hold for every input, let fast-check try hundreds of random inputs, and watch shrinking find the smallest one that breaks the AI's code. Bounds, sums, round trips and no-change properties.

    • Write a property with generators, a rule and fc.assert
    • Read a counterexample and its shrink steps
    • Use bounds, sums, round trips and no-change properties
    • Combine properties with exact examples
  7. Integration and Contract Tests

    See an over-mocked unit test pass while the real API changes. Catch it with an integration test and a contract test, make contracts strict, keep mocks honest, and learn what to mock and what to run for real.

    • Explain how a mock can hide a real API change
    • Write an integration test that runs real parts together
    • Write a strict contract with required fields and types
    • Check real responses and mock data against the same contract
  8. User-Flow Checks

    See every unit test pass while checkout is broken. Write flow tests that act like a shopper, find things by role and label, cover happy and unhappy paths, choose the right test for each bug, and export your first verification pack.

    • Explain why unit tests can pass while a user flow is broken
    • Write a flow test that finds things by role and label
    • Test the happy path and the unhappy path
    • Pick unit, contract or flow tests for each kind of bug