Testing & Verification tutorial

Treat every AI change as a claim. Prove it or catch it with tests that fail on wrong code, in a real test lab in your browser.

Start the course

Course contents

Verifying AI Changes

  1. Done Is a Claim — Why an AI agent's "done" is a claim, not proof. Four ways a test stays green on wrong code, the one question that tells strong tests from weak ones, and the evidence ladder from weakest to strongest proof.
  2. Reading an AI Diff — Read an AI change line by line. Match every claim in the agent's summary to the diff, spot weakened, deleted and skipped tests, catch scope creep, and turn each unbacked claim into a test.
  3. Spec First: Tests Before Code — Turn a ticket into acceptance criteria and tests before the agent writes code. See every new test fail on the old code first, then pass, and catch edge cases hidden in words like "or more".
  4. Reviewing AI Code — Review AI changes with a short checklist. Catch made-up APIs, off-by-one mistakes, missing edge cases, regular expressions that look right, and scope creep, without raising false alarms.
  5. Auditing AI-Written Tests — Measure AI-written tests for real. See why 100% coverage can hide weak tests, run mutation testing and watch mutants survive, then raise the score with exact checks, remove over-mocks and spot saved answers with the bug inside.
  6. Property-Based Testing — Write rules that must hold for every input, let fast-check try hundreds of random inputs, and watch shrinking find the smallest one that breaks the AI's code. Bounds, sums, round trips and no-change properties.
  7. Integration and Contract Tests — See an over-mocked unit test pass while the real API changes. Catch it with an integration test and a contract test, make contracts strict, keep mocks honest, and learn what to mock and what to run for real.
  8. User-Flow Checks — See every unit test pass while checkout is broken. Write flow tests that act like a shopper, find things by role and label, cover happy and unhappy paths, choose the right test for each bug, and export your first verification pack.

Verification as a System

  1. Verification Gates for Agents — Turn rules into gates. See how an agent fakes green, build a test-diff guard that blocks skipped, loosened and deleted tests, write the definition of done as code with clear reasons, and wire the gates into Claude Code, Codex, Cursor and CI.
  2. Characterization Tests — Before an AI refactor, pin what old code does today. Build a golden table of edge cases, compare old and new code on hundreds of random orders, keep quirks on purpose, and follow a safe order of steps for every refactor.
  3. Security and Dependency Checks — Catch security bugs in AI changes with tests and scanners: HTML and SQL injection, secrets typed into code, and made-up or look-alike package names (slopsquatting and typosquatting). Then add them to your gates.
  4. Flaky Tests and AI — A test that is green on one run and red on another is a bug report, not bad luck. Run one AI test 300 times and find the runs that fail, fix the code instead of retrying, pass the clock in as an input, and stop tests from sharing state.
  5. Reviewing an AI Pull Request — Take one real AI pull request through every gate in order: read the claim, run the test-diff guard, write tests from the ticket and see them fail first, fix and check with planted bugs and mutation, then let the definition of done give the verdict and write a review with evidence.
  6. Capstone: Your Verification Pack — Use every gate at once. Run your guard, fail-first, mutation, secret and escaping checks on ten seeded AI changes, accept or reject each with cited evidence, verify a new function to an 80% mutation score, and build a verification pack from your own project's answers.

Testing in .NET

  1. Testing in .NET with xUnit — The course ideas in .NET: xUnit facts and theories, Assert.Equal and Assert.Throws, decimal for money, fakes behind interfaces, and the AI mistakes to look for in a C# diff.
  2. Spec First and Mutation Testing in .NET — Make C# tests fail on the old code first, turn ticket criteria into xUnit tests, and measure them with mutants: Stryker.NET in real projects, hand-made mutants in the lab, and how to read the report.
  3. Properties and Integration in .NET — Property tests with FsCheck and shrinking, API contracts with System.Text.Json, integration tests with WebApplicationFactory, and characterization tests with old-vs-new checks before an AI refactor of C# code.
  4. .NET Gates, Security and Capstone — The gates for .NET teams: a test-diff guard for xUnit and Stryker.NET config, the definition of done in C#, dotnet test and Stryker.NET in CI, secrets in appsettings.json, SQL and HTML injection, NuGet package checks, and a short capstone.