Animation: four mutants of the shipping code go into the weak test suite and all survive. The same four go into the strong suite and all are killed.

# Auditing AI-Written Tests Course: [Testing & Verification](https://www.devtutorials.ai/learn/testing-verification) · Section: Verifying AI Changes Source: https://www.devtutorials.ai/learn/testing-verification/verifying-ai-changes/auditing-ai-tests > Measure AI-written tests for real. See why 100% coverage can hide weak tests, run mutation testing and watch mutants survive, then raise the score with exact checks, remove over-mocks and spot saved answers with the bug inside. ## What Auditing Tests Means **Audit** = a careful check of something that others will trust. AI agents write a lot of tests. Many look complete. Some even run every line of the code. But do they catch bugs? In this lesson, you will measure that for real: - with **coverage**, which tells you which lines ran - with **mutation testing**, which tells you which bugs your tests would catch You will also spot three test smells that AI tests often have: - weak checks - over-mocking - saved answers with the bug inside **Simple lesson:** a test suite is evidence only if it fails on wrong code. Mutation testing measures that. ### A Real-World Example Think about a security guard at a building gate. - The guard's report says: "I checked every person who came in." That is like coverage. - But did the guard look at their ID cards, or just wave at them? - The only way to know: send in a person with a fake ID. Does the guard stop them? Mutation testing sends fake IDs to your tests. Each fake ID is a small bug. - **Unit test** = checks one small piece of code, like one function. Fast. - **Integration test** = checks that two or more pieces work together, like your code and a real API. - **End-to-end test** = checks a whole user journey in the running app, like "add to cart, then pay". This lesson is about unit tests. **Integration and Contract Tests** and **User-Flow Checks** cover the other two. - **Mock** = a fake part that stands in for a real one, and records how it was called. In Vitest: `vi.fn()`. - **Stub** = a fake part that only returns a fixed answer. - **Fake** = a simple working copy, like an in-memory list instead of a real database. They are useful at the edges, like the network or the clock. They are risky when they replace the code you are testing. ## What You Will Learn In this lesson, you will learn: 1. Why 95% coverage can still mean weak tests. 2. What a **mutant** is, and what the **mutation score** means. 3. How to raise the mutation score with exact checks at the edges. 4. How **over-mocking** hides real bugs. 5. Why expected values copied from a run are dangerous. ## Coverage Is Not Quality ### What Coverage Measures **Coverage** = the share of lines (or branches) that ran during the tests. - 100% coverage means every line ran at least once. - It does **not** mean every line was checked. - A test with `toBeDefined()` runs the code, then checks almost nothing. ### What a Mutant Is **Mutant** = a copy of your code with one small change, made on purpose. Examples of mutations: - `>=` becomes `>` - `499` becomes `500` - `40` becomes `41` - `Math.round` becomes `Math.floor` If a test fails on a mutant, the mutant is **killed**. If every test still passes, the mutant **survived**. **Mutation score** = killed mutants ÷ all mutants. Higher is better.

Same code, same coverage. Only the strong suite notices the changes.

### Watch Mutants Survive The agent wrote three tests for `shippingFee`. Every line of the code runs. Press **▶ Run tests**. Green. Now press **🧬 Run mutation**. - **The Mutants tab:** most mutants survived. The score is low. - **Why:** `toBeGreaterThan(0)` and `toBeDefined()` accept almost any answer. - **Click a line number:** the editor jumps to the line that was changed. ### Exact Answers at the Edges Here are four better tests for the same code. Press **🧬 Run mutation**. - **The Mutants tab:** every mutant is killed. - **Why:** each test has an exact answer (`toBe(40)`), and they sit on the edges: ₹498 and ₹499, 5 kg and 6 kg. **Two moves raise the mutation score:** - Replace loose checks (`toBeGreaterThan`, `toBeDefined`, `toBeTruthy`) with exact ones (`toBe`, `toEqual`). - Test both sides of every limit: ₹498 and ₹499, 5 kg and 6 kg. **This lab's mutation engine is a small teaching copy.** Real projects use **Stryker**. Stryker makes many more kinds of mutants and can test only the lines that changed. The idea is the same. **Simple lesson:** coverage says the code ran. Mutation score says the tests would notice if it were wrong. ## Over-Mocking **Over-mocking** = faking so much that the real code under test never runs. Agents often mock more than people do. A mock that returns the expected answer turns any test green. Press **▶ Run tests**. Green. - **The mock:** `vi.fn().mockReturnValue(18)` gives 18, so the total is 118. - **The real code:** `gstOn` uses 12%, not 18%. It never ran. - **Now try:** add a test with no mock: `expect(orderTotal(100)).toBe(118);`. Run again. Red: 112. **Mock at the edges only.** - Fine to mock: the network, the clock, random numbers, emails. - Not fine: the function you are testing, or plain maths it uses. - Ask: "Does the real code still run in this test?" **Simple lesson:** if the mock gives the answer, the test only checks the mock. ## Saved Answers With the Bug Inside Sometimes the expected value is copied from a run of the code. A **snapshot** test saves output like this to a file, so later runs can compare against it. - That is fine when you have checked the saved output. - It is dangerous when the code was already wrong. The bug gets saved as the "right" answer. - People call this **blessing** a snapshot. In this lab, the saved answer is written into the test as an object. Press **▶ Run tests**. Green. - **Look closely:** `total: 100`. With 18% GST on ₹100, the total should be ₹118. - **Now try:** change it to `total: 118` and run again. Red. The code forgot to add the GST. **Simple lesson:** check every saved answer by hand before you trust it. ## Your Turn: Raise the Score Press **🧬 Run mutation**. Read which mutants survived. - For each survivor, add a test where that change gives a different answer. - Run mutation again. Aim for 80% or more. **Simple lesson:** every surviving mutant points to a test you have not written yet. ## TL;DR 1. **Coverage** = which lines ran. It does not say if they were checked. 2. **Mutant** = a copy of the code with one small change. 3. **Killed** = a test failed on the mutant. **Survived** = all tests still passed. 4. **Mutation score** = killed ÷ all mutants. Teams often ask for 80% on changed code. 5. **Raise it** with exact checks, on both sides of every limit. 6. **Over-mocking** = the mock gives the answer, so the real code never runs. 7. **Blessed snapshot** = a saved answer copied from buggy code. ## Common mistakes | Mistake | Wrong idea | Correct idea | What goes wrong | |---|---|---|---| | Trusting coverage | "95% coverage means good tests." | Measure the mutation score too. | Bugs pass through lines that ran but were not checked. | | Loose checks | "toBeGreaterThan(0) is safe." | Use the exact answer. | ₹41 or ₹9999 also passes. | | Mocking the unit | "Mock the tax to keep it simple." | Mock only the edges, like the network. | The 12% bug ships. | | Copying output | "The code printed it, so it is the answer." | Work the answer out by hand. | The bug is saved inside the test. | | Chasing 100% | "Every mutant must die." | Aim for a set threshold, like 80%. | Time is lost on mutants that change nothing. | ## Check Your Understanding ## Exercises ## What to Remember - Coverage tells you what ran. Mutation score tells you what was checked. - Each surviving mutant is a test you still need. - Use exact answers on both sides of every limit. - Mock only at the edges. The code under test must really run. - Check saved answers by hand before you trust them. ## Further Reading - [Stryker Mutator](https://stryker-mutator.io/): mutation testing for JavaScript, TypeScript and .NET. - [Vitest mocking guide](https://vitest.dev/guide/mocking.html): when and how to use `vi.fn`. Next: **Property-Based Testing**. You will write rules for hundreds of random inputs. Then the tool finds the smallest input that breaks them.