Animation: on the left the detector light blinks, which looks fine. On the right smoke rises to the detector and no alarm sounds.

# Done Is a Claim Course: [Testing & Verification](https://www.devtutorials.ai/learn/testing-verification) · Section: Verifying AI Changes Source: https://www.devtutorials.ai/learn/testing-verification/verifying-ai-changes/done-is-a-claim > Why an AI agent's "done" is a claim, not proof. Four ways a test stays green on wrong code, the one question that tells strong tests from weak ones, and the evidence ladder from weakest to strongest proof. ## What "Done" Means An AI coding agent often ends its work with a happy message. For example: - "Done! ✅" - "All tests pass." - "10% discount works." That message is a **claim**. It is not proof. In this course, you will learn to ask one question about every change: > What evidence says this really works? **Simple lesson:** "done" is something the agent says. Evidence is something you can check. ### A Real-World Example Think about a smoke detector on the ceiling. - Its small light blinks every minute. - The blink tells you the battery is fine. - But does it beep when there is smoke? The blink cannot tell you. - The only real check: put smoke near it, and listen for the alarm. A test that cannot fail is like that blinking light. It looks fine. It proves nothing.

A blinking light is not a working alarm. A green test is not working code.

Now meet an AI change. - The agent added a discount function. - It wrote a test. - It says "Done! All tests pass." Press **▶ Run tests**. - **The Tests tab:** the test passes. Green. - **The test file:** it compares the answer with itself. It can never fail. - **Now try:** change the test to `.toBe(450)` and run it again. It turns red. The code gives 490, not 450. - **The bug:** the code takes ₹10 off, not 10%. **Simple lesson:** the agent's test was green, and the code was still wrong. **Test** = a small piece of code that checks other code gives the right answer. Most tests have three steps. People call them **Arrange, Act, Assert**: 1. **Arrange:** set up the input. For example, a price of ₹100. 2. **Act:** run the code you want to check. 3. **Assert:** check the answer with `expect(...)`. - A test **passes** (green) when every check is true. - A test **fails** (red) when a check is false. - A **test runner** runs all the tests and shows the results. This course uses Vitest. Press **▶ Run tests**. Then change `18` to `20` and run it again. The test turns red, and the message says what it got. ## What You Will Learn In this lesson, you will learn: 1. Why "done" is a claim, not proof. 2. Four ways a test can be green while the code is wrong. 3. The one question that tells a strong test from a weak one. 4. The evidence ladder: eight kinds of proof, from weakest to strongest. 5. How to judge five real-looking AI changes yourself. ## Claims and Evidence ### Claim **Claim** = something someone says is true. Examples of claims: - The agent writes "Fixed the bug." - The summary says "Handles an empty cart." - A teammate says "I tested it." A claim may be true. You just cannot tell yet. ### Evidence **Evidence** = something you can run or read that shows the claim is true. Examples of evidence: - A test that fails on wrong code and passes on right code. - A test that failed before the fix and passes after it. - The app working when you click through it. ### Oracle **Oracle** = the part of a test that knows the right answer. - In `expect(applyDiscount(500, 10)).toBe(450)`, the oracle is `450`. - You work it out yourself: 10% of 500 is 50, and 500 minus 50 is 450. - A test with a weak oracle cannot spot a wrong answer. **Where the right answer comes from matters.** - Work it out by hand, or take it from the ticket. - Never take it from the code you are testing. - If the code is wrong, its answer is wrong too. **Simple lesson:** a claim says it works. Evidence shows it works. The oracle is what makes evidence count. ## Green Is Not Proof A test can pass while the code is wrong. Here are four common ways. AI agents make all four. ### A Test That Cannot Fail You saw this one already. - `expect(applyDiscount(500, 10)).toBe(applyDiscount(500, 10))` - Both sides run the same code. They are always equal. - So the test is green for right code and wrong code. This is called a **tautology**: a check that is always true. ### A Test With No Check Press **▶ Run tests**. - **The Tests tab:** "1 passed (0 checks)". - **Why:** the test calls `isFreeDelivery(499)` but never uses `expect`. - **The bug it hides:** the rule says ₹499 **or more** is free. The code uses `>`, so ₹499 pays. - **Now try:** add `expect(isFreeDelivery(499)).toBe(true);` inside the test and run it again. ### A Test That Copies the Code Press **▶ Run tests**. - **The test:** it works out `expected` with `Math.floor`, the same formula as the code. - **The problem:** if the formula is wrong, the test is wrong in the same way. - **The truth:** 2.678 should round to 2.68. `Math.floor` gives 2.67. - **Now try:** replace `expected` with `2.68` and run it again. ### A Skipped Test Press **▶ Run tests**. - **The Tests tab:** "1 passed | 1 skipped". - **Why:** `it.skip` tells the runner not to run the test at all. - **The bug it hides:** the cart ignores the quantity. 3 pens at ₹10 should cost 30. - **Now try:** change `it.skip` to `it` and run again. **Agents sometimes skip a failing test to get green.** - Look for `.skip`, `.todo` or "skipped" in the output. - Look for a test that was deleted, or a check that was made weaker. - A skipped test did not run, so it caught nothing. ### A Test With Real Numbers Now the same discount bug, with a strong test. Press **▶ Run tests**. - **The Tests tab:** red. "expected 490 to be 450". - **Why it works:** `450` was worked out by hand, not by the code. - **Now try:** open `src/discount.ts`. Fix the code to `Math.round(price * (100 - percent)) / 100`. Run again. Green. Here is the difference between a weak test and a strong test, side by side.

A weak test passes on both versions. A strong test fails on the buggy one.

**Simple lesson:** green only means the tests found nothing. It does not mean there was nothing to find. ## The Key Question ### Would This Test Fail If the Code Were Wrong? Ask this about every test. It is the most useful question in this course. - If **yes**, the test is real evidence. - If **no**, the test is only decoration. You can answer it for real. Plant a bug in the code and run the tests. A good test turns red. ### Plant Bugs and Watch This lab hides five buggy versions of `applyDiscount`. Each one is a mistake an AI could make. Press **▶ Run tests**, then press **🐞 Plant bugs**. - **The Bug check tab:** the test catches only 1 of 5 bugs. - **Why:** with ₹100 and 10%, "₹10 off" and "10% off" give the same answer, 90. - **Now try:** add a second test with ₹500. Plant the bugs again. More are caught. - **The Evidence tab:** each rung lights up when its evidence exists. **Pick inputs where wrong answers look different.** - ₹100 hides many mistakes, because 10% of 100 is 10. - ₹199.99 shows rounding mistakes, because it has paise. - 0% shows "special case" mistakes. **Simple lesson:** a test is only as good as the bugs it can catch. ## The Evidence Ladder Some evidence is stronger than other evidence. The **evidence ladder** puts it in order, from weakest at the bottom to strongest at the top.

Each rung adds proof the rung below it cannot give.

Click each rung to see what it proves and what it misses. - **Rungs 1 to 3** are where most AI changes stop. - **Rungs 4 to 8** are what this course teaches you to add. - **You do not always need all eight.** A small text change needs less proof than a payment change. **Simple lesson:** always know which rung your evidence is on. Climb higher when the change matters more. ## Is AI-Written Code Worse? ### What Studies Found Recent research looked at tests written by AI agents. - One 2026 study looked at about 86,000 test changes made by five coding agents. It found that about four in five had weak checks or no real checks. - Another study of over a million commits found that agents often use more **mocks** than people do. A mock is a fake part that stands in for a real one. You learn about mocks in **Integration and Contract Tests**. - Giving AI feedback from planted bugs made its tests much stronger in one study. ### The Other Side - A 2026 report found that recent tests written by one AI model were about as strong as tests written by people. - So the lesson is not "AI tests are bad". - The lesson is: **measure every test the same way, no matter who wrote it.** **This course is about checking, not blaming.** A test written by a person can be just as weak. The same question works for both: would it fail if the code were wrong? **Simple lesson:** do not trust or distrust a test because of who wrote it. Check what it can catch. ## Practice: Five Green AI Changes An agent made five changes to a shop app. Every change came with tests, and every test is green. Three of them are broken. 1. Press **▶ Run the agent's tests**. All five turn green. 2. Open each change. Read the code and the test. 3. Pick **Really works** or **Broken** for each one. 4. Press **🔍 Reveal the truth**. A hidden test with hand-worked numbers runs on each change. Things to look for: - **Does the test compare the code with itself?** - **Does the test copy the code's formula?** - **Does it try the edge?** For example, exactly ₹499. - **What evidence came with it?** "Failed first" and "checked in the app" are higher rungs. **Simple lesson:** green tests looked the same for all five. Only reading the tests showed the difference. ## How This Course Works Every lesson follows the same steps: 1. An AI change looks done. 2. Evidence shows it is not. 3. You build the check that catches it. The lessons in this course: - **Done Is a Claim** (this lesson) - **Reading an AI Diff** - **Spec First: Tests Before Code** - **Reviewing AI Code** - **Auditing AI-Written Tests** - **Property-Based Testing** - **Integration and Contract Tests** - **User-Flow Checks** - **Verification Gates for Agents** - **Characterization Tests** - **Security and Dependency Checks** - **Capstone: Your Verification Pack** - Four .NET lessons come in a later release. At the end, you export a **verification pack** for your own project. It works with Claude Code, Codex and Cursor. **Simple lesson:** each lesson adds one rung of evidence you can use on real AI changes. ## Use It at Work Do this on the next change an agent says is "done": - **Run the tests yourself.** `npx vitest run`. Do not trust a summary. - **Look for skips.** `npx vitest run --reporter=verbose` lists every test, skipped ones too. - **Ask the agent for proof:** ```text Show me the exact test command you ran and its full output. List the tests you added for this change, and what each one checks. ``` - **Check the claim against the ladder:** types clean, tests ran, new tests exist for the change, nothing skipped. ## TL;DR 1. **Claim** = something the agent says, like "done". 2. **Evidence** = something you can run or read that shows it is true. 3. **Oracle** = the part of a test that knows the right answer. Work it out by hand. 4. **Green is not proof.** Four ways a test passes on wrong code: - It compares the code with itself. - It has no `expect`. - It copies the code's formula. - It is skipped. 5. **The key question** = would this test fail if the code were wrong? 6. **Evidence ladder** = eight kinds of proof, from "the agent says done" to "a human reviewed it". ## Common mistakes | Mistake | Wrong idea | Correct idea | What goes wrong | |---|---|---|---| | Trusting "Done!" | "The agent said it works." | It is a claim. Look for evidence. | Broken code ships with a confident message. | | Trusting green | "All tests pass, so it works." | Green means the tests found nothing. Check what they look for. | A weak test hides a real bug. | | Expected value from the code | "I ran the function and copied its answer into the test." | Work out the answer by hand or from the ticket. | The test locks in the bug. | | Ignoring "skipped" | "1 skipped is fine." | A skipped test ran nothing. Find out why it was skipped. | The one test that would catch the bug never runs. | | Easy inputs only | "I tested ₹100 and it worked." | Use inputs where wrong answers look different: paise, 0, edges. | Different bugs give the same answer and slip through. | | Blaming the author | "A person wrote it, so it is fine." | Measure every test the same way. | Weak human tests get no review. | ## Check Your Understanding ## Exercises ## What to Remember - "Done" is a claim. Ask what evidence says it works. - A green test only means the tests found nothing. - Work out expected values by hand. Never copy them from the code. - Watch for tests that cannot fail, tests with no check, copied formulas and skipped tests. - Ask about every test: would it fail if the code were wrong? - Know which rung of the evidence ladder you are on. ## Further Reading - [Vitest documentation](https://vitest.dev/guide/): the test runner this course's lab copies. - [Stryker Mutator](https://stryker-mutator.io/): a real tool that plants bugs in your code to test your tests. You use it in **Auditing AI-Written Tests**. - [fast-check](https://fast-check.dev/): a tool that tries hundreds of random inputs. You use it in **Property-Based Testing**. Next: **Reading an AI Diff**. You will check what an AI change really did, line by line, and catch claims its summary cannot back up.