Animation: a row of CI runs lights up one by one. Most are green. A few are red. One run alone looks fine, but many runs show the red ones.

# Flaky Tests and AI Course: [Testing & Verification](https://www.devtutorials.ai/learn/testing-verification) · Section: Verification as a System Source: https://www.devtutorials.ai/learn/testing-verification/verification-system/flaky-tests > A test that is green on one run and red on another is a bug report, not bad luck. Run one AI test 300 times and find the runs that fail, fix the code instead of retrying, pass the clock in as an input, and stop tests from sharing state. ## Green Once Is Not Green An agent runs the tests once. They are green. It says "done". But some tests are green on one run and red on the next, with no code change. **Flaky test** = a test that sometimes passes and sometimes fails on the same code. Agents make flaky tests often: - they use the real clock, like `new Date()` - they use `Math.random()` without a seed - they let tests share one list, one file or one database When a flaky test goes red, many people (and agents) just press "re-run" until it is green. **Simple lesson:** a flaky test is a bug report. Find the cause. Do not retry until green. ### A Real-World Example Think about a school bell that rings late one day in twenty. - On most days, nobody notices. - On the day of the exam, half the class is late. - "Just ring it again" does not fix the wire. Someone has to open the box. A flaky test is that bell. Re-running it hides the loose wire. ## What You Will Learn In this lesson, you will learn: 1. Why one green run is not evidence. 2. How to rerun a test on many **seeded** runs and list the runs that fail. 3. Why the fix is usually in the code, not the test. 4. How to pass the clock in, so tests choose the time. 5. How to stop tests from sharing state. ## One Run vs Many Runs ### Green on One Run **Seed** = a starting number for random values. The same seed always gives the same "random" numbers. In this lab, seed 7 means "CI run number 7". Press **▶ Run tests**. - **The Tests tab:** green. - **`AI_SUMMARY.md`:** "Tests pass ✅". - **The question:** was that luck? ### The Same Test on 300 Runs Now the same check runs once for each seed from 1 to 300. Press **▶ Run tests**. - **The Tests tab:** red. The failing runs are `66, 77, 221, 232`. - **4 runs in 300** gave two seats the same pickup code. - **This is a real bug.** Two people in one booking get the same code.

One run shows one box. Many runs show the red ones.

### Fix the Code, Not the Test The wrong fixes: - re-run until green - add `retry: 3` to the test - delete the test The right fix: make the codes unique inside one booking. Press **▶ Run tests**. Green on all 300 runs. Read `src/seats.ts`: it skips a code it already used. **Retries hide bugs.** A test that passes "on the second try" still failed once. That failure happened to a real user too. Use reruns to **find** a flake, never to make it green. **Simple lesson:** run a new test many times. If it ever fails, the code or the test has a bug. ## Time: Pass the Clock In ### A Test That Depends on the Real Clock The agent wrote `greet()` with `new Date()` inside. Press **▶ Run tests**. Green, at any hour. - **The first test** (in the comment) was green only before noon. In the afternoon CI, it went red. - **The "fix":** the test now copies the code's own `if` rules. - **The problem:** it checks the code against itself. If the rule is wrong, the test is wrong the same way. ### Pass the Time In Change the code so the time is an **input**: `greet(now: Date)`. Now each test picks its own time. Press **▶ Run tests**, then **🐞 Plant bugs**. - **The Tests tab:** green on every run, at every hour. - **The Bug check tab:** 1 of 5 caught. The tests use 9:00 and 16:59, so the edges at 12:00 and 17:00, very early hours and the evening are never tested. - **Now try:** add tests for `3:00`, `11:59`, `12:00` and `17:00`. Plant bugs again until 5 of 5 are caught. **The same trick works for randomness.** Pass a `rand` function in, like `seatCodes(2, rand)`. Tests pass a seeded one. The app passes `Math.random`. **Simple lesson:** if a test needs to control something, make it an input. ## Shared State: Tests That Depend on Order **Shared state** = data that more than one test can change, like a list at the top of a file. Press **▶ Run tests**. - **The Tests tab:** "a new cart is empty" fails: expected 0, got 1. - **Why:** the first test added a pen to the same list. - **Alone,** the second test passes. That is why it looks flaky: it depends on what ran first. Fix it with `beforeEach`. It runs before every test. Press **▶ Run tests**. Green, in any order. **Real test runners can shuffle the order.** Vitest has `--sequence.shuffle`. Run it in CI now and then. A test that breaks when shuffled depends on shared state. **Simple lesson:** every test sets up what it needs. No test relies on another. ## Use It at Work - **Rerun a new test many times** before you trust it: ```bash for i in $(seq 50); do npx vitest run tests/seats.test.ts || break; done ``` - **Shuffle the order:** `npx vitest run --sequence.shuffle`. - **Control the clock** when you cannot pass it in: `vi.useFakeTimers()` and `vi.setSystemTime(new Date(2026, 10, 13, 9, 0))`. - **Turn off CI retries** for new tests, or report every retry. A retry is a failure. - **Tell the agent the rule:** ```text Do not call new Date() or Math.random() inside the code you write. Take the time and the random function as parameters, so tests can pass fixed values. Every test must set up its own data in beforeEach. ``` ## TL;DR 1. **Flaky test** = sometimes passes, sometimes fails, on the same code. 2. **One green run** is not evidence. Run a new test many times. 3. **Seed** = a starting number that makes random values repeatable. 4. **Retries hide bugs.** Use reruns to find a flake, then fix the cause. 5. **Time and randomness** are inputs. Tests pass fixed values. 6. **Shared state** is reset in `beforeEach`, so order does not matter. ## Common mistakes | Mistake | Wrong idea | Correct idea | What goes wrong | |---|---|---|---| | Retry until green | "It passed on the second run." | It failed once. Find out why. | The bug ships. Users hit it. | | `new Date()` in a test | "It works when I run it." | Pass a fixed time in. | Red every afternoon, or at midnight. | | Test copies the code | "Expected = the same `if` rules." | Write expected values by hand. | Wrong rule, wrong test, still green. | | Shared list or file | "Test B never runs first." | Reset in `beforeEach`. | Red when the order changes. | | Seed hides a real bug | "Seed 7 passes, so we are fine." | Try many seeds. | The bug lives on the other seeds. | ## Check Your Understanding ## Exercises ## What to Remember - One green run is a claim. Many green runs are evidence. - A flaky test is a bug report. Find the cause. - Never retry until green. - Pass time and randomness in. Tests choose fixed values. - Reset shared state before every test. ## Further Reading - [Vitest sequence options](https://vitest.dev/config/#sequence): shuffle the test order to find hidden order problems. - [Vitest fake timers](https://vitest.dev/api/vi.html#vi-usefaketimers): control time when you cannot pass it in. - [Martin Fowler: Eradicating non-determinism in tests](https://martinfowler.com/articles/nonDeterminism.html): the common causes and fixes. Next: **Reviewing an AI Pull Request**. You will take one real AI pull request through every gate, in order, and write a review with evidence.