A new test is green once. What should you do before trusting it?

Run it many times, for example on many seeds

A test fails on 4 runs out of 300. What is the best next step?

Find the cause; it may be a real bug in the code

How do you test code that says 'Good morning' before 12:00?

Pass the time in, like greet(new Date(2026, 10, 13, 11, 59))

Copy the if rules into the test

Test B passes alone but fails after test A. What is the likely cause?

Shared state that test A changed

Why is a test that copies the code's own rules weak?

If the rule is wrong, the test is wrong in the same way and stays green

Flaky Tests and AI

By Syncfusion®, Inc. · Last updated November 13, 2026

A test that is green on one run and red on another is a bug report, not bad luck. Run one AI test 300 times and find the runs that fail, fix the code instead of retrying, pass the clock in as an input, and stop tests from sharing state.

Green Once Is Not Green

An agent runs the tests once. They are green. It says "done".

But some tests are green on one run and red on the next, with no code change.

Flaky test = a test that sometimes passes and sometimes fails on the same code.

Agents make flaky tests often:

they use the real clock, like new Date()
they use Math.random() without a seed
they let tests share one list, one file or one database

When a flaky test goes red, many people (and agents) just press "re-run" until it is green.

Simple lesson: a flaky test is a bug report. Find the cause. Do not retry until green.

A Real-World Example

Think about a school bell that rings late one day in twenty.

On most days, nobody notices.
On the day of the exam, half the class is late.
"Just ring it again" does not fix the wire. Someone has to open the box.

A flaky test is that bell. Re-running it hides the loose wire.

What You Will Learn

In this lesson, you will learn:

Why one green run is not evidence.
How to rerun a test on many seeded runs and list the runs that fail.
Why the fix is usually in the code, not the test.
How to pass the clock in, so tests choose the time.
How to stop tests from sharing state.

One Run vs Many Runs

Green on One Run

Seed = a starting number for random values. The same seed always gives the same "random" numbers. In this lab, seed 7 means "CI run number 7".

Press ▶ Run tests.

The Tests tab: green.
AI_SUMMARY.md: "Tests pass ✅".
The question: was that luck?

The Same Test on 300 Runs

Now the same check runs once for each seed from 1 to 300.

Press ▶ Run tests.

The Tests tab: red. The failing runs are 66, 77, 221, 232.
4 runs in 300 gave two seats the same pickup code.
This is a real bug. Two people in one booking get the same code.

Animation: a row of CI runs lights up one by one. Most are green. A few are red. One run alone looks fine, but many runs show the red ones. 30 CI runs of the same test, same code

Green: the run the agent saw Red: the runs your users will hit

One run shows one box. Many runs show the red ones.

Fix the Code, Not the Test

The wrong fixes:

re-run until green
add retry: 3 to the test
delete the test

The right fix: make the codes unique inside one booking.

Press ▶ Run tests. Green on all 300 runs. Read src/seats.ts: it skips a code it already used.

Retries hide bugs. A test that passes "on the second try" still failed once. That failure happened to a real user too. Use reruns to find a flake, never to make it green.

Simple lesson: run a new test many times. If it ever fails, the code or the test has a bug.

Time: Pass the Clock In

A Test That Depends on the Real Clock

The agent wrote greet() with new Date() inside.

Press ▶ Run tests. Green, at any hour.

The first test (in the comment) was green only before noon. In the afternoon CI, it went red.
The "fix": the test now copies the code's own if rules.
The problem: it checks the code against itself. If the rule is wrong, the test is wrong the same way.

Pass the Time In

Change the code so the time is an input: greet(now: Date). Now each test picks its own time.

Press ▶ Run tests, then 🐞 Plant bugs.

The Tests tab: green on every run, at every hour.
The Bug check tab: 1 of 5 caught. The tests use 9:00 and 16:59, so the edges at 12:00 and 17:00, very early hours and the evening are never tested.
Now try: add tests for 3:00, 11:59, 12:00 and 17:00. Plant bugs again until 5 of 5 are caught.

The same trick works for randomness. Pass a rand function in, like seatCodes(2, rand). Tests pass a seeded one. The app passes Math.random.

Simple lesson: if a test needs to control something, make it an input.

Shared State: Tests That Depend on Order

Shared state = data that more than one test can change, like a list at the top of a file.

Press ▶ Run tests.

The Tests tab: "a new cart is empty" fails: expected 0, got 1.
Why: the first test added a pen to the same list.
Alone, the second test passes. That is why it looks flaky: it depends on what ran first.

Fix it with beforeEach. It runs before every test.

Press ▶ Run tests. Green, in any order.

Real test runners can shuffle the order. Vitest has --sequence.shuffle. Run it in CI now and then. A test that breaks when shuffled depends on shared state.

Simple lesson: every test sets up what it needs. No test relies on another.

Use It at Work

Rerun a new test many times before you trust it:

for i in $(seq 50); do npx vitest run tests/seats.test.ts || break; done

Shuffle the order: npx vitest run --sequence.shuffle.
Control the clock when you cannot pass it in: vi.useFakeTimers() and vi.setSystemTime(new Date(2026, 10, 13, 9, 0)).
Turn off CI retries for new tests, or report every retry. A retry is a failure.
Tell the agent the rule:

Do not call new Date() or Math.random() inside the code you write.
Take the time and the random function as parameters, so tests can pass fixed values.
Every test must set up its own data in beforeEach.

TL;DR

Flaky test = sometimes passes, sometimes fails, on the same code.
One green run is not evidence. Run a new test many times.
Seed = a starting number that makes random values repeatable.
Retries hide bugs. Use reruns to find a flake, then fix the cause.
Time and randomness are inputs. Tests pass fixed values.
Shared state is reset in beforeEach, so order does not matter.

Common mistakes

Mistake	Wrong idea	Correct idea	What goes wrong
Retry until green	"It passed on the second run."	It failed once. Find out why.	The bug ships. Users hit it.
`new Date()` in a test	"It works when I run it."	Pass a fixed time in.	Red every afternoon, or at midnight.
Test copies the code	"Expected = the same `if` rules."	Write expected values by hand.	Wrong rule, wrong test, still green.
Shared list or file	"Test B never runs first."	Reset in `beforeEach`.	Red when the order changes.
Seed hides a real bug	"Seed 7 passes, so we are fine."	Try many seeds.	The bug lives on the other seeds.

Check Your Understanding

Exercises

What to Remember

One green run is a claim. Many green runs are evidence.
A flaky test is a bug report. Find the cause.
Never retry until green.
Pass time and randomness in. Tests choose fixed values.
Reset shared state before every test.

Flaky Tests and AI

Green Once Is Not Green

A Real-World Example

What You Will Learn

One Run vs Many Runs

Green on One Run

The Same Test on 300 Runs

Fix the Code, Not the Test

Time: Pass the Clock In

A Test That Depends on the Real Clock

Pass the Time In

Shared State: Tests That Depend on Order

Use It at Work

TL;DR

Common mistakes

Check Your Understanding

Exercises

What to Remember

Further Reading