Reliability: Timeouts, Retries and Circuit Breakers

By the Syncfusion DevTutorials team · Last updated October 20, 2026

Keep working when dependencies fail: cascading failures, timeouts, retries with backoff and jitter, circuit breakers, fallbacks, bulkheads, backpressure and load shedding.

Imagine you call a restaurant to book a table, but nobody answers.

You would not stay on the phone forever. After waiting for a while, you hang up. You might try calling again a few minutes later, but not every second. If you learn the restaurant is closed, you stop calling and choose another place. And if the kitchen is already overloaded, it is better for the restaurant to say "we can't take more orders right now" than to accept more orders and make every customer wait two hours.

Reliable software systems work in exactly the same way.

When systems communicate with other services, those services can become slow, unavailable, overloaded, or fail completely. To handle these situations, reliable systems use timeouts, retries with backoff, circuit breakers, bulkheads, and backpressure.

The main idea: failures are normal. Stop waiting on slow services, retry carefully, avoid calling services that are clearly unhealthy, and prevent one failure from spreading through the entire system.

What You Will Learn

In this lesson, you will learn:

1. Why failures are normal in distributed systems.
2. What cascading failures are and how they spread.
3. How to choose timeouts.
4. When retries help and when they make problems worse.
5. How backoff and jitter reduce retry storms.
6. How circuit breakers protect systems.
7. How fallbacks and graceful degradation improve reliability.
8. How bulkheads, backpressure, and load shedding prevent overload.

Failures Are Normal

Many beginners imagine failures as rare events.

In reality, modern applications depend on many services:

• Payment providers
• Databases
• SMS providers
• Email services
• Mapping services
• Internal APIs

Each service may be highly reliable on its own. However, when dozens of services work together, there is almost always something having a problem somewhere in the system.

If each service is up 99.9% of the time, the chance that all of them are up at once drops fast as you add more:

What to Notice

• Reliability decreases as more dependencies are added.
• Even highly reliable services fail occasionally.
• Distributed systems must expect failures.
• The goal is not to eliminate failures, but to limit their impact.

Cascading Failures

How One Slow Service Breaks Everything

A slow service is often more dangerous than a completely failed service.

A failed service returns an error immediately.

A slow service causes callers to wait.

While callers wait, they hold resources such as worker threads, memory, and network connections.

QuickBite's checkout service has 100 workers.

Each worker can process one request at a time.

These workers are shared across:

• Card payments
• Wallet payments
• Cash orders

Card checkouts arrive at 50 a second. Busy workers = calls arriving each second × how long each call waits. Step through what happens as the provider slows down:

What to Notice

• Slow calls consume workers for longer periods.
• More waiting means more busy workers.
• Eventually no workers remain for other requests.
• Card payment problems begin affecting unrelated features.

When all workers become occupied waiting for the payment provider, even cash orders stop working.

This chain reaction is called a cascading failure.

A problem in one dependency spreads into healthy parts of the system.

Timeouts

Give Up On Slow Calls

A timeout defines the maximum amount of time a service will wait for a response.

If the timeout is reached, the request is treated as failed.

Suppose the provider's p99 response time is 400 ms. A timeout of about 1 second leaves room for slower requests while still protecting the system:

• With a 1-second timeout: 50 calls × 1 second = 50 busy workers.
• Without it, during an 8-second slowdown: 50 calls × 8 seconds = 400 busy workers.

What to Notice

• Timeouts protect resources.
• Faster failure is often better than waiting forever.
• Every network call should have a timeout.

Retries

When Retries Help

Many failures are temporary.

Examples include:

• Network interruptions
• Server restarts
• Brief overload
• Temporary connection failures

Trying again after a short delay often succeeds.

However, retries should only happen when:

• The operation is idempotent.
• The error is temporary.
• Retry count is limited.

Trying again will not change the outcome.

Retry Storms

Retries are dangerous because every retry creates additional load.

Imagine every request retries three times immediately.

What to Notice

• Retries multiply traffic.
• Immediate retries increase overload.
• Too many retries can keep a service down longer.
• Retry storms can become worse than the original failure.

Backoff and Jitter

Backoff

Backoff increases the wait time between retries.

Example:

This is called exponential backoff.

Jitter

If thousands of servers retry at exactly the same time, they create traffic spikes.

Jitter adds randomness so retries happen at slightly different times.

With a cap of 2 seconds, the waits double each time: 100, 200, 400, 800, 1,600 ms. Jitter then multiplies each wait by a random number between 0 and 1, so one server might wait 60, 180, 280, 800, 1,280 ms while its neighbour waits different amounts.

The code behind the animation:

The right way: each retry waits about twice as long, never more than 2 seconds

const baseMs = 100;
const capMs = 2000;
const jitter = [0.6, 0.9, 0.7, 1.0, 0.8]; // in real code: Math.random(), here fixed so the output repeats

const waits = [];
for (let attempt = 0; attempt < 5; attempt++) {
  const backoff = Math.min(capMs, baseMs * 2 ** attempt); // 100, 200, 400, 800, 1600
  waits.push(Math.round(backoff * jitter[attempt]));
}
console.log(`Wait before each retry (ms): ${waits.join(", ")}`);

Expected output:

Wait before each retry (ms): 60, 180, 280, 800, 1280

What to Notice

• Backoff reduces pressure on struggling services.
• Jitter spreads retry traffic.
• Together they help services recover faster.

Circuit Breakers

Stop Calling A Service That Is Down

If a service has failed repeatedly, continuing to call it only wastes resources.

A circuit breaker detects this situation and temporarily stops sending requests.

The circuit breaker has three states.

Closed

Everything is healthy.

Requests are allowed.

Open

Too many failures have occurred.

Requests are rejected immediately.

Half-Open

After a waiting period:

• Allow a small number of test requests.
• If they succeed, close the breaker.
• If they fail, open it again.

Step through a breaker that opens after 3 failures in a row:

The code behind the animation:

After 3 failures in a row, the breaker opens and stops calling

function makeBreaker(threshold) {
  let failures = 0;
  let state = "closed";
  return {
    record(result) {
      if (state === "open") return "open (call skipped)";
      failures = result === "fail" ? failures + 1 : 0;
      if (failures >= threshold) state = "open";
      return state;
    },
  };
}

const breaker = makeBreaker(3);
const results = ["ok", "fail", "fail", "fail", "fail", "ok"];
console.log(results.map((r) => `${r}->${breaker.record(r)}`).join("  "));

Expected output:

ok->closed  fail->closed  fail->closed  fail->open  fail->open (call skipped)  ok->open (call skipped)

What to Notice

• Open breakers fail fast.
• Workers remain available.
• The dependency gets time to recover.

Fallbacks And Graceful Degradation

A failure does not always have to produce an error page.

Sometimes a reduced experience is better than a broken experience.

This is called graceful degradation.

What to Notice

• Some functionality is better than none.
• Fallbacks improve user experience during incidents.
• Critical operations remain available.

Bulkheads

Ships use bulkheads to stop flooding from spreading between sections.

Software uses the same idea.

Different dependencies receive separate resource pools.

What to Notice

• Shared pools allow failures to spread.
• Separate pools isolate failures.
• One failing dependency cannot consume all resources.

Backpressure And Load Shedding

Sometimes nothing is broken.

There is simply more work than the system can handle.

Backpressure

Backpressure tells callers to slow down.

Examples:

• Rejecting requests when queues are full.
• Returning "try again later".
• Pausing new work temporarily.

Load Shedding

Load shedding deliberately drops less important work.

For QuickBite:

What to Notice

• Not all requests are equally important.
• Protect business-critical features first.
• Serving most customers quickly is better than serving everyone slowly.

Try It: Kill a Server

Three app servers share the lunch traffic. Kill one at second 20 and press Play to watch the next 40 seconds unfold, second by second.

Try this order:

1. Kill app-2, nothing else on: almost every order fails. A dead server answers every request instantly with an error, so to a "least connections" load balancer it looks like the least busy server, and it gets nearly all the traffic.
2. Turn on health checks: after 2 seconds the dead server is skipped. But the other two now get 360 orders a second for 320 of capacity. Their queues grow, waits climb past the 1-second timeout, and the servers keep working on requests whose customers have already given up. Soon almost nothing succeeds: the failure has spread to the healthy servers. This is a cascading failure.
3. Turn on backpressure: each server queues at most 20 requests and turns the rest away at once. A small share of orders gets "try again", and everyone else is served quickly.
4. Let app-2 restart at second 40: the queues drain and errors drop back to zero.

Try It: The Payment Provider Breaks

Pick an incident, then turn on the protections one at a time. Watch the red bars: those are seconds when checkout ran out of workers and every order failed.

Try this order:

1. Slow provider, nothing on: checkout runs out of workers for 27 seconds.
2. Add the timeout: card payments still fail during the incident, but cash and wallet orders keep working.
3. Add 3 retries: worse again. Retries against a dead provider use up the workers.
4. Add the circuit breaker: the breaker stops calling after two bad seconds. Workers run out for only 5 seconds in total, instead of 20.
5. Switch to "fails half the calls": now retries really help, and backoff lowers the peak load.

Common Mistakes

Practice

What to Remember

• Failures are normal.
• A slow dependency can trigger cascading failures.
• Every external call needs a timeout.
• Retries should be limited and use backoff and jitter.
• Circuit breakers stop wasted work.
• Fallbacks keep systems usable.
• Bulkheads isolate failures.
• Backpressure manages overload.
• Load shedding protects critical features.
• Reliability is not about preventing every failure. It is about limiting the impact when failures happen.

Further Reading

These are the sources this lesson was checked against:

• AWS Builders' Library: Timeouts, Retries and Backoff with Jitter: how Amazon sets timeouts and retries without causing retry storms.
• Martin Fowler: Circuit Breaker: the closed, open and half-open states explained.
• Google SRE Book: Addressing Cascading Failures: how overload spreads, and load shedding to stop it.
• Microsoft Azure Architecture Center: Bulkhead Pattern: separate resource pools for each dependency.

Next: Observability, where you will learn how to detect, investigate, and understand system problems using metrics, logs, and traces.