# Rate Limiting

Course: [High-Level Design](https://www.devtutorials.ai/learn/system-design/hld) · Section: HLD Foundations
Source: https://www.devtutorials.ai/learn/system-design/hld/foundations/rate-limiting

> Protect busy endpoints: choosing keys and limits, 429 and Retry-After, fixed windows and their edge problem, sliding windows, token buckets, shared counters and well-behaved clients.

Think about a popular ride at an amusement park.

A gatekeeper allows about 10 people to enter every minute. If a small group arrives together, they can still get in because a few seats are available. But if one person tries to go through the gate 100 times, the gatekeeper asks them to come back later. This keeps the ride running safely and fairly for everyone.

A **rate limiter** works like that gatekeeper. It counts how many requests each user sends and politely blocks any extra requests before they reach your servers.

**The main idea:** decide how many requests each user can send during a period of time, allow short bursts of activity, and stop extra requests early with a clear "slow down" message.

## What You Will Learn

In this lesson, you will learn:

1. Why rate limiting is important.
2. What can be rate limited, such as users, IP addresses, and API keys.
3. How to tell clients to slow down using 429 responses and Retry-After.
4. How the Fixed Window method works and the limitation it has at window boundaries.
5. How the Sliding Window method works.
6. How the Token Bucket method works and why it handles bursts effectively.
7. Where rate limiters are placed and how multiple servers share the same count.
8. How clients should respond when they reach a rate limit.

In Lesson 4, the load balancer distributed traffic across multiple servers. However, this does not help if a single bot sends 1,000 requests every second. A rate limiter stops that traffic before it is distributed to the servers.

## Why Limit Requests?

Without a limit, one user can use more than their fair share of system resources. That user might be:

- **a bot** trying thousands of coupon codes;
- **a buggy app** stuck in a loop and retrying every millisecond;
- **a scraper** copying every menu;
- **a real customer** repeatedly pressing "Refresh" during a slow period.

Any of these can slow the system down for other users or increase operating costs. A rate limiter helps keep things **fair** and **safe** for everyone.

## What to Limit By

A limit needs a **key**, which means a way to identify who is being counted.

| Key | Good for | Watch out for |
| --- | --- | --- |
| User id | Logged-in customers | Bots can create many accounts |
| IP address | Visitors who are not logged in | Many people may share the same office or mobile network IP |
| API key | Partner apps, such as a restaurant's till system | Keys can leak |

Different endpoints need different limits:

| QuickBite endpoint | Sensible limit | Why |
| --- | --- | --- |
| Browse menus | 100 a minute per user | Cheap and users often scroll quickly |
| Apply a coupon | 5 a minute per user | Helps stop coupon-guessing bots |
| Log in | 5 a minute per IP | Helps stop password-guessing attempts |
| Place an order | 10 a minute per user | Most people do not place 10 orders in a minute |

## Saying "Slow Down"

When a request goes over the limit, the server responds with **429 Too Many Requests**. A good response also includes a **Retry-After** header, which tells the client how many seconds to wait before trying again.

This is much better than a generic error because the client knows exactly what happened and when it can try again.

## Method 1: Fixed Window

### How It Works

Split time into fixed periods, such as one minute. Count requests in the current period. When the count reaches the limit, reject the remaining requests until the next period begins.

It is simple and efficient because it needs only one counter per user per minute.

### The Edge Problem

The window resets based on the clock. This means a user can send a full window's worth of requests at 0:59 and another full window's worth at 1:00.

#### The wrong way first: trust the fixed window at the edge

Step through a bot that sends 10 login tries at 0:59 and 10 more at 1:00:

Twice the limit was allowed in only two seconds. For a login page, that means a bot can try twice as many passwords in a very short time.

## Method 2: Sliding Window

### How It Works

A **sliding window** always looks at the **last 60 seconds** from the current moment instead of the current clock minute. One approach is to store request times and count only those that happened during the last minute.

#### The right way: count the last 60 seconds

Step through the same 20 tries with a sliding window:

The code behind the animation:

```js runnable title="The right way: a sliding window allows only 10 in any 60 seconds"
const limit = 10;
const windowSec = 60;
const times = [...Array(10).fill(59), ...Array(10).fill(60)];

const log = []; // times of requests we allowed
let allowed = 0;
for (const t of times) {
  while (log.length && log[0] <= t - windowSec) log.shift(); // forget old requests
  if (log.length < limit) {
    log.push(t);
    allowed++;
  }
}
console.log(`Allowed ${allowed} requests, refused ${times.length - allowed}`);
```

Expected output:

```text
Allowed 10 requests, refused 10
```

Keeping every timestamp uses more memory for busy users. In real systems, a **sliding window counter** is often used to estimate the same result using fewer resources. The main idea remains the same: look at the last 60 seconds instead of the current clock minute.

## Method 3: Token Bucket

### How It Works

The **token bucket** is one of the most popular rate-limiting methods and is used by many large APIs.

1. Each user has a bucket that can hold a fixed number of **tokens**, such as 20.
2. Tokens are added to the bucket at a steady rate, such as 10 per second, until the bucket becomes full.
3. Each request uses one token.
4. If no token is available, the request is rejected with a 429 response.

It only needs two values per user: the number of tokens left and the time the bucket was last refilled.

### Why It Handles Bursts Well

The bucket **size** controls the largest burst that is allowed. The **refill rate** controls the steady request speed.

This means a user who has been inactive for some time can send a short burst of requests, but cannot continue sending requests at a high rate forever. Step through a bucket of 5 that refills 1 token a second:

The code behind the animation:

```js runnable title="A bucket of 5, refilling 1 a second: a burst of 5, then 1 a second"
const capacity = 5;
const refillPerSec = 1;
let tokens = capacity;
let last = 0;

function allow(t) {
  tokens = Math.min(capacity, tokens + (t - last) * refillPerSec); // top up for time passed
  last = t;
  if (tokens >= 1) {
    tokens -= 1;
    return "ok";
  }
  return "429";
}

const times = [0, 0, 0, 0, 0, 0, 0, 1, 2, 2];
console.log(times.map((t) => `${t}s:${allow(t)}`).join("  "));
```

Expected output:

```text
0s:ok  0s:ok  0s:ok  0s:ok  0s:ok  0s:429  0s:429  1s:ok  2s:ok  2s:429
```

Five requests were allowed immediately. The sixth and seventh requests were rejected. After that, one token becomes available every second, so approximately one request per second can continue.

### Telling the Client When to Retry

The bucket can also calculate how long a client should wait before enough tokens become available again:

| Tokens needed | Tokens now | Refill rate | Retry-After |
| --- | --- | --- | --- |
| 1 | 0 | 0.5 a second | 2 s |
| 5 | 2 | 1 a second | 3 s |

The wait is (tokens needed − tokens now) ÷ refill rate, rounded up to a whole second.

### Try It: Protect the Coupon Endpoint

Choose a traffic pattern and a limiter. The lab runs 30 seconds of traffic through the limiter and then sends it to the coupon service, which can handle around 50 requests per second. Press **Play** to watch what happens second by second.

Try this order:

1. **A bot attack, no limiter:** requests pile up in front of the coupon service, causing delays and timeouts.
2. **A bot attack, token bucket:** the bot is restricted to the allowed rate, keeping the service responsive.
3. **A short burst, fixed window, then sliding window:** compare how each method handles sudden bursts of traffic.

## Where the Rate Limiter Lives

### At the Front Door

Place the limiter as early as possible so that unnecessary requests never reach your application servers or database. This is usually the **API gateway**, which sits in front of the rest of the system.

### Sharing One Count

QuickBite runs multiple gateways. If each gateway keeps its own count, a user could receive the full limit from every gateway.

To avoid this, counts are stored in a **shared store**, usually a fast in-memory database such as Redis.

This creates another challenge: two gateways may update the same count at the same time. Step through what goes wrong, and the fix:

The solution is an **atomic** update. The store increases the count and returns the new value in a single operation. Redis's `INCR` command provides this behavior.

## How Clients Should React

Rate limiting works best when clients cooperate.

A well-behaved client:

- reads the **Retry-After** header and waits for the specified time;
- waits slightly longer after repeated refusals using **backoff**;
- shows users a clear message such as "Too many tries. Please wait a moment."

A poorly written client that retries immediately after every 429 response only wastes its own available requests.

Use more than one limit for sensitive endpoints. A bot spreading login attempts across thousands of accounts might stay below every individual user limit. Adding per-IP and overall limits helps prevent this.

**Easy way to remember:** the bucket size controls the largest burst, and the refill rate controls the steady request speed. No token means no request.

## Common Mistakes

| Mistake | Wrong | Correct | Why it happens |
|---|---|---|---|
| One limit for everything | 100 a minute on menus, logins and coupons alike. | Strict limits on logins and coupons, looser ones on browsing. | Cheap and risky endpoints need very different protection. |
| Fixed window for sensitive pages | A per-minute counter that resets on the clock. | A token bucket or a sliding window. | Bursts at the window edge let twice the limit through. |
| Unhelpful errors | Returning 500, or an empty page. | Return 429 with a Retry-After header. | Clients cannot tell "slow down" from "broken", so they retry blindly. |
| Counting on each gateway | Every gateway keeps its own counter. | Keep counts in a shared store, updated atomically. | Separate or racing counters give users more than the limit. |
| A single key | Limiting by user id only. | Combine user, IP and overall limits on logins and coupons. | Bots spread their requests across many accounts or many IPs. |

## Practice

## What to Remember

- A **rate limiter** controls how many requests a user can send during a period of time.
- Choose a **key** such as a user, IP address, or API key, and apply limits per endpoint.
- Reject extra requests using **429** and **Retry-After**.
- **Fixed window:** simple, but allows bursts at window boundaries.
- **Sliding window:** checks the last N seconds and avoids edge bursts.
- **Token bucket:** bucket size controls bursts; refill rate controls steady speed.
- Place limits at the **gateway**.
- Use a **shared store** with **atomic updates** when multiple servers are involved.
- Good clients wait and retry responsibly.

## Further Reading

These are the sources this lesson was checked against:

- [Stripe Docs: Rate Limits](https://docs.stripe.com/rate-limits): how a large payments API limits requests and how clients should handle 429 responses.
- [MDN: 429 Too Many Requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429): the status code and the Retry-After header.
- [Redis Documentation: INCR](https://redis.io/docs/latest/commands/incr/): the atomic counter commonly used for rate limiting, with a rate limiter pattern.
- [System Design Primer](https://github.com/donnemartin/system-design-primer): where rate limiting fits among other building blocks.

Next: **Reliability**, where you will learn how to keep QuickBite running even when a dependent service becomes slow or unavailable.