# Design a Payment System

Course: [High-Level Design](https://www.devtutorials.ai/learn/system-design/hld) · Section: HLD Case Studies
Source: https://www.devtutorials.ai/learn/system-design/hld/case-studies/payments

> Charge cards exactly once: payment states, idempotency keys, a double-entry ledger, background charging and daily reconciliation.

A customer taps "Pay ₹450". The app shows a spinner. QuickBite asks a payment gateway to charge the customer's card, and the gateway usually responds within a few hundred milliseconds.

Most of the time, everything works correctly. The challenge is what happens when things go wrong. What if the gateway charged the card but the response never arrived? What if the customer taps Pay twice? What if a server crashes in the middle of the payment?

Payment systems are designed around those problems.

**The main idea:** give every payment a unique identity, record every step, never guess after a timeout, and make every retry safe.

## What You Will Learn

In this case study, you will:

1. Write the requirements for checkout payments.
2. Estimate payment traffic and understand gateway delays.
3. Model payments as a state machine.
4. Use idempotency keys to prevent duplicate charges.
5. Record money in a double-entry ledger.
6. Process charges in the background and reconcile payments safely.
7. Build and test the design under peak traffic.

This lesson builds on Leaders and Replicas, Strong Consistency, Idempotency, Timeouts and Retries.

## Step 1: Requirements

Before designing the system, define what it must do.

### What It Must Do (Functional)

- Charge customers for orders.
- Support full and partial refunds.
- Pay restaurants and riders.
- Track payment status.
- Show payment results to customers.
- Maintain a complete payment history.

### How Well It Must Do It (Non-Functional)

- A customer should never be charged twice for one order.
- Every payment must be traceable.
- Checkout should respond within 100 ms.
- Payment confirmation should arrive within a few seconds.
- No payment should be lost during failures.
- The system should remain available when servers fail.

### Out of Scope

QuickBite does not store card numbers. Card information goes directly from the customer's device to the payment gateway, which returns a token that QuickBite uses for future requests. This reduces security and compliance requirements.

## Step 2: Estimate

Before choosing an architecture, estimate payment traffic.

Assume QuickBite processes one million orders per day.

A festival sale such as Diwali packs orders into a few minutes. Take the peak as about 130 times the average:

| Number | Value |
|---|---|
| Average | 1,000,000 ÷ 86,400 = **11.6 checkouts a second** |
| Diwali peak | 11.6 × 130, about **1,500 a second** |

Now consider the payment gateway. Suppose each gateway request takes about 300 ms.

If the checkout request waits for the gateway, each request holds a server thread for 0.3 s. Say each server has 4 threads:

| Number | Value |
|---|---|
| Threads waiting on the gateway at any moment | 1,500 × 0.3 = **450** |
| Servers needed just to wait | 450 ÷ 4 = **113** |

What to notice:

- Payment gateways are relatively slow.
- Waiting inside the checkout request wastes server capacity.
- Server count grows quickly as payment volume increases.
- Charging cards should not happen directly inside the checkout request.

## Step 3: A Payment Is a State Machine

Many beginners think a payment has only two outcomes:

- Paid
- Not paid

Real payment systems are more complicated. When a customer clicks Pay, the payment does not instantly become successful or failed. Instead, it moves through several stages while QuickBite and the payment gateway communicate with each other.

That is why payments are modeled as a state machine.

Common states include:

- Created
- Pending
- Succeeded
- Failed
- Unknown

A payment starts in the **Created** state. When the gateway request is sent, it moves to **Pending**. If the gateway confirms the charge, the payment becomes **Succeeded**. If the gateway rejects the payment, it becomes **Failed**.

The most important state is **Unknown**.

Imagine QuickBite sends a charge request to the gateway. The gateway successfully charges the customer's card. However, before the success response reaches QuickBite, the network connection fails.

Now QuickBite does not know what happened. The customer may already have been charged, the payment may have failed, and both possibilities are still valid.

This situation is called the **Unknown** state.

Unknown means:

- The request was sent.
- No response was received.
- The charge may or may not have happened.

A timeout does not mean failure. It means the system does not know what happened.

Never treat a timeout as a failed payment. The safe action is to ask the gateway for the payment status using the same payment identity.

## Step 4: Idempotency Keys

Suppose a customer taps the Pay button twice. Or suppose a network timeout causes QuickBite to retry the payment.

Without protection, the gateway may receive multiple charge requests and charge the customer multiple times.

To prevent this, every payment receives an idempotency key.

Think of the idempotency key as the payment's unique identity. An idempotency key is generated before the first payment attempt and reused for every retry.

Example: `order-881-charge`

The payment gateway remembers this key.

If the same key arrives again:

- No new charge is created.
- The gateway returns the original result.

No matter how many retries happen, the gateway treats them as the same payment.

Step through one lost answer, one retry and one double tap:

Here is the same idea in code:

```js runnable title="The same key never charges twice"
const seen = new Map(); // key -> result, kept by the gateway
let charged = 0;

function charge(key, amount) {
  if (seen.has(key)) return seen.get(key);
  charged += amount;
  const result = `charged ₹${amount}`;
  seen.set(key, result);
  return result;
}

console.log(charge("order-881-charge", 450)); // answer lost on the way back
console.log(charge("order-881-charge", 450)); // retry
console.log(charge("order-881-charge", 450)); // customer tapped Pay again
console.log(`Total taken from the card: ₹${charged}`);
```

Expected output:

```text
charged ₹450
charged ₹450
charged ₹450
Total taken from the card: ₹450
```

What to notice:

- Retries are safe.
- Double taps are safe.
- Duplicate requests never create duplicate charges.

## Step 5: A Double-Entry Ledger

Payments must be recorded accurately.

A common beginner design is to keep only a balance. First it says `Customer Balance = ₹450`. Later it says `Customer Balance = ₹0`.

The problem is that the history is gone. Nobody can see how the money moved.

A ledger solves this problem. Instead of storing only balances, every movement of money is recorded. This creates a complete history of every transaction.

The safest approach is a double-entry ledger.

A double-entry ledger records both sides of every movement:

- Money leaves one account.
- Money enters another account.

Because both sides are recorded, mistakes become easier to detect.

The table shows every entry for order 881. The customer pays ₹450. The restaurant gets ₹380 and the rider ₹25:

Here is the same idea in code:

```js runnable title="Every payment's entries add up to zero"
const entries = [
  { order: 881, account: "customer card", amount: -450 },
  { order: 881, account: "quickbite holding", amount: 450 },
  { order: 881, account: "quickbite holding", amount: -405 },
  { order: 881, account: "restaurant 4471", amount: 380 },
  { order: 881, account: "rider 92", amount: 25 },
];

const total = entries.reduce((t, e) => t + e.amount, 0);
const holding = entries.filter((e) => e.account === "quickbite holding").reduce((t, e) => t + e.amount, 0);
console.log(`Sum of all entries: ${total}`);
console.log(`QuickBite keeps: ₹${holding}`);
```

Expected output:

```text
Sum of all entries: 0
QuickBite keeps: ₹45
```

What to notice:

- Every entry has an equal and opposite entry.
- The ledger always balances.
- The total should always equal zero.
- Errors become easy to detect.

The ledger should be append-only. Existing entries are never changed. Refunds create new entries that move money in the opposite direction.

Strong consistency is important because payment data cannot be lost. Imagine a customer is charged ₹450 but the payment record disappears because of a database failure. The money moved, but the system has no record of it.

For this reason, payment systems prioritize correctness over speed. A slower payment is acceptable. Missing money is not.

## Step 6: Charge in the Background and Reconcile

The checkout request should not wait for slow gateway operations.

Instead, it should do only the work that must happen immediately:

1. Create a payment record.
2. Generate an idempotency key.
3. Record the payment in the ledger.
4. Add a charge job to a queue.
5. Return a processing response.

The customer sees **Processing...**

The payment remains in the processing state until workers determine whether it succeeded, failed, or remains unknown.

Background workers handle the slow gateway communication.

Workers:

- Call the gateway.
- Update payment status.
- Record results in the ledger.

If the gateway responds:

- Mark the payment as succeeded or failed.

If the request times out:

- Mark the payment as unknown.
- Ask the gateway for the status.
- Retry only if the gateway confirms the payment never happened.

Step through one payment that times out:

Even carefully designed payment systems can become inconsistent.

For example:

- QuickBite says a payment succeeded.
- The gateway says it failed.

Or:

- QuickBite says a payment is unknown.
- The gateway says it succeeded.

Reconciliation is the process of comparing both systems and finding these differences.

Every night, reconciliation compares QuickBite's records with the gateway's records.

The table compares four payments, side by side:

What to notice:

- Reconciliation finds mismatches.
- Unknown payments can be resolved later.
- Gateway records and internal records remain synchronized.

Payment gateways may also send webhooks when payment status changes. Treat webhooks as idempotent events because the same webhook may be delivered more than once.

## Step 7: Build It on the Canvas and Test It

Open the **design sandbox**.

The starting design includes:

- 2 load balancers
- 4 payment servers
- A two-copy ledger
- Direct gateway calls

The simulator generates payment traffic and failures.

| Check | Passes when |
|---|---|
| Peak load | Checkout p99 at most 100 ms, under 0.1% errors and payments confirmed within 3 seconds |
| Lose any one machine | Any load balancer, payment server, ledger copy or worker fails. Errors under 1%, p99 under 250 ms and confirmations within 5 seconds |
| Cost | At most $1,200 a month |
| Requirements | Timeouts must be reconciled and retries must reuse the same idempotency key |

Run the starting design first.

Then add:

- A payment queue
- Charge workers
- Background processing
- Idempotent retries
- Reconciliation

Compare the results.

What you will probably find:

- Charging inside the checkout request wastes server capacity.
- One ledger copy creates a single point of failure.
- Too few workers delay confirmations.
- Idempotency keys prevent duplicate charges.
- Reconciliation is required for unknown payments.

A design that passes:

- 2 load balancers
- 2 payment servers
- 2 ledger copies
- Payment queue
- 2 charge workers

This design costs about **$1,040 per month**, keeps checkout latency around **22 ms**, and confirms payments in about **1.4 seconds**.

## Common Mistakes

| Mistake | Wrong | Correct | Why It Happens |
|---|---|---|---|
| Timeout equals failure | Retry immediately after a timeout | Treat the payment as unknown and verify status | Most software treats timeouts as failures |
| No idempotency key | Every retry creates a new charge | Reuse the same payment identity | Retries are added later |
| Single balance field | Update balances directly | Use a ledger | Simpler implementation |
| Charging inside checkout | Checkout waits on the gateway | Use background workers | Easier to build initially |
| No reconciliation | Ignore mismatches | Compare records regularly | Problems are rare and easy to overlook |

## Practice

## What to Remember

- Payments are designed for correctness, not speed.
- A payment is a state machine.
- A timeout does not mean failure.
- Unknown is a real payment state.
- Every payment needs a unique identity.
- Idempotency keys make retries safe.
- Double-entry ledgers keep records correct.
- Ledgers should be append-only.
- Charge cards in the background.
- Reconciliation catches problems that other protections miss.

## Further Reading

These are the sources this lesson was checked against:

- [Stripe: Idempotent requests](https://docs.stripe.com/api/idempotent_requests): how a key makes a retried payment request safe.
- [Stripe: Designing robust and predictable APIs with idempotency](https://stripe.com/blog/idempotency): timeouts, retries and why "unknown" happens.
- [Martin Fowler: Accounting patterns](https://martinfowler.com/eaaDev/AccountingNarrative.html): accounts, entries and why ledgers are append-only.

Next: **Video Streaming**, where QuickBite serves recipe videos to large numbers of users.