Design a Notification System

By the Syncfusion DevTutorials team · Last updated

Send order updates by push, SMS and email: queue the work, a queue per channel, provider rate limits, retries with backoff and dedupe keys.

"Your order is confirmed." "Your rider has picked up your food." "Your rider is 2 minutes away." Every QuickBite order sends messages like these through push notifications, SMS or email.

QuickBite does not send these messages directly. Push notifications are delivered by Apple and Google, SMS messages are delivered by an SMS provider and emails are delivered by an email service.

These providers are outside QuickBite's control.

They can be slow, they can fail and they may have rate limits. For example, the SMS provider accepts only 500 texts per second.

The main idea: respond to the order service immediately, place notification work into queues and make every notification safe to retry.

What You Will Learn

In this case study, you will:

  1. Write the requirements for a notification system.
  2. Estimate notification traffic and identify provider limits.
  3. Use queues so the order service never waits on providers.
  4. Separate channels so slow notifications do not block fast ones.
  5. Retry failed sends safely using backoff and dedupe keys.
  6. Build and test the design during peak traffic.

This lesson builds on Queues, At-Least-Once Delivery, Idempotency, Timeouts, Retries and Backoff.

Step 1: Requirements

Before designing the system, start by defining what it must do.

What It Must Do (Functional)

  • Send notifications by push, SMS or email.
  • Respect customer notification preferences.
  • Populate notification templates with order information.
  • Track notification delivery history.
  • Support multiple languages.
  • Process order updates automatically.

How Well It Must Do It (Non-Functional)

  • The order service should receive a response within 50 ms.
  • Push notifications should arrive within 500 ms at p99.
  • Customers should never receive duplicate messages.
  • Failed notifications should be retried automatically.
  • The system should continue working if a server fails.
  • The design should remain cost effective during peak traffic.

Out of Scope

This lesson does not include:

  • Marketing campaigns
  • Scheduled notifications
  • Promotional messaging
  • Recommendation notifications

The focus is transactional order notifications.

Step 2: Estimate

Before designing the architecture, estimate notification demand.

Assume QuickBite processes 1,000 orders per second during the dinner rush.

Each order sends about 2 push messages, 0.6 texts and 0.9 emails. The chart compares what each channel needs with what its provider accepts:

Push needs 2,000 a second (limit 5,000) and email 900 (limit 2,000). SMS needs 600 a second, but the provider accepts only 500.

What to notice:

  • Push notifications stay within the provider limit.
  • Email stays within the provider limit.
  • SMS demand exceeds the provider's limit.
  • QuickBite cannot make the SMS provider faster.
  • SMS notifications must wait in a queue during busy periods.

Now estimate the backlog.

SMS gets 100 texts a second behind (600 − 500). Over a 10-minute rush:

NumberValue
Backlog at the end of the rush100 × 600 s = 60,000 texts
Wait for the last text (500 a second, after the rush)60,000 ÷ 500 = 120 s, about 2 minutes

What to notice:

  • SMS messages will build up during peak traffic.
  • Some SMS notifications will be delayed.
  • A delay is acceptable for SMS.
  • Push notifications should not be delayed because of SMS traffic.

Step 3: Never Make the Order Service Wait

A common mistake is sending notifications directly from the order service.

In that design:

  1. An order update occurs.
  2. The order service calls the provider.
  3. The provider responds slowly.
  4. The order service waits.

This creates unnecessary latency.

Instead, the notification service should perform only a few quick tasks:

  1. Validate the request.
  2. Check notification preferences.
  3. Create a notification job.
  4. Place the job into a queue.
  5. Return success immediately.

The slow work happens later.

Background workers read jobs from the queue and communicate with providers.

This keeps ordering fast even when providers are slow or unavailable.

Step 4: One Queue or One per Channel?

A single queue looks simple but creates a serious problem.

Imagine push notifications and SMS messages share the same queue.

During peak traffic:

  • SMS messages pile up.
  • SMS workers wait on provider limits.
  • Push notifications get stuck behind SMS jobs.

This is called head-of-line blocking.

The solution is to use one queue for each channel.

For example:

  • Push queue
  • SMS queue
  • Email queue

Each queue has its own workers.

Push notifications continue moving quickly even when SMS messages are delayed.

How many workers are busy at once? Push jobs take 10 ms each. In a shared queue, an SMS job holds its worker for 100 ms while it waits on the rate limit. In its own queue, the SMS job waits in the queue instead, and holds a worker for only 10 ms.

DesignWorkers busy at once
One shared queue(2,000 × 10 ms + 600 × 100 ms) ÷ 1,000 = 80
A queue per channel(2,000 × 10 ms + 600 × 10 ms) ÷ 1,000 = 26

What to notice:

  • Shared queues waste worker capacity.
  • Slow channels affect fast channels.
  • Separate queues improve performance and reliability.

Separate queues also support priorities. Critical notifications such as order updates can use high-priority queues, while promotional campaigns can use lower-priority queues.

Step 5: Retries, Backoff and Duplicates

Providers sometimes fail.

Examples include:

  • Timeouts
  • Temporary outages
  • Rate limit responses
  • Network problems

Workers should retry failed sends automatically.

Instead of retrying immediately, use exponential backoff with jitter.

Step through four retries of one failed text:

Here is the same idea in code:

Backoff with jitter spreads retries out

let seed = 123456789;
const random = () => ((seed = (seed * 16807) % 2147483647) / 2147483647);

for (let attempt = 1; attempt <= 4; attempt++) {
  const cap = 1000 * 2 ** (attempt - 1); // 1 s, 2 s, 4 s, 8 s
  const wait = Math.round(random() * cap); // full jitter
  console.log(`retry ${attempt}: wait ${wait} ms (up to ${cap})`);
}

Expected output:

retry 1: wait 218 ms (up to 1000)
retry 2: wait 1913 ms (up to 2000)
retry 3: wait 3318 ms (up to 4000)
retry 4: wait 4494 ms (up to 8000)

Exponential backoff prevents large numbers of retries from hitting the provider at the same time.

Retries create another problem.

Sometimes the notification succeeds but the success response is lost.

The worker thinks the request failed and retries.

Without protection, the customer receives duplicates.

The solution is a dedupe key.

A dedupe key uniquely identifies a notification.

Example: order-881:picked-up:sms

The same key is reused on every retry.

Step through three tries of the same text:

Here is the same idea in code:

A dedupe key turns three tries into one text

const sent = new Set();
let textsDelivered = 0;

function send(key) {
  if (sent.has(key)) return "skipped (already sent)";
  sent.add(key);
  textsDelivered++;
  return "sent";
}

const key = "order-881:picked-up:sms";
console.log(`try 1: ${send(key)}`);
console.log(`try 2: ${send(key)}`);
console.log(`try 3: ${send(key)}`);
console.log(`Texts the rider got: ${textsDelivered}`);

Expected output:

try 1: sent
try 2: skipped (already sent)
try 3: skipped (already sent)
Texts the rider got: 1

What to notice:

  • Retries are safe.
  • Duplicate messages are prevented.
  • The same notification can be processed multiple times without sending multiple messages.

Because queues usually provide at-least-once delivery, duplicate jobs can occur naturally. Dedupe keys are required for customer-facing notifications.

Step 6: Build It on the Canvas and Test It

Open the design sandbox.

The starting design includes:

  • 2 notification API servers
  • Direct calls to providers
  • No queues
  • No workers

The simulator sends:

  • 2,000 push notifications per second
  • 600 SMS notifications per second
  • 900 email notifications per second
CheckPasses when
Peak loadAPI p99 at most 50 ms, under 0.1% errors and push delivered within 500 ms
Lose any one machineAny API server or worker dies. Errors under 1%, API p99 under 150 ms and push within 1 second
CostAt most $700 a month
RequirementsQueues between APIs and providers, with dedupe-safe retries

Send every QuickBite order update on time

Run the starting design first.

Then add:

  • Notification queues
  • Channel-specific queues
  • Background workers
  • Retry logic
  • Dedupe keys

Compare the results.

What you will probably find:

  • Calling providers directly makes the order path slow.
  • One shared queue causes push notifications to wait behind SMS traffic.
  • SMS rate limits create backlogs during rush periods.
  • Retry logic is required for reliable delivery.
  • Dedupe keys prevent duplicate notifications.

A design that passes:

  • 2 notification API servers
  • Push queue
  • SMS queue
  • Email queue
  • 6 notification workers
  • Dedupe-safe retries

This design costs about $580 per month, keeps API latency around 9 ms and delivers push notifications in about 108 ms.

Common Mistakes

MistakeWrongCorrectWhy It Happens
Sending directly to providersThe order request waits on the providerQueue the job and return immediatelyDirect integration looks simpler
One shared queuePush waits behind SMS trafficUse separate queuesWorks fine until traffic increases
Blind retriesRetry without knowing whether the send succeededUse dedupe keysTimeouts often look like failures
Retry stormsThousands of retries happen togetherUse backoff and jitterFailures often happen at the same time
Ignoring provider limitsAssume providers accept unlimited trafficDesign around rate limitsLimits appear during peak traffic

Practice

What to Remember

  • Queue first, send later.
  • Providers are outside your control.
  • Separate queues prevent head-of-line blocking.
  • Use retries for reliability.
  • Use exponential backoff and jitter.
  • Every notification should have a dedupe key.
  • Design around provider limits.
  • Keep customer-facing operations fast.

Further Reading

These are the sources this lesson was checked against:

Next: File Storage, where QuickBite stores menu photos and receipts.