Queues, Streams and Idempotency

By the Syncfusion DevTutorials team · Last updated October 20, 2026

Hand slow work to queues: producers, consumers and acks, absorbing bursts, queues vs pub/sub, streams and offsets, at-least-once delivery, idempotency keys and dead-letter queues.

Picture a busy restaurant kitchen at lunch.

Waiters do not stand next to the cook until each dish is ready. They clip the order ticket onto a rail and go back to their tables. Cooks take the next ticket whenever they are free. When the rush comes, the rail gets longer, but no customer is turned away, and no ticket is lost.

That rail is a queue. It works like a waiting line for tasks.

One part of the system puts work into the queue and moves on. Another part takes the work from the queue and processes it later.

The main idea: put slow or risky work into a queue so customers do not have to wait for it, and traffic spikes do not overwhelm the system. Then make each step safe to repeat, because queues can sometimes deliver the same message more than once.

What You Will Learn

In this lesson, you will learn:

1. Why making customers wait for every step is a problem.
2. What producers, queues and consumers are.
3. How a queue absorbs a burst of traffic.
4. The difference between a queue and pub/sub.
5. What a stream is, and why it keeps old messages.
6. Why messages can arrive more than once.
7. What idempotency means, and how idempotency keys stop double charges.
8. What a dead-letter queue is.

The Problem with Doing Everything at Once

When a customer taps "Place order", QuickBite must:

• charge the card;
• tell the restaurant;
• email a receipt.

If checkout does all of this before answering, the customer waits for the slowest step.

Even worse, if the email service is down, the whole order may fail, even though sending an email has nothing to do with preparing food.

Step through checkout both ways:

Waiting for every step takes 2,000 ms. Saving the order, queueing the work and answering takes 50 ms. With a queue, the customer sees "Order placed" almost immediately. The other work happens a moment later in the background.

How a Queue Works

The Three Parts

A queue system has three main parts:

• A producer creates messages and puts them into the queue. In this example, checkout creates the message "order 42 placed".
• The queue stores messages safely until somebody is ready to process them.
• A consumer (or worker) takes messages from the queue and performs the work.

After the work is finished, the consumer sends a "done" signal called an acknowledgement, or ack.

Only after receiving the ack does the queue remove the message.

If a worker crashes before sending the ack, the message stays in the queue and can be delivered again. This is how queues avoid losing work.

A Queue Absorbs Bursts

Workers handle 20 orders a second.

For five seconds at lunch, 45 orders a second arrive.

Without a queue, the extra orders might fail because workers cannot keep up.

With a queue, the extra orders wait safely until workers become available.

The chart shows how many orders wait in the queue at the end of each second:

The queue grows during the burst and shrinks afterwards.

Nothing is lost.

But if the queue keeps growing for a long time, you need more workers.

Try It: A Queue for Orders

Change the worker speed, then make workers crash now and then. Watch the "Double charges" number.

Queues and Pub/Sub

There are two common ways to hand out messages.

Queue

A queue gives each message to one worker.

Use it when work should happen only once.

Examples:

• charge a card;
• send an email;
• generate an invoice.

Each task is handled by one worker.

Pub/Sub

Pub/Sub means Publish and Subscribe.

A pub/sub system gives each message to every subscriber.

The message is usually an event, a short note that something happened.

For example, an OrderPlaced event.

The following systems may all need this event:

• restaurant app;
• rider matching service;
• analytics service.

Every subscriber receives its own copy.

With pub/sub, the producer does not need to know who is listening. A new team can subscribe tomorrow without changing checkout at all.

Streams: A Queue That Remembers

A normal queue removes a message after it is acknowledged.

A stream is different.

A stream keeps messages for days, weeks, or even months.

Kafka is a well-known example.

Because messages stay around, systems can read them again later.

Each reader keeps its own bookmark, called an offset.

An offset is simply the position of the next message the reader will read.

Different readers can be at different places in the stream.

Step through three readers on the same stream:

Rider matching reads only order 5. Analytics reads orders 2 and 3. The new report reads all five orders from the beginning.

Streams are useful when:

• many teams need the same events;
• you need to rebuild reports;
• you need to replay old events after fixing a bug;
• you want to recover from failures.

Order Within a Key

Messages for the same order must stay in order.

For example: OrderPlaced, then OrderPaid, then OrderDelivered. A reader must never see "delivered" before "placed".

Streams keep order within a partition.

That is why all messages for the same order id are usually sent to the same partition.

Messages Can Arrive Twice

Why It Happens

Most queues use at-least-once delivery.

This means:

• messages are not lost;
• messages can sometimes arrive more than once.

Imagine this happens:

1. A worker receives an order.
2. The worker charges the card.
3. The worker crashes.
4. The queue never receives the ack.
5. The queue sends the message again.

The card could be charged twice.

You might expect every message to be processed only once.

In real systems, that is difficult to guarantee because servers crash and networks fail.

The practical solution is to make processing safe to repeat.

Idempotency

An action is idempotent if doing it multiple times produces the same final result.

Examples:

✅ Set the order status to "Paid"

✅ Set the order status to "Delivered"

Not idempotent:

❌ Charge ₹280

❌ Add ₹50 to a wallet

Setting a status repeatedly gives the same result.

Charging money repeatedly charges the customer again and again.

The wrong way first: trust the queue to deliver once

Step through what happens when the worker trusts the queue to deliver each message once:

Three charges for two real orders: order-42 was charged 2 times.

The right way: an idempotency key

An idempotency key is a unique ID for one piece of work.

Examples: order-42, payment-982, refund-174.

Before doing the work, the system checks whether that ID has already been processed.

If it has, the system skips the duplicate request.

Step through the same deliveries with a key check:

The code behind the animation:

The right way: remember which keys were charged and skip repeats

const deliveries = ["order-42", "order-43", "order-42"];

const alreadyCharged = new Set(); // in real life: a database table with a unique key
const charges = [];
for (const orderId of deliveries) {
  if (alreadyCharged.has(orderId)) continue; // seen this key before: skip it
  charges.push(orderId);
  alreadyCharged.add(orderId);
}
console.log(`Charges made: ${charges.length} for 2 real orders`);

Expected output:

Charges made: 2 for 2 real orders

Now there are 2 charges for 2 real orders.

Many payment providers support this directly.

If the same idempotency key arrives twice, they return the original result instead of charging the card again.

In simple words: give every piece of work a unique name, and never do the same work twice.

When a Message Keeps Failing

Some messages fail every time.

For example:

• a broken address;
• invalid data;
• missing information.

If the queue retries forever, workers waste time on work that will never succeed.

After several failed attempts, the message is moved to a separate dead-letter queue.

A dead-letter queue is simply a parking area for failed messages.

Engineers can inspect the message, find the problem, fix it, and replay the message if needed.

Step through three messages with a limit of 3 attempts:

The code behind the animation:

Messages that fail 3 times go to the dead-letter queue

const maxAttempts = 3;
const messages = [
  { id: "order-1", failures: 0 },
  { id: "order-2", failures: 5 }, // broken address: fails every time
  { id: "order-3", failures: 1 }, // failed once, then worked
];

const done = [];
const deadLetters = [];
for (const m of messages) {
  if (m.failures >= maxAttempts) deadLetters.push(m.id);
  else done.push(m.id);
}
console.log(`Processed: ${done.join(", ")}`);
console.log(`Dead-letter queue: ${deadLetters.join(", ")}`);

Expected output:

Processed: order-1, order-3
Dead-letter queue: order-2

Processed: order-1 and order-3. Dead-letter queue: order-2.

Common Mistakes

Practice

What to Remember

• A queue is a waiting line for work.
• Producers put messages into the queue.
• Consumers take messages from the queue and process them.
• An ack tells the queue the work is finished.
• Queues absorb traffic spikes without losing work.
• Queue: one message goes to one worker.
• Pub/Sub: one message goes to many subscribers.
• A stream keeps old messages and allows replaying them later.
• Most systems use at-least-once delivery.
• Messages can arrive more than once.
• Idempotency makes processing safe to repeat.
• Idempotency keys stop duplicate work and duplicate charges.
• Dead-letter queues store messages that keep failing.
• Monitor queue length and message age to find problems early.

Further Reading

These are the sources this lesson was checked against:

• Amazon SQS Developer Guide: At-Least-Once Delivery: why standard queues can deliver a message more than once.
• Stripe API: Idempotent Requests: how a payment provider uses idempotency keys to avoid double charges.
• Apache Kafka Documentation: Introduction: streams, topics, partitions and offsets.
• Designing Data-Intensive Applications (Martin Kleppmann): message brokers, logs and exactly-once trade-offs.

Next: Rate Limiting, where you protect QuickBite's busiest endpoints from bots and runaway apps.