Design LLM Inference Serving

By the Syncfusion DevTutorials team · Last updated October 20, 2026

Run an AI support assistant on GPUs: prefill and decode, time to first token, streaming, continuous batching and answer caching.

"Where is my order?" "Can I change the address?" "The rider says they are outside but I can't see them." QuickBite's support team handles thousands of these questions every evening. To reduce support workload and improve response times, QuickBite introduces an AI-powered support assistant based on a Large Language Model (LLM).

Unlike the systems covered so far, LLM serving has a completely different challenge.

Most backend systems are limited by CPUs, databases or network traffic. LLM systems are limited by GPUs, which are extremely powerful but also extremely expensive.

Another difference is response generation. Traditional APIs produce a response almost instantly. LLMs generate responses one token at a time, meaning a complete answer can take several seconds.

The design question becomes:

How can we keep expensive GPUs busy enough to justify their cost while making customers feel like answers arrive instantly?

The main idea: start showing tokens as soon as they are generated, process many conversations together on the same GPU and avoid GPU usage whenever a cached answer can be used.

What You Will Learn

In this case study, you will:

1. Write the requirements for an AI support assistant.
2. Estimate GPU workload and response generation costs.
3. Understand Time to First Token (TTFT) and Full Answer Time.
4. Stream responses instead of waiting for completion.
5. Use continuous batching to maximize GPU utilization.
6. Cache common answers to reduce GPU demand.
7. Build and validate a scalable LLM serving architecture.

Step 1: Requirements

Before designing the system, define exactly what the assistant must do.

What It Must Do (Functional)

• Answer customer support questions inside chat.
• Use order information to provide personalized responses.
• Access current order status and delivery information.
• Escalate conversations to human agents when required.
• Handle common support requests automatically.
• Continue conversations across multiple messages.

How Well It Must Do It (Non-Functional)

• The first words of an answer should appear within 800 ms at p99.
• Complete answers should arrive within 15 seconds at p99.
• GPU utilization should remain efficient because GPUs are expensive.
• The system must remain affordable while handling peak traffic.
• Customers must be rate limited to prevent abuse.
• The platform should continue operating when a GPU server fails.

Out of Scope

The first version does not include:

• Custom model training
• Voice support
• Image generation
• Multimodal interactions
• Autonomous task execution

The focus of this lesson is efficient LLM inference serving.

Step 2: Estimate

LLM inference happens in two stages.

First, the model reads the prompt. This is called the prefill phase and is usually fast.

The prompt may include:

• System instructions
• Support policies
• Order details
• Customer messages

After reading the prompt, the model generates an answer. This is called the decode phase.

Unlike traditional software, the model generates one token at a time.

For one typical chat: the prompt is 600 tokens (instructions, order details and the question) and the answer is 150 tokens. Reading the prompt takes about 100 ms. Writing takes 20 ms per answer token, so 150 × 20 = 3,000 ms.

What to notice:

• Reading the prompt is fast.
• Writing the answer takes most of the time.
• Most LLM latency comes from token generation.

Now consider peak traffic.

Suppose QuickBite receives 30 new support chats every second.

Each chat spends about 3 seconds writing. With 30 new chats a second, 30 × 3 = 90 chats are being written at any moment. If one GPU wrote one chat at a time, QuickBite would need 90 GPUs.

What to notice:

• Many responses are being generated at the same time.
• One chat per GPU would require a huge number of GPUs.
• GPU cost becomes impractical very quickly.
• The rest of the design focuses on reducing GPU usage while keeping response times low.

Step 3: Stream the Answer

A common mistake is waiting for the entire answer before sending anything to the customer.

In that design:

1. Customer asks a question.
2. The GPU generates the full answer.
3. Customer waits.
4. The complete answer appears.

Even if the response only takes a few seconds, it feels slow.

Instead, stream tokens as they are generated.

The flow becomes:

1. Customer asks a question.
2. The GPU generates the first token.
3. The first token is sent immediately.
4. Additional tokens continue to stream.

Two metrics matter:

• Time to First Token (TTFT): how long it takes before the customer sees the first words.
• Full Answer Time: how long it takes to receive the complete answer.

Streaming does not make the GPU faster.

The answer still takes the same amount of time to generate.

However, customers see progress almost immediately, which makes the experience feel much faster.

Step 4: Continuous Batching

Running one conversation per GPU wastes most of the GPU's computing power.

A GPU can process multiple conversations at the same time.

Batching allows multiple conversations to share the same GPU.

Continuous batching improves this even further.

Instead of waiting for an entire batch to finish, conversations can join and leave active batches dynamically.

Step through how a batch fills and how chats join and leave it:

One chat at a time, a GPU finishes an answer every 1.8 seconds: 0.6 answers a second. With 32 chats in a batch, each answer takes about 3 seconds, but 32 run together: 32 ÷ 3 = 10.7 answers a second per GPU.

What to notice:

• Batching dramatically increases throughput.
• The same GPU can process many more conversations.
• Infrastructure cost drops significantly.

One important limit is GPU memory.

Each active conversation keeps a working memory called the KV Cache.

The KV Cache uses GPU memory and limits how many conversations can fit into a batch.

That is why batches are measured in dozens of conversations, not thousands.

Step 5: Do Not Ask the GPU at All

The cheapest inference request is the one that never reaches the model.

Many customer questions are repetitive.

Examples include:

• Where is my order?
• How do I request a refund?
• Why is my order delayed?
• How can I contact the rider?

These questions often depend on information the platform already knows.

An answer cache can store responses for common situations.

The cache key may include:

• Question category
• Order status
• Delivery state

When a matching answer exists:

1. Read the answer from the cache.
2. Return it immediately.
3. Skip GPU inference.

Follow 30 chats a second through an answer cache that knows 40% of the answers:

What this does to the GPU bill (at $1,500 a GPU a month):

What to notice:

• Caching reduces GPU usage dramatically.
• Lower GPU usage means lower cost.
• Many common questions can be answered without inference.

Step 6: Build It on the Canvas and Test It

Open the design sandbox.

The canvas starts with:

• 2 load balancers
• 2 chat API servers
• 2 GPU servers
• No streaming
• One conversation per GPU

The simulator runs:

• 30 new chats every second
• Response generation workloads
• Machine failures
• Cost constraints

What you will probably find:

• One conversation per GPU is extremely expensive.
• Streaming improves user experience immediately.
• Continuous batching greatly increases throughput.
• Answer caching reduces GPU demand.
• Too few GPUs create long response times.

A design that passes:

• 2 load balancers
• 2 chat API servers
• Answer cache
• 3 GPU servers
• Continuous batching
• Streaming responses

This design balances:

• User experience
• Reliability
• GPU utilization
• Infrastructure cost

while remaining within budget.

Common Mistakes

Practice

What to Remember

• LLM inference consists of reading the prompt and generating tokens.
• Most latency comes from token generation.
• Measure both Time to First Token and Full Answer Time.
• Stream tokens to improve perceived performance.
• Continuous batching dramatically improves GPU utilization.
• GPU memory limits batch sizes through KV Cache usage.
• Cache common questions whenever possible.
• Never cache customer-specific responses.
• Rate limit users to protect GPU capacity.
• Efficient GPU utilization is the key to affordable LLM serving.

Further Reading

These are the sources this lesson was checked against:

• vLLM documentation: continuous batching and memory management for LLM serving.
• Anyscale: How continuous batching enables 23x throughput in LLM inference: why batching changes GPU throughput so much.
• MDN: Using server-sent events: streaming text from server to browser.

Next: Capstone, where you design the complete QuickBite platform using everything learned throughout the High-Level Design course.