Partitioning and Consistent Hashing

By the Syncfusion DevTutorials team · Last updated October 20, 2026

Split data across machines: partitioning vs replication, choosing a partition key, range vs hash splits and hot spots, why hash mod N fails, the consistent hash ring, virtual nodes and hot keys.

Imagine a school library with one hug card catalogue.

When the library contains only a few hundred books, a single drawer and one librarian are enough. As the collection grows to millions of books, the drawer becomes difficult to manage and a single librarian cannot keep up with the workload.

To solve the problem, the library splits the catalogue into multiple drawers. One drawer might contain books beginning with A-F, another G-M, and another N-Z. Each drawer has its own librarian, allowing many people to search at the same time.

Databases solve growth problems in a similar way.

Instead of storing all data on one machine, data can be split across several machines. This process is called partitioning (or sharding).

The main idea: divide large datasets into smaller slices so that each machine handles only part of the work, and choose a splitting strategy that minimizes data movement when machines are added or removed.

What You Will Learn

In this lesson, you will learn:

1. When a single database is no longer sufficient.
2. The difference between partitioning and replication.
3. What a partition key is and how to choose one.
4. How range partitioning works and why it creates hot spots.
5. How hash partitioning works.
6. Why hash mod N causes large amounts of data movement.
7. How consistent hashing solves the resizing problem.
8. What virtual nodes are and why they help.
9. How to handle extremely popular data items known as hot keys.

When One Database Is Not Enough

In previous lessons, QuickBite's database easily fit on a single machine.

However, imagine QuickBite expands across many cities and stores data for several years.

Here is the estimate after growing to 20 cities:

At this scale:

• A single machine may not have enough storage.
• A single machine may not support the required write throughput.
• A machine failure becomes increasingly costly.

Eventually, the data must be divided across multiple machines.

Partitioning vs Replication

Partitioning and replication are often confused because both involve multiple machines.

However, they solve different problems.

Replication

Replication creates complete copies of the data.

Benefits:

• Fault tolerance.
• High availability.
• Read scalability.

Every replica contains the entire dataset.

Partitioning

Partitioning divides data into slices.

Benefits:

• More storage capacity.
• More write capacity.
• Improved scalability.

Each machine stores only a portion of the data.

Most large systems use both techniques together.

A partition is usually replicated onto multiple machines for reliability.

In simple terms: replication creates additional copies of data. Partitioning divides data into pieces.

The Partition Key

To partition data, the system must decide which machine owns each record.

The value used to make that decision is called the partition key.

A good partition key should:

• Distribute data evenly.
• Distribute traffic evenly.
• Keep related data together whenever possible.

These goals often compete with each other.

Choosing the correct partition key is one of the most important decisions in system design.

Range Partitioning

How Range Partitioning Works

Range partitioning assigns ranges of values to different machines.

Examples:

• Cities:

  • Bengaluru → Machine 1
  • Chennai → Machine 2
  • Kochi → Machine 3

• Time:

  • January → Machine 1
  • February → Machine 2
  • March → Machine 3

Range partitioning is easy to understand and simplifies some queries.

Related data is often stored together.

The Hot Spot Problem

Real-world traffic is rarely distributed evenly.

Some cities receive far more orders than others.

The chart shows each machine's share when orders are split by city:

One machine now handles most of the workload.

This imbalance creates a hot spot.

A hot spot occurs when one partition receives significantly more traffic than the others.

The same issue occurs when partitioning by date because today's partition receives nearly all current writes.

Hash Partitioning

How It Works

Instead of grouping by ranges, hash partitioning uses a hash function.

A hash function converts values such as an order ID, a restaurant ID or a user ID into numbers that appear random.

Partition ownership is then determined using the hash value.

The goal is to spread records evenly across all machines.

Benefits

Hash partitioning generally:

• Distributes data evenly.
• Distributes traffic evenly.
• Reduces hot spots.

The Trade-Off

Questions involving the partition key remain efficient.

Questions using other fields may require contacting every partition.

This is known as a scatter-gather query.

Step through three questions on 8 machines:

This illustrates an important rule:

Choose the partition key based on the application's most important access patterns.

If restaurants frequently ask "Show all my orders", then partitioning by restaurant ID may be a good choice.

The Problem with Hash Mod N

The Traditional Approach

Machine = Hash(Key) % NumberOfMachines

This provides an even distribution while the machine count remains unchanged.

What Happens When Machines Change?

Adding or removing a machine changes the modulo calculation.

As a result, many records move to different machines.

The chart counts what happens to 1,000 orders when a fifth machine joins four:

Most data moves even though only one new machine was added.

This creates:

• Large data migrations.
• Increased network traffic.
• Temporary performance degradation.

A better solution is needed.

Consistent Hashing

The Hash Ring

Consistent hashing places both machines and data on a circular ring.

The process works as follows:

1. Hash each machine to a position on the ring.
2. Hash each data item to a position on the ring.
3. Assign the data item to the first machine encountered when moving clockwise.

This creates a stable mapping between data and machines.

Why It Works Better

When a new machine joins the ring:

• Only nearby data moves.
• Most existing data remains where it already lives.

Step through adding a machine to a small ring numbered 0 to 99:

The code behind the animation:

A new machine at 55 takes only the orders between 41 and 55

// A tiny ring numbered 0 to 99. Machines sit at these points.
function owner(position, machines) {
  const sorted = [...machines].sort((a, b) => a.at - b.at);
  const next = sorted.find((m) => m.at >= position); // first one clockwise
  return (next ?? sorted[0]).name; // past the last? wrap round to the first
}

const before = [{ name: "A", at: 10 }, { name: "B", at: 40 }, { name: "C", at: 70 }];
const after = [...before, { name: "D", at: 55 }];

const orders = [5, 25, 45, 50, 60, 85, 95];
const moved = orders.filter((p) => owner(p, before) !== owner(p, after));
console.log(`Before: ${orders.map((p) => `${p}→${owner(p, before)}`).join("  ")}`);
console.log(`After:  ${orders.map((p) => `${p}→${owner(p, after)}`).join("  ")}`);
console.log(`Moved: ${moved.join(", ")}`);

Expected output:

Before: 5→A  25→B  45→C  50→C  60→C  85→A  95→A
After:  5→A  25→B  45→D  50→D  60→C  85→A  95→A
Moved: 45, 50

Before: 5→A, 25→B, 45→C, 50→C, 60→C, 85→A, 95→A. After adding D at 55, only 45 and 50 move, to D.

Only the records that belong to the new machine's section of the ring change ownership.

Everything else remains untouched.

This dramatically reduces rebalancing work.

Virtual Nodes

A ring with only one position per machine may distribute data unevenly.

For example, with one position each, the slices could come out like this:

This imbalance creates performance issues.

The solution is virtual nodes.

Instead of placing a machine on the ring once, place each machine at many points, for example 100 positions each for A, B and C.

The many positions average out over the ring, producing a much more even distribution.

Virtual nodes also allow stronger servers to own more data simply by assigning them more virtual nodes.

Try It: Add or Remove a Server

Experiment with:

• Hash mod N
• Consistent hashing
• Virtual nodes

Observe how:

• Hash mod N reshuffles most data.
• Consistent hashing moves only a small portion.
• More virtual nodes create a more balanced distribution.

In simple terms: consistent hashing ensures that adding a server moves only the data that the new server should own.

Hot Keys

The Remaining Problem

Even when data is perfectly distributed, one individual item may become extremely popular.

Imagine a restaurant called Famous Biryani during a major sporting event.

Thousands of requests target the same restaurant.

All those requests still map to the same partition.

This creates a hot key.

A hot key can overwhelm an otherwise healthy partition.

Handling Hot Keys

Option 1: Cache the Data

If the item is read frequently:

• Store it in a cache.
• Serve most requests without touching the database.

This is usually the preferred solution.

Option 2: Split the Key

Break one logical key into multiple physical keys.

For example: famous-biryani#0, famous-biryani#1, famous-biryani#2 and famous-biryani#3.

Step through 4,000 orders for one restaurant during a big match:

The code behind the animation:

Splitting one hot key four ways spreads 4,000 orders across 4 machines

const hotOrders = 4000; // orders for one restaurant during a big match
const parts = 4;

const perMachine = {};
for (let i = 0; i < hotOrders; i++) {
  const key = `famous-biryani#${i % parts}`; // add a small suffix
  perMachine[key] = (perMachine[key] ?? 0) + 1;
}
console.log(perMachine);

Expected output:

{
  "famous-biryani#0": 1000,
  "famous-biryani#1": 1000,
  "famous-biryani#2": 1000,
  "famous-biryani#3": 1000
}

Each of the four keys gets 1,000 orders.

The workload is now distributed across several partitions.

The trade-off is increased query complexity because reads must combine multiple partitions.

Hot-key splitting should be used only when necessary.

Common Mistakes

Practice

What to Remember

• Partitioning divides data into slices, while replication creates copies.
• A partition key determines where each record is stored.
• Range partitioning is simple but can create hot spots.
• Hash partitioning distributes data more evenly.
• Queries that do not use the partition key may require scatter-gather operations.
• Hash mod N causes large amounts of data movement when machine counts change.
• Consistent hashing minimizes data movement during scaling events.
• Virtual nodes improve balance across the hash ring.
• Hot keys can overload individual partitions even when overall distribution is balanced.
• Hot keys are commonly handled through caching or key splitting.

Further Reading

These are the sources this lesson was checked against:

• Amazon's Dynamo paper (2007): consistent hashing with virtual nodes in a real production system.
• Designing Data-Intensive Applications (Martin Kleppmann): partitioning by key range and by hash, hot spots and rebalancing.
• System Design Primer: Sharding: a short overview of sharding and its trade-offs.

Next: CAP, PACELC and Quorums, where you will learn how distributed systems behave when communication between machines fails and how consistency, availability, and quorum-based decisions affect system design.