# Reranking Course: [Retrieval-Augmented Generation](https://www.devtutorials.ai/learn/rag) · Section: Context Refinement Source: https://www.devtutorials.ai/learn/rag/context-refinement/reranking > Two-stage retrieval and rerankers: high-recall first stage, high-precision second stage, pool sizing and latency, MMR diversity, cross-encoders vs bi-encoders, joint attention, reranker models and score normalization. Hiring for a job usually happens in two rounds. First, someone quickly reviews hundreds of CVs and selects the applicants who look most promising. Then an interviewer spends more time with those selected candidates and chooses the best few. Interviewing every applicant would take too long, but choosing people based only on a quick scan would not be very accurate. Two-stage retrieval works in the same way. A fast first stage finds a broad set of possible matches. Then a slower, smarter reranker looks at those matches more carefully and puts them in a better order. The main idea: Stage 1 (vector search, keyword search, or hybrid search) focuses on high recall, which means getting the correct chunks somewhere into a pool of candidates, usually 20–100 chunks. Stage 2 uses a reranker, often a cross-encoder, to read the question and each candidate chunk together and score how well they match. This improves precision, meaning the most relevant chunks move to the top. You can adjust the pool size to balance accuracy, latency, and cost. You can also normalise scores and keep the final results diverse instead of returning several nearly identical chunks. ## A Real-World Example A Real-World Example Brightgrid's gold set revealed an important pattern. For 90% of questions, the correct chunk appeared somewhere in the top 20 results returned by hybrid search. However, it appeared in the top 3 results only 70% of the time, and those top 3 chunks were the ones sent to the model. For the question, "Is water damage covered by the warranty?", the top 3 results were the warranty overview, the safety article, and a general returns article. The chunk that actually stated "Damage from water … is not covered" was ranked 7th. The team added a reranker. Hybrid search first returned the top 30 candidates. A small cross-encoder then rescored those 30 candidates, and only the best 3 were sent to the model. The correct chunk moved to rank 1. Top-3 accuracy increased from 70% to 86%, while adding only about 120 ms of extra latency. In simple words: search widely first, then examine the results carefully. ## What You Will Learn In this lesson, you will learn: 1. Two-stage architecture: fast broad retrieval, then deep reranking. 2. Trade-offs: latency vs accuracy, candidate pool size, cost and throughput. 3. Pool management: diversity controls (MMR) and over-retrieval. 4. Cross-encoders: joint query-chunk attention, bi-encoder vs cross-encoder, reranker models. 5. The reranking pipeline: rescoring, score normalization and final ranking. In the RAG Lab, turn on **Rerank (two-stage)** in Retrieval settings. The lab then takes a bigger pool (at least 10) from the first stage and rescores it with a simple **Tier 1 reranker** that reads the question and each chunk together and rewards word coverage and closeness. Real systems use a cross-encoder model for this step. The idea and the pipeline are the same. ## Two-Stage Architecture ### Stage 1: Fast Broad Retrieval (High Recall) The first stage must search the entire index quickly. To do that, it uses methods such as vector search, BM25, or hybrid search, which rely on pre-computed representations. Its goal is not to produce the perfect top result. Instead, its goal is to make sure the correct chunks are included somewhere in the candidate pool. This stage is usually measured using recall@pool, which tells you how often the correct chunk appears anywhere within the top N retrieved chunks. ### Stage 2: Deep Candidate Reranking (High Precision) The second stage only works with the candidate pool returned by stage 1. Because the pool is much smaller, it can spend more time analysing each candidate. The reranker reads both the question and a candidate chunk together and produces a new relevance score. It then reorders the candidates based on those scores. This stage is often measured using precision@k or hit@k, where k is the number of chunks ultimately sent to the model. ```js runnable title="Stage 1 finds them, stage 2 ranks them" // Two-stage retrieval: a fast first stage keeps many candidates (high recall), // a careful second stage reorders them (high precision). const relevant = new Set(["warranty#1", "warranty#3"]); const stage1 = [ "cancel-order#1", "glow-strip-setup#1", "brightgrid-plus#1", "warranty#3", "returns-policy#2", "safety#2", "warranty#1", "energy-use#1", ]; // Stand-in for a reranker's scores: it reads question and chunk together, so it's more accurate. const rerankScore = { "warranty#1": 0.94, "warranty#3": 0.9, "returns-policy#2": 0.41, "cancel-order#1": 0.22, "safety#2": 0.2, "brightgrid-plus#1": 0.05, "glow-strip-setup#1": 0.04, "energy-use#1": 0.03, }; const stage2 = [...stage1].sort((a, b) => rerankScore[b] - rerankScore[a]); const precisionAt = (list, k) => list.slice(0, k).filter((id) => relevant.has(id)).length / k; const recallAt = (list, k) => list.slice(0, k).filter((id) => relevant.has(id)).length / relevant.size; console.log( `stage 1 top 3: ${stage1.slice(0, 3).join(", ")} precision@3 ${precisionAt(stage1, 3).toFixed(2)}`, ); console.log( `stage 1 top 8 recall@8: ${recallAt(stage1, 8).toFixed(2)} (both relevant chunks are somewhere in the pool)`, ); console.log( `stage 2 top 3: ${stage2.slice(0, 3).join(", ")} precision@3 ${precisionAt(stage2, 3).toFixed(2)}`, ); ``` Expected output: ```text stage 1 top 3: cancel-order#1, glow-strip-setup#1, brightgrid-plus#1 precision@3 0.00 stage 1 top 8 recall@8: 1.00 (both relevant chunks are somewhere in the pool) stage 2 top 3: warranty#1, warranty#3, returns-policy#2 precision@3 0.67 ``` Stage 1's top 3 contained neither relevant chunk, but its pool of 8 held both. Stage 2 moved them to the top. ## Tradeoff Optimization ### Latency vs Accuracy Balancing A reranker adds extra processing time because it must evaluate each candidate individually. Small cross-encoders running on a CPU may take a few milliseconds per question-chunk pair. Larger models running on a GPU may provide better accuracy but require additional resources. When designing a system, set a latency budget for the entire response and decide how much of that budget reranking can use. ### Candidate Pool Sizing A bigger pool raises the chance the right chunk is in it, but recall gains shrink while reranking time keeps growing: ```js runnable title="Recall and time for different pool sizes" // Sizing the candidate pool: a bigger pool finds more, but reranking costs time per candidate. const rerankMsPerCandidate = 4; // a small cross-encoder on a CPU const recallAtPool = { 10: 0.82, 20: 0.9, 50: 0.96, 100: 0.98, 200: 0.985 }; // measured on a gold set console.log("pool recall rerank time"); for (const [pool, recall] of Object.entries(recallAtPool)) { const ms = Number(pool) * rerankMsPerCandidate; console.log( `${pool.padStart(4)} ${recall.toFixed(3)} ${String(ms).padStart(6)} ms${ms > 250 ? " (over a 250 ms budget)" : ""}`, ); } console.log( "Here a pool of about 50 is the sweet spot: most of the recall, inside the budget.", ); ``` Expected output: ```text pool recall rerank time 10 0.820 40 ms 20 0.900 80 ms 50 0.960 200 ms 100 0.980 400 ms (over a 250 ms budget) 200 0.985 800 ms (over a 250 ms budget) Here a pool of about 50 is the sweet spot: most of the recall, inside the budget. ``` Measure **recall@pool** on your gold set, and pick the smallest pool that gets most of the available recall within your budget. 20–100 is typical. ### Compute Cost Allocation across Stages Spend computing resources where they provide the most benefit. In many systems, this means using a fast and scalable first stage and a stronger reranker on a much smaller pool. Hosted reranking services are often priced according to the number of candidates processed, so unnecessarily large pools can increase costs. ### Throughput Scaling in Two-Stage Search Throughput means the number of questions a system can process each second. Rerankers are often the bottleneck. Common strategies for improving throughput include: Batching multiple candidate pairs into a single reranker call. Caching previously computed reranking results. Skipping reranking when the first-stage results already show strong confidence. ## Candidate Pool Management ### Over-Retrieval Sizing Strategies Over-retrieval means retrieving more candidates than you plan to keep. For example, you might retrieve 30 candidates but only send the best 3 to the model. This gives the reranker room to improve the ranking and helps compensate for filtering or deduplication later in the pipeline. A common starting point is: Pool size = k × 5 to k × 10 Then adjust it using results from your gold set. ### Candidate Pool Diversity Controls A reranker scores each candidate independently. Because of this, very similar chunks often receive very similar scores and may dominate the final results. MMR (Maximal Marginal Relevance) helps solve this problem. It selects results one at a time, balancing: Relevance to the question. Difference from already selected results. A parameter called lambda controls the balance: 1.0 = relevance only. Lower values = more diversity. ```js runnable title="MMR picks relevant and different chunks" // MMR (Maximal Marginal Relevance): pick results that are relevant AND different from // what's already picked, so the top 3 isn't three copies of one fact. const cands = [ { id: "reset-password#1", rel: 0.9, words: ["forgot", "password", "email", "link"] }, { id: "forgot-password#1", rel: 0.89, words: ["forgot", "password", "email", "link"] }, { id: "two-step#2", rel: 0.7, words: ["backup", "codes", "phone"] }, { id: "reset-password#2", rel: 0.6, words: ["spam", "folder", "email"] }, ]; const sim = (a, b) => a.words.filter((w) => b.words.includes(w)).length / Math.max(a.words.length, b.words.length); function mmr(list, k, lambda) { const picked = []; const rest = [...list]; while (picked.length < k && rest.length) { let best = 0; let bestScore = -Infinity; rest.forEach((c, i) => { const redundancy = picked.length ? Math.max(...picked.map((p) => sim(c, p))) : 0; const s = lambda * c.rel - (1 - lambda) * redundancy; if (s > bestScore) { bestScore = s; best = i; } }); picked.push(rest.splice(best, 1)[0]); } return picked.map((c) => c.id).join(", "); } console.log("relevance only (lambda 1.0):", mmr(cands, 3, 1)); console.log("MMR (lambda 0.7):", mmr(cands, 3, 0.7)); ``` Expected output: ```text relevance only (lambda 1.0): reset-password#1, forgot-password#1, two-step#2 MMR (lambda 0.7): reset-password#1, two-step#2, reset-password#2 ``` ## Cross-Encoder Mechanics ### Bi-Encoder vs Cross-Encoder Tradeoffs A bi-encoder converts the question and each chunk into vectors separately. Chunk vectors are generated ahead of time and stored in the index, making retrieval fast. However, the question and chunk never directly interact. Similarity is usually measured using cosine similarity between vectors. A cross-encoder receives both the question and chunk together as a single input and outputs a relevance score. Because it can directly compare the words and meaning of both texts, it is usually more accurate. However, it must run for every question-chunk pair at search time, making it too slow for searching an entire index. ```js runnable title="Where the model work happens" // Bi-encoder vs cross-encoder: where the model work happens. // A bi-encoder embeds every chunk once, ahead of time, and only the question at search time. // A cross-encoder must read the question together with each candidate, at search time. function modelCalls({ chunks, questions, pool }) { const bi = { ahead: chunks, perQuestion: 1 }; const cross = { ahead: 0, perQuestion: pool }; return { bi: bi.ahead + bi.perQuestion * questions, cross: cross.ahead + cross.perQuestion * questions, crossOnAll: chunks * questions, // reranking everything, with no first stage }; } const c = modelCalls({ chunks: 100000, questions: 1000, pool: 50 }); console.log(`bi-encoder search: ${c.bi.toLocaleString("en-US")} model calls`); console.log( `cross-encoder on a pool of 50: ${c.cross.toLocaleString("en-US")} model calls`, ); console.log( `cross-encoder on every chunk: ${c.crossOnAll.toLocaleString("en-US")} model calls`, ); console.log( "That's why the cross-encoder only reranks a small pool from a fast first stage.", ); ``` Expected output: ```text bi-encoder search: 101,000 model calls cross-encoder on a pool of 50: 50,000 model calls cross-encoder on every chunk: 100,000,000 model calls That's why the cross-encoder only reranks a small pool from a fast first stage. ``` | | Bi-encoder | Cross-encoder | | --- | --- | --- | | Input | One text at a time | Question + chunk together | | Output | A vector | A relevance score | | Work at search time | Embed the question once | One model run per candidate | | Accuracy | Good | Better, especially on details | | Used for | Stage 1 search | Stage 2 reranking | ### Joint Query-Chunk Attention Scoring To understand why a cross-encoder is usually more accurate, it helps to understand attention. Attention is a mechanism used by transformer models that helps the model decide which words are important when understanding text. Instead of looking at words in isolation, the model looks at how words relate to one another. In a cross-encoder, the question and the chunk are placed into one input sequence. This allows words in the question to directly interact with words in the chunk. For example, imagine the question is: Is water damage covered by the warranty? And the chunk contains: Damage from water is not covered under the warranty. Because both texts are processed together, the model can directly connect: "water damage" in the question "damage from water" in the chunk It can also recognise important details such as: exact matches similar meanings negative statements such as "not covered" which sentence actually answers the question This direct interaction between the question and the chunk is one of the main reasons cross-encoders often produce better relevance rankings than bi-encoders. ### Self-Attention Across Query and Passage Pairs A cross-encoder combines the question and chunk into a single input that looks like this:[CLS] question [SEP] chunk [SEP] Where: [CLS] is a special token placed at the beginning. [SEP] is a special token used to separate different pieces of text. The model reads the entire sequence as one piece of input. During processing, words from the question can attend to words in the chunk, and words in the chunk can attend to words in the question. After processing the sequence, the model produces an output for the [CLS] token. This output is converted into a single relevance score that represents how well the chunk answers the question. Cross-encoders also have an input length limit, often around 512 tokens. If a chunk is too long, part of it may be cut off before processing. This is another reason why good chunk sizes matter. If important information is truncated, the reranker may not see the answer and could assign a lower score. ### Specialized Reranker Models (Cohere, BGE) Rerankers are trained using large collections of questions paired with both relevant and irrelevant passages. This training helps them learn what makes a passage a good answer. Common options include: Hosted APIs, such as Cohere Rerank. These are easy to use, support multiple languages, and are managed by the provider. Open-source models, such as BGE rerankers and MS MARCO cross-encoders like ms-marco-MiniLM-L-6-v2. These can be run on your own infrastructure and some smaller models can even run in a browser using Transformers.js. When choosing a reranker, evaluate: retrieval quality on your gold set supported languages latency operating cost The best choice depends on your application's requirements. ## Reranking Pipeline ### Rescoring Candidate Chunks The pipeline: A typical reranking pipeline works as follows: Stage 1 retrieves a candidate pool, such as 30 chunks, after applying any filters. A (question, chunk) pair is created for every candidate. These pairs are sent to the reranker, preferably in a single batch. The reranker returns one relevance score for each pair. The candidates are reordered using those new scores. ### Score Normalization Rerankers often return raw scores called logits. A logit is simply the model's raw relevance score before any conversion. Logits can be positive or negative and may vary greatly between models. Raw logits are useful internally, but they are difficult to interpret directly. To make them easier to use, they are often normalised. Two common approaches are: Sigmoid Converts logits into values between 0 and 1. Scores closer to 1 indicate higher relevance. A score above 0.5 is often treated as relevant. Min-max normalisation Rescales scores within a single result list. The lowest score becomes 0 and the highest becomes 1. Useful for display and blending scores. Does not represent absolute relevance. ```js runnable title="Sigmoid and min-max normalization" // Score normalization: rerankers output raw scores (logits). Turn them into 0..1 values // so you can use one threshold and compare across questions. const logits = { "warranty#1": 7.2, "warranty#3": 5.9, "returns-policy#2": -1.4, "safety#2": -4.8, }; const sigmoid = (x) => 1 / (1 + Math.exp(-x)); const values = Object.values(logits); const lo = Math.min(...values); const hi = Math.max(...values); console.log("chunk logit sigmoid min-max"); for (const [id, x] of Object.entries(logits)) { console.log( `${id.padEnd(18)} ${x.toFixed(1).padStart(5)} ${sigmoid(x).toFixed(3).padStart(7)} ${((x - lo) / (hi - lo)).toFixed(3).padStart(7)}`, ); } console.log( "Keep chunks with sigmoid >= 0.5:", Object.keys(logits) .filter((id) => sigmoid(logits[id]) >= 0.5) .join(", "), ); ``` Expected output: ```text chunk logit sigmoid min-max warranty#1 7.2 0.999 1.000 warranty#3 5.9 0.997 0.892 returns-policy#2 -1.4 0.198 0.283 safety#2 -4.8 0.008 0.000 Keep chunks with sigmoid >= 0.5: warranty#1, warranty#3 ``` ### Final Context Rank Assignment After reranking: Sort candidates by reranker score. Remove candidates that fall below a relevance threshold. Apply MMR or deduplication if needed. Keep the top k chunks. Send those chunks to the context-packing stage. It is also useful to keep both: Stage 1 retrieval scores Stage 2 reranker scores in your trace or logs. This makes it easier to understand what changes the reranker made and to troubleshoot retrieval problems. A reranker can only reorder chunks that are already in the candidate pool. If the correct chunk never reaches Stage 2, the problem is usually in Stage 1. In that case, improve chunking, embeddings, hybrid search, or query transformation rather than tuning the reranker. Reranking adds latency to every question. Always measure its impact using your gold set. If accuracy improves only slightly, consider skipping reranking or enabling it only when Stage 1 results have low confidence. Compare retrieval with and without reranking in the lab, then open the Trace view to inspect both stages: First stage → Candidate pool → Reranker → Final top-k chunks A reranker can't rescue a chunk that never made it into the pool. If the right chunk isn't in stage 1's top N, fix stage 1 first: chunking, embeddings, hybrid search or query transformation. Reranking adds latency to **every** question. Measure the gain on your gold set. If top-k accuracy barely moves, skip the reranker, or only use it when stage 1 isn't confident. Compare search with and without the rerank step in the lab, then open the **Trace** to see the two stages: ## Common mistakes - **A pool that's too small**, so the reranker has nothing good to promote. - **Chunks longer than the reranker's input limit**, so the key sentence gets cut off. - **Using min-max scores as absolute relevance**: the best of bad results still scores 1.0. - **Forgetting filters** on the pool, so the reranker promotes chunks the user can't see. - **No latency budget**, so reranking quietly doubles response time. ## Exercises Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer. ## What to Remember - **Two stages:** a fast first stage for **recall** (a pool of 20–100), then a reranker for **precision** at the top. - Size the **pool** from recall@pool and your **latency budget**. **Batch** reranker calls, cache, or skip when stage 1 is confident. - **Over-retrieve** to leave room for filters and deduplication. Use **MMR** for variety. - A **cross-encoder** reads the question and chunk **together** (joint attention), so it's more accurate but too slow for whole-index search. Options include hosted (Cohere) and open (BGE, MS MARCO) rerankers. - **Normalize** reranker scores (sigmoid), apply a threshold, and keep both stages' scores in the trace. Next: **Context Packing**, fitting the best chunks into the prompt, in the right order.