# Top-K Retrieval Course: [Retrieval-Augmented Generation](https://www.devtutorials.ai/learn/rag) · Section: Retrieval Engine Source: https://www.devtutorials.ai/learn/rag/retrieval-engine/top-k-retrieval > Turning a question into the right few chunks: score cutoffs, top-k selection, dynamic thresholds, the search flow and its latency, deduplication, out-of-distribution checks, fallbacks and corrective retrieval. A shop assistant who always brings you exactly three items, whether or not the shop has what you asked for, is not very helpful. Ask for "blue running shoes, size 44" in a shop that only sells hats, and you still get three hats. A good assistant brings only things that really match, says "we do not have that" when nothing fits, and asks a colleague or checks the back room when the first look comes up short. Retrieval is the same. **Top-k** search always returns k chunks. Your job is to keep the good ones, drop the noise, and know when to try again or say "I do not know". **The main idea:** top-k retrieval turns the question into a vector, searches the index, ranks by distance, and fetches the chunks' text and metadata. Then quality control takes over: **score cutoffs** (fixed or dynamic) remove weak matches, **deduplication** removes copies, **out-of-distribution** checks catch questions far from your content, and **fallbacks** like corrective retrieval try again before giving up. ## A Real-World Example Brightgrid noticed something odd in its logs: for every question, the assistant received exactly three chunks, even for "What is the CEO's favourite colour?" The model got privacy, contact and safety articles, tried to be helpful, and wrote a confusing answer about data privacy. The team added three rules: 1. **A score cutoff.** Chunks below a minimum similarity are dropped. If nothing is left, the app answers "I do not know based on the help center" without calling the model at all. 2. **Deduplication.** The two near-identical password articles no longer fill two of the three slots. 3. **A corrective retry.** If the best score is weak, the app expands the question with synonyms and searches once more before giving up. "My light has no internet" now finds the Wi-Fi article. Unhelpful answers dropped by a third, and model costs fell too, because refused questions never reached the model. **In simple words:** take only what really matches, and have a plan for when nothing does. ## What You Will Learn In this lesson, you will learn: 1. Retrieval thresholding: score cutoffs, top-k selection, noise filtering and dynamic thresholds. 2. The search execution flow: query vectorization, index search and ranking, payload extraction and latency tracking. 3. Retrieval quality control: out-of-distribution matches, duplicates, low-confidence results and fallback retrieval. The RAG Lab's Ask and Search tabs now have **Retrieval settings**. The **Score cutoff** slider drops results that score below a share of the best score. Open the **Trace** tab after a search to see each step and how many candidates survived it. ## Retrieval Thresholding ### Top-K Candidate Selection **k** is how many chunks you keep. Small k (3–5) keeps prompts short and focused. Larger k (10–20) gives the model more to work with, but adds noise and cost. A common pattern is to retrieve more than you need, say 20, then cut and rerank (next chapter) down to the best 3–5. ### Similarity Score Cutoffs Top-k has a blind spot: it returns k results **no matter what**. A fixed **minimum score** removes chunks that are not really related: ```js runnable title="Top-k alone vs top-k with a cutoff" // Top-k always returns k results, even when none are relevant. A score cutoff fixes that. const results = { "How long is the warranty?": [ { id: "warranty#1", score: 0.82 }, { id: "warranty#3", score: 0.61 }, { id: "safety#2", score: 0.34 }, ], "What is the CEO's favourite colour?": [ { id: "privacy#1", score: 0.21 }, { id: "contact-support#1", score: 0.19 }, { id: "safety#1", score: 0.17 }, ], }; const MIN_SCORE = 0.4; for (const [q, hits] of Object.entries(results)) { const kept = hits.filter((h) => h.score >= MIN_SCORE); console.log(q); console.log(" top-3 only: ", hits.map((h) => h.id).join(", ")); console.log( " with cutoff: ", kept.map((h) => h.id).join(", ") || "(nothing: answer 'I do not know')", ); } ``` Expected output: ```text How long is the warranty? top-3 only: warranty#1, warranty#3, safety#2 with cutoff: warranty#1, warranty#3 What is the CEO's favourite colour? top-3 only: privacy#1, contact-support#1, safety#1 with cutoff: (nothing: answer 'I do not know') ``` Choose the cutoff by looking at your **gold set**: what scores do correct chunks usually get, and what do unrelated ones get? Score ranges differ between embedding models, so a cutoff must be chosen per model. ### Filtering Noise Candidates **Noise** is a chunk that matched for the wrong reason: a shared common word ("Brightgrid", "lamp"), a navigation line, or a vague, general paragraph. Cutoffs remove much of it. Clean ingestion (Chapter 2) and good chunking remove the rest at the source. ### Dynamic Score Radius Thresholds A fixed cutoff can be too strict for some questions and too loose for others. **Dynamic** thresholds adapt to each result list: - **Relative cutoff:** keep chunks scoring at least, say, 80% of the best one. - **Largest gap ("elbow"):** cut where scores suddenly drop between two neighbours. ```js runnable title="Relative cutoff and largest gap" // Dynamic thresholds: compare scores with the best one, and cut at a big drop. const hits = [ { id: "brightgrid-plus#2", score: 0.81 }, { id: "brightgrid-plus#1", score: 0.77 }, { id: "schedules#1", score: 0.52 }, { id: "energy-use#2", score: 0.5 }, { id: "billing-questions#1", score: 0.47 }, ]; // Rule 1: keep scores within 80% of the best. const best = hits[0].score; const relative = hits.filter((h) => h.score >= 0.8 * best); console.log("within 80% of best:", relative.map((h) => h.id).join(", ")); // Rule 2: cut at the largest gap between neighbouring scores (the "elbow"). let cutAt = hits.length; let biggest = 0; for (let i = 1; i < hits.length; i++) { const gap = hits[i - 1].score - hits[i].score; if (gap > biggest) { biggest = gap; cutAt = i; } } console.log( `largest gap ${biggest.toFixed(2)} after rank ${cutAt}:`, hits .slice(0, cutAt) .map((h) => h.id) .join(", "), ); ``` Expected output: ```text within 80% of best: brightgrid-plus#2, brightgrid-plus#1 largest gap 0.25 after rank 2: brightgrid-plus#2, brightgrid-plus#1 ``` ## Search Execution Flow ### Query Vectorization The cleaned question is turned into a vector with the **same embedding model** used for the chunks (Chapter 3). In keyword search, the "vector" is the list of query terms instead. ### Index Search & Distance Ranking The index returns the nearest chunk ids with their scores, best first. With flat search that is exact. With HNSW or IVF it is approximate (Lesson 10). ### Candidate Payload Extraction Indexes usually return only **ids and scores**. The next step fetches each chunk's **payload**: its text and metadata (title, section, dates, access tags). Fetch only the fields you need: the text for the prompt, metadata for filtering and citations. ### Index Lookup Latency Tracking Each step takes time. Measure it, so you know where to optimize. In most small systems, **embedding the question** is the slowest part of retrieval, and generation is the slowest part overall. ```js runnable title="Steps and their latency budget" // The search flow, step by step, with a latency budget for each step. const steps = [ { step: "1 clean the query", ms: 0.2 }, { step: "2 embed the query", ms: 12 }, { step: "3 search the index", ms: 4 }, { step: "4 rank by distance", ms: 0.3 }, { step: "5 fetch payloads (text, metadata)", ms: 3 }, ]; const budget = 50; let total = 0; for (const s of steps) { total += s.ms; console.log(`${s.step.padEnd(36)} ${String(s.ms).padStart(5)} ms`); } console.log( `${"total".padEnd(36)} ${total.toFixed(1).padStart(5)} ms of a ${budget} ms budget`, ); const slowest = steps.reduce((a, b) => (b.ms > a.ms ? b : a)); console.log("slowest step:", slowest.step); ``` Expected output: ```text 1 clean the query 0.2 ms 2 embed the query 12 ms 3 search the index 4 ms 4 rank by distance 0.3 ms 5 fetch payloads (text, metadata) 3 ms total 19.5 ms of a 50 ms budget slowest step: 2 embed the query ``` Track latency **percentiles**, not just averages. The "p95" is the time 95% of searches beat. A system with a 20 ms average but a 900 ms p95 feels slow to one user in twenty. ## Retrieval Quality Control ### Removing Out-of-Distribution Matches An **out-of-distribution** question is far from everything in your corpus: "write a poem", "what's the weather", a question in another language your corpus does not cover. Its best score will be much lower than usual. Compare the best score with what normal questions get, and treat a very low best score as "no relevant content". ### Deduplicating Identical Vectors Near-identical chunks produce near-identical vectors and scores, so they arrive together and fill your top-k with one fact. Remove duplicates by comparing **normalized text** (or a hash of it), or by checking whether two vectors are almost identical, with a cosine above about 0.98. ```js runnable title="Remove duplicates, flag far-away questions" // Quality control: remove duplicate chunks, and flag questions far from everything. const hits = [ { id: "reset-password#1", text: "Press Forgot password and enter your email.", score: 0.84, }, { id: "forgot-password#1", text: "Press Forgot password and enter your email.", score: 0.84, }, { id: "two-step-sign-in#2", text: "Use one of your 10 backup codes.", score: 0.55 }, ]; const seen = new Set(); const unique = hits.filter((h) => { const key = h.text.toLowerCase().replace(/\s+/g, " ").trim(); if (seen.has(key)) { return false; } seen.add(key); return true; }); console.log("after dedup:", unique.map((h) => h.id).join(", ")); // Out-of-distribution: the best score is far below what normal questions get. const typicalBest = 0.7; for (const [q, bestScore] of [ ["How do I reset my password?", 0.84], ["Write me a poem about the sea", 0.18], ]) { console.log( `${q.padEnd(30)} best ${bestScore} -> ${bestScore < typicalBest / 2 ? "out of distribution: do not answer from these" : "normal"}`, ); } ``` Expected output: ```text after dedup: reset-password#1, two-step-sign-in#2 How do I reset my password? best 0.84 -> normal Write me a poem about the sea best 0.18 -> out of distribution: do not answer from these ``` ### Handling Low-Confidence Query Returns When the best result is weak but not useless, you have options: - **Answer carefully**, and say that the sources may not fully cover the question. - **Ask the user** to clarify: "Do you mean the Lumo Lamp or the Glow Strip?" - **Retry** with a better query (below). ### Fallback Retrieval Mechanisms A **fallback** is a second plan when the first search fails: - Switch search type: if vector search is weak, try keyword search, or the other way round. - **Widen** the scope: include outer-volume sources such as tickets, or remove an optional filter. - **Corrective retrieval:** grade the results, and if they are weak, **rewrite or expand** the query and search again. You will learn more ways to rewrite queries in Lesson 14. - **Hand over** to a person when nothing works. ```js runnable title="Retry with an expanded query" // Fallbacks and corrective retrieval: check the results, and try again if they are weak. const index = { "wifi-troubleshooting#2": "move the lamp closer to the router and use a 2.4 ghz wi-fi network", "hub-pairing#2": "hold the button for 5 seconds to pair a light with the hub", }; const synonyms = { internet: ["wi-fi", "network", "router"], light: ["lamp"] }; function search(q) { const words = q.toLowerCase().match(/[a-z0-9.-]+/g); return Object.entries(index) .map(([id, t]) => ({ id, score: words.filter((w) => t.includes(w)).length / words.length, })) .sort((a, b) => b.score - a.score)[0]; } function retrieve(q) { let best = search(q); console.log(`try 1 "${q}" -> ${best.id} (${best.score.toFixed(2)})`); if (best.score >= 0.4) { return best.id; } // Corrective step: the result looks weak, so expand the query and search again. const expanded = q + " " + q .split(" ") .flatMap((w) => synonyms[w] ?? []) .join(" "); best = search(expanded); console.log(`try 2 "${expanded}" -> ${best.id} (${best.score.toFixed(2)})`); return best.score >= 0.4 ? best.id : "no good match: hand over to a person"; } console.log("result:", retrieve("my light has no internet")); ``` Expected output: ```text try 1 "my light has no internet" -> hub-pairing#2 (0.20) try 2 "my light has no internet lamp wi-fi network router" -> wifi-troubleshooting#2 (0.44) result: wifi-troubleshooting#2 ``` Limit retries, for example to one or two. Each retry adds latency and cost, and a question with no answer in your corpus will not get one by searching five more times. Compare a plain search with one that expands the query, in the lab: ## Common mistakes - **No cutoff**, so unrelated chunks reach the model and get "used". - **A cutoff borrowed from another model**, which removes everything or nothing. - **Fetching full documents** when only the chunk text and a few fields are needed. - **No refusal path** when every score is low. - **Duplicates** that make one fact look like strong agreement between sources. ## Exercises Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer. ## What to Remember - Top-k search: **vectorize the query → search the index → rank by distance → fetch payloads**. Track the latency of each step. - Top-k **always** returns k. Add a **minimum score**, chosen per model from your gold set. - **Dynamic thresholds** (relative to the best score, or the largest gap) adapt to each question. - **Deduplicate** near-identical chunks, and treat unusually low best scores as **out of distribution**. - Plan **fallbacks**: switch search type, widen the scope, **corrective retrieval** with an expanded query, or hand over to a person. Limit retries. Next: **Hybrid Search**, combining keyword and vector search for the best of both.