# Hybrid Search Course: [Retrieval-Augmented Generation](https://www.devtutorials.ai/learn/rag) · Section: Retrieval Engine Source: https://www.devtutorials.ai/learn/rag/retrieval-engine/hybrid-search > Combining keyword and vector search: TF-IDF and BM25, dense search, normalizing and weighting scores, Reciprocal Rank Fusion with k=60, alpha calibration, parallel execution, acronym boosts and exact-match overrides. Think of two friends helping you find a song. One remembers lyrics word for word: say "yellow submarine" and they know it instantly, but describe "that happy Beatles song about a boat" and they're lost. The other remembers the feel of songs: they get the happy-boat description, but mix up songs with similar moods. Ask both, and combine what they say, and you'll find the song almost every time. **Keyword search** is the first friend. **Vector search** is the second. **Hybrid search** asks both and merges their answers. **The main idea:** **sparse** keyword search (TF-IDF and **BM25**) is great for exact words, names, codes and acronyms. **Dense** vector search is great for meaning and paraphrases. Hybrid search runs both, often in parallel, and merges the two result lists, either with **Reciprocal Rank Fusion (RRF)**, which uses ranks only, usually with k=60, or with a **weighted** sum of normalized scores, using a weight called **alpha**. Boosts and exact-match overrides protect technical terms. ## A Real-World Example After Brightgrid switched to vector search, most questions improved, but two kinds got worse: - "**E42** error on my hub": the error code means nothing to the embedding model, so the article about error E42 ranked below general hub articles. - "**BG Hub** reset": "BG" is Brightgrid's internal short name, and the embedding did not connect it to "Brightgrid". Keyword search handled both perfectly, but failed on "my light won't hook up to the internet", which vector search handled well. The team switched to hybrid search: BM25 and vector search run side by side, and RRF merges the lists. They also added an **exact-match override** for error codes and an **acronym list** ("BG" means "Brightgrid"). Both kinds of question now work. **In simple words:** words for precision, meaning for recall, and a fair way to combine them. ## What You Will Learn In this lesson, you will learn: 1. Sparse keyword search: TF-IDF and BM25. 2. Dense vector search, and how to combine sparse and dense scores. 3. Reciprocal Rank Fusion: rank-based merging, the k constant, and weighted alpha calibration. 4. Hybrid pipelines: running both in parallel, keyword boosting for acronyms, exact-match overrides and merging result lists. In the RAG Lab, choose **hybrid** search. It needs the embedding model, which loads from your browser cache if you have it. Under **Retrieval settings**, switch **Hybrid fusion** between **RRF** and **weighted**, and move the **α** slider. Each result shows how it was found, like "keyword+vector". ## Keyword & Vector Combination ### Term Frequency-Inverse Document Frequency (TF-IDF) **TF-IDF** scores how important a word is to a chunk: - **TF (term frequency):** how often the word appears in this chunk. More often means more relevant. - **IDF (inverse document frequency):** how **rare** the word is across all chunks. Words that appear everywhere, like "the" or "Brightgrid", tell you little. Rare words, like "flickering", tell you a lot. TF-IDF = TF × IDF. A query's score for a chunk is the sum over its words. ```js runnable title="TF-IDF for three words" // TF-IDF: a word counts more if it is frequent in this chunk (TF) and rare overall (IDF). const chunks = { a: "the lamp blinks blue when the lamp is ready", b: "the hub pairs up to 50 lights", c: "the lamp uses 9 watts", }; const N = Object.keys(chunks).length; const words = (t) => t.split(" "); const df = (w) => Object.values(chunks).filter((t) => words(t).includes(w)).length; for (const w of ["the", "lamp", "blinks"]) { const tf = words(chunks.a).filter((x) => x === w).length; const idf = Math.log(N / df(w)); console.log( `${w.padEnd(7)} tf ${tf} in ${df(w)} of ${N} chunks idf ${idf.toFixed(2)} tf-idf ${(tf * idf).toFixed(2)}`, ); } ``` Expected output: ```text the tf 2 in 3 of 3 chunks idf 0.00 tf-idf 0.00 lamp tf 2 in 2 of 3 chunks idf 0.41 tf-idf 0.81 blinks tf 1 in 1 of 3 chunks idf 1.10 tf-idf 1.10 ``` ### Sparse Keyword Search (BM25) **BM25** is the standard keyword-ranking formula, used by most search engines. It improves TF-IDF in two ways: - **Saturation (k1):** the 10th time a word appears adds much less than the 2nd. A chunk cannot win just by repeating a word. - **Length normalization (b):** long chunks contain more words by chance, so their scores are scaled down. ```js runnable title="BM25 saturation and length" // BM25 improves TF-IDF in two ways: repeated words give less and less extra score // (saturation, k1), and long chunks are scored down (length normalization, b). const k1 = 1.2; const b = 0.75; const avgLen = 20; const idf = 1.5; function bm25Term(tf, len) { return idf * ((tf * (k1 + 1)) / (tf + k1 * (1 - b + (b * len) / avgLen))); } console.log("word count short chunk (10 words) long chunk (60 words)"); for (const tf of [1, 2, 5, 10]) { console.log( `${String(tf).padStart(10)} ${bm25Term(tf, 10).toFixed(2).padStart(22)} ${bm25Term(tf, 60).toFixed(2).padStart(21)}`, ); } ``` Expected output: ```text word count short chunk (10 words) long chunk (60 words) 1 1.89 0.83 2 2.40 1.32 5 2.87 2.06 10 3.07 2.54 ``` It's called **sparse** because a keyword vector has one slot per word in the vocabulary, and almost all of them are zero. ### Dense Vector Similarity Search **Dense** search compares embeddings (Chapter 3). It finds paraphrases and related wording, but it can miss exact tokens: product codes, error numbers, names, new words the model never saw, and short acronyms. | | Sparse (BM25) | Dense (vectors) | | --- | --- | --- | | Strong at | Exact words, codes, names, acronyms | Meaning, synonyms, paraphrases | | Weak at | Different words for the same idea | Rare tokens, codes, exact phrases | | Needs | An inverted index of words | An embedding model and a vector index | | Explainable | Yes: which words matched | Less: "the vectors were close" | ### Combining Sparse and Dense Scores The two scores live on **different scales**. BM25 scores might run from 0 to 15, cosine from 0 to 1. You cannot simply add them. Two common approaches: 1. **Normalize, then weight.** Rescale each list to 0–1 (for example with min-max), then compute `alpha × vector + (1 − alpha) × keyword`. 2. **Ignore scores, use ranks.** That's RRF, below. ```js runnable title="Weighted fusion with different alpha values" // Combine a keyword list and a vector list. Scores use different scales, // so either normalize them (weighted) or use ranks only (RRF). const keyword = [ { id: "BG-error-E42", score: 9.1 }, { id: "wifi#2", score: 4.2 }, { id: "hub#1", score: 2.0 }, ]; const vector = [ { id: "wifi#2", score: 0.83 }, { id: "wifi#1", score: 0.81 }, { id: "BG-error-E42", score: 0.42 }, ]; function minMax(list) { const s = list.map((h) => h.score); const lo = Math.min(...s); const hi = Math.max(...s); return Object.fromEntries(list.map((h) => [h.id, (h.score - lo) / (hi - lo)])); } function weighted(alpha) { const k = minMax(keyword); const v = minMax(vector); const ids = [...new Set([...keyword, ...vector].map((h) => h.id))]; return ids .map((id) => ({ id, s: alpha * (v[id] ?? 0) + (1 - alpha) * (k[id] ?? 0) })) .sort((a, b) => b.s - a.s) .map((x) => `${x.id} ${x.s.toFixed(2)}`) .join(" | "); } for (const alpha of [0, 0.5, 1]) { console.log(`alpha ${alpha}: ${weighted(alpha)}`); } ``` Expected output: ```text alpha 0: BG-error-E42 1.00 | wifi#2 0.31 | hub#1 0.00 | wifi#1 0.00 alpha 0.5: wifi#2 0.65 | BG-error-E42 0.50 | wifi#1 0.48 | hub#1 0.00 alpha 1: wifi#2 1.00 | wifi#1 0.95 | BG-error-E42 0.00 | hub#1 0.00 ``` At alpha 0 only keywords count, and the error-code article wins. At alpha 1 only vectors count, and it sinks to the bottom. In between, you get a blend. ## Reciprocal Rank Fusion (RRF) ### Rank-Based Score Merging **RRF** gives each chunk `1 / (k + rank)` from each list it appears in, and adds them up. It only looks at **positions**, so it does not care whether scores are BM25 numbers or cosines. A chunk that appears high in **both** lists rises to the top. ### RRF Constant Optimization (k=60) The constant **k** controls how much the very top ranks dominate. With **k = 60** (the value from the original RRF paper, and the usual default), rank 1 and rank 5 score fairly similarly, so agreement between lists matters most. With a small k, rank 1 counts much more. ```js runnable title="RRF with k = 60 and k = 1" // Reciprocal Rank Fusion: score = sum of 1 / (k + rank) over the lists. Only ranks matter. const keyword = ["BG-error-E42", "wifi#2", "hub#1"]; const vector = ["wifi#2", "wifi#1", "BG-error-E42"]; function rrf(lists, k) { const score = {}; for (const list of lists) { list.forEach((id, i) => { score[id] = (score[id] ?? 0) + 1 / (k + i + 1); }); } return Object.entries(score) .sort((a, b) => b[1] - a[1]) .map(([id, s]) => `${id} ${s.toFixed(4)}`) .join(" | "); } console.log("k = 60:", rrf([keyword, vector], 60)); console.log("k = 1: ", rrf([keyword, vector], 1)); console.log( "A chunk found by both lists rises to the top. Small k gives rank 1 much more weight.", ); ``` Expected output: ```text k = 60: wifi#2 0.0325 | BG-error-E42 0.0323 | wifi#1 0.0161 | hub#1 0.0159 k = 1: wifi#2 0.8333 | BG-error-E42 0.7500 | wifi#1 0.3333 | hub#1 0.2500 A chunk found by both lists rises to the top. Small k gives rank 1 much more weight. ``` ### Balancing Exact Matches vs Semantic Matches Both methods let you lean one way: - With **weighted** fusion, raise **alpha** for more meaning, or lower it for more exact words. - With **RRF**, give each list a **weight**, for example `1.0 × keyword + 0.8 × vector`, or retrieve more candidates from one side. ### Weighted Score Alpha Calibration Don't guess alpha. **Calibrate** it with your gold set: try 0, 0.1, …, 1 and keep the value with the best hit rate. Questions full of codes, like a technical support desk, often prefer a lower alpha. Chatty, natural-language questions prefer a higher one. Start with **RRF (k = 60)**. It needs no tuning and handles different score scales safely. Move to weighted fusion only if your gold set shows a clear gain at some alpha. ## Hybrid Search Pipelines ### Parallel Sparse-Dense Execution Keyword and vector search do not depend on each other, so run them **at the same time**. Total latency is then the slower of the two, not their sum. Retrieve a few more candidates than you need from each, for example 2 × k, so the fusion has enough to work with. ### Keyword Boosting for Technical Acronyms Acronyms and short names, like "BG", "2FA" and "SKU", often mean nothing to embedding models. Keep an **acronym list** for your domain, expand acronyms in the query ("BG" becomes "Brightgrid"), and give matches on those exact terms a **boost**. ### Exact String Match Overrides Some strings must win when they match exactly: error codes, order numbers, part numbers, and quoted phrases. Detect them with a pattern, like `E42` or `BG-1042`. If a chunk contains the exact string, move it to the top, or add a large bonus. ```js runnable title="Acronym expansion and an exact-match override" // Exact matches matter for codes, model numbers and acronyms. Boost them, or override. const query = "error E42 on the BG Hub"; const hits = [ { id: "hub-pairing#1", text: "The Brightgrid Hub connects up to 50 lights.", score: 0.78, }, { id: "error-codes#7", text: "Error E42 means the hub lost its radio link.", score: 0.61, }, { id: "wifi#1", text: "Most failures happen on 5 GHz networks.", score: 0.59 }, ]; const codes = query.match(/\b[A-Z]{1,3}\d{1,4}\b/g) ?? []; // like E42 const acronyms = { BG: "Brightgrid" }; const expanded = query .split(" ") .map((w) => acronyms[w] ?? w) .join(" "); console.log("codes found:", codes.join(", "), "| expanded query:", expanded); const boosted = hits .map((h) => { const exact = codes.some((c) => h.text.includes(c)); return { ...h, score: exact ? h.score + 1 : h.score, exact }; }) .sort((a, b) => b.score - a.score); boosted.forEach((h) => console.log( `${h.id.padEnd(15)} ${h.score.toFixed(2)}${h.exact ? " <- exact code match, moved to the top" : ""}`, ), ); ``` Expected output: ```text codes found: E42 | expanded query: error E42 on the Brightgrid Hub error-codes#7 1.61 <- exact code match, moved to the top hub-pairing#1 0.78 wifi#1 0.59 ``` ### Merging Search Result Lists When merging lists, take care to: - **Deduplicate** by chunk id. RRF does this naturally by adding scores per id. - **Keep provenance**: remember which list found each chunk ("keyword", "vector", or both). It is very useful for debugging. - **Apply filters before fusion**, so both lists respect permissions and status. Hybrid search can make debugging harder: when a result is wrong, check each list separately first. The RAG Lab shows how each result was found, like "keyword", "vector" or "keyword+vector", so you can see which side brought it in. Compare the modes on the same question in the lab. Keyword and vector work alone; hybrid merges them: ## Common mistakes - **Adding unnormalized scores**, so BM25's bigger numbers decide everything. - **Too few candidates per list**, so fusion has nothing to merge. - **No exact-match handling** for codes and IDs. - **Calibrating alpha on a handful of questions**, so it does not generalize. - **Different filters on each side**, which leaks restricted chunks through one of the lists. ## Exercises Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer. ## What to Remember - **TF-IDF** rewards words frequent in a chunk and rare overall. **BM25** adds saturation (k1) and length normalization (b). - **Sparse** (keyword) search is strong on exact terms. **Dense** (vector) search is strong on meaning. Hybrid search uses both. - Scores use different scales. **Normalize and weight** with alpha, or use **RRF**, which only uses ranks. Start with **k = 60**. - **Calibrate** alpha or list weights with your gold set. - Run both searches **in parallel**, **expand acronyms**, **boost or override** exact codes, apply filters before fusing, and keep provenance. Next: **Metadata Filtering**, restricting search by region, date, category, tenant and permissions.