# Corpus Engineering

Course: [Retrieval-Augmented Generation](https://www.devtutorials.ai/learn/rag) · Section: RAG Foundations
Source: https://www.devtutorials.ai/learn/rag/foundations/corpus-engineering

> Choosing and cleaning the documents a RAG system searches: document and ticket corpora, core set vs outer volume, near-duplicates, outdated contradictions, unanswerable questions, frozen snapshots, gold sets and reference chunks.

Think about a library. If the shelves contain repeated books, outdated editions, and unchecked notes, even a skilled librarian may find the wrong information.

A well-organised library keeps trusted books, removes unnecessary copies, marks old editions clearly, and checks whether important subjects are covered.

A RAG system needs the same kind of care.

The complete collection of documents that a RAG system can search is called a corpus. Corpus engineering means selecting, organising, checking, and maintaining those documents so the system can retrieve reliable information.

This work includes selecting trusted documents, removing duplicate or outdated information, and checking for missing or conflicting answers.

A snapshot is a saved version of the corpus used for testing. A gold set contains test questions, expected answers, and supporting sources. Together, they help the team measure whether changes improve the RAG system.

**The main idea:** choose a small, trusted **core** set first and treat everything else as **outer volume**. Find and fix **near-duplicates**, **outdated** documents that contradict current ones, and questions the corpus **can't answer**. Then **freeze** a snapshot and build a **gold set** of questions with known answers and the exact chunks that hold them, so you can measure every change.

## A Real-World Example

Brightgrid added its help-centre articles and five years of support tickets to its RAG system. The team expected more documents to produce better answers, but the quality became worse.

Several problems appeared:

An old support ticket contained a previous 14-day return policy, so the assistant sometimes gave outdated information.
Two articles contained almost the same password instructions. Both appeared in the search results and pushed out other useful information.
When customers asked questions that the documents could not answer, the system still retrieved unrelated content.

The problem was not the language model. The searchable documents needed to be organised more carefully.

The team kept current help-centre articles as the main trusted collection, searched old support tickets only when needed, combined duplicate content, and marked outdated documents as archived.

The team also created a gold set of 30 test questions with known answers and sources. This allowed the team to check whether future changes improved or reduced the quality of the system.

**In simple words:** clean shelves first, then measure.

## What You Will Learn

In this lesson, you will learn:

1. How to choose and structure a corpus: document collections, ticket corpora, and the core set vs outer volume.
2. How to handle quality problems: near-duplicates, outdated vs current contradictions, and unanswerable or out-of-domain questions.
3. How to freeze a corpus snapshot, define a gold answer set, and map each answer to its reference chunks.

The **RAG Lab** now holds two kinds of documents: 24 help-center **articles** and 6 solved support **tickets**. Its new **Corpus check** tab finds near-duplicates and conflicts, freezes a snapshot, and runs a 10-question gold set against the search.

## Corpus Selection & Structure

### Document Collections & Support Ticket Corpora

Most RAG projects mix two kinds of sources:

- **Document collections:** help articles, policies, manuals, product pages, wikis. They are written on purpose, reviewed, and meant to be **true**.
- **Ticket and conversation corpora:** support tickets, chat logs, emails, forum threads. They are written fast, by many people, about **one customer's case**. They're rich in real problems and real wording, but they can be wrong, outdated or about an exception.

Both are useful. Tickets show how customers actually describe problems ("the lamp blinks blue but the app never finds it"), which helps search. But a ticket is evidence of **what happened once**, not of **what the rule is**.

### Core Knowledge Set vs Outer Volume

Split your corpus into two layers:

| | Core knowledge set | Outer volume |
| --- | --- | --- |
| What | Reviewed, owned, current documents | Everything else: tickets, chats, old wikis, forums |
| Examples | Help center, official policies | Ticket 4812, a forum thread |
| Trust | The source of truth | Helpful examples, never policy |
| Size | Small (tens to thousands) | Large (thousands to millions) |
| Default in search | Always on | Off, or ranked lower |

```js runnable title="Split core from outer volume"
// Core knowledge set vs outer volume: what to index first.
const corpus = [
  { id: "returns-policy", kind: "article", owner: "support team", reviewed: true },
  { id: "shipping-times", kind: "article", owner: "support team", reviewed: true },
  { id: "warranty", kind: "article", owner: "legal", reviewed: true },
  { id: "ticket-4812", kind: "ticket", owner: null, reviewed: false },
  { id: "ticket-4907", kind: "ticket", owner: null, reviewed: false },
  { id: "ticket-5001", kind: "ticket", owner: null, reviewed: false },
  { id: "forum-thread-77", kind: "forum", owner: null, reviewed: false },
];

// Core: owned, reviewed, the source of truth. Outer: everything else.
const core = corpus.filter((d) => d.owner && d.reviewed);
const outer = corpus.filter((d) => !(d.owner && d.reviewed));

console.log(`core  (${core.length}):`, core.map((d) => d.id).join(", "));
console.log(`outer (${outer.length}):`, outer.map((d) => d.id).join(", "));
console.log(
  "Rule: answers should cite core documents. Outer volume can add examples, not policy.",
);
```

Expected output:

```text
core  (3): returns-policy, shipping-times, warranty
outer (4): ticket-4812, ticket-4907, ticket-5001, forum-thread-77
Rule: answers should cite core documents. Outer volume can add examples, not policy.
```

**Start with the core set.** Get it right, measure it, and only then add outer volume where it clearly helps. A small, clean corpus beats a huge, messy one.

Now ask the same question with the support tickets included. Ticket 3120, from 2023, says returns are only accepted within 14 days. Watch it push its way into the sources.

## Managing Corpus Quality & Noise

### Handling Near-Duplicate Documents
Near-duplicates are documents that contain almost the same information with small differences.

For example, a knowledge base may contain:

Two copies of the same help article
An old and a new version of a policy
The same answer repeated across several support tickets

Near-duplicates can reduce retrieval quality. They may take up several search-result positions, leaving less room for other useful content. If the copies contain slightly different information, the model may also receive conflicting details.

One way to identify near-duplicates is to compare short, overlapping groups of words called shingles. For example, the text "forgot password and email" can produce groups such as "forgot password", "password and", and "and email".

Jaccard similarity measures how many of these word groups two documents share. The score ranges from:

0, when the documents share no word groups
1, when the compared word groups are identical

Documents with a high score can be reviewed and then merged, removed, or marked as different versions.

```js runnable title="Find near-duplicates"
// Near-duplicates: compare documents by their overlapping 2-word phrases (shingles).
const docs = {
  "reset-password":
    "on the sign-in screen press forgot password and enter your email we send a reset link that works for 30 minutes",
  "forgot-password":
    "on the sign-in screen choose forgot password and type your email address a reset link is sent to you and works for 30 minutes",
  "two-step-sign-in":
    "open account then security then two-step sign-in you can use an authenticator app",
};

function shingles(text, n = 2) {
  const w = text.split(" ");
  const set = new Set();
  for (let i = 0; i + n <= w.length; i++) {
    set.add(w.slice(i, i + n).join(" "));
  }
  return set;
}

// Jaccard similarity: shared phrases divided by all distinct phrases.
function jaccard(a, b) {
  const shared = [...a].filter((x) => b.has(x)).length;
  return shared / (a.size + b.size - shared);
}

const ids = Object.keys(docs);
for (let i = 0; i < ids.length; i++) {
  for (let j = i + 1; j < ids.length; j++) {
    const s = jaccard(shingles(docs[ids[i]]), shingles(docs[ids[j]]));
    console.log(
      `${ids[i]} vs ${ids[j]}: ${s.toFixed(2)}${s > 0.3 ? "  <- near-duplicate" : ""}`,
    );
  }
}
```

Expected output:

```text
reset-password vs forgot-password: 0.34  <- near-duplicate
reset-password vs two-step-sign-in: 0.00
forgot-password vs two-step-sign-in: 0.00
```

Once you find them, **merge** them into one document, **delete** the extra copy, or keep both but **deduplicate** at search time so only one makes the top k.

### Outdated vs Current Document Contradictions

Old documents rarely disappear. Policies get new versions, products get renamed, and old tickets stay in the archive. When an old and a new document disagree, RAG may retrieve either one, and the model has no way to know which is true.

Three defenses work well together:

1. **Label every document** with an `updated` date and a `status` (current or archived). You'll learn about metadata in Chapter 2.
2. **Filter** archived documents out of search by default, like the RAG Lab does.
3. **Detect conflicts:** find documents on the same topic that state different facts, and have a person decide which one is right.

```js runnable title="Detect and resolve a contradiction"
// Outdated vs current: two documents disagree. Detect it, then prefer the current one.
const docs = [
  {
    id: "returns-policy-2023",
    updated: "2023-02-10",
    status: "archived",
    text: "Products can be returned within 14 days.",
  },
  {
    id: "returns-policy",
    updated: "2026-03-01",
    status: "current",
    text: "You can return any product within 30 days of delivery.",
  },
];

const days = (t) => Number((t.match(/(\d+) days/) ?? [])[1]);
const values = docs.map((d) => ({
  id: d.id,
  days: days(d.text),
  updated: d.updated,
  status: d.status,
}));
console.log(
  values.map((v) => `${v.id}: ${v.days} days (${v.updated}, ${v.status})`).join("\n"),
);

const conflict = new Set(values.map((v) => v.days)).size > 1;
console.log("conflict:", conflict);

// Resolution rule: current beats archived, then the newest date wins.
const winner = [...values].sort(
  (a, b) =>
    (a.status === "current" ? -1 : 1) - (b.status === "current" ? -1 : 1) ||
    b.updated.localeCompare(a.updated),
)[0];
console.log("use:", winner.id, "->", winner.days, "days");
```

Expected output:

```text
returns-policy-2023: 14 days (2023-02-10, archived)
returns-policy: 30 days (2026-03-01, current)
conflict: true
use: returns-policy -> 30 days
```

"Newest wins" is a good default, but not a law. Sometimes a newer document is a draft, a regional exception, or simply a mistake. Automatic rules should **flag** conflicts. A person who owns the content should **resolve** them.

With **Include archived articles** on, the 2023 policy competes with the current one. Compare the sources with and without it.

### Unanswerable & Out-of-Domain Queries

Not every question has an answer in your corpus:

- **Unanswerable:** about your domain, but the corpus doesn't say. "What is Brightgrid's stock price?" is about Brightgrid, but no help article covers it.
- **Out of domain:** not about your domain at all. "Write me a poem about cats."

The danger is that search **always returns something**, its best guess, even when nothing is relevant. Then the model is tempted to answer from it. Corpus work helps in two ways. First, you **list** the questions users really ask that the corpus can't answer, and either add content or decide to refuse. Second, you can measure how much of a question's vocabulary appears in the corpus at all:

```js runnable title="Is the question even covered?"
// Unanswerable and out-of-domain questions: check if the corpus even talks about it.
const corpusText = [
  "return any product within 30 days of delivery",
  "standard shipping takes 3 to 5 business days",
  "every lamp has a 2 year warranty",
  "brightgrid plus costs 3.99 a month",
].join(" ");
const vocab = new Set(corpusText.split(" "));
const STOP = new Set([
  "what",
  "is",
  "the",
  "how",
  "do",
  "does",
  "a",
  "of",
  "for",
  "to",
  "i",
  "my",
  "can",
  "you",
]);

function coverage(question) {
  const words = (question.toLowerCase().match(/[a-z0-9]+/g) ?? []).filter(
    (w) => !STOP.has(w),
  );
  const known = words.filter((w) => vocab.has(w));
  return {
    known: known.length,
    total: words.length,
    ratio: known.length / Math.max(1, words.length),
  };
}

for (const q of [
  "How long is the lamp warranty?",
  "What is the stock price of Brightgrid?",
  "Can you write me a poem about cats?",
]) {
  const c = coverage(q);
  const label =
    c.ratio >= 0.5
      ? "in domain"
      : c.ratio > 0
        ? "probably unanswerable"
        : "out of domain";
  console.log(`${q.padEnd(40)} ${c.known}/${c.total} known words -> ${label}`);
}
```

Expected output:

```text
How long is the lamp warranty?           2/3 known words -> in domain
What is the stock price of Brightgrid?   1/3 known words -> probably unanswerable
Can you write me a poem about cats?      0/5 known words -> out of domain
```

This simple check is a rough signal, not a final answer. Chapter 4 adds score thresholds and Chapter 6 adds refusal rules.

## Corpus Freezing & Gold Standards

### Freezing Corpus Snapshots

You can't measure progress on a target that keeps moving. If articles change while you're testing a new chunking method, you won't know whether a better score came from your change or from the edited articles.

A **frozen snapshot** is a copy of the corpus at one moment, with a **fingerprint**: a short code computed from the exact text. If even one character changes, the fingerprint changes. Every test result is then recorded together with the snapshot fingerprint it was measured on.

### Gold Answer Set Definition

A **gold set** is a list of test questions, each with:

- The **question**, written the way real users ask it.
- The **correct answer**, written down by a person who knows the topic.
- The **reference chunks** that contain the answer (see below).

A good gold set mixes easy questions, tricky wording, questions that need two chunks, and **unanswerable** questions where the correct result is a refusal. Start with 20 to 50 questions. That's enough to catch most problems.

### Reference Chunk Mapping

For each gold question, write down the **chunk ids** that hold the answer, like `returns-policy#1`. This lets you test **retrieval on its own**: did the right chunk appear in the top k? You don't need a model, and the check is exact and deterministic.

```js runnable title="Freeze, then test with a gold set"
// Freeze a snapshot, then test retrieval against a gold set.
const snapshot = {
  "returns-policy#1": "You can return any product within 30 days of delivery.",
  "shipping-times#1": "Standard shipping takes 3 to 5 business days.",
  "warranty#1": "Every lamp has a 2-year warranty against defects.",
};

// A fingerprint (FNV-1a hash) of the exact text. If anything changes, it changes.
function fingerprint(obj) {
  let h = 2166136261;
  for (const ch of JSON.stringify(obj)) {
    h = Math.imul(h ^ ch.charCodeAt(0), 16777619) >>> 0;
  }
  return h.toString(16).padStart(8, "0");
}
console.log(
  "snapshot",
  fingerprint(snapshot),
  "with",
  Object.keys(snapshot).length,
  "chunks",
);

// The gold set: questions, their correct answer, and the chunk that holds it.
const gold = [
  { q: "how many days to return", answer: "30 days", chunk: "returns-policy#1" },
  {
    q: "how long does standard shipping take",
    answer: "3 to 5 business days",
    chunk: "shipping-times#1",
  },
  { q: "what is the stock price", answer: null, chunk: null },
];

const retrieve = (q) =>
  Object.keys(snapshot).find(
    (id) =>
      q.split(" ").filter((w) => w.length > 3 && snapshot[id].toLowerCase().includes(w))
        .length >= 2,
  ) ?? null;

for (const g of gold) {
  const got = retrieve(g.q);
  console.log(
    `${got === g.chunk ? "PASS" : "FAIL"}  "${g.q}" -> ${got ?? "nothing"} (want ${g.chunk ?? "nothing"})`,
  );
}
```

Expected output:

```text
snapshot 6b295e3c with 3 chunks
PASS  "how many days to return" -> returns-policy#1 (want returns-policy#1)
PASS  "how long does standard shipping take" -> shipping-times#1 (want shipping-times#1)
PASS  "what is the stock price" -> nothing (want nothing)
```

Re-check reference chunks whenever you change **chunking**. A new chunking method gives new chunk ids, so `returns-policy#1` might now be `returns-policy#2`. Map the gold set to chunk text or to document and section, not only to chunk numbers.

Run the lab's gold set and freeze a snapshot. Then edit an article in the **Knowledge base** tab and come back: the fingerprint will no longer match.

## Common mistakes

- **More data equals better answers.** Extra noisy documents often lower accuracy.
- **Trusting tickets as rules.** A ticket describes one case, often an exception, sometimes a mistake.
- **No archive status.** Without it, retrieval has no way to prefer current documents.
- **Gold sets with only easy questions.** Real users ask vague, misspelled and unanswerable questions.
- **Forgetting to re-map reference chunks** after changing chunking, so the gold set quietly reports wrong results.

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer.

## What to Remember

- The **corpus** sets the ceiling for answer quality.
- Keep a small, reviewed **core set** as the source of truth. **Outer volume** like tickets adds examples, not policy.
- Find **near-duplicates** with shingles and Jaccard similarity, then merge or deduplicate them.
- Label documents with **dates and status**, filter **archived** ones, and **flag** contradictions for a person to resolve.
- Expect **unanswerable** and **out-of-domain** questions, and plan for them.
- **Freeze** a snapshot with a fingerprint, and keep a **gold set** of questions, answers and **reference chunks** to measure every change.

Next: **Chapter 2, Document Ingestion**, turning PDFs, Word files and web pages into clean text.