Corpus Engineering

By the Syncfusion DevTutorials team · Last updated October 20, 2026

Choosing and cleaning the documents a RAG system searches: document and ticket corpora, core set vs outer volume, near-duplicates, outdated contradictions, unanswerable questions, frozen snapshots, gold sets and reference chunks.

Think about a library. If the shelves contain repeated books, outdated editions, and unchecked notes, even a skilled librarian may find the wrong information.

A well-organised library keeps trusted books, removes unnecessary copies, marks old editions clearly, and checks whether important subjects are covered.

A RAG system needs the same kind of care.

The complete collection of documents that a RAG system can search is called a corpus. Corpus engineering means selecting, organising, checking, and maintaining those documents so the system can retrieve reliable information.

This work includes selecting trusted documents, removing duplicate or outdated information, and checking for missing or conflicting answers.

A snapshot is a saved version of the corpus used for testing. A gold set contains test questions, expected answers, and supporting sources. Together, they help the team measure whether changes improve the RAG system.

The main idea: choose a small, trusted core set first and treat everything else as outer volume. Find and fix near-duplicates, outdated documents that contradict current ones, and questions the corpus can't answer. Then freeze a snapshot and build a gold set of questions with known answers and the exact chunks that hold them, so you can measure every change.

A Real-World Example

Brightgrid added its help-centre articles and five years of support tickets to its RAG system. The team expected more documents to produce better answers, but the quality became worse.

Several problems appeared:

An old support ticket contained a previous 14-day return policy, so the assistant sometimes gave outdated information. Two articles contained almost the same password instructions. Both appeared in the search results and pushed out other useful information. When customers asked questions that the documents could not answer, the system still retrieved unrelated content.

The problem was not the language model. The searchable documents needed to be organised more carefully.

The team kept current help-centre articles as the main trusted collection, searched old support tickets only when needed, combined duplicate content, and marked outdated documents as archived.

The team also created a gold set of 30 test questions with known answers and sources. This allowed the team to check whether future changes improved or reduced the quality of the system.

In simple words: clean shelves first, then measure.

What You Will Learn

In this lesson, you will learn:

1. How to choose and structure a corpus: document collections, ticket corpora, and the core set vs outer volume.
2. How to handle quality problems: near-duplicates, outdated vs current contradictions, and unanswerable or out-of-domain questions.
3. How to freeze a corpus snapshot, define a gold answer set, and map each answer to its reference chunks.

Corpus Selection & Structure

Document Collections & Support Ticket Corpora

Most RAG projects mix two kinds of sources:

• Document collections: help articles, policies, manuals, product pages, wikis. They are written on purpose, reviewed, and meant to be true.
• Ticket and conversation corpora: support tickets, chat logs, emails, forum threads. They are written fast, by many people, about one customer's case. They're rich in real problems and real wording, but they can be wrong, outdated or about an exception.

Both are useful. Tickets show how customers actually describe problems ("the lamp blinks blue but the app never finds it"), which helps search. But a ticket is evidence of what happened once, not of what the rule is.

Core Knowledge Set vs Outer Volume

Split your corpus into two layers:

Split core from outer volume

// Core knowledge set vs outer volume: what to index first.
const corpus = [
  { id: "returns-policy", kind: "article", owner: "support team", reviewed: true },
  { id: "shipping-times", kind: "article", owner: "support team", reviewed: true },
  { id: "warranty", kind: "article", owner: "legal", reviewed: true },
  { id: "ticket-4812", kind: "ticket", owner: null, reviewed: false },
  { id: "ticket-4907", kind: "ticket", owner: null, reviewed: false },
  { id: "ticket-5001", kind: "ticket", owner: null, reviewed: false },
  { id: "forum-thread-77", kind: "forum", owner: null, reviewed: false },
];

// Core: owned, reviewed, the source of truth. Outer: everything else.
const core = corpus.filter((d) => d.owner && d.reviewed);
const outer = corpus.filter((d) => !(d.owner && d.reviewed));

console.log(`core  (${core.length}):`, core.map((d) => d.id).join(", "));
console.log(`outer (${outer.length}):`, outer.map((d) => d.id).join(", "));
console.log(
  "Rule: answers should cite core documents. Outer volume can add examples, not policy.",
);

Expected output:

core  (3): returns-policy, shipping-times, warranty
outer (4): ticket-4812, ticket-4907, ticket-5001, forum-thread-77
Rule: answers should cite core documents. Outer volume can add examples, not policy.

Start with the core set. Get it right, measure it, and only then add outer volume where it clearly helps. A small, clean corpus beats a huge, messy one.

Now ask the same question with the support tickets included. Ticket 3120, from 2023, says returns are only accepted within 14 days. Watch it push its way into the sources.

Managing Corpus Quality & Noise

Handling Near-Duplicate Documents

Near-duplicates are documents that contain almost the same information with small differences.

For example, a knowledge base may contain:

Two copies of the same help article An old and a new version of a policy The same answer repeated across several support tickets

Near-duplicates can reduce retrieval quality. They may take up several search-result positions, leaving less room for other useful content. If the copies contain slightly different information, the model may also receive conflicting details.

One way to identify near-duplicates is to compare short, overlapping groups of words called shingles. For example, the text "forgot password and email" can produce groups such as "forgot password", "password and", and "and email".

Jaccard similarity measures how many of these word groups two documents share. The score ranges from:

0, when the documents share no word groups 1, when the compared word groups are identical

Documents with a high score can be reviewed and then merged, removed, or marked as different versions.

Find near-duplicates

// Near-duplicates: compare documents by their overlapping 2-word phrases (shingles).
const docs = {
  "reset-password":
    "on the sign-in screen press forgot password and enter your email we send a reset link that works for 30 minutes",
  "forgot-password":
    "on the sign-in screen choose forgot password and type your email address a reset link is sent to you and works for 30 minutes",
  "two-step-sign-in":
    "open account then security then two-step sign-in you can use an authenticator app",
};

function shingles(text, n = 2) {
  const w = text.split(" ");
  const set = new Set();
  for (let i = 0; i + n <= w.length; i++) {
    set.add(w.slice(i, i + n).join(" "));
  }
  return set;
}

// Jaccard similarity: shared phrases divided by all distinct phrases.
function jaccard(a, b) {
  const shared = [...a].filter((x) => b.has(x)).length;
  return shared / (a.size + b.size - shared);
}

const ids = Object.keys(docs);
for (let i = 0; i < ids.length; i++) {
  for (let j = i + 1; j < ids.length; j++) {
    const s = jaccard(shingles(docs[ids[i]]), shingles(docs[ids[j]]));
    console.log(
      `${ids[i]} vs ${ids[j]}: ${s.toFixed(2)}${s > 0.3 ? "  <- near-duplicate" : ""}`,
    );
  }
}

Expected output:

reset-password vs forgot-password: 0.34  <- near-duplicate
reset-password vs two-step-sign-in: 0.00
forgot-password vs two-step-sign-in: 0.00

Once you find them, merge them into one document, delete the extra copy, or keep both but deduplicate at search time so only one makes the top k.

Outdated vs Current Document Contradictions

Old documents rarely disappear. Policies get new versions, products get renamed, and old tickets stay in the archive. When an old and a new document disagree, RAG may retrieve either one, and the model has no way to know which is true.

Three defenses work well together:

1. Label every document with an updated date and a status (current or archived). You'll learn about metadata in Chapter 2.
2. Filter archived documents out of search by default, like the RAG Lab does.
3. Detect conflicts: find documents on the same topic that state different facts, and have a person decide which one is right.

Detect and resolve a contradiction

// Outdated vs current: two documents disagree. Detect it, then prefer the current one.
const docs = [
  {
    id: "returns-policy-2023",
    updated: "2023-02-10",
    status: "archived",
    text: "Products can be returned within 14 days.",
  },
  {
    id: "returns-policy",
    updated: "2026-03-01",
    status: "current",
    text: "You can return any product within 30 days of delivery.",
  },
];

const days = (t) => Number((t.match(/(\d+) days/) ?? [])[1]);
const values = docs.map((d) => ({
  id: d.id,
  days: days(d.text),
  updated: d.updated,
  status: d.status,
}));
console.log(
  values.map((v) => `${v.id}: ${v.days} days (${v.updated}, ${v.status})`).join("\n"),
);

const conflict = new Set(values.map((v) => v.days)).size > 1;
console.log("conflict:", conflict);

// Resolution rule: current beats archived, then the newest date wins.
const winner = [...values].sort(
  (a, b) =>
    (a.status === "current" ? -1 : 1) - (b.status === "current" ? -1 : 1) ||
    b.updated.localeCompare(a.updated),
)[0];
console.log("use:", winner.id, "->", winner.days, "days");

Expected output:

returns-policy-2023: 14 days (2023-02-10, archived)
returns-policy: 30 days (2026-03-01, current)
conflict: true
use: returns-policy -> 30 days

With Include archived articles on, the 2023 policy competes with the current one. Compare the sources with and without it.

Unanswerable & Out-of-Domain Queries

Not every question has an answer in your corpus:

• Unanswerable: about your domain, but the corpus doesn't say. "What is Brightgrid's stock price?" is about Brightgrid, but no help article covers it.
• Out of domain: not about your domain at all. "Write me a poem about cats."

The danger is that search always returns something, its best guess, even when nothing is relevant. Then the model is tempted to answer from it. Corpus work helps in two ways. First, you list the questions users really ask that the corpus can't answer, and either add content or decide to refuse. Second, you can measure how much of a question's vocabulary appears in the corpus at all:

Is the question even covered?

// Unanswerable and out-of-domain questions: check if the corpus even talks about it.
const corpusText = [
  "return any product within 30 days of delivery",
  "standard shipping takes 3 to 5 business days",
  "every lamp has a 2 year warranty",
  "brightgrid plus costs 3.99 a month",
].join(" ");
const vocab = new Set(corpusText.split(" "));
const STOP = new Set([
  "what",
  "is",
  "the",
  "how",
  "do",
  "does",
  "a",
  "of",
  "for",
  "to",
  "i",
  "my",
  "can",
  "you",
]);

function coverage(question) {
  const words = (question.toLowerCase().match(/[a-z0-9]+/g) ?? []).filter(
    (w) => !STOP.has(w),
  );
  const known = words.filter((w) => vocab.has(w));
  return {
    known: known.length,
    total: words.length,
    ratio: known.length / Math.max(1, words.length),
  };
}

for (const q of [
  "How long is the lamp warranty?",
  "What is the stock price of Brightgrid?",
  "Can you write me a poem about cats?",
]) {
  const c = coverage(q);
  const label =
    c.ratio >= 0.5
      ? "in domain"
      : c.ratio > 0
        ? "probably unanswerable"
        : "out of domain";
  console.log(`${q.padEnd(40)} ${c.known}/${c.total} known words -> ${label}`);
}

Expected output:

How long is the lamp warranty?           2/3 known words -> in domain
What is the stock price of Brightgrid?   1/3 known words -> probably unanswerable
Can you write me a poem about cats?      0/5 known words -> out of domain

This simple check is a rough signal, not a final answer. Chapter 4 adds score thresholds and Chapter 6 adds refusal rules.

Corpus Freezing & Gold Standards

Freezing Corpus Snapshots

You can't measure progress on a target that keeps moving. If articles change while you're testing a new chunking method, you won't know whether a better score came from your change or from the edited articles.

A frozen snapshot is a copy of the corpus at one moment, with a fingerprint: a short code computed from the exact text. If even one character changes, the fingerprint changes. Every test result is then recorded together with the snapshot fingerprint it was measured on.

Gold Answer Set Definition

A gold set is a list of test questions, each with:

• The question, written the way real users ask it.
• The correct answer, written down by a person who knows the topic.
• The reference chunks that contain the answer (see below).

A good gold set mixes easy questions, tricky wording, questions that need two chunks, and unanswerable questions where the correct result is a refusal. Start with 20 to 50 questions. That's enough to catch most problems.

Reference Chunk Mapping

For each gold question, write down the chunk ids that hold the answer, like returns-policy#1. This lets you test retrieval on its own: did the right chunk appear in the top k? You don't need a model, and the check is exact and deterministic.

Freeze, then test with a gold set

// Freeze a snapshot, then test retrieval against a gold set.
const snapshot = {
  "returns-policy#1": "You can return any product within 30 days of delivery.",
  "shipping-times#1": "Standard shipping takes 3 to 5 business days.",
  "warranty#1": "Every lamp has a 2-year warranty against defects.",
};

// A fingerprint (FNV-1a hash) of the exact text. If anything changes, it changes.
function fingerprint(obj) {
  let h = 2166136261;
  for (const ch of JSON.stringify(obj)) {
    h = Math.imul(h ^ ch.charCodeAt(0), 16777619) >>> 0;
  }
  return h.toString(16).padStart(8, "0");
}
console.log(
  "snapshot",
  fingerprint(snapshot),
  "with",
  Object.keys(snapshot).length,
  "chunks",
);

// The gold set: questions, their correct answer, and the chunk that holds it.
const gold = [
  { q: "how many days to return", answer: "30 days", chunk: "returns-policy#1" },
  {
    q: "how long does standard shipping take",
    answer: "3 to 5 business days",
    chunk: "shipping-times#1",
  },
  { q: "what is the stock price", answer: null, chunk: null },
];

const retrieve = (q) =>
  Object.keys(snapshot).find(
    (id) =>
      q.split(" ").filter((w) => w.length > 3 && snapshot[id].toLowerCase().includes(w))
        .length >= 2,
  ) ?? null;

for (const g of gold) {
  const got = retrieve(g.q);
  console.log(
    `${got === g.chunk ? "PASS" : "FAIL"}  "${g.q}" -> ${got ?? "nothing"} (want ${g.chunk ?? "nothing"})`,
  );
}

Expected output:

snapshot 6b295e3c with 3 chunks
PASS  "how many days to return" -> returns-policy#1 (want returns-policy#1)
PASS  "how long does standard shipping take" -> shipping-times#1 (want shipping-times#1)
PASS  "what is the stock price" -> nothing (want nothing)

Run the lab's gold set and freeze a snapshot. Then edit an article in the Knowledge base tab and come back: the fingerprint will no longer match.

Common mistakes

• More data equals better answers. Extra noisy documents often lower accuracy.
• Trusting tickets as rules. A ticket describes one case, often an exception, sometimes a mistake.
• No archive status. Without it, retrieval has no way to prefer current documents.
• Gold sets with only easy questions. Real users ask vague, misspelled and unanswerable questions.
• Forgetting to re-map reference chunks after changing chunking, so the gold set quietly reports wrong results.

Exercises

Each exercise ends with a value on the last line. Press Check to compare it with the expected answer.

What to Remember

• The corpus sets the ceiling for answer quality.
• Keep a small, reviewed core set as the source of truth. Outer volume like tickets adds examples, not policy.
• Find near-duplicates with shingles and Jaccard similarity, then merge or deduplicate them.
• Label documents with dates and status, filter archived ones, and flag contradictions for a person to resolve.
• Expect unanswerable and out-of-domain questions, and plan for them.
• Freeze a snapshot with a fingerprint, and keep a gold set of questions, answers and reference chunks to measure every change.

Next: Chapter 2, Document Ingestion, turning PDFs, Word files and web pages into clean text.