# RAG Evaluation

Course: [Retrieval-Augmented Generation](https://www.devtutorials.ai/learn/rag) · Section: Production
Source: https://www.devtutorials.ai/learn/rag/production/rag-evaluation

> Measuring RAG with numbers: Precision@K, Recall@K, hit rate and MRR; faithfulness, context relevance and citation accuracy; gold chunk matching, test suites, the RAG Triad and LLM-as-judge.

Think about evaluating a restaurant. Asking only “Was the food good?” does not explain what worked or what went wrong. The restaurant needs to check separate areas, such as ingredient quality, food preparation, order accuracy, and service.

A RAG system should also be evaluated in separate parts. Asking only “Is the assistant good?” is too broad to identify the cause of a problem.

RAG evaluation checks areas such as:

Retrieval quality: Did the system find the information needed to answer the question?
Answer faithfulness: Does the answer use only information supported by the retrieved sources?
Citation accuracy: Do the citations point to sources that support the related claims?
Answer relevance: Does the response directly address the user’s question?

Checking each area separately helps the team identify whether a problem comes from retrieval, answer generation, or citation handling.

**The main idea:** measure **retrieval** with **Precision@K**, **Recall@K**, **hit rate** and **MRR** against a gold set's reference chunks. Measure **generation** with **faithfulness** (no claims beyond the context), **context relevance** and **citation accuracy**. Automate it with **grader contracts**, **test suites** and the **RAG Triad**, and use **LLM-as-judge** carefully, checking the judge against human labels.

## A Real-World Example

Brightgrid changed its chunking method, embedding model, and prompt during the same week. After the changes were released, customer complaints increased. Because several parts had changed together, the team could not tell which change caused the problem.

To investigate, the team created an evaluation suite, which is a set of tests used to measure the RAG system.

The suite included:

60 gold questions with expected answers and supporting source chunks
Questions that the knowledge base should refuse to answer
Retrieval tests to check whether the correct chunks appeared in the results
Answer tests to check whether responses were relevant and supported by the sources
Citation tests to check whether each claim pointed to the correct source

The team tested each change separately using the same questions. The results showed that the new chunking method had split some tables into incomplete pieces. This made the correct information harder to retrieve. The new embedding model and prompt were not causing the problem.

The team restored the previous chunking method, improved how tables were handled, and began running the evaluation suite before every release.

**In simple words:** measure each part separately, on the same questions, every time something changes.

## What You Will Learn

In this lesson, you will learn:

1. Retrieval metrics: Precision@K, Recall@K, Mean Reciprocal Rank (MRR) and hit rate.
2. Generation and grounding metrics: answer faithfulness, context relevance and citation accuracy.
3. Grader contracts and outcome-based testing: gold chunk matching, automated test suites, the RAG Triad, and LLM-as-judge.

The RAG Lab's **Corpus check** tab runs the 10-question gold set and now reports **hit@3**, **MRR** and **recall@3** for the answerable questions, plus how many no-answer questions were refused. Switch between keyword, vector and hybrid search, and compare the numbers.

## Retrieval Metrics

### Precision@K and Recall@K

The letter K represents the number of top search results being checked. For example, K = 3 means that the first three retrieved chunks are evaluated.

A relevant chunk is a chunk that contains information needed to answer the question.

Precision@K measures how many of the top K results are relevant. A higher precision means fewer unrelated chunks are sent to the model.
Recall@K measures how many of all the relevant chunks appear within the top K results. A higher recall means the system found more of the information needed for the answer.

```js runnable title="Precision and recall at different k"
// Precision@K and Recall@K for one question.
const relevant = new Set(["returns-policy#1", "returns-policy#3"]); // from the gold set
const retrieved = [
  "returns-policy#1",
  "cancel-order#2",
  "returns-policy#3",
  "shipping-times#1",
  "warranty#1",
];

function precisionAt(k) {
  return retrieved.slice(0, k).filter((id) => relevant.has(id)).length / k;
}
function recallAt(k) {
  return retrieved.slice(0, k).filter((id) => relevant.has(id)).length / relevant.size;
}
for (const k of [1, 3, 5]) {
  console.log(
    `k=${k}: precision ${precisionAt(k).toFixed(2)}, recall ${recallAt(k).toFixed(2)}`,
  );
}
```

Expected output:

```text
k=1: precision 1.00, recall 0.50
k=3: precision 0.67, recall 1.00
k=5: precision 0.40, recall 1.00
```

The two pull against each other: a bigger K usually raises recall and lowers precision. Measure both, at the K you actually send to the model.

### Mean Reciprocal Rank (MRR) & Hit Rate

These metrics check whether the retrieval system finds a relevant chunk and how high that chunk appears in the results.

Hit Rate@K

Hit Rate@K, also called Hit@K, measures the percentage of questions that have at least one relevant chunk within the top K results.

For example, Hit@3 checks whether a relevant chunk appears anywhere in the first three results.

Reciprocal Rank

The reciprocal rank is based on the position of the first relevant chunk:

Position 1 gives a score of 1
Position 2 gives a score of 0.5
Position 3 gives a score of about 0.33
No relevant chunk gives a score of 0
Mean Reciprocal Rank

Mean Reciprocal Rank, or MRR, is the average reciprocal rank across all test questions.

A higher MRR means that the first relevant chunk usually appears closer to the top of the search results.

```js runnable title="Hit rate and MRR on a small gold set"
// Hit rate and MRR over a small gold set.
const runs = [
  {
    q: "return window",
    relevant: ["returns-policy#1"],
    got: ["returns-policy#1", "cancel-order#2", "warranty#1"],
  },
  {
    q: "refund time",
    relevant: ["returns-policy#3"],
    got: ["returns-policy#1", "returns-policy#3", "cancel-order#1"],
  },
  {
    q: "plus price",
    relevant: ["brightgrid-plus#2"],
    got: ["energy-use#2", "schedules#1", "brightgrid-plus#2"],
  },
  {
    q: "wifi fix",
    relevant: ["wifi-troubleshooting#2"],
    got: ["hub-pairing#2", "schedules#1", "lumo-lamp-setup#1"],
  },
];

let hits = 0;
let rrSum = 0;
for (const r of runs) {
  const rank = r.got.findIndex((id) => r.relevant.includes(id)) + 1; // 0 means not found
  hits += rank > 0 ? 1 : 0;
  rrSum += rank > 0 ? 1 / rank : 0;
  console.log(
    `${r.q.padEnd(13)} first relevant at rank ${rank || "-"}  reciprocal ${rank ? (1 / rank).toFixed(2) : "0.00"}`,
  );
}
console.log(
  `hit rate@3 ${(hits / runs.length).toFixed(2)}, MRR ${(rrSum / runs.length).toFixed(2)}`,
);
```

Expected output:

```text
return window first relevant at rank 1  reciprocal 1.00
refund time   first relevant at rank 2  reciprocal 0.50
plus price    first relevant at rank 3  reciprocal 0.33
wifi fix      first relevant at rank -  reciprocal 0.00
hit rate@3 0.75, MRR 0.46
```

## Generation & Grounding Metrics

These metrics check whether the retrieved information is useful and whether the final answer is supported by that information.

### Answer Faithfulness (Hallucination Detection)

Faithfulness, also called groundedness, measures whether the factual claims in an answer are supported by the retrieved context.
For example, if an answer contains five factual claims and four are supported by the retrieved sources, its faithfulness score is: 4 supported claims ÷ 5 factual claims = 0.8
A claim is unfaithful when the retrieved context does not support it. The claim may be correct in the real world, but the RAG system cannot verify it from the sources provided for that answer.

### Context Relevance Scoring

Context relevance measures how closely the retrieved information relates to the user’s question.

High context relevance means that most retrieved chunks help answer the question. Low context relevance means that the prompt contains unrelated or unnecessary information, which may distract the model and increase processing time and cost.

Context relevance can be evaluated even when no expected answer is available, making it useful for checking questions from real users.

```js runnable title="Faithfulness and context relevance"
// Faithfulness: share of answer sentences supported by the context.
// Context relevance: share of retrieved chunks that are about the question.
const context = [
  "You can return any product within 30 days of delivery.",
  "Refunds take 5 business days after the parcel arrives.",
  "The Lumo Lamp uses 9 watts.",
];
const answer = [
  "You can return it within 30 days.",
  "Refunds take 5 business days.",
  "You also get free shipping both ways.",
];
const words = (t) =>
  t
    .toLowerCase()
    .match(/[a-z0-9]+/g)
    .filter((w) => w.length > 2);
const ctxWords = new Set(context.flatMap(words));

const supported = answer.map(
  (s) => words(s).filter((w) => ctxWords.has(w)).length / words(s).length >= 0.6,
);
console.log("supported sentences:", supported.map((x) => (x ? "yes" : "NO")).join(", "));
console.log(
  "faithfulness:",
  (supported.filter(Boolean).length / answer.length).toFixed(2),
);

const qTerms = ["return", "refund", "days"];
const relevantChunks = context.filter((c) =>
  qTerms.some((t) => c.toLowerCase().includes(t)),
);
console.log("context relevance:", (relevantChunks.length / context.length).toFixed(2));
```

Expected output:

```text
supported sentences: yes, yes, NO
faithfulness: 0.67
context relevance: 0.67
```

### Citation Accuracy Verification

**Citation accuracy** asks: does each cited source really contain the fact it's cited for? An answer can be faithful to the context as a whole but cite the wrong chunk, which misleads anyone who clicks the citation. Count correct citations divided by all citations.

```js runnable title="Score citation accuracy"
// Citation accuracy: does each cited source really contain the claim's key facts?
const sources = {
  S1: "Return within 30 days of delivery.",
  S2: "Refunds take 5 business days.",
};
const claims = [
  { text: "Returns are accepted for 30 days", cite: "S1", facts: ["30 days"] },
  { text: "Refunds take 5 business days", cite: "S1", facts: ["5 business days"] },
  { text: "Refunds take 5 business days", cite: "S2", facts: ["5 business days"] },
  { text: "Returns are accepted for 60 days", cite: "S1", facts: ["60 days"] },
];
let correct = 0;
for (const c of claims) {
  const ok = c.facts.every((f) => (sources[c.cite] ?? "").includes(f));
  correct += ok ? 1 : 0;
  console.log(`${ok ? "correct" : "WRONG  "} ${c.cite} for "${c.text}"`);
}
console.log(`citation accuracy: ${(correct / claims.length).toFixed(2)}`);
```

Expected output:

```text
correct S1 for "Returns are accepted for 30 days"
WRONG   S1 for "Refunds take 5 business days"
correct S2 for "Refunds take 5 business days"
WRONG   S1 for "Returns are accepted for 60 days"
citation accuracy: 0.50
```

## Grader Contracts & Outcome-Based Testing

Outcome-based testing checks whether the RAG system produces the expected result. It evaluates both the information retrieved by the system and the final answer given to the user.

### Gold Answer Chunk Matching

A gold set contains prepared test questions. For each question, it records:

The chunks that contain the required information
The important facts expected in the answer
Whether the system should answer or refuse

The expected source chunks are called reference chunks.

Two automatic checks can be performed using this information:

Chunk matching: Checks whether the reference chunks appear within the top retrieved results. This helps measure whether the retrieval system found the required information.

Answer matching: Checks whether the generated answer contains the expected facts. Exact checks are useful for important values such as names, dates, prices, and quantities.

For example, if the expected answer says that a product can be returned within 30 days, the test can check whether the generated answer contains that value.

More complex qualities, such as whether the answer correctly explains a condition or exception, may require a model-based grader or human review.

### Automated Test Case Suites

An automated test suite is a collection of tests that checks the RAG system whenever something changes.

Run the same tests after changing the documents, chunking method, embedding model, retrieval settings, prompt, or language model.

A useful test suite includes:

Test groups: Organise questions into categories such as simple facts, multi-part questions, questions with no answer, misleading questions, and boundary cases.
Quality thresholds: Set the minimum acceptable score for important metrics. If a new version falls below a threshold, review it before release.
Test history: Save the results together with the corpus version and system settings. This allows the team to compare different test runs fairly.

### LLM-as-Judge

Some qualities, such as "is this sentence supported?" or "does this answer the question?", are hard to check with simple rules. An **LLM-as-judge** asks a strong model to grade answers. Make it reliable with a **grader contract**:

- A fixed **rubric** with clear criteria and a small scale (like 1–3).
- A fixed **output format** (JSON), so scores can be parsed and averaged.
- The judge sees **only what it needs**: question, context and answer.
- **Temperature 0**, and the same judge model and prompt for every comparison.

Then **check the judge**: label a sample of answers by hand, and measure how often the judge agrees. If agreement is low, fix the rubric before trusting the scores.

```js runnable title="A grader contract, judge agreement and the RAG triad"
// LLM-as-judge with a grader contract, checked against human labels.
// The contract: a fixed rubric and a fixed JSON output, so scores are comparable.
const rubric = {
  scale: [1, 2, 3],
  criteria: {
    groundedness: "every claim is supported by the context",
    answerRelevance: "the answer addresses the question",
  },
  output: '{"groundedness": 1-3, "answerRelevance": 1-3, "reason": "one sentence"}',
};
console.log(
  "criteria:",
  Object.keys(rubric.criteria).join(", "),
  "| output:",
  rubric.output,
);

// Judge scores vs human scores on six answers (groundedness).
const judge = [3, 3, 1, 2, 3, 1];
const human = [3, 2, 1, 2, 3, 1];
const agree = judge.filter((s, i) => s === human[i]).length / judge.length;
console.log(`judge agrees with humans on ${Math.round(agree * 100)}% of answers`);

// The RAG triad for one question: context relevance, groundedness, answer relevance.
const triad = { contextRelevance: 0.67, groundedness: 0.67, answerRelevance: 1.0 };
const weakest = Object.entries(triad).sort((a, b) => a[1] - b[1])[0];
console.log(
  "triad:",
  Object.entries(triad)
    .map(([k, v]) => `${k} ${v.toFixed(2)}`)
    .join(", "),
  "| weakest:",
  weakest[0],
);
```

Expected output:

```text
criteria: groundedness, answerRelevance | output: {"groundedness": 1-3, "answerRelevance": 1-3, "reason": "one sentence"}
judge agrees with humans on 83% of answers
triad: contextRelevance 0.67, groundedness 0.67, answerRelevance 1.00 | weakest: contextRelevance
```

### RAG Triad Metrics (Relevance, Groundedness, Answer Relevance)

The **RAG Triad** checks the three links between question, context and answer:

| Link | Metric | A low score means |
| --- | --- | --- |
| Question → context | **Context relevance** | Retrieval brought the wrong or noisy chunks |
| Context → answer | **Groundedness** (faithfulness) | The model added unsupported claims |
| Answer → question | **Answer relevance** | The answer is off-topic or incomplete |

The weakest link tells you where to work: retrieval, prompting or the model.

Start small and real: 30–60 gold questions taken from actual user logs beat 1,000 synthetic ones. Add every production failure to the gold set. Within months, your suite covers the questions that actually matter.

Judges have biases: they may prefer longer answers, answers that sound confident, or text written in their own style. Never compare systems with **different** judges or rubrics, and re-check judge agreement whenever you change the judge model.

## Common mistakes

- **Changing several things at once**, so you can't tell which one helped or hurt.
- **Only measuring final answers**, so retrieval problems look like model problems.
- **Gold sets that drift**: chunk ids change after re-chunking and nobody updates the references.
- **Judges without contracts**, which give inconsistent scores.
- **Ignoring precision**: high recall at k = 20 means little if the model only sees the top 3.

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer.

## What to Remember

- **Retrieval metrics**: **Precision@K** (noise), **Recall@K** (coverage), **hit rate** (found at all), **MRR** (found early). They are computed from the gold set's **reference chunks**.
- **Generation metrics**: **faithfulness** (claims supported by context), **context relevance** (context about the question), **citation accuracy** (each citation correct).
- The **RAG Triad**: context relevance, groundedness and answer relevance. The weakest link shows where to work.
- Use **LLM-as-judge** with a **grader contract** (rubric, JSON, temperature 0), and check it against **human labels**.
- Run an **automated suite** with groups and thresholds on **every change**, and record the snapshot and settings with each run.

Next: **Production Operations**, tracing, monitoring, maintenance and caching in a live system.