# Engineering Knowledge Assistant

Course: [Retrieval-Augmented Generation](https://www.devtutorials.ai/learn/rag) · Section: Development Workflows
Source: https://www.devtutorials.ai/learn/rag/dev-workflows/engineering-knowledge-assistant

> Building an internal RAG assistant over engineering docs for UI/UX, frontend, backend, security and testing: what to index, team-based access, secret redaction, the gold set in CI and end-to-end UI tests.

Engineering teams often store useful information across many places, including design-system guides, API documentation, security procedures, and testing instructions. As a result, developers may struggle to find answers to common questions, such as:

Which colour should be used for an error message?
How does pagination work in a list API?
What should a developer do after accidentally committing a secret key?

An internal RAG assistant can search the organisation’s engineering documents and provide an answer using the most relevant information. This helps developers find existing guidance without searching through several websites and documents.

However, the assistant must be managed carefully. Some documents, such as operational runbooks, may have restricted access. Documents may also contain passwords, secret keys, or other sensitive information that should not be retrieved. Changes to documents, chunking, or retrieval settings can also reduce answer quality over time.

**The main idea:** build an **engineering knowledge assistant** with the same pipeline as a customer assistant, plus extra care for internal content. Index the **design system, frontend, backend, security and testing** docs; apply **team-based access**; **redact secrets** before indexing; run the **gold set in CI** so quality changes block merges; and test the **chat UI end to end**. Developers get fast, cited answers about UI, frontend, backend, security and testing, without leaking what they shouldn't see.

## A Real-World Example

Brightgrid’s platform team creates a RAG assistant that searches the company’s Engineering Handbook.

During the first week, the assistant helps the team in several ways:

A frontend developer asks which colour token should be used for error messages. The assistant answers color-danger and provides a link to the supporting design-system page.
A backend developer asks how list endpoints should handle pagination. An old guide recommends page numbers, while the current guide recommends cursors. Because archived documents are excluded from retrieval, the assistant uses the current guide.
An old setup document contains a sandbox API key. The ingestion process removes the key before indexing the document, so the assistant cannot retrieve or reveal it.
A team member changes the chunk size. The automated evaluation shows that Hit@3 falls from 0.90 to 0.82, which means the correct information appears less often in the first three results. The quality check blocks the change before it is released.

In simple words:
 An internal RAG assistant helps developers find everyday technical information. To use it safely, the organisation must protect restricted content, remove secrets, exclude outdated documents, and test every important change.

## What You Will Learn

In this lesson, you will learn:

1. What to index for an engineering assistant across the stack.
2. How to apply team-based access to internal docs.
3. How to redact secrets before indexing.
4. How to run the gold set in CI.
5. How to test the chat interface end to end.

Switch the RAG Lab's knowledge base to **Engineering handbook** (top of the lab). It holds 13 internal docs: design tokens, accessibility, UX states, frontend components, data fetching, API conventions, a retired pagination doc, database guidelines, secrets handling, auth rules, testing strategy, assistant quality checks and an on-call runbook. Every tab works with it, including the gold set in **Corpus check**.

## What to Index

| Area | Docs | Example question |
| --- | --- | --- |
| **UI/UX design** | Design tokens, accessibility standard, UX states | Which token should error messages use? |
| **Frontend** | Component conventions, data fetching, streaming | How should streamed answers be rendered? |
| **Backend** | API conventions, database guidelines | How should list endpoints paginate? |
| **Security** | Secrets handling, auth rules | What do I do if an API key leaks? |
| **Testing** | Testing strategy, assistant quality checks | Which tests does a new endpoint need? |
| **Operations** | Runbooks | What should I check if errors spike? |

```js runnable title="Route developer questions to handbook docs"
// Which handbook doc answers each developer question? (keyword retrieval)
const docs = {
  "design-tokens": "color tokens primary danger error text spacing radius",
  "accessibility-standard": "wcag contrast label placeholder keyboard focus button",
  "api-conventions": "rest paths plural errors status pagination cursor limit",
  "secrets-handling": "secrets keys tokens rotate leak git history",
  "testing-strategy": "unit integration playwright endpoints tests ci flaky",
};
const questions = [
  "Which token is used for error text?",
  "How do list endpoints paginate with a cursor?",
  "What if a key leaks into git?",
  "Which tests do endpoints need in CI?",
];
const words = (t) => t.toLowerCase().match(/[a-z]+/g);
for (const q of questions) {
  const ranked = Object.entries(docs)
    .map(([id, text]) => [id, words(q).filter((w) => text.split(" ").includes(w)).length])
    .sort((a, b) => b[1] - a[1]);
  console.log(`${q.padEnd(46)} -> ${ranked[0][0]}`);
}
```

Expected output:

```text
Which token is used for error text?            -> design-tokens
How do list endpoints paginate with a cursor?  -> api-conventions
What if a key leaks into git?                  -> secrets-handling
Which tests do endpoints need in CI?           -> testing-strategy
```

The second answer comes from `api-conventions` (cursors), not the archived `api-pagination-v1` (page numbers). Tick **Include archived articles** in the Ask tab and ask again to see the conflict the status filter prevents.

## Team-Based Access

Not every engineering doc is for everyone. Security incident playbooks, on-call runbooks with infrastructure details, and unreleased designs may be restricted:

```js runnable title="Filter docs by team"
// Team-based access: some engineering docs are restricted.
const docs = [
  { id: "api-conventions", teams: ["all"] },
  { id: "on-call-runbook", teams: ["platform", "sre"] },
  { id: "security-incident-playbook", teams: ["security"] },
  { id: "design-tokens", teams: ["all"] },
];
function visibleTo(user) {
  return docs
    .filter((d) => d.teams.includes("all") || d.teams.some((t) => user.teams.includes(t)))
    .map((d) => d.id);
}
console.log("frontend dev:", visibleTo({ teams: ["web"] }).join(", "));
console.log("SRE:         ", visibleTo({ teams: ["sre"] }).join(", "));
```

Expected output:

```text
frontend dev: api-conventions, design-tokens
SRE:          api-conventions, on-call-runbook, design-tokens
```

The rules from the previous lesson apply: teams come from the **session**, filters run **before retrieval**, and caches are keyed by permission set.

Internal does not mean safe. An assistant that every employee can use, built over every page of the wiki, will happily answer contractors' questions from a restricted runbook. Tag documents with teams at ingestion, and default to restricted when a document has no tag.

## Secrets in Docs

Engineering docs often contain things that shouldn't be there: sample keys, connection strings, passwords in setup steps. Redact them **at ingestion**, so they never reach chunks, vectors, prompts or answers:

```js runnable title="Redact secrets before indexing"
// Redact secrets before indexing, so they can never be retrieved or quoted.
const page = `To call the billing sandbox use key acme_live_4f9XyZ81QwErTy00.
Ask #payments for the production key. DB password: hunter2hunter2`;

const redacted = page
  .replace(/acme_live_[A-Za-z0-9]{8,}/g, "[REDACTED_KEY]")
  .replace(/(password:\s*)\S+/gi, "$1[REDACTED]");
console.log(redacted);
console.log(/acme_live_|hunter2/.test(redacted) ? "LEAK" : "safe to index");
```

Expected output:

```text
To call the billing sandbox use key [REDACTED_KEY].
Ask #payments for the production key. DB password: [REDACTED]
safe to index
```

- Run a secret scanner over every document before chunking.
- Replace matches with a marker such as `[REDACTED_KEY]`, and alert the doc owner.
- Remember the rule from **Secrets handling**: a leaked secret must be **rotated**, not just removed.

## Quality in CI

### The Gold Set as a Test

Treat the gold set as a test suite. Any change to prompts, chunking, embeddings or the index runs it, and a meaningful drop blocks the merge:

```js runnable title="Fail CI when quality drops"
// The gold set as a CI test: fail the build if retrieval quality drops.
const baseline = { hitAt3: 0.9, refusals: 1.0 };
const thisRun = { hitAt3: 0.82, refusals: 1.0 };
const maxDrop = 0.05;

const drop = baseline.hitAt3 - thisRun.hitAt3;
console.log(`hit@3: ${baseline.hitAt3} -> ${thisRun.hitAt3} (drop ${drop.toFixed(2)})`);
console.log(`refusals: ${baseline.refusals} -> ${thisRun.refusals}`);
const ok = drop <= maxDrop && thisRun.refusals >= baseline.refusals;
console.log(
  ok
    ? "CI: pass"
    : "CI: FAIL - block the merge and check the latest chunking or prompt change",
);
```

Expected output:

```text
hit@3: 0.9 -> 0.82 (drop 0.08)
refusals: 1 -> 1
CI: FAIL - block the merge and check the latest chunking or prompt change
```

Press **Run the gold set** in Corpus check. Eight questions: seven answerable ones across UI, frontend, backend, security and testing, and one ("What is the office address?") that must be refused.

Store the gold set next to the code, review changes to it like code, and print the failing questions in the CI log. A red build that says "lost: How should list endpoints paginate?" is fixed in minutes; one that only says "hit@3 0.82" takes hours.

## Testing the Chat UI

Retrieval tests don't catch interface bugs: chips that don't render, a refusal shown as an error, a Stop button that does nothing. Add a few **end-to-end** journeys with a browser testing tool such as Playwright:

```js runnable title="End-to-end journeys as data"
// End-to-end checks for the chat UI, written as data a Playwright test could run.
const journeys = [
  {
    ask: "Which color token should error messages use?",
    expect: { textIncludes: "color-danger", chips: 1 },
  },
  { ask: "What is the office address?", expect: { refused: true, chips: 0 } },
];
const fakeUi = {
  "Which color token should error messages use?": {
    text: "Use color-danger (#B91C1C) for errors.",
    chips: 1,
    refused: false,
  },
  "What is the office address?": {
    text: "I don't know based on the engineering handbook.",
    chips: 0,
    refused: true,
  },
};
for (const j of journeys) {
  const ui = fakeUi[j.ask];
  const ok =
    (j.expect.textIncludes ? ui.text.includes(j.expect.textIncludes) : true) &&
    ui.chips === j.expect.chips &&
    ui.refused === Boolean(j.expect.refused);
  console.log(`${ok ? "PASS" : "FAIL"} ${j.ask}`);
}
```

Expected output:

```text
PASS Which color token should error messages use?
PASS What is the office address?
```

Good journeys: one answerable question per area, one refusal, one restricted-doc question for a user without access, and one network-error case.

## Common mistakes

- **Indexing the whole wiki without tags.** Restricted pages become answers for everyone.
- **Keeping archived guides searchable.** Old conventions come back as answers.
- **Secrets in indexed docs.** Redact at ingestion and rotate leaked secrets.
- **No gold set in CI.** Quality drops silently with every "small" change.
- **Testing retrieval but not the UI.** Add end-to-end journeys.
- **No owner per doc.** Answers go stale when nobody updates the source.

## Check Your Understanding

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer.

## What to Remember

- An **engineering assistant** answers everyday UI/UX, frontend, backend, security and testing questions from internal docs.
- Filter out **archived** guidance so old conventions don't return.
- Apply **team-based access** from the session, with **default-deny** for untagged docs.
- **Redact secrets at ingestion**, and rotate anything that leaked.
- Run the **gold set in CI** and block merges on quality drops.
- Add **end-to-end UI tests** for chips, refusals and errors.

This completes the Development Workflows chapter and the RAG course. Return to the **RAG Reference** page any time for formulas, defaults and checklists.