Engineering Knowledge Assistant

By the Syncfusion DevTutorials team · Last updated October 19, 2026

Building an internal RAG assistant over engineering docs for UI/UX, frontend, backend, security and testing: what to index, team-based access, secret redaction, the gold set in CI and end-to-end UI tests.

Engineering teams often store useful information across many places, including design-system guides, API documentation, security procedures, and testing instructions. As a result, developers may struggle to find answers to common questions, such as:

Which colour should be used for an error message? How does pagination work in a list API? What should a developer do after accidentally committing a secret key?

An internal RAG assistant can search the organisation’s engineering documents and provide an answer using the most relevant information. This helps developers find existing guidance without searching through several websites and documents.

However, the assistant must be managed carefully. Some documents, such as operational runbooks, may have restricted access. Documents may also contain passwords, secret keys, or other sensitive information that should not be retrieved. Changes to documents, chunking, or retrieval settings can also reduce answer quality over time.

The main idea: build an engineering knowledge assistant with the same pipeline as a customer assistant, plus extra care for internal content. Index the design system, frontend, backend, security and testing docs; apply team-based access; redact secrets before indexing; run the gold set in CI so quality changes block merges; and test the chat UI end to end. Developers get fast, cited answers about UI, frontend, backend, security and testing, without leaking what they shouldn't see.

A Real-World Example

Brightgrid’s platform team creates a RAG assistant that searches the company’s Engineering Handbook.

During the first week, the assistant helps the team in several ways:

A frontend developer asks which colour token should be used for error messages. The assistant answers color-danger and provides a link to the supporting design-system page. A backend developer asks how list endpoints should handle pagination. An old guide recommends page numbers, while the current guide recommends cursors. Because archived documents are excluded from retrieval, the assistant uses the current guide. An old setup document contains a sandbox API key. The ingestion process removes the key before indexing the document, so the assistant cannot retrieve or reveal it. A team member changes the chunk size. The automated evaluation shows that Hit@3 falls from 0.90 to 0.82, which means the correct information appears less often in the first three results. The quality check blocks the change before it is released.

In simple words: An internal RAG assistant helps developers find everyday technical information. To use it safely, the organisation must protect restricted content, remove secrets, exclude outdated documents, and test every important change.

What You Will Learn

In this lesson, you will learn:

1. What to index for an engineering assistant across the stack.
2. How to apply team-based access to internal docs.
3. How to redact secrets before indexing.
4. How to run the gold set in CI.
5. How to test the chat interface end to end.

What to Index

Route developer questions to handbook docs

// Which handbook doc answers each developer question? (keyword retrieval)
const docs = {
  "design-tokens": "color tokens primary danger error text spacing radius",
  "accessibility-standard": "wcag contrast label placeholder keyboard focus button",
  "api-conventions": "rest paths plural errors status pagination cursor limit",
  "secrets-handling": "secrets keys tokens rotate leak git history",
  "testing-strategy": "unit integration playwright endpoints tests ci flaky",
};
const questions = [
  "Which token is used for error text?",
  "How do list endpoints paginate with a cursor?",
  "What if a key leaks into git?",
  "Which tests do endpoints need in CI?",
];
const words = (t) => t.toLowerCase().match(/[a-z]+/g);
for (const q of questions) {
  const ranked = Object.entries(docs)
    .map(([id, text]) => [id, words(q).filter((w) => text.split(" ").includes(w)).length])
    .sort((a, b) => b[1] - a[1]);
  console.log(`${q.padEnd(46)} -> ${ranked[0][0]}`);
}

Expected output:

Which token is used for error text?            -> design-tokens
How do list endpoints paginate with a cursor?  -> api-conventions
What if a key leaks into git?                  -> secrets-handling
Which tests do endpoints need in CI?           -> testing-strategy

The second answer comes from api-conventions (cursors), not the archived api-pagination-v1 (page numbers). Tick Include archived articles in the Ask tab and ask again to see the conflict the status filter prevents.

Team-Based Access

Not every engineering doc is for everyone. Security incident playbooks, on-call runbooks with infrastructure details, and unreleased designs may be restricted:

Filter docs by team

// Team-based access: some engineering docs are restricted.
const docs = [
  { id: "api-conventions", teams: ["all"] },
  { id: "on-call-runbook", teams: ["platform", "sre"] },
  { id: "security-incident-playbook", teams: ["security"] },
  { id: "design-tokens", teams: ["all"] },
];
function visibleTo(user) {
  return docs
    .filter((d) => d.teams.includes("all") || d.teams.some((t) => user.teams.includes(t)))
    .map((d) => d.id);
}
console.log("frontend dev:", visibleTo({ teams: ["web"] }).join(", "));
console.log("SRE:         ", visibleTo({ teams: ["sre"] }).join(", "));

Expected output:

frontend dev: api-conventions, design-tokens
SRE:          api-conventions, on-call-runbook, design-tokens

The rules from the previous lesson apply: teams come from the session, filters run before retrieval, and caches are keyed by permission set.

Secrets in Docs

Engineering docs often contain things that shouldn't be there: sample keys, connection strings, passwords in setup steps. Redact them at ingestion, so they never reach chunks, vectors, prompts or answers:

Redact secrets before indexing

// Redact secrets before indexing, so they can never be retrieved or quoted.
const page = `To call the billing sandbox use key acme_live_4f9XyZ81QwErTy00.
Ask #payments for the production key. DB password: hunter2hunter2`;

const redacted = page
  .replace(/acme_live_[A-Za-z0-9]{8,}/g, "[REDACTED_KEY]")
  .replace(/(password:\s*)\S+/gi, "$1[REDACTED]");
console.log(redacted);
console.log(/acme_live_|hunter2/.test(redacted) ? "LEAK" : "safe to index");

Expected output:

To call the billing sandbox use key [REDACTED_KEY].
Ask #payments for the production key. DB password: [REDACTED]
safe to index

• Run a secret scanner over every document before chunking.
• Replace matches with a marker such as [REDACTED_KEY], and alert the doc owner.
• Remember the rule from Secrets handling: a leaked secret must be rotated, not just removed.

Quality in CI

The Gold Set as a Test

Treat the gold set as a test suite. Any change to prompts, chunking, embeddings or the index runs it, and a meaningful drop blocks the merge:

Fail CI when quality drops

// The gold set as a CI test: fail the build if retrieval quality drops.
const baseline = { hitAt3: 0.9, refusals: 1.0 };
const thisRun = { hitAt3: 0.82, refusals: 1.0 };
const maxDrop = 0.05;

const drop = baseline.hitAt3 - thisRun.hitAt3;
console.log(`hit@3: ${baseline.hitAt3} -> ${thisRun.hitAt3} (drop ${drop.toFixed(2)})`);
console.log(`refusals: ${baseline.refusals} -> ${thisRun.refusals}`);
const ok = drop <= maxDrop && thisRun.refusals >= baseline.refusals;
console.log(
  ok
    ? "CI: pass"
    : "CI: FAIL - block the merge and check the latest chunking or prompt change",
);

Expected output:

hit@3: 0.9 -> 0.82 (drop 0.08)
refusals: 1 -> 1
CI: FAIL - block the merge and check the latest chunking or prompt change

Press Run the gold set in Corpus check. Eight questions: seven answerable ones across UI, frontend, backend, security and testing, and one ("What is the office address?") that must be refused.

Testing the Chat UI

Retrieval tests don't catch interface bugs: chips that don't render, a refusal shown as an error, a Stop button that does nothing. Add a few end-to-end journeys with a browser testing tool such as Playwright:

End-to-end journeys as data

// End-to-end checks for the chat UI, written as data a Playwright test could run.
const journeys = [
  {
    ask: "Which color token should error messages use?",
    expect: { textIncludes: "color-danger", chips: 1 },
  },
  { ask: "What is the office address?", expect: { refused: true, chips: 0 } },
];
const fakeUi = {
  "Which color token should error messages use?": {
    text: "Use color-danger (#B91C1C) for errors.",
    chips: 1,
    refused: false,
  },
  "What is the office address?": {
    text: "I don't know based on the engineering handbook.",
    chips: 0,
    refused: true,
  },
};
for (const j of journeys) {
  const ui = fakeUi[j.ask];
  const ok =
    (j.expect.textIncludes ? ui.text.includes(j.expect.textIncludes) : true) &&
    ui.chips === j.expect.chips &&
    ui.refused === Boolean(j.expect.refused);
  console.log(`${ok ? "PASS" : "FAIL"} ${j.ask}`);
}

Expected output:

PASS Which color token should error messages use?
PASS What is the office address?

Good journeys: one answerable question per area, one refusal, one restricted-doc question for a user without access, and one network-error case.

Common mistakes

• Indexing the whole wiki without tags. Restricted pages become answers for everyone.
• Keeping archived guides searchable. Old conventions come back as answers.
• Secrets in indexed docs. Redact at ingestion and rotate leaked secrets.
• No gold set in CI. Quality drops silently with every "small" change.
• Testing retrieval but not the UI. Add end-to-end journeys.
• No owner per doc. Answers go stale when nobody updates the source.

Check Your Understanding

Exercises

Each exercise ends with a value on the last line. Press Check to compare it with the expected answer.

What to Remember

• An engineering assistant answers everyday UI/UX, frontend, backend, security and testing questions from internal docs.
• Filter out archived guidance so old conventions don't return.
• Apply team-based access from the session, with default-deny for untagged docs.
• Redact secrets at ingestion, and rotate anything that leaked.
• Run the gold set in CI and block merges on quality drops.
• Add end-to-end UI tests for chips, refusals and errors.

This completes the Development Workflows chapter and the RAG course. Return to the RAG Reference page any time for formulas, defaults and checklists.