# Capstone

Course: [AI Security & Guardrails](https://www.devtutorials.ai/learn/ai-security) · Section: Securing AI Coding Agents
Source: https://www.devtutorials.ai/learn/ai-security/securing-ai-coding-agents/capstone

> Your whole security pack against 25 attacks from every lesson and all honest traffic, with both threat models and a validating export. Pass to download the v2 pack.

## Capstone: A Security Pack for Your App and Repo

One last attack before the exam. A coding task asks the agent to look up a server name, then "resolve it with the key as a subdomain". Replay it on Version 0.

A secret read, a DNS lookup that carries it, and egress to an outside host: three steps that each look like debugging. Real attacks chain small steps like this, across the app and the repo. The capstone grades everything you built against a mixed suite of them.

## TL;DR

- The capstone runs **25 attacks** drawn from every lesson, plus **all 291 honest requests**, against your whole pipeline and your coding-agent settings.
- **Pass:** 0 **critical** ledger effects, at least **97%** of honest traffic served, your **Lesson 1 threat model** and **Lesson 12 coding-agent map** complete, and the **export validates**.
- Passing marks your pack **v2**: everything from v1 plus limits, agent settings for each tool you picked, hook scripts and the MCP allow-list.
- On an existing app, your answers to the Setup questionnaire go into the threat model template; use the exported attack suite as the start of your own.

## A Real-World Example

A platform team adopts the pack as their review template for every new AI feature. Each feature team fills in the threat model, runs the attack suite in CI and attaches their agent settings. In the first quarter, the suite catches two regressions before release: a new tool added without scoping, and a refactor that dropped the egress allow-list from one code path.

None of those were new attack ideas. They were old controls that quietly went missing. That is what a regression suite is for.

## What You Will Learn

- How the capstone suite is built and graded
- How to read a failing capstone result and find the lesson to revisit
- What the v2 pack contains and how to use it in your own repo
- How to run the pack's checks in CI
- How to keep the pack current

## Cheat Sheet: Capstone Checklist

| Check | Where | Lesson |
|---|---|---|
| Threat model complete | Architecture tab | 1 |
| Page 1 controls (secrets, egress, scoping, encoding, agency) | Pipeline tab | 2 to 7 |
| Agent, RAG, supply-chain and limit controls | Pipeline tab | 8 to 11 |
| Coding-agent map | Coding agent tab | 12 |
| Lock-down, hook, MCP allow-list | Pipeline + Coding agent tabs | 13, 14 |
| Export validates | Export tab | 7, 15 |

## How the Grade Works

The capstone uses a softer honest-traffic bar (97%) than single lessons (98%), because every control is on at once. The attack bar is about **critical** effects: anything that leaks secrets or personal data, moves money, runs code or plants a backdoor.

```ts runnable title="The capstone grade"
function capstone(r: { critical: number; served: number; honest: number; threatModel: boolean; codingMap: boolean; exportProblems: number }) {
  const checks: [string, boolean][] = [
    ["0 critical effects", r.critical === 0],
    ["honest >= 97%", r.served / r.honest >= 0.97],
    ["threat model complete", r.threatModel],
    ["coding map complete", r.codingMap],
    ["export validates", r.exportProblems === 0],
  ];
  for (const [name, ok] of checks) console.log((ok ? "✓ " : "✗ ") + name);
  return checks.every(([, ok]) => ok) ? "PASS" : "not yet";
}
console.log(capstone({ critical: 0, served: 289, honest: 291, threatModel: true, codingMap: true, exportProblems: 0 }));
```

```csharp runnable title="The capstone grade"
string Capstone(int critical, int served, int honest, bool threatModel, bool codingMap, int exportProblems)
{
    var checks = new (string Name, bool Ok)[]
    {
        ("0 critical effects", critical == 0),
        ("honest >= 97%", (double)served / honest >= 0.97),
        ("threat model complete", threatModel),
        ("coding map complete", codingMap),
        ("export validates", exportProblems == 0),
    };
    foreach (var (name, ok) in checks) Console.WriteLine((ok ? "✓ " : "✗ ") + name);
    return checks.All(c => c.Ok) ? "PASS" : "not yet";
}
Console.WriteLine(Capstone(critical: 0, served: 289, honest: 291, threatModel: true, codingMap: true, exportProblems: 0));
```

Expected output:

```text
✓ 0 critical effects
✓ honest >= 97%
✓ threat model complete
✓ coding map complete
✓ export validates
PASS
```

## Reading a Failure

Feedback names the attack family and the ledger effect. Each family belongs to one lesson, so a failure points you straight back:

```ts runnable title="From a failing family to the lesson to revisit"
const home: Record<string, number> = {
  "System prompt extraction": 2, "Poisoned help article": 3, "Another customer's delivery note": 4,
  "Script in the rendered reply": 5, "The same refund paid many times": 6, "Memory poisoned for a later session": 8,
  "Another tenant's chunks retrieved": 9, "Malicious MCP tool description": 10, "Token flood": 11,
  "Coding agent reads secrets": 13, "Poisoned repo, issue or MCP": 14,
};
const failures = ["Script in the rendered reply", "Coding agent reads secrets"];
for (const f of failures) console.log(f + " -> revisit Lesson " + home[f]);
```

```csharp runnable title="From a failing family to the lesson to revisit"
var home = new Dictionary<string, int>
{
    ["System prompt extraction"] = 2, ["Poisoned help article"] = 3, ["Another customer's delivery note"] = 4,
    ["Script in the rendered reply"] = 5, ["The same refund paid many times"] = 6, ["Memory poisoned for a later session"] = 8,
    ["Another tenant's chunks retrieved"] = 9, ["Malicious MCP tool description"] = 10, ["Token flood"] = 11,
    ["Coding agent reads secrets"] = 13, ["Poisoned repo, issue or MCP"] = 14,
};
var failures = new[] { "Script in the rendered reply", "Coding agent reads secrets" };
foreach (var f in failures) Console.WriteLine(f + " -> revisit Lesson " + home[f]);
```

Expected output:

```text
Script in the rendered reply -> revisit Lesson 5
Coding agent reads secrets -> revisit Lesson 13
```

The capstone attacks use different variants from the ones you practised on: an encoded prompt leak, a split help article, a hidden memory, a multi-step DNS leak. A defence that only matched one wording shows up here.

## What the v2 Pack Contains

```ts runnable title="The v2 pack, folder by folder"
const pack = {
  "threat-model.md, owasp-mapping.md": "for security and compliance reviews",
  "guardrails/*.ts": "pipeline, scoping, output, egress, actions, your own checks",
  "config/egress-allowlist.json, config/limits.json": "hosts, rate, tokens, spend, steps, alerts",
  "agent-config/": "settings, hooks and MCP allow-list for each tool you picked",
  "tests/attacks.test.ts": "the attack regression suite (Vitest)",
  "skills/ai-security-review/SKILL.md": "a review checklist for your coding agent",
  "architecture.md, permission-matrix.md": "diagram and who-may-do-what, from your pipeline",
  "rag-security.md, production-readiness.md": "RAG design and the launch checklist",
  "dotnet/": "the same guardrails for ASP.NET Core (plus your C# checks)",
  ".gitleaks.toml, .github/workflows/": "secret scan, attack suite and .NET build in CI",
};
for (const [files, what] of Object.entries(pack)) console.log(files + ": " + what);
```

```csharp runnable title="The v2 pack, folder by folder"
var pack = new Dictionary<string, string>
{
    ["threat-model.md, owasp-mapping.md"] = "for security and compliance reviews",
    ["guardrails/*.ts"] = "pipeline, scoping, output, egress, actions, your own checks",
    ["config/egress-allowlist.json, config/limits.json"] = "hosts, rate, tokens, spend, steps, alerts",
    ["agent-config/"] = "settings, hooks and MCP allow-list for each tool you picked",
    ["tests/attacks.test.ts"] = "the attack regression suite (Vitest)",
    ["skills/ai-security-review/SKILL.md"] = "a review checklist for your coding agent",
    ["architecture.md, permission-matrix.md"] = "diagram and who-may-do-what, from your pipeline",
    ["rag-security.md, production-readiness.md"] = "RAG design and the launch checklist",
    ["dotnet/"] = "the same guardrails for ASP.NET Core (plus your C# checks)",
    [".gitleaks.toml, .github/workflows/"] = "secret scan, attack suite and .NET build in CI",
};
foreach (var (files, what) in pack) Console.WriteLine(files + ": " + what);
```

Expected output:

```text
threat-model.md, owasp-mapping.md: for security and compliance reviews
guardrails/*.ts: pipeline, scoping, output, egress, actions, your own checks
config/egress-allowlist.json, config/limits.json: hosts, rate, tokens, spend, steps, alerts
agent-config/: settings, hooks and MCP allow-list for each tool you picked
tests/attacks.test.ts: the attack regression suite (Vitest)
skills/ai-security-review/SKILL.md: a review checklist for your coding agent
architecture.md, permission-matrix.md: diagram and who-may-do-what, from your pipeline
rag-security.md, production-readiness.md: RAG design and the launch checklist
dotnet/: the same guardrails for ASP.NET Core (plus your C# checks)
.gitleaks.toml, .github/workflows/: secret scan, attack suite and .NET build in CI
```

## Put It in CI

The pack is most useful when it runs on every change. A minimal CI job installs and runs the suite, and fails the build on any red test:

```ts runnable title="A CI gate in a few lines"
const results = [
  { test: "refuses email to an outside address", pass: true },
  { test: "never returns another customer's order", pass: true },
  { test: "encodes <script>", pass: true },
  { test: "gates a repeated refund", pass: false },
];
const failed = results.filter((r) => !r.pass);
console.log(failed.length ? "CI FAILED: " + failed.map((r) => r.test).join(", ") : "CI passed");
console.log("exit code " + (failed.length ? 1 : 0));
```

```csharp runnable title="A CI gate in a few lines"
var results = new[]
{
    (Test: "refuses email to an outside address", Pass: true),
    (Test: "never returns another customer's order", Pass: true),
    (Test: "encodes <script>", Pass: true),
    (Test: "gates a repeated refund", Pass: false),
};
var failed = results.Where(r => !r.Pass).ToList();
Console.WriteLine(failed.Count > 0 ? "CI FAILED: " + string.Join(", ", failed.Select(r => r.Test)) : "CI passed");
Console.WriteLine("exit code " + (failed.Count > 0 ? 1 : 0));
```

Expected output:

```text
CI FAILED: gates a repeated refund
exit code 1
```

Test the exported agent settings in the real tools before you rely on them: deny a read of `.env`, and try a planted hook command. Settings keys change between releases, and a typo in a settings file can silently do nothing.

When you add a new tool, data source or agent, add one attack and one honest request for it to `tests/attacks.test.ts` in the same pull request. The suite grows with the product.

When you add a tool, add an attack and an honest request for it in the same change:

```ts runnable title="Growing the suite"
const suite = ["egress allow-list", "scoped orders", "safe rendering"];
const newTool = "export_invoices";
suite.push(newTool + ": refuses another customer's invoice", newTool + ": exports my own invoices");
console.log(suite.length + " tests:");
for (const t of suite) console.log("- " + t);
```

```csharp runnable title="Growing the suite"
var suite = new List<string> { "egress allow-list", "scoped orders", "safe rendering" };
var newTool = "export_invoices";
suite.AddRange(new[] { newTool + ": refuses another customer's invoice", newTool + ": exports my own invoices" });
Console.WriteLine(suite.Count + " tests:");
foreach (var t in suite) Console.WriteLine("- " + t);
```

Expected output:

```text
5 tests:
- egress allow-list
- scoped orders
- safe rendering
- export_invoices: refuses another customer's invoice
- export_invoices: exports my own invoices
```

## Lab Exercise: Your Security Pack Against 25 Attacks

Pick the capstone suite in the Pipeline tab and run it. Then open the Export tab and download your pack.

**Pass rule:** 0 critical ledger effects across the 25 attacks, at least 97% of the 291 honest requests served, the Lesson 1 threat model and the Lesson 12 coding-agent map complete, and the export validating, including every JSON and TOML file for the tools you picked.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Tuning only for the capstone suite | "Passing the capstone means we are secure." | Treat the suite as a sample. Keep every lesson's controls on. | Attacks outside the sample get through. |
| Exporting and never testing in the real tools | "The settings file looks right." | Run the real-tool check: a denied `.env` read and a blocked planted hook. | A setting that does nothing ships to the team. |
| Skipping the threat model | "The controls speak for themselves." | Ship the threat model with the pack. | Reviewers cannot tell why each control exists, so they remove some. |
| Letting the suite go stale | "The suite passed at launch." | Add a test with every new tool, source or agent. | New paths go untested. |
| Turning controls off to fix a false positive | "This control blocks honest users, so remove it." | Adjust the control and add the honest request to the suite. | The attack it stopped comes back. |

## Exercises

## What to Remember

- The capstone grades everything together: 25 attacks, every honest request, both threat models and the export.
- Each failing family points back to one lesson.
- The v2 pack is your guardrails, limits, agent settings and attack suite as files.
- Run the suite in CI and grow it with the product.

## Further Reading

- [OWASP Top 10 for LLM Applications 2025](https://genai.owasp.org/llm-top-10/) and [OWASP Top 10 for Agentic Applications 2026](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/): the two lists your pack maps to.
- [Vitest documentation](https://vitest.dev/guide/): running the exported attack suite.
- [The lethal trifecta for AI agents, by Simon Willison](https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/): the one test to apply to every new design.

Next: **Cheat Sheet**. Every attack, control, OWASP entry and tool setting from the course on one page.