AI Security & Guardrails tutorial

Attack an AI feature in a safe range, then add guardrails that really hold: architecture, permissions and checks outside the model, not just prompt rules.

Start the course

Course contents

Securing AI Features

  1. Threat Modelling — Every token the model reads is a possible instruction. Draw the trust boundaries of an AI feature, spot the lethal trifecta, and map each risk to the OWASP Top 10 for LLM apps.
  2. Direct Injection & Jailbreaks — One customer message prints the system prompt and its secrets. See why prompt rules and classifiers leak, then stop every variant with architecture, policy in code and an output check, without blocking real customers.
  3. Indirect Injection — A poisoned help-centre article makes the agent email customer data out, and the customer never sees it. Label untrusted text, take capabilities away and add an egress allow-list until no variant gets data out.
  4. Sensitive Data & PII — The agent shows one customer another customer's order and writes it to the logs. Move scoping out of the prompt and into the tool layer, filter other people's details from replies, and redact the logs.
  5. Improper Output Handling — A review runs a script in the next reader's browser, and a report request leaks every customer's rows. Encode what the chat renders and replace model-written SQL with a validated filter and parameters.
  6. Excessive Agency — The agent pays a $420 refund nobody approved and gives a 100% discount on request. Remove the tools the job does not need, scope the rest and gate refunds in code, while valid refunds still go through.
  7. Guardrail Pipeline — Put every control from Lessons 2 to 6 into one pipeline, measure it on mixed traffic with two numbers, choose a classifier threshold from its curve, and export your v1 security pack with an attack suite for Vitest.

Agents and Production

  1. Agentic Risks — A memory planted in one session leaks data in the next, a subagent obeys a ticket, and a loop runs 500 tool calls. Add memory provenance, handoff checks and a step cap that honest tasks fit under.
  2. RAG & Vector Weaknesses — Poisoned community posts answer policy questions and a vector search returns another customer's ticket. Quarantine untrusted documents at ingestion and filter every search by tenant.
  3. Supply Chain — An MCP tool description tells the agent to email conversations out, and a model invents a package name an attacker registered. Pin MCP servers to reviewed descriptions and check packages before install.
  4. Unbounded Consumption & Monitoring — A 60,000-token message, a request burst, a spend run-up and a runaway loop. Size per-user limits above real use, check cost before the call, and alert on attack signatures in traces.

Securing AI Coding Agents

  1. Coding-Agent Threat Model — Your coding agent reads issues, comments, READMEs and MCP descriptions, holds your secrets and has a shell. Map what it reads, runs and sends, and find the lethal trifecta.
  2. Permissions & Sandboxing — The agent reads .env and curls it out. Deny secret reads, sandbox the shell, allow-list the network and commands, so build, test and install still work, in each tool's real settings files.
  3. Poisoned Repos, Issues & MCP — A README installs a backdoor, an issue asks for your .env and a repo ships its own MCP server. Block risky actions with PreToolUse hooks and allow-list project MCP servers.
  4. Capstone — Your whole security pack against 25 attacks from every lesson and all honest traffic, with both threat models and a validating export. Pass to download the v2 pack.

Reference

  1. Cheat Sheet — Every attack, control, OWASP entry, linter rule and coding-agent setting from the AI Security course on one page.