# Reliability and Operations Course: [Model Context Protocol](https://www.devtutorials.ai/learn/mcp) ยท Section: Production Source: https://www.devtutorials.ai/learn/mcp/production/reliability-and-operations > Running MCP in production: classifying failures, error frames, safe retries, versioning and deprecation, health checks, audit logs, emergency disable, evaluation, cost and an improvement loop. A delivery company doesn't buy trucks and hope they'll keep running. It checks them each morning, plans for flat tyres, records each trip, and can take a truck off the road when a problem appears. At the end of the month, it reviews which routes ran late and looks for ways to improve them. Production MCP needs the same kind of care. Servers can go down, hosts and servers can support different protocol versions, a model can choose the wrong tool, and costs can rise. Operations work helps teams notice these problems, find their cause, and limit their impact. **The main idea:** recognize common failures, retry only calls that are safe to repeat, manage protocol and tool versions carefully, check server health, keep useful audit logs, provide an emergency disable switch, and regularly evaluate tool choices and cost. ## A Real-World Example On a Monday morning, Brightgrid's support assistant started answering, "I can't look that up right now," to every order question. The team investigated: 1. The helpdesk server had been updated over the weekend and now supported only protocol version `2026-07-28`. 2. One older host still requested `2025-11-25`. Its requests failed with error `-32022`, which means the two sides had no shared protocol version. 3. The host showed a friendly message but recorded no useful details, so the team needed three hours to find the cause. The team improved the rollout and monitoring: - The server keeps supporting the **previous version** for a while after an upgrade. - A **health check** calls `server/discover` regularly from each host and alerts when there is no shared version. - Logs include the **error code**, method, and server name for each failure. This incident shows why operations need both prevention and visibility: supporting older versions reduces disruption, while health checks and useful logs help find problems sooner. **In simple words:** expect failures, notice them quickly, and make them easier to fix. ## What You Will Learn In this lesson, you will learn: 1. How to classify common failures: connectivity, capability mismatch, bad arguments, and tool errors. 2. How to read error frames and choose when to retry. 3. How to version protocol features and tools without surprising hosts. 4. How to monitor health, keep audit logs, disable a tool, and apply least privilege in operations. 5. How to evaluate tool choice and track cost. 6. How to use a repeatable improvement workflow. ## Common Failures When an MCP interaction fails, start by identifying what kind of failure it is. Different failures need different fixes: a network problem needs a connectivity check, while a tool error may need different arguments or a fix in the tool itself. ### Four Kinds of Problems | Kind | What you see | Usual cause | | --- | --- | --- | | **Connectivity** | Connection refused, timeout, no tools listed | Server not running, wrong command, wrong URL, firewall | | **Capability mismatch** | `-32601` method not found, `-32021` missing client capability, `-32022` unsupported version | One side doesn't support what the other expects | | **Bad arguments** | A result with `isError: true` saying "Input validation error" | The model sent arguments that don't match `inputSchema` | | **Tool errors** | A normal result with `isError: true` | The tool ran but failed: order not found, API down | Bad arguments are a kind of **tool error**, not a protocol error. The host or SDK can return a result with `isError: true`, giving the model a chance to correct its arguments. Error `-32602` (invalid params) is for a request that is malformed in another way, such as using an unknown tool name. ### Reading Error Frames The table separates failures by what the client receives. Here is the key distinction, first introduced in Chapter 3: - A **protocol error** comes back as a JSON-RPC `error` with a `code` and `message`. Something was wrong with the request itself. - A **tool error** comes back as a normal `result` with `isError: true`. The request was valid, but the tool's work failed. The host can show the text to the model so it can decide what to do next. ```js runnable title="Sort failures by kind" // Sort failures into kinds, because each kind needs a different fix. const failures = [ { error: { code: -32601, message: "Method not found" } }, { result: { isError: true, content: [{ type: "text", text: "Input validation error: order_id is required" }], }, }, { error: { code: -32022, message: "Unsupported protocol version" } }, { error: { code: -32021, message: "Missing required client capability" } }, { result: { isError: true, content: [{ type: "text", text: "Order BG-9 not found" }] }, }, { network: "ECONNREFUSED" }, ]; function classify(f) { if (f.network) { return [ "connectivity", "check the server is running and the URL or command is right", ]; } if (f.result?.isError) { const text = f.result.content[0].text; return text.startsWith("Input validation") ? ["bad arguments", "show the text to the model so it can fix the arguments"] : ["tool error", "show the text to the model so it can try again"]; } const fixes = { "-32601": ["unknown method", "check capabilities before calling"], "-32602": ["invalid params", "check the tool name and request shape"], "-32022": ["version mismatch", "pick a version both sides support"], "-32021": ["capability mismatch", "declare the capability or skip the feature"], }; return fixes[f.error.code] ?? ["other", "log it and alert"]; } for (const f of failures) { const [kind, fix] = classify(f); console.log(kind.padEnd(20), "->", fix); } ``` Expected output: ```text unknown method -> check capabilities before calling bad arguments -> show the text to the model so it can fix the arguments version mismatch -> pick a version both sides support capability mismatch -> declare the capability or skip the feature tool error -> show the text to the model so it can try again connectivity -> check the server is running and the URL or command is right ``` In the **MCP Lab**, call `lookup_order` with `order_id` set to `"hello"`, and then with no arguments at all (`{}`). Open the Inspector and compare both frames. Both are results with `isError: true`: the first comes from the tool itself, and the second comes from the SDK checking `inputSchema`. The model can use the returned text to understand the problem. ### Retrying Safely Once you know the failure type, ask whether repeating the call could safely help. A brief network drop or a busy server answering `503` may be temporary. Repeating a call that changes data can cause the action to happen twice. - Retry only **read-only** calls or **idempotent** calls. An idempotent call has the same effect whether it runs once or more than once. A retry of `cancel_order` or `send_email` after a timeout could repeat the action. - Wait **longer each time** with **exponential backoff**, and add a little randomness called **jitter** so many clients do not retry at the same moment. - **Stop** after a few tries and tell the user what happened. ```js runnable title="Plan a safe retry" // Retry only safe, temporary failures, waiting longer each time. const temporary = new Set(["ECONNRESET", "ETIMEDOUT", "HTTP 503"]); function plan(call, maxTries = 4, baseMs = 200) { if (!call.readOnly && !call.idempotent) { return "no retry: the tool changes data, so ask the user first"; } if (!temporary.has(call.failure)) { return "no retry: this failure will not fix itself"; } const waits = []; for (let i = 0; i < maxTries - 1; i++) { waits.push(baseMs * 2 ** i); } return `retry after ${waits.join("ms, ")}ms (plus a little random jitter)`; } console.log("lookup_order, 503: ", plan({ readOnly: true, failure: "HTTP 503" })); console.log("lookup_order, 400: ", plan({ readOnly: true, failure: "HTTP 400" })); console.log("cancel_order, reset:", plan({ readOnly: false, idempotent: false, failure: "ECONNRESET" })); ``` Expected output: ```text lookup_order, 503: retry after 200ms, 400ms, 800ms (plus a little random jitter) lookup_order, 400: no retry: this failure will not fix itself cancel_order, reset: no retry: the tool changes data, so ask the user first ``` ## Versioning Protocol and tool changes can strand clients that have not upgraded yet. Versioning gives clients and servers a way to stay compatible while changes roll out. ### Protocol Versions Each request carries a protocol version. Servers list the versions they support in `server/discover`. If the client and server share no version, the server answers `-32022`. To upgrade without breaking older hosts: 1. Add the new version **alongside** the old one. 2. Watch logs to see which hosts still use the old version. 3. Remove the old version only after those hosts have moved on. ### Your Own Tool Versions Tools change too. Adding an **optional** argument is safe for existing callers. Renaming a tool, removing an argument, or changing what a result means can break clients. For those changes, add a new tool (such as `lookup_order_v2`), keep the old tool for a while, and send `notifications/tools/list_changed` so hosts refresh their lists. ### Deprecation in the Spec A **deprecated** feature is on its way out but remains supported for a transition period. In the course's 2026-07-28 spec, deprecated features keep working for at least **12 months** and are listed in a deprecated-features registry. You saw examples in Chapter 5: Roots, Sampling, and Logging. Check the registry when planning upgrades. In the **MCP Lab**, open the Inspector and click the `server/discover` row. Its result lists `supportedVersions` and `capabilities`, the same fields a health check reads. ## Operations Versioning helps clients stay compatible during changes. Operations help teams notice when a running server is unavailable or no longer usable. ### Health Checks A **health check** is a small, regular test that asks whether a server is alive and ready for the host's needs. Calling `server/discover` checks more than whether the process responds: - Does it answer, and how quickly? - Does it share a protocol version with the host? - Does it have the capabilities the host needs, such as `tools`? ```js runnable title="Judge a health check" // A health check: call server/discover and judge the answer. const supported = ["2026-07-28", "2025-11-25"]; function health(name, reply, ms) { if (!reply) { return `${name}: DOWN (no answer)`; } const shared = reply.supportedVersions.filter((v) => supported.includes(v)); if (shared.length === 0) { return `${name}: DEGRADED (no shared protocol version)`; } if (ms > 1000) { return `${name}: DEGRADED (slow: ${ms} ms)`; } const tools = reply.capabilities.tools ? "tools" : "no tools"; return `${name}: OK (${shared[0]}, ${tools}, ${ms} ms)`; } console.log( health( "helpdesk", { supportedVersions: ["2026-07-28"], capabilities: { tools: {} } }, 42, ), ); console.log(health("wiki", { supportedVersions: ["2024-11-05"], capabilities: {} }, 30)); console.log( health( "billing", { supportedVersions: ["2026-07-28"], capabilities: { tools: {} } }, 2400, ), ); console.log(health("reports", null, 0)); ``` Expected output: ```text helpdesk: OK (2026-07-28, tools, 42 ms) wiki: DEGRADED (no shared protocol version) billing: DEGRADED (slow: 2400 ms) reports: DOWN (no answer) ``` ### Audit Logs An **audit log** records who did what, and when. It helps answer questions such as which user called a tool, which server handled it, and whether the call succeeded. For each tool call, log: - The time, user, host, and server. - The tool name and a **safe summary** of the arguments. - The result: success, tool error, or protocol error, and how long it took. Never log secrets, tokens, or full personal data. Use **OpenTelemetry** to connect a tool call in the host to the work it caused in the server. In the 2026-07-28 spec, `_meta` keys such as `traceparent` can carry trace context. ### Emergency Disable Monitoring tells you when something is wrong; an emergency disable switch lets an operator limit the damage while the team investigates. When a tool misbehaves, you need to stop it **now**, not after a new release. Keep an operator switch that: - Hides the tool from `tools/list` and sends `list_changed`. - Rejects calls to it with a clear tool error, including calls from hosts that still have the old list. ```js runnable title="Turn a tool off at once" // Emergency disable: an operator switch that hides a tool everywhere at once. const disabled = new Set(); const tools = ["lookup_order", "cancel_order", "book_return"]; function visibleTools() { return tools.filter((t) => !disabled.has(t)); } function callTool(name) { if (disabled.has(name)) { return { isError: true, text: `${name} is turned off for now. Please contact support.`, }; } return { isError: false, text: `${name} ran` }; } console.log("before:", visibleTools().join(", ")); disabled.add("cancel_order"); // a bug was found, so the operator flips the switch console.log("after: ", visibleTools().join(", ")); console.log("old client still calls it ->", callTool("cancel_order").text); ``` Expected output: ```text before: lookup_order, cancel_order, book_return after: lookup_order, book_return old client still calls it -> cancel_order is turned off for now. Please contact support. ``` An off switch that has never been tried is not an off switch. Test it on a quiet day, so you know it works on a bad one. ### Least Privilege in Operations The same least-privilege principle applies to operations: use separate credentials for each server and environment, prefer read-only access where possible, use short-lived tokens, and regularly remove access nobody needs. This limits the damage if a credential is exposed or misused. ## Evaluation and Cost Health checks and logs show whether the system is working. Evaluation and cost tracking show whether it is working well for users and at a reasonable price. ### Measure Tool Choice An **evaluation** (or **eval**) is a set of test questions with expected answers or actions. For MCP, a key question is: *Did the model pick the right tool and provide the right arguments?* Run the eval when you change tool names, descriptions, or the set of servers. ### Track Cost Each turn uses tokens for tool descriptions, results, and the answer. Track token use per question and per tool. Large tool lists and huge results are often major costs, as Chapter 7 explains. ```js runnable title="Score a small evaluation" // A tiny evaluation: did the model pick the right tool, and what did it cost? const cases = [ { question: "Where is order BG-1042?", expected: "lookup_order", picked: "lookup_order", tokens: 1800, }, { question: "Cancel BG-1043", expected: "cancel_order", picked: "cancel_order", tokens: 2100, }, { question: "How do returns work?", expected: "search", picked: "lookup_order", tokens: 2600, }, { question: "Book a return for BG-1042", expected: "book_return", picked: "book_return", tokens: 1900, }, ]; const right = cases.filter((c) => c.picked === c.expected).length; const tokens = cases.reduce((sum, c) => sum + c.tokens, 0); const pricePerMillion = 3; // dollars, example only console.log( `correct tool: ${right}/${cases.length} (${Math.round((right / cases.length) * 100)}%)`, ); console.log(`tokens: ${tokens}, cost: $${((tokens / 1e6) * pricePerMillion).toFixed(4)}`); for (const c of cases.filter((c) => c.picked !== c.expected)) { console.log(`miss: "${c.question}" -> picked ${c.picked}, wanted ${c.expected}`); } ``` Expected output: ```text correct tool: 3/4 (75%) tokens: 8400, cost: $0.0252 miss: "How do returns work?" -> picked lookup_order, wanted search ``` ## A Failure Story Here is an illustrative story that ties the operations practices together: 1. **Signal:** the eval score dropped from 95% to 70% after a release. 2. **Look:** the misses were all "how do returns work?" questions. The model called `lookup_order` instead of the wiki's `search`. 3. **Cause:** the new release had renamed the wiki tool from `search_articles` to `search`, with a one-word description. 4. **Fix:** a clearer description: "Search help-center articles about returns, shipping and setup. Use for general questions, not specific orders." 5. **Guard:** the failing questions were added to the eval, so the problem can be caught if it returns. ## The Improvement Workflow The story gives a repeatable pattern for responding to production problems. Make this a habit: 1. **Measure**: health checks, logs, eval score, cost. 2. **Find** the biggest problem. 3. **Classify** it: connectivity, mismatch, arguments, tool error, wrong tool choice, or cost. 4. **Fix** the smallest thing that solves it. 5. **Add a test** so it stays fixed. 6. **Repeat.** ## Common mistakes - **Swallowing errors.** A friendly message for users is good, but the log must keep the code and useful details. - **Retrying writes.** A timeout does not mean the action failed. It might have worked, so retry only if the operation is idempotent. - **Breaking changes without notice.** Renaming a tool can break saved prompts and evals. - **Only checking "is it up?"** A server can be running but unusable because of a version mismatch. - **Skipping evals.** Without them, you cannot tell whether a description change helped or hurt. ## Exercises Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer. ## What to Remember - Four kinds of failure: **connectivity**, **capability mismatch**, **bad arguments**, and **tool errors**. Protocol errors use `error.code`. Tool errors and bad arguments use `isError: true`, so the host can give the model useful feedback. - **Retry** only safe calls, with backoff and jitter, and stop after a few tries. - Add new protocol versions **beside** old ones. Deprecated spec features last at least **12 months**. - Run **health checks** with `server/discover`, keep **audit logs** without secrets, and keep a tested **emergency disable** switch. - **Evaluate** tool choice and **track cost**. Use the loop: measure, find, classify, fix, test, repeat. Next: the **MCP Reference**, a page to keep open while you build.