# Integrations: Public API and Webhooks

Course: [High-Level Design](https://www.devtutorials.ai/learn/system-design/hld) · Section: B2B SaaS Architectures
Source: https://www.devtutorials.ai/learn/system-design/hld/b2b-saas/integrations

> Customers connect Northwind to their own systems. Deliver every webhook with retries and a dead-letter queue, sign it, keep one broken endpoint from slowing the rest, and rate-limit the API per tenant.

## Other Systems Depend on You Now

Acme Retail wants every closed ticket in its data warehouse. Litware wants new tickets in its chat tool. Contoso syncs every customer record into its CRM every night. B2B customers do not just use your product; they **connect** it to everything else they run.

That makes two promises. Your **public API** lets their code call you, so it must stay stable and must not let one customer's script slow the others. Your **webhooks** call their code when something happens, so they must arrive, arrive once in practice, be provably from you, and never be held up by another customer's broken server.

**The main idea:** write each webhook to an outbox with the change, deliver from a queue per endpoint with retries and backoff, dead-letter what still fails, sign every message, and rate-limit the API by tenant.

## TL;DR

- **Outbox:** write the event in the same database transaction as the change, so a crash never loses it.
- **Retries with backoff** (1, 2, 4, 8 s…), then a **dead-letter queue** the customer can replay.
- **One queue per endpoint**: a slow or dead server only delays its own events.
- **Sign** every webhook (HMAC-SHA256 with the endpoint's secret, plus a timestamp) so receivers can reject fakes and replays.
- **Rate-limit the API per tenant**, return HTTP 429 with `Retry-After`, and **version** the API so old integrations keep working.

## A Real-World Example

A SaaS company sent all webhooks from one shared queue. One large customer's endpoint started taking 30 seconds to answer. Within an hour every customer's webhooks were an hour late, and thousands of integrations that used webhooks to trigger shipping, alerts and billing were silently behind. The fix was a queue per endpoint and a timeout of a few seconds.

## What You Will Learn

1. How to deliver a webhook reliably.
2. How signatures let receivers trust a webhook.
3. Why one slow endpoint must not share a queue with everyone.
4. How to rate-limit a public API per tenant.
5. How to change an API without breaking integrations.

## Delivering Every Webhook

Webhooks are delivered **at least once**: a retry after a lost reply means the receiver can see an event twice. Give every event a unique id and tell customers to ignore ids they have already processed. Order is not guaranteed either; include a sequence number or the record's version so a receiver can ignore an older update that arrives late.

Set a short timeout (a few seconds) on every delivery, and never follow redirects to private addresses. A webhook URL is typed in by a customer: it could point at your own internal network. Resolve the address and refuse private and internal ranges before you connect.

## Signing: Proving It Came from You

Compare signatures with a constant-time comparison, and let an endpoint have two secrets for a while, so customers can rotate them without downtime.

## One Slow Endpoint Must Not Slow the Rest

If an endpoint keeps failing for days, pause it and email the customer, instead of retrying forever.

## The Public API: Limits per Tenant

Version the API (for example `/v1/` in the path, or a dated version header), add fields freely, and never remove or rename one inside a version. Publish a deprecation date for old versions and tell each tenant which of its integrations still use them: you can see that in your API logs.

## Try It: Deliver Every Webhook Without Letting One Customer Slow the Rest

1. Start with no fixes. How many events are lost, and why?
2. Turn on retries and the dead-letter queue. What happens to the healthy endpoint's delivery time?
3. Switch to one queue per endpoint. Then sign the webhooks and change a character of the body in the receiver box.
4. Turn on the per-tenant API limit.

**Pass rule:** 0 events lost (each is delivered or dead-lettered), every flaky-endpoint event delivered, the healthy endpoint's p95 within 2 s, webhooks signed, and no other tenant's API requests rejected while Contoso's script runs.

Many customers cannot accept inbound webhooks at all (firewalls, on-premises systems). Offer a way to **pull** the same events too: an events API with a cursor ("give me everything after event 18,204"). It is the same event store, read the other way.

## Check Yourself

## Common mistakes

| Mistake | Wrong | Correct | Why it happens |
|---|---|---|---|
| Send inside the request | A slow receiver slows the save | Outbox, then a dispatcher | It is the shortest code path |
| No dead-letter queue | Events vanish after the last retry | Keep them; let the customer replay | Failures seem rare in testing |
| Shared delivery queue | One broken endpoint delays everyone | Queue per endpoint | One queue is simpler to run |
| Unsigned webhooks | Anyone can post fake events | HMAC signature + timestamp | The URL feels private |
| Breaking API changes | A field renamed in place | New version; old one kept until its deprecation date | The change looks small from the inside |

## Practice

## What to Remember

- Save each event with its change, then deliver it from a queue.
- Retry with backoff, dead-letter what still fails, and let customers replay it.
- Give each endpoint its own queue and a short timeout.
- Sign every webhook with a per-endpoint secret and a timestamp.
- Rate-limit the API per tenant, and version it.

## Further Reading

- [Standard Webhooks specification](https://www.standardwebhooks.com/): signing, ids and timestamps that many providers share.
- [Stripe: Receive webhook events](https://docs.stripe.com/webhooks): retries, signatures and duplicate events from the receiver's side.
- [IETF draft: RateLimit header fields for HTTP](https://datatracker.ietf.org/doc/draft-ietf-httpapi-ratelimit-headers/): telling API clients their limits.

Next: **Data Residency, Regions and Cells**. Fabrikam EU's contract says its data never leaves the EU. You find every copy, route every request to the right cell, and move the tenant to a new cell without losing a change.