# Workflow and Automation Engines

Course: [High-Level Design](https://www.devtutorials.ai/learn/system-design/hld) · Section: B2B SaaS Architectures
Source: https://www.devtutorials.ai/learn/system-design/hld/b2b-saas/workflow-engines

> Let each tenant automate its own work with trigger, condition and action rules, then stop the rule that triggers itself, the retry that pages someone twice and the import that takes every worker.

## When a Ticket Is Urgent, Page the Lead

Acme Retail wants a simple thing: **when a ticket becomes urgent, page the on-call lead**. Contoso wants every imported ticket tagged. Fabrikam wants tickets closed after 7 quiet days. Fifty tenants, hundreds of different wishes, and none of them should need a Northwind developer.

So Northwind ships an **automation engine**: customers build their own rules in the product, and shared workers run them. That is powerful, and it brings three new ways to fail: a rule can trigger itself forever, a retry can do something twice, and one tenant's rules can keep everyone else's waiting.

**The main idea:** store rules as tenant data (trigger, condition, action), run actions from a queue, record what caused each change so rules cannot loop, give every action an idempotency key so retries are safe, and cap how many workers one tenant can use.

## TL;DR

- A rule is **data**: when (trigger event), if (condition), then (action). Each tenant has its own rules.
- The engine listens to events, finds the tenant's matching rules and puts actions on a **queue**. Workers run them.
- A rule whose action fires its own trigger **loops forever**. Record who caused each change and stop chains with a **loop guard**.
- Workers retry failed actions. Send an **idempotency key** with each action so a retry never does the work twice.
- Give each tenant a **limit** on shared workers, so a bulk import cannot block everyone.

## A Real-World Example

A help desk customer adds a rule: "when a ticket is updated, add the note *We are looking into it*". Adding a note updates the ticket. Within a minute, one ticket has 40,000 notes, the customer's email notifications have gone out 40,000 times, and the shared workers are busy with nothing else. The vendor adds an emergency kill switch that night, and a loop guard the next week.

## What You Will Learn

1. How rules are stored and run.
2. Why actions run on a queue.
3. How a rule triggers itself, and how a loop guard stops it.
4. How idempotency keys make retries safe.
5. How per-tenant limits keep shared workers fair.

## Rules Are Tenant Data

A rule is a row, not code: `(tenant, trigger, condition, action, enabled)`. This is the same idea as custom fields in Lesson 4: the product reads the tenant's definitions at run time, so 50 tenants can have 50 different setups on one codebase.

Keep conditions to a small, safe language: compare fields, check a value changed, combine with AND and OR. Never run customer-written code inside your workers without a real sandbox, time limits and memory limits.

## Loops: a Rule That Triggers Itself

Loops also hide across rules: rule A updates a field that triggers rule B, which triggers rule A. A depth limit catches those too.

Every automated change should carry its **cause**: which rule, which event, how deep in the chain. Without that, the engine cannot tell an agent's edit from its own echo. Keep a per-tenant kill switch as well, so support can stop a runaway rule in seconds.

## Retries Without Doing It Twice

Actions call other systems: a pager, an email service, a customer's webhook. Calls fail, and replies get lost. Workers must retry, and the other side must be able to say "I have already done that".

The key must be the same on every try of the same action: build it from the tenant, the record, the rule and the event, never from the time or a random number.

## One Tenant Must Not Take Every Worker

This is the noisy-neighbour problem from Lesson 3, in a new place. The same tools apply: per-tenant concurrency limits, fair queues, and a dedicated pool for a whale tenant that automates heavily.

## Try It: Make the Rule Engine Safe to Share

1. With no fixes, follow Ana's single change. How many rule runs does it cause?
2. Turn on the loop guard. Then look at the pages: why does the lead get some pages twice?
3. Turn on idempotency keys, then the per-tenant limit. What happens to Contoso's import time?

**Pass rule:** at most 10 rule runs for one change, all 20 urgent tickets paged exactly once, other tenants wait at most 1 s, and Contoso's import still finishes within 120 s.

Scheduled rules ("close tickets after 7 quiet days") need a durable timer: store the due time in the database and let a scheduler pick up due rows. An in-memory timer is lost when the server restarts. The help desk case study uses this for SLA timers.

## Check Yourself

## Common mistakes

| Mistake | Wrong | Correct | Why it happens |
|---|---|---|---|
| No loop guard | A rule updates the record that triggers it | Skip changes made by the same rule; cap the chain depth | Each rule looks fine on its own |
| Random idempotency keys | A new key on each retry | Key from tenant, record, rule and event | The key is generated where the call is made |
| Inline actions | The save waits for the email service | Queue the action | It is less code at first |
| Shared, unlimited workers | One import uses every worker | Per-tenant limits or fair queues | Tests run with one tenant |
| No kill switch | Engineers edit the database to stop a rule | A per-tenant "pause automations" switch | Nobody plans for the first runaway rule |

## Practice

## What to Remember

- Rules are tenant data: trigger, condition, action.
- Run actions from a queue, with retries.
- Record the cause of every automated change, and guard against loops.
- Use stable idempotency keys so retries never repeat work.
- Limit each tenant's share of the workers.

## Further Reading

- [Stripe: Designing robust and predictable APIs with idempotency](https://stripe.com/blog/idempotency): idempotency keys and safe retries.
- [Temporal: Durable execution](https://docs.temporal.io/temporal): running long workflows that survive crashes and retries.
- [AWS Builders' Library: Timeouts, retries and backoff with jitter](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/): how to retry without making an outage worse.

Next: **Integrations: Public API and Webhooks**. Customers connect Northwind to their own systems. You deliver every webhook, sign it, and stop one broken endpoint from slowing everyone else.