Observability: Metrics, Logs and Traces

By the Syncfusion DevTutorials team · Last updated October 20, 2026

See what your system is doing: the three pillars, golden signals, structured logs, distributed traces, and alerting on SLO burn rate instead of noise.

Think about driving a car at night.

The dashboard shows your speed and fuel level. A warning light appears when the engine becomes too hot. If something goes wrong, a mechanic can connect a diagnostic tool and see exactly which part failed and when it happened. Without the dashboard and warning lights, you would only know there was a problem when the car stopped working.

Observability is the software equivalent of the dashboard, warning lights, and diagnostic tool. It helps you understand what your system is doing from the outside and discover why something went wrong without relying on guesswork.

The main idea: collect measurements (metrics), events (logs), and request journeys (traces), connect them using a shared identifier, and alert people when customers are being affected, not simply because a machine appears slightly busy.

What You Will Learn

In this lesson, you will learn:

1. What observability means and how it differs from monitoring.
2. The three pillars of observability: metrics, logs, and traces.
3. The most important metrics to watch first: the four golden signals.
4. How to create logs that are easy to search and analyze.
5. How traces and spans show the path of a request through a system.
6. How a trace id connects logs, metrics, and traces together.
7. How to use SLO burn-rate alerts so alerts are meaningful.

Monitoring and Observability

The terms are related, but they are not the same.

• Monitoring answers questions you already know to ask, such as "Is the error rate above 1%?"
• Observability helps answer questions you did not think of beforehand, such as "Why are orders slow only for Android customers in Kochi?"

Both are important. Monitoring tells you that a problem exists. Observability helps you understand what the problem is and why it happened.

The Three Pillars

• Metrics are measurements collected over time, such as requests per second, error rate, and p99 latency. They are inexpensive to store and are ideal for dashboards and alerts.
• Logs are records of individual events, such as "order 881 saved" or "payment timed out." They provide detailed information.
• Traces follow a single request as it moves through different services, showing how much time is spent in each step.

In simple terms: metrics tell you something is wrong, traces tell you where, and logs tell you why.

Metrics

Kinds of Metrics

• A counter only increases, such as total orders or total errors.
• A gauge can increase or decrease, such as the number of busy workers or queue length.
• A histogram records how values are distributed, making it possible to calculate p50 and p99 values (Lesson 1).

Most useful metrics come from comparing counters over time.

Divide errors by requests for each minute and the problem stands out:

The raw numbers look similar, but the percentage clearly shows that something went wrong at 12:33.

The Four Golden Signals

With hundreds of available metrics, where should you start? Google's SRE book recommends focusing on four key signals for every service.

• Latency: how long requests take to complete. Monitor p50 and p99 values, and track failed requests separately.
• Traffic: how many requests the service receives.
• Errors: how many requests fail.
• Saturation: how heavily the service is being used, such as worker usage, memory usage, or queue length.

If these four signals look healthy, the service is usually healthy as well.

Logs

Structured Logs

Logs written as plain text are easy to create but difficult to search.

The wrong way first: search free-text logs

Three log lines, and a search for the text "881":

The search also returns user 8812. In a system generating millions of logs each day, this kind of extra noise makes troubleshooting much more difficult.

The right way: log fields, not sentences

A structured log stores information as named fields, typically using JSON. This allows you to search for exact values instead of matching text.

The same three events as structured logs, and a search for the field orderId = 881:

The code behind the animation:

The right way: structured logs let you ask for orderId 881 exactly

const logs = [
  { level: "info", event: "order_saved", orderId: 881, ms: 12 },
  { level: "info", event: "login", userId: 8812 },
  { level: "error", event: "payment_failed", orderId: 881, reason: "timeout" },
];

const found = logs.filter((entry) => entry.orderId === 881);
console.log(`Lines found: ${found.length}`);
found.forEach((entry) => console.log(`  ${entry.level}: ${entry.event}`));

Expected output:

Lines found: 2
  info: order_saved
  error: payment_failed

Log Levels and What Not to Log

Assign a level to each log entry so that important information can be filtered easily.

• error: something failed and requires attention;
• warn: something unusual happened, but the operation still succeeded;
• info: a normal but important event;
• debug: detailed information mainly used by developers and usually disabled in production.

Never log sensitive information such as card numbers, passwords, or full addresses. Logs are often copied, archived, and stored for long periods.

Traces

Spans

A trace records the complete journey of a single request. Each individual step is called a span, which represents a specific piece of work with a start time and duration.

Spans can contain other spans. For example, "Load cart" may contain a "SELECT cart items" database query because the query occurs during the cart-loading process.

When displayed visually, traces often appear as waterfall timelines.

Finding the Slow Part

A long-running span is not always the real problem. What matters is how much time a span spends doing its own work, excluding any child spans. This is called self time.

Here is one slow checkout request as a waterfall. Own work = the span's total time minus the time of the spans inside it:

The problem is in the query itself, not in the cart-loading code. The query is likely missing an index, as discussed in Lesson 6.

Try It: Find the Slow Span

Choose a request and identify which span is causing the slowdown. When you find the correct span, you can also view the log entry that shares the same trace id.

Press Play to replay the request step by step and identify the slow component.

One Id to Tie It All Together

The Trace Id

When a request enters the system, the first service assigns it a unique trace id, such as a3ce929d0e0e4736. Every service then passes this id to the next service and records it in all logs and spans.

Passing the id between services is known as context propagation. The W3C standard uses a header called traceparent for this purpose.

This makes it possible to gather every log entry and trace related to a single request across the entire system.

Step through how one trace id turns scattered logs from three services into one story:

Without the trace id, these log entries would be difficult to find among millions of others.

Alerting That Means Something

Alert on Symptoms, Not Causes

An alert should wake someone up only when customers are affected, or are likely to be affected soon.

• Good: "Checkout errors are consuming the monthly error budget too quickly."
• Bad: "Server 7 CPU is at 85%." This may not be a problem at all.

Too many unnecessary alerts create alert fatigue, where important alerts start to get ignored.

Burn Rate

In Lesson 1, a 99.9% SLO creates an error budget of 0.1% per month. The burn rate measures how quickly that budget is being used.

• burn rate 1: the budget is being used at the planned rate;
• burn rate 5: the budget is being used five times faster than planned.

For example, if 0.5% of orders are failing right now:

• burn rate = 0.5% failing ÷ 0.1% budget = 5.0;
• at that speed, the month's budget is gone in 30 ÷ 5 = 6 days.

Page or Ticket?

A common recommendation from the Google SRE Workbook is to page someone immediately if the burn rate remains above 14.4 for an hour. This consumes 2% of the monthly error budget in just one hour.

Slower burn rates can usually be handled by creating a ticket for investigation.

A month has 30 × 24 = 720 hours, so one hour at burn rate b uses b ÷ 720 of the budget:

Common Mistakes

Practice

What to Remember

• Observability helps you investigate new questions about a running system, not just the questions you planned for.
• Metrics tell you something is wrong, traces tell you where, and logs tell you why.
• Start with the four golden signals: latency, traffic, errors, and saturation.
• Use structured logs with log levels, and never log sensitive information.
• A trace contains multiple spans. Check self time to find the true source of delays.
• Pass a trace id through every service and include it in logs and traces.
• Alert using SLO burn rate. Page for fast burns and create tickets for slower ones.

Further Reading

These are the sources this lesson was checked against:

• Google SRE Book: Monitoring Distributed Systems: the four golden signals, and alerting on symptoms.
• Google SRE Workbook: Alerting on SLOs: burn rates, and the 14.4 rule for paging.
• OpenTelemetry: Observability Primer: metrics, logs, traces and spans explained.
• W3C Trace Context: the standard traceparent header for passing trace ids between services.

You have completed HLD Foundations. Next, you will move on to the HLD Case Studies, where you will apply everything learned in these twelve lessons to design real systems, starting with a URL shortener.