Observability - Complete Deep Dive

Prerequisites: Performance Metrics, Heartbeat & Health Checks Used in: the last five minutes of every senior interview — “how would you know this system is healthy?”

Monitoring vs Observability

Say this: “Monitoring tells me that something is wrong; observability lets me find out why without shipping new code. The practical test is whether I can answer a question I didn’t anticipate — if I have to add a metric and redeploy to debug, I have monitoring, not observability.”

The three pillars

The workflow that ties them together:

Metric alert fires: "checkout p99 > 1s"
  → dashboard: which service? → payments
  → trace: where in the request? → 800ms in the fraud-check call
  → logs for that trace ID → "fraud model cache miss, falling back to sync scoring"
                            ↑ root cause, in about 90 seconds

This is the answer to “how would you debug a latency regression?” — metrics to detect, traces to localize, logs to explain. Naming that sequence is worth more than listing tools.

Metrics

The four golden signals (Google SRE)

⚠️ Split latency by success and failure. A fast-failing 500 looks like a latency improvement if you bucket everything together — errors returning in 2ms will pull your p99 down while the system burns.

RED vs USE — two lenses, different subjects

Say this: “RED for every service — rate, errors, duration — because that’s what the user experiences. USE for every resource behind it, because saturation is the leading indicator: queue depth and pool utilization rise before latency does, which is what makes them useful for autoscaling and for paging early.”

Metric types

⚠️ Prefer histograms over summaries. You cannot merge per-instance p99s into a fleet p99 (see Performance Metrics). Histograms merge; quantiles don’t.

Cardinality explosion — the #1 real-world observability incident

Every unique combination of label values is a separate time series.

http_requests_total{method, status, endpoint}
  5 methods × 10 statuses × 50 endpoints = 2,500 series          ← fine

...now add user_id:
  2,500 × 1,000,000 users = 2,500,000,000 series                 ← your metrics
                                                                    system is dead

Never put these in a metric label: user ID, request ID, trace ID, email, session ID, full URL path with IDs in it (/orders/12345), raw error message, timestamp.

Where high-cardinality data belongs: logs and traces. They’re indexed for it; metrics aren’t.

❌ metric label: endpoint="/orders/12345"
✅ metric label: endpoint="/orders/:id"     ← normalize the template
✅ trace attribute / log field: order_id=12345

Say this: “I’d keep metric labels low-cardinality — normalized route templates, status class, region — and push per-request identifiers into traces and logs. Cardinality explosion from putting a user ID in a label is the most common way teams take down their own monitoring, right at the moment they need it.”

Logs

Structured, always

❌ log.info("User " + userId + " failed payment of " + amt + " reason " + r);
   → unqueryable; you're writing regexes at 3am

✅ log.info("payment_failed",
       Map.of("user_id", userId, "amount_cents", amt,
              "reason", r, "trace_id", traceId));
   → queryable: reason:insufficient_funds AND amount_cents:>10000

Levels, used consistently

⚠️ If everything is ERROR, nothing is. Log-level discipline is what makes “error rate” a usable signal.

The costs people forget

• Volume dominates the bill. 10K rps × 2KB per log = 20MB/s = ~1.7TB/day. At that scale, DEBUG logging in production is a five-figure monthly line item.
• Logging is synchronous unless you make it async. A blocking write to a slow disk or a full log pipeline adds latency to every request — logging has caused real outages this way.
• PII in logs is a compliance problem. Redact at the emitter, not in the pipeline, because once it’s stored you have a deletion obligation.

Sampling strategy: log all errors, sample successes (1%), and support dynamic log levels so you can turn DEBUG on for one service or one user without a redeploy.

Distributed tracing

trace_id: abc123  (one user request, propagated through every hop)

[gateway ──────────────────────────────────────────── 450ms]
  [auth ── 20ms]
  [orders ─────────────────────────────────── 400ms]
     [db query ── 30ms]
     [inventory ────── 90ms]
     [payments ──────────────────── 260ms]     ← there it is
        [fraud-check ──────── 200ms]           ← actually, there

Propagation is the whole game. If any service drops the trace header, the trace breaks there and everything downstream looks like an unrelated trace. Instrument it once in a shared HTTP/gRPC client (or get it free from a service mesh) rather than per-endpoint — and remember queues: trace context must be carried in message headers too, or async work becomes invisible.

Sampling:

Say this: “Tail-based sampling if the platform supports it — I want 100% of errors and slow traces and 1% of the boring ones. Head-based sampling at 1% is cheap but statistically guarantees you miss most of the rare failures you’re trying to debug.”

Alerting

Alert on symptoms, not causes

❌ "CPU > 80%"                 → often fine, and doesn't mean users hurt
❌ "a pod restarted"           → that's the system self-healing
❌ "disk 70% full"             → not urgent unless it's growing fast

✅ "checkout error rate > 1% for 5 min"      → users are failing
✅ "p99 latency > 1s for 10 min"             → users are waiting
✅ "queue depth growing for 15 min"          → users will fail soon
✅ "error budget burn rate 14× normal"       → we'll blow the SLO in 2 days

The test for a good alert: if it fires and nobody needs to do anything, it should not have fired. Every non-actionable page trains the on-call to ignore alerts — that’s alert fatigue, and it’s how real incidents get missed.

Symptom vs cause, in practice

Page on symptoms (user-visible). Route causes to dashboards and tickets. One symptom alert plus a good dashboard beats twenty cause alerts, because the twenty all fire together during an incident and bury the signal.

Burn-rate alerting (the SRE approach)

Instead of “error rate > 1%,” alert on how fast you’re consuming the error budget:

SLO: 99.9% over 30 days → error budget = 43 minutes

Burn rate 1×   → you'll exactly use the budget in 30 days   → fine
Burn rate 14×  → budget gone in ~2 days                     → page now
Burn rate 6×   → budget gone in ~5 days                     → ticket, not a page

Two windows (a fast one to catch spikes, a slow one to catch slow bleeds) gives you high precision and recall — far better than a fixed threshold that either pages on every blip or misses a slow degradation.

Things worth alerting on that people forget

• Absence of data — a metric going silent usually means the exporter died, and a silent system looks perfectly healthy on a dashboard
• Certificate expiry at 30 days
• Consumer lag growing (the earliest signal that async processing is falling behind)
• DLQ depth > 0 — messages are being permanently dropped from the happy path
• Replication lag exceeding your read-your-writes assumption
• Cron/batch job didn’t run — nothing failing means nothing alerting

What to instrument in an HLD answer

When asked “how would you monitor this?”, walk the request path:

⚠️ Business metrics catch what technical metrics miss. A deploy that makes the checkout button invisible produces a perfect technical dashboard — 200s everywhere, low latency — while revenue goes to zero. “Orders per minute vs the same time last week” catches it in minutes. Mentioning this is a strong senior signal.

The two saturation metrics most worth alerting on: queue depth and connection pool utilization. Both rise before latency does, which makes them leading rather than lagging indicators.

Interview application

“How would you know this system is healthy?” “RED metrics per service and USE per resource, with the SLO defined on the user-facing indicator — checkout success rate and p99 latency at the edge. I’d page on symptoms using multi-window burn-rate alerts against the error budget, not on CPU. For debugging, distributed tracing with W3C context propagated through HTTP and message headers, tail-sampled so I keep every error and slow trace, and structured logs carrying the trace ID so I can jump from a trace span straight to its logs.”

“A customer says the app is slow. Walk me through it.” “Check whether it’s them or everyone: is the fleet p99 elevated, and does it slice to a region, client version, or tenant? If it’s global, find the slow hop in a trace — that gives me the service and often the specific call. Then the logs for that trace ID explain why. If the metrics look clean, I suspect something outside my measurement: DNS, mobile radio latency, or a client-side regression — which is why I want RUM at the client, not just server-side timing.”

“How do you avoid alert fatigue?” “Only page on symptoms that need immediate human action, use burn-rate windows so brief blips don’t page, route everything else to dashboards and tickets, and review every page in the postmortem asking ‘was this actionable?’ — deleting alerts is real work that teams under-invest in.”

Common Interview Questions

Q: “Metrics, logs, or traces — where do you start?” A: “Metrics to detect and quantify, traces to localize, logs to explain. They’re a workflow, not alternatives. If I could only have one for a distributed system I’d take traces, because they contain enough structure to derive rough metrics, whereas metrics can never tell me where in a call graph the time went.”

Q: “What’s cardinality explosion?” A: “Every unique label-value combination is a separate time series. Adding a user ID label to a metric multiplies your series count by your user count and kills the metrics backend — usually during an incident, when you need it most. High-cardinality identifiers belong in traces and logs; metric labels should be normalized route templates and low-cardinality dimensions.”

Q: “Why not alert on CPU?” A: “High CPU often means good utilization, and low CPU can coexist with terrible latency from lock contention or a saturated connection pool. CPU is a cause, not a symptom. I’d alert on user-visible symptoms and keep CPU on the dashboard for diagnosis.”

Q: “How do you trace a request through async processing?” A: “Propagate trace context in the message headers, not just HTTP headers, and link the consumer span to the producer span. Without that, everything after the queue looks like an unrelated trace and the async half of your system is invisible.”

Q: “What’s the most important cache metric?” A: “Hit ratio — it directly tells me whether the cache is earning its cost. Paired with eviction rate: a falling hit ratio with a high eviction rate means the cache is too small, whereas a falling hit ratio with low evictions means the access pattern changed or TTLs are too short.”

Q: “How much do you sample?” A: “Tail-based: 100% of errors and slow traces, ~1% of successes. Head-based sampling at a flat 1% is cheaper but means you almost never captured the rare slow request you’re being asked about.”

Q: “What would you alert on for a Kafka consumer?” A: “Consumer lag and its trend — absolute lag spikes during a deploy and that’s fine, but lag growing steadily means arrival rate exceeds processing rate and you’ll never catch up. Also oldest-message age, DLQ depth above zero, and the absence of processing metrics entirely, since a dead consumer emits nothing and looks calm.”

Decision Framework