Deployment & Reliability - Complete Deep Dive

Prerequisites: Performance Metrics, Heartbeat & Health Checks Used in: the “how would you operate this?” part of every senior design interview

Why this shows up in HLD interviews

Junior answers stop at the architecture diagram. Senior interviews ask: how do you ship a change to this without downtime? What happens when a region dies? How do you migrate that schema while both versions are running?

These questions separate “I’ve designed systems” from “I’ve operated them.”

Deploy vs Release

The most useful distinction in this whole area:

Deploy  = the new code is running on the servers
Release = users can actually see the new behaviour

Feature flags decouple them. Deploy the code dark on Monday, flip it on for 1% of users on Wednesday, roll to 100% on Friday. If it’s bad, flip the flag — no rollback deploy needed, which is orders of magnitude faster and safer.

Say this: “I’d decouple deploy from release with feature flags. That means rollback is a config change taking effect in seconds, not a redeploy, and I can enable a risky path for internal users first.”

Deployment strategies

Rolling

Default in Kubernetes. Cheap and simple, but both versions serve traffic simultaneously for the whole rollout — so version N and N+1 must be mutually compatible. That constraint is the source of most deploy incidents, and it’s why the migration section below matters.

Blue-Green

Two identical environments. Flip the load balancer. Instant rollback is the selling point.

⚠️ The catch nobody mentions: the database. Blue and green usually share it, so a schema change can’t be “flipped back.” Blue-green gives you instant code rollback, not instant data rollback. Say that out loud and you sound experienced.

Canary

Send a small slice of real traffic to the new version and watch error rate, p99, and business metrics before proceeding.

1%  → wait 10 min, compare metrics vs control  → ok?
5%  → wait 20 min                              → ok?
25% → wait 30 min                              → ok?
100%
                    any regression → route back to 0%

Requirements people forget:

• Metrics tagged by version, or you can’t tell the canary is worse
• Enough traffic for statistical significance — 1% of 100 rps is 1 rps, which tells you nothing
• Automated analysis and rollback, otherwise it’s just a slow deploy with a human staring at a dashboard
• Sticky assignment, or a user bouncing between versions sees inconsistent behaviour

Shadow / traffic mirroring

Duplicate production traffic to the new version, throw away its responses. Perfect for validating a rewrite or load-testing with realistic traffic.

⚠️ Side effects are the trap. If the shadow service writes to the real DB, sends real emails, or charges real cards, you’ve doubled everything. Shadow requires either a separate datastore or a strictly read-only path.

Say this: “For a risky change I’d canary with automated metric analysis. For a full rewrite of a service, shadow traffic first so I validate correctness and capacity against real load with zero user risk, then canary the real cutover.”

Zero-downtime schema migrations

This is the single most-asked “how do you actually operate this” question, and the answer is a pattern worth memorizing: expand → migrate → contract.

You can never do a breaking schema change in one step, because during a rolling deploy old and new code both run.

Renaming a column (the canonical example)

❌ Wrong:  ALTER TABLE users RENAME name TO full_name;
           → old instances still SELECT name → instant 500s

✅ Right, four deploys:

1. EXPAND    Add full_name (nullable). Deploy code that WRITES BOTH,
             READS name. Old code unaffected.
2. BACKFILL  Copy name → full_name in batches (throttled, resumable).
3. MIGRATE   Deploy code that READS full_name, still writes both.
             Verify. This is the safe rollback point.
4. CONTRACT  Deploy code that only uses full_name. Then drop name.

Every step is independently deployable and independently reversible. Slower — but nothing breaks.

The rules

⚠️ Long-running ALTERs take locks. On a hot table, an ALTER that needs an ACCESS EXCLUSIVE lock will queue behind existing queries and block everything behind it — a 30-second migration becomes a 10-minute outage. Set a lock_timeout and retry rather than letting a migration pile up a queue.

Say this: “Schema changes go through expand-migrate-contract with dual writes, because during a rolling deploy old and new code run simultaneously. I set a lock_timeout on DDL so a migration fails fast instead of blocking the table, and I backfill in throttled batches so I don’t saturate replication.”

The same pattern applies to APIs and events

• API: add the new field, support both shapes, migrate clients, then remove — you can’t remove a field while old clients exist
• Kafka events: only make backward-compatible schema changes (add optional fields with defaults); enforce it with a schema registry so a bad producer can’t break every consumer
• Caches: version the cache key (v2:user:42), so a format change is a cold cache rather than a deserialization crash on every read

Reliability patterns

Timeouts, retries, circuit breakers, bulkheads

These four are always asked together because none works alone.

Timeout budgets must decrease down the stack:

client        3000ms
 gateway      2500ms
  service     2000ms
   downstream 1500ms
    database   500ms

If any inner timeout exceeds its caller's, the caller gives up first —
the inner work continues, consuming resources for a response nobody will read.

Retry rules:

• Only retry idempotent operations, or non-idempotent ones with an idempotency key
• Exponential backoff with jitter — without jitter, retries synchronize into waves
• Retry budget: cap retries at ~10% of total requests. Unbounded retries during a partial outage triple your load at exactly the wrong moment
• Don’t retry at every layer. Three layers each retrying 3× is 27 requests for one user action

# Full jitter — the version AWS recommends
delay = random.uniform(0, min(cap, base * 2 ** attempt))

Graceful degradation

Recommendation service down  → show generic popular items, not an error page
Search down                  → show browse categories
Avatar service down          → default avatar
Payment down                 → queue the order, confirm asynchronously
Analytics down               → drop the events (nobody notices)

Say this: “I’d rank features by criticality. The checkout path must work; recommendations, badges, and ‘people also viewed’ are all degradable. Each non-critical dependency gets a short timeout and a static fallback so it can never fail the page.”

Load shedding vs backpressure

Shed the right requests: drop by priority (free tier before paid, batch before interactive) and shed early — at the edge, before you’ve spent CPU on the request. Shedding after doing 90% of the work is the worst of both worlds.

⚠️ Queues don’t fix overload, they hide it. An unbounded queue converts “fast failures” into “slow successes nobody is waiting for anymore.” Always bound queues, and drop items older than the client’s timeout — serving a response after the client gave up is pure waste.

Multi-region and disaster recovery

Active-active’s real problem is writes. Two regions accepting writes to the same row means conflicts, and you have to pick a resolution strategy up front:

• Partition by user/tenant so each record has one home region — by far the most common real answer
• CRDTs for data types that converge mathematically (counters, sets)
• Last-write-wins — simple, and it silently loses writes under clock skew
• Single write region + read replicas everywhere — the pragmatic 90% answer

Say this: “I’d shard users by home region so writes are always local and there’s no conflict resolution to get wrong. Cross-region is async replication for read locality and DR. That gives me active-active availability without active-active write conflicts.”

⚠️ Untested DR doesn’t exist. If you’ve never failed over, your RTO is a guess. Say you’d run game days / scheduled failover drills — that single sentence is worth a lot in an interview.

Failover mechanics

Automatic vs manual failover:

The honest answer: automatic for stateless and read paths, human-in-the-loop for the write primary unless you have a consensus-based system (Raft) that makes it genuinely safe.

Health checks done right

⚠️ The two classic mistakes:

1. Liveness checking a downstream dependency. The DB has a blip → every pod fails liveness → Kubernetes restarts your entire fleet → now you have a cold-cache thundering herd on top of a DB problem. Liveness must only test the process itself.
2. Readiness that’s expensive. A health endpoint that runs a real query gets hit by every LB every few seconds across every instance — that’s real load, and it fails first under stress, taking your fleet out of rotation exactly when you need it.

Graceful shutdown — the other half of zero-downtime deploys:

SIGTERM received
  → fail readiness immediately (LB stops sending new requests)
  → wait for the LB's deregistration delay (it's not instant!)
  → finish in-flight requests (drain, with a deadline)
  → close connections, flush buffers
  → exit

Skipping the deregistration wait is why “zero-downtime” deploys still show a spike of 502s: the LB is still routing to a pod that already stopped accepting.

Interview application

“How do you deploy this without downtime?” “Rolling deploys behind readiness probes and graceful shutdown with a drain period. Any schema change goes through expand-migrate-contract with dual writes so N and N+1 coexist. Risky behavioural changes ship behind a feature flag, canaried at 1% with automated metric comparison, so rollback is a flag flip in seconds rather than a redeploy.”

“What happens if the primary region goes down?” “Writes are partitioned by user home region, so losing a region takes out writes for that slice only. Each region has an async cross-region replica, so we promote it — RPO in the tens of seconds because replication is async, RTO a few minutes because promotion plus fencing is scripted, and I’d verify the RTO with quarterly drills. Reads keep working everywhere from local replicas the whole time.”

“How do you prevent one slow service taking everything down?” “Per-dependency bulkheads so a slow dependency can only exhaust its own pool, timeouts that decrease going down the stack, a circuit breaker so we fail fast instead of queueing during an outage, and a static fallback for anything non-critical. Plus retry budgets — capped at ~10% of traffic with jittered backoff — so retries can’t amplify a partial failure into a full one.”

Common Interview Questions

Q: “Blue-green or canary?” A: “Canary for most changes — it exposes real-traffic bugs with a small blast radius and costs almost nothing. Blue-green when I need instant, total rollback and can afford 2× capacity. Note that neither solves database rollback; that’s what expand-migrate-contract is for.”

Q: “How do you roll back a bad deploy?” A: “Best case it’s not a deploy at all — flip the feature flag. Otherwise redeploy the previous image. The real constraint is the database: if the migration was expand-only and backward-compatible, code rollback is safe. If it was destructive, you can’t roll back, which is exactly why migrations are never destructive in the same release.”

Q: “How do you handle a schema migration on a table with 500M rows?” A: “Never a blocking ALTER. Add the new column nullable, backfill in throttled batches with a resumable cursor while watching replication lag, dual-write from the app, switch reads, then contract. Indexes go up with CREATE INDEX CONCURRENTLY. Every DDL gets a lock_timeout so it fails fast rather than blocking the table behind it.”

Q: “What’s the difference between liveness and readiness?” A: “Liveness failure restarts the container; readiness failure just pulls it from the load balancer. So readiness is for transient conditions like warmup, and liveness is only for unrecoverable states. Pointing liveness at a shared dependency causes fleet-wide restart storms.”

Q: “How do you know a canary is bad?” A: “Compare the canary against the control on the same metrics over the same window — error rate, p99, and a business metric like conversion, tagged by version. Automate the decision, because a human watching a dashboard misses slow regressions and won’t catch it at 2am.”

Q: “What’s your RPO and RTO?” A: “RPO is how much data I can lose, set by replication mode — async cross-region gives me seconds. RTO is how long I can be down, set by how automated promotion is — scripted with fencing gets me to minutes. And I’d state that both are only real if we drill them.”

Q: “Retries seem obviously good — when are they bad?” A: “During a partial outage they’re how you finish the job. Retrying a non-idempotent write duplicates it; retrying without jitter synchronizes clients into waves; retrying at every layer multiplies 3× into 27×; and retrying an overloaded dependency guarantees it stays overloaded. Retries need idempotency, jittered backoff, a budget, and a circuit breaker.”

Decision Framework