CAP Theorem - Complete Deep Dive

Prerequisites: Database Concepts, Scalability Used in: Key-Value Store, Digital Wallet, Chat System

What is CAP Theorem?

CAP theorem states that a distributed system can only guarantee TWO of these three properties simultaneously:

• Consistency — Every read gets the most recent write (all nodes see the same data at the same time)
• Availability — Every request gets a response (even if it’s stale data)
• Partition Tolerance — System works even when network communication between nodes breaks

The catch: Network partitions WILL happen in distributed systems (routers fail, cables break, cloud AZs lose connectivity). So Partition Tolerance is not optional — you must have P.

This means your real choice is: C or A (during a partition).

During normal operation: You get all three (C + A + P)
During a network partition: You must choose C OR A

Choose C (Consistency): System refuses to serve potentially stale data → some requests fail
Choose A (Availability): System serves whatever data it has → some responses might be stale

The Analogy

Imagine two bank branches (Node A and Node B) that share account data. The phone line between them breaks (partition).

If you choose Consistency (CP):

• Customer at Branch B wants to withdraw $500
• Branch B can’t verify the balance with Branch A (partition)
• Branch B says: “Sorry, system is unavailable. Come back later.”
• No wrong transaction happens, but service is down.

If you choose Availability (AP):

• Customer at Branch B wants to withdraw $500
• Branch B can’t verify with Branch A, but serves anyway using its last known balance
• Customer withdraws $500
• Meanwhile, Customer ALSO withdraws at Branch A (balance was already $500)
• Total withdrawn: $1000 from a $500 account (inconsistency!)
• But both branches stayed “available.”

CP vs AP Systems

CP Systems (Consistency + Partition Tolerance)

During a partition, the system becomes unavailable rather than serve stale data.

When to choose CP: Financial transactions, inventory management, any system where wrong data is worse than no data.

AP Systems (Availability + Partition Tolerance)

During a partition, the system continues serving requests but might return stale data.

When to choose AP: Social media feeds, product catalogs, DNS, session stores — where stale data is acceptable for a few seconds.

Consistency Models (Spectrum)

It’s not binary (consistent vs inconsistent). There’s a spectrum:

For interviews, you only need to know:

• Strong consistency (CP) — for money/critical data
• Eventual consistency (AP) — for everything else
• Read-your-writes — compromise (after write, read from primary; otherwise read from replica)

Real-World Examples in System Design

Example 1: Digital Wallet (CP)

User A has $100.
User A transfers $100 to User B.
User A tries to spend $100 again.

CP (correct): Second spend is rejected — balance is 0.
AP (dangerous): Second spend might succeed if nodes are partitioned → $100 created from nothing.

Decision: Financial systems MUST be CP. Lost availability (retry later) is better than lost money.

Example 2: Instagram Feed (AP)

User posts a photo.
Follower opens app 2 seconds later.

CP: Follower sees "loading..." until all replicas are consistent.
AP: Follower might not see the photo for 2-5 seconds (eventual consistency). That's fine.

Decision: Social feeds use AP. Seeing a post 3 seconds late doesn’t matter. Unavailability would be worse.

Example 3: Chat System (Compromise)

Message ordering within a conversation: needs consistency (can't show messages out of order).
Read receipts: eventual consistency is fine (delay of 1-2s is OK).
Online status: eventual consistency (green dot can be 10s stale).

Decision: Different parts of the same system can have different consistency requirements.

PACELC Theorem (Extended CAP)

CAP only talks about what happens DURING a partition. PACELC adds: what happens during NORMAL operation?

PAC: During Partition → choose Availability or Consistency
ELC: Else (no partition) → choose Latency or Consistency

Translation: Even without partitions, you trade latency for consistency. Strong consistency requires waiting for all replicas to acknowledge. Eventual consistency returns immediately (faster) but might be stale.

How to Talk About CAP in Interviews

Don’t say: “My system is CP” (too simplistic)

Do say: “For the payment/wallet path, I’d use strong consistency (writes to single Postgres primary, reads from primary after writes). For the feed/notification path, I’d use eventual consistency with read replicas (AP), accepting 1-2s staleness. Different components have different consistency requirements.”

The key insight: You don’t choose CP or AP for the whole system. You choose PER COMPONENT based on what that component does.

Common Interview Questions

Q: “Is your system CP or AP?” A: “It depends on the component. The transaction ledger is CP (strong consistency — we can’t lose or duplicate money). The user feed is AP (eventual consistency — seeing a post 2s late is fine). The notification system is AP (we’d rather deliver eventually than fail completely).”

Q: “What happens if your database primary goes down?” A: “If CP: writes fail until a new primary is elected (seconds to minutes). The system is unavailable for writes but no data is corrupted. If AP: writes go to a replica, but we risk conflicts when the original primary comes back.”

Q: “How do you handle stale reads?” A: “For critical reads (after a write): read from primary (strong consistency). For general browsing: read from replica (eventually consistent, ~100ms lag). Use read-your-writes pattern: tag the user with a version number after writing, read from primary if version hasn’t caught up on replica.”

Q: “Can you have strong consistency AND high availability?” A: “During normal operation, yes (PACELC: EC). During a network partition, no — you must choose. In practice, partitions are rare and short-lived, so most of the time you get both.”

Decision Framework