Designing ChatGPT (AI Conversational Assistant)

Difficulty: Advanced Prerequisites:Message Queues and Caching

TL;DR

An AI chat system accepts user prompts, assembles conversation context, routes to GPU inference servers, and streams the response token-by-token back to the user via SSE. The hard parts: managing context windows (128K tokens max), scaling expensive GPU inference, streaming partial responses in real-time, and handling millions of concurrent conversations.

flowchart LR
    USER["User"]:::client
    API["Chat Service"]:::service
    QUEUE["Inference Queue"]:::async
    GPU["GPU Pool<br/>LLM Inference"]:::service
    DB[("Conversation DB")]:::data

    USER -->|"1. Send prompt"| API
    API -->|"2. Assemble context"| DB
    API -->|"3. Enqueue for inference"| QUEUE
    QUEUE -->|"4. Pick job"| GPU
    GPU -->|"5. Stream tokens via SSE"| USER

    classDef client fill:#4c3a5e,stroke:#818cf8,color:#e2e8f0
    classDef service fill:#1a3a2a,stroke:#4ade80,color:#e2e8f0
    classDef async fill:#3b1f5e,stroke:#c084fc,color:#e2e8f0
    classDef data fill:#3b3520,stroke:#fbbf24,color:#e2e8f0

In 3 sentences: User sends a prompt → Chat Service fetches conversation history, assembles a context window, and enqueues an inference request. A GPU server picks the job, runs the LLM, and streams tokens one-by-one back to the user via SSE. The full response is persisted to the conversation DB for future context.

1. Understanding the Problem

An AI chat system like ChatGPT lets users have multi-turn conversations with a large language model. Unlike a traditional request-response API, the LLM generates output token-by-token (taking 5-30 seconds for a full response), so the system must stream partial results back to the user in real-time. The model has a fixed context window (e.g., 128K tokens) so the system must intelligently manage which parts of the conversation history to include. GPU inference is extremely expensive ($2-4/hour per GPU) so efficient scheduling and scaling is critical.

Real examples: ChatGPT (OpenAI), Claude (Anthropic), Gemini (Google), Copilot (Microsoft).

1.5. Naive First Cut

flowchart LR
    USER["User"]:::client
    API["API Server"]:::service
    LLM["LLM Model<br/>single GPU"]:::service
    DB[("One DB")]:::data

    USER --> API
    API --> LLM
    API --> DB

    classDef client fill:#4c3a5e,stroke:#818cf8,color:#e2e8f0
    classDef service fill:#1a3a2a,stroke:#4ade80,color:#e2e8f0
    classDef data fill:#3b3520,stroke:#fbbf24,color:#e2e8f0

User sends prompt → API calls LLM synchronously → waits 10-30 seconds → returns full response at once.

How it breaks:

• User stares at blank screen for 30 seconds waiting for full response (terrible UX — ChatGPT shows words appearing in real-time)
• One GPU serves one request at a time. With 1M concurrent users → need 1M GPUs (impossible, each costs $2-4/hour)
• No context management — sending entire conversation history to LLM exceeds context window after 10 messages
• API server thread is blocked for 30 seconds per request → thread pool exhaustion at 100 concurrent users
• No rate limiting → one user can monopolize GPU resources
• No conversation persistence — refresh the page, history is gone

1.7. Prior Art We’re Drawing From

• OpenAI Streaming API — Uses Server-Sent Events to stream tokens as they’re generated. Each token is a separate SSE event with a delta field. Client renders incrementally. The connection stays open for the full generation duration (5-60 seconds). (OpenAI API Docs)
• vLLM (UC Berkeley) — High-throughput LLM serving engine using PagedAttention. Batches multiple requests on a single GPU by managing KV-cache memory like virtual memory pages. Achieves 2-4x throughput vs naive serving. (vLLM Paper)
• Anthropic’s Constitutional AI — Multi-layer safety filtering: input classifier → model generation → output classifier. Harmful prompts are rejected before reaching the model; harmful outputs are filtered before reaching the user.
• NVIDIA Triton Inference Server — Production inference orchestrator that handles request batching, model versioning, GPU scheduling, and health monitoring. The standard for deploying LLMs at scale.

2. Functional Requirements

Core (Top 3)

1. Send prompt and receive streaming response — user types a message, LLM generates a response streamed token-by-token in real-time (<200ms to first token)
2. Multi-turn conversation with context — the model remembers previous messages in the conversation and responds coherently across turns
3. Conversation persistence — users can view chat history, continue old conversations, and access them from any device

Below the Line

• Multiple model selection (GPT-4, GPT-4o-mini, etc.)
• File/image upload and multimodal input
• Code execution (sandbox)
• Plugins / tool use / function calling
• Sharing conversations via link
• Search across conversation history

3. Non-Functional Requirements

Core

Below the Line

• Multi-region deployment for global latency
• Cost optimization (batch similar requests, use cheaper models for simple queries)
• Compliance (data residency, GDPR deletion)

Scale Estimation

• Users: 100M weekly active, 10M daily active, 1M concurrent peak
• Requests: 500K prompts/minute at peak
• Avg response: 500 tokens (~375 words), takes ~10 seconds to generate
• Context per request: avg 2000 tokens of history + 200 token prompt = 2200 input tokens
• GPU requirement: At 30 req/sec per GPU (with batching via vLLM) → ~280 GPUs at peak
• Storage: 10M conversations/day × avg 4KB = 40GB/day new conversation data

Technology Choices

4. Core Entities

• User — id, tier (free/plus/enterprise), usage counters
• Conversation — id, userId, title, model, createdAt, updatedAt
• Message — id, conversationId, role (user/assistant/system), content, tokenCount, createdAt
• InferenceJob — id, conversationId, prompt (assembled context), status (queued/running/completed/failed), model, createdAt

5. API / System Interface

POST /v1/chat/completions  (streaming)
  Headers: Authorization: Bearer <token>
  Body: { conversationId, message: "explain binary search", model: "gpt-4o" }
  Response: SSE stream
    event: token
    data: {"delta": "Binary"}

    event: token
    data: {"delta": " search"}

    event: token
    data: {"delta": " is"}
    ...
    event: done
    data: {"messageId": "msg_abc", "usage": {"promptTokens": 2200, "completionTokens": 487}}

GET /v1/conversations
  Response: [{ id, title, model, updatedAt }]  (paginated, cursor-based)

GET /v1/conversations/:id/messages
  Response: [{ id, role, content, createdAt }]  (paginated)

DELETE /v1/conversations/:id
  Response: 204 No Content

Security notes:

• JWT auth on all endpoints. userId extracted from token.
• Rate limiting enforced before inference queue (don’t waste GPU on over-limit users)
• Input content goes through safety classifier before reaching the model

6. High-Level Design

FR1: Send Prompt and Receive Streaming Response

The core UX: user sends a message, words start appearing on screen within 500ms, and continue flowing at reading speed until complete.

New components:

1. Chat Service — Orchestrates the request: authenticates, rate-limits, assembles context, enqueues inference job, and relays the token stream back to the client.
   💡 This service is stateless and horizontally scalable. It holds an open SSE connection to the client while waiting for tokens from the GPU.
2. Inference Queue — Buffers requests between Chat Service and GPU pool. Priority-based (paid users > free). Absorbs burst traffic.
3. GPU Inference Pool — Fleet of GPU servers running vLLM. Picks jobs from queue, runs the model, streams tokens back.
4. Token Relay — Mechanism for GPU to send tokens back to the specific Chat Service instance holding the user’s SSE connection. Uses Redis Pub/Sub (channel = jobId).

flowchart LR
    USER["User Browser"]:::client
    CS["Chat Service"]:::service
    RQ["Inference Queue<br/>Redis Streams"]:::async
    GPU["GPU Pool<br/>vLLM"]:::service
    PS["Redis Pub/Sub<br/>token relay"]:::async
    DB[("Conversation DB")]:::data

    USER -->|"1. POST prompt via SSE"| CS
    CS -->|"2. Enqueue inference job"| RQ
    RQ -->|"3. GPU picks job"| GPU
    GPU -->|"4. Publish tokens"| PS
    PS -->|"5. Relay tokens"| CS
    CS -->|"6. Stream tokens via SSE"| USER
    CS -->|"7. Save message on done"| DB

    classDef client fill:#4c3a5e,stroke:#818cf8,color:#e2e8f0
    classDef service fill:#1a3a2a,stroke:#4ade80,color:#e2e8f0
    classDef async fill:#3b1f5e,stroke:#c084fc,color:#e2e8f0
    classDef data fill:#3b3520,stroke:#fbbf24,color:#e2e8f0

Step-by-step flow:

1. User sends prompt → Chat Service authenticates (JWT), checks rate limit (Redis counter)
2. Chat Service fetches conversation history from DB, assembles the context window (system prompt + history + new message)
3. Chat Service creates an inference job and pushes to Redis Streams (priority queue: paid > free)
4. A GPU server (running vLLM) picks the job from the queue
5. GPU starts generating tokens. For EACH token generated, it publishes to Redis Pub/Sub channel tokens:{jobId}
6. Chat Service is subscribed to tokens:{jobId} — receives each token and immediately writes it to the user’s SSE connection
7. User’s browser receives tokens one-by-one via EventSource.onmessage → renders each word as it arrives
8. When generation completes, GPU publishes a done event. Chat Service saves the full response to Conversation DB, closes the SSE connection.

Why Redis Pub/Sub for token relay? The GPU server and the Chat Service instance holding the user’s SSE connection are different machines. We need a way to route tokens from GPU → specific Chat Service instance. Redis Pub/Sub with channel = jobId achieves this with sub-ms latency. Fire-and-forget is fine because if a token is lost, the user sees a tiny gap — acceptable for streaming text.

FR2: Multi-Turn Conversation with Context

The LLM is stateless — it doesn’t “remember” anything. Every request must include the full conversation history (up to the context window limit). The Chat Service manages this.

New components:

1. Context Assembler — Builds the prompt sent to the LLM by combining: system prompt + conversation history + new user message. Handles truncation if history exceeds context window.
2. Context Cache (Redis) — Caches the assembled context for active conversations. Avoids re-fetching from DB on every turn.

flowchart LR
    CS["Chat Service"]:::service
    CC["Context Cache<br/>Redis"]:::data
    DB[("Conversation DB")]:::data
    ASM["Context Assembler"]:::service

    CS -->|"1. Check cached context"| CC
    CS -->|"2. Fallback read messages"| DB
    CS -->|"3. Assemble prompt"| ASM
    ASM -->|"4. Return assembled context"| CS

    classDef service fill:#1a3a2a,stroke:#4ade80,color:#e2e8f0
    classDef data fill:#3b3520,stroke:#fbbf24,color:#e2e8f0

How context assembly works:

Context window = 128K tokens (model limit)
System prompt = 500 tokens (fixed instructions)
New user message = 200 tokens
Available for history = 128K - 500 - 200 = 127,300 tokens

Strategy:
  1. Include ALL recent messages (starting from newest, going backwards)
  2. Stop when cumulative tokens exceed the available budget
  3. If conversation is very long (>127K tokens of history):
     - Include last 20 messages verbatim (recent context)
     - Include a SUMMARY of older messages (compressed by a smaller model or pre-computed)
     - Summary is stored in Redis/DB, updated every 10 turns by a background worker

Why this matters: Without proper context management, after 10 long messages the context window overflows and the LLM either errors or loses early context silently.

FR3: Conversation Persistence

Users expect: close the tab → reopen → conversation is there. Switch devices → same conversations available.

Storage:

DynamoDB:
  Table: conversations
    PK: userId
    SK: conversationId
    Fields: title, model, createdAt, updatedAt, messageCount

  Table: messages
    PK: conversationId
    SK: createdAt (or messageId with timestamp prefix)
    Fields: role, content, tokenCount

Why DynamoDB:

• Partition by userId → user’s conversations are co-located
• Messages partitioned by conversationId → fetch one conversation’s history fast
• No complex joins (just “give me messages for conversation X”)
• Scales infinitely for write-heavy workload (millions of new messages per minute)

7. Potential Deep Dives

Deep Dive 1: Token Streaming — How It Actually Works

Problem: LLM generates one token every 15-30ms (30-70 tokens/second). User needs to see each token immediately, not wait 10 seconds for the full response.

Bad: Wait for full generation → return as one response. User waits 10-30 seconds staring at nothing.

Good: Buffer tokens in batches of 5-10, send every 100ms. Slightly choppy but works.

Great: Stream EACH token individually via SSE the instant it’s generated. User sees smooth, word-by-word appearance — exactly like ChatGPT’s UX.

Implementation:

GPU (vLLM) generates tokens:
  token 1: "Binary"   → publish to Redis Pub/Sub channel tokens:{jobId}
  token 2: " search"  → publish
  token 3: " is"      → publish
  token 4: " a"       → publish
  ...

Chat Service (subscribed to tokens:{jobId}):
  receives token → immediately writes to SSE connection:
    res.write('event: token\ndata: {"delta":"Binary"}\n\n')
    res.write('event: token\ndata: {"delta":" search"}\n\n')
    ...

Browser:
  const source = new EventSource('/v1/chat/completions?...');
  source.onmessage = (e) => {
    const { delta } = JSON.parse(e.data);
    appendToScreen(delta);  // word appears instantly
  };

Deep Dive 2: GPU Inference Scaling

Problem: GPUs cost $2-4/hour each. LLM inference takes 10-30 seconds per request. You can’t give each user a dedicated GPU. How do you serve 1M concurrent users with 300 GPUs?

Bad: One request per GPU at a time. 1M users / 30 sec per request = need 30M GPU-seconds per second = impossible.

Good: Request batching — collect 8-32 requests and run them through the model together (batch inference). GPU utilization goes from 10% to 70%+. Throughput: 30-100 requests/sec per GPU.

Great: Continuous batching with vLLM PagedAttention. Requests don’t wait for the whole batch to finish. As soon as one request in the batch completes (shorter response), a new request takes its slot. GPU stays 90%+ utilized continuously.

Math with batching:

• 1 GPU handles ~50 requests/sec (with continuous batching)
• 500K requests/min at peak = ~8300 requests/sec
• GPUs needed: 8300 / 50 = ~166 GPUs at peak
• With 2x headroom: ~330 GPUs

Cost: 330 GPUs × $3/hour = $1000/hour = $720K/month at peak. This is why ChatGPT Plus costs $20/month — GPU inference is genuinely expensive.

Deep Dive 3: Rate Limiting for AI (Token-Based, Not Just Request-Based)

Problem: A user asking “explain quantum computing in detail” costs 100x more GPU time than “hi.” Request-count limits alone don’t capture true cost.

Bad: Limit to 40 requests per 3 hours regardless of size. User sends 40 “write me a 5000-word essay” requests → exhausts far more GPU than intended.

Good: Limit by token count (input + output). Free tier: 100K tokens per day. Paid: 1M tokens per day. Accurately reflects cost.

Great: Dual limits: request count AND token count. Plus priority queuing — paid users’ jobs are dequeued before free users’. During peak load, free tier gets slower (higher queue wait) while paid stays fast.

Implementation:

Redis keys per user:
  ratelimit:{userId}:requests   → INCR, TTL=3h, max=40
  ratelimit:{userId}:tokens     → INCRBY(token_count), TTL=24h, max=100000

On each request:
  1. Check request count → reject if over limit
  2. Estimate input tokens (fast tokenizer) → check token budget → reject if over
  3. After response: add actual output tokens to the counter

Deep Dive 4: Safety and Content Moderation

Problem: Users will try to jailbreak the model (make it say harmful things), generate illegal content, or extract training data.

Solution: Two-stage filtering pipeline:

User prompt
    ↓
INPUT CLASSIFIER (fast, lightweight model — 5ms):
  - Is this a harmful prompt? (violence, illegal activity, personal info extraction)
  - Score: 0.0 (safe) to 1.0 (harmful)
  - If score > 0.7 → reject immediately, don't waste GPU time
    ↓
LLM generates response
    ↓
OUTPUT CLASSIFIER (same fast model — 5ms):
  - Is this response harmful even though the prompt seemed ok?
  - If score > 0.7 → replace with "I can't help with that" message
    ↓
User receives safe response

Why two stages: Input filter catches obvious attacks. But clever jailbreaks slip through → output filter is the safety net. Both are fast (5ms) because they’re small classifier models, not the full LLM.

Deep Dive 5: Context Summarization for Long Conversations

Problem: After 50 messages, the conversation exceeds 128K tokens. You can’t send everything to the LLM. But dropping old messages means the model “forgets.”

Bad: Truncate — just send the last 20 messages. Model loses all earlier context.

Good: Sliding window — last 20 messages verbatim + drop the rest. Simple but information loss.

Great: Sliding window + rolling summary. A background worker periodically summarizes older messages into a compressed paragraph. The assembled prompt becomes:

[System prompt: 500 tokens]
[Summary of messages 1-30: "User asked about binary search, then discussed quicksort vs mergesort..." — 200 tokens]
[Messages 31-50: verbatim — 5000 tokens]
[New user message: 200 tokens]
Total: ~5900 tokens (fits easily in 128K)

The summary is updated every 10 new messages by a cheaper, faster model (GPT-4o-mini) running as a background job. Stored in Redis for fast access.

8. Final Architecture

flowchart TD
    subgraph Clients
        WEB["Web Browser"]:::client
        APP["Mobile App"]:::client
    end

    subgraph Edge
        GW["API Gateway<br/>auth + rate limit"]:::edge
    end

    subgraph Services
        CS["Chat Service<br/>context assembly + SSE"]:::service
        SAFETY["Safety Classifier"]:::service
        SUMM["Summarization Worker"]:::service
    end

    subgraph Inference
        QUEUE["Inference Queue<br/>Redis Streams"]:::async
        GPU1["GPU Server 1<br/>vLLM"]:::service
        GPU2["GPU Server 2<br/>vLLM"]:::service
        GPUN["GPU Server N<br/>vLLM"]:::service
    end

    subgraph Data
        CONV[("Conversation DB<br/>DynamoDB")]:::data
        CACHE[("Context Cache<br/>Redis")]:::data
        PS["Redis Pub/Sub<br/>token relay"]:::async
    end

    WEB --> GW
    APP --> GW
    GW --> CS
    CS --> SAFETY
    CS --> CACHE
    CS --> CONV
    CS --> QUEUE
    QUEUE --> GPU1
    QUEUE --> GPU2
    QUEUE --> GPUN
    GPU1 --> PS
    GPU2 --> PS
    GPUN --> PS
    PS --> CS
    CS -->|"SSE stream"| WEB
    CS -->|"SSE stream"| APP
    SUMM --> CONV
    SUMM --> CACHE

    classDef client fill:#4c3a5e,stroke:#818cf8,color:#e2e8f0
    classDef edge fill:#1e3a5f,stroke:#60a5fa,color:#e2e8f0
    classDef service fill:#1a3a2a,stroke:#4ade80,color:#e2e8f0
    classDef async fill:#3b1f5e,stroke:#c084fc,color:#e2e8f0
    classDef data fill:#3b3520,stroke:#fbbf24,color:#e2e8f0

How it works end-to-end:

1. User sends prompt → API Gateway authenticates, rate-limits
2. Chat Service runs prompt through Safety Classifier → reject if harmful
3. Chat Service assembles context: fetch from Redis cache (or DynamoDB fallback) + new message
4. Chat Service enqueues inference job to Redis Streams (priority: paid > free)
5. A GPU server picks the job, starts generating tokens via vLLM
6. Each token is published to Redis Pub/Sub (channel = jobId)
7. Chat Service receives tokens, relays to user via SSE — words appear in real-time
8. On completion: save full assistant message to DynamoDB, update context cache
9. Background: Summarization Worker periodically compresses old conversations for efficient future context

Key Takeaways

• Stream tokens via SSE — the defining UX of ChatGPT. Never wait for full response.
• GPU batching (vLLM/PagedAttention) — batch multiple requests per GPU, 10-50x better utilization
• Context window management — sliding window + rolling summary keeps conversations coherent without overflow
• Priority queue — paid users get faster inference during peak; free users wait longer but still served
• Two-stage safety filter — input classifier before GPU (save cost) + output classifier after (safety net)

Related Designs

• Chat System (WhatsApp) — message persistence, delivery guarantees (different: human-to-human, not AI)
• Notification System — push delivery patterns
• Job Scheduler — queue-based job processing with priority

Related Concepts

• Message Queues → — Redis Streams for inference queue
• Caching → — context cache for active conversations
• Rate Limiting → — token-based rate limiting for AI workloads

Written by SystemCraft Interview Prep Platform

Free system design, LLD, and DSA resources built by engineers who went through the interview grind at Indian and international tech companies.