ChatGPT-Style Conversational AI Service

0) Problem Restatement

Design a product like ChatGPT. A user types a message and the answer streams back word by word. Conversations are saved, so the user can come back later on any device and continue. The answers come from large language models (LLMs) running on GPUs, which are expensive and limited, so the system has to share them fairly and stay up even when demand is higher than capacity.

A good way to answer: start with a minimal version (no history, just question in and answer out), then add conversations, streaming, limits and reliability.

1) Requirements

1.1 Functional

• Send a message and receive a streamed response.
• Keep conversations: list, open, rename, delete.
• Continue a conversation, which means the model sees the earlier messages.
• Stop generating, and regenerate an answer.
• Different plans (free vs paid) get different models and limits.

1.2 Non-Functional

• Time to first token under ~1 second in normal load.
• High availability: degrade gracefully (smaller model, queue, clear message) instead of failing.
• Scale: tens of millions of daily users.
• Safety: filter harmful inputs and outputs.

1.3 Scale Estimates

• 50M daily users × 10 messages = 500M messages/day ≈ 6,000/sec, peak ~20K/sec.
• An average answer is 400 tokens. At ~50 tokens/sec per stream, each answer takes about 8 seconds, so ~50K–150K streams are open at the same time.
• The GPU fleet is the bottleneck. Each GPU server handles only a limited number of concurrent requests (e.g., 50–100 with batching), so we need thousands of GPU servers.
• Storage: 500M messages × 2 KB ≈ 1 TB/day of conversation text.

1.4 API Design

• POST /v1/conversations/{id}/messages with { content, model? } → an SSE stream (Server-Sent Events: one long HTTP response that sends small chunks as they are ready) of { delta: "Hello" } … { done: true, message_id }.
• GET /v1/conversations?cursor= and GET /v1/conversations/{id}.
• POST /v1/messages/{id}/stop.

2) High-Level Architecture

2.1 Overview

• API Gateway: login, rate limits per user and plan, and SSE connections.
• Chat Service: loads conversation history, builds the prompt (context), applies safety checks, and calls the inference layer.
• Inference Router: picks a model cluster with free capacity and handles queueing.
• Model servers (GPU): run the LLM, batching many requests together to use the GPU well.
• Conversation Store: messages by conversation (e.g., Cassandra/DynamoDB), plus a cache for recent conversations.
• Safety service: checks prompts and outputs.

2.2 Architecture Diagram

flowchart LR
    U["Web / Mobile"] -->|"SSE"| GW["API Gateway - auth, rate limits"]
    GW --> CS["Chat Service"]
    CS --> DB[("Conversation Store")]
    CS --> SF["Safety Service"]
    CS --> R["Inference Router"]
    R --> Q[("Priority queues per model")]
    Q --> M1["GPU cluster - large model"]
    Q --> M2["GPU cluster - small model"]
    M1 -->|"tokens"| CS
    M2 -->|"tokens"| CS

3) Data Model

conversations: conversation_id, user_id, title, created_at, updated_at, model
messages:      conversation_id (partition), message_id (time-ordered), role (user/assistant),
               content, token_count, parent_message_id (for regenerate branches), status

Partitioning by conversation_id keeps a whole conversation together, so loading it is one fast query.

4) Key Flows

4.1 Sending a message

1. The gateway checks the user's rate limit (e.g., 40 messages / 3 hours on the free plan).
2. The Chat Service saves the user message, then loads recent messages.
3. Context assembly: models have a maximum input size (the context window). If the conversation is too long, keep the system prompt plus the newest messages, and replace the old part with a short summary.
4. A safety check runs on the input.
5. The router sends the request to a GPU cluster. Tokens stream back and are forwarded to the user through SSE as they arrive.
6. When the stream ends, save the full assistant message (and generate a conversation title in the background for new chats).

4.2 Stop and disconnect

If the user clicks stop or closes the tab, the Chat Service cancels the GPU request, which frees capacity, and saves the partial answer.

5) Deep Dive A — GPUs are the bottleneck

• Batching: a GPU is much more efficient when it serves many requests at once. Model servers use continuous batching: new requests join the running batch as soon as another one finishes, instead of waiting for the whole batch.
• Queues and backpressure: when all GPUs are busy, requests wait in a queue with a timeout. If the queue is too long, reject early with a friendly "at capacity" message rather than letting everyone time out.
• Priorities: paid users' queue is served first, while free users still get a guaranteed share so they are not starved.
• Fallback: when the big model is overloaded, route free users to a smaller, cheaper model.
• Autoscaling is slow (GPU machines take minutes to start and load a model), so keep some spare capacity and plan ahead for daily peaks.

6) Deep Dive B — Reliability of long streams

• A stream can take 30+ seconds, and many things can break in the middle. Save tokens in chunks as they arrive, so a page refresh can resume by reading what has been generated so far.
• If a GPU server dies mid-answer, the router retries on another server (starting over) and the UI shows a "regenerating" state.
• Make message sends idempotent with a client-generated message ID, so a network retry does not create two answers.
• For multi-region, users are served in their nearest region. Conversation data is replicated so a region failure only interrupts in-flight streams.

7) Trade-offs & Alternatives

8) Common Follow-up Questions

• "How do you add memory across conversations?" Store extracted user facts separately and retrieve the relevant ones into the prompt, the same way retrieval-augmented generation (RAG) works.
• "How do you count cost?" Count input and output tokens per request, add them to per-user and per-org usage counters, and use those for limits and billing.
• "Enterprise version?" Add SSO, per-organization data isolation, audit logs, and a setting to disable training on customer data.

9) Wrap-Up

Stream answers over SSE from a Chat Service that loads history, builds the context and calls an inference router. GPUs are the scarce resource, so use continuous batching, priority queues, early rejection and smaller-model fallback. Store conversations partitioned by conversation ID, save streamed tokens as they arrive, and make sends idempotent so failures mid-answer are recoverable.