AI Prompt Playground (OpenAI / Anthropic Console)

0) Problem Restatement

Design a web-based prompt playground, like the OpenAI Playground or the Anthropic Console. A signed-in developer writes a prompt (system message plus user messages), picks a model and settings (temperature, max tokens), clicks Run, and watches the answer stream in real time. They can save prompts, keep versions, compare runs side by side, and come back later to see history.

Anthropic also asked about very large prompts (hundreds of KB to MB): where should that content live?

1) Requirements

1.1 Functional

• An editor for prompts with variables (e.g., {{customer_name}}).
• Run against a chosen model with settings, and stream the output.
• Save prompts, create versions and view run history.
• Compare outputs (e.g., two models side by side).
• Stop a running generation.

1.2 Non-Functional

• Responsive: time to first token under a second or two, and a smooth UI while streaming.
• Durable: saved prompts and runs are never lost.
• Handles large prompts without slowing everything else.
• Cost and abuse control: rate limits and spend limits per user or organization.

1.3 Scale Estimates

• 1M monthly developers, 100K daily active, ~20 runs each → 2M runs/day ≈ 25/sec, peak 100/sec.
• A typical prompt is ~5 KB, but some are 1 MB+. Outputs are ~2 KB.
• Storage: 2M runs × ~10 KB ≈ 20 GB/day (much more if large prompts are copied into every run, which we'll avoid).

1.4 API Design

• POST /v1/prompts { name, messages, model, params } → { prompt_id, version: 1 }
• POST /v1/prompts/{id}/versions (save a new version)
• POST /v1/runs { prompt_version_id | inline messages, model, params, variables } → SSE stream of tokens, then { run_id, usage }
• POST /v1/runs/{id}/stop
• GET /v1/prompts/{id}/runs?cursor=

2) High-Level Architecture

2.1 Overview

• Frontend (React): the editor, a streaming output panel and a compare view. Autosaves drafts locally.
• Playground API: auth, rate limits and spend checks, saving prompts and versions, and starting runs.
• Run Service: builds the final request (fills in variables), calls the model gateway, streams tokens back over SSE, and records usage.
• Model Gateway: the same inference service the public API uses (routing, batching, quotas).
• Metadata DB (Postgres): prompts, versions and runs.
• Object storage: large prompt bodies and outputs, stored by content hash.

2.2 Architecture Diagram

flowchart LR
    B["Browser - editor + stream view"] -->|"save"| API["Playground API"]
    B -->|"run - SSE"| RS["Run Service"]
    API --> DB[("Postgres - prompts, versions, runs")]
    API --> OS[("Object storage - large bodies by hash")]
    RS --> DB
    RS --> OS
    RS --> MG["Model Gateway"]
    MG --> GPU["Model servers"]
    RS -->|"usage"| BILL["Usage and spend limits"]

3) Data Model

CREATE TABLE prompts (prompt_id UUID PRIMARY KEY, org_id UUID, owner_id UUID, name TEXT, latest_version INT);
CREATE TABLE prompt_versions (
  version_id UUID PRIMARY KEY, prompt_id UUID, version INT,
  model TEXT, params JSONB,
  body_inline JSONB,         -- small prompts stored directly
  body_ref TEXT,             -- 'sha256:ab12...' when the body is large (object storage)
  body_bytes INT, created_at TIMESTAMP
);
CREATE TABLE runs (
  run_id UUID PRIMARY KEY, version_id UUID, user_id UUID, model TEXT, params JSONB,
  variables JSONB, output_inline TEXT, output_ref TEXT,
  status TEXT, input_tokens INT, output_tokens INT, latency_ms INT, created_at TIMESTAMP
);

Versions are immutable. Editing creates a new version, so every run points to exactly what was sent.

4) Key Flows

4.1 Run with streaming

1. The browser posts the run request and keeps the SSE connection open.
2. The Run Service checks the rate limit and spend limit, loads the version (from inline or object storage), fills in variables, and calls the model gateway with streaming on.
3. Each token chunk is forwarded to the browser right away. The service also buffers the output.
4. On finish (or stop), it saves the run with the output and token usage, and adds the cost to the org's spend.

4.2 Stop

The browser calls stop (or closes the connection). The Run Service cancels the upstream request so GPU time isn't wasted, and saves the partial output with status stopped.

5) Deep Dive A — Very large prompts

• Threshold rule: bodies under ~64 KB go inline in Postgres (fast, simple). Larger bodies go to object storage, keyed by their SHA-256 hash, with only the reference in the DB. This keeps DB rows small and fast.
• Deduplication for free: many runs and versions reuse the same big document, and hashing stores it once.
• Upload directly: the browser uploads large content to object storage with a pre-signed URL, then saves the version with the hash, so the API servers don't proxy megabytes.
• Editor performance: for a 1 MB prompt, use an editor that virtualizes rendering (only draws visible lines), and diff versions on the server.
• Model limits: check the token count against the model's context window before sending, and warn early.

6) Deep Dive B — Compare, reliability and cost

• Compare mode: fire the same prompt at two models in parallel, with two streams in one SSE connection (tagged by run ID) or two connections.
• Streaming reliability: if the connection drops, the run keeps going on the server (for a short time) and the output is saved. The UI can reload the finished run by ID.
• Cost control: per-user and per-org rate limits, a monthly spend cap with alerts at 80% and 100%, and max_tokens limits.
• History at scale: runs are append-only. Partition the runs table by month and archive old outputs to object storage.

7) Trade-offs & Alternatives

8) Common Follow-up Questions

• "Sharing prompts?" Add permissions at the prompt level (private, org, link), and see the prompt-sharing design.
• "Evaluations?" Let users attach a dataset of inputs, run a version against all of them as a batch job, and score the outputs.
• "Multi-tenant enterprise?" Scope all data by org_id, add SSO, and optionally turn off saving of run contents for sensitive orgs.

9) Wrap-Up

The browser editor saves immutable prompt versions to Postgres, keeping large bodies in object storage by content hash. The Run Service fills in variables, calls the shared model gateway and streams tokens back over SSE, while saving the output and usage at the end. Rate limits and spend caps control cost, and stopping a run cancels the upstream generation.