Multi-Tenant Data Ingestion and Processing Platform

0) Problem Restatement

Amazon asked: design a platform that ingests data from many customers (tenants) and runs processing pipelines on it (validate, transform, enrich, store). Tenants range from tiny (a few MB/day) to huge (TBs/day). The platform must keep each tenant's data isolated, stop one tenant from slowing everyone else ("noisy neighbor"), support per-tenant configuration, and be reliable.

1) Requirements

• Ingest via API, file uploads and streaming connectors.
• Per-tenant schemas and pipeline configuration (which transforms, where to deliver).
• Isolation: tenants never see each other's data, with per-tenant encryption keys.
• Fairness: quotas and fair scheduling. Big tenants can't starve small ones.
• Reliability: retries, dead-letter queues and replay. Per-tenant visibility and cost tracking.

2) Architecture

flowchart LR
    T["Tenants - API, files, connectors"] --> GW["Ingestion gateway - auth, quotas"]
    GW --> BUF[("Durable buffer - partitioned by tenant")]
    BUF --> SCH["Fair scheduler - per-tenant queues"]
    SCH --> W["Shared worker pool"]
    SCH --> DW["Dedicated workers - premium tenants"]
    CFG[("Tenant config + schemas")] --> W
    W --> OUT[("Per-tenant storage - separate prefixes/keys")]
    W -->|"failures"| DLQ[("Per-tenant dead-letter")]
    W --> MET["Metrics + cost per tenant"]

3) Key Design Points

• Tenant identity everywhere: every record carries tenant_id from authentication (never from the payload). Storage paths, topics or partitions, and encryption keys are per tenant.
• Quotas at the gateway: requests/sec and bytes/day per tenant (by plan). Over-quota → throttle with 429, or accept into a lower-priority lane.
• Fair scheduling: instead of one shared FIFO queue (where a tenant dumping 1 TB blocks everyone), keep per-tenant queues, and have the scheduler take work round-robin or weighted-fair across tenants. Each tenant also has a max concurrency.
• Tenancy tiers:
• Shared (pooled) workers and storage for most tenants: cheap.
• Dedicated workers, or even separate clusters, for large or regulated tenants: strong isolation, predictable performance, but higher cost.
• Per-tenant config: schemas (validated with a registry), transform steps and destinations, versioned. Config changes apply to new batches, and bad configs are caught by validation plus a dry run.
• Reliability: at-least-once processing with idempotent writes (dedupe keys), retries with backoff, and per-tenant dead-letter queues so one tenant's bad data doesn't block others. Replay from the durable buffer.

4) Isolation and Security

• Logical isolation (tenant ID in every query, row-level security) for shared stores, and physical isolation (separate buckets, databases or accounts) for high tiers.
• Per-tenant KMS keys, so a tenant's data can be crypto-shredded on offboarding.
• Access logs and audit per tenant.

5) Observability and Cost

• Per-tenant dashboards: volume, lag, errors, DLQ size and quota usage.
• Cost attribution: CPU-seconds, bytes stored and processed per tenant, used for pricing and for spotting abusive patterns.

6) Wrap-Up

Authenticate tenants at a gateway that enforces quotas, buffer data durably partitioned by tenant, and use a fair scheduler over per-tenant queues (with per-tenant concurrency caps) feeding a shared worker pool, plus dedicated capacity for big or regulated tenants. Carry tenant identity through every step, isolate storage and keys per tenant, apply versioned per-tenant schemas and configs, process at-least-once with idempotent writes and per-tenant dead-letter queues, and track metrics and cost per tenant.