Content Moderation System (Harmful Post Detection)

0) Problem Restatement

Design a system that finds harmful content on a social platform. For example (asked at Meta): posts or ads selling weapons, as well as hate speech, nudity, spam and scams. Content can be text, images or video. The system checks new posts when they are uploaded. Clear violations are blocked, uncertain ones go to human reviewers, and users can report posts and appeal decisions.

The core tension: if we block too much, innocent people get hurt (false positives); if we block too little, harmful content spreads (false negatives).

1) Requirements

1.1 Functional

• Score every new post, comment, ad and profile for multiple policies (weapons, hate, nudity, spam, ...).
• Take actions: allow, reduce reach, blur with a warning, remove, ban the account.
• Human review queues with tools and priorities.
• User reports and appeals.
• Re-scan old content when policies or models change.

1.2 Non-Functional

• Low latency at upload for high-risk checks (a few hundred ms), so bad content doesn't go live even briefly.
• Scale: billions of items per day.
• Accuracy tuned per policy (weapons sales: high recall; borderline humor: careful).
• Auditability: why was this removed?

1.3 Scale Estimates

• 2B new items/day ≈ 23K/sec, peaks of 100K/sec.
• Images: ~40% of items. Video is fewer but much heavier (sample frames).
• If 0.5% go to human review, that's 10M reviews/day, too many. So ML must handle most cases, and humans handle the uncertain middle.

1.4 API Design

• Internal: POST /v1/moderate { content_id, type, text, media_urls, author_id } → { decision: allow|hold|block, labels: [{ policy: "weapons_sale", score: 0.93 }] }
• POST /v1/reports { content_id, reason }
• POST /v1/appeals { content_id, message }

2) High-Level Architecture

2.1 Overview

• Fast inline checks (at upload): hash matching against known bad content (e.g., PhotoDNA-style perceptual hashes), blocked keywords, and a fast text and image classifier.
• Async deep checks: heavier multimodal models (text + image + video frames + author signals), run within seconds to minutes after posting.
• Decision engine: combines scores with per-policy thresholds and account history, then chooses an action.
• Review system: queues by policy, language and severity, with a reviewer UI.
• Feedback loop: reviewer decisions become training labels.

2.2 Architecture Diagram

flowchart LR
    UP["New post / ad"] --> FAST["Fast checks: hash match, keywords, light model"]
    FAST -->|"clear violation"| BLK["Block"]
    FAST -->|"publish"| PUB["Post goes live"]
    FAST --> K[("Kafka - content events")]
    K --> DEEP["Deep multimodal models"]
    DEEP --> DEC["Decision engine - thresholds per policy"]
    RPT["User reports"] --> DEC
    DEC -->|"high score"| ACT["Remove / reduce reach"]
    DEC -->|"uncertain"| RQ[("Human review queues")]
    RQ --> REV["Reviewers"]
    REV --> ACT
    REV --> LBL[("Labels for retraining")]
    LBL --> TRAIN["Model training"]

3) Data Model

content_scores:  content_id, model_version, policy, score, created_at
decisions:       content_id, action, policy, source (model|reviewer|appeal), actor, reason, created_at
review_tasks:    task_id, content_id, policy, priority, language, status, assigned_to, sla_due
known_hashes:    hash, policy, source   -- known violating images/videos

4) Key Flows

4.1 A new post with an image of a gun for sale

1. Inline: check the perceptual hash against known bad images. No match. The text classifier sees "selling", "DM for price" and a weapon-related term, giving a weapons-sale score of 0.7, which is not high enough to block instantly.
2. The post is published (or held briefly, for high-risk surfaces like ads) and an event is sent to Kafka.
3. Deep model: an image model detects a firearm, the text is a sales intent, and the account is new with a pattern of similar posts. The combined score is 0.95.
4. The decision engine applies the weapons-sale threshold (block at 0.9) → remove and notify the author with the reason and an appeal link.
5. If the score were 0.6–0.9, it would go to the review queue instead, with priority based on predicted reach (viral posts first).

4.2 User report

Reports add a signal and can push content into review sooner. Many reports from trusted reporters raise priority.

5) Deep Dive A — Thresholds and precision/recall

• Each policy has its own thresholds, set from labeled data. For weapons sales we want high recall (catch almost all), so we send more to review. For satire or borderline content, favor precision to avoid wrongly removing speech.
• Use graded actions: at moderate scores, reduce reach or add a warning instead of removing.
• Measure prevalence: sample random content daily and have experts label it to estimate "what % of views were of violating content". This tells us how well the whole system works, not just the model.

6) Deep Dive B — Adversaries and scale

• Evasion: people misspell words, put text in images, crop or recolor images. Countermeasures: OCR on images, perceptual hashes that survive small edits, embeddings that capture meaning, and account-level signals (new account, many similar posts, links to known bad groups).
• Video: sample frames (e.g., 1 per second plus scene changes) and audio transcripts, and scan the most-viewed videos more deeply.
• Model updates: when a new model or policy ships, re-scan recent content in the background, starting with the most-viewed.
• Reviewer wellbeing and quality: blur by default, limit exposure time, and double-review a sample to measure reviewer accuracy.

7) Trade-offs & Alternatives

8) Common Follow-up Questions

• "Ads vs posts?" Ads are paid and higher risk, so review them before they go live. Posts go live and are checked right after, unless they're high-risk.
• "Multiple languages?" Multilingual models, plus review queues routed by language.
• "How do appeals work?" A different reviewer re-checks the item. If overturned, restore the content and add the case as a training label.

9) Wrap-Up

Run fast checks at upload (hash matching, keywords, a light model) to stop obvious violations, then deep multimodal models asynchronously. A decision engine applies per-policy thresholds to allow, limit, remove or send to human review, prioritized by reach. Feed reviewer decisions back into training, fight evasion with OCR, perceptual hashes and account signals, and measure success with sampled prevalence.