ML Pipeline to Predict Document Aspect Ratio (Adobe)

0) Problem Restatement

Adobe asked a machine-learning engineer to design an end-to-end pipeline: given a document that has images and text (like a flyer, social post or slide), predict the best aspect ratio for it (e.g., 1:1, 4:5, 16:9, 9:16), perhaps to suggest a layout or crop for different platforms. The interview checks whether you can go from a vague product goal to data, features, model, evaluation and serving.

1) Clarify the Problem

• Output: a fixed set of common ratios (classification: 1:1, 4:5, 16:9, 9:16, A4), or any number (regression on width/height). A fixed set usually matches product needs better, so classification it is, with probabilities, so we can suggest the top 2.
• Input: the document's images (the main image, all images), text (length, headlines), and layout elements (counts, positions).
• Success: users accept the suggestion (product metric), and offline accuracy / top-2 accuracy (ML metric).

2) Data

• Labels: past documents where users picked a final aspect ratio (implicit labels from product logs), plus a smaller set labeled by designers for quality.
• Cleaning: remove templates the user never edited (they just kept the default), and balance classes (1:1 might dominate).
• Splits: split by user or template, not randomly, so near-duplicate documents don't leak between train and test.

3) Features

• Image features: embeddings from a pretrained vision model (e.g., CLIP or a ViT). Also the main image's own aspect ratio, faces and salient objects (where the important content is).
• Text features: character and word counts, number of text blocks, headline length, and text embeddings (short vs long copy matters: long text prefers taller formats).
• Layout features: number of elements, their bounding-box spread, and how much margin exists.
• Context: the target platform, if known (Instagram story → 9:16), and the user's past choices.

4) Model

• Baseline: gradient-boosted trees (e.g., XGBoost) on hand-crafted features plus pooled embeddings. It's fast, strong and explainable.
• Better: a multimodal neural network, with image and text encoders (frozen or fine-tuned) plus a small MLP on combined features, trained with cross-entropy.
• Calibration: make the probabilities meaningful (temperature scaling), so the UI can show "confident" vs "try these two".

5) Pipeline

flowchart LR
    LOGS["Product logs - final ratios"] --> DS["Dataset builder - clean, balance, split"]
    LABEL["Designer labels"] --> DS
    DS --> FE["Feature extraction - image + text + layout"]
    FE --> FS[("Feature store")]
    FS --> TR["Training + eval"]
    TR --> REG[("Model registry")]
    REG --> SRV["Model server"]
    APP["Editor app"] -->|"document"| SRV
    SRV -->|"top-2 ratios + confidence"| APP
    APP -->|"accepted / changed"| LOGS

6) Evaluation

• Offline: accuracy, top-2 accuracy, per-class precision/recall, and a confusion matrix (mixing up 4:5 and 1:1 is less bad than 16:9 vs 9:16). Check slices too: document types and languages.
• Online: an A/B test where treatment shows suggestions. Measure suggestion acceptance, time to finish a design, and export rates.

7) Serving

• Latency target: under ~200 ms, since the suggestion appears while editing.
• Compute image embeddings once when an image is added (cache by image hash), so prediction only runs the small head model.
• Batch requests on GPU servers, or run a distilled small model on-device for speed and privacy.
• Log inputs, predictions and user choices for monitoring and retraining.

8) Monitoring and Retraining

• Watch the distribution of predictions and inputs (e.g., a new platform trend like more vertical video) for drift.
• Retrain monthly (or when drift is detected) with new accepted and changed choices, and promote only if offline metrics and a small online test improve.

9) Wrap-Up

Frame it as classification over common ratios with calibrated probabilities. Build labels from users' final choices (cleaned and split by user) plus designer labels, and use image embeddings, text statistics and embeddings, layout and context features. Start with gradient-boosted trees, then a multimodal network. Evaluate with top-2 accuracy, confusion analysis and an online A/B test, serve with cached image embeddings for low latency, and monitor drift for regular retraining.