Autonomous Vehicle Camera Perception Pipeline

0) Problem Restatement

Design the camera perception pipeline for a self-driving car (asked at NVIDIA). Several cameras (e.g., 8, around the car) capture images many times per second. The pipeline must turn them into a list of objects (cars, pedestrians, cyclists, lanes, traffic lights), with position, speed and uncertainty, and hand that to the planning system within a strict time limit, running on the car's own computer. Trace the data from physical cameras to the planner.

1) Requirements

• 8 cameras at 30 frames per second, high resolution.
• Detect and track objects around the car, and estimate distance and velocity.
• End-to-end latency (photon to planner) under ~100 ms, with small jitter.
• Keep working safely if a camera fails or a stage is late.
• Log data for offline training and debugging.

1.1 Rough Budget (example)

2) Pipeline

flowchart LR
    CAM["8 cameras - hardware-triggered"] --> CAP["Capture - timestamps"]
    CAP --> PRE["Preprocess on GPU - undistort, resize"]
    PRE --> NN["Perception models - detection, segmentation, depth"]
    NN --> POST["Post-process - NMS, 3D boxes"]
    POST --> FUS["Fusion - cameras + radar/lidar"]
    FUS --> TRK["Tracking - object IDs, velocity"]
    TRK --> PLAN["Planning"]
    CAP --> LOG[("Data logger - selected clips")]
    HM["Health monitor"] --> PLAN

3) Key Stages Explained

1. Synchronized capture: all cameras are triggered by hardware at the same instant and stamped with a shared clock (e.g., PTP time sync). Without this, objects appear in different places in different cameras.
2. Calibration: each camera's intrinsics (lens) and extrinsics (position and angle on the car) turn pixels into 3D rays. Calibration is checked online, since cameras shift slightly over time.
3. Preprocessing on GPU: convert raw sensor data, fix lens distortion, resize and normalize. Keep data on the GPU (zero-copy) to avoid slow memory transfers.
4. Inference: a multi-camera model (e.g., bird's-eye-view networks) detects objects, lanes and traffic lights, and estimates depth. Models are optimized (TensorRT, lower precision like FP16/INT8) and batched across cameras.
5. Post-processing: remove duplicate boxes (non-max suppression) and produce 3D boxes with confidence scores.
6. Fusion: combine with radar (good speed measurement) and lidar (good distance), if present.
7. Tracking: a tracker (e.g., Kalman filter) links detections over time into objects with IDs and velocities, and smooths noise.
8. Output: an object list with timestamps and uncertainty, sent to planning.

4) Real-Time Engineering

• Deadlines, not queues: if a frame is late, drop it and use the next one. Planning needs the freshest data, not a backlog.
• Scheduling: fixed priorities and pinned CPU cores/GPU streams for perception. Nothing else (like logging) may steal time from the critical path.
• Determinism: pre-allocate memory, avoid dynamic allocation and garbage collection in the loop, and use bounded execution times.
• Timestamps everywhere: every output carries the capture time, so planning can compensate for latency (predicting where objects are right now).

5) Safety and Degraded Operation

• Health monitor: checks every stage's latency and output sanity (e.g., too few detections, frozen images). If a camera fails, mark its field of view as "unknown" and tell planning, which slows down or pulls over.
• Redundancy: overlapping camera views, plus radar/lidar, so no single sensor failure leaves a blind spot. Critical compute may be duplicated.
• Fail-safe: if perception stops producing valid output within the deadline, planning triggers a minimal-risk maneuver.

6) Data Logging for Training

• Continuously record into a ring buffer. When something interesting happens (a hard brake, disagreement between sensors, the driver taking over), save the clip plus metadata. You can't upload everything.
• Upload saved clips when the car is parked and on Wi-Fi. They feed labeling, retraining and regression tests.

7) Wrap-Up

Trigger all cameras together on a shared clock, calibrate them, preprocess on the GPU, run optimized multi-camera models, then post-process, fuse with other sensors and track objects over time, all inside a ~100 ms budget. Treat it as a real-time system (drop late frames, fixed priorities, no dynamic allocation, timestamps everywhere). Add a health monitor, sensor redundancy and fail-safe behavior, and log interesting clips from a ring buffer for training.