Security Monitoring Framework for Cloud Infrastructure

0) Problem Restatement

Google asked: design a security monitoring framework for cloud infrastructure. Collect security telemetry from everywhere (cloud audit logs, IAM changes, network flow logs, host and container events, application auth logs), detect threats (compromised credentials, privilege escalation, data exfiltration, crypto-mining), alert and prioritize, support investigations, and ideally respond automatically to common cases.

1) Architecture

flowchart LR
    SRC["Cloud audit logs, IAM, VPC flows, host agents, app auth logs"] --> COL["Collectors"]
    COL --> K[("Kafka - raw security events")]
    K --> NORM["Normalize + enrich (asset owner, geo-IP, threat intel)"]
    NORM --> DET["Streaming detections - rules + anomaly models"]
    NORM --> LAKE[("Security data lake - long retention")]
    DET --> ALERT["Alert manager - dedupe, score, route"]
    ALERT --> SOC["SOC analysts - cases"]
    ALERT --> SOAR["Automated response playbooks"]
    SOC --> LAKE

2) Collection and Normalization

• Pull or stream from every source: cloud provider audit logs (who did what via the API), IAM and policy changes, network flow logs, DNS logs, host/EDR agents (process launches, file changes), Kubernetes audit logs, and application login logs.
• Normalize into one schema (e.g., OCSF or ECS): actor, action, resource, source_ip, result, timestamp.
• Enrich: asset owner and criticality, user role, geo-IP, threat-intel matches (known bad IPs or domains). This makes detections and triage far better.

3) Detection

• Rules (detection-as-code), versioned in Git, tested against sample logs, and reviewed:
• "10+ failed logins then a success from a new country within 10 minutes" (a windowed rule).
• "IAM policy grants *:* admin to a new principal."
• "Security logging was disabled."
• "Large data transfer from a storage bucket to an unknown external IP."
• Anomaly detection: baselines per user or service (usual regions, API calls, data volume), flagging big deviations.
• Correlation: combine weak signals into strong ones (a suspicious login + a new access key created + an unusual API burst = likely account takeover).
• Streaming engine (e.g., Flink) for real-time rules, plus scheduled queries over the data lake for slower patterns.
• Coverage: map detections to MITRE ATT&CK techniques to see gaps.

4) Alerting and Response

• Deduplicate and group related alerts into one case, score them by severity × asset criticality × confidence, and route to on-call.
• Automated response (SOAR) for clear cases: disable a leaked access key, isolate a compromised VM, force re-authentication. Risky actions need a human to approve.
• Tuning: track alert precision per rule, and fix or retire noisy rules. Alert fatigue is the biggest practical problem.

5) Storage and Investigation

• Hot searchable storage for ~30–90 days (fast queries during incidents), and cheaper long-term storage for 1+ years (compliance, long investigations).
• Protect the monitoring pipeline itself: separate accounts, immutable log storage, and alerts if log sources go silent (attackers often disable logging).

6) Wrap-Up

Collect security telemetry from all cloud, network, host and app sources into a streaming pipeline, normalize it to one schema, and enrich it with asset, identity, geo and threat-intel context. Run version-controlled rules, windowed and correlation detections, and behavioral anomaly models in real time (plus scheduled hunts over a security data lake), then dedupe and score alerts into routed cases, automate responses for clear cases, tune rules to fight alert fatigue, and protect the pipeline's own integrity.