Malicious IP and URL Detection Service

0) Problem Restatement

LinkedIn asked two related questions:

1. Malicious IP detection: watch application and network events, detect bad source IPs (scrapers, credential stuffing, DDoS bots), and give every request a low-latency decision: allow, challenge (CAPTCHA), rate-limit or block. It must work across multiple regions.
2. Malicious URL checking: when users share links, check them using an existing (slow, rate-limited) isMalicious(url) API, with caching, rate limiting and fault tolerance.

Both share the same shape: a fast decision path backed by a cache, fed by a slower detection path.

1) Requirements

• Decision API: check(ip | url) → { action, reason, ttl } in under ~5 ms at very high QPS.
• Detection: find new bad actors from signals within seconds to minutes.
• Propagate new blocks to all regions quickly (under ~30 s).
• Minimize false positives, and support allowlists and appeals.
• For URLs: don't overload the external API, and handle its outages.

1.1 Scale Estimates

• 500K requests/sec across regions need a decision → decisions must be local (in-memory) lookups.
• Known bad IPs: millions. Known URL verdicts: hundreds of millions (cache the popular ones).

2) Architecture

flowchart LR
    REQ["Incoming requests"] --> EDGE["Edge / gateway - local decision cache"]
    EDGE -->|"miss or suspicious"| DS["Decision Service"]
    DS --> RC[("Regional cache - Redis")]
    EDGE -->|"request events"| K[("Kafka - security events")]
    K --> DET["Detection: streaming features + rules + model"]
    TI["Threat intel feeds"] --> DET
    DET -->|"new verdicts"| VS[("Verdict store - global")]
    VS -->|"replicate + push"| RC
    RC -->|"push invalidations"| EDGE
    DS -->|"URL cache miss - rate limited"| EXT["isMalicious API"]

3) Fast Path (decisions)

• Each gateway keeps an in-memory set of blocked and challenged IPs (and CIDR ranges), refreshed by push from the regional cache. The lookup is a hash or prefix-tree (for IP ranges) lookup, taking microseconds.
• Entries have a TTL (e.g., block for 1 hour), because IPs get reused and bad actors move on.
• The action depends on confidence: low → rate-limit, medium → challenge, high → block.
• Allowlists (known partners, internal IPs) always win.

4) Detection Path (finding bad actors)

• Streaming features per IP: requests/minute, failed logins/minute, number of distinct accounts tried, error rates, user-agent diversity, geo mismatch. These are computed in a stream processor (Flink) with sliding windows.
• Rules: "over 50 failed logins across over 20 accounts in 5 minutes → block 1h".
• ML model scores IPs on combined features, and threat-intel feeds add known bad IPs.
• A verdict (ip, action, expires_at, reason) is written to the global verdict store and pushed to all regions. Publish it on a global topic, so each region updates its Redis and gateways within seconds.

5) URL Checking with a Slow External API

1. Normalize the URL (lowercase host, remove tracking parameters) and hash it.
2. Look it up in the cache (malicious: longer TTL; clean: shorter TTL, since sites can get compromised later).
3. On a miss: call isMalicious through a rate-limited client (token bucket for the API's quota), with request coalescing (many users sharing the same new URL → one call).
4. If the API is slow or down: circuit breaker + fallback, e.g., allow the post but mark the URL "unverified", and re-check asynchronously, removing or warning if it turns out bad. For high-risk surfaces (like DMs to many users), fail closed or hold the message.
5. Re-check popular URLs periodically.

6) Deep Dive — False positives and adversaries

• Shared IPs (mobile carriers, corporate NAT, VPNs): one IP may carry thousands of real users. Prefer challenges over blocks for them, and combine the IP with device or account signals.
• Rotating IPs (botnets): detect by behavior patterns and fingerprints, not just IP, and block whole ranges or ASNs (network owners) when evidence is strong.
• Appeals and monitoring: track how often challenged users solve the challenge (real users do, bots don't), and provide unblock tools.
• Fail-open vs fail-closed: if the decision service is down, gateways keep using their last known lists (still protected), and new detections simply pause.

7) Trade-offs & Alternatives

8) Wrap-Up

Make decisions locally at the gateway from in-memory sets of blocked or challenged IPs (with TTLs and allowlists), fed by a regional cache that receives pushed verdicts from a global store within seconds. Produce verdicts with a streaming detection pipeline (windowed per-IP features, rules, models and threat intel). For URLs, normalize and cache verdicts, call the slow external API through a rate-limited, coalescing client with a circuit breaker, and choose fail-open or fail-closed per surface.