Scaling a Rules Engine for High Traffic (Rippling)

0) Problem Restatement

Rippling asked: scale a rules engine that evaluates customer-defined rules against a high volume of events in real time. Example rules in an HR/IT platform:

• "When an employee's department changes to Engineering, add them to the GitHub org and the #eng Slack channel."
• "When a new hire's start date is within 7 days, order a laptop."
• "If an expense > $500 and the category is travel, require manager approval."

1) Rules as Data

{ "rule_id": "r-91", "tenant": "acme", "version": 3, "enabled": true,
  "trigger": { "event": "employee.updated", "field_changed": "department" },
  "condition": { "all": [ { "field": "employee.department", "op": "==", "value": "Engineering" },
                          { "field": "employee.country", "op": "in", "value": ["US", "CA"] } ] },
  "actions": [ { "type": "add_to_group", "app": "github", "group": "eng" } ] }

• Trigger: which event type (and optionally which field change) can fire this rule. This is the key to indexing.
• Condition: an expression tree, compiled when saved into fast predicates.
• Actions: side effects run by action workers.

2) Architecture

flowchart LR
    EV["Domain events"] --> K[("Kafka - by tenant")]
    K --> EVAL["Evaluators - stateless, autoscaled"]
    RS["Rule service - save, validate, compile"] --> RDB[("Rules DB - versioned")]
    RS -->|"rule changed"| RC[("Compiled rule cache per tenant")]
    RC --> EVAL
    EVAL --> AQ[("Action queue")]
    AQ --> AW["Action workers - idempotent, retries"]
    AW --> APPS["Integrations - Slack, GitHub, payroll"]

3) Making Evaluation Fast

1. Index rules by trigger per tenant: (tenant, event_type) → rules. Most events match only a handful of rules. Optionally index further by the changed field or a simple equality in the condition (department == Engineering) for a hash lookup.
2. Compile conditions once (to closures or bytecode), instead of re-parsing JSON on every event.
3. Cache compiled rule sets in evaluator memory, keyed by tenant and version. Invalidate via a "rules changed" event, with a periodic version check as backup.
4. Enrich once: load the event's entity (employee record) once, and let all candidate rules read from it.
5. Partition by tenant in Kafka, so each tenant's events are processed in order. Scale evaluators by the number of partitions and lag.

4) Correctness Issues

• Idempotent actions: an action carries a key (rule_id, rule_version, event_id), so retries or reprocessing don't add someone to a group twice or order two laptops.
• Ordering: per-tenant (or per-entity) ordering, so "department changed to Eng, then back to Sales" doesn't apply in the wrong order.
• Rule updates: evaluation uses the rule version current when the event was processed, and records which version fired (for audit and explainability).
• Loops: an action can generate events that trigger rules again. Limit the chain depth, and detect repeating cycles.
• Consistency vs latency: a rule edit may take a few seconds to reach all evaluators (the cache refresh). Acceptable in most cases. For strict needs, include the version check in the hot path.

5) Wrap-Up

Store rules as versioned data with a trigger, a compiled condition and actions, index them per tenant by event type (and simple equality keys) so each event is evaluated only against relevant rules, and cache compiled rule sets in stateless evaluators that consume a tenant-partitioned event stream. Run actions through a queue with idempotency keys and retries, keep per-entity ordering, record which rule version fired, and guard against rule loops.