Smart Grid Control over Unreliable Devices

0) Problem Restatement

Design a central system that watches and controls millions of smart devices on a power grid: EV chargers, air conditioners, fridges, water heaters and hospital equipment. All of them talk to us over the unreliable public internet. Devices report their power use and status, and those reports can be delayed, duplicated or out of order. When total load gets too high, the system sends commands like "reduce all ACs in region X by 20%". Some devices are offline, some users have opted out or overridden, and critical devices (hospital equipment) must never be turned down. Finally, the system must reconcile: which devices actually did what we asked?

OpenAI asked several versions of this (monitoring, controller, command dispatch, "underspecified power-plant prompt"). The skills tested are handling unreliable networks and designing safe control loops.

1) Requirements

1.1 Functional

• Ingest device telemetry: power draw, state and settings.
• A live view of load per region.
• Policies: rules that decide when and how to reduce load, respecting device type, user preferences and exemptions.
• Send commands to groups of devices, and track acknowledgements and actual effect.
• Reconcile the expected vs actual response, and retry or escalate.

1.2 Non-Functional

• Safety first: never curtail exempt devices, respect user overrides, and cap how much and for how long a device is curtailed.
• Correct under unreliable delivery: duplicates, reordering, long offline periods.
• Timely: react to overload within seconds to a minute.
• Scale: millions of devices.

1.3 Scale Estimates

• 5M devices reporting every 30 seconds → ~170K reports/sec.
• A regional event: a command to 500K devices within a minute.

1.4 API Design

• Device → cloud: POST /telemetry (or MQTT) { device_id, seq, ts, watts, state, desired_version_applied }
• Cloud → device: MQTT topic devices/{id}/desired → { version, target: { max_watts: 1200 }, valid_until }
• Operator: POST /v1/events { region, reduce_pct: 20, duration_min: 30, device_types: ["ac"] }

2) High-Level Architecture

2.1 Overview

• Device gateway (MQTT): authenticates each device with a certificate and keeps connections open.
• Telemetry pipeline: Kafka → stream processor → per-device latest state + per-region load aggregates.
• Policy engine: watches load and forecasts, and decides targets per device group, filtering out exempt and opted-out devices.
• Device shadow / desired-state store: for each device, the desired state (what we want) and the reported state (what it says it did).
• Command dispatcher: pushes desired-state changes, and retries until acknowledged or expired.
• Reconciler: compares expected vs measured load reduction.

2.2 Architecture Diagram

flowchart LR
    D["Devices"] <-->|"MQTT"| GW["Device Gateway"]
    GW --> K[("Kafka - telemetry")]
    K --> SP["Stream processor - dedupe, order by seq"]
    SP --> SH[("Device shadows - reported state")]
    SP --> AG[("Region load aggregates")]
    PE["Policy Engine - safety rules"] --> AG
    PE --> SH
    PE -->|"new desired state"| DS[("Desired state store")]
    DS --> CD["Command Dispatcher - retry until ack"]
    CD --> GW
    RC["Reconciler"] --> SH
    RC --> DS
    RC --> AG

3) Handling Unreliable Reports

• Every report carries a device sequence number and the device's timestamp.
• Duplicates: ignore reports with a seq we've already processed for that device.
• Out of order: only update "latest state" if seq is newer. Older reports still go to history storage at their own timestamp.
• Late data for aggregates: region load uses event time with a short watermark (e.g., 30 seconds), and marks regions with many silent devices as "uncertain" instead of treating silence as zero load.
• Clock problems: device clocks drift, so the server records the receive time too, and flags big differences.

4) Sending Commands Safely: Desired State, not "Do X Now"

Instead of fire-and-forget commands, use a desired-state model (like AWS IoT device shadows or Kubernetes):

• We set desired = { version: 18, max_watts: 1200, valid_until: 15:30 } for each device (or group).
• The device applies it and reports applied_version: 18.
• Idempotent: resending version 18 changes nothing, and a device ignores versions older than what it already has.
• Offline devices: when they reconnect, they fetch the latest desired state. If valid_until has passed, it no longer applies, so an old curtailment never kicks in hours late.
• Expiry built in: every curtailment has an end time, so a device that loses connection returns to normal by itself (a safety default).

5) Deep Dive A — Safety and user respect

• Exemptions are checked in the policy engine and on the device: hospital and medical equipment is never curtailed, whatever the server says.
• User override: users can opt out of an event, and the device reports override=true. We respect it and pick other devices to reach the target.
• Limits: max curtailment depth and duration per device per day, and comfort bounds (e.g., the thermostat never above 28°C).
• Gradual actions: step changes in waves (10% of devices at a time) to avoid a rebound when everyone turns back on at once. Stagger the end times too.

6) Deep Dive B — Reconciliation and verification

• For each event, compute the expected reduction: sum of targeted devices × their expected drop.
• Measure the actual reduction from telemetry and region meters.
• Devices that didn't acknowledge, or acknowledged but didn't reduce, get a retry, and if they keep failing they're marked unreliable (and less relied on next time).
• If the region is still over target, the policy engine recruits more devices from the next priority group.
• Everything is logged for audits and for settlement (customers may be paid for participating).

7) Trade-offs & Alternatives

8) Wrap-Up

Ingest telemetry through an authenticated MQTT gateway into Kafka, deduplicate and order it with per-device sequence numbers, and keep device shadows and region load aggregates. Let a policy engine with strict safety rules choose targets, and express commands as versioned, expiring desired states that devices apply idempotently, even after being offline. Roll changes out in waves, and continuously reconcile expected vs actual load reduction, retrying or recruiting more devices when needed.