When the logs said the lights went off and the radio said otherwise

A success that was not one

Two lines from the same second of the same log file, found while reading through an unremarkable morning's output:

08:25:35.954   Full Court light confirmed off
08:25:36       genOnOff.off ... timed out after 10000ms

The application reported success 0.85 seconds before the radio gave up trying.

Nothing crashed and nothing alerted. The system had been treating an acknowledged command as a completed one, and an acknowledgement only proves that the message was accepted for delivery. It proves nothing about the light. In the failure case the floodlights stayed on and the log insisted they were off.

Positive confirmation replaced acknowledgement

Command acknowledgement was removed as a success signal entirely. The system now waits for the device to republish its own state, requires that report to be strictly newer than the command that caused it, and raises on timeout or on disagreement. A light is off when the light says it is off.

The confirmation timeout is 12 seconds, chosen deliberately to sit above zigbee2mqtt's own 10 second ZCL timeout. If the two were closer, the confirmation could expire while the transport was still legitimately retrying, and the system would trade false successes for false failures.

Failures that are reported nowhere else

Some radio-level problems appear only on the zigbee2mqtt bridge logging topic and on no device topic at all. The system subscribes to it, so a class of failure that is otherwise completely silent becomes visible.

The lux gate fails closed

Ambient light readings decide whether the floodlights are needed. Readings go stale, and a stale reading is more dangerous than no reading, because it looks valid. Freshness is tiered at 15, 20 and 30 minutes with per-sensor calibration, and when the data is too old the system refuses to act rather than guessing from the last number it saw.

Choices made against the site, not the datasheet

• Zigbee on channel 11, chosen against the venue's Wi-Fi on channel 9, so the two do not contend for the same part of the 2.4 GHz band.
• Inrush staggering at T-15, T-10 and T, so banks of floodlights do not strike simultaneously.
• Multi-process-safe rotating logs with interprocess file locking and peer-rotation detection, and atomic status snapshots written through os.replace, so a reader never sees a half-written file.
• systemd-managed with automatic restart, running as a non-root user.

The client can check the devices without a VPN

Device state is published over MQTT and surfaced on a small access-gated web dashboard the client opens in a browser. The alternative was installing a VPN client on their machines, which works right up until the one time they need to check something and are not sitting at that particular desk. A page behind an access gate costs nothing to use and nothing to support.

The transport abstraction, proven rather than asserted

The system moved from Wi-Fi relays to Zigbee in July 2026 after the relays began failing intermittently. In the migration commit the main application file has a diff of zero lines. The abstraction was load-bearing, and swapping the entire radio layer demonstrated it. No unrecovered switching failure since.

AI-assisted, human-accountable. AI tools contributed significantly to the implementation described here. The specification, the review of every change, the production operation and the accountability are mine.