Replacing the Engine Mid-Flight: A Zero-Downtime Migration Playbook

Replacing a production decision engine in an always-on real-time system requires more than correct code: it requires evidence that behavior remains equivalent while serving traffic continues. This talk presents a practical playbook for migrating an engine without a maintenance window, organized around four invariants—availability, decision behavior, reversibility, and configuration parity. It argues that code review and test fixtures alone cannot establish production equivalence because live traffic diversity, changing configuration, and slow deployment rollback introduce risks that are difficult to expose before cutover.
At the center of the playbook is a migration control loop—observe, compare, stage, switch, and learn—supported by four controls. Shadow execution and replay run the new engine alongside the legacy system, comparing meaningful outcomes rather than raw response bytes. A semantic-diff process normalizes unstable fields and classifies differences as expected, unexpected, or rollout-blocking. Switchable routing separates evaluation from serving authority, so traffic can move between engines without a code deployment. Continuous configuration checks verify both structural parity and behavioral parity, while staged rollout advances by isolated risk and defined gates rather than percentage alone. Each stage includes live-signal monitoring, stop conditions, a tested rollback action, and recovery verification.
The talk also addresses shadow side effects, noisy diffs, telemetry gaps, operational burden, and unclear rollback ownership. It explains that safe migration depends on preventing shadow paths from mutating state, sharing operational responsibility, and preserving the legacy path until the new engine earns authority. Zero downtime carries costs: duplicate systems, capacity headroom, diff storage, triage, rollback rehearsals, and delayed simplification. The reusable method follows eight moves, from defining invariants and inventorying behavior to shadowing, diffing, checking drift, switching, staging, and retiring safely. Its central principle is to make each step observable, bounded, and reversible.