Multi-Tenant Migration with Progressive Cutover

This migration looked ordinary from far away and pretty unforgiving up close. I had to move 150 B2B SaaS tenants off a legacy monolithic database onto new infrastructure, around 10 million rows across more than 30 tables, enough foreign-key complexity that getting table order wrong could cause referential drift or a broken load, and in practice a roughly two-year effort. The business expectation was simple and not negotiable: no downtime, minimal customer impact, and immediate rollback if anything looked off.

The hardest constraint dictated the core design. These were enterprise tenants running 24/7, so both systems had to be live at the same time, if someone hit a major problem on the new version, they had to fall back to the old one with zero operational delay. "We'll fix it in an hour" is not acceptable at that tier. So I built a dual-write layer: every write went to both databases during transition, with reads switched per tenant behind a feature flag, so any tenant could roll back instantly.

I would rather tell the parts that went wrong than pretend it was clean. My first cut wrote records one at a time, and that stalled hard once a big tenant's row counts grew. The fix was a bulk-create API, but it did not exist on the destination yet, so I made the case to the destination team with evidence from the stall, got it onto their roadmap, and used it. And one wave came out horribly wrong and had to be redone: older builds stored first and last name in a single field, a later version split them in two, and the legacy data for the internal users (drivers, internal-facing, so nobody had cared to clean it) was never reconciled. The new UI surfaces names right on the screen, so I could not leave the mess; I purged that migrated tenant, fixed the handling, and re-ran it. The honest root cause is that I did not know the entire database, I designed from the recent, well-formed sample data, and older records were structured differently. First and last name was just one of many such surprises, and a large share of the two years went into exactly that: upstream data hygiene nobody had done on time, surfacing one wave at a time.

The rest of the machinery held. I split tenants into 20 risk-ordered waves so every hard move was informed by easier ones. An automated dependency graph produced FK-safe load order instead of a hand-written sequence. Reconciliation, row counts, checksums, business-rule checks, ran in parallel and paused a wave when drift crossed a threshold; the drift it did catch came mostly from messy real-world data like emojis and special characters showing up where clean text was expected, not from design gaps.

There were real costs. Dual-write is awkward and doubles the write path. Temporary routing logic is awkward. Reconciliation frameworks take time to build. I gave up short-term elegance, because the business did not need elegance, it needed continuity, and instant fallback for tenants who cannot afford to go dark. The result was zero downtime, incidents down from 20 to 2 during the migration, MTTR down 30%, and every tenant migrated on schedule with rollback maintained throughout. The lesson in one line: the migration got safer the moment I stopped trying to make it look simple and started building explicit controls around the parts that were never simple at all.