Start typing to search this publication.
techcirkle
Open menu

Subscribe to techcirkle

Get new posts delivered straight to your inbox.

Shadow Mode: Migrating Live Customers to a New Billing System

The build is not the risky part of a billing project. Cutover is. Here is the parallel-run technique that keeps customers from noticing.

techcirkle
6 min read
Cover image for Shadow Mode: Migrating Live Customers to a New Billing System
Invoice management and billing software on a computer screen
Invoice management and billing software on a computer screen

Billing projects do not usually fail during the build. They fail in the two weeks after cutover, when invoices go out and a few hundred customers discover that their amount changed.

The customers you are migrating are mid-contract, mid-cycle and occasionally mid-dispute. Unlike almost any other migration, there is no soft launch, no percentage rollout to a tolerant cohort, and no quiet rollback — an invoice that has been sent has been sent.

Shadow mode is how you de-risk it. The technique is simple to describe and the discipline is where projects fail.

The mechanism

The new billing system computes everything and sends nothing.

                 ┌─ old system ──► invoices ──► customers
usage events ──┤
                 └─ new system ──► invoices ──► diff store (sent nowhere)

Both systems consume the same event stream for the same accounts. At the end of each cycle, you diff them line by line. Not invoice totals — line items, with their quantities, rates, discounts, tax and rounding.

Totals matching is not sufficient. We have seen a cycle where every total matched and two line items had been swapped between a usage charge and a credit, cancelling out. Compare at the line level or you are not comparing.

What to diff on

account_id
invoice_period
line_item_type
quantity
unit_rate
amount_minor
tax_amount_minor
currency

Write the diff output somewhere queryable, with the stable identity of each line item so you can track the same discrepancy across cycles. A CSV emailed to the team on the first of the month is not a diff pipeline, and you will want to ask "is this the same problem as last month" repeatedly.

Every discrepancy is a finding, not a bug to be squashed

This is the part teams get wrong. The instinct, on seeing a mismatch, is to change the new system until it matches the old one. Sometimes that is right. Often it is not, and you need to know which before deciding.

Discrepancies fall into four categories:

  • New system bug. Fix the new system.

  • Undocumented behaviour in the old system. A rounding convention, a grandfathered discount, a proration rule nobody wrote down. Decide deliberately whether to replicate it. Some of these are features; some are bugs you have been shipping for three years.

  • Old system bug. You have been under- or over-billing. This needs a commercial decision, not an engineering one, and it should go to whoever owns revenue before you touch code.

  • Manual adjustment. Somebody intervened in 2024 and the new system has no way of knowing. These should be modelled explicitly as adjustments, not reverse-engineered into rating logic.

In practice that second category is the largest, and surfacing it is the main value of the whole exercise. The old system's real behaviour is only partly in its code; the rest is in configuration drift and people's habits.

How long to run it

Two to three complete billing cycles, with the last one clean.

"Clean" means every discrepancy is either resolved or explicitly accepted with a documented reason. A cycle with four open unexplained diffs is not clean, and shipping on it means you will debug those four in production with customers attached.

For monthly billing that is two to three months of overlap. It typically works out at twenty to thirty percent of total project effort, which looks like a lot on a plan and is the cheapest insurance in the project. Teams that compress it spend the same time afterwards, under pressure, with an angry support queue.

Annual contracts are the awkward case — you cannot wait a year. The workable approach is to recompute historical annual invoices from the event store in backfill mode and diff against what was actually sent, which gives you the same signal without the wait, provided your new system is replayable.

Replayability is the prerequisite

Shadow mode only works if the new system can compute an invoice for an arbitrary past period from stored events and the contract as it existed then. That means:

  • A rating function with no side effects and no database reads inside it.

  • Contracts stored as immutable versions with validity intervals, never mutated in place.

  • An event store that holds what happened and no computed prices.

If the new system cannot recompute history, you cannot shadow it, and you are reduced to testing against synthetic data — which tests your assumptions rather than your customers' actual contracts. That distinction is usually where the surprises live.

The cutover itself

When the last cycle is clean:

  1. Freeze configuration changes on both systems.

  2. Flip which system sends. One switch, one owner, one rollback plan.

  3. Keep the old system readable, not running, for at least a year.

  4. Watch the support queue rather than the dashboard for the first week. Customers report billing problems to support, not to your metrics.

Step three matters more than it looks. Somebody will ask about an old invoice, and "we migrated and cannot show you that" is a bad answer to give a customer or an auditor.

Where generation helps

The diff harness, the fixture generation from historical invoices, the backfill scripts and the reconciliation reporting are all well-specified, schema-heavy work and genuinely fast to generate. That is a meaningful chunk of the migration effort.

What should stay human: deciding what each discrepancy means. That is a commercial judgement about contracts, and the whole point of the exercise is that the information needed to make it is not written down anywhere.

The wider build-versus-buy framing and cost ranges are in TechCirkle's guide to small business software for billing. The architectural side, including replayability, sits in our custom software development work.

Frequently Asked Questions

What is shadow mode in a billing migration?

Running the new billing system alongside the old one on the same event stream, computing every invoice but sending none of them, then diffing the two outputs line by line each cycle. It surfaces undocumented behaviour in the old system before any customer sees a changed invoice.

How long should a billing system run in parallel?

Two to three complete billing cycles, with the final one clean — meaning every discrepancy is either fixed or explicitly accepted with a documented reason. For monthly billing that is two to three months, typically twenty to thirty percent of total project effort.

Should I diff invoice totals or line items?

Line items, including quantity, unit rate, discount, tax and rounding. Totals can match while individual lines are wrong in offsetting ways, which we have seen happen. Comparing totals gives false confidence at exactly the moment you need real confidence.

What do I do when the old system turns out to be wrong?

Escalate it as a commercial decision rather than fixing it quietly. If you have been under-billing or over-billing, whoever owns revenue needs to decide whether to correct, backdate or absorb it. That is not an engineering call, and making it quietly in code creates a worse problem.

How do you shadow-test annual billing without waiting a year?

Recompute historical annual invoices from the event store in backfill mode and diff against what was actually sent. This requires the new system to be replayable — a pure rating function, versioned contracts and an event store without stored prices — but it gives the same signal immediately.

Can I skip the parallel run if the new system has good test coverage?

Tests validate your assumptions about contracts. The parallel run validates the contracts themselves, including the undocumented behaviour that is not in anyone's specification. The two are not substitutes, and the second is where the surprises are.

Subscribe to techcirkle