How sync-up migration works

This article explains the design of the follow-up pass VaultMe runs after an initial migration, and the limits that design carries.

What is a sync-up migration?

A sync-up migration (delta migration) is a follow-up pass over a migration VaultMe has already run. It copies source items that appeared since the previous run, and it applies source-side changes to destination copies VaultMe created earlier. It is not a second migration and not a fresh attempt at the same one — it continues the migration that already exists, and it only ever adds to or updates the destination content that migration produced. Where this pass fits into the wider operating model is sketched in "How the VaultMe platform works".

Every migration gets at least one sync-up pass, and VaultMe starts the first one automatically as soon as the initial run finishes. That first pass exists because an initial run takes time to complete: while VaultMe was working through the source, mail kept arriving and account holders kept editing files, so there is nearly always recent content the initial run could not have included. Later passes work the same way from a different starting point — each one copies what changed in the source since the previous pass completed, rather than looking at the whole source again.

Three things a sync-up pass is deliberately not:

• It is not continuous synchronization. Nothing watches the source in between. A pass is a discrete run with a beginning and an end, and until the next one starts, changes in the source simply accumulate.
• It is not a standing backup of the source. A pass only ever adds source-side changes to the destination. It does not restore destination content that someone deleted, and it stops being available once the migration's own record of what it copied expires.
• It is not a mirror. A pass reads the source and writes the destination, and that is the only direction in which anything flows. It does not compare the two sides and make them match.

The job is narrow and finite: bridge the gap between the initial run and cutover, the point at which the customer stops relying on the source and starts relying on the destination. Sync-up exists so that gap can last as long as the project needs without the destination going stale in the meantime. Planning the switch itself — routing mail, retiring the source, and the customer-side follow-up that goes with it — is a separate exercise, covered in "Planning the cutover and domain switch".

Why is it designed this way?

Every decision in sync-up migration follows from one fact about the initial run: it works from an index built at the beginning. VaultMe lists the discoverable items in the connected source and then processes that list, as described in "How the VaultMe platform works". Anything that appears in the source after the list closes is not part of that run. The initial migration is therefore a snapshot of the source as it stood near the start, not as it stands when the run finishes.

That gap is structural rather than a defect, and no better implementation would remove it. VaultMe is built so the source stays usable while a migration runs — nobody has to stop sending mail or freeze their files — and a source in active use is a source that keeps changing. The very property that makes migrations non-disruptive guarantees that content will accumulate behind the snapshot, and the longer the run, the more of it there is.

Given a gap that is certain to exist, the follow-up has to close it, and close it cheaply. Copying the whole source again would take roughly as long as the initial run and would grow the destination each time, so the follow-up is incremental instead: it works out what is new or changed and touches only that.

That also rules out the obvious alternative of simply running the migration a second time. VaultMe does not inspect existing destination content to work out what is already there. For every source item in scope it writes a fresh destination copy, whether or not an identical item is already sitting in the destination from an earlier run — and it makes no difference whether that earlier content was put there by VaultMe or by something else. Running the same source into the same destination again therefore produces a second, independent set of copies rather than filling in the difference. Catching up has to happen inside the original migration, using the record that migration kept of what it wrote.

Keeping the two sides in step indefinitely is ruled out from the other end. VaultMe is a migration tool, and the point of a migration is a destination that stands on its own. After cutover, people work in the destination: they file mail, reorganize folders, edit documents. Continuing to push source-side changes into a destination that is in active use means overwriting real work whenever the same item changed on both sides, and the longer it continues, the more often that happens. The window is bounded on purpose, because the correct end state is two independent sides rather than two sides held together.

Three properties come out of that reasoning, and everything a sync-up pass does or refuses to do follows from them: a pass works from the original migration's own record of what it copied, it carries source-side changes only, and it stays available for a limited time rather than forever.

How does the index make sync-up work?

The mechanism behind all of this is the migration index. Beyond listing the items to process, it records a pairing: for every source item VaultMe copied, the specific destination copy VaultMe created for it. The initial run builds those pairings as it works, and every sync-up pass that completes updates them, adding pairings for items it has just copied and refreshing what the index knows about the ones it already had.

Everything a pass decides, it decides from those pairings. To find out whether a source item has already been copied, VaultMe looks it up in the index rather than searching the destination for something that resembles it. To find out whether a copied item needs updating, it compares the source item against what the index recorded. VaultMe deliberately does not rely on provider-side signals such as message identifiers, file names, or a provider's own feed of recent changes, because those signals are not reliable enough to base matching on — a renamed file is still the same file, and its name is a poor way to recognize it.

Deciding from its own record rather than from the state of the destination gives sync-up three properties worth knowing before a pass runs:

• Unchanged items are never copied twice. When the index records a source item as migrated and unchanged, a pass leaves it alone. An extra pass over a source where little has happened finishes quickly and adds almost nothing to the destination.
• A deleted destination copy is not recreated. If a migrated message or file is deleted on the destination side, the index still records its source item as migrated and unchanged, so no later pass brings it back. Deleting destination copies in the expectation that a pass will restore them does not work.
• A pass is not a reconciliation of the two sides. VaultMe does not compare the source and the destination and resolve the differences. Only changes on the source side flow, and only forward.

Those pairings belong to one migration and are not shared. A different migration — one that migrates the same source into the same destination, say — has its own index and knows nothing about the copies the first one made. This is why a migration cannot be resumed by starting another one, and why the life of the index sets the life of sync-up.

What does a sync-up copy — and what does it leave alone?

A pass looks at the source, works out what has changed since the previous run, and applies those changes to the destination. In practice that splits into two kinds of work: creating destination copies for source items that were not there before, and updating destination copies whose source items changed. Concretely, a pass carries forward:

• New source items. Each one gets a destination copy placed by the same rules the initial run used — the same content organization method, the same destination structure, the same handling of naming conflicts. A message that arrived a week after the initial run ends up where it would have gone had it arrived a week before.
• Changed content. When the content of an already-copied source item changes, VaultMe updates its paired destination copy instead of adding a second one alongside it.
• Relocations, where the destination allows them. If a copied item is moved somewhere else in the source — into a different folder, for example — VaultMe applies the same relocation to its paired copy, as far as the destination provider's capabilities and the migration's organization rules permit.
• Updates to supported attributes. Source-side changes to metadata VaultMe can migrate, such as a display name or a status, are applied to the paired copy wherever the destination supports that attribute.
• Missing structure. A pass relies on the destination structure the initial run created. If part of it is no longer there, VaultMe attempts to recreate what it needs before writing.

What a pass leaves alone matters just as much. Deleting an item in the source does not delete its paired destination copy: the copy stays where it is, which is one of the ways the two sides drift apart over a long project. Content that only ever existed in the destination is never touched at all, because a pass reads nothing but the source. A message delivered straight to the destination after mail routing was switched over, for instance, has no source counterpart, so no pass has any reason to look at it.

The one thing a pass will overwrite is destination-side editing of a copy it made. When a source-side change is applied to a paired destination copy, concurrent changes made to that copy on the destination side are replaced by the incoming version. Where the destination keeps a revision history for that kind of item, the overwritten version can sometimes be recovered from it; otherwise those edits are lost.

Together those rules make one common cutover shape safe and two others unsafe. Switching mail delivery to the destination and then running a final pass works: the pass sweeps up the last of what reached the source before the switch and leaves everything delivered to the destination after it untouched. Forwarding mail from the source to the destination while a pass is still running does the opposite: each message that arrives during the overlap lands in the destination twice — once by forwarding, and once when the pass copies the source's own copy of it. Reaching for a new migration of the same user accounts to catch up does not work either, because a new migration writes its own fresh copy of everything.

What limits a sync-up?

The first limit is time. VaultMe keeps a migration's index for 30 days after its most recent completed run — the initial migration, or the latest sync-up pass, whichever finished last. A migration that cuts over within a few weeks never runs into that boundary. Beyond it, each pass that completes inside the current window refreshes the index and starts a new 30 days, so a project that completes a pass at least once a month keeps its ability to catch up. Let the window elapse and the index is deleted, along with the pairings that make an incremental pass possible.

There is no way back from that. Once the index is gone, catching up means running a new migration of the same source into the same destination, and because that migration has no knowledge of the copies the previous one made, it writes a complete second set. On a project that stretches over months, keeping the window open is therefore a real scheduling obligation. It is also worth being clear about what the refresh is for: it holds the bridge open for as long as the project needs, not indefinitely, and the reasons are in "Why is it designed this way?".

The remaining limits are practical:

• Both connections have to be live. A pass reads the source and writes the destination, so it cannot run unless both sides are still connected to VaultMe. Connections can drop for reasons outside VaultMe's control, such as a password reset or a change to MFA policy; what a drop means and how it is resolved is covered in the disconnection chapter of "How VaultMe authentication works".
• An extra pass runs for a batch, not for one account. The control for starting an additional pass acts on the batch rather than on a single user account you single out, and it covers the accounts in that batch that are not currently processing — during a partly finished batch, the ones that have already completed. Running a pass for one selected account is not available in the app; support can do it on request at support@vaultme.com.
• An unusually large volume of change may be cancelled. Additional passes are meant to carry a reasonable amount of catch-up work. Where the volume of source-side change goes well beyond that, support can cancel the pass.
• A reasonable pass is not charged separately. Running one adds no fee of its own; it is part of the migration that was already paid for. How that fits the wider pricing model is in "Pricing and payment".

What are the trade-offs?

The design buys a specific set of things:

• The source stays in use throughout. Because whatever changes during the initial run is picked up afterwards, no one has to stop working in the source while VaultMe copies it.
• Catching up is cheap. A pass is incremental, so one run over a quiet source finishes quickly, and items that have not changed are not written to the destination a second time.
• No second migration is needed. Reusing the original migration's index means catch-up does not involve running — and paying for — a fresh migration, and does not leave duplicates behind.
• A long runway with a short cutover becomes realistic. The bulk of the copying can happen weeks before anyone switches over, leaving a final pass rather than a full migration to run at the moment that matters most.

And it gives up a specific set of things:

• It is not a reconciliation. A destination copy someone deleted stays gone, and a source item someone deleted stays in the destination. Neither side is made to match the other.
• Destination-side work can be overwritten. Anything edited on a copy VaultMe made is at risk until the last pass has run, which is the reason not to settle into the destination before then.
• The window has to be kept alive. On a long project, completing a pass inside every 30-day window is somebody's job, and missing it is not a recoverable mistake.

One combination has to be avoided outright: forwarding mail from the source to the destination while sync-up passes are still running. Both mechanisms put the same message in the destination — forwarding delivers a copy as it arrives, and the next pass copies the source's own copy of it, because a pass reads the source without checking whether an equivalent message is already in the destination. The result is two copies of everything that arrives during the overlap. Pick one: either forward and stop running passes, or keep running passes and leave delivery pointed at the source until the switch. Sequencing mail flow around a cutover belongs to "Planning the cutover and domain switch".

The same logic decides when to stop. Once people are working in the destination, every further pass has more opportunity to overwrite something real, while the value of what it carries forward keeps falling as the source goes quiet. The intended ending is to stop running passes soon after cutover, let the two sides become independent, and retire the source on the customer's own schedule.