GeoTech

Engineering / Case study

Moving 52 mailboxes with zero loss: the missing Message-ID trap and a per-message diff

GeoTech Engineering · · 7 min read

EmailIMAPimapsyncMigration

In September 2026 we moved the mailboxes of several companies from a shared-hosting provider onto our own mail platform. The tooling is mature and the plan was simple: copy everything, compare counts, switch DNS. Halfway through, the copy tool told us that nothing was missing on a mailbox where 78 messages had not arrived. This is how that happens, how we found it, and the small comparison script we now run on every migration before we call it done.

52mailboxes
18,587messages on the old server
0messages missing
28exact duplicates, merged

TL;DR

The problem

An email migration has one job: every message that was on the old server is on the new one, in the same folder, with the same date and flags. Customers rarely look at old mail on day one, so a gap can stay invisible for months, until someone searches for a contract from last year and it is not there. By then the old server is gone.

The standard tool for IMAP-to-IMAP copies is imapsync. We ran version 2.319 in its official container, once per mailbox, with these options:

imapsync --host1 OLD --user1 "$email" --passfile1 p1 \
         --host2 127.0.0.1 --user2 "$email" --passfile2 p2 \
         --automap --syncinternaldates \
         --useheader Message-Id

--automap maps special folders between servers (the old server's Trash became Deleted Items), --syncinternaldates keeps the original received dates, and nothing is ever deleted on either side, so the copy can be repeated as often as needed. --useheader Message-Id tells imapsync to recognise a message it has already copied by that one header, which is fast and stable across servers.

Why the usual check is not enough

The usual check is a table of message counts per mailbox: old server, new server, difference. Our first batch of 18 mailboxes came out at 8,463 on the old server and 8,452 on the new one. Eleven short. Every one of the 11 turned out to be a duplicate: the same message stored twice on the old server, which imapsync correctly copied once. That is not a loss, but a count table alone cannot tell you so.

The other usual check is imapsync's own summary line, Messages found in host1 not in host2. It said 0 on every mailbox. We believed it for the first batch. On the second batch we should not have.

The trap: messages with no Message-ID

Message-ID is optional in practice. Old messages imported from other clients, some automated notifications, drafts and a few sent items simply do not carry one. With --useheader Message-Id, imapsync has no key for those messages, so it does not copy them. And because its "not in host2" check uses the same key, it cannot see them either. Both the copy and the check are blind to exactly the same messages, so they agree with each other perfectly.

On the largest mailbox of the second batch the first pass transferred 5,357 of 5,435 messages and still printed:

Messages transferred                    : 5357
Messages found in host1 not in host2    : 0 messages
Detected 0 errors

Zero errors, zero missing, 78 messages short. Some of a gap like that can be duplicates, so the number by itself proved nothing either way. We only learned which messages were really absent, and why, once we stopped asking imapsync and compared the two servers ourselves (the script is below).

What we tried, including a mistake

Our first real mistake was earlier and more mundane. The imapsync container runs as an unprivileged user by default, and our password files were readable by root only (mode 600, as they should be). Every mailbox failed with exit code 66 before copying anything. The fix was to run the container as root rather than loosen the file permissions: credentials stay locked down, and the throwaway container is the thing that adapts.

For the Message-ID gap we first considered patching the missing messages by hand, folder by folder. That is slow and does not scale to 34 mailboxes. Instead we added an --allheaders switch to our wrapper script that simply drops --useheader Message-Id, so imapsync falls back to its default identification, which combines several headers and works for messages without a Message-ID. Re-running is safe because nothing is ever deleted and already-copied messages are skipped. After the re-run the second batch stood at 10,124 on the old server and 10,107 on the new one, and the remaining 17 were exact duplicates.

The fix: an independent, per-message diff

The lesson was that the tool that does the copy should not be the tool that certifies it. We wrote a short Python script that logs in to both servers read-only, walks every selectable folder and builds a key for each message:

def norm(name):
    return (name.replace("Deleted Items", "Trash")
                .replace("Junk E-mail", "Junk")
                .replace("Sent Items", "Sent"))

# one FETCH per folder: no bodies, no subjects
FIELDS = "(UID RFC822.SIZE FLAGS INTERNALDATE BODY.PEEK[HEADER.FIELDS (MESSAGE-ID)])"
key = mid or f"NOID:{norm(folder)}:{size}"
missing = [k for k in src if k not in dst]

It fetches only sizes, flags, dates and the one header, never subjects or bodies, so it can run against a customer's mailbox without anyone reading their mail. BODY.PEEK matters: a plain BODY fetch would mark every message as read on the source server. For anything missing it prints the folder, size, flags, date and whether the message had a Message-ID, which is usually enough to see the pattern at a glance.

Duplicates fall out naturally: two copies of the same message produce the same key, so they count once on each side and never show up as "missing".

Measured result

BatchMailboxesOld serverNew serverDifferenceMissing
1188,4638,45211 duplicates0
23410,12410,10717 duplicates0
Total5218,58718,559280

Checklist for your own migration

  1. Copy, never move. Leave the old server untouched and keep it for at least two weeks after the DNS switch.
  2. Do not identify messages by Message-ID alone. If you use it for speed on re-runs, do a final pass without it.
  3. Never let the copy tool certify its own work. Run an independent diff with a key that still works when the header is missing.
  4. Normalise folder names before comparing, or every renamed special folder looks like a loss.
  5. Treat a count gap as a question, not an answer: duplicates on the source make honest copies look short.
  6. Fetch metadata with BODY.PEEK only. Your check should not change read flags or expose message content.
  7. Sweep again after the MX change, and keep sweeping until no new mail reaches the old server.
  8. Keep password files root-only and adapt the tooling to them, not the other way round.

Need this?

GeoTech runs a managed business email platform: your own domain, your own DKIM and SPF, webmail and phone setup on our hostnames, and migrations from shared hosting with a per-message proof that nothing was lost before we switch your DNS. Talk to us.