In September 2026 we moved the mailboxes of several companies from a shared-hosting provider onto our own mail platform. The tooling is mature and the plan was simple: copy everything, compare counts, switch DNS. Halfway through, the copy tool told us that nothing was missing on a mailbox where 78 messages had not arrived. This is how that happens, how we found it, and the small comparison script we now run on every migration before we call it done.
TL;DR
- We copied 52 mailboxes across 5 domains with imapsync, in two batches.
- To make re-runs fast we told imapsync to identify messages by their
Message-IDheader. Messages that have no such header were silently skipped, and imapsync's own "found in host1 not in host2" check reported 0, because that check uses the same identification. - Comparing totals did not settle it either: the old server held 28 exact duplicates, so a gap in counts can be harmless or a loss, and you cannot tell which from a number.
- The fix: re-run with imapsync's default identification, then prove the result with an independent, per-message diff keyed on Message-ID or folder + size when the header is absent. Result: 0 missing on every mailbox checked.
The problem
An email migration has one job: every message that was on the old server is on the new one, in the same folder, with the same date and flags. Customers rarely look at old mail on day one, so a gap can stay invisible for months, until someone searches for a contract from last year and it is not there. By then the old server is gone.
The standard tool for IMAP-to-IMAP copies is imapsync. We ran version 2.319 in its official container, once per mailbox, with these options:
imapsync --host1 OLD --user1 "$email" --passfile1 p1 \
--host2 127.0.0.1 --user2 "$email" --passfile2 p2 \
--automap --syncinternaldates \
--useheader Message-Id
--automap maps special folders between servers (the old server's Trash became Deleted Items), --syncinternaldates keeps the original received dates, and nothing is ever deleted on either side, so the copy can be repeated as often as needed. --useheader Message-Id tells imapsync to recognise a message it has already copied by that one header, which is fast and stable across servers.
Why the usual check is not enough
The usual check is a table of message counts per mailbox: old server, new server, difference. Our first batch of 18 mailboxes came out at 8,463 on the old server and 8,452 on the new one. Eleven short. Every one of the 11 turned out to be a duplicate: the same message stored twice on the old server, which imapsync correctly copied once. That is not a loss, but a count table alone cannot tell you so.
The other usual check is imapsync's own summary line, Messages found in host1 not in host2. It said 0 on every mailbox. We believed it for the first batch. On the second batch we should not have.
The trap: messages with no Message-ID
Message-ID is optional in practice. Old messages imported from other clients, some automated notifications, drafts and a few sent items simply do not carry one. With --useheader Message-Id, imapsync has no key for those messages, so it does not copy them. And because its "not in host2" check uses the same key, it cannot see them either. Both the copy and the check are blind to exactly the same messages, so they agree with each other perfectly.
On the largest mailbox of the second batch the first pass transferred 5,357 of 5,435 messages and still printed:
Messages transferred : 5357
Messages found in host1 not in host2 : 0 messages
Detected 0 errors
Zero errors, zero missing, 78 messages short. Some of a gap like that can be duplicates, so the number by itself proved nothing either way. We only learned which messages were really absent, and why, once we stopped asking imapsync and compared the two servers ourselves (the script is below).
What we tried, including a mistake
Our first real mistake was earlier and more mundane. The imapsync container runs as an unprivileged user by default, and our password files were readable by root only (mode 600, as they should be). Every mailbox failed with exit code 66 before copying anything. The fix was to run the container as root rather than loosen the file permissions: credentials stay locked down, and the throwaway container is the thing that adapts.
For the Message-ID gap we first considered patching the missing messages by hand, folder by folder. That is slow and does not scale to 34 mailboxes. Instead we added an --allheaders switch to our wrapper script that simply drops --useheader Message-Id, so imapsync falls back to its default identification, which combines several headers and works for messages without a Message-ID. Re-running is safe because nothing is ever deleted and already-copied messages are skipped. After the re-run the second batch stood at 10,124 on the old server and 10,107 on the new one, and the remaining 17 were exact duplicates.
The fix: an independent, per-message diff
The lesson was that the tool that does the copy should not be the tool that certifies it. We wrote a short Python script that logs in to both servers read-only, walks every selectable folder and builds a key for each message:
- the
Message-IDheader if there is one; - otherwise
NOID:<folder>:<size in bytes>, with folder names normalised so that Trash and Deleted Items, Junk and Junk E-mail, Sent and Sent Items compare as the same folder.
def norm(name):
return (name.replace("Deleted Items", "Trash")
.replace("Junk E-mail", "Junk")
.replace("Sent Items", "Sent"))
# one FETCH per folder: no bodies, no subjects
FIELDS = "(UID RFC822.SIZE FLAGS INTERNALDATE BODY.PEEK[HEADER.FIELDS (MESSAGE-ID)])"
key = mid or f"NOID:{norm(folder)}:{size}"
missing = [k for k in src if k not in dst]
It fetches only sizes, flags, dates and the one header, never subjects or bodies, so it can run against a customer's mailbox without anyone reading their mail. BODY.PEEK matters: a plain BODY fetch would mark every message as read on the source server. For anything missing it prints the folder, size, flags, date and whether the message had a Message-ID, which is usually enough to see the pattern at a glance.
Duplicates fall out naturally: two copies of the same message produce the same key, so they count once on each side and never show up as "missing".
Measured result
| Batch | Mailboxes | Old server | New server | Difference | Missing |
|---|---|---|---|---|---|
| 1 | 18 | 8,463 | 8,452 | 11 duplicates | 0 |
| 2 | 34 | 10,124 | 10,107 | 17 duplicates | 0 |
| Total | 52 | 18,587 | 18,559 | 28 | 0 |
- The per-message diff reported 0 missing on every mailbox that showed any difference in batch 2, and on a spot check of the three largest mailboxes from batch 1, which had been copied with the Message-ID option before we knew about the trap.
- Original received dates were preserved, and every folder came across: inbox, sent, drafts, spam, deleted, archives, and custom folders including ones with Arabic names.
- Mail kept arriving at the old server until each domain's MX record moved. A sweep two days later copied 23 new messages for one domain and the diff again showed 3,209 unique messages on both sides, 0 missing. For that domain we ran the sweep nightly until its MX moved.
Checklist for your own migration
- Copy, never move. Leave the old server untouched and keep it for at least two weeks after the DNS switch.
- Do not identify messages by
Message-IDalone. If you use it for speed on re-runs, do a final pass without it. - Never let the copy tool certify its own work. Run an independent diff with a key that still works when the header is missing.
- Normalise folder names before comparing, or every renamed special folder looks like a loss.
- Treat a count gap as a question, not an answer: duplicates on the source make honest copies look short.
- Fetch metadata with
BODY.PEEKonly. Your check should not change read flags or expose message content. - Sweep again after the MX change, and keep sweeping until no new mail reaches the old server.
- Keep password files root-only and adapt the tooling to them, not the other way round.
Need this?
GeoTech runs a managed business email platform: your own domain, your own DKIM and SPF, webmail and phone setup on our hostnames, and migrations from shared hosting with a per-message proof that nothing was lost before we switch your DNS. Talk to us.