sync: IMAP and JMAP derive disjoint message keys, so engines do not converge across types #407

New Issue

2026-08-12T16:03:45Z

KofTwentyTwo commented

2026-08-12 16:03:45 +00:00

(Migrated from github.com)

Found during the final review of #394 (PR #405). The identity ladder is in place, but the two backends do not meet on it.

Email::derive_message_key resolves EMAILID (RFC 8474) → X-GM-MSGID → (Message-ID + content hash) → folder-scoped surrogate.

JMAP passes its Email object id as email_id, so it enters at tier 1. That is right: a JMAP id is server-assigned and stable across mailboxes within the account, the same guarantee EMAILID gives.
IMAP hardcodes email_id: None, gm_msgid: None at every derivation site, because the FETCH never asks for them. It can only ever reach tier 3 (Message-ID + content hash) or tier 4 (surrogate).

So two engines syncing the same account through different protocols mint disjoint key spaces for the same messages. ADR 0002's headline promise — that N correct clients converge because the servers coordinate — does not hold across engine types today. It holds within a protocol.

This is known-open rather than a defect: nothing regressed, and the ladder is built to pick the stronger tier up for free once the identifiers are fetched.

Work

Request EMAILID in the IMAP FETCH where the server advertises OBJECTID, and X-GM-MSGID where it advertises X-GM-EXT-1, with per-server capability detection.
Blocked on #406 — lifting stored rows to a stronger tier repoints every placement, and the repoint path currently orphans the old message row and strands its decrypted body in messages_fts. Do not land 1 before 406.
Extend the test server to serve both identifiers so the tier-1 path is exercised offline.
Say this out loud in ADR 0002. The ADR presents content-addressed identity as decided; it should record that cross-protocol convergence waits on the FETCH work, so the roadmap does not read as claiming more than is true.

Cross-engine convergence is what #397's ≥3-engine harness is meant to prove; it cannot prove it across protocols until this lands.

Found during the final review of #394 (PR #405). The identity ladder is in place, but the two backends do not meet on it. `Email::derive_message_key` resolves `EMAILID` (RFC 8474) → `X-GM-MSGID` → (`Message-ID` + content hash) → folder-scoped surrogate. - **JMAP** passes its Email object id as `email_id`, so it enters at **tier 1**. That is right: a JMAP id is server-assigned and stable across mailboxes within the account, the same guarantee `EMAILID` gives. - **IMAP** hardcodes `email_id: None, gm_msgid: None` at every derivation site, because the FETCH never asks for them. It can only ever reach **tier 3** (Message-ID + content hash) or **tier 4** (surrogate). So two engines syncing the same account through different protocols mint **disjoint key spaces** for the same messages. ADR 0002's headline promise — that N correct clients converge because the servers coordinate — does not hold across engine types today. It holds within a protocol. This is known-open rather than a defect: nothing regressed, and the ladder is built to pick the stronger tier up for free once the identifiers are fetched. ### Work 1. Request `EMAILID` in the IMAP FETCH where the server advertises `OBJECTID`, and `X-GM-MSGID` where it advertises `X-GM-EXT-1`, with per-server capability detection. 2. **Blocked on #406** — lifting stored rows to a stronger tier repoints every placement, and the repoint path currently orphans the old message row and strands its decrypted body in `messages_fts`. Do not land 1 before 406. 3. Extend the test server to serve both identifiers so the tier-1 path is exercised offline. 4. Say this out loud in ADR 0002. The ADR presents content-addressed identity as decided; it should record that cross-protocol convergence waits on the FETCH work, so the roadmap does not read as claiming more than is true. Cross-engine convergence is what #397's ≥3-engine harness is meant to prove; it cannot prove it across protocols until this lands.

KofTwentyTwo commented

2026-08-14 23:48:44 +00:00

(Migrated from github.com)

Implemented in #416, merged to dev.

IMAP now requests EMAILID where the server advertises OBJECTID and X-GM-MSGID where it advertises X-GM-EXT-1, so it reaches identity tier 1 instead of only tiers 3-4. An IMAP-derived key and a JMAP-derived key for the same object id are now equal — pinned by a convergence test built to fail if either side changes tier or domain label.

The blocker was real: imap-proto 0.16.7's msg_att is a closed alt(...) and async-imap's decoder turns a nom error into a fatal io::Error, so naively adding EMAILID would have killed the connection against exactly the OBJECTID servers this targets. Resolved by extending the hand-rolled raw layer (the precedent imap_raw.rs set for ENABLE/VANISHED), reading the attribute beneath ImapStream via Session's AsMut<T> transport access so it never reaches the decoder.

IDENTITY_SCHEMA_VERSION goes 2 -> 3: cached mail rebuilds and re-syncs on next open rather than re-keying incrementally through the placement repoint arm. Accounts, credentials and rules survive.

Implemented in #416, merged to `dev`. IMAP now requests `EMAILID` where the server advertises `OBJECTID` and `X-GM-MSGID` where it advertises `X-GM-EXT-1`, so it reaches identity tier 1 instead of only tiers 3-4. An IMAP-derived key and a JMAP-derived key for the same object id are now equal — pinned by a convergence test built to fail if either side changes tier or domain label. The blocker was real: `imap-proto 0.16.7`'s `msg_att` is a closed `alt(...)` and `async-imap`'s decoder turns a nom error into a fatal `io::Error`, so naively adding `EMAILID` would have killed the connection against exactly the OBJECTID servers this targets. Resolved by extending the hand-rolled raw layer (the precedent `imap_raw.rs` set for `ENABLE`/`VANISHED`), reading the attribute beneath `ImapStream` via `Session`'s `AsMut<T>` transport access so it never reaches the decoder. `IDENTITY_SCHEMA_VERSION` goes 2 -> 3: cached mail rebuilds and re-syncs on next open rather than re-keying incrementally through the placement repoint arm. Accounts, credentials and rules survive.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/nuncio#407