feat(imap): key messages by their server object id #416

2026-08-14T23:41:38Z

KofTwentyTwo commented

2026-08-14 23:41:38 +00:00

(Migrated from github.com)

IMAP hardcoded email_id: None, gm_msgid: None into derive_message_key, so it could only ever reach identity tier 3 (Message-ID + content hash) or tier 4 (folder-scoped surrogate). JMAP passes its Email object id and enters tier 1. One account synced through both protocols therefore minted disjoint key spaces, and ADR 0002's convergence claim was not true across protocols.

RFC 8474 exists so a server implementing both reports the same object id through both. This makes the keys equal.

What changed

Per-capability negotiation. EMAILID is requested only where the server advertises OBJECTID, X-GM-MSGID only where it advertises X-GM-EXT-1, mirroring the QRESYNC/CONDSTORE ladder in plan_folder_sync. A server advertising neither behaves exactly as today — no behaviour change, no extra round trip.
EMAILID is read through the raw layer. imap-proto 0.16.7's msg_att parser is a closed alt(...) with no catch-all, and async-imap's decoder turns the resulting nom error into a fatal io::Error that closes the session — so naively adding EMAILID to a typed FETCH would break every fetch against exactly the OBJECTID servers this targets. imap_raw now writes the command to and reads the reply off the transport beneath ImapStream, with its own scanner, so the attribute never reaches imap_proto::parser::parse_response. Both preconditions (client read buffer drained; tag prefix that cannot collide with async-imap's IdGenerator) are documented at the call site. X-GM-MSGID already parses, so tier 2 uses the existing Fetch::gmail_msg_id() accessor.
Test server advertises and serves both, so tier 1 is exercised entirely offline.
Docs corrected: derive_message_key's convergence section, the place() doc in imap.rs, and ADR 0002 — stating precisely what does and does not converge rather than overclaiming.

Heads-up: this bumps the identity schema

IDENTITY_SCHEMA_VERSION goes 2 → 3. Lifting stored rows to a stronger tier changes every message_key, so cached mail rebuilds on next open via the existing reconcile_schema_version machinery. Config and credentials survive. This is deliberately not left to the placement upsert's repoint arm — that reap is a safety net, not a migration mechanism.

Tests

Offline throughout. An OBJECTID server yields IdentitySource::EmailId; the same message fetched from a second folder yields the same message_key; a Gmail-shaped server yields GmailMsgId; a server advertising neither is unchanged; and an IMAP-derived key from an EMAILID equals a JMAP-derived key from the same object id, pinned against a golden so it fails if either side changes tier or domain label.

Gate green: cargo fmt --all -- --check, cargo check-all, cargo test-all (704 passed, 0 failed), plus RUST_TEST_THREADS=2 cargo test-all three times with zero failures.

Closes #407

IMAP hardcoded `email_id: None, gm_msgid: None` into `derive_message_key`, so it could only ever reach identity tier 3 (`Message-ID` + content hash) or tier 4 (folder-scoped surrogate). JMAP passes its Email object id and enters tier 1. One account synced through both protocols therefore minted disjoint key spaces, and ADR 0002's convergence claim was not true across protocols. RFC 8474 exists so a server implementing both reports the *same* object id through both. This makes the keys equal. ## What changed - **Per-capability negotiation.** `EMAILID` is requested only where the server advertises `OBJECTID`, `X-GM-MSGID` only where it advertises `X-GM-EXT-1`, mirroring the QRESYNC/CONDSTORE ladder in `plan_folder_sync`. A server advertising neither behaves exactly as today — no behaviour change, no extra round trip. - **`EMAILID` is read through the raw layer.** `imap-proto` 0.16.7's `msg_att` parser is a closed `alt(...)` with no catch-all, and `async-imap`'s decoder turns the resulting nom error into a fatal `io::Error` that closes the session — so naively adding `EMAILID` to a typed FETCH would break every fetch against exactly the OBJECTID servers this targets. `imap_raw` now writes the command to and reads the reply off the transport beneath `ImapStream`, with its own scanner, so the attribute never reaches `imap_proto::parser::parse_response`. Both preconditions (client read buffer drained; tag prefix that cannot collide with `async-imap`'s `IdGenerator`) are documented at the call site. `X-GM-MSGID` already parses, so tier 2 uses the existing `Fetch::gmail_msg_id()` accessor. - **Test server** advertises and serves both, so tier 1 is exercised entirely offline. - **Docs corrected:** `derive_message_key`'s convergence section, the `place()` doc in `imap.rs`, and ADR 0002 — stating precisely what does and does not converge rather than overclaiming. ## Heads-up: this bumps the identity schema `IDENTITY_SCHEMA_VERSION` goes 2 → 3. Lifting stored rows to a stronger tier changes every `message_key`, so **cached mail rebuilds on next open** via the existing `reconcile_schema_version` machinery. **Config and credentials survive.** This is deliberately not left to the placement upsert's repoint arm — that reap is a safety net, not a migration mechanism. ## Tests Offline throughout. An OBJECTID server yields `IdentitySource::EmailId`; the same message fetched from a second folder yields the same `message_key`; a Gmail-shaped server yields `GmailMsgId`; a server advertising neither is unchanged; and an IMAP-derived key from an `EMAILID` equals a JMAP-derived key from the same object id, pinned against a golden so it fails if either side changes tier or domain label. Gate green: `cargo fmt --all -- --check`, `cargo check-all`, `cargo test-all` (704 passed, 0 failed), plus `RUST_TEST_THREADS=2 cargo test-all` three times with zero failures. Closes #407

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/nuncio#416