fix(store): deterministic message surrogate identity (C1) #337

2026-07-31T18:44:24Z

KofTwentyTwo commented

2026-07-31 18:44:24 +00:00

(Migrated from github.com)

Closes #281 (WS-B1) — the message-identity fix (audit Critical C1). GATE for the Message-shape work.

Problem

messages.id TEXT PRIMARY KEY was the sole key and Email.id = "imap-uid-{uid}", but IMAP UIDs are unique only per mailbox+UIDVALIDITY. The same UID in two folders (INBOX 5 / Sent 5) or two accounts collapsed to imap-uid-5 and silently overwrote via INSERT OR REPLACE. The id shape also leaked internal encoding on the wire.

Change

Surrogate id = SHA-256 (existing sha2 dep, via nuncio-core) over length-prefixed (account_id, folder_id, uid_validity, remote_id), hex-encoded. Deterministic → a re-sync recomputes the same id and upserts the same row (fire-once filter guard holds). Email::surrogate_id lives on the model.
New columns messages.remote_id/uid_validity (NOT NULL DEFAULT ''), backfill-safe ALTER TABLE migration mirroring ensure_accounts_smtp_columns (legacy rows seeded remote_id = id), plus UNIQUE(account_id, folder_id, uid_validity, remote_id) — the real correctness guard.
IMAP/JMAP populate remote_id/uid_validity and set Email.id to the surrogate. JMAP uses a "jmap" UIDVALIDITY sentinel (documented).
Outbox executor builds RemoteMutationSpec from the row's remote_id/uid_validity columns (stops parsing imap-uid-); the IMAP backend addresses by remote_id and still SELECT-verifies UIDVALIDITY before mutating (#252 safety preserved). JMAP addresses by object id via remote_id.
No proto change — Message.id stays a string, only its value is now opaque; nuncio.proto + golden untouched.

Regression tests

message_identity_prevents_cross_folder_and_cross_account_collision (nuncio-store) — the exact C1 collision: same UID across folders/accounts persist as distinct rows; re-sync upserts one row; columns round-trip.
surrogate_id_distinguishes_the_cross_folder_and_cross_account_collision + surrogate_id_is_deterministic_and_opaque (nuncio-core).
Existing outbox/sync/JMAP wiremock e2e updated to assert addressing via remote_id and opaque ids.

Gate (3 separate commands, all green, 0 ignored)

cargo fmt --all -- --check — clean
cargo check-all (clippy --all-targets --workspace -D warnings) — 0 warnings
cargo test-all — all suites pass, 0 failed, 0 ignored (nunciod lib 123; nuncio-store, nuncio-mail, nuncio-core, nuncio-filter, nuncio-cli, nuncio-cal, nuncio-contacts, and all e2e suites green). Pre-commit hook re-ran the full gate on commit.

No fabricated success; all protocols mocked; no live network.

Closes #281 (WS-B1) — the message-identity fix (audit Critical C1). **GATE** for the Message-shape work. ## Problem `messages.id TEXT PRIMARY KEY` was the sole key and `Email.id = "imap-uid-{uid}"`, but IMAP UIDs are unique only per mailbox+UIDVALIDITY. The same UID in two folders (INBOX 5 / Sent 5) or two accounts collapsed to `imap-uid-5` and silently overwrote via `INSERT OR REPLACE`. The id shape also leaked internal encoding on the wire. ## Change - **Surrogate id** = `SHA-256` (existing `sha2` dep, via `nuncio-core`) over length-prefixed `(account_id, folder_id, uid_validity, remote_id)`, hex-encoded. Deterministic → a re-sync recomputes the same id and upserts the same row (fire-once filter guard holds). `Email::surrogate_id` lives on the model. - **New columns** `messages.remote_id`/`uid_validity` (`NOT NULL DEFAULT ''`), backfill-safe `ALTER TABLE` migration mirroring `ensure_accounts_smtp_columns` (legacy rows seeded `remote_id = id`), plus `UNIQUE(account_id, folder_id, uid_validity, remote_id)` — the real correctness guard. - **IMAP/JMAP** populate `remote_id`/`uid_validity` and set `Email.id` to the surrogate. JMAP uses a `"jmap"` UIDVALIDITY sentinel (documented). - **Outbox executor** builds `RemoteMutationSpec` from the row's `remote_id`/`uid_validity` columns (stops parsing `imap-uid-`); the IMAP backend addresses by `remote_id` and still SELECT-verifies UIDVALIDITY before mutating (#252 safety preserved). JMAP addresses by object id via `remote_id`. - **No proto change** — `Message.id` stays a `string`, only its value is now opaque; `nuncio.proto` + golden untouched. ## Regression tests - `message_identity_prevents_cross_folder_and_cross_account_collision` (nuncio-store) — the exact C1 collision: same UID across folders/accounts persist as distinct rows; re-sync upserts one row; columns round-trip. - `surrogate_id_distinguishes_the_cross_folder_and_cross_account_collision` + `surrogate_id_is_deterministic_and_opaque` (nuncio-core). - Existing outbox/sync/JMAP wiremock e2e updated to assert addressing via `remote_id` and opaque ids. ## Gate (3 separate commands, all green, 0 ignored) - `cargo fmt --all -- --check` — clean - `cargo check-all` (clippy `--all-targets --workspace -D warnings`) — 0 warnings - `cargo test-all` — all suites pass, 0 failed, 0 ignored (nunciod lib 123; nuncio-store, nuncio-mail, nuncio-core, nuncio-filter, nuncio-cli, nuncio-cal, nuncio-contacts, and all e2e suites green). Pre-commit hook re-ran the full gate on commit. No fabricated success; all protocols mocked; no live network.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/nuncio#337