feat(store): split message identity from mailbox placements #405

2026-08-12T16:03:11Z

KofTwentyTwo commented

2026-08-12 16:03:11 +00:00

(Migrated from github.com)

Splits a message's identity from its placements, the last core piece of ADR 0002. Closes #394; also carries #386 and #387, which it depends on.

messages    (message_key PK, account_id, subject, sender, recipient,
             received_at, body_plain, body_html, message_id,
             content_hash, identity_source)
placements  (account_id, folder_id, uidvalidity, uid, message_key, read_flag,
             PRIMARY KEY (account_id, folder_id, uidvalidity, uid))
filter_fired(rule_id, message_key, fired_at, PRIMARY KEY (rule_id, message_key))

surrogate_id(account, folder, uidvalidity, uid) made the same message in a different folder a different message, so a move minted a new row and orphaned the old one. Identity is now content-addressed and folder-independent: EMAILID (RFC 8474) → X-GM-MSGID → (Message-ID and content hash) → the old folder-scoped surrogate as a last resort. Each tier is domain-separated and length-prefixed, and account-scoped at every level.

Why the shape is what it is

A Message-ID never keys storage alone. It is public, echoed in the References of every reply, and forgeable — with an upsert, a crafted message could address and overwrite one the user already trusts, and evade filtering by never being classified new. Tier 3 requires a content hash too; missing either, derivation falls through to the surrogate. The same Message-ID with different content is correctly two messages, which also covers list-copy-vs-direct and drafts.

uidvalidity is in the placement primary key. UIDs restart at 1 after a bump; without it, the first message of the new generation overwrites whichever old one shared its UID.

The message row is first-write-wins, and the FTS row is written only on the insert that created the message. A second sighting — the normal case for a message also in another folder — leaves the stored body alone.

Evaluation is per placement; acting is once per message. WHERE FOLDER = 'INBOX' has no answer for a message in three folders, so FilterEngine now takes a PlacedEmail. But MOVE/FLAG converge under repetition and FORWARD/CALL WEBHOOK do not — they hit third parties. So each matched rule takes an atomic (rule_id, message_key) claim before any side effect, and fails closed: an unverifiable claim is never a licence to act.

The FTS plaintext-orphan rule. messages_fts holds decrypted bodies, reaped by a trigger on AFTER DELETE ON messages. Dropping one placement of a multi-folder message leaves the body searchable; dropping the last reaps the message row, and the trigger clears the index. The trigger stays on messages and must never move to placements — inverted, deleted mail leaves readable plaintext behind. Both directions are pinned by tests, and all three plausible mutations (unconditional reap, no reap, trigger relocated) were verified to break the correct assertion.

Migration: your cached mail resets

IDENTITY_SCHEMA_VERSION goes 1 → 2. On next open the cached message store is rebuilt and mail re-syncs from the server, which is authoritative anyway. Accounts, credentials, filter rules and WORM audit records are preserved; cached mail, sync checkpoints and the filter execution ledger are not. Per ADR 0002's pre-release posture, this beats carrying a permanently two-tier store in which old rows are invisible to every guarantee the model provides.

Merge order

This branch merges the enumeration chain, so it lands last:

#384 → #401 → #402 → #403 → this PR

It also carries #398 (#386) and #400 (#387) as its declared ADR dependencies. The chain was merged in mid-flight rather than after, because both sides rewrote the same functions in backend.rs / imap.rs / jmap.rs — doing it once, with both sides fresh, immediately exposed a bug that a later merge would have buried (below).

Bugs found along the way

Absence reporting matched zero rows. The incoming side computed removals/present via surrogate_id; post-split those keys named nothing, so ghost cleanup would have logged a removal count and deleted nothing. Now placement_key_for returns a PlacementKey directly.
A SELECT omitting UIDVALIDITY could wipe a folder. present keys with an empty generation match no stored placement, so a caller obeying the contract deletes everything. Now forced to present = None / removals = [] with a warn!, mirroring the existing checkpoint guard.
existing_placements had no chunking while the caller passed a whole folder — SQLite's 32,766 bind cap meant a first sync of any mailbox over 8,191 messages failed permanently, every retry failing identically. Now chunked internally so no caller can trip it.
A deferred read-then-write transaction was a live defect, not theory: restoring it reproducibly fails with database is locked. Both sites now use BEGIN IMMEDIATE, matching the pattern already established elsewhere in db.rs.
The outbox guessed which copy to mutate. With no record of which occupancy a rule matched, it addressed placements.first() — so a rule matching the INBOX copy could MOVE/DELETE the Archive copy. It now refuses, permanently and without touching the server.

Wire contract

Unchanged. The only .proto edits are comments. Message.id's value changes for every message, which is invisible to the byte-exact descriptor golden — the golden passes without regeneration. Message.folder_id now names the placement the message was reached through (the requested folder for a listing; the lowest-sorting folder id for a lookup by id). A field carrying the full placement set is #395's scope.

ListFolders/ListMessages carry no account_id, so the gRPC layer iterates accounts and merges by folder id, reproducing today's account-blind behaviour exactly. The store beneath is correctly account-scoped.

Known-open, deliberately

IMAP and JMAP derive disjoint key spaces. IMAP does not yet request EMAILID/X-GM-MSGID, so it only ever reaches tier 3 or 4, while a JMAP object id enters at tier 1. Two engines on one account therefore do not converge across engine types yet. Documented at derive_message_key and at IMAP's place().
Adding those FETCH items is not free. It would change every stored key, sending each re-synced placement through the upsert's repoint arm, orphaning the old message row and stranding its decrypted body in messages_fts. The repoint path needs a reap first.
JMAP still reports present: None and so still accumulates ghost messages; it needs Email/changes.
filter_fired is never reaped — bounded by (rules × messages) and cleared by the schema reset, but a deleted-and-re-arriving message never re-fires.
Triage is now once per (rule, message) forever. A second run over the same rules reports matched_count: 0. Stated in the RPC's doc comment.
export_mbox no longer emits X-Nuncio-Folder — the folder is a placement property now, and restoring it needs a Placement in the signature.

Verification

cargo fmt --all -- --check clean · cargo check-all 0 errors · cargo test-all 680 passed / 0 failed.

Every task was implemented test-first, reviewed, fixed, and re-reviewed, then the whole branch reviewed once more. Where a property mattered, it was verified by mutation rather than assertion — including two cases where a test that looked right could not actually catch its own bug and had to be rebuilt.

Reviewers: the load-bearing parts are the reap logic in delete_placements, the claim gating in apply_filter_actions, and the present: None vs Some(vec![]) distinction end to end. Those three are where a silent regression would hurt most.

Splits a message's **identity** from its **placements**, the last core piece of [ADR 0002](https://github.com/KofTwentyTwo/nuncio/pull/385). Closes #394; also carries #386 and #387, which it depends on. ``` messages (message_key PK, account_id, subject, sender, recipient, received_at, body_plain, body_html, message_id, content_hash, identity_source) placements (account_id, folder_id, uidvalidity, uid, message_key, read_flag, PRIMARY KEY (account_id, folder_id, uidvalidity, uid)) filter_fired(rule_id, message_key, fired_at, PRIMARY KEY (rule_id, message_key)) ``` `surrogate_id(account, folder, uidvalidity, uid)` made the same message in a different folder a different message, so a move minted a new row and orphaned the old one. Identity is now content-addressed and folder-independent: `EMAILID` (RFC 8474) → `X-GM-MSGID` → (`Message-ID` **and** content hash) → the old folder-scoped surrogate as a last resort. Each tier is domain-separated and length-prefixed, and account-scoped at every level. ## Why the shape is what it is **A `Message-ID` never keys storage alone.** It is public, echoed in the `References` of every reply, and forgeable — with an upsert, a crafted message could address and overwrite one the user already trusts, and evade filtering by never being classified new. Tier 3 requires a content hash too; missing either, derivation falls through to the surrogate. The same `Message-ID` with different content is correctly two messages, which also covers list-copy-vs-direct and drafts. **`uidvalidity` is in the placement primary key.** UIDs restart at 1 after a bump; without it, the first message of the new generation overwrites whichever old one shared its UID. **The message row is first-write-wins**, and the FTS row is written only on the insert that created the message. A second sighting — the normal case for a message also in another folder — leaves the stored body alone. **Evaluation is per placement; acting is once per message.** `WHERE FOLDER = 'INBOX'` has no answer for a message in three folders, so `FilterEngine` now takes a `PlacedEmail`. But `MOVE`/`FLAG` converge under repetition and `FORWARD`/`CALL WEBHOOK` do not — they hit third parties. So each matched rule takes an atomic `(rule_id, message_key)` claim before any side effect, and **fails closed**: an unverifiable claim is never a licence to act. **The FTS plaintext-orphan rule.** `messages_fts` holds *decrypted* bodies, reaped by a trigger on `AFTER DELETE ON messages`. Dropping one placement of a multi-folder message leaves the body searchable; dropping the last reaps the message row, and the trigger clears the index. The trigger stays on `messages` and must never move to `placements` — inverted, deleted mail leaves readable plaintext behind. Both directions are pinned by tests, and all three plausible mutations (unconditional reap, no reap, trigger relocated) were verified to break the correct assertion. ## Migration: your cached mail resets `IDENTITY_SCHEMA_VERSION` goes **1 → 2**. On next open the cached message store is rebuilt and mail re-syncs from the server, which is authoritative anyway. **Accounts, credentials, filter rules and WORM audit records are preserved; cached mail, sync checkpoints and the filter execution ledger are not.** Per ADR 0002's pre-release posture, this beats carrying a permanently two-tier store in which old rows are invisible to every guarantee the model provides. ## Merge order This branch merges the enumeration chain, so it lands last: ``` #384 → #401 → #402 → #403 → this PR ``` It also carries #398 (#386) and #400 (#387) as its declared ADR dependencies. The chain was merged in mid-flight rather than after, because both sides rewrote the same functions in `backend.rs` / `imap.rs` / `jmap.rs` — doing it once, with both sides fresh, immediately exposed a bug that a later merge would have buried (below). ## Bugs found along the way - **Absence reporting matched zero rows.** The incoming side computed `removals`/`present` via `surrogate_id`; post-split those keys named nothing, so ghost cleanup would have logged a removal count and deleted nothing. Now `placement_key_for` returns a `PlacementKey` directly. - **A SELECT omitting UIDVALIDITY could wipe a folder.** `present` keys with an empty generation match no stored placement, so a caller obeying the contract deletes everything. Now forced to `present = None` / `removals = []` with a `warn!`, mirroring the existing checkpoint guard. - **`existing_placements` had no chunking** while the caller passed a whole folder — SQLite's 32,766 bind cap meant a first sync of any mailbox over 8,191 messages failed permanently, every retry failing identically. Now chunked internally so no caller can trip it. - **A deferred read-then-write transaction was a live defect**, not theory: restoring it reproducibly fails with `database is locked`. Both sites now use `BEGIN IMMEDIATE`, matching the pattern already established elsewhere in `db.rs`. - **The outbox guessed which copy to mutate.** With no record of which occupancy a rule matched, it addressed `placements.first()` — so a rule matching the INBOX copy could `MOVE`/`DELETE` the Archive copy. It now refuses, permanently and without touching the server. ## Wire contract **Unchanged.** The only `.proto` edits are comments. `Message.id`'s *value* changes for every message, which is invisible to the byte-exact descriptor golden — the golden passes without regeneration. `Message.folder_id` now names the placement the message was reached through (the requested folder for a listing; the lowest-sorting folder id for a lookup by id). A field carrying the full placement set is #395's scope. `ListFolders`/`ListMessages` carry no `account_id`, so the gRPC layer iterates accounts and merges by folder id, reproducing today's account-blind behaviour exactly. The store beneath is correctly account-scoped. ## Known-open, deliberately - **IMAP and JMAP derive disjoint key spaces.** IMAP does not yet request `EMAILID`/`X-GM-MSGID`, so it only ever reaches tier 3 or 4, while a JMAP object id enters at tier 1. Two engines on one account therefore do not converge across engine types yet. Documented at `derive_message_key` and at IMAP's `place()`. - **Adding those FETCH items is not free.** It would change every stored key, sending each re-synced placement through the upsert's repoint arm, orphaning the old message row and stranding its decrypted body in `messages_fts`. The repoint path needs a reap first. - **JMAP still reports `present: None`** and so still accumulates ghost messages; it needs `Email/changes`. - **`filter_fired` is never reaped** — bounded by (rules × messages) and cleared by the schema reset, but a deleted-and-re-arriving message never re-fires. - **Triage is now once per `(rule, message)` forever.** A second run over the same rules reports `matched_count: 0`. Stated in the RPC's doc comment. - **`export_mbox` no longer emits `X-Nuncio-Folder`** — the folder is a placement property now, and restoring it needs a `Placement` in the signature. ## Verification `cargo fmt --all -- --check` clean · `cargo check-all` 0 errors · `cargo test-all` **680 passed / 0 failed**. Every task was implemented test-first, reviewed, fixed, and re-reviewed, then the whole branch reviewed once more. Where a property mattered, it was verified by mutation rather than assertion — including two cases where a test that *looked* right could not actually catch its own bug and had to be rebuilt. Reviewers: the load-bearing parts are the reap logic in `delete_placements`, the claim gating in `apply_filter_actions`, and the `present: None` vs `Some(vec![])` distinction end to end. Those three are where a silent regression would hurt most.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/nuncio#405