feat(sync): enumerate IMAP changes over a four-rung capability ladder #402

2026-08-10T21:51:09Z

KofTwentyTwo commented

2026-08-10 21:51:09 +00:00

(Migrated from github.com)

Summary of Changes

Fixes the ghost-message defect. This is the highest-value piece of ADR 0002 and it helps a single engine as much as ten.

Stacked on PR #401 → #384. It also merges #400 (story/schema-version-and-token-scheme) for the checkpoint scheme it extends. GitHub retargets as those land; review the final commit.

The defect

Sync could only ever add. It resolved a UID boundary, fetched forward, and stored the results — nothing ever asked what stopped being there. A message moved or deleted by any other client (another daemon, a phone, webmail) stayed local forever.

The contract

FolderChanges { upserts, removals, present: Option<Vec<String>>, next_state }

present is an Option deliberately. None and Some(vec![]) are not the same: None means the pass was incremental and cannot speak to absence; Some(vec![]) means the folder is genuinely empty. Conflating them deletes a mailbox.

sync_messages remains as a provided method so no implementation can satisfy the trait by supplying only the fetch-only view and quietly never reporting a removal.

The ladder — per mailbox, not per account

NOMODSEQ is a mailbox property (RFC 7162 §3.1.2.2), so one folder can be unable to answer mod-sequence queries on a server that otherwise can.

Rung	Removals come from	Provider
QRESYNC	server states them (`VANISHED`)	Fastmail/Cyrus, Dovecot
CONDSTORE	`CHANGEDSINCE` + UID-set enumeration	Gmail
UID-set diff	UID boundary + enumeration	Exchange
Full	everything fetched	no usable checkpoint

Gmail advertises CONDSTORE and not QRESYNC. A two-rung ladder would have pushed the largest provider onto full sweeps against its published 2,500 MB/day per-account cap — the correction that came out of the protocol review.

Enumeration asks for UID only — no bodies, no flags — so detecting removals costs one short line per message. A mailbox that can supply a mod-sequence gets a mod-sequence checkpoint even after a full pass, so the next sync starts on a higher rung than this one did.

UIDVALIDITY

A change is not a downgrade to a lesser rung — it is a full pass. UIDs restart, so every stored boundary names a different message. The dangerous reading would be to enumerate the new UID space, find none of the stored ids, and conclude the folder was emptied. Pinned by a test.

Also here

The raw collector gains a bounded read. A server that accepts a command then stops answering must surface as a stall, not hang the sync — the same failure the typed fetch path already bounds. Caught by an existing test that started hanging.

Type of Change

Bug fix (non-breaking change fixing an issue)
New feature (non-breaking change adding functionality)
Breaking change (fix or feature causing existing functionality to change)
Documentation update

MailBackend gains sync_changes; syncs now delete local rows the server no longer has. Internal only — no .proto change.

Related Issues

Refs #390. Design: ADR 0002 (PR #385). Depends on #384, #388, #387.

Quality Verification Checklist

I have verified code formatting (cargo fmt --all -- --check)
I have verified clippy lints (cargo clippy --all-targets --workspace -- -D warnings), plus --features test-support
I have executed the full test suite (cargo test --workspace) and all tests pass (624 passed, 0 failed)
Business logic is implemented in headless engine crates, keeping UI presentation shells skinny
No secrets, keys, or credentials are added to tracked files

Notes for review

The store now deletes. delete_messages + message_ids_in_folder (scoped by account and folder — folder ids are not globally unique, and diffing across accounts would delete one account's mail because the other's server did not mention it). The messages_ad trigger keeps FTS clean, so a deleted message cannot survive as a searchable plaintext orphan.
5 new wire tests, including every_rung_reports_a_message_that_disappeared which runs all three server profiles.
FetchRange/resolve_fetch_range deleted, superseded by Checkpoint/choose_rung. Their tests were rewritten against the new API rather than dropped.
Three existing assertions retargeted, not weakened. They counted UID FETCH commands assuming one enumeration plus one body fetch; there is now an additional UID-only enumeration. They now assert on the body fetch narrowing, and one explicitly asserts the enumeration pass is present.
Inline scripted responders gained CAPABILITY, since the planner negotiates before selecting. Deliberately not added to the responder that scripts capability absence — it must keep control of what it advertises.
JMAP reports present: None — it does not use Email/changes yet, so it cannot claim to have seen the complete set. Honest rather than convenient; JMAP removals are follow-up work.
Known cost: the enumeration runs every non-QRESYNC pass. ADR 0002 anticipates making it periodic; deferred rather than guessed at.

## Summary of Changes **Fixes the ghost-message defect.** This is the highest-value piece of ADR 0002 and it helps a single engine as much as ten. > Stacked on PR #401 → #384. It also merges #400 (`story/schema-version-and-token-scheme`) for the checkpoint scheme it extends. GitHub retargets as those land; review the final commit. ### The defect Sync could only ever *add*. It resolved a UID boundary, fetched forward, and stored the results — nothing ever asked what stopped being there. A message moved or deleted by **any** other client (another daemon, a phone, webmail) stayed local forever. ### The contract ```rust FolderChanges { upserts, removals, present: Option<Vec<String>>, next_state } ``` `present` is an `Option` deliberately. **`None` and `Some(vec![])` are not the same**: `None` means the pass was incremental and cannot speak to absence; `Some(vec![])` means the folder is genuinely empty. Conflating them deletes a mailbox. `sync_messages` remains as a *provided* method so no implementation can satisfy the trait by supplying only the fetch-only view and quietly never reporting a removal. ### The ladder — per mailbox, not per account `NOMODSEQ` is a mailbox property (RFC 7162 §3.1.2.2), so one folder can be unable to answer mod-sequence queries on a server that otherwise can. | Rung | Removals come from | Provider | | --- | --- | --- | | QRESYNC | server states them (`VANISHED`) | Fastmail/Cyrus, Dovecot | | CONDSTORE | `CHANGEDSINCE` + UID-set enumeration | **Gmail** | | UID-set diff | UID boundary + enumeration | **Exchange** | | Full | everything fetched | no usable checkpoint | Gmail advertises CONDSTORE and **not** QRESYNC. A two-rung ladder would have pushed the largest provider onto full sweeps against its published 2,500 MB/day per-account cap — the correction that came out of the protocol review. Enumeration asks for **UID only** — no bodies, no flags — so detecting removals costs one short line per message. A mailbox that can supply a mod-sequence gets a mod-sequence checkpoint *even after a full pass*, so the next sync starts on a higher rung than this one did. ### UIDVALIDITY A change is **not** a downgrade to a lesser rung — it is a full pass. UIDs restart, so every stored boundary names a different message. The dangerous reading would be to enumerate the new UID space, find none of the stored ids, and conclude the folder was emptied. Pinned by a test. ### Also here The raw collector gains a **bounded read**. A server that accepts a command then stops answering must surface as a stall, not hang the sync — the same failure the typed fetch path already bounds. Caught by an existing test that started hanging. ## Type of Change - [x] Bug fix (non-breaking change fixing an issue) - [x] New feature (non-breaking change adding functionality) - [x] Breaking change (fix or feature causing existing functionality to change) - [ ] Documentation update `MailBackend` gains `sync_changes`; syncs now delete local rows the server no longer has. Internal only — no `.proto` change. ## Related Issues Refs #390. Design: [ADR 0002](docs/adr/0002-multi-engine-sync-model.md) (PR #385). Depends on #384, #388, #387. ## Quality Verification Checklist - [x] I have verified code formatting (`cargo fmt --all -- --check`) - [x] I have verified clippy lints (`cargo clippy --all-targets --workspace -- -D warnings`), plus `--features test-support` - [x] I have executed the full test suite (`cargo test --workspace`) and all tests pass (**624 passed, 0 failed**) - [x] Business logic is implemented in headless engine crates, keeping UI presentation shells skinny - [x] No secrets, keys, or credentials are added to tracked files ## Notes for review - **The store now deletes.** `delete_messages` + `message_ids_in_folder` (scoped by account *and* folder — folder ids are not globally unique, and diffing across accounts would delete one account's mail because the other's server did not mention it). The `messages_ad` trigger keeps FTS clean, so a deleted message cannot survive as a searchable plaintext orphan. - **5 new wire tests**, including `every_rung_reports_a_message_that_disappeared` which runs all three server profiles. - **`FetchRange`/`resolve_fetch_range` deleted**, superseded by `Checkpoint`/`choose_rung`. Their tests were rewritten against the new API rather than dropped. - **Three existing assertions retargeted**, not weakened. They counted `UID FETCH` commands assuming one enumeration plus one body fetch; there is now an additional UID-only enumeration. They now assert on the *body* fetch narrowing, and one explicitly asserts the enumeration pass **is** present. - **Inline scripted responders gained `CAPABILITY`**, since the planner negotiates before selecting. Deliberately not added to the responder that scripts capability *absence* — it must keep control of what it advertises. - **JMAP reports `present: None`** — it does not use `Email/changes` yet, so it cannot claim to have seen the complete set. Honest rather than convenient; JMAP removals are follow-up work. - Known cost: the enumeration runs every non-QRESYNC pass. ADR 0002 anticipates making it periodic; deferred rather than guessed at.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/nuncio#402