existing_placements binds four parameters per key into one row-value
IN (...) list, and its caller hands it an entire folder: FolderChanges
accumulates every batch into one Vec, so the Full rung -- an ordinary
first sync, and the unrecognised-checkpoint fallback -- passes the whole
mailbox. SQLITE_MAX_VARIABLE_NUMBER is 32766, so past 8191 placements
the statement fails, the folder sync aborts before writing a checkpoint,
and every retry fails identically. A first sync of any large mailbox was
permanently broken.
Split the batch internally at 4000 keys and union the results, so the
limit is unreachable from outside rather than one forgetful caller away.
existing_message_ids had the same shape at a 32766 ceiling and is
chunked the same way; it stays because it is public API drawing the
identity-vs-occupancy distinction, not because it has callers.
Also correct three comments that documented traps as free:
- The IMAP batching doc claimed the working set stays bounded regardless
of mailbox size. Batching bounds each request, not memory; the parsed
messages accumulate across batches.
- Adding the EMAILID/X-GM-MSGID FETCH items is not free: it repoints
every stored message's key, orphaning the old messages row and
stranding its decrypted body in messages_fts. It needs the repoint
path to reap first, or it leaks plaintext.
- IMAP never requests the tier-1/2 identifiers, so an IMAP-derived key
and a JMAP-derived key for the same message differ. Recorded on
derive_message_key and on the IMAP place() helper.
De-link ADR 0002 in the roadmap: the file exists on no merge target, so
the markdown now matches the rendered page, which already used plain
bold.
WS-B, WS-C and WS-E grew when ADR 0002 broke the multi-engine sync model into
#386-#397, and the 2026-08-07 snapshot never caught up. Counts now match the
live milestones. Nothing is marked closed: the identity split and its chain are
green on a branch, and the numbers move when it merges.
Refs #394
The surrogate tier hashes the folder, so a message with no usable Message-ID
-- including any fetched headers-only -- gets a folder-scoped id that changes
when it moves. Saying ids survive a move unconditionally was false for exactly
the messages most likely to have one.
Message.id is no longer a folder-scoped surrogate: it is content-addressed
from the strongest identity the server offered, so it survives a move.
Message.folder_id names the one occupancy the message was reached through,
since a message can sit in several folders at once.
Triage's contract is stated too -- it claims each (rule, message) pair once
and for all, so a second run over the same rules matches nothing.
Refs #394
An outbox row records only a message key, so for a message occupying
several mailboxes there is no way to know which copy the rule matched.
Addressing whichever placement sorted first meant a rule that matched
the INBOX copy could MOVE or DELETE the Archive one -- against mail the
user never targeted, with no undo, and Gmail labels make multi-placement
ordinary rather than exotic. Unresolvable addressing is no more a licence
to act than an unverifiable claim: it now fails permanently, naming the
competing mailboxes. Single-placement mutations are unaffected.
Also from review: PreviewRule no longer substitutes an invented inbox
occupancy when a stored message's placements cannot be read, which let
WHERE FOLDER match a location the message does not have; explicit
VANISHED removals no longer need ambient account context, since a
PlacementKey names its own account; and both Mail read RPCs now state
that they list across configured accounts only, so removing an account
hides its cached mail by design rather than by accident.
Covers the cross-account merge shims, which nothing exercised: summed
counts for a shared folder id, merged ordering across page boundaries,
and a page boundary that only shows up when one account owns the folder.
Message identity is folder-independent, so a message already stored from
INBOX has a known key when it turns up in Archive. Classifying "is this
new?" by message key answered "seen" for a mailbox nothing had ever
filtered, and the arrival silently fired no rule at all -- no error, no
log. fetch_and_persist now classifies on PlacementKey via
existing_placements, persists through save_email_at, and reconciles
absence with placements_in_folder/delete_placements.
Evaluation is therefore per placement, so acting has to be gated
separately: apply_filter_actions takes the placement it evaluates
against and must win claim_filter_fire, keyed on message identity,
before any action runs. MOVE and FLAG converge under repetition;
FORWARD and CALL WEBHOOK land on third parties and do not. The claim
fails closed -- an unverifiable claim skips the actions rather than
licensing them. MARK READ now names the occupancy it was evaluated
against, since IMAP \Seen is per-mailbox.
The daemon's gRPC and outbox surfaces are ported only as far as
compilation and honest behaviour require, and say so in place: the
outbox recovers its remote addressing from a placement and warns when
the row cannot say which one the rule matched, and the Mail read RPCs
merge across configured accounts pending an account on the wire.
A SELECT that omits [UIDVALIDITY] left every placement key this pass built
carrying an empty generation. None of them match placements stored under a real
one, and under the `present` contract -- delete any placement absent from this
list -- that reads as "the folder holds nothing", so the caller would delete the
whole mailbox. The pass now declines to report absence at all, mirroring the
checkpoint path, which already refuses to write a validity-less value, and warns
so the reason is visible. Upserts are still surfaced.
[UIDVALIDITY] is REQUIRED on SELECT (RFC 3501 section 6.3.1), so this is a guard
against a broken server rather than an expected path -- but the impact is a
wiped mailbox.
Also pin the identity tier on the two branches that fetch no octets. The
headers-only branch drops its hash because it covers only the header section;
publishing it would key the message on the wrong octets and mint a second key
once the body arrived, resurrecting the message as a duplicate. The fixtures now
carry a Message-ID so those branches reaching the surrogate is a real assertion
rather than an accident of the fixture, and the full-body path is asserted to
reach the content tier for contrast.
Give PlacementKey and Placement::key() direct tests: all four coordinates
reproduced, the read flag excluded, each coordinate distinguishing, usable as a
set key and sortable, and a serde round trip.
Reconciles IMAP change enumeration with the message/placement identity split.
The two lines both rewrote how a folder sync reports itself, and resolving them
separately would have meant migrating FolderChanges twice.
FolderChanges wins as the structure, retyped for placements:
upserts: Vec<PlacedMessage> (identity, content, and the occupancy)
removals: Vec<PlacementKey>
present: Option<Vec<PlacementKey>>
next_state: String (unchanged)
Absence is per-occupancy because that is all a folder can speak to. A UID that
vanished from one mailbox says nothing about copies elsewhere, and a message
key cannot be reconstructed from a UID anyway now that identity depends on
which tier the server supported -- so the incoming side's surrogate-per-UID
removals would have named rows the store no longer writes.
PlacementKey moves to nuncio-core so both nuncio-mail and nuncio-store can name
one type; nuncio-store re-exports it and does not gain a reverse dependency.
The store's read-side (Email, Placement) alias becomes MessageWithPlacement, so
it no longer collides with nuncio-mail's PlacedMessage in the sync path.
The four-rung capability ladder is preserved intact, per mailbox: QRESYNC,
CONDSTORE, UID-set diff, full resync. JMAP keeps present: None -- it still has
no Email/changes and genuinely cannot speak to absence.
The incoming message_ids_in_folder and delete_messages are dropped: they read
messages.folder_id, which the placement schema removed, and delete_messages
would reap a message on one folder's silence. placements_in_folder and
delete_placements already do the placement-correct equivalent.
The IMAP and JMAP backends no longer mint a folder-scoped surrogate id for
every fetched message. They derive the account-scoped message key by the
identity precedence and report it as a PlacedMessage: the Email (identity and
content), the tier that produced the key, and the Placement the fetch found it
in. Read state moves onto the placement, where IMAP \Seen already lives.
IMAP does not request EMAILID (RFC 8474) or X-GM-MSGID yet, so identity rests
on the Message-ID/content pair where a Message-ID was captured and on the
folder-scoped surrogate otherwise; adding those FETCH items later lifts stored
rows to a stronger tier on the next resync. A JMAP Email id is server-assigned
and account-stable, the same guarantee EMAILID gives, so JMAP enters at the top
tier instead.
MimeParserAdapter::parse_mime drops its id and folder_id arguments: identity
cannot be settled until the parsed Message-ID and content hash are in hand
alongside the transport's coordinates, so the caller assigns it.
Five defects found in review:
- existing_placements never pinned that a never-saved key is absent, and
its negative case was vacuous (no Archive row existed to over-report).
Over-reporting makes genuinely new mail look already-seen and be
skipped by the sync classifier. Both are now real assertions, backed
by a count so a dropped IN filter fails.
- save_email_at and delete_placements both read then write in one
deferred transaction. In WAL mode that aborts with SQLITE_BUSY, which
busy_timeout does not retry; twenty racing saves reproducibly failed.
Both now use BEGIN IMMEDIATE, matching the WORM and ledger appenders,
with a concurrency regression test.
- list_messages_page claimed the folder join was one-to-one. It is not:
the placements key permits one message in one folder twice, and the
message-only cursor made the second row unreachable across a page
boundary. The keyset now carries the full placement address.
- list_messages_for_export matched a folder id without its account,
reintroducing the cross-account collision this model removes.
The old name tripped clippy::should_implement_trait, which fails the
workspace lint gate and therefore CI. Implementing std::str::FromStr
instead would force a Result and a new error type when the only failure
is "not one of four known tokens" -- Option already says that. The
round-trip test now also pins that an unrecognised token returns None.
A server that stops reporting a UID has said the message left that
mailbox, not that it ceased to exist -- a move reports exactly this in
the source folder. So a placement is dropped unconditionally and the
message only when its last placement goes.
That reap is load-bearing for confidentiality: messages_fts holds
decrypted bodies and is cleared by the messages_ad trigger on messages,
so deleting the message row is the only thing that clears the index.
Skip it and a deleted message leaves its plaintext body readable to
anyone with filesystem access; invert it and search breaks for mail that
still exists. Both directions are pinned by tests.
Reads now separate what a message is from where it sits: get_message
returns identity and content alone, list_messages joins through
placements and hands back the placement it found the message via, and
folders are derived by grouping placements per account -- folder ids are
not globally unique, so grouping across accounts would report one merged
INBOX holding two accounts' mail. Read state is updated per placement
because IMAP \Seen is per-mailbox. An FTS hit now names a message key,
not a placement, since the index holds one row per message.
A message's content is now written once, keyed on its folder-independent
message key, while each mailbox it occupies gets its own placement row.
The message row is first-write-wins so a later sighting claiming a known
key cannot rewrite a body the user already trusts, and the plaintext FTS
row is written only on the insert that actually created the message.
The placement is upserted, because its read flag genuinely changes.
A message can occupy several folders at once, so FOLDER and ACCOUNT
conditions have no single answer at the message level. PlacedEmail
pairs a message with the one mailbox occupancy under evaluation;
evaluate_condition, evaluate, evaluate_with_timeout, and preview now
take a PlacedEmail and read Folder/Account off its Placement, while
content fields still read the Email. Callers fan out over a message's
placements and evaluate each independently.
Split the messages table into identity/content only (message_key
primary key, identity_source column) and a new placements table
holding one row per (account, folder, uidvalidity, uid) mailbox
occupancy. uidvalidity is part of the placements primary key because
UIDs restart at 1 after a bump and would otherwise collide with rows
from the prior generation.
Add filter_fired as a fire-once ledger for filter actions, keyed on
message identity so a message arriving in a second folder cannot
re-trigger a side-effecting rule that already ran against it. Add
claim_filter_fire (atomic INSERT ... ON CONFLICT DO NOTHING),
has_filter_fired, and filter_fire_count as public store methods so
other crates can observe and use the ledger without reaching the
connection pool.
Bump IDENTITY_SCHEMA_VERSION to 2 so reconcile_schema_version rebuilds
the cached store on upgrade, and extend reset_cached_message_store to
clear placements and filter_fired alongside messages. Retire the now
-obsolete ensure_messages_identity_columns additive migration and its
test: the remote_id/uid_validity columns and the identity index it
backfilled no longer exist on messages, and placements' own primary
key now provides the uniqueness guarantee it used to build. Update
backfill_message_fts and the messages_ad trigger for the message_key
rename, keeping the trigger on messages (not placements) so dropping
one placement of a still-present message never orphans its FTS row.
export_mbox stopped emitting X-Nuncio-Folder when folder_id moved off
Email onto Placement, but the reason lived only in an SDD report, not
the code. Add a comment at the removal site so a future reader knows
it's intentional and what restoring it would require (threading a
Placement into export_mbox's signature).
Also broaden message_key_is_account_scoped_at_every_tier, which
claimed to cover every identity tier but only exercised EmailId;
it now asserts account-scoping at all four tiers.
Identity cannot currently be reconstructed for a stored message. The parser
never reads Message-ID, the Email model has nowhere to put it, the messages
table has no column for it, and the raw octets are dropped once parsed. Any
later attempt to give a message a folder-independent identity therefore has
nothing to work from except a re-download of the entire mailbox.
Capture both now, as nullable columns nothing reads yet, so the data starts
accruing while the design that consumes it is still being built.
Normalization is fixed and shared rather than left to each call site: strip all
ASCII whitespace (a msg-id cannot legally contain any, so this unfolds a wrapped
header without knowing how the transport folded it), strip one pair of angle
brackets, and preserve case, since only the domain half is case-insensitive and
lowercasing wholesale would merge ids the origin considers distinct. Engines
that normalize differently would hold the same message under different bytes,
and no comparison between their caches would mean anything.
Captured at every ingest path, including the two that do not parse a full
message. The envelope-only path takes Message-ID from ENVELOPE rather than
skipping capture. The headers-only path keeps the id recovered from those
headers but discards the hash parse_mime computed over them, because
content_hash is defined over a message's full octets and publishing a
header-only digest under that name would quietly mean something else.
The hash exists to keep Message-ID honest: it is public, echoed in every
References header, and reused legitimately -- a mailing-list copy and a direct
copy share one, and a draft keeps one across every save -- so the pair
distinguishes messages that the id alone would merge.
No backfill. Legacy rows stay NULL, which reads as "not captured" rather than
"this message has no Message-ID"; inventing a value would be worse than
admitting the gap.
Refs #386
james.maes