KofTwentyTwo/nuncio
2
0 Fork 0
Code Issues 73 Pull Requests 12 Actions Packages Projects Releases Wiki Activity

Compare commits

merge into: KofTwentyTwo/nuncio:main
KofTwentyTwo/nuncio:main KofTwentyTwo/nuncio:qualification/central-actions-20261031 KofTwentyTwo/nuncio:feature/nuncio-google-first-rebuild KofTwentyTwo/nuncio:dependabot/cargo/tokio-rustls-0.26.5 KofTwentyTwo/nuncio:dependabot/cargo/keyring-4.2.0 KofTwentyTwo/nuncio:dependabot/cargo/flate2-1.1.10 KofTwentyTwo/nuncio:dependabot/cargo/mail-parser-0.11.8 KofTwentyTwo/nuncio:dependabot/cargo/quick-xml-0.42.0 KofTwentyTwo/nuncio:dependabot/cargo/aes-gcm-0.11.1 KofTwentyTwo/nuncio:dependabot/cargo/futures-0.3.34 KofTwentyTwo/nuncio:dev KofTwentyTwo/nuncio:dependabot/cargo/async-trait-0.1.92 KofTwentyTwo/nuncio:story/monitor-and-health KofTwentyTwo/nuncio:dependabot/cargo/clap-4.6.6 KofTwentyTwo/nuncio:dependabot/cargo/lettre-0.11.23 KofTwentyTwo/nuncio:dependabot/github_actions/github/codeql-action-4.37.4 KofTwentyTwo/nuncio:rc KofTwentyTwo/nuncio:wip/unreviewed-account-tls-sync
..
pull from: KofTwentyTwo/nuncio:dev
KofTwentyTwo/nuncio:qualification/central-actions-20261031 KofTwentyTwo/nuncio:feature/nuncio-google-first-rebuild KofTwentyTwo/nuncio:dependabot/cargo/tokio-rustls-0.26.5 KofTwentyTwo/nuncio:dependabot/cargo/keyring-4.2.0 KofTwentyTwo/nuncio:dependabot/cargo/flate2-1.1.10 KofTwentyTwo/nuncio:dependabot/cargo/mail-parser-0.11.8 KofTwentyTwo/nuncio:dependabot/cargo/quick-xml-0.42.0 KofTwentyTwo/nuncio:dependabot/cargo/aes-gcm-0.11.1 KofTwentyTwo/nuncio:dependabot/cargo/futures-0.3.34 KofTwentyTwo/nuncio:dev KofTwentyTwo/nuncio:dependabot/cargo/async-trait-0.1.92 KofTwentyTwo/nuncio:story/monitor-and-health KofTwentyTwo/nuncio:dependabot/cargo/clap-4.6.6 KofTwentyTwo/nuncio:dependabot/cargo/lettre-0.11.23 KofTwentyTwo/nuncio:dependabot/github_actions/github/codeql-action-4.37.4 KofTwentyTwo/nuncio:main KofTwentyTwo/nuncio:rc KofTwentyTwo/nuncio:wip/unreviewed-account-tls-sync
136 Commits
main .. dev
Author SHA1 Message Date
james.maes a268fd5a00 Merge pull request #431 from KofTwentyTwo/story/wire-contacts-sync 
feat(contacts): drive contacts sync from stored CardDAV accounts
 2026-08-14 21:48:47 -05:00
james.maes 6cdf7be0a7 Merge pull request #430 from KofTwentyTwo/fix/dav-namespace-xml 
fix(dav): parse multistatus with namespaces, not substrings
 2026-08-14 21:46:11 -05:00
james.maes e444ffd0c0 feat(contacts): drive contacts sync from stored CardDAV accounts 
Contacts sync had a real, wiremock-tested CardDAV client that production
could never reach: `AccountConfig` had no way to express a CardDAV
address book, so `Contacts/Sync` always fell to its honest-error branch
unless a test double was injected.

Add CardDAV as a first-class account protocol, mirroring how CalDAV is
already resolved and driven:

- `AccountProtocol::CardDav` and `Transport::CardDav(DavTransport)`, plus
  `is_caldav()`/`is_carddav()` so callers pick a specific DAV flavour
  rather than the ambiguous `is_dav()`. Calendar sync now filters on
  `is_caldav()` so a CardDAV account can never be mistaken for a calendar.
- Persistence reuses the existing `collection_url` column, distinguished
  by the `protocol` discriminator; no schema change.
- Proto gains `ACCOUNT_PROTOCOL_CARDDAV` and a `carddav` transport case
  (both additive), with the committed descriptor golden regenerated.
- `contacts_sync::sync_carddav_account` builds a real `CardDavClient` from
  the persisted collection URL and a credential read just-in-time from the
  OS keyring, zeroized on drop.
- `Contacts/Sync` resolves configured CardDAV accounts and returns a typed
  `NotConfigured` (FAILED_PRECONDITION) when none matches -- including
  when the named account exists but speaks another protocol -- instead of
  an untyped internal error or a fabricated zero count.
- `account add --protocol carddav --collection-url ...` in the CLI.

Offline E2E proves the production path: a CardDAV account added over the
real Accounts API, synced through a real client against wiremock, with
credentials resolved from a mocked vault.

Refs #255
 2026-08-14 21:41:15 -05:00
James Maes fb03f2eec2 fix(dav): parse multistatus with namespaces, not substrings 
The CalDAV and CardDAV clients found their payloads by splitting the
response body on the literal strings `<c:calendar-data>` and
`<card:address-data>`. That is only correct for servers that happen to
pick the same prefixes Nuncio does. A compliant server that binds the
namespaces to `D:`/`CAL:`, or declares `DAV:` as the default namespace,
produced zero events or zero contacts and reported success -- the user
saw an empty calendar with nothing in the logs. An element whose name
merely contains the searched one (`<c:calendar-data-summary>`) was read
as if it were the payload, and a property returned under a `404`
propstat was treated as a value.

Add `nuncio_core::dav`, a namespace-aware `multistatus` walker built on
quick-xml (already a workspace dependency, previously declared but
unused) and shared by both clients, since RFC 4791 and RFC 6352 return
the same document shape. Elements are matched on their resolved
`{namespace}local-name`; `<propstat>` status is honoured; and a body
that cannot be traversed is now a `MalformedResponse` error rather than
an empty result indistinguishable from an empty collection. Responses
that carry no readable value are counted and logged at `warn` with the
href and reason, so an omission is visible instead of silent.

Refs #257
 2026-08-14 21:35:31 -05:00
james.maes 349a3dd0b2 Merge pull request #429 from KofTwentyTwo/fix/bulk-path-corrupt-body 
fix(store): survive one corrupt body on bulk read paths
 2026-08-14 21:14:37 -05:00
james.maes c484fb048e fix(store): survive one corrupt body on bulk read paths 
A single unauthenticatable body ciphertext took down whole operations:
the FTS backfill runs inside migrate(), so one bit-rotted row stopped
the daemon booting at all, and listing, triage and export each failed
outright over one bad row out of fifty.

Single-item get_message stays fail-closed and is unchanged. It names one
message, so there is no other row left to hand back, and a message with
an empty body reads exactly like a genuinely empty one.

Bulk paths now return each row as MessageRead (Ok(email) or
Err(UnreadableBody)) instead of failing the call. The row keeps its slot,
which is what lets keyset cursors and chunk boundaries stay exact, and no
row ever comes back carrying a body the store could not authenticate.
Each failure is logged at warn! naming only the message key.

The daemon reports the omissions on the wire rather than swallowing them:
unreadable_message_ids on ListMessagesResponse and ExportResponse (an
archive outlives the store, so its gap is named), unreadable_count on
TriageProgress, and the CLI prints all three. Triage never evaluates
rules against a body that failed authentication. The FTS backfill skips
the row entirely rather than indexing an empty body, so every subsequent
open re-reports it and a repaired row heals itself.

Refs #265
 2026-08-14 21:07:14 -05:00
james.maes 74954f79db Merge pull request #428 from KofTwentyTwo/fix/nl-scheduler-honesty 
fix(cal): refuse phrases the scheduler cannot resolve exactly
 2026-08-14 21:04:00 -05:00
james.maes f5d8c8a153 fix(cal): refuse phrases the scheduler cannot resolve exactly 
NaturalLanguageScheduler ignored the requested weekday and time
entirely, always returning the reference instant plus 24 hours, and
never errored on unrecognised phrasing. Every input produced a
plausible-looking event at the wrong time, which a user only discovers
by missing the meeting.

Replace it with a small, exactly-specified grammar:

  <summary> today|tomorrow|YYYY-MM-DD at <time> for <duration>

Everything outside that grammar returns an error. Weekday phrasings
("next Tuesday") are refused as genuinely ambiguous rather than
resolved, since the two common readings differ by a week. A time of
day and a duration are both mandatory, so neither is defaulted.

Resolution now requires an explicit IANA time zone; a bare Unix
timestamp cannot anchor "3pm". Wall-clock times go through the zone's
DST rules and both anomalies are reported: times inside a
spring-forward gap and times inside a fall-back overlap are refused
rather than silently shifted or disambiguated. Durations are added as
elapsed time, so events spanning a transition keep their real length.

parse now returns ParsedSchedulingIntent instead of a CalendarEvent
with invented "default" account and calendar ids. There are no
in-tree callers, so no behaviour visible to a client changes.

Refs #258
 2026-08-14 20:52:26 -05:00
james.maes 8ee0884e51 Merge pull request #427 from KofTwentyTwo/chore/cli-polish 
refactor(cli): drop the runner's dead ephemeral store
 2026-08-14 20:42:48 -05:00
james.maes 1e48bc47a6 Merge pull request #425 from KofTwentyTwo/fix/defabricate-tls-and-size 
fix: fail honestly on unreadable TLS modes and mis-sized mail
 2026-08-14 20:33:14 -05:00
james.maes 3508b5e10a refactor(cli): drop the runner's dead ephemeral store 
The headless runner still built an ephemeral DatabaseEngine, an EventBus,
and a TempDir that no command read -- a remnant of the CLI-holds-its-own-store
design that predates the gRPC API. Nothing but two dead_code accessors touched
them, so remove the fields, the accessors, and the now-unreachable RunnerError.

With no store to open, construction is neither fallible nor async, so the
constructors lose their Result and async, and 'ephemeral' no longer describes
them: they are renamed connect/connect_with/connect_with_token_override.

Doc comments that contrasted the daemon's persistent store against 'this
runner's own ephemeral local db' now described a field that does not exist;
they are rewritten to say the CLI keeps no store at all.
 2026-08-14 20:28:27 -05:00
james.maes 98b59b8a98 fix(filter): size counts all body parts and attachments 
The NSQL size field was computed from body_plain alone, so an HTML-only
message measured zero bytes and a short note carrying a large attachment
measured only the note. Every size comparison then answered confidently
and wrongly on exactly the mail such rules are written to catch.

size is now the total byte size of the content the store holds: both body
parts plus every attachment payload. Both parts are summed because a
multipart/alternative message genuinely carries both on the wire. Headers
and MIME framing are excluded and documented as excluded -- the raw RFC822
octets are not retained and no column carries their length, so the figure
is a consistent lower bound rather than a guess.

Refs #256
 2026-08-14 20:22:58 -05:00
james.maes ce49515892 fix(store): reject unreadable stored TLS modes on account read 
tls_mode_from_db coerced any unrecognized imap_tls_mode/smtp_tls_mode
token to ImplicitTls on the path every ordinary account read takes, so a
corrupted or tampered row silently handed callers a transport the operator
never configured -- a STARTTLS account read back as implicit TLS, with no
error and no log.

Parse now fails closed with DatabaseError::Corrupted naming the account,
the column, and the offending value. list_accounts already returned a
Result, so no caller signature changes and no caller panics; get_account
inherits the behaviour through it.

Refs #271
 2026-08-14 20:22:53 -05:00
james.maes f6165484e8 Merge pull request #424 from KofTwentyTwo/fix/bound-daemon-keyring 
fix(daemon): bound nunciod's own keyring reads
 2026-08-14 20:13:58 -05:00
james.maes 789b059f89 Merge pull request #423 from KofTwentyTwo/fix/recovery-salvage-flake 
fix(test): close the store pool before salvaging its database
 2026-08-14 20:13:29 -05:00
james.maes 90eab25081 fix(daemon): bound nunciod's own keyring reads 
nunciod mostly mints the key material it reads, so the OS keyring
normally answers without ceremony. That is a habit, not a guarantee: a
keychain locked by screen-lock, a restored or migrated keychain whose
ACL no longer trusts the binary, a re-signed or rebuilt nunciod, or a
Linux Secret Service with no unlocked collection all turn the read into
a wait on something nobody can answer. Unbounded, that left the daemon
hung at startup with no error and no log line.

Bound both of the daemon's key-material reads -- the gRPC bearer token
minted at startup and the webhook signing key provisioned on first CALL
WEBHOOK dispatch -- so each terminates with an honest, logged error that
names the key material and the likely cause. Startup still fails closed
through the same path it already used; only the hang case now reaches
it.

Lift the bounded read the CLI already uses into nuncio-store so both
binaries share one implementation. It still runs on a detached OS thread
behind a timeout rather than spawn_blocking: a keyring read cannot be
cancelled, and Tokio joins its blocking pool at shutdown, which would
merely relocate the hang to process exit.

The daemon deliberately gains no environment override. It mints and owns
the authoritative token and must never accept one from its environment.

Refs #421
 2026-08-14 20:05:31 -05:00
james.maes 9daec8a731 fix(test): close the store pool before salvaging its database 
test_stage_2_backup_creation_and_stage_3_table_salvage built its fixture
engine inside a block and let it drop rather than closing it. A dropped
sqlx pool tears its connections down asynchronously, so SQLite's
close-time WAL handling -- which operates on -wal/-shm by path -- could
land after salvage had unlinked those paths and created a fresh database
at the same location. That pairs the new, empty main file with a -wal
describing the old one, and replaying it reads past end-of-file:
SQLITE_IOERR_SHORT_READ (522), surfaced as the observed
Query(Database(SqliteError { code: 522, message: "disk I/O error" })).

close() awaits the teardown, so the window ceases to exist rather than
merely narrowing. This mirrors what the production caller
(DatabaseEngine::close_and_salvage) already does and what the sibling
stage-1 detection test already did. Record the precondition on
SqliteRecoveryEngine::salvage so the next caller does not repeat it.
 2026-08-14 20:05:13 -05:00
james.maes 4859424a51 Merge pull request #420 from KofTwentyTwo/fix/headless-cli-token 
fix(cli): let the CLI take its gRPC token from the environment
 2026-08-14 19:53:59 -05:00
james.maes 318aa9664b Merge pull request #418 from KofTwentyTwo/perf/list-messages-placements 
perf(grpc): batch ListMessages placements; fix JMAP parse folder
 2026-08-14 19:51:49 -05:00
james.maes 4732d6e786 fix(cli): let the CLI take its gRPC token from the environment 
nunciod mints the gRPC bearer token into the OS keyring, so the keychain
item's ACL trusts nunciod. nuncio-cli is a different binary, so reading
that item raises an interactive consent prompt. Nothing can answer that
prompt from CI, ssh, or a background shell, and the blocking keyring read
never returns -- the CLI hung with no output and no error.

Add an opt-in NUNCIO_GRPC_TOKEN override: when set and non-empty the CLI
uses it verbatim and never touches the keyring. The keyring remains the
default and is otherwise unchanged.

The keyring read itself can no longer stall forever either. It now runs on
a detached OS thread behind a bounded timeout that reports what is blocking
and how to bypass it. A detached thread rather than spawn_blocking, because
Tokio joins its blocking pool during runtime shutdown, which would just move
the hang to process exit.

Collapse the eight copies of the token-resolution block onto that one
resolver so no call site can drift.

Refs #419
 2026-08-14 19:46:23 -05:00
james.maes 4bbb23be3d fix(jmap): parse Email/get for the folder being synced 
`parse_email_get_response` labelled every placement it produced with a
fixed `"inbox"`, so a caller outside the sync loop got occupancies keyed
by a mailbox the account may never sync -- and a placement keyed by a
folder that is never enumerated can never be matched again.

Take the folder id as a parameter and drop the private in-folder variant
the sync loop had been calling to work around the fixed label. Callers
now say which mailbox they are parsing for.
 2026-08-14 19:41:55 -05:00
james.maes 9cc9292912 perf(grpc): batch placement reads for ListMessages 
Populating `Message.placements` cost one query per message, so a page of
fifty messages cost fifty-one round trips against the store. Add a
batched store read that fetches a whole set of message keys in one
statement per chunk and groups the result by key, and use it from the
listing path.

The batch chunks at the existing bind-parameter ceiling and orders by
folder id within each key, so the first placement remains the message's
canonical mailbox exactly as the per-message read had it.
 2026-08-14 19:40:16 -05:00
james.maes b1282bb71f Merge pull request #417 from KofTwentyTwo/story/mutation-rpcs 
feat(mail): mutation RPCs, typed conflicts, and the placement set
 2026-08-14 18:57:56 -05:00
james.maes d05659e05b Merge pull request #416 from KofTwentyTwo/story/imap-tier1-identity 
feat(imap): key messages by their server object id
 2026-08-14 18:48:43 -05:00
james.maes 619d37f3ab test(mail): cover placement-addressed mutations and conflicts offline 
Prove the new wire surface end to end without a network: a multi-mailbox
message refuses an unaddressed move and accepts an addressed one; the
queued outbox row really carries the occupancy the caller named; naming
an occupancy the message does not hold is refused rather than re-aimed;
reads report every occupancy while folder_id still names the one they
arrived through; conflicts survive from record through resolution; and
filters_enabled round-trips over UpdateAccount.

Also pin the executor half: a mutation whose payload names a placement
dispatches against exactly that occupancy instead of hitting the
ambiguity refusal.

Refs #395
 2026-08-14 18:47:43 -05:00
james.maes 93f00cdd0b feat(imap): key messages by their server object id 
IMAP hardcoded no object id into the message key, so it could only ever
reach the Message-ID+content tier or the folder-scoped surrogate, while
JMAP entered at the top tier from the server's Email id. One account
synced through both protocols therefore minted disjoint key spaces.

RFC 8474 EMAILID is the same identifier JMAP exposes, so requesting it
makes the two keys equal. Negotiate per capability: EMAILID only where
the server advertises OBJECTID, X-GM-MSGID only where it advertises
X-GM-EXT-1. A server offering neither behaves exactly as before.

EMAILID cannot go through the typed client: imap-proto's msg_att parser
is a closed alt with no catch-all, and async-imap turns the resulting
nom error into a fatal io::Error that closes the session. The raw layer
reads it off the transport beneath ImapStream instead, so an attribute
the crate cannot parse can never reach its decoder.

Lifting stored rows to a stronger tier changes every message_key, so
IDENTITY_SCHEMA_VERSION moves to 3 and the existing reconciliation
rebuilds the cached store rather than repointing rows one at a time.
 2026-08-14 18:41:10 -05:00
james.maes 5402dfb038 feat(mail): make messages hand-drivable over the wire 
The engine already executed verified remote mutations, but the Mail
service had no way to ask for one, so a human could not move, delete or
flag a message at all. Add MoveMessage, DeleteMessage and FlagMessage,
each addressing a specific mailbox occupancy rather than a bare message
key: identity is content-addressed and folder-independent, so one message
can sit in several mailboxes, and aiming a MOVE or DELETE at the wrong
occupancy is destructive and has no undo. The outbox already refused an
ambiguously-addressed mutation permanently; the queued row now carries
the occupancy its intent was formed against, which turns that refusal
into something a caller can avoid instead of a dead end. A named
occupancy is honoured only after confirming the message still holds it.

Conflicts become a typed error and durable state rather than a hopeful
field. Append ERROR_REASON_CONFLICT (13) and surface it through the
existing ErrorInfo status-details mechanism, so a client predating it
takes the error path instead of misreading success; a conflict is raised
only on the server's positive evidence, never on an unverifiable outcome.
Because the event stream is a lossy broadcast with no resumption, the
conflict is also written to a queryable table, with ListConflicts and
ResolveConflict over it.

Message now carries its complete placement set. folder_id keeps its
meaning -- the occupancy the message was reached through -- so this is
purely additive; its doc comment now points at the new field instead of
saying no such field exists.

filters_enabled reaches the wire. It was per-account and settable only by
hand-editing the accounts row, which made filters untestable by hand; it
is now on AccountConfig and flippable through UpdateAccount, still
defaulting to off because filters take irreversible remote actions.

CLI verbs land alongside: mail move/delete/flag/unflag, mail
conflicts/resolve, and account filters.

Refs #395
 2026-08-14 18:36:40 -05:00
james.maes ff06925dce Merge pull request #415 from KofTwentyTwo/docs/path-to-manual-use 
docs: chart the shortest route to a hand-drivable daemon
 2026-08-14 18:15:47 -05:00
james.maes b6987746ab docs: chart the shortest route to a hand-drivable daemon 
The engine is ahead of its contract: it executes verified mutations but the
Mail service has no RPC to ask for one, so a human cannot move, delete or
flag anything. By this repo's definition of done the placement work has an
engine and nothing else.

Records the three steps that buy manual usability, what works and what does
not if you boot it today, the 81-issue backlog behind them, and the two test
traps (tracing callsite Interest, tokio::time::pause over real I/O) that are
invisible at default parallelism.
 2026-08-14 18:08:52 -05:00
james.maes 36cacc07e2 Merge pull request #414 from KofTwentyTwo/story/jmap-email-changes 
feat(jmap): enumerate changes so deleted mail stops haunting the store
 2026-08-14 17:56:47 -05:00
james.maes 7ba4d45641 Merge pull request #413 from KofTwentyTwo/chore/placement-followups 
chore: placement model follow-ups (reap on repoint, claim lifetime, sync naming)
 2026-08-14 17:52:41 -05:00
james.maes b4292b00b0 feat(jmap): enumerate changes so deleted mail stops haunting the store 
JMAP's folder sync could only ever add. It returned `removals: []` and
`present: None` unconditionally, so mail deleted or moved by any other
client -- another daemon, a phone, webmail -- stayed in the local store
forever and every JMAP account accumulated ghosts indefinitely.

`sync_changes` now resumes through `Email/changes` (RFC 8621 s4.2 over
the RFC 8620 s5.2 core method):

- `created`/`updated` become upserts, with bodies fetched by `Email/get`
- `destroyed` becomes placement removals
- `hasMoreChanges` is followed to convergence, bounded so a server that
  never advances its `newState` cannot pin the sync task
- `cannotCalculateChanges` falls back to a full enumeration, the one
  pass entitled to report `present: Some(...)` because it is the only
  one that genuinely looked at the whole mailbox

An ordinary incremental pass still reports `present: None`: it never
enumerated the folder, and claiming otherwise would delete every message
that simply did not change.

`Email/changes` is account-wide, so a folder pass sees ids belonging to
other mailboxes. `Email/get` now requests `mailboxIds`, which both keeps
those ids from being placed here and reconciles away a message another
client moved out of this folder.

Checkpoints are scheme-tagged (`v1:jmapstate:{state}`) like the IMAP
ones, since a JMAP state is an opaque token with no self-describing
shape. An untagged or unknown token re-enumerates rather than being
handed back to a server that never issued it.

Placements now carry the mailbox that was actually synced. Removals are
addressed by that same key, so a placement stored under any other folder
id could never be reconciled away again.

Refs #408
 2026-08-14 17:49:46 -05:00
james.maes fdbd1a51ca chore(mail): rename folder sync to match what it returns 
Refs #410
 2026-08-14 17:40:02 -05:00
james.maes 1a94716abb docs(store): state the lifetime of a filter fire claim 
Refs #409
 2026-08-14 17:37:44 -05:00
james.maes 62e1b40998 fix(store): reap the message a repointed placement strands 
Refs #406
 2026-08-14 17:37:19 -05:00
james.maes 024cb2e9ad Merge pull request #399 from KofTwentyTwo/story/filters-enabled-per-account 
fix(filter): make filter execution single-owner per account, default off
 2026-08-14 15:38:29 -05:00
james.maes 9a872245c4 Merge remote-tracking branch 'origin/dev' into story/filters-enabled-per-account 
dev split message identity from placement, made filter evaluation per
placement, and replaced the schema-migration helpers with create-only
schemas. This branch's single-owner filter execution had to be re-seated
on top of all three.

The account gate stays where it was in intent -- a daemon that does not
own an account runs none of that account's rule actions -- but now sits
outside the per-placement fire-once claim rather than beside a per-message
one. On the sync path the gate is checked with the new-arrival test before
`apply_filter_actions` is reached, so a non-owner never takes a claim it
has already decided not to act on; leaving the claim unconsumed keeps it
available to the daemon that does own the account. Triage does the same
one level up: a message whose account this daemon does not own is counted
as scanned but its placements are never even read, and a daemon owning no
account still refuses the RPC outright instead of streaming an empty scan.

`filters_enabled` is now only a column in the created `accounts` schema;
the additive `ALTER TABLE` helper and its two migration tests are dropped,
since dev creates schemas and never upgrades them. The flag's persistence
is still covered -- the identity-rebuild test now asserts filter ownership
survives a rebuild, and the store round-trip assertions are unchanged.

The non-owner sync test keeps every assertion it had, translated to the
placement API: the mail is still stored, the occupancy's read flag is
still unflipped, and no execution log or outbox mutation is written.
 2026-08-14 15:30:49 -05:00
james.maes c40f6f05cd Merge pull request #385 from KofTwentyTwo/docs/multi-engine-sync-model 
docs(adr): record the convergent multi-engine sync model
 2026-08-14 15:21:58 -05:00
james.maes bcf6c280c7 Merge pull request #405 from KofTwentyTwo/story/message-placement-split 
feat(store): split message identity from mailbox placements
 2026-08-14 15:20:55 -05:00
james.maes 25a7c4a001 Merge remote-tracking branch 'origin/dev' into story/message-placement-split 
# Conflicts:
#	crates/nuncio-store/src/db.rs
 2026-08-14 15:13:29 -05:00
james.maes 73c85c0eeb Merge pull request #403 from KofTwentyTwo/story/verified-mutations 
feat(mail): report what a mutation proved, not that it was accepted
 2026-08-14 15:08:37 -05:00
james.maes 77eee1a7d1 Merge remote-tracking branch 'origin/dev' into story/verified-mutations 2026-08-14 14:55:25 -05:00
james.maes 48a77ef55d Merge pull request #402 from KofTwentyTwo/story/imap-change-enumeration 
feat(sync): enumerate IMAP changes over a four-rung capability ladder
 2026-08-14 14:55:23 -05:00
james.maes c17dbcb15b Merge remote-tracking branch 'origin/dev' into story/imap-change-enumeration 
# Conflicts:
#	crates/nuncio-store/src/db.rs
 2026-08-14 14:42:25 -05:00
james.maes f4e9189e0f Merge pull request #401 from KofTwentyTwo/story/imap-raw-command-layer 
feat(mail): add a raw IMAP layer for ENABLE, tagged codes, and VANISHED
 2026-08-14 14:38:52 -05:00
james.maes fcf62e1105 Merge remote-tracking branch 'origin/dev' into story/imap-raw-command-layer 2026-08-14 14:10:28 -05:00
james.maes 9d58883632 Merge pull request #384 from KofTwentyTwo/story/imap-mock-harness 
test(mail): stateful IMAP mock harness + fix a flaky webhook capture test
 2026-08-14 14:09:51 -05:00
james.maes a747b3a32e Merge remote-tracking branch 'origin/dev' into story/imap-mock-harness 
# Conflicts:
#	crates/nuncio-filter/src/test_tracing.rs
#	crates/nuncio-filter/src/webhook.rs
 2026-08-14 13:42:20 -05:00
james.maes c712adbe6e Merge pull request #411 from KofTwentyTwo/fix/test-flakes-tracing-race 
fix(test): install one global capture subscriber; delete schema migrations
 2026-08-14 09:46:38 -05:00
james.maes bbebd576d8 fix(test): time out outbox mutations in real time, not on a paused clock 
The two hung-mutation tests paused the Tokio clock so auto-advance would
jump the 30s per-item timeout. That cannot be made reliable: the executor
does real SQLite I/O on a blocking pool inside the same window, and
auto-advance fires whenever the runtime looks idle. It also fires sqlx's
own pool-acquire timer, which defaults to the same 30 seconds -- turning a
healthy list_pending_mutations into PoolTimedOut, whose error arm returns
an all-zero summary. That is exactly the shape CI reported: completed 0
and failed 0 both as expected, retried 0 instead of 1.

Make the per-item timeout an overridable method on RemoteExecutionEnv and
give the hanging test env 250ms, so the tests wait it out in real time and
no test pauses the clock. The hang is infinite, so a loaded runner makes
these slower, never wrong.

50 runs at --test-threads=2 under 16-way CPU load: 0 failures.
 2026-08-12 13:10:12 -05:00
james.maes 15226b62a1 refactor(store): create the schema, never migrate it 
Four ALTER TABLE helpers upgraded a pre-existing on-disk schema: smtp
columns, TLS-mode columns, the DAV collection_url, and the messages
identity columns. Every column they add is already declared in the
CREATE TABLE IF NOT EXISTS in the same migrate(), so on any database
this engine actually creates they were unconditional no-ops.

Nothing is released and every store is a local development database, so
there is no older schema to meet. Delete the helpers and the three tests
that existed only to exercise them.

backfill_message_fts stays: salvage restores rows straight into messages,
bypassing save_email's index write, so a recovered database really does
have content the trigram index has never seen. Its doc no longer claims
the legacy justification.

Removes 495 lines, and with them the migrate_logs flake outright.
 2026-08-12 12:32:20 -05:00
james.maes 1cd1bcc13c fix(test): capture logs through a global subscriber in 4 more crates 
Ports the nunciod fix (99a8d1a) to nuncio-cal, nuncio-contacts,
nuncio-store, and nuncio-filter: install the tracing capture
subscriber once as the process default and route events to a
per-thread buffer, instead of a thread-local subscriber that a
racing callsite's cached Interest::never() can permanently defeat
under a parallel test harness.

nuncio-cal and nuncio-contacts get the same structured Recorder
shape as nunciod. nuncio-store gains a formatted-text test_tracing
helper (its assertions match on rendered level tags and substrings)
used by vault.rs; the two migration-log CapturedLogs sites in db.rs
are left untouched because those tests are being deleted separately.
nuncio-filter keeps its dependency-free hand-rolled Subscriber but
makes it global and per-thread-routed, replacing the local
CapturingSubscriber in engine.rs and webhook.rs.
 2026-08-12 12:26:46 -05:00
james.maes 99a8d1a96f fix(test): capture logs through one global subscriber, not a thread-local 
tracing resolves a callsite's Interest once per process and caches it. A
callsite first executed on a thread with no subscriber caches never and is
skipped forever, before any thread-local subscriber is consulted. Under the
parallel harness, whichever thread reached a callsite first decided whether
any other test could capture it, so every log-capture assertion in this
crate was a coin flip -- reproducible at --test-threads=2, which is what CI
runners give us.

Install the capture layer once as the global default and route events to a
per-thread recorder instead. Rebuild the interest cache once at install so
callsites already resolved against the no-op dispatcher are recoverable.

Measured on nunciod: ~90% of runs red at --test-threads=2 before, 0/12 after.
 2026-08-12 11:47:34 -05:00
james.maes 4f6a024d3b docs(roadmap): reconcile the prose WS total with the table 2026-08-12 10:46:08 -05:00
james.maes 79f357864a fix(store): chunk IN-list lookups below SQLite's bind cap 
existing_placements binds four parameters per key into one row-value
IN (...) list, and its caller hands it an entire folder: FolderChanges
accumulates every batch into one Vec, so the Full rung -- an ordinary
first sync, and the unrecognised-checkpoint fallback -- passes the whole
mailbox. SQLITE_MAX_VARIABLE_NUMBER is 32766, so past 8191 placements
the statement fails, the folder sync aborts before writing a checkpoint,
and every retry fails identically. A first sync of any large mailbox was
permanently broken.

Split the batch internally at 4000 keys and union the results, so the
limit is unreachable from outside rather than one forgetful caller away.
existing_message_ids had the same shape at a 32766 ceiling and is
chunked the same way; it stays because it is public API drawing the
identity-vs-occupancy distinction, not because it has callers.

Also correct three comments that documented traps as free:

- The IMAP batching doc claimed the working set stays bounded regardless
  of mailbox size. Batching bounds each request, not memory; the parsed
  messages accumulate across batches.
- Adding the EMAILID/X-GM-MSGID FETCH items is not free: it repoints
  every stored message's key, orphaning the old messages row and
  stranding its decrypted body in messages_fts. It needs the repoint
  path to reap first, or it leaks plaintext.
- IMAP never requests the tier-1/2 identifiers, so an IMAP-derived key
  and a JMAP-derived key for the same message differ. Recorded on
  derive_message_key and on the IMAP place() helper.

De-link ADR 0002 in the roadmap: the file exists on no merge target, so
the markdown now matches the rendered page, which already used plain
bold.
 2026-08-12 10:45:28 -05:00
james.maes 7fadc04662 docs(roadmap): refresh WS counts and record the ADR 0002 split 
WS-B, WS-C and WS-E grew when ADR 0002 broke the multi-engine sync model into
#386-#397, and the 2026-08-07 snapshot never caught up. Counts now match the
live milestones. Nothing is marked closed: the identity split and its chain are
green on a branch, and the numbers move when it merges.

Refs #394
 2026-08-12 10:27:43 -05:00
james.maes 1be51907ab docs(proto): stop overstating when a message id survives a move 
The surrogate tier hashes the folder, so a message with no usable Message-ID
-- including any fetched headers-only -- gets a folder-scoped id that changes
when it moves. Saying ids survive a move unconditionally was false for exactly
the messages most likely to have one.
 2026-08-12 10:21:11 -05:00
james.maes dbe7acdb72 docs(proto): describe content-addressed ids and placement-scoped folders 
Message.id is no longer a folder-scoped surrogate: it is content-addressed
from the strongest identity the server offered, so it survives a move.
Message.folder_id names the one occupancy the message was reached through,
since a message can sit in several folders at once.

Triage's contract is stated too -- it claims each (rule, message) pair once
and for all, so a second run over the same rules matches nothing.

Refs #394
 2026-08-12 10:12:56 -05:00
james.maes 4f1dc296f1 fix(outbox): refuse an ambiguously-addressed mailbox mutation 
An outbox row records only a message key, so for a message occupying
several mailboxes there is no way to know which copy the rule matched.
Addressing whichever placement sorted first meant a rule that matched
the INBOX copy could MOVE or DELETE the Archive one -- against mail the
user never targeted, with no undo, and Gmail labels make multi-placement
ordinary rather than exotic. Unresolvable addressing is no more a licence
to act than an unverifiable claim: it now fails permanently, naming the
competing mailboxes. Single-placement mutations are unaffected.

Also from review: PreviewRule no longer substitutes an invented inbox
occupancy when a stored message's placements cannot be read, which let
WHERE FOLDER match a location the message does not have; explicit
VANISHED removals no longer need ambient account context, since a
PlacementKey names its own account; and both Mail read RPCs now state
that they list across configured accounts only, so removing an account
hides its cached mail by design rather than by accident.

Covers the cross-account merge shims, which nothing exercised: summed
counts for a shared folder id, merged ordering across page boundaries,
and a page boundary that only shows up when one account owns the folder.
 2026-08-12 09:58:22 -05:00
james.maes 1260dd0a09 feat(sync): classify arrivals per placement and claim fires once 
Message identity is folder-independent, so a message already stored from
INBOX has a known key when it turns up in Archive. Classifying "is this
new?" by message key answered "seen" for a mailbox nothing had ever
filtered, and the arrival silently fired no rule at all -- no error, no
log. fetch_and_persist now classifies on PlacementKey via
existing_placements, persists through save_email_at, and reconciles
absence with placements_in_folder/delete_placements.

Evaluation is therefore per placement, so acting has to be gated
separately: apply_filter_actions takes the placement it evaluates
against and must win claim_filter_fire, keyed on message identity,
before any action runs. MOVE and FLAG converge under repetition;
FORWARD and CALL WEBHOOK land on third parties and do not. The claim
fails closed -- an unverifiable claim skips the actions rather than
licensing them. MARK READ now names the occupancy it was evaluated
against, since IMAP \Seen is per-mailbox.

The daemon's gRPC and outbox surfaces are ported only as far as
compilation and honest behaviour require, and say so in place: the
outbox recovers its remote addressing from a placement and warns when
the row cannot say which one the rule matched, and the Mail read RPCs
merge across configured accounts pending an account on the wire.
 2026-08-12 09:36:16 -05:00
james.maes 1d113ef11a fix(mail): report no absence when a mailbox omits UIDVALIDITY 
A SELECT that omits [UIDVALIDITY] left every placement key this pass built
carrying an empty generation. None of them match placements stored under a real
one, and under the `present` contract -- delete any placement absent from this
list -- that reads as "the folder holds nothing", so the caller would delete the
whole mailbox. The pass now declines to report absence at all, mirroring the
checkpoint path, which already refuses to write a validity-less value, and warns
so the reason is visible. Upserts are still surfaced.

[UIDVALIDITY] is REQUIRED on SELECT (RFC 3501 section 6.3.1), so this is a guard
against a broken server rather than an expected path -- but the impact is a
wiped mailbox.

Also pin the identity tier on the two branches that fetch no octets. The
headers-only branch drops its hash because it covers only the header section;
publishing it would key the message on the wrong octets and mint a second key
once the body arrived, resurrecting the message as a duplicate. The fixtures now
carry a Message-ID so those branches reaching the surrogate is a real assertion
rather than an accident of the fixture, and the full-body path is asserted to
reach the content tier for contrast.

Give PlacementKey and Placement::key() direct tests: all four coordinates
reproduced, the read flag excluded, each coordinate distinguishing, usable as a
set key and sortable, and a serde round trip.
 2026-08-12 09:07:18 -05:00
james.maes e558b22c32 Merge branch 'story/verified-mutations' into story/message-placement-split 
Reconciles IMAP change enumeration with the message/placement identity split.
The two lines both rewrote how a folder sync reports itself, and resolving them
separately would have meant migrating FolderChanges twice.

FolderChanges wins as the structure, retyped for placements:

  upserts:  Vec<PlacedMessage>       (identity, content, and the occupancy)
  removals: Vec<PlacementKey>
  present:  Option<Vec<PlacementKey>>
  next_state: String                 (unchanged)

Absence is per-occupancy because that is all a folder can speak to. A UID that
vanished from one mailbox says nothing about copies elsewhere, and a message
key cannot be reconstructed from a UID anyway now that identity depends on
which tier the server supported -- so the incoming side's surrogate-per-UID
removals would have named rows the store no longer writes.

PlacementKey moves to nuncio-core so both nuncio-mail and nuncio-store can name
one type; nuncio-store re-exports it and does not gain a reverse dependency.
The store's read-side (Email, Placement) alias becomes MessageWithPlacement, so
it no longer collides with nuncio-mail's PlacedMessage in the sync path.

The four-rung capability ladder is preserved intact, per mailbox: QRESYNC,
CONDSTORE, UID-set diff, full resync. JMAP keeps present: None -- it still has
no Email/changes and genuinely cannot speak to absence.

The incoming message_ids_in_folder and delete_messages are dropped: they read
messages.folder_id, which the placement schema removed, and delete_messages
would reap a message on one folder's silence. placements_in_folder and
delete_placements already do the placement-correct equivalent.
 2026-08-12 08:56:01 -05:00
james.maes bc72571f7a feat(mail): derive message identity and emit placements 
The IMAP and JMAP backends no longer mint a folder-scoped surrogate id for
every fetched message. They derive the account-scoped message key by the
identity precedence and report it as a PlacedMessage: the Email (identity and
content), the tier that produced the key, and the Placement the fetch found it
in. Read state moves onto the placement, where IMAP \Seen already lives.

IMAP does not request EMAILID (RFC 8474) or X-GM-MSGID yet, so identity rests
on the Message-ID/content pair where a Message-ID was captured and on the
folder-scoped surrogate otherwise; adding those FETCH items later lifts stored
rows to a stronger tier on the next resync. A JMAP Email id is server-assigned
and account-stable, the same guarantee EMAILID gives, so JMAP enters at the top
tier instead.

MimeParserAdapter::parse_mime drops its id and folder_id arguments: identity
cannot be settled until the parsed Message-ID and content hash are in hand
alongside the transport's coordinates, so the caller assigns it.
 2026-08-11 15:43:08 -05:00
james.maes adef7fc1b6 fix(store): harden placement queries, paging, and write locking 
Five defects found in review:

- existing_placements never pinned that a never-saved key is absent, and
  its negative case was vacuous (no Archive row existed to over-report).
  Over-reporting makes genuinely new mail look already-seen and be
  skipped by the sync classifier. Both are now real assertions, backed
  by a count so a dropped IN filter fails.
- save_email_at and delete_placements both read then write in one
  deferred transaction. In WAL mode that aborts with SQLITE_BUSY, which
  busy_timeout does not retry; twenty racing saves reproducibly failed.
  Both now use BEGIN IMMEDIATE, matching the WORM and ledger appenders,
  with a concurrency regression test.
- list_messages_page claimed the folder join was one-to-one. It is not:
  the placements key permits one message in one folder twice, and the
  message-only cursor made the second row unreachable across a page
  boundary. The keyset now carries the full placement address.
- list_messages_for_export matched a folder id without its account,
  reintroducing the cross-account collision this model removes.
 2026-08-11 15:31:06 -05:00
james.maes eb506bad53 refactor(core): rename IdentitySource::from_str to from_token 
The old name tripped clippy::should_implement_trait, which fails the
workspace lint gate and therefore CI. Implementing std::str::FromStr
instead would force a Result and a new error type when the only failure
is "not one of four known tokens" -- Option already says that. The
round-trip test now also pins that an unrecognised token returns None.
 2026-08-11 15:31:06 -05:00
james.maes d4815ef768 feat(store): reap a message only when its last placement goes 
A server that stops reporting a UID has said the message left that
mailbox, not that it ceased to exist -- a move reports exactly this in
the source folder. So a placement is dropped unconditionally and the
message only when its last placement goes.

That reap is load-bearing for confidentiality: messages_fts holds
decrypted bodies and is cleared by the messages_ad trigger on messages,
so deleting the message row is the only thing that clears the index.
Skip it and a deleted message leaves its plaintext body readable to
anyone with filesystem access; invert it and search breaks for mail that
still exists. Both directions are pinned by tests.
 2026-08-11 15:15:05 -05:00
james.maes 882d2f12a2 feat(store): read messages through their placements 
Reads now separate what a message is from where it sits: get_message
returns identity and content alone, list_messages joins through
placements and hands back the placement it found the message via, and
folders are derived by grouping placements per account -- folder ids are
not globally unique, so grouping across accounts would report one merged
INBOX holding two accounts' mail. Read state is updated per placement
because IMAP \Seen is per-mailbox. An FTS hit now names a message key,
not a placement, since the index holds one row per message.
 2026-08-11 15:14:50 -05:00
james.maes 031961ebc1 feat(store): write message identity and placement separately 
A message's content is now written once, keyed on its folder-independent
message key, while each mailbox it occupies gets its own placement row.
The message row is first-write-wins so a later sighting claiming a known
key cannot rewrite a body the user already trusts, and the plaintext FTS
row is written only on the insert that actually created the message.
The placement is upserted, because its read flag genuinely changes.
 2026-08-11 15:05:14 -05:00
james.maes da7b67ba01 feat(filter): evaluate rules against a placement, not a message 
A message can occupy several folders at once, so FOLDER and ACCOUNT
conditions have no single answer at the message level. PlacedEmail
pairs a message with the one mailbox occupancy under evaluation;
evaluate_condition, evaluate, evaluate_with_timeout, and preview now
take a PlacedEmail and read Folder/Account off its Placement, while
content fields still read the Email. Callers fan out over a message's
placements and evaluate each independently.
 2026-08-11 14:58:19 -05:00
james.maes 09c3069d57 feat(store): add placements, fire-once claims, and bump identity schema 
Split the messages table into identity/content only (message_key
primary key, identity_source column) and a new placements table
holding one row per (account, folder, uidvalidity, uid) mailbox
occupancy. uidvalidity is part of the placements primary key because
UIDs restart at 1 after a bump and would otherwise collide with rows
from the prior generation.

Add filter_fired as a fire-once ledger for filter actions, keyed on
message identity so a message arriving in a second folder cannot
re-trigger a side-effecting rule that already ran against it. Add
claim_filter_fire (atomic INSERT ... ON CONFLICT DO NOTHING),
has_filter_fired, and filter_fire_count as public store methods so
other crates can observe and use the ledger without reaching the
connection pool.

Bump IDENTITY_SCHEMA_VERSION to 2 so reconcile_schema_version rebuilds
the cached store on upgrade, and extend reset_cached_message_store to
clear placements and filter_fired alongside messages. Retire the now
-obsolete ensure_messages_identity_columns additive migration and its
test: the remote_id/uid_validity columns and the identity index it
backfilled no longer exist on messages, and placements' own primary
key now provides the uniqueness guarantee it used to build. Update
backfill_message_fts and the messages_ad trigger for the message_key
rename, keeping the trigger on messages (not placements) so dropping
one placement of a still-present message never orphans its FTS row.
 2026-08-11 14:29:11 -05:00
james.maes 2c9ed5a218 docs(core): explain dropped mbox folder header, broaden test 
export_mbox stopped emitting X-Nuncio-Folder when folder_id moved off
Email onto Placement, but the reason lived only in an SDD report, not
the code. Add a comment at the removal site so a future reader knows
it's intentional and what restoring it would require (threading a
Placement into export_mbox's signature).

Also broaden message_key_is_account_scoped_at_every_tier, which
claimed to cover every identity tier but only exercised EmailId;
it now asserts account-scoping at all four tiers.
 2026-08-11 14:06:12 -05:00
james.maes 590e9a40f2 feat(core): separate message identity from mailbox placement 2026-08-11 14:00:45 -05:00
james.maes 1aacf74509 docs(plan): record pre-flight rulings on gate, ledger access, scoping 2026-08-11 13:56:45 -05:00
james.maes d138dbf65d docs(plan): plan the message/placement identity split 
Refs #394
 2026-08-11 13:46:56 -05:00
james.maes 99c01e5e0e Merge branch 'story/schema-version-and-token-scheme' into story/message-placement-split 2026-08-11 13:41:58 -05:00
james.maes 5a03ab13e3 feat(mail): report what a mutation proved, not that it was accepted 
A mutation reported success whenever the server answered OK. IMAP answers OK
for a UID command that matched nothing (RFC 3501 6.4.8), so a mutation that
lost a race to another client -- the message already moved, already deleted --
was recorded as completed having done nothing. The user's intent was silently
dropped and the outbox row was marked done.

Two states cannot express this. "Succeeded" and "failed" leave out the case
that actually occurs: the server accepted the command and its response does not
establish what happened. Collapsing that into either neighbour is the bug, so
mutations now return Applied, Conflict, or Unknown.

Applied requires positive evidence. A COPYUID naming destination UIDs proves a
move; a removal report proves a delete; an echoed FETCH proves a flag change.
Conflict requires positive evidence too -- a MODIFIED response code -- because
MODIFIED means "changed by someone else", never "gone". Everything else is
Unknown, which is the honest reading of an absent COPYUID: only SHOULD for MOVE
(RFC 6851 4.3), legitimately omitted for a UIDNOTSTICKY destination, and
indistinguishable from a UID set that matched nothing.

The outbox treats Unknown as retryable. A mutation that did apply and is
re-attempted is a no-op; one that did not and is recorded complete is
unrecoverable, so the asymmetry decides it. Conflict is terminal and visible
instead: the intent was formed against state that no longer exists, and
re-issuing it would either do nothing or overwrite someone else's change.

The delete path now reads its proof through the collecting layer, because a
QRESYNC-enabled server reports VANISHED rather than EXPUNGE (RFC 6851 4.4) and
the typed expunge stream only yields the latter -- reading it alone reports
nothing removed on every successful delete.

Refs #391
 2026-08-10 17:06:02 -05:00
james.maes 1f75527563 feat(sync): enumerate IMAP changes over a four-rung capability ladder 
Sync could only ever add. It resolved a UID boundary, fetched forward from it,
and stored what came back; nothing ever asked what stopped being there. A
message moved or deleted by any other client -- another daemon, a phone,
webmail -- stayed in the local store forever, and every account accumulated
ghosts. This is the defect, and it affects a single engine just as much as ten.

A folder pass now reports what changed, what vanished, and honestly whether it
was able to observe absence at all. The last part is why `present` is an
Option: None means the pass was incremental and cannot speak to absence, while
Some(empty) means the folder is genuinely empty. Conflating those deletes a
mailbox.

Four rungs, chosen per mailbox rather than per account, because NOMODSEQ is a
mailbox property (RFC 7162 3.1.2.2) and one folder can be unable to answer
mod-sequence queries on a server that otherwise can. QRESYNC has the server
state removals outright. CONDSTORE narrows the fetch but never volunteers them,
so absence comes from enumerating what remains -- this is Gmail's rung, which
advertises CONDSTORE and not QRESYNC, and a two-rung ladder would have pushed
the largest provider onto full sweeps against a published daily byte cap.
Neither extension leaves a UID boundary for arrivals plus an enumeration for
removals; that is Exchange. No usable checkpoint fetches everything.

The enumeration passes ask for UID only -- no bodies, no flags -- so detecting
removals costs one short line per message rather than a download. A mailbox
that can supply a mod-sequence gets a mod-sequence checkpoint even after a full
pass, so the next sync starts on a higher rung than this one did.

A UIDVALIDITY change is not a downgrade to a lesser rung, it is a full pass:
UIDs restart, so every stored boundary names a different message. The dangerous
reading would be to enumerate the new UID space, find none of the stored ids in
it, and conclude the folder was emptied.

The raw collector also gains a bounded read. A server that accepts a command
and then stops answering must surface as a stall rather than hanging the sync,
which is the same failure the typed fetch path already bounds.

Refs #390
 2026-08-10 16:50:48 -05:00
james.maes 9551f51b55 Merge branch 'story/schema-version-and-token-scheme' into story/imap-change-enumeration 2026-08-10 16:21:54 -05:00
james.maes bf19b9f3a6 feat(mail): add a raw IMAP layer for ENABLE, tagged codes, and VANISHED 
The typed async-imap API cannot express the four responses that decide whether
a sync saw everything or whether a mutation actually happened. ENABLE does not
exist, and RFC 7162 3.2.3 makes a server answer BAD to a QRESYNC SELECT without
it. select()'s parser has no arm for VANISHED, so a deleted message is simply
never learned about. And run_command_and_check_ok discards the tagged response
code, which is exactly where RFC 4315 puts COPYUID and RFC 7162 puts MODIFIED.

Hand-rolled rather than forked. run_command and read_response are public and
imap-proto already parses Vanished and CopyUid, so what was missing is a
collector that reads to the tagged response and keeps what arrived on the way.
That is a hundred lines against maintaining a fork for the same result.

Two subtleties the collector exists to get right. COPYUID arrives in an
UNTAGGED OK for MOVE (RFC 6851 4.3 advises exactly that), so a collector
reading only the tagged response misses it on every conforming server and then
has no proof the move happened. And UID EXPUNGE answers with VANISHED rather
than EXPUNGE once QRESYNC is enabled (4.4), so counting EXPUNGE alone reports
failure on every successful delete; both are gathered.

MODIFIED is the one real gap. imap-proto 0.16.7 has no ResponseCode variant for
it, and ResponseData's raw bytes are behind a private module, so it cannot be
read from the parse tree at all. Its resp_text is opt(resp_text_code) followed
by free text, so an unrecognised code is not an error -- the whole
"[MODIFIED 7,9]" survives in the information string, and that is where it is
recovered from. Parsing a response code out of a human-readable field is not
how this should work forever; upstreaming the variant is the real fix. It is
deterministic, it is pinned by a test against a server emitting the real
syntax, and the alternative is no conflict detection at all.

Absent COPYUID is deliberately not treated as failure. A UID set matching
nothing succeeds having done nothing (RFC 3501 6.4.8), and a conforming server
may also omit the code for a UIDNOTSTICKY destination, so absence means unknown.

Also fixes the mock server reading the first parenthesised group of a STORE as
its flag list, which made a conditional store apply UNCHANGEDSINCE and the
mod-sequence as flag names. The flag list is always the final argument.

Refs #388
 2026-08-10 16:08:45 -05:00
james.maes 1ec1e90772 feat(store): gate the identity schema and tag sync tokens with their scheme 
Two silent-failure paths, both cheap to close and both load-bearing for the
sync work that follows.

The schema is built entirely from CREATE TABLE IF NOT EXISTS, so an older
binary opens a newer database successfully and writes the old model into it.
Nothing detects that; the file ends up holding two mutually invisible truths.
A user_version generation now gates opening: newer than this build is a
fail-closed error, and older rebuilds the cached message store.

Rebuilding rather than migrating is the pre-release decision recorded in ADR
0002. Message identity cannot be recomputed for rows already stored, so the
alternative is keeping them under the old model forever -- a store where half
the mail is invisible to every guarantee the new model provides. Accounts,
credentials and rules are kept; cached mail, checkpoints and the execution
ledger are rebuilt from the server, which is authoritative regardless. The
ledger goes with them because its hash chain is keyed on message ids that would
no longer resolve, and a chain pointing at absent rows proves nothing.

Separately, folder checkpoints now carry a scheme tag. The stored value was
"{uidvalidity}:{uidnext}" and the QRESYNC checkpoint that replaces it is
"{uidvalidity}:{highestmodseq}" -- identical shape, different meaning. An
upgraded engine reading one as the other hands the server a UIDNEXT where it
expects a MODSEQ; UIDNEXT is normally the larger number, so the server answers
"nothing changed since" and every message below that point is never enumerated
again. That is silent, permanent mail loss with no error anywhere. Any token
whose scheme this build does not recognise, including an untagged one, now
forces a full re-fetch: over-fetching is recoverable, misreading is not.

Refs #387
 2026-08-10 15:34:24 -05:00
james.maes c6eca910c4 docs(adr): decide the pre-release posture and drop the migration compromise 
The first draft reasoned like a shipped product: preserve every stored row,
never change behaviour without a path, keep old and new daemons interoperable.
Nothing is released, so none of that is owed -- and paying for it distorted the
model in one place that matters.

Message identity cannot be derived for rows already stored, because Message-ID
was never persisted and the octets are discarded after parse. Honouring those
rows meant keeping them under their old surrogate forever: permanently
un-deduped, invisible to every cross-folder guarantee the design exists to
provide, and leaving the store two-tier for as long as the install survives.
Resetting the message store and re-syncing from the server -- which is
authoritative regardless -- costs a developer one sync and buys a single
coherent model.

The audit ledger follows the same reasoning rather than gaining a permanent
namespace boundary: a hash chain whose entries reference ids that no longer
resolve has no audit value, so it is reinitialised with the store it describes.

This also removes the dependency that shaped the build order. Placements waited
on captured identity accumulating only because history had to be preserved;
now capture need only land one merge ahead of the model consuming it. The order
is recorded explicitly, along with the note that change enumeration still ships
alone and still fixes ghost messages for a single engine.

The posture section is written to be deleted at the first real release, when
every one of these questions becomes a genuine obligation.
 2026-08-10 15:29:47 -05:00
james.maes ae8815263a fix(filter): make filter execution single-owner per account, default off 
Every daemon syncing an account decides for itself that a message is new,
because "new" means "not in this daemon's store". So every daemon fires the
same rules against the same message. MOVE, COPY and FLAG survive that -- ten
moves converge on one outcome -- but FORWARD and CALL WEBHOOK land on a third
party with no shared state to compare against. Ten daemons send ten forwards
and page an on-call engineer ten times per alert. Nothing in the outbox
constrains this either: pending_remote_mutations has no uniqueness of any kind.

The engine cannot fix this by itself. The servers coordinate state, not
actions, so there is no resource to compare-and-set against for an effect that
leaves the system. Until a server-side claim is available, ownership has to be
declared: filters_enabled on the account, and exactly one daemon per account
turns it on.

Default off, including for accounts that already exist. An account already
configured on several machines would otherwise have all of them start
forwarding the moment they upgrade. Off costs one setting to correct; a sent
forward cannot be recalled. The salvage path deliberately does not recover the
flag either, for the same reason.

A non-owning daemon still syncs and stores everything -- it is a full mail
client, it simply does not act. Triage is bound by the same rule since it
produces the identical side effects, and refuses outright when this daemon owns
no account rather than streaming a scan that reports progress and applies
nothing, which would read as "no rules matched".

Webhooks now carry an Idempotency-Key stable across retries of a rule/message
pair, so a redelivery caused by a lost response can be discarded rather than
acted on twice. It is not yet stable across daemons -- rule ids are random per
daemon -- which single-owner execution is what covers today.

Refs #389
 2026-08-10 14:57:58 -05:00
james.maes 3de75493d5 feat(mail): capture Message-ID and a content hash at ingest 
Identity cannot currently be reconstructed for a stored message. The parser
never reads Message-ID, the Email model has nowhere to put it, the messages
table has no column for it, and the raw octets are dropped once parsed. Any
later attempt to give a message a folder-independent identity therefore has
nothing to work from except a re-download of the entire mailbox.

Capture both now, as nullable columns nothing reads yet, so the data starts
accruing while the design that consumes it is still being built.

Normalization is fixed and shared rather than left to each call site: strip all
ASCII whitespace (a msg-id cannot legally contain any, so this unfolds a wrapped
header without knowing how the transport folded it), strip one pair of angle
brackets, and preserve case, since only the domain half is case-insensitive and
lowercasing wholesale would merge ids the origin considers distinct. Engines
that normalize differently would hold the same message under different bytes,
and no comparison between their caches would mean anything.

Captured at every ingest path, including the two that do not parse a full
message. The envelope-only path takes Message-ID from ENVELOPE rather than
skipping capture. The headers-only path keeps the id recovered from those
headers but discards the hash parse_mime computed over them, because
content_hash is defined over a message's full octets and publishing a
header-only digest under that name would quietly mean something else.

The hash exists to keep Message-ID honest: it is public, echoed in every
References header, and reused legitimately -- a mailing-list copy and a direct
copy share one, and a draft keeps one across every save -- so the pair
distinguishes messages that the id alone would merge.

No backfill. Legacy rows stay NULL, which reads as "not captured" rather than
"this message has no Message-ID"; inventing a value would be worse than
admitting the gap.

Refs #386
 2026-08-10 14:48:37 -05:00
james.maes 85c5d46ba0 docs(adr): record the convergent multi-engine sync model 
The engine syncs as though it is the only client of an account: a forward-only
UID high-water mark that cannot observe deletion, and fire-and-forget mutations
that report success when a UID command silently did nothing. A second client --
another machine, a phone, or webmail -- makes both assumptions false, so the
local view diverges and nothing detects it.

ADR 0002 decides how a daemon behaves as one of several clients. The core
choice is to build no coordination layer: IMAP and WebDAV already coordinate
concurrent clients, so the work is finishing the sync model rather than adding
multi-engine support, and correctness at ten engines follows from correctness
at one. The exception is side-effecting filter actions, which land on third
parties with no shared resource to compare against; those get a single owner.

A draft was reviewed independently for protocol correctness, codebase fit, and
adversarially. All three falsified load-bearing claims, and the corrections are
material enough to record rather than summarise: Gmail has no QRESYNC and
Exchange has neither QRESYNC nor CONDSTORE, so the ladder needs four rungs
selected per mailbox; an absent COPYUID is conforming on success and cannot mean
conflict; keying identity on a Message-ID hash would have let a forged message
overwrite a trusted body; and the proposed convergence test could not have
detected the design's largest hole. The review evidence lands alongside the ADR
because several findings cost real effort to establish and would otherwise be
rediscovered expensively.

Documentation only. No code or contract changes.
 2026-08-10 14:26:59 -05:00
james.maes 83636efcdb fix(filter): stop the webhook log-capture tests failing at random 
The two webhook telemetry tests installed a hand-rolled tracing::Subscriber
with set_default and asserted on what it recorded. Roughly three runs in ten
they captured nothing at all and failed.

Callsite Interest is cached process-globally. A raw Subscriber installed
thread-locally does not participate in the way tracing_subscriber's registry
handles that cache, so whichever sibling test reached the dispatch callsites
first decided whether these two could ever see them -- an order dependency,
and therefore a coin flip under a parallel test harness.

Two narrower fixes were tried and measured, and both made it worse:
rebuild_interest_cache() after set_default failed five runs in six, because
the rebuild is itself global and concurrent capture tests clobber each other;
returning Interest::sometimes() from register_callsite failed nine in ten,
because callsite registration consults the global dispatcher and never asks a
thread-local subscriber at all.

Adopt the registry-backed capture helper nuncio-cal and nuncio-contacts
already use, which does not exhibit the problem. The two tests become plain
#[test] driving a current-thread runtime built inside the capture scope, since
a #[tokio::test] creates its runtime outside it. Assertions now match on level
and named fields rather than substrings of a formatted line, so "the host was
logged" no longer passes because the host happened to appear in some other
field.

Measured: 0 failures in 12 stress runs, against 3 in 10 before.
 2026-08-10 14:14:41 -05:00
james.maes ac017ba3c4 test(mail): add a stateful multi-connection IMAP server for offline tests 
The IMAP client had no mock. wiremock is HTTP-only, so JMAP and the DAV
protocols have one and IMAP does not; what stood in for it was a scripted
responder rewritten inline in each test, holding one connection and no
mailbox state.

That shape cannot express the property the sync work needs to assert: two
clients observing each other's writes through shared server state. It also
cannot reproduce the response forms that decide whether a mutation actually
applied -- COPYUID, VANISHED, and a tagged [MODIFIED ...] -- so a client that
reports success for a no-op cannot currently be caught by a test.

The server binds a real loopback listener, shares mailbox state across every
accepted connection, and speaks IMAP4rev1 plus the parts of RFC 7162, RFC 4315,
and RFC 6851 the engine exercises. Capability advertisement is selectable,
because the sync ladder branches on it and a suite that only ever runs against
a QRESYNC server proves nothing about the path Gmail and Exchange take.

Behind a `test-support` feature so it stays out of production builds while
remaining available to dependent crates' integration suites.

Covers: shared state across connections; a move of an already-moved UID
returning OK with no COPYUID; UNCHANGEDSINCE reporting MODIFIED in the tagged
response; QRESYNC SELECT reporting VANISHED; QRESYNC refused without ENABLE and
on a CONDSTORE-only server; VANISHED versus EXPUNGE on delete.
 2026-08-10 13:57:12 -05:00
James Maes a997220a98 docs: add Nuncio Monitor implementation plan 
Twelve tasks across four phases: outbox account attribution, System.Shutdown,
System.GetHealth, then the monitor crate.
 2026-08-07 13:46:05 -05:00
James Maes 1e5f19c6ae docs: add Nuncio Monitor design spec 
In-repo dev instrument (crates/nuncio-monitor) plus the three engine
changes it needs: System.Shutdown, pending_remote_mutations.account_id,
and System.GetHealth (implements #296).
 2026-08-07 13:36:46 -05:00
James Maes d54720439e chore: drop dead npm dependabot path, ignore scratchpad 
The Tauri GUI npm tree moved to _reference/ and is out of the workspace,
so /crates/nuncio-gui/ui no longer exists for dependabot to update.
 2026-08-07 12:51:59 -05:00
James Maes 418aa7052e docs: reconcile roadmap and handoff with M6.5, WS and OBS 
Roadmap and its rendered page carried no mention of M6.5, the WS-A..WS-F
workstreams, or the completed OBS epic. AGENTS.md becomes a pointer to
CLAUDE.md instead of a near-verbatim copy.
 2026-08-07 12:51:12 -05:00
James Maes a79ec42e10 chore: back-merge main into dev (rustls 0.23.43) 2026-08-07 12:40:40 -05:00
james.maesandJames Maes 99351310ec feat(core): redaction policy, Redacted<T>, and no-secrets-in-logs canary (#378) 
Add a Redacted<T> newtype in nuncio-core whose Debug, Display, and serde
render a fixed placeholder, so a wrapped credential cannot leak through a
derived Debug or serde_json; the value is reachable only via an explicit
expose_secret(). Migrate the CalDAV/CardDAV account auth tokens to it and
drop their hand-written redacting Debug impls.

Document the logging redaction policy (what must never be logged vs what
is allowed) in docs/LOGGING.md and the redact module doc.

Add a cross-crate no-secrets canary that installs a capturing subscriber,
drives auth reject, account add, mock and JMAP-wiremock sync, mark-read,
send, a filter rule-fire, a webhook dispatch, and CalDAV/CardDAV sync with
sentinel secrets/PII, and asserts none appear in captured telemetry.

Refs #363

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 20:07:03 -05:00
james.maesandJames Maes ee63a62f17 feat(cli): wire verbosity flags and render typed daemon errors (#377) 
Wire -v/-vv/-vvv to a real stderr tracing subscriber (previously parsed
but never used), with NUNCIO_LOG/RUST_LOG override precedence matching
the daemon. Decode the WS-A1 ErrorInfo from a failed RPC's status
details and render the gRPC code plus the typed reason alongside the
message, in both text and --json output, instead of dropping the code
or dumping a raw Status. Document daemon log levels, the rotating log
file location, JSON log output, and request_id correlation in
docs/RUNNING.md.

Refs #365

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 19:41:49 -05:00
james.maesandJames Maes 58f321451d feat(proto): real GetStatus health/readiness surface (#376) 
Turn System.GetStatus into a live health/readiness surface backed by
real daemon state: process uptime, accounts_loaded, unread_count,
last_error, outbox_depth, per-account sync state (idle/syncing/error
with last-synced/last-error), and top-level ready/db_healthy.

- proto: extend GetStatusResponse; add AccountSyncState message + SyncState
  enum; regenerate the descriptor golden.
- store: add cheap ping (SELECT 1), unread-message count, and pending
  outbox-mutation count.
- sync dispatcher: retain each account's last completed outcome and expose
  a live per-account sync-status snapshot.
- daemon: populate every field from live state; DB read failures degrade
  honestly (reflected via db_healthy), never fabricated.
- cli: render the full health surface in text and JSON.

Refs #364

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 19:40:52 -05:00
james.maesandJames Maes a668c802e1 feat(mail): instrument protocol syncs and log destructive IMAP ops (#374) 
Wrap IMAP/JMAP folder and message sync in tracing spans keyed by account
and folder, logging completion counts (fetched/oversized/skipped) so a
sync run's outcome is visible without a debugger. SMTP send attempts and
their outcome are logged with the recipient COUNT only, never addresses.

Every destructive IMAP mutation (MOVE, DELETE/UID-scoped EXPUNGE) is
logged with its UID set and the UIDVALIDITY guard's pass/fail result
BEFORE the command reaches the server, so an audit trail exists even
when the guard then refuses to act. JMAP mirrors the same pre-logging
for its Email/set move/destroy, keyed by the JMAP object id instead of
a UID (JMAP has no UIDVALIDITY-equivalent to guard).

Refs #361

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 19:19:17 -05:00
james.maesandJames Maes ba2eb726d9 fix(cal): surface dropped VEVENTs; instrument DAV syncs + store lifecycle; redact DAV config Debug (#373) 
An unparseable VEVENT inside a CalDAV multistatus response was silently
discarded, so a partial sync failure looked identical to a calendar with
fewer events. It now logs a WARN with the VEVENT's UID (if extractable)
and a short parse-error reason, never the raw calendar body.

CalDAV and CardDAV round trips are wrapped in spans and log completion
counts (fetched/parsed/dropped) keyed by account/calendar id only.
Additive SQLite column migrations and first-use vault key minting now
log at INFO, naming the table/column or key purpose without ever
exposing key material.

CalDavAccountConfig and CardDavAccountConfig held an auth_token behind a
derived Debug, which would have printed the raw token verbatim from any
future debug! log or panic message. Both now use a manual Debug impl
that redacts the token.

Refs #362

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 19:17:45 -05:00
james.maesandJames Maes da072bf25d feat(nunciod): domain + lifecycle log coverage across handlers, syncs, outbox, shutdown (#372) 
Add meaningful domain events to the 8 gRPC services (account
add/update/remove/probe, mark-read, sync dispatch, contact create,
rule create/update/delete/import/export, triage, export, audit verify),
instrument the previously-silent send/calendar_sync/contacts_sync
routines, log outbox mutation attempt+result at the executor decision
level, and trace every graceful-shutdown drain step plus the
ShuttingDown event emission. Domain events inherit the OBS-2 per-RPC
span (request_id) rather than duplicating its entry/exit lines. Only
ids, counts, transport/op/rule kinds, folder names, and status are
logged -- never bodies, subjects, addresses, credentials, or tokens.

Refs #359

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 18:59:44 -05:00
james.maesandJames Maes 9f39fb095e feat(nunciod): per-RPC tracing span with request-id correlation and auth logging (#371) 
Add a tower Layer on the tonic server that wraps every RPC in a root span
carrying a generated request_id, the gRPC method, and the peer address. The
inner service (routing, the bearer-auth interceptor, and the handler) runs
inside that span, so all events emitted while handling a request inherit the
request_id. The span opens with a DEBUG event and closes, once the response
body drains, with an INFO event carrying the terminal gRPC status and elapsed
time.

Propagate the correlation span into the SyncDispatcher's detached sync tasks so
a sync triggered by Mail.Sync logs under the triggering request_id.

Log bearer-auth accept (DEBUG) and reject (WARN, with a fixed reason) in the
interceptor, carrying only the peer -- never the token, the authorization
header, or anything derived from them. The constant-time compare is unchanged.

Refs #358

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 18:39:01 -05:00
james.maesandJames Maes 2a385b984c fix(filter): surface rule-eval timeouts and instrument engine + webhook egress (#369) 
Replace the silent unwrap_or_default() on the ReDoS evaluation timeout
with an explicit, per-rule timeout that logs a warn! carrying the
rule's identity instead of quietly folding a timed-out evaluation into
"no match". The per-rule evaluation now runs on a blocking-pool task so
the timeout can actually preempt a slow synchronous evaluation, rather
than racing a timer a non-yielding future would never let fire; a
timed-out rule is documented and logged as non-matching, and the rest
of the batch still runs.

Add DEBUG rule-fire logging (rule id/name + action kinds, never the
destination data an action carries) and INFO on reload_rules. Wrap
webhook dispatch in a tracing span logging the target host, and log
every SSRF/CIDR/DNS-rebind block decision as warn! with host/blocked-ip
and reason; allowed egress logs at debug. No message bodies, secrets,
or full URLs are logged.

Refs #360

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 18:34:51 -05:00
james.maesandJames Maes eb117d2f47 refactor(proto): model account transport as a oneof (#367) 
Make an invalid account transport mix unrepresentable: an account now
carries exactly one transport instead of a flat bag of optional
imap/jmap/dav fields.

- proto: AccountConfig gains `oneof transport { ImapSmtpTransport |
  JmapTransport | DavTransport }`; the flat fields (protocol,
  server_host/port, use_tls, imap/smtp tls modes, smtp_host/port,
  collection_url) are removed and their field numbers reserved. Golden
  descriptor regenerated; contract_stability passes.
- core: AccountConfig carries a `Transport` enum; validate() checks the
  active variant and preserves the cleartext-to-remote-host refusal per
  transport.
- store: physical accounts columns stay flat; the existing `protocol`
  column is the transport-kind discriminator. Each variant round-trips.
- daemon: grpc maps core Transport <-> proto oneof both ways; an unset
  transport returns typed VALIDATION_FAILED, never a panic.
- cli: `account add --protocol` gates transport flags; list/show render
  the active transport; --json stays coherent.

Refs #278

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 18:13:13 -05:00
james.maesandJames Maes a261f8e61a feat(nunciod): layered tracing subscriber with EnvFilter, rotating file, and JSON (#368) 
Replace the stderr-only tracing_subscriber::fmt::init() bootstrap with a
layered Registry subscriber: a reloadable EnvFilter (NUNCIO_LOG > RUST_LOG >
info) exposed via a reload::Handle for future runtime level control, a
human-readable stderr console layer, and a non-blocking rotating daily file
sink under <data_dir>/logs/. NUNCIO_LOG_FORMAT=json switches the file sink to
JSON. Malformed filters and unwritable log dirs degrade gracefully instead of
crashing the daemon; the WorkerGuard is returned and bound in main for the
process lifetime.

Refs #357

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 18:12:16 -05:00
james.maesandJames Maes 9b69b7f1b1 feat(nunciod): bounded-concurrency sync dispatcher with per-account timeout (#366) 
Route every production account sync through one SyncDispatcher that caps
simultaneous syncs (NUNCIO_SYNC_CONCURRENCY, default 4), coalesces a
duplicate in-flight sync for the same account, applies a per-account
timeout (NUNCIO_SYNC_TIMEOUT, default 300s), and races in-flight syncs
against the shared shutdown signal. The client Mail.Sync RPC and the
daemon's CoreCommand sync consumer now share a single dispatcher instance
so one slow account can no longer head-of-line-block the others.

Refs #292

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 17:58:29 -05:00
james.maesandJames Maes db60f30018 feat(proto): enum hygiene for rule export format and account TLS (#356) 
Introduce a dedicated RuleExportFormat enum (SQL/JSON) for ExportRules,
replacing the stringly "sql"/"json" format field; UNSPECIFIED is rejected
with invalid_argument rather than silently defaulting. Reuse of the
mailbox Export service's ExportFormat enum was rejected since its members
(MBOX/EML_ZIP/JSON/JSONL) describe portable mailbox output, not rule
rendering, and the two have no overlapping meaning.

Drop AccountConfig.use_tls from the wire contract (reserving field 7) and
from nuncio_core::AccountConfig: it duplicated imap_tls_mode/smtp_tls_mode
and was never read for behavior anywhere in the daemon, CLI, or store --
only ever written and then ignored. The SQLite accounts.use_tls column is
kept (with a default) for on-disk compatibility with existing rows; no
code reads it anymore.

Regenerate the nuncio.v1 descriptor golden.

Refs #277

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 17:28:22 -05:00
james.maesandJames Maes 9bfffe59da feat(nunciod): bound gRPC server concurrency/streams/timeout/message size (#355) 
Add sensible resource bounds to the tonic Server builder so a
misbehaving or malicious client cannot exhaust the daemon: a
per-connection concurrency limit, a per-call timeout generous enough
for the longest legitimate unary sync, an HTTP/2 concurrent-stream
cap, a per-service inbound message-size limit raised above tonic's
default to fit real mail attachments, and TCP/HTTP2 keepalive.

Refs #304

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 17:27:02 -05:00
james.maesandJames Maes d1a6f8f417 feat(store): implement contact search over contacts_fts (#354) 
Add search_contacts/search_contacts_page to SearchEngine, mirroring the
existing message and event search queries and their keyset pagination.
The contacts_fts virtual table and its insert/update/delete triggers
already existed but were unused; new tests prove the triggers keep the
index in sync across insert, update, and delete, that ranked hits
correctly include/exclude matches, and that sanitized/edge queries do
not error.

Refs #318

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 17:05:21 -05:00
james.maesandJames Maes f85893e128 feat(proto): enforce id-naming law and return full config from AddAccount (#353) 
Audit every message across nuncio.proto for the id-naming convention
(an entity's own identifier is `id`; a reference to another entity is
`<noun>_id`) and fix the outliers: RemoveAccountRequest, TestAccountConnectionRequest,
DeleteRuleRequest, and UpdateRuleRequest carried a bare `id` referencing an
account or rule rather than naming it; MessageSearchHit did the same for
the message it matched. AddAccountResponse now returns the persisted
AccountConfig (never the password), matching how CreateRule/CreateContact
return their created object instead of a bare id.

Document Message.id (and the other entity ids) as an opaque,
server-assigned token clients must never parse or construct, reflecting
the surrogate id introduced for messages.

Regenerates the nuncio.v1 descriptor golden for the renamed/changed fields.

Refs #276

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 17:04:11 -05:00
james.maesandJames Maes 0567ddea1d feat(mail): bound IMAP fetch by batch count and per-message size (#352) 
Enumerate a folder's UIDs and sizes cheaply (no bodies), then pull full
bodies in explicit UID-set batches of at most MAX_MESSAGES_PER_FETCH_BATCH
so the transient working set stays bounded regardless of mailbox size.
Messages whose RFC822.SIZE exceeds MAX_MESSAGE_BODY_BYTES are fetched
headers-only and recorded with an empty body plus a WARN naming the uid
and size, rather than buffered whole or silently dropped.

Refs #301

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 16:54:37 -05:00
james.maesandJames Maes 834f6a8ddc feat(proto): keyset pagination for all list/search RPCs (#350) 
Apply one AIP-158 keyset pagination convention (page_size/page_token/
next_page_token, opaque cursor, not offset) to every list/search RPC:
ListMessages, ListFolders, ListRules, ListEvents, ListContacts,
SearchMessages, and reconcile Audit.ListRecords off its unstable
limit/offset. A malformed page_token returns a typed VALIDATION_FAILED
ErrorInfo (InvalidArgument). Regenerate the nuncio.v1 golden descriptor.

Refs #275

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 16:42:25 -05:00
james.maesandJames Maes 3687bc3f08 feat(nunciod): bound outbox per-item execution with a timeout (#349) 
Wrap each outbox mutation's execution in a per-item timeout raced against
the shutdown signal, so one hung remote op (a stuck IMAP/JMAP call or
webhook) can no longer freeze delivery for every other pending mutation
across every account. A timed-out item is dispositioned exactly like any
other transient failure: retryable, bumping retry_count, never completed.

Refs #290

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 16:31:35 -05:00
james.maesandJames Maes 64cff6d8f6 feat(proto): unify time fields on Timestamp/Duration (#348) 
Message.received_at, CalendarEvent.start_time/end_time,
CalendarSyncRequest/ListEventsRequest.start_window/end_window,
FilterRule.created_at/updated_at, FilterExecutionLog.matched_at,
AccountConfig.sync_interval_secs, PreviewRuleResponse.execution_time_us,
and AuditRecord.timestamp_ns all mixed unix-seconds and nanosecond int64
scalars, leaving the unit ambiguous at the wire level. Move every wall-clock
field to google.protobuf.Timestamp and every duration to
google.protobuf.Duration, add prost-types as the standard well-known-types
crate, and add nuncio_proto::time as the single conversion point between
the store's native integer instants and the wire types (including a
lossless unix-nanos split/join for the WORM audit ledger's sub-second
precision). Update the daemon's gRPC mappers and the CLI's request
construction/rendering accordingly, and regenerate the descriptor golden.

Refs #274

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 16:00:48 -05:00
james.maesandJames Maes 885c13e4cc feat(nunciod): single-instance advisory lock on the DB path (#346) 
Acquire an exclusive OS advisory lock on a sidecar file beside the
database path (<db_path>.lock) before opening the store, so a second
nunciod pointed at the same database fails fast with an actionable
error instead of both instances writing to the same SQLite file.
Previously only an incidental gRPC port-bind collision offered any
protection against this.

Uses fs4 (flock on Unix, LockFileEx on Windows) since std has no
portable file-lock primitive. The guard is held for the process
lifetime and dropped at clean shutdown, releasing the lock; the OS
also releases it automatically if the process dies uncleanly, so
there is no PID-file staleness to reason about.

Refs #291

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 15:55:13 -05:00
james.maesandJames Maes c660ed6d06 feat(cal): add CalendarEvent::to_ical serializer (#345) 
Adds the CalDAV write-back inverse of IcalParserAdapter::parse_ical: a
ToIcal extension trait wraps a single VEVENT (UID, DTSTAMP, DTSTART,
DTEND, SUMMARY, LOCATION, RRULE) in a VCALENDAR shell, escaping TEXT
values per RFC 5545 3.3.11.

Refs #310

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 15:52:07 -05:00
james.maesandJames Maes 49119a4ce1 refactor(core): remove dead EventBus ReloadFilters/unread_count arms (#344) 
CoreCommand::ReloadFilters was a silent no-op: nothing in the workspace
sends it, since filter reload actually happens via
FiltersGrpcService::reload_engine_from_store. AppState::unread_count was
written by MarkRead but never read by any consumer -- GetStatusResponse
only carries engine_status and version, so it never reached the wire.

Refs #324

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 15:38:59 -05:00
james.maesandJames Maes aa5c1c35e9 fix(filter): pin webhook dispatch to the checked IP to close DNS-rebind TOCTOU (#343) 
Refs #307

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 14:15:12 -05:00
james.maesandJames Maes 01e7acd97d feat(proto): typed errors (ErrorInfo/ErrorReason) via Status details (#342) 
* feat(proto): add typed error taxonomy and Status helper

Introduce a frozen, machine-readable error taxonomy for the nuncio.v1
contract: an ErrorReason enum and an ErrorInfo message carrying a reason,
optional metadata, and a human message. A nuncio-proto errors helper builds
a tonic::Status that encodes ErrorInfo into the grpc-status-details-bin
trailer with the canonical gRPC code per reason, and decodes the reason back
on the client side. Regenerate the committed descriptor golden.

Refs #273

* feat(nunciod): return typed ErrorInfo details from error sites

Convert a representative set of daemon gRPC error returns to carry a typed
ErrorInfo in the status-details trailer via the nuncio-proto errors helper:
account/message/event/contact/rule not-found (with the offending id as
metadata), NSQL validation failures, and the calendar not-configured
precondition. Canonical status codes are unchanged, so existing error-path
assertions still hold. Add an offline over-the-wire E2E test that decodes the
reason and metadata from a returned Status against the real daemon server.

Refs #273

---------

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 14:12:44 -05:00
james.maesandJames Maes dfc7cc9358 perf(sync): batch new-message detection to remove the is_new_message N+1 (#341) 
Replace the per-message get_message existence check in fetch_and_persist
with a single existing_message_ids(chunk_ids) lookup per fetched folder
chunk, plus a per-pass seen-set to keep duplicate ids within one chunk
firing at most once. Fire-once semantics for filter actions are
unchanged: a message is new iff it was absent from the store before this
sync pass and this is its first occurrence in the current chunk.

Refs #325

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 14:11:05 -05:00
james.maesandJames Maes f44412f1ab fix(store): deterministic message surrogate identity (C1) (#337) 
* fix(store): give messages a deterministic surrogate identity

IMAP UIDs are unique only per mailbox+UIDVALIDITY, yet messages.id was the
sole key and Email.id was "imap-uid-{uid}", so the same UID in two folders or
accounts collapsed to one row and silently overwrote (audit C1). The id shape
also leaked internal encoding on the wire.

Messages now carry an opaque surrogate id = SHA-256 over length-prefixed
(account, folder, uid_validity, remote_id), plus first-class remote_id and
uid_validity columns and a UNIQUE(account, folder, uid_validity, remote_id)
index (backfill-safe migration seeding legacy rows from the old id). The IMAP
and JMAP backends populate remote_id/uid_validity and hash the id; the outbox
executor recovers remote addressing from the columns instead of parsing the
id, still SELECT-verifying UIDVALIDITY before mutating. Re-sync recomputes the
same id, so it upserts one row and the fire-once filter guard still holds.

Message.id stays a proto string; only its value becomes opaque (no proto
change).

Refs #281

* fix(store): make message-identity migration atomic and self-healing

The identity migration ran ADD COLUMN, backfill, and CREATE UNIQUE INDEX as
separate autocommit statements with the backfill gated on having just added
the column. A crash between the ADD and the backfill could leave legacy rows
at remote_id='' with the column present, so the next startup skipped the
backfill and the index creation then collided on
(account, folder, uid_validity='', remote_id='') -- wedging the daemon.

Run the whole step in one transaction (a crash rolls back atomically) and make
the backfill unconditional (idempotent UPDATE ... WHERE remote_id = '') so any
committed partial state is completed before the index is built. No-op on a
fresh or already-migrated database.

Refs #281

---------

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 13:57:22 -05:00
james.maesandJames Maes 6d62c09fcd feat(nunciod): graceful shutdown with WAL-checkpointing close (#340) 
Install a Ctrl+C/SIGTERM signal handler and share one shutdown signal
across the real-sync command consumer, the outbox worker, the
update-check loop, and the gRPC server, so a clean exit drains every
task and forces the WAL checkpoint via DatabaseEngine::close() instead
of the OS killing the process mid-write. The signal also finally
triggers the existing CoreCommand::Shutdown / CoreEvent::ShuttingDown
event-bus path outside of tests.

Refs #289

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 13:55:17 -05:00
james.maesandJames Maes 546cebe723 fix(core): reject cleartext Plain TLS to non-loopback hosts (#335) 
* fix(core): reject cleartext Plain TLS to non-loopback hosts

AccountConfig::validate silently accepted TlsMode::Plain for imap/smtp
against any host, so a misconfig could send credentials in cleartext
to a remote server. Plain is now rejected unless the host is loopback
(127.0.0.1, ::1, or localhost); ImplicitTls/StartTls are unaffected.

Refs #297

* fix(core): reject cleartext CalDAV collection_url on remote hosts

The CalDAV collection_url validation only checked the scheme prefix,
so http://<remote-host>/... passed even though it sends the account's
Basic-auth credentials in cleartext. Reuse the same loopback-only rule
as the imap/smtp TlsMode::Plain check: plain http is now accepted only
when the URL's host is loopback (127.0.0.0/8, ::1, or localhost);
https to any host, and http to a loopback host, remain valid.

Refs #297

---------

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 13:35:15 -05:00
james.maesandJames Maes 84c1a6b8fe fix(cli): render clean error messages instead of raw tonic::Status (#336) 
Replace ~28 CLI error paths that formatted a tonic::Status directly
(dumping its Debug output, including internal metadata) with the
server-provided status.message(), falling back to a short "<code>
error" form when the message is empty.

Refs #321

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 13:30:13 -05:00
james.maesandJames Maes e3d3413736 feat: production-wire Calendar sync via first-class CalDAV accounts (#272) 
* feat(proto): add CALDAV account protocol and collection_url

Add ACCOUNT_PROTOCOL_CALDAV = 3 and a generic AccountConfig.collection_url
(field 14) so a CalDAV account can be a first-class account entry. The
field is deliberately generic (DAV collection URL) for CardDAV reuse.
Regenerated the committed descriptor golden; additive within nuncio.v1.

Refs #254 #178

* feat(core): model CalDAV accounts with a generic collection URL

Add AccountProtocol::CalDav and AccountConfig.collection_url, plus an
is_dav() helper. Validation is now protocol-aware: a DAV account requires
a non-empty http(s) collection_url and skips the IMAP/SMTP endpoint
checks, while mail accounts are unchanged.

Refs #254 #178

* feat(store): persist account collection_url with backfill-safe migration

Add a nullable collection_url column to the accounts table via an
ensure_accounts_dav_columns migration that mirrors the SMTP/TLS-mode
column backfills, and round-trip it through save_account/list_accounts/
get_account. Salvage preserves collection_url alongside the TLS modes.

Refs #254 #178

* feat(daemon): wire the production CalDAV sync path

Calendar/Sync with no injected backend now resolves configured CalDAV
accounts (filtered by protocol, honoring a targeted account_id), reads
each credential from the keyring vault, builds a real CalDavClient from
the persisted collection_url, and syncs events via
calendar_sync::sync_caldav_account. The honest error now fires only when
no CalDAV account is configured. AddAccount validates CalDAV accounts
(http(s) collection_url + password) via the protocol-aware core
validation. The mail sync/outbox paths skip DAV accounts, which have no
mail backend. The CalendarEngineOverrides seam is retained for tests.

Refs #254 #178

* feat(cli): create and list CalDAV accounts

account add gains --protocol (imap-smtp/jmap/caldav) and --collection-url;
a caldav account is rejected before dialing the daemon if the collection
URL is missing, and prompts for a password like a mail account. account
list renders CalDAV accounts with their collection URL.

Refs #254 #178

* test(daemon): prove the production CalDAV sync path offline

Add a full-daemon E2E that configures a CalDAV account over the real
Accounts API and drives Calendar/Sync with NO injected backend, so the
daemon builds a real CalDavClient against a WireMock CalDAV server and
persists the fetched events, immediately visible via ListEvents. Also
asserts the honest error for an unconfigured account and that Calendar
stays behind its BearerAuthInterceptor. Fully offline: WireMock + mock
keyring, no live network.

Refs #254 #178

* test(daemon): add collection_url to account test fixtures

Fill in the new AccountConfig.collection_url field in the orchestrator
and send test fixtures so they construct the mail account model.

Refs #254 #178

---------

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 12:41:24 -05:00
james.maesandJames Maes 167955288b fix(store): preserve audit tables and real TLS modes during salvage (#270) 
Salvage previously dropped worm_audit_records and filter_execution_logs
entirely, losing tamper-evident audit/ledger history on recovery, and
hardcoded every salvaged account's imap_tls_mode/smtp_tls_mode back to
ImplicitTls regardless of the account's real configured mode. Salvage
now carries both ledger tables through recovery verbatim, including
their hash-chain columns, and reads each account's real TLS modes from
the salvaged row, WARN-logging and falling back only when a raw value
is genuinely unrecognized.

Refs #248

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 12:01:40 -05:00
james.maesandJames Maes 335c543683 fix(store): surface unparseable filter rules instead of dropping them (#269) 
list_filter_rules silently dropped stored rules that failed to re-parse,
making a rule disappear from enforcement with no trace. Now each
unparseable row is WARN-logged with its rule id and skipped, so the
failure is visible instead of indistinguishable from "never uploaded".

Also stop nunciod's boot-time filter load from collapsing a genuine
database error into an empty rule set: load_initial_filter_rules
propagates the error (ERROR-logged) instead of unwrap_or_default(),
so a DB failure fails startup visibly rather than quietly disabling
all filtering.

Refs #247

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 11:51:21 -05:00
james.maesandJames Maes 545f06fa7b fix(store): fail export when its WORM audit write fails (#268) 
Export previously discarded the WORM audit write's result with `let _
= ...`, so a failed audit record still produced a successful export --
an unaudited export was indistinguishable from an audited one,
defeating the tamper-evidence the WORM log exists to provide.

Refs #246

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 11:40:15 -05:00
james.maesandJames Maes e1c66da4df fix(filter): use unique rule ids to prevent silent overwrite (#267) 
CreateRule derived a rule's id deterministically from
sha256(name + nsql), so two rules with identical name and NSQL text
produced the same id. save_filter_rule persists via INSERT OR REPLACE
with no existence check, so the second create silently overwrote the
first rule with no warning.

Generate each rule's id from a CSPRNG (the same OsRng nuncio-store
already uses to source key material) instead of deriving it from the
rule's content, so identical name+NSQL always yields distinct ids.
UpdateRule and list_filter_rules already reassign the id from the
persisted row after parsing, so they are unaffected.

Refs #249

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 11:32:10 -05:00
james.maesandJames Maes e75f469f6b fix(store): make WORM/ledger appends atomic (#266) 
save_filter_execution_log and append_worm_audit_record read the latest
chain hash and then insert as two separate statements with no lock,
letting concurrent appends read the same prev-hash and fork the
tamper-evident hash chain. Wrap each read+insert in a BEGIN IMMEDIATE
transaction so SQLite serializes the chain read against the dependent
insert.

Refs #245 #164

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 11:22:57 -05:00
james.maesandJames Maes dfaec91bd4 fix(store): fail closed on decrypt failure in decrypt_text_at_rest (#264) 
decrypt_text_at_rest previously swallowed any hex-decode, AEAD-decrypt,
or UTF-8 failure with a fallback to an empty string, so a corrupted or
tampered body column read back as an empty body instead of surfacing
an error -- defeating the AEAD's integrity guarantee exactly where it
matters: detecting tampering on read. Change its signature to return
Result<String, CipherError> and propagate failures through every read
call site in db.rs as the new DatabaseError::Decryption variant.

Refs #237 #162

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 11:07:10 -05:00
james.maesandJames Maes 025e992c11 feat: build the outbox remote-mutation executor (filters made live) (#263) 
* feat(mail): add remote-mutation ops to MailBackend (IMAP/JMAP)

Add apply_mutation to the MailBackend trait plus RemoteMutationKind/
RemoteMutationSpec covering move/copy/flag/delete. IMAP addresses by UID
parsed from imap-uid-{uid}, guards on the folder checkpoint's UIDVALIDITY
before acting, and uses UID MOVE with a UID COPY+STORE+EXPUNGE fallback.
JMAP uses Email/set (keywords/mailboxIds patch, destroy) and confirms the
id was genuinely updated/destroyed. MockMailBackend records applied
mutations. Scripted-duplex (IMAP) and wiremock (JMAP) tests, offline only.

Refs #252

* feat(nunciod): execute outbox remote mutations for real

Replace the no-op outbox worker with a real executor that drains each
pending remote mutation against the resolved account's backend: mailbox
mutations (move/copy/flag/unflag/delete) via MailBackend::apply_mutation,
FORWARD via the SMTP MessageSender, and CALL WEBHOOK via the SSRF-hardened
dispatcher (signing key provisioned from the vault). A mutation is marked
completed ONLY on genuine success; transient failures retry up to
MAX_RETRIES then fail, permanent failures (gone message, unknown action)
fail immediately. Offline E2E proves move/flag/delete/forward/webhook end
to end and that a failed op is never marked completed.

Refs #252

* fix(mail): scope EXPUNGE to target UID; fail closed without UIDPLUS

A bare EXPUNGE removes every \Deleted message in the mailbox, so the
DELETE path and the no-MOVE-capability move fallback could collaterally
destroy other messages a different client had flagged \Deleted. Replace
the mailbox-wide expunge with RFC 4315 UID EXPUNGE scoped to the target
UID, gated on the UIDPLUS capability; without UIDPLUS the op fails closed
with an honest error rather than falling back to a destructive bare
EXPUNGE. Tests now assert the exact scoped 'UID EXPUNGE <uid>' command
(a bare EXPUNGE is rejected) and cover the non-UIDPLUS fail-closed case.

Also provision the webhook signing key lazily so a vault issue with that
key can no longer disable the mailbox-mutation and forward paths.

Refs #252

---------

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 10:55:21 -05:00
james.maesandJames Maes aabda08069 refactor(mail): remove fake/dead protocol code paths (#262) 
Delete crates/nuncio-cal/src/carddav.rs, an orphaned duplicate
CardDavClient/Contact that made no network call and was unused outside
its own tests -- the real CardDAV client lives in nuncio-contacts.

Make credential-less ImapEngine/JmapEngine sync_folders/sync_messages
return an honest MailError::AuthError instead of returning hardcoded
folders/messages. This also fixes a latent bug in ImapEngine::sync_folders:
a genuinely empty real mailbox list previously fell through to fake
canned folders instead of being returned as-is.

Remove send_email from the MailBackend trait: production sends route
through MessageSender/SmtpTransportEngine exclusively, and the IMAP/JMAP
implementations were both no-op fabricated successes with no real
transport behind them.

Migrate the tests that exercised the removed fixture paths to real
fixtures: a scripted loopback IMAP server drives ImapEngine::sync_folders
through a real LOGIN/LIST/LOGOUT exchange, and credential-less calls are
now asserted to fail honestly rather than fabricate data.

Refs #253

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 10:05:49 -05:00
james.maesandJames Maes 17774acb16 refactor(core): remove fabricated e2ee/ai/plugin modules (#261) 
An audit found e2ee.rs, ai.rs, and plugin.rs implemented no real
functionality: signature verification did no cryptography, the AI
engine returned canned strings with no model call, and the plugin
runtime executed no code. These modules were pub-exported but wired
nowhere in the workspace, and their tests asserted on the fabricated
output rather than real behavior. E2EE, local AI, and plugins are
deferred features; deferred features must be absent, not fake-present.

A workspace-wide grep confirmed no code outside these three files and
their lib.rs declarations referenced any of their exported symbols.

Refs #244 #79 #80 #81

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-31 09:52:51 -05:00
james.maesandJames Maes fc5d6b337e fix(filter): enforce real IP/CIDR SSRF checks on webhook targets (#243) 
Replace the substring blocklist in pass 6 with parsed IP-literal range
checks and add a dispatch-time DNS-resolution check so hostnames that
resolve into private/metadata ranges are rejected before connecting.

Refs #203

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-30 13:10:06 -05:00
james.maesandJames Maes 3c11760a83 fix(update): fail closed on missing or unfetchable SHA256SUMS (#242) 
download_and_verify previously returned the downloaded archive unverified
when a release had no SHA256SUMS.txt asset, or when fetching it returned a
non-2xx status. Restructure so the only path to Ok is an exact SHA-256
match: absent asset, non-success fetch, filename-not-found, and hash
mismatch all return Err. Adds a distinct ChecksumUnavailable variant and
wiremock coverage for each failure mode plus the matching happy path.

Refs #202

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-30 12:55:59 -05:00
james.maesandJames Maes ec2b7b3ea6 feat(security): reject non-loopback gRPC bind addresses (#240) 
serve() now parses the requested address and rejects it with a new
GrpcServeError::NonLoopbackAddress before ever calling TcpListener::bind,
so a misconfigured NUNCIO_GRPC_ADDR can't expose the bearer-token-only
API beyond the local host. Also adds one consolidated canary test that
dials all eight mounted services with no token and asserts
Unauthenticated, giving reviewers a single checkpoint to update whenever
a service is added to serve_on_listener_with_overrides.

Refs #201

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-30 11:25:32 -05:00
james.maesandJames Maes 83d8b99c86 fix(store): validate WORM/ledger key length from the vault (#239) 
Mirror the existing storage-key length check for the WORM audit and
filter-ledger HMAC keys: resolve_engine_keys now converts each key's
Vec<u8> into a Zeroizing<[u8; 32]>, failing closed with a distinct
DatabaseError::KeyProvisioning message when a vault entry decodes to
the wrong length. Previously only the storage key was validated, so a
corrupted or hand-edited vault entry of any length would be silently
accepted as an HMAC signing key.

Refs #200

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-29 22:19:14 -05:00
james.maesandJames Maes 03c02ae9df feat(security): zeroize long-lived engine key material on drop (#238) 
Wrap DatabaseEngine's storage/WORM/ledger keys, WebhookDispatcher's
HMAC secret, and add_account's transient password in
zeroize::Zeroizing so their backing memory is overwritten once the
holding struct drops, instead of lingering as plain heap bytes for
the rest of the process lifetime.

Refs #199, #163

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-29 22:10:01 -05:00
james.maesandJames Maes 74fa4fc7b8 fix(store): fail closed on encryption failure in save_email (#236) 
encrypt_text_at_rest previously swallowed any AEAD failure with
unwrap_or_default(), silently persisting an empty ciphertext string
that is indistinguishable from a genuinely empty body on read-back.
Change its signature to return Result<String, CipherError> and
propagate failures through save_email as DatabaseError::Encryption,
resolved before the transaction begins so no row is ever written for
a message whose body failed to encrypt.

Refs #198, #162

Co-authored-by: James Maes <jmaes@dmdbrands.com>
 2026-07-29 22:00:52 -05:00
119 changed files with 38781 additions and 4229 deletions
Expand all files Collapse all files

No files matched your search

.github/dependabot.yml
+4 -13
View File
@@ -13,20 +13,11 @@ updates:
commit-message:
prefix: "deps(cargo)"
# 2. Tauri v2 Desktop GUI UI (npm)
- package-ecosystem: "npm"
directory: "/crates/nuncio-gui/ui"
schedule:
interval: "daily"
time: "04:00"
open-pull-requests-limit: 10
labels:
- "dependencies"
- "gui"
commit-message:
prefix: "deps(npm)"
# The Tauri GUI's npm tree moved out of the workspace to `_reference/` and is
# being rebuilt as a separate client repo, so there is no npm ecosystem to
# update here. Re-add an npm entry in the client repo, not this one.
# 3. GitHub Actions Workflows
# 2. GitHub Actions Workflows
- package-ecosystem: "github-actions"
directory: "/"
schedule:
.gitignore
+1
View File
@@ -36,3 +36,4 @@ _reference/**/gen/
**/node_modules/
**/dist/
trivy_err*.log
scratchpad/
AGENTS.md
+11
View File
@@ -0,0 +1,11 @@
# AGENTS.md
The guidance for this repository lives in [`CLAUDE.md`](CLAUDE.md). It is
tool-agnostic despite the filename: what it documents is the repo's
architecture, build and quality gates, conventions, and testing standards.
**Read [`CLAUDE.md`](CLAUDE.md) before working here**, whichever agent you are.
This file is deliberately a pointer, not a copy. It previously held a
near-verbatim duplicate of `CLAUDE.md` differing only in the title line, which
would have drifted out of sync the first time either file was edited.
Cargo.lock
Generated
+151 -2
View File
@@ -886,6 +886,15 @@ dependencies = [
"cfg-if",
]
[[package]]
name = "crossbeam-channel"
version = "0.5.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d85363c37faeca707aef026efa9f3b34d077bce547e48f770770625c6013679e"
dependencies = [
"crossbeam-utils",
]
[[package]]
name = "crossbeam-queue"
version = "0.3.13"
@@ -998,6 +1007,12 @@ dependencies = [
"zeroize",
]
[[package]]
name = "deranged"
version = "0.5.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c"
[[package]]
name = "derivative"
version = "2.2.0"
@@ -1314,6 +1329,16 @@ dependencies = [
"percent-encoding",
]
[[package]]
name = "fs4"
version = "0.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8640e34b88f7652208ce9e88b1a37a2ae95227d84abec377ccd3c5cfeb141ed4"
dependencies = [
"rustix 1.1.4",
"windows-sys 0.59.0",
]
[[package]]
name = "fs_extra"
version = "1.3.0"
@@ -2213,6 +2238,15 @@ dependencies = [
"encoding_rs",
]
[[package]]
name = "matchers"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d1525a2a28c7f4fa0fc98bb91ae755d1e2d1505079e05539e35bc876b5d65ae9"
dependencies = [
"regex-automata",
]
[[package]]
name = "matchit"
version = "0.8.4"
@@ -2400,6 +2434,12 @@ dependencies = [
"num-traits",
]
[[package]]
name = "num-conv"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441"
[[package]]
name = "num-integer"
version = "0.1.46"
@@ -2459,7 +2499,6 @@ dependencies = [
"chrono-tz 0.10.4",
"icalendar",
"nuncio-core",
"quick-xml",
"reqwest",
"rrule",
"serde",
@@ -2467,6 +2506,7 @@ dependencies = [
"thiserror 1.0.69",
"tokio",
"tracing",
"tracing-subscriber",
"wiremock",
]
@@ -2493,6 +2533,7 @@ dependencies = [
"tokio-stream",
"tonic",
"tracing",
"tracing-subscriber",
]
[[package]]
@@ -2502,7 +2543,6 @@ dependencies = [
"async-trait",
"chrono",
"nuncio-core",
"quick-xml",
"regex",
"reqwest",
"serde",
@@ -2512,6 +2552,7 @@ dependencies = [
"thiserror 1.0.69",
"tokio",
"tracing",
"tracing-subscriber",
"wiremock",
]
@@ -2526,6 +2567,7 @@ dependencies = [
"futures",
"hex",
"hmac",
"quick-xml",
"regex",
"reqwest",
"serde",
@@ -2544,6 +2586,7 @@ dependencies = [
name = "nuncio-filter"
version = "0.1.0"
dependencies = [
"aes-gcm",
"arc-swap",
"async-trait",
"chrono",
@@ -2561,6 +2604,7 @@ dependencies = [
"tokio",
"tracing",
"wiremock",
"zeroize",
]
[[package]]
@@ -2573,6 +2617,7 @@ dependencies = [
"lettre",
"mail-parser",
"nuncio-core",
"nuncio-mail",
"reqwest",
"rustls",
"rustls-pki-types",
@@ -2584,6 +2629,7 @@ dependencies = [
"tokio-rustls",
"tokio-stream",
"tracing",
"tracing-test",
"webpki-roots",
"wiremock",
]
@@ -2592,7 +2638,9 @@ dependencies = [
name = "nuncio-proto"
version = "0.1.0"
dependencies = [
"bytes",
"prost",
"prost-types",
"protoc-bin-vendored",
"thiserror 1.0.69",
"tokio",
@@ -2622,6 +2670,7 @@ dependencies = [
"thiserror 1.0.69",
"tokio",
"tracing",
"tracing-subscriber",
"zeroize",
]
@@ -2632,7 +2681,11 @@ dependencies = [
"arc-swap",
"async-trait",
"bytes",
"fs4",
"hex",
"http",
"http-body",
"http-body-util",
"nuncio-cal",
"nuncio-cli",
"nuncio-contacts",
@@ -2650,9 +2703,12 @@ dependencies = [
"tokio",
"tokio-stream",
"tonic",
"tower",
"tracing",
"tracing-appender",
"tracing-subscriber",
"wiremock",
"zeroize",
]
[[package]]
@@ -2961,6 +3017,12 @@ dependencies = [
"zerovec",
]
[[package]]
name = "powerfmt"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "439ee305def115ba05938db6eb1644ff94165c5ab5e9420d1c1bcedbba909391"
[[package]]
name = "ppv-lite86"
version = "0.2.21"
@@ -4203,6 +4265,12 @@ version = "2.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292"
[[package]]
name = "symlink"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a7973cce6668464ea31f176d85b13c7ab3bba2cb3b77a2ed26abd7801688010a"
[[package]]
name = "syn"
version = "1.0.109"
@@ -4329,6 +4397,36 @@ dependencies = [
"cfg-if",
]
[[package]]
name = "time"
version = "0.3.54"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3e1d5e639ff6bab73cb6885cc7e7b1de96c3f32c68ec55f3952614bec1092244"
dependencies = [
"deranged",
"num-conv",
"powerfmt",
"serde_core",
"time-core",
"time-macros",
]
[[package]]
name = "time-core"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109"
[[package]]
name = "time-macros"
version = "0.2.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7e689342a48d2ea927c87ea50cabf8594854bf940e9310208848d680d668ed85"
dependencies = [
"num-conv",
"time-core",
]
[[package]]
name = "tinystr"
version = "0.8.3"
@@ -4574,6 +4672,19 @@ dependencies = [
"tracing-core",
]
[[package]]
name = "tracing-appender"
version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "050686193eb999b4bb3bc2acfa891a13da00f79734704c4b8b4ef1a10b368a3c"
dependencies = [
"crossbeam-channel",
"symlink",
"thiserror 2.0.19",
"time",
"tracing-subscriber",
]
[[package]]
name = "tracing-attributes"
version = "0.1.31"
@@ -4606,18 +4717,56 @@ dependencies = [
"tracing-core",
]
[[package]]
name = "tracing-serde"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "704b1aeb7be0d0a84fc9828cae51dab5970fee5088f83d1dd7ee6f6246fc6ff1"
dependencies = [
"serde",
"tracing-core",
]
[[package]]
name = "tracing-subscriber"
version = "0.3.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cb7f578e5945fb242538965c2d0b04418d38ec25c79d160cd279bf0731c8d319"
dependencies = [
"matchers",
"nu-ansi-term",
"once_cell",
"regex-automata",
"serde",
"serde_json",
"sharded-slab",
"smallvec",
"thread_local",
"tracing",
"tracing-core",
"tracing-log",
"tracing-serde",
]
[[package]]
name = "tracing-test"
version = "0.2.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "19a4c448db514d4f24c5ddb9f73f2ee71bfb24c526cf0c570ba142d1119e0051"
dependencies = [
"tracing-core",
"tracing-subscriber",
"tracing-test-macro",
]
[[package]]
name = "tracing-test-macro"
version = "0.2.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ad06847b7afb65c7866a36664b75c40b895e318cea4f71299f013fb22965329d"
dependencies = [
"quote",
"syn 2.0.119",
]
[[package]]
Cargo.toml
+8 -1
View File
@@ -40,7 +40,8 @@ serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
thiserror = "1.0"
tracing = "0.1"
tracing-subscriber = "0.3"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
tracing-appender = "0.2"
sqlx = { version = "0.8", features = ["runtime-tokio", "sqlite"] }
ratatui = "0.28"
crossterm = { version = "0.28", features = ["event-stream"] }
@@ -79,11 +80,17 @@ flate2 = "1.0"
tar = "0.4"
zip = { version = "8.6", default-features = false, features = ["deflate"] }
tonic = "0.14"
tower = "0.5"
http = "1"
http-body = "1"
http-body-util = "0.1"
tonic-prost = "0.14"
prost = "0.14"
prost-types = "0.14"
tonic-prost-build = "0.14"
protoc-bin-vendored = "3.2"
subtle = "2.6"
fs4 = "0.13"
crates/nuncio-cal/Cargo.toml
+1 -1
View File
@@ -21,8 +21,8 @@ chrono-tz.workspace = true
rrule.workspace = true
chrono.workspace = true
reqwest.workspace = true
quick-xml.workspace = true
[dev-dependencies]
tempfile.workspace = true
wiremock.workspace = true
tracing-subscriber.workspace = true
crates/nuncio-cal/src/caldav.rs
+375 -23
View File
@@ -1,8 +1,11 @@
//! CalDAV (RFC 4791) / WebDAV REPORT client and XML query generator.
use async_trait::async_trait;
use nuncio_core::dav::{parse_multistatus, CALDAV_NS};
use nuncio_core::model::CalendarEvent;
use nuncio_core::redact::Redacted;
use serde::{Deserialize, Serialize};
use tracing::{debug, info, instrument, warn};
use crate::backend::CalendarBackend;
use crate::parser::{CalendarError, IcalParserAdapter};
@@ -13,7 +16,7 @@ use crate::parser::{CalendarError, IcalParserAdapter};
/// `caldav_url` must already resolve to a specific calendar collection (e.g.
/// `https://caldav.example.com/dav/calendars/user/jmaes/work/`) -- PROPFIND-based
/// `calendar-home-set` auto-discovery is out of scope for this client.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct CalDavAccountConfig {
/// Nuncio account identifier this calendar collection belongs to.
pub account_id: String,
@@ -22,8 +25,26 @@ pub struct CalDavAccountConfig {
/// Basic-auth username (or app-specific username, per provider).
pub username: String,
/// Basic-auth secret (password or app-specific token) resolved from the OS keyring by the
/// caller -- never stored anywhere else in plaintext.
pub auth_token: String,
/// caller -- never stored anywhere else in plaintext. Wrapped in [`Redacted`] so a derived
/// `Debug`, a `Display`, or `serde` can never disclose it to a log; the raw value is
/// reachable only through an explicit `expose_secret` call.
pub auth_token: Redacted<String>,
}
/// Best-effort extraction of the `UID` property from a raw (possibly malformed) VEVENT/
/// VCALENDAR block, used only to correlate a dropped-VEVENT log line with the source event --
/// never returns the summary, attendees, description, or any other calendar content.
fn extract_vevent_uid(raw_ics: &str) -> Option<String> {
raw_ics.lines().find_map(|line| {
let line = line.trim();
let (name, value) = line.split_once(':')?;
let bare_name = name.split(';').next().unwrap_or(name);
if bare_name.eq_ignore_ascii_case("UID") {
Some(value.trim().to_string())
} else {
None
}
})
}
/// CalDAV client protocol engine managing WebDAV REPORT queries against a real server.
@@ -65,33 +86,74 @@ impl CalDavClient {
)
}
/// Parse CalDAV WebDAV XML `<multistatus>` response containing embedded VEVENT data.
/// Parse a CalDAV WebDAV XML `<multistatus>` response containing embedded VEVENT data.
///
/// The document is traversed with a namespace-aware parser and `calendar-data` is matched
/// on `{urn:ietf:params:xml:ns:caldav}calendar-data`, so the server's choice of prefix (or
/// of a default namespace) is irrelevant and a body this client cannot read is an error
/// rather than an empty -- and indistinguishable from genuinely empty -- event list.
///
/// A VEVENT block that fails to parse is dropped from the returned events but never
/// silently: it is logged at `warn` (with its UID, if extractable, and a short parse-error
/// reason -- never the raw VEVENT body) so a sync that quietly lost calendar data is
/// visible in telemetry instead of just showing up as a smaller-than-expected event count.
/// A `<response>` that carried no readable `calendar-data` at all (a `404` propstat, say)
/// is logged the same way.
pub fn parse_multistatus_response(
&self,
calendar_id: &str,
raw_xml: &str,
) -> Result<Vec<CalendarEvent>, CalendarError> {
let mut events = Vec::new();
self.parse_multistatus_response_counted(calendar_id, raw_xml)
.map(|(events, _dropped)| events)
}
// Extract <c:calendar-data> or <calendar-data> text blocks
for block in raw_xml.split("<c:calendar-data>") {
if let Some((ics_data, _)) = block.split_once("</c:calendar-data>") {
let clean_ics = ics_data.trim();
if !clean_ics.is_empty() {
let event_id = format!("caldav-evt-{}", events.len() + 1);
if let Ok(event) = IcalParserAdapter::parse_ical(
&event_id,
&self.config.account_id,
calendar_id,
clean_ics,
) {
events.push(event);
}
/// Same parse as [`Self::parse_multistatus_response`], additionally reporting how many
/// calendar entries were lost -- unparseable VEVENTs plus unreadable responses -- so
/// callers can log round-trip counts.
fn parse_multistatus_response_counted(
&self,
calendar_id: &str,
raw_xml: &str,
) -> Result<(Vec<CalendarEvent>, usize), CalendarError> {
let report = parse_multistatus(raw_xml, CALDAV_NS, "calendar-data")
.map_err(|e| CalendarError::MalformedResponse(e.to_string()))?;
let mut dropped = report.skipped.len();
for skipped in &report.skipped {
warn!(
account_id = %self.config.account_id,
calendar_id = %calendar_id,
href = %skipped.href.as_deref().unwrap_or("unknown"),
reason = %skipped.reason,
"dropped CalDAV response with no readable calendar-data"
);
}
let mut events = Vec::new();
for (index, entry) in report.values.iter().enumerate() {
let event_id = format!("caldav-evt-{}", index + 1);
match IcalParserAdapter::parse_ical(
&event_id,
&self.config.account_id,
calendar_id,
&entry.value,
) {
Ok(event) => events.push(event),
Err(err) => {
dropped += 1;
warn!(
account_id = %self.config.account_id,
calendar_id = %calendar_id,
uid = %extract_vevent_uid(&entry.value).unwrap_or_else(|| "unknown".to_string()),
reason = %err,
"dropped unparseable VEVENT"
);
}
}
}
Ok(events)
Ok((events, dropped))
}
/// Format a unix timestamp as the `YYYYMMDDTHHMMSSZ` form RFC 4791's
@@ -113,6 +175,11 @@ impl CalDavClient {
/// returned. A server or network failure surfaces as [`CalendarError::TransportFailed`];
/// a malformed TZID inside a returned VEVENT surfaces as
/// [`CalendarError::UnresolvableTimezone`] via [`IcalParserAdapter::parse_ical`].
#[instrument(
name = "caldav_sync",
skip(self),
fields(account_id = %self.config.account_id, calendar_id = %calendar_id)
)]
pub async fn fetch_remote_events(
&self,
calendar_id: &str,
@@ -128,12 +195,21 @@ impl CalDavClient {
let report_method = reqwest::Method::from_bytes(b"REPORT")
.map_err(|e| CalendarError::TransportFailed(format!("invalid HTTP method: {e}")))?;
debug!(
account_id = %self.config.account_id,
calendar_id = %calendar_id,
"issuing CalDAV REPORT calendar-query"
);
let response = self
.http
.request(report_method, &self.config.caldav_url)
.header("Content-Type", "application/xml; charset=utf-8")
.header("Depth", "1")
.basic_auth(&self.config.username, Some(&self.config.auth_token))
.basic_auth(
&self.config.username,
Some(self.config.auth_token.expose_secret()),
)
.body(body)
.send()
.await
@@ -153,7 +229,17 @@ impl CalDavClient {
.await
.map_err(|e| CalendarError::TransportFailed(e.to_string()))?;
self.parse_multistatus_response(calendar_id, &raw_xml)
let (events, dropped) = self.parse_multistatus_response_counted(calendar_id, &raw_xml)?;
info!(
account_id = %self.config.account_id,
calendar_id = %calendar_id,
fetched = events.len() + dropped,
parsed = events.len(),
dropped,
"completed CalDAV sync"
);
Ok(events)
}
}
@@ -179,7 +265,7 @@ mod tests {
account_id: "acct-1".to_string(),
caldav_url: "https://caldav.example.com/calendars/work/".to_string(),
username: "jmaes".to_string(),
auth_token: "app-token-secret".to_string(),
auth_token: Redacted::new("app-token-secret".to_string()),
}
}
@@ -221,10 +307,276 @@ END:VCALENDAR</c:calendar-data>
assert_eq!(events[0].location, Some("Boardroom".to_string()));
}
#[test]
fn parse_multistatus_response_surfaces_a_warn_for_a_dropped_vevent_instead_of_silence() {
use crate::test_tracing::with_recorder;
let client = CalDavClient::new(test_config());
// A TZID that cannot be resolved to a real IANA zone; the VEVENT fails to parse but
// the one alongside it (plain UTC) must still come through.
let xml_response = r#"<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav">
<d:response>
<d:propstat>
<d:prop>
<c:calendar-data>BEGIN:VCALENDAR
VERSION:2.0
BEGIN:VEVENT
UID:evt-unresolvable-tzid
SUMMARY:Confidential Merger Talks
DTSTART;TZID=Not/ARealZone:20240101T090000
DTEND;TZID=Not/ARealZone:20240101T100000
END:VEVENT
END:VCALENDAR</c:calendar-data>
</d:prop>
</d:propstat>
</d:response>
<d:response>
<d:propstat>
<d:prop>
<c:calendar-data>BEGIN:VCALENDAR
VERSION:2.0
BEGIN:VEVENT
SUMMARY:Good Event
DTSTART:20240301T120000Z
DTEND:20240301T130000Z
END:VEVENT
END:VCALENDAR</c:calendar-data>
</d:prop>
</d:propstat>
</d:response>
</d:multistatus>"#;
let (recorder, result) =
with_recorder(|| client.parse_multistatus_response("cal-work", xml_response));
let events = result.expect("the unparseable VEVENT is dropped, not fatal");
assert_eq!(events.len(), 1);
assert_eq!(events[0].summary, "Good Event");
let warnings: Vec<_> = recorder
.events()
.into_iter()
.filter(|e| e.level == tracing::Level::WARN)
.collect();
assert_eq!(warnings.len(), 1, "exactly one VEVENT should be dropped");
assert_eq!(
warnings[0].fields.get("uid").map(String::as_str),
Some("evt-unresolvable-tzid")
);
assert!(warnings[0].message().contains("dropped"));
// The dropped VEVENT's summary (potential PII) must never appear in telemetry.
for value in recorder.all_field_values() {
assert!(!value.contains("Confidential Merger Talks"));
}
}
/// The same document, written the three ways a compliant server may write it. Before
/// namespace-aware parsing, only the middle one produced any events at all.
#[test]
fn parse_multistatus_response_reads_any_prefix_binding_of_the_caldav_namespace() {
let client = CalDavClient::new(test_config());
let event_ics = "BEGIN:VCALENDAR\nVERSION:2.0\nBEGIN:VEVENT\nSUMMARY:Product Planning\nEND:VEVENT\nEND:VCALENDAR";
let uppercase = format!(
r#"<?xml version="1.0" encoding="utf-8"?>
<D:multistatus xmlns:D="DAV:" xmlns:CAL="urn:ietf:params:xml:ns:caldav">
<D:response><D:href>/evt1.ics</D:href><D:propstat><D:prop>
<CAL:calendar-data>{event_ics}</CAL:calendar-data>
</D:prop><D:status>HTTP/1.1 200 OK</D:status></D:propstat></D:response>
</D:multistatus>"#
);
let default_ns = format!(
r#"<?xml version="1.0" encoding="utf-8"?>
<multistatus xmlns="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav">
<response><href>/evt1.ics</href><propstat><prop>
<c:calendar-data>{event_ics}</c:calendar-data>
</prop><status>HTTP/1.1 200 OK</status></propstat></response>
</multistatus>"#
);
let odd_prefix = format!(
r#"<?xml version="1.0" encoding="utf-8"?>
<ns0:multistatus xmlns:ns0="DAV:" xmlns:ns1="urn:ietf:params:xml:ns:caldav">
<ns0:response><ns0:href>/evt1.ics</ns0:href><ns0:propstat><ns0:prop>
<ns1:calendar-data>{event_ics}</ns1:calendar-data>
</ns0:prop><ns0:status>HTTP/1.1 200 OK</ns0:status></ns0:propstat></ns0:response>
</ns0:multistatus>"#
);
for (label, xml) in [
("uppercase prefixes", &uppercase),
("default namespace", &default_ns),
("generated prefixes", &odd_prefix),
] {
let events = client
.parse_multistatus_response("cal-work", xml)
.unwrap_or_else(|e| panic!("{label} parses: {e}"));
assert_eq!(events.len(), 1, "{label}");
assert_eq!(events[0].summary, "Product Planning", "{label}");
}
}
#[test]
fn parse_multistatus_response_warns_when_a_response_has_no_readable_calendar_data() {
use crate::test_tracing::with_recorder;
let client = CalDavClient::new(test_config());
// The first response's property merely *contains* the searched name; the second
// returns the real property under a 404 propstat, which is not a value.
let xml_response = r#"<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav">
<d:response>
<d:href>/lookalike.ics</d:href>
<d:propstat><d:prop>
<c:calendar-data-summary>NOT THE PAYLOAD</c:calendar-data-summary>
</d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat>
</d:response>
<d:response>
<d:href>/gone.ics</d:href>
<d:propstat><d:prop>
<c:calendar-data>BEGIN:VCALENDAR
END:VCALENDAR</c:calendar-data>
</d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat>
</d:response>
</d:multistatus>"#;
let (recorder, result) =
with_recorder(|| client.parse_multistatus_response("cal-work", xml_response));
let events = result.expect("an unreadable response is not fatal");
assert!(events.is_empty(), "neither response carries calendar-data");
let warnings: Vec<_> = recorder
.events()
.into_iter()
.filter(|e| e.level == tracing::Level::WARN)
.collect();
assert_eq!(warnings.len(), 2, "both omissions must be visible");
let hrefs: Vec<_> = warnings
.iter()
.filter_map(|w| w.fields.get("href").cloned())
.collect();
assert!(hrefs.contains(&"/lookalike.ics".to_string()));
assert!(hrefs.contains(&"/gone.ics".to_string()));
assert!(warnings
.iter()
.any(|w| w.fields.get("reason").is_some_and(|r| r.contains("404"))));
}
#[test]
fn parse_multistatus_response_rejects_a_body_it_cannot_read_instead_of_returning_empty() {
let client = CalDavClient::new(test_config());
// A malformed document: the `<d:response>` element is never closed.
let truncated = r#"<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:"><d:response><d:href>/a.ics</d:href>"#;
assert!(matches!(
client.parse_multistatus_response("cal-work", truncated),
Err(CalendarError::MalformedResponse(_))
));
// A body that is not a multistatus at all (an intercepting proxy's error page).
let not_dav = "<html><body>502 Bad Gateway</body></html>";
assert!(matches!(
client.parse_multistatus_response("cal-work", not_dav),
Err(CalendarError::MalformedResponse(_))
));
}
#[test]
fn format_query_timestamp_produces_ical_utc_form() {
let formatted =
CalDavClient::format_query_timestamp(1_704_067_200).expect("valid timestamp");
assert_eq!(formatted, "20240101T000000Z");
}
#[test]
fn debug_impl_redacts_auth_token() {
let config = test_config();
let rendered = format!("{config:?}");
assert!(!rendered.contains("app-token-secret"));
assert!(rendered.contains("<redacted>"));
}
#[test]
fn fetch_remote_events_logs_completion_counts_including_a_drop() {
use crate::test_tracing::with_recorder;
use wiremock::matchers::{method, path};
use wiremock::{Mock, MockServer, ResponseTemplate};
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("current-thread runtime");
let (recorder, result) = with_recorder(|| {
runtime.block_on(async {
let mock_server = MockServer::start().await;
let xml_response = r#"<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav">
<d:response>
<d:propstat>
<d:prop>
<c:calendar-data>BEGIN:VCALENDAR
VERSION:2.0
BEGIN:VEVENT
UID:evt-bad
DTSTART;TZID=Not/ARealZone:20240101T090000
END:VEVENT
END:VCALENDAR</c:calendar-data>
</d:prop>
</d:propstat>
</d:response>
<d:response>
<d:propstat>
<d:prop>
<c:calendar-data>BEGIN:VCALENDAR
VERSION:2.0
BEGIN:VEVENT
SUMMARY:Good Event
DTSTART:20240301T120000Z
DTEND:20240301T130000Z
END:VEVENT
END:VCALENDAR</c:calendar-data>
</d:prop>
</d:propstat>
</d:response>
</d:multistatus>"#;
Mock::given(method("REPORT"))
.and(path("/calendars/work/"))
.respond_with(ResponseTemplate::new(207).set_body_string(xml_response))
.mount(&mock_server)
.await;
let mut config = test_config();
config.caldav_url = format!("{}/calendars/work/", mock_server.uri());
let client = CalDavClient::new(config);
client
.fetch_remote_events("cal-work", 1_700_000_000, 1_800_000_000)
.await
})
});
let events = result.expect("one good VEVENT still comes through");
assert_eq!(events.len(), 1);
let completion = recorder
.events()
.into_iter()
.find(|e| e.message().contains("completed CalDAV sync"))
.expect("a completion event is logged");
assert_eq!(
completion.fields.get("fetched").map(String::as_str),
Some("2")
);
assert_eq!(
completion.fields.get("parsed").map(String::as_str),
Some("1")
);
assert_eq!(
completion.fields.get("dropped").map(String::as_str),
Some("1")
);
}
}
crates/nuncio-cal/src/carddav.rs
-148
View File
@@ -1,148 +0,0 @@
//! CardDAV (vCard RFC 6350) contact sync and XML query parser.
use crate::parser::CalendarError;
use serde::{Deserialize, Serialize};
/// Contact entity representation synced over CardDAV.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct Contact {
pub id: String,
pub name: String,
pub email: String,
pub phone: Option<String>,
pub organization: Option<String>,
}
/// CardDAV client protocol engine managing vCard address book queries.
pub struct CardDavClient {
account_id: String,
}
impl CardDavClient {
/// Create a new `CardDavClient` bound to an account ID.
pub fn new(account_id: &str) -> Self {
Self {
account_id: account_id.to_string(),
}
}
/// Construct a standard CardDAV `<addressbook-query>` XML payload.
pub fn build_addressbook_query() -> String {
r#"<?xml version="1.0" encoding="utf-8" ?>
<card:addressbook-query xmlns:d="DAV:" xmlns:card="urn:ietf:params:xml:ns:carddav">
<d:prop>
<d:getetag />
<card:address-data />
</d:prop>
</card:addressbook-query>"#
.to_string()
}
/// Parse raw vCard string into domain [`Contact`] entity.
pub fn parse_vcard(id: &str, raw_vcard: &str) -> Result<Contact, CalendarError> {
let mut name = String::new();
let mut email = String::new();
let mut phone = None;
let mut organization = None;
for line in raw_vcard.lines() {
let line = line.trim();
if let Some(val) = line.strip_prefix("FN:") {
name = val.trim().to_string();
} else if let Some(val) = line.strip_prefix("EMAIL:") {
email = val.trim().to_string();
} else if line.starts_with("EMAIL;") {
if let Some((_, val)) = line.split_once(':') {
email = val.trim().to_string();
}
} else if let Some(val) = line.strip_prefix("TEL:") {
phone = Some(val.trim().to_string());
} else if line.starts_with("TEL;") {
if let Some((_, val)) = line.split_once(':') {
phone = Some(val.trim().to_string());
}
} else if let Some(val) = line.strip_prefix("ORG:") {
organization = Some(val.trim().to_string());
}
}
if name.is_empty() && email.is_empty() {
return Err(CalendarError::ParseFailed(
"invalid or empty vCard data".to_string(),
));
}
if name.is_empty() {
name = email.clone();
}
Ok(Contact {
id: id.to_string(),
name,
email,
phone,
organization,
})
}
/// Parse CardDAV WebDAV XML `<multistatus>` response containing embedded vCard data.
pub fn parse_multistatus_response(&self, raw_xml: &str) -> Result<Vec<Contact>, CalendarError> {
let mut contacts = Vec::new();
for block in raw_xml.split("<card:address-data>") {
if let Some((vcard_data, _)) = block.split_once("</card:address-data>") {
let clean_vcard = vcard_data.trim();
if !clean_vcard.is_empty() {
let contact_id = format!("{}-contact-{}", self.account_id, contacts.len() + 1);
if let Ok(contact) = Self::parse_vcard(&contact_id, clean_vcard) {
contacts.push(contact);
}
}
}
}
Ok(contacts)
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn parse_vcard_valid_structure() {
let vcard = r#"BEGIN:VCARD
VERSION:4.0
FN:James Maes
EMAIL:james.maes@kof22.com
TEL:+1-555-0199
ORG:Kof22 Labs
END:VCARD"#;
let contact = CardDavClient::parse_vcard("c-1", vcard).expect("parse vcard");
assert_eq!(contact.name, "James Maes");
assert_eq!(contact.email, "james.maes@kof22.com");
assert_eq!(contact.phone.as_deref(), Some("+1-555-0199"));
assert_eq!(contact.organization.as_deref(), Some("Kof22 Labs"));
}
#[test]
fn parse_multistatus_carddav_xml() {
let client = CardDavClient::new("acct-1");
let xml = r#"<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:" xmlns:card="urn:ietf:params:xml:ns:carddav">
<d:response>
<card:address-data>BEGIN:VCARD
FN:Alice Dev
EMAIL:alice@nuncio.mx
END:VCARD</card:address-data>
</d:response>
</d:multistatus>"#;
let contacts = client
.parse_multistatus_response(xml)
.expect("parse multistatus");
assert_eq!(contacts.len(), 1);
assert_eq!(contacts[0].name, "Alice Dev");
}
}
crates/nuncio-cal/src/ical.rs
+182
View File
@@ -0,0 +1,182 @@
//! iCalendar (RFC 5545) VEVENT serializer -- the write-back inverse of
//! [`crate::parser::IcalParserAdapter`].
//!
//! [`CalendarEvent`] stores `start_time`/`end_time` as resolved UTC unix timestamps (the parser
//! already collapses any `TZID`/floating/`Z` form down to a single UTC instant), so this
//! serializer always emits `Z`-suffixed UTC `DATE-TIME` values. That is a lossless encoding of
//! what the model actually holds: re-parsing the emitted text yields the same UTC instant no
//! matter which form the original payload used.
use chrono::{DateTime, Utc};
use nuncio_core::model::CalendarEvent;
/// Adds RFC 5545 serialization to [`CalendarEvent`].
///
/// `CalendarEvent` is defined in `nuncio-core`, so this is an extension trait rather than an
/// inherent impl -- Rust forbids inherent impls on types owned by another crate. Importing the
/// trait makes `event.to_ical()` read identically to an inherent method at call sites.
pub trait ToIcal {
/// Serialize this event as a complete iCalendar payload: a single `VEVENT` wrapped in a
/// `VCALENDAR`, the shape a CalDAV `PUT` request body expects.
fn to_ical(&self) -> String;
}
impl ToIcal for CalendarEvent {
fn to_ical(&self) -> String {
let dtstamp = format_utc_datetime(Utc::now().timestamp());
let dtstart = format_utc_datetime(self.start_time);
let dtend = format_utc_datetime(self.end_time);
let mut vevent = String::new();
vevent.push_str("BEGIN:VEVENT\r\n");
vevent.push_str(&format!("UID:{}\r\n", escape_text(&self.id)));
vevent.push_str(&format!("DTSTAMP:{}\r\n", dtstamp));
vevent.push_str(&format!("DTSTART:{}\r\n", dtstart));
vevent.push_str(&format!("DTEND:{}\r\n", dtend));
vevent.push_str(&format!("SUMMARY:{}\r\n", escape_text(&self.summary)));
if let Some(location) = &self.location {
vevent.push_str(&format!("LOCATION:{}\r\n", escape_text(location)));
}
if let Some(rrule) = &self.rrule {
// RRULE is a structured value (RFC 5545 3.3.10), not TEXT -- it must not go through
// TEXT escaping, and the parser reads it back verbatim via `properties().get`.
vevent.push_str(&format!("RRULE:{}\r\n", rrule));
}
vevent.push_str("END:VEVENT\r\n");
let mut vcalendar = String::new();
vcalendar.push_str("BEGIN:VCALENDAR\r\n");
vcalendar.push_str("VERSION:2.0\r\n");
vcalendar.push_str("PRODID:-//Nuncio//Calendar Engine//EN\r\n");
vcalendar.push_str(&vevent);
vcalendar.push_str("END:VCALENDAR\r\n");
vcalendar
}
}
/// Format a UTC unix timestamp as an RFC 5545 `DATE-TIME` in UTC form (`YYYYMMDDTHHMMSSZ`).
fn format_utc_datetime(unix_ts: i64) -> String {
DateTime::<Utc>::from_timestamp(unix_ts, 0)
.map(|dt| dt.format("%Y%m%dT%H%M%SZ").to_string())
// `from_timestamp` only returns `None` for a timestamp outside chrono's representable
// range; falling back to the Unix epoch keeps this a total function without panicking,
// since a corrupt in-memory timestamp is not something this serializer can repair.
.unwrap_or_else(|| "19700101T000000Z".to_string())
}
/// Escape a `TEXT` value per RFC 5545 3.3.11: backslash, semicolon, and comma are escaped with a
/// leading backslash, and embedded newlines become the two-character literal `\n`. Order matters
/// -- backslash must be escaped first so the escaping added for the other characters is not
/// itself re-escaped.
fn escape_text(value: &str) -> String {
let mut escaped = String::with_capacity(value.len());
for ch in value.chars() {
match ch {
'\\' => escaped.push_str("\\\\"),
';' => escaped.push_str("\\;"),
',' => escaped.push_str("\\,"),
'\n' => escaped.push_str("\\n"),
'\r' => {}
other => escaped.push(other),
}
}
escaped
}
#[cfg(test)]
mod tests {
use super::*;
use crate::parser::IcalParserAdapter;
fn event(
summary: &str,
start_time: i64,
end_time: i64,
rrule: Option<&str>,
location: Option<&str>,
) -> CalendarEvent {
CalendarEvent {
id: "evt-roundtrip".to_string(),
account_id: "acct-1".to_string(),
calendar_id: "cal-1".to_string(),
summary: summary.to_string(),
start_time,
end_time,
rrule: rrule.map(str::to_string),
location: location.map(str::to_string),
}
}
fn roundtrip(original: &CalendarEvent) -> CalendarEvent {
let ics = original.to_ical();
IcalParserAdapter::parse_ical(
&original.id,
&original.account_id,
&original.calendar_id,
&ics,
)
.expect("serialized iCalendar must parse back")
}
/// Proves the serializer is the true inverse of [`IcalParserAdapter::parse_ical`] across the
/// event shapes CalDAV write-back must support: a plain timed event, an all-day event
/// (whole-day UTC window), an event whose UTC instant originated from a `TZID` source (the
/// model has already resolved it to UTC, so round-tripping through `to_ical` must preserve
/// that same instant), and a recurring event carrying an `RRULE`.
#[test]
fn to_ical_roundtrips_through_parse_ical_for_all_event_shapes() {
let timed = event(
"Standup",
1_709_294_400,
1_709_298_000,
None,
Some("Room 5"),
);
assert_eq!(roundtrip(&timed), timed);
// Whole-day window: 2024-03-01T00:00:00Z .. 2024-03-01T23:59:59Z, matching the parser's
// own DATE-only convention (start of day / end of day).
let all_day = event("Conference", 1_709_251_200, 1_709_337_599, None, None);
assert_eq!(roundtrip(&all_day), all_day);
// 2024-07-15 09:00 America/New_York (EDT, UTC-4) resolves to this UTC instant; the model
// only ever stores the resolved UTC epoch, so serializing and re-parsing must land on
// exactly the same instant even though the TZID itself is not retained.
let from_tzid = event("Standup EDT", 1_721_048_400, 1_721_052_000, None, None);
assert_eq!(roundtrip(&from_tzid), from_tzid);
let recurring = event(
"Weekly Sync",
1_709_294_400,
1_709_298_000,
Some("FREQ=WEEKLY;BYDAY=MO"),
Some("Conference Room 101"),
);
assert_eq!(roundtrip(&recurring), recurring);
}
#[test]
fn to_ical_escapes_rfc5545_special_characters_in_text_fields() {
let special = event(
"Comma, semicolon; backslash\\ and\nnewline",
1_709_294_400,
1_709_298_000,
None,
Some("Loc; with, chars"),
);
let ics = special.to_ical();
assert!(ics.contains("SUMMARY:Comma\\, semicolon\\; backslash\\\\ and\\nnewline\r\n"));
assert!(ics.contains("LOCATION:Loc\\; with\\, chars\r\n"));
assert_eq!(roundtrip(&special), special);
}
#[test]
fn to_ical_wraps_a_single_vevent_in_a_vcalendar_shell() {
let plain = event("Solo Event", 1_709_294_400, 1_709_298_000, None, None);
let ics = plain.to_ical();
assert!(ics.starts_with("BEGIN:VCALENDAR\r\n"));
assert!(ics.contains("VERSION:2.0\r\n"));
assert!(ics.trim_end().ends_with("END:VCALENDAR"));
assert_eq!(ics.matches("BEGIN:VEVENT").count(), 1);
}
}
crates/nuncio-cal/src/lib.rs
+4 -2
View File
@@ -5,16 +5,18 @@
pub mod backend;
pub mod caldav;
pub mod carddav;
pub mod ical;
pub mod mock;
pub mod nlp;
pub mod parser;
pub mod rrule;
pub mod scheduling;
#[cfg(test)]
mod test_tracing;
pub use backend::CalendarBackend;
pub use caldav::{CalDavAccountConfig, CalDavClient};
pub use carddav::{CardDavClient, Contact};
pub use ical::ToIcal;
pub use mock::MockCalendarBackend;
pub use nlp::{NaturalLanguageScheduler, NlpError, ParsedSchedulingIntent};
pub use parser::{CalendarError, IcalParserAdapter};
crates/nuncio-cal/src/nlp.rs
+664 -60
View File
@@ -1,101 +1,705 @@
//! Natural Language Calendar Booking NLP Parser.
//! A deliberately small, exact natural-language scheduling parser.
//!
//! The grammar below is the *entire* set of phrasings this parser understands.
//! Anything outside it returns an error. That is the point: a scheduler that
//! guesses produces a plausible-looking event at the wrong instant, and the user
//! only discovers the mistake by missing the meeting. Refusing is recoverable;
//! silently booking the wrong time is not.
//!
//! # Grammar
//!
//! ```text
//! <summary> <day> at <time> for <duration>
//! ```
//!
//! * `<summary>` — free text, must be non-empty. The `at`/`for` separators are
//! matched from the right, so a summary may itself contain them
//! ("Prep for review tomorrow at 9am for 30 minutes").
//! * `<day>` — `today`, `tomorrow`, or an ISO calendar date `YYYY-MM-DD`.
//! * `<time>` — `noon`, `midnight`, `3pm`, `3:15pm`, `9am`, `9:05am`, or a
//! 24-hour `HH:MM`. A bare number (`at 3`) is rejected as ambiguous.
//! * `<duration>` — `<n> minute(s)|min(s)` or `<n> hour(s)|hr(s)`, non-zero and
//! no longer than 24 hours.
//!
//! Both `at <time>` and `for <duration>` are mandatory. Without a time there is
//! no instant to resolve, and a defaulted duration is a guess about the user's
//! intent, so neither is invented.
//!
//! # Deliberately unsupported
//!
//! Weekday phrasings (`next Tuesday`, `this Friday`, `on Tuesday`) are rejected
//! as ambiguous rather than resolved. English speakers do not agree on whether
//! "next Tuesday" means the Tuesday that is coming or the Tuesday of the
//! following week, and the two readings differ by a whole week. There is no
//! interpretation this parser could pick that would not be wrong for a large
//! share of users, so it asks instead.
//!
//! # Time zones
//!
//! Resolution requires an explicit IANA time zone: "3pm" is not an instant until
//! you know where the user is, and a Unix timestamp alone does not carry that.
//! Wall-clock times are resolved through the zone's DST rules, and the two
//! local-time anomalies are reported rather than papered over:
//!
//! * A time inside a spring-forward gap does not exist → [`NlpError::NonexistentLocalTime`].
//! * A time inside a fall-back overlap occurs twice → [`NlpError::AmbiguousLocalTime`].
//!
//! Durations are added as absolute elapsed seconds, so an event that spans a DST
//! transition keeps its real length and its wall-clock end time shifts, which is
//! what a physical meeting actually does.
use nuncio_core::model::CalendarEvent;
use chrono::{Duration, LocalResult, NaiveDate, TimeZone};
use chrono_tz::Tz;
use serde::{Deserialize, Serialize};
use thiserror::Error;
/// Parsed natural language scheduling intent result.
#[derive(Debug, Clone, Serialize, Deserialize)]
/// The longest event this parser will accept, in minutes.
const MAX_DURATION_MINUTES: u32 = 24 * 60;
/// A successfully parsed scheduling request.
///
/// This is a parse result, not a calendar event: it deliberately carries no
/// account or calendar identity, because the parser has no way to know which
/// calendar the user meant and must not invent one.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct ParsedSchedulingIntent {
/// Event title, with the trailing scheduling phrase removed.
pub summary: String,
/// Start instant, Unix seconds UTC.
pub start_time: i64,
/// End instant, Unix seconds UTC (`start_time` plus the parsed duration).
pub end_time: i64,
/// Parsed duration in minutes.
pub duration_minutes: u32,
pub location: Option<String>,
pub attendees: Vec<String>,
}
/// Errors originating from natural language scheduling parsing.
#[derive(Error, Debug, PartialEq, Eq)]
/// Why a scheduling string could not be resolved to an exact instant.
#[derive(Error, Debug, Clone, PartialEq, Eq)]
pub enum NlpError {
#[error("failed to parse scheduling string: {0}")]
ParseFailed(String),
#[error("empty scheduling query provided")]
EmptyQuery,
#[error("no event title found before the scheduling phrase")]
MissingSummary,
#[error("no time of day given; expected `... at <time> ...`")]
MissingTime,
#[error("no duration given; expected `... for <n> minutes`")]
MissingDuration,
#[error(
"`{0}` is ambiguous: it can mean either the coming weekday or the one a \
week later. Use `today`, `tomorrow`, or an explicit YYYY-MM-DD date"
)]
AmbiguousWeekday(String),
#[error("unsupported day `{0}`; expected `today`, `tomorrow`, or YYYY-MM-DD")]
UnsupportedDay(String),
#[error("unsupported time `{0}`; expected `noon`, `midnight`, `3pm`, `3:15pm`, or `HH:MM`")]
UnsupportedTime(String),
#[error("unsupported duration `{0}`; expected `<n> minutes` or `<n> hours`, up to 24 hours")]
UnsupportedDuration(String),
#[error("{date} {time} does not exist in {zone}: the clocks skip forward over it")]
NonexistentLocalTime {
date: String,
time: String,
zone: String,
},
#[error("{date} {time} occurs twice in {zone}: the clocks fall back over it")]
AmbiguousLocalTime {
date: String,
time: String,
zone: String,
},
#[error("the reference timestamp {0} is not a representable instant")]
InvalidReference(i64),
#[error("the resolved instant overflows the representable range")]
InstantOutOfRange,
}
/// Natural Language Scheduling NLP Parser.
/// Parser for the small scheduling grammar documented at the module level.
pub struct NaturalLanguageScheduler;
impl NaturalLanguageScheduler {
/// Parse human input string (e.g. "Coffee with Bob next Tuesday at 2pm for 45 minutes") into [`CalendarEvent`].
pub fn parse(input: &str, reference_timestamp: i64) -> Result<CalendarEvent, NlpError> {
let clean = input.trim();
if clean.is_empty() {
/// Resolve `input` to an exact instant, relative to `reference_timestamp`
/// (Unix seconds) interpreted in `zone`.
///
/// `zone` is required rather than defaulted: relative days and wall-clock
/// times are meaningless without it, and assuming a zone is the specific
/// mistake that produces confidently wrong bookings.
///
/// Returns [`NlpError`] for every phrasing outside the documented grammar.
/// It never falls back to an approximation.
pub fn parse(
input: &str,
reference_timestamp: i64,
zone: Tz,
) -> Result<ParsedSchedulingIntent, NlpError> {
let original = input.trim();
if original.is_empty() {
return Err(NlpError::EmptyQuery);
}
// Extract duration if specified (e.g., "45 minutes", "1 hour")
let duration_secs = if clean.contains("45 min") || clean.contains("45 minutes") {
45 * 60
} else if clean.contains("30 min") || clean.contains("30 minutes") {
30 * 60
} else if clean.contains("1 hour") || clean.contains("60 minutes") {
60 * 60
} else {
30 * 60 // Default 30 minutes
};
// ASCII lowercasing preserves byte length, so byte offsets found in the
// lowered copy stay valid in `original` and the summary keeps its case.
let lowered = original.to_ascii_lowercase();
// Determine summary (strip timing phrases)
let summary = if let Some((head, _)) = clean.split_once(" next ") {
head.to_string()
} else if let Some((head, _)) = clean.split_once(" tomorrow ") {
head.to_string()
} else if let Some((head, _)) = clean.split_once(" at ") {
head.to_string()
} else {
clean.to_string()
};
let (head, duration_text) =
split_last(&lowered, original, " for ").ok_or(NlpError::MissingDuration)?;
let duration_minutes = parse_duration(&duration_text.to_ascii_lowercase())?;
// Standard 24h offset calculation for "next" / "tomorrow"
let start = reference_timestamp + 86400;
let end = start + duration_secs;
let lowered_head = head.to_ascii_lowercase();
let (summary_and_day, time_text) =
split_last(&lowered_head, head, " at ").ok_or(NlpError::MissingTime)?;
let (hour, minute) = parse_time_of_day(&time_text.to_ascii_lowercase())?;
Ok(CalendarEvent {
id: format!("nlp-evt-{}", uuid_or_timestamp(start)),
account_id: "default".to_string(),
calendar_id: "default".to_string(),
summary: summary.trim().to_string(),
start_time: start,
end_time: end,
rrule: None,
location: None,
let reference = zone
.timestamp_opt(reference_timestamp, 0)
.single()
.ok_or(NlpError::InvalidReference(reference_timestamp))?;
let (summary, date) = split_day(summary_and_day, reference.date_naive())?;
if summary.is_empty() {
return Err(NlpError::MissingSummary);
}
let start = resolve_local(zone, date, hour, minute)?;
let end = start
.checked_add_signed(Duration::minutes(i64::from(duration_minutes)))
.ok_or(NlpError::InstantOutOfRange)?;
Ok(ParsedSchedulingIntent {
summary,
start_time: start.timestamp(),
end_time: end.timestamp(),
duration_minutes,
})
}
}
fn uuid_or_timestamp(ts: i64) -> String {
format!("evt-{ts}")
/// Split `original` at the last occurrence of `sep` located in `lowered`.
///
/// `lowered` must be the ASCII-lowercased form of `original` so their byte
/// offsets line up.
fn split_last<'a>(lowered: &str, original: &'a str, sep: &str) -> Option<(&'a str, &'a str)> {
let at = lowered.rfind(sep)?;
Some((&original[..at], &original[at + sep.len()..]))
}
/// Turn the trailing day word of `text` into a concrete date, returning the
/// remaining text as the summary.
fn split_day(text: &str, reference_date: NaiveDate) -> Result<(String, NaiveDate), NlpError> {
let trimmed = text.trim_end();
let (head, last) = match trimmed.rsplit_once(char::is_whitespace) {
Some((head, last)) => (head, last),
None => ("", trimmed),
};
let lowered = last.to_ascii_lowercase();
let date = match lowered.as_str() {
"today" => reference_date,
"tomorrow" => reference_date
.succ_opt()
.ok_or(NlpError::InstantOutOfRange)?,
other => {
if let Ok(explicit) = NaiveDate::parse_from_str(other, "%Y-%m-%d") {
explicit
} else if is_weekday(other) {
// Rebuild the phrase with any qualifier ("next", "this", "on")
// so the error quotes back what the user actually wrote.
let qualifier = head
.rsplit_once(char::is_whitespace)
.map_or(head, |(_, word)| word);
let phrase = if matches!(
qualifier.to_ascii_lowercase().as_str(),
"next" | "this" | "on" | "coming"
) {
format!("{qualifier} {last}")
} else {
last.to_string()
};
return Err(NlpError::AmbiguousWeekday(phrase));
} else {
return Err(NlpError::UnsupportedDay(last.to_string()));
}
}
};
Ok((head.trim().to_string(), date))
}
fn is_weekday(word: &str) -> bool {
matches!(
word,
"monday"
| "mon"
| "tuesday"
| "tue"
| "tues"
| "wednesday"
| "wed"
| "thursday"
| "thu"
| "thurs"
| "friday"
| "fri"
| "saturday"
| "sat"
| "sunday"
| "sun"
)
}
/// Parse `noon`, `midnight`, a 12-hour `am`/`pm` time, or a 24-hour `HH:MM`.
///
/// A bare number is rejected: "at 3" gives no way to tell 03:00 from 15:00.
fn parse_time_of_day(text: &str) -> Result<(u32, u32), NlpError> {
let raw = text.trim();
let compact: String = raw.chars().filter(|c| !c.is_whitespace()).collect();
let unsupported = || NlpError::UnsupportedTime(raw.to_string());
match compact.as_str() {
"noon" => return Ok((12, 0)),
"midnight" => return Ok((0, 0)),
_ => {}
}
if let Some(clock) = compact
.strip_suffix("am")
.map(|c| (c, false))
.or_else(|| compact.strip_suffix("pm").map(|c| (c, true)))
{
let (digits, is_pm) = clock;
let (hour, minute) = split_clock(digits, 1..=12, unsupported)?;
let hour = match (hour, is_pm) {
(12, false) => 0,
(12, true) => 12,
(h, false) => h,
(h, true) => h + 12,
};
return Ok((hour, minute));
}
// 24-hour form must carry the colon, so a bare hour never parses.
if !compact.contains(':') {
return Err(unsupported());
}
split_clock(&compact, 0..=23, unsupported)
}
/// Split `H`, `H:MM`, or `HH:MM` into an hour within `hours` and a minute.
fn split_clock(
text: &str,
hours: std::ops::RangeInclusive<u32>,
unsupported: impl Fn() -> NlpError,
) -> Result<(u32, u32), NlpError> {
let (hour_text, minute_text) = match text.split_once(':') {
Some((h, m)) => (h, m),
None => (text, "0"),
};
if hour_text.is_empty() || !hour_text.chars().all(|c| c.is_ascii_digit()) {
return Err(unsupported());
}
// A minute field is always exactly two digits; "3:5" is a typo, not 03:05.
if text.contains(':')
&& (minute_text.len() != 2 || !minute_text.bytes().all(|b| b.is_ascii_digit()))
{
return Err(unsupported());
}
let hour: u32 = hour_text.parse().map_err(|_| unsupported())?;
let minute: u32 = minute_text.parse().map_err(|_| unsupported())?;
if !hours.contains(&hour) || minute > 59 {
return Err(unsupported());
}
Ok((hour, minute))
}
/// Parse `<n> minutes` / `<n> hours` into a minute count.
fn parse_duration(text: &str) -> Result<u32, NlpError> {
let raw = text.trim();
let unsupported = || NlpError::UnsupportedDuration(raw.to_string());
let mut parts = raw.split_whitespace();
let (Some(count_text), Some(unit), None) = (parts.next(), parts.next(), parts.next()) else {
return Err(unsupported());
};
let count: u32 = count_text.parse().map_err(|_| unsupported())?;
let minutes = match unit {
"minute" | "minutes" | "min" | "mins" => count,
"hour" | "hours" | "hr" | "hrs" => count.checked_mul(60).ok_or_else(unsupported)?,
_ => return Err(unsupported()),
};
if minutes == 0 || minutes > MAX_DURATION_MINUTES {
return Err(unsupported());
}
Ok(minutes)
}
/// Resolve a wall-clock time in `zone`, reporting both DST anomalies honestly.
fn resolve_local(
zone: Tz,
date: NaiveDate,
hour: u32,
minute: u32,
) -> Result<chrono::DateTime<Tz>, NlpError> {
let local = date
.and_hms_opt(hour, minute, 0)
.ok_or(NlpError::InstantOutOfRange)?;
match zone.from_local_datetime(&local) {
LocalResult::Single(instant) => Ok(instant),
LocalResult::None => Err(NlpError::NonexistentLocalTime {
date: date.to_string(),
time: format!("{hour:02}:{minute:02}"),
zone: zone.name().to_string(),
}),
LocalResult::Ambiguous(_, _) => Err(NlpError::AmbiguousLocalTime {
date: date.to_string(),
time: format!("{hour:02}:{minute:02}"),
zone: zone.name().to_string(),
}),
}
}
#[cfg(test)]
mod tests {
use super::*;
use chrono_tz::{America::New_York, Europe::London};
// 2026-03-05 10:00 EST — a plain winter weekday, well clear of any DST edge.
const REF_NY: i64 = 1_772_722_800;
// 2026-03-07 12:00 EST — the day before US DST begins (2026-03-08 02:00).
const REF_NY_BEFORE_SPRING_FORWARD: i64 = 1_772_902_800;
// 2026-10-31 12:00 EDT — the day before US DST ends (2026-11-01 02:00).
const REF_NY_BEFORE_FALL_BACK: i64 = 1_793_462_400;
// 2026-03-05 10:00 GMT — same wall clock as REF_NY, a different instant.
const REF_LONDON: i64 = 1_772_704_800;
fn parse(input: &str, reference: i64, zone: Tz) -> ParsedSchedulingIntent {
NaturalLanguageScheduler::parse(input, reference, zone).unwrap()
}
fn err(input: &str, reference: i64, zone: Tz) -> NlpError {
NaturalLanguageScheduler::parse(input, reference, zone).unwrap_err()
}
// --- supported grammar: exact instants -------------------------------
#[test]
fn parse_valid_nlp_string_creates_event() {
let input = "Coffee with Bob next Tuesday at 2pm for 45 minutes";
let ref_ts = 1700000000;
let event = NaturalLanguageScheduler::parse(input, ref_ts).unwrap();
assert_eq!(event.summary, "Coffee with Bob");
assert_eq!(event.end_time - event.start_time, 45 * 60);
fn tomorrow_at_a_pm_time_resolves_to_the_exact_instant() {
let intent = parse(
"Coffee with Bob tomorrow at 2pm for 45 minutes",
REF_NY,
New_York,
);
assert_eq!(intent.summary, "Coffee with Bob");
// 2026-03-06 14:00 EST.
assert_eq!(intent.start_time, 1_772_823_600);
assert_eq!(intent.end_time, 1_772_823_600 + 45 * 60);
assert_eq!(intent.duration_minutes, 45);
}
#[test]
fn parse_empty_query_returns_error() {
let err = NaturalLanguageScheduler::parse(" ", 1700000000).unwrap_err();
assert_eq!(err, NlpError::EmptyQuery);
fn noon_and_midnight_resolve_to_the_exact_instant() {
let noon = parse("Lunch tomorrow at noon for 1 hour", REF_NY, New_York);
assert_eq!(noon.start_time, 1_772_816_400); // 2026-03-06 12:00 EST
assert_eq!(noon.duration_minutes, 60);
let midnight = parse("Deploy tomorrow at midnight for 30 mins", REF_NY, New_York);
assert_eq!(midnight.start_time, 1_772_773_200); // 2026-03-06 00:00 EST
}
#[test]
fn today_resolves_against_the_reference_date_not_the_wall_clock() {
let intent = parse("Retro today at 9:30am for 30 minutes", REF_NY, New_York);
assert_eq!(intent.summary, "Retro");
// 2026-03-05 09:30 EST — before the reference instant, and still exact.
assert_eq!(intent.start_time, 1_772_721_000);
}
#[test]
fn an_explicit_iso_date_resolves_to_the_exact_instant() {
let intent = parse(
"Standup 2026-03-08 at 01:30 for 120 minutes",
REF_NY,
New_York,
);
assert_eq!(intent.summary, "Standup");
assert_eq!(intent.start_time, 1_772_951_400); // 2026-03-08 01:30 EST
}
#[test]
fn separators_are_matched_from_the_right_so_summaries_may_contain_them() {
let intent = parse(
"Prep for review at HQ tomorrow at 2pm for 45 minutes",
REF_NY,
New_York,
);
assert_eq!(intent.summary, "Prep for review at HQ");
assert_eq!(intent.start_time, 1_772_823_600);
}
#[test]
fn every_documented_duration_unit_is_accepted() {
for (phrase, expected) in [
("15 minutes", 15),
("15 minute", 15),
("15 mins", 15),
("15 min", 15),
("2 hours", 120),
("1 hour", 60),
("2 hrs", 120),
("1 hr", 60),
] {
let input = format!("Sync tomorrow at 2pm for {phrase}");
assert_eq!(
parse(&input, REF_NY, New_York).duration_minutes,
expected,
"{phrase}"
);
}
}
#[test]
fn every_documented_time_form_is_accepted() {
for (phrase, expected_hour, expected_minute) in [
("3pm", 15, 0),
("3:15pm", 15, 15),
("9am", 9, 0),
("9:05am", 9, 5),
("12am", 0, 0),
("12pm", 12, 0),
("00:00", 0, 0),
("23:59", 23, 59),
("noon", 12, 0),
("midnight", 0, 0),
] {
let input = format!("Sync tomorrow at {phrase} for 30 minutes");
let intent = parse(&input, REF_NY, New_York);
let expected = New_York
.with_ymd_and_hms(2026, 3, 6, expected_hour, expected_minute, 0)
.unwrap()
.timestamp();
assert_eq!(intent.start_time, expected, "{phrase}");
}
}
// --- time zones -------------------------------------------------------
#[test]
fn the_same_phrase_in_a_different_zone_is_a_different_instant() {
let input = "Sync tomorrow at 2pm for 30 minutes";
let ny = parse(input, REF_NY, New_York);
let london = parse(input, REF_LONDON, London);
assert_eq!(ny.start_time, 1_772_823_600); // 2026-03-06 14:00 EST
assert_eq!(london.start_time, 1_772_805_600); // 2026-03-06 14:00 GMT
// Same wall clock, five hours apart — the zone is load-bearing, so a
// parser that ignored it would be wrong for one of these callers.
assert_eq!(ny.start_time - london.start_time, 5 * 3600);
}
#[test]
fn a_time_inside_the_spring_forward_gap_is_refused() {
let error = err(
"Sync tomorrow at 2:30am for 30 minutes",
REF_NY_BEFORE_SPRING_FORWARD,
New_York,
);
assert_eq!(
error,
NlpError::NonexistentLocalTime {
date: "2026-03-08".to_string(),
time: "02:30".to_string(),
zone: "America/New_York".to_string(),
}
);
}
#[test]
fn a_time_just_after_the_spring_forward_gap_resolves_exactly() {
let intent = parse(
"Sync tomorrow at 3am for 30 minutes",
REF_NY_BEFORE_SPRING_FORWARD,
New_York,
);
// 2026-03-08 03:00 EDT — one hour of real time after 01:00 EST, not two.
assert_eq!(intent.start_time, 1_772_953_200);
}
#[test]
fn a_time_inside_the_fall_back_overlap_is_refused() {
let error = err(
"Sync tomorrow at 1:30am for 30 minutes",
REF_NY_BEFORE_FALL_BACK,
New_York,
);
assert_eq!(
error,
NlpError::AmbiguousLocalTime {
date: "2026-11-01".to_string(),
time: "01:30".to_string(),
zone: "America/New_York".to_string(),
}
);
}
#[test]
fn a_duration_spanning_a_dst_transition_keeps_its_real_length() {
let intent = parse(
"Standup 2026-03-08 at 01:30 for 120 minutes",
REF_NY,
New_York,
);
// Elapsed time is exactly two hours...
assert_eq!(intent.end_time - intent.start_time, 120 * 60);
// ...but the clocks jumped, so the wall-clock end is 04:30, not 03:30.
let end = New_York.timestamp_opt(intent.end_time, 0).unwrap();
assert_eq!(
end.format("%Y-%m-%d %H:%M %Z").to_string(),
"2026-03-08 04:30 EDT"
);
}
// --- refusals ---------------------------------------------------------
#[test]
fn empty_and_blank_queries_are_refused() {
assert_eq!(err("", REF_NY, New_York), NlpError::EmptyQuery);
assert_eq!(err(" ", REF_NY, New_York), NlpError::EmptyQuery);
}
#[test]
fn weekday_phrasings_are_refused_as_ambiguous_rather_than_guessed() {
for (input, quoted) in [
("Coffee next Tuesday at 2pm for 45 minutes", "next Tuesday"),
("Coffee this Friday at 2pm for 45 minutes", "this Friday"),
("Coffee on Monday at 2pm for 45 minutes", "on Monday"),
("Coffee Tuesday at 2pm for 45 minutes", "Tuesday"),
] {
assert_eq!(
err(input, REF_NY, New_York),
NlpError::AmbiguousWeekday(quoted.to_string()),
"{input}"
);
}
}
#[test]
fn a_missing_time_or_duration_is_refused_rather_than_defaulted() {
assert_eq!(
err("Coffee tomorrow at 2pm", REF_NY, New_York),
NlpError::MissingDuration
);
assert_eq!(
err("Coffee tomorrow for 45 minutes", REF_NY, New_York),
NlpError::MissingTime
);
assert_eq!(
err("Coffee with Bob", REF_NY, New_York),
NlpError::MissingDuration
);
}
#[test]
fn a_missing_summary_is_refused() {
assert_eq!(
err("tomorrow at 2pm for 45 minutes", REF_NY, New_York),
NlpError::MissingSummary
);
}
#[test]
fn unsupported_day_phrasings_are_refused() {
for phrase in [
"next week",
"in 3 days",
"the 5th",
"eod",
"yesterday",
"fortnight",
] {
let input = format!("Sync {phrase} at 2pm for 30 minutes");
let error = err(&input, REF_NY, New_York);
assert!(
matches!(error, NlpError::UnsupportedDay(_)),
"{phrase} produced {error:?}"
);
}
}
#[test]
fn a_bare_hour_is_refused_because_it_cannot_be_disambiguated() {
for phrase in [
"3",
"15",
"half past three",
"teatime",
"3 o'clock",
"25:00",
"13pm",
] {
let input = format!("Sync tomorrow at {phrase} for 30 minutes");
let error = err(&input, REF_NY, New_York);
assert!(
matches!(error, NlpError::UnsupportedTime(_)),
"{phrase} produced {error:?}"
);
}
}
#[test]
fn malformed_minute_fields_are_refused() {
for phrase in ["3:5pm", "3:60pm", "3:", ":30", "3:015"] {
let input = format!("Sync tomorrow at {phrase} for 30 minutes");
let error = err(&input, REF_NY, New_York);
assert!(
matches!(error, NlpError::UnsupportedTime(_)),
"{phrase} produced {error:?}"
);
}
}
#[test]
fn unsupported_or_out_of_range_durations_are_refused() {
for phrase in [
"a while",
"0 minutes",
"1.5 hours",
"25 hours",
"1441 minutes",
"30",
"half an hour",
"2 days",
] {
let input = format!("Sync tomorrow at 2pm for {phrase}");
let error = err(&input, REF_NY, New_York);
assert!(
matches!(error, NlpError::UnsupportedDuration(_)),
"{phrase} produced {error:?}"
);
}
}
#[test]
fn an_invalid_calendar_date_is_refused() {
let error = err("Sync 2026-02-30 at 2pm for 30 minutes", REF_NY, New_York);
assert!(matches!(error, NlpError::UnsupportedDay(_)), "{error:?}");
}
#[test]
fn the_boundary_duration_of_exactly_24_hours_is_accepted() {
let intent = parse("Hackathon tomorrow at noon for 24 hours", REF_NY, New_York);
assert_eq!(intent.duration_minutes, MAX_DURATION_MINUTES);
assert_eq!(intent.end_time - intent.start_time, 24 * 3600);
}
}
crates/nuncio-cal/src/parser.rs
+6
View File
@@ -22,6 +22,12 @@ pub enum CalendarError {
#[error("unresolvable TZID timezone reference: {0}")]
UnresolvableTimezone(String),
/// The server answered, but its body is not a WebDAV `multistatus` document this client
/// can traverse. Distinct from an empty result: a response that cannot be read must never
/// be reported as a calendar with no events in it.
#[error("malformed CalDAV multistatus response: {0}")]
MalformedResponse(String),
/// CalDAV network/transport-layer failure (connection, TLS, or an unexpected HTTP status)
/// distinct from a payload that connected fine but failed to parse.
#[error("CalDAV transport failure: {0}")]
crates/nuncio-cal/src/test_tracing.rs
+201
View File
@@ -0,0 +1,201 @@
//! Shared in-test `tracing` capture helper.
//!
//! Records every span (with its fields) and every event (with its level,
//! fields, and message) into an in-memory buffer, so tests can assert on what
//! the code logged -- including the negative assertion that a secret never
//! appears in any recorded field. Fully offline: no I/O.
//!
//! The subscriber is installed **once, globally**, and events are routed to a
//! per-thread recorder. That split is deliberate and load-bearing: a
//! thread-local subscriber cannot work here, because `tracing` caches each
//! callsite's `Interest` process-wide on first use. See
//! [`install_global_capture`].
#![cfg(test)]
use std::cell::RefCell;
use std::collections::HashMap;
use std::sync::{Arc, Mutex, OnceLock};
use tracing::field::{Field, Visit};
use tracing::span::Attributes;
use tracing::{Event, Id, Level, Subscriber};
use tracing_subscriber::layer::Context as LayerContext;
use tracing_subscriber::prelude::*;
use tracing_subscriber::registry::LookupSpan;
/// A captured span: its name and the string-rendered fields it opened with.
#[derive(Debug, Clone)]
pub(crate) struct RecordedSpan {
#[allow(dead_code)]
pub name: String,
pub fields: HashMap<String, String>,
}
/// A captured event: its level and its string-rendered fields (including the
/// `message`).
#[derive(Debug, Clone)]
pub(crate) struct RecordedEvent {
pub level: Level,
pub fields: HashMap<String, String>,
}
impl RecordedEvent {
/// The event's `message` field, or the empty string if it had none.
pub fn message(&self) -> &str {
self.fields.get("message").map(String::as_str).unwrap_or("")
}
}
/// In-memory sink of everything the subscriber captured during a test.
#[derive(Debug, Default)]
pub(crate) struct Recorder {
spans: Mutex<Vec<RecordedSpan>>,
events: Mutex<Vec<RecordedEvent>>,
}
impl Recorder {
/// A snapshot of every captured span.
pub fn spans(&self) -> Vec<RecordedSpan> {
self.spans.lock().unwrap_or_else(|e| e.into_inner()).clone()
}
/// A snapshot of every captured event.
pub fn events(&self) -> Vec<RecordedEvent> {
self.events
.lock()
.unwrap_or_else(|e| e.into_inner())
.clone()
}
/// Every field value recorded across all spans and events, so a test can
/// assert that a secret never appears anywhere in the telemetry.
pub fn all_field_values(&self) -> Vec<String> {
let mut values = Vec::new();
for span in self.spans().iter() {
values.extend(span.fields.values().cloned());
}
for event in self.events().iter() {
values.extend(event.fields.values().cloned());
}
values
}
}
struct FieldVisitor(HashMap<String, String>);
impl Visit for FieldVisitor {
fn record_str(&mut self, field: &Field, value: &str) {
self.0.insert(field.name().to_string(), value.to_string());
}
fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
self.0
.entry(field.name().to_string())
.or_insert_with(|| format!("{value:?}"));
}
}
thread_local! {
/// The recorder collecting telemetry for the test running on THIS thread,
/// or `None` on any thread not currently inside [`with_recorder`].
///
/// Routing per thread is what keeps the single global subscriber from
/// letting concurrent tests read each other's telemetry.
static ACTIVE_RECORDER: RefCell<Option<Arc<Recorder>>> = const { RefCell::new(None) };
}
/// Run `f` against the recorder installed for this thread, if any.
fn with_active<R>(f: impl FnOnce(&Recorder) -> R) -> Option<R> {
ACTIVE_RECORDER.with(|slot| slot.borrow().as_ref().map(|rec| f(rec)))
}
struct CaptureLayer;
impl<S> tracing_subscriber::Layer<S> for CaptureLayer
where
S: Subscriber + for<'a> LookupSpan<'a>,
{
fn on_new_span(&self, attrs: &Attributes<'_>, _id: &Id, _ctx: LayerContext<'_, S>) {
with_active(|rec| {
let mut visitor = FieldVisitor(HashMap::new());
attrs.record(&mut visitor);
rec.spans
.lock()
.unwrap_or_else(|e| e.into_inner())
.push(RecordedSpan {
name: attrs.metadata().name().to_string(),
fields: visitor.0,
});
});
}
fn on_event(&self, event: &Event<'_>, _ctx: LayerContext<'_, S>) {
with_active(|rec| {
let mut visitor = FieldVisitor(HashMap::new());
event.record(&mut visitor);
rec.events
.lock()
.unwrap_or_else(|e| e.into_inner())
.push(RecordedEvent {
level: *event.metadata().level(),
fields: visitor.0,
});
});
}
}
/// Install the capturing subscriber as the process-wide default, exactly once.
///
/// It MUST be global rather than thread-local, and this is the whole reason the
/// helper is shaped this way. `tracing` resolves each callsite's `Interest`
/// **once for the process** and caches it; a callsite first executed on a thread
/// with no subscriber caches `Interest::never()` and is then skipped forever --
/// before any later thread-local subscriber is ever consulted. Under a parallel
/// test harness, whichever thread reaches a callsite first therefore decides
/// whether any other test can ever capture it, which made every log-capture
/// assertion in this crate a coin flip.
///
/// A single global subscriber removes the race: every callsite registers against
/// a real subscriber, so interest is always enabled, and per-thread routing
/// (`ACTIVE_RECORDER`) decides what is actually recorded.
fn install_global_capture() {
static INSTALLED: OnceLock<()> = OnceLock::new();
INSTALLED.get_or_init(|| {
let subscriber = tracing_subscriber::registry().with(CaptureLayer);
// A global default may already be set by another component of the same
// test binary; capturing still works through the thread-local routing
// above, so a lost race here is not fatal.
let _ = tracing::subscriber::set_global_default(subscriber);
// Callsites executed before this point resolved their interest against
// the no-op dispatcher and cached "never". Recompute so they can be
// captured from here on. Done once, at install -- not per test.
tracing::callsite::rebuild_interest_cache();
});
}
/// Resets this thread's recorder slot on drop, so a panicking test cannot leak
/// its recorder into whatever the harness runs next on the same thread.
struct RecorderGuard;
impl Drop for RecorderGuard {
fn drop(&mut self) {
ACTIVE_RECORDER.with(|slot| *slot.borrow_mut() = None);
}
}
/// Runs `f` with telemetry from THIS thread captured, returning the recorder
/// alongside `f`'s result.
///
/// Only this thread's telemetry is recorded, so any async work driven inside
/// `f` must run on the calling thread (e.g. a current-thread runtime's
/// `block_on`) to be captured -- unchanged from the previous thread-local
/// implementation. What changed is that the subscriber itself is now global;
/// see [`install_global_capture`] for why that is load-bearing.
pub(crate) fn with_recorder<T>(f: impl FnOnce() -> T) -> (Arc<Recorder>, T) {
install_global_capture();
let recorder = Arc::new(Recorder::default());
ACTIVE_RECORDER.with(|slot| *slot.borrow_mut() = Some(recorder.clone()));
let _guard = RecorderGuard;
let value = f();
(recorder, value)
}
crates/nuncio-cal/tests/wiremock_caldav_test.rs
+184 -2
View File
@@ -3,6 +3,7 @@
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use nuncio_cal::{CalDavAccountConfig, CalDavClient, CalendarBackend, CalendarError};
use nuncio_core::model::CalendarEvent;
use wiremock::matchers::{method, path};
use wiremock::{Mock, MockServer, ResponseTemplate};
@@ -60,7 +61,7 @@ async fn wiremock_caldav_report_fetches_and_resolves_tzid_events() {
account_id: "acct-wm-1".to_string(),
caldav_url: format!("{}/calendars/work/", mock_server.uri()),
username: "james.maes".to_string(),
auth_token: "wiremock-app-token".to_string(),
auth_token: "wiremock-app-token".to_string().into(),
};
let client = CalDavClient::new(config);
@@ -106,7 +107,7 @@ async fn wiremock_caldav_report_server_error_surfaces_as_transport_failure() {
account_id: "acct-wm-2".to_string(),
caldav_url: format!("{}/calendars/broken/", mock_server.uri()),
username: "james.maes".to_string(),
auth_token: "wiremock-app-token".to_string(),
auth_token: "wiremock-app-token".to_string().into(),
};
let client = CalDavClient::new(config);
@@ -117,3 +118,184 @@ async fn wiremock_caldav_report_server_error_surfaces_as_transport_failure() {
assert!(matches!(err, CalendarError::TransportFailed(_)));
}
/// The same two events as [`multistatus_body`], serialized the way a server that binds the DAV
/// and CalDAV namespaces to different prefixes would write them. A substring-matching parser
/// reads zero events out of this and calls it success; a namespace-aware one reads two.
fn multistatus_body_with_uppercase_prefixes() -> String {
multistatus_body()
.replace("<d:", "<D:")
.replace("</d:", "</D:")
.replace(r#"xmlns:d="DAV:""#, r#"xmlns:D="DAV:""#)
.replace("<c:", "<CAL:")
.replace("</c:", "</CAL:")
.replace(
r#"xmlns:c="urn:ietf:params:xml:ns:caldav""#,
r#"xmlns:CAL="urn:ietf:params:xml:ns:caldav""#,
)
}
/// The same two events again, this time with `DAV:` as the document's default namespace, so no
/// DAV element carries a prefix at all.
fn multistatus_body_with_default_namespace() -> String {
multistatus_body()
.replace("<d:", "<")
.replace("</d:", "</")
.replace(r#"xmlns:d="DAV:""#, r#"xmlns="DAV:""#)
}
/// Serve `body` from a mock CalDAV endpoint and run one real `REPORT` round trip against it.
async fn fetch_against_body(
body: String,
account_id: &str,
) -> Result<Vec<CalendarEvent>, CalendarError> {
let mock_server = MockServer::start().await;
Mock::given(method("REPORT"))
.and(path("/calendars/work/"))
.respond_with(ResponseTemplate::new(207).set_body_string(body))
.mount(&mock_server)
.await;
let config = CalDavAccountConfig {
account_id: account_id.to_string(),
caldav_url: format!("{}/calendars/work/", mock_server.uri()),
username: "james.maes".to_string(),
auth_token: "wiremock-app-token".to_string().into(),
};
CalDavClient::new(config)
.fetch_remote_events("cal-work", 1_700_000_000, 1_800_000_000)
.await
}
#[tokio::test]
async fn wiremock_caldav_report_reads_the_same_document_under_any_namespace_prefix() {
for (label, body) in [
(
"uppercase prefixes",
multistatus_body_with_uppercase_prefixes(),
),
(
"default namespace",
multistatus_body_with_default_namespace(),
),
] {
let events = fetch_against_body(body, "acct-wm-ns")
.await
.unwrap_or_else(|e| panic!("{label} fetches: {e}"));
assert_eq!(events.len(), 2, "{label}");
assert!(
events
.iter()
.any(|e| e.summary == "Wiremock CalDAV Standup"),
"{label}"
);
}
}
#[tokio::test]
async fn wiremock_caldav_report_ignores_a_property_that_merely_contains_the_searched_name() {
let body = r#"<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav">
<d:response>
<d:href>/calendars/work/lookalike.ics</d:href>
<d:propstat>
<d:prop>
<c:calendar-data-summary>BEGIN:VCALENDAR
BEGIN:VEVENT
SUMMARY:Decoy
END:VEVENT
END:VCALENDAR</c:calendar-data-summary>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
<d:response>
<d:href>/calendars/work/real.ics</d:href>
<d:propstat>
<d:prop>
<c:calendar-data>BEGIN:VCALENDAR
VERSION:2.0
BEGIN:VEVENT
SUMMARY:Wiremock UTC Review
DTSTART:20240301T120000Z
DTEND:20240301T130000Z
END:VEVENT
END:VCALENDAR</c:calendar-data>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
</d:multistatus>"#;
let events = fetch_against_body(body.to_string(), "acct-wm-lookalike")
.await
.expect("fetch succeeds");
assert_eq!(events.len(), 1, "the decoy element is not calendar-data");
assert_eq!(events[0].summary, "Wiremock UTC Review");
}
#[tokio::test]
async fn wiremock_caldav_report_treats_a_404_propstat_as_no_value() {
let body = r#"<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav">
<d:response>
<d:href>/calendars/work/gone.ics</d:href>
<d:propstat>
<d:prop>
<c:calendar-data>BEGIN:VCALENDAR
VERSION:2.0
BEGIN:VEVENT
SUMMARY:Should Not Appear
DTSTART:20240301T120000Z
DTEND:20240301T130000Z
END:VEVENT
END:VCALENDAR</c:calendar-data>
</d:prop>
<d:status>HTTP/1.1 404 Not Found</d:status>
</d:propstat>
</d:response>
<d:response>
<d:href>/calendars/work/real.ics</d:href>
<d:propstat>
<d:prop>
<c:calendar-data>BEGIN:VCALENDAR
VERSION:2.0
BEGIN:VEVENT
SUMMARY:Wiremock UTC Review
DTSTART:20240301T120000Z
DTEND:20240301T130000Z
END:VEVENT
END:VCALENDAR</c:calendar-data>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
</d:multistatus>"#;
let events = fetch_against_body(body.to_string(), "acct-wm-404")
.await
.expect("fetch succeeds");
assert_eq!(events.len(), 1, "a 404 propstat carries no value");
assert_eq!(events[0].summary, "Wiremock UTC Review");
}
#[tokio::test]
async fn wiremock_caldav_report_malformed_body_surfaces_as_an_error_not_an_empty_calendar() {
// Well-formed HTTP, unreadable body: the `<d:response>` element is never closed.
let truncated = r#"<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:"><d:response><d:href>/calendars/work/a.ics</d:href>"#;
let err = fetch_against_body(truncated.to_string(), "acct-wm-malformed")
.await
.expect_err("a body this client cannot read must not look like an empty calendar");
assert!(matches!(err, CalendarError::MalformedResponse(_)));
// A 200 response that is not a multistatus at all (an intercepting captive portal, say).
let not_dav = "<html><body>Sign in to continue</body></html>";
let err = fetch_against_body(not_dav.to_string(), "acct-wm-html")
.await
.expect_err("a non-multistatus body must not look like an empty calendar");
assert!(matches!(err, CalendarError::MalformedResponse(_)));
}
crates/nuncio-cli/Cargo.toml
+1
View File
@@ -25,6 +25,7 @@ serde.workspace = true
serde_json.workspace = true
thiserror.workspace = true
tracing.workspace = true
tracing-subscriber.workspace = true
rpassword.workspace = true
chrono.workspace = true
hex.workspace = true
crates/nuncio-cli/src/args.rs
+116 -23
View File
@@ -70,14 +70,18 @@ pub struct Cli {
#[arg(long, global = true, help = "Target a specific configured account ID")]
pub account: Option<String>,
/// Enable verbose log output to stderr.
/// Increase log verbosity on stderr. Repeatable: `-v` = info, `-vv` =
/// debug, `-vvv` = trace. Omitted entirely, the CLI logs only warnings.
/// `NUNCIO_LOG`/`RUST_LOG`, if set, override this (see
/// [`crate::logging`]).
#[arg(
short,
long,
global = true,
help = "Enable detailed verbose execution logs on stderr"
action = clap::ArgAction::Count,
help = "Increase log verbosity on stderr (-v info, -vv debug, -vvv trace)"
)]
pub verbose: bool,
pub verbose: u8,
/// Subcommand resource noun.
#[command(subcommand)]
@@ -202,11 +206,21 @@ pub enum UpdateSubcommand {
pub enum AccountSubcommand {
/// List all configured mail and calendar accounts.
List,
/// Add and configure a new mail account securely.
/// Add and configure a new mail or CalDAV account securely.
Add {
/// Email address or account username.
#[arg(short, long, help = "Email address or account username")]
email: String,
/// Account protocol: imap-smtp (default), jmap, caldav, or carddav. A
/// caldav or carddav account requires --collection-url and ignores the
/// IMAP/SMTP flags.
#[arg(long, default_value = "imap-smtp", help = "Account protocol")]
protocol: String,
/// Fully-qualified DAV collection URL -- a CalDAV calendar collection
/// or a CardDAV address book (required for a caldav or carddav
/// account; ignored for mail accounts).
#[arg(long, help = "DAV collection URL (caldav/carddav only)")]
collection_url: Option<String>,
/// IMAP server hostname.
#[arg(long, default_value = "mail.kof22.com", help = "IMAP server hostname")]
imap_host: String,
@@ -238,6 +252,26 @@ pub enum AccountSubcommand {
#[arg(short, long, help = "Unique account identifier")]
id: String,
},
/// Turn the filter engine on or off for one account.
Filters {
/// Unique account identifier.
#[arg(short, long, help = "Unique account identifier")]
id: String,
/// Run filters against this account's incoming mail.
#[arg(
long,
help = "Enable filters for this account",
conflicts_with = "disable"
)]
enable: bool,
/// Stop running filters against this account's incoming mail.
#[arg(
long,
help = "Disable filters for this account",
conflicts_with = "enable"
)]
disable: bool,
},
/// Edit an existing account profile configuration.
Edit {
/// Unique account identifier.
@@ -331,6 +365,66 @@ pub enum MailSubcommand {
#[arg(long, help = "Mark the message as unread", conflicts_with = "read")]
unread: bool,
},
/// Move one occupancy of a message into another folder.
Move {
/// Unique message identifier.
#[arg(short, long, help = "Unique message identifier")]
id: String,
/// Destination folder identifier.
#[arg(short, long, help = "Destination folder identifier")]
to: String,
/// Which mailbox occupancy to act on, as
/// `account/folder/uidvalidity/uid`. Required when the message occupies
/// more than one mailbox; `mail read` lists them.
#[arg(long, help = "Occupancy as account/folder/uidvalidity/uid")]
placement: Option<String>,
},
/// Delete one occupancy of a message. Other occupancies survive.
Delete {
/// Unique message identifier.
#[arg(short, long, help = "Unique message identifier")]
id: String,
/// Which mailbox occupancy to act on, as
/// `account/folder/uidvalidity/uid`.
#[arg(long, help = "Occupancy as account/folder/uidvalidity/uid")]
placement: Option<String>,
},
/// Flag one occupancy of a message.
Flag {
/// Unique message identifier.
#[arg(short, long, help = "Unique message identifier")]
id: String,
/// Which mailbox occupancy to act on, as
/// `account/folder/uidvalidity/uid`.
#[arg(long, help = "Occupancy as account/folder/uidvalidity/uid")]
placement: Option<String>,
},
/// Remove the flag from one occupancy of a message.
Unflag {
/// Unique message identifier.
#[arg(short, long, help = "Unique message identifier")]
id: String,
/// Which mailbox occupancy to act on, as
/// `account/folder/uidvalidity/uid`.
#[arg(long, help = "Occupancy as account/folder/uidvalidity/uid")]
placement: Option<String>,
},
/// List mutations the server refused because another client changed the
/// message first.
Conflicts {
/// Also list conflicts that have already been resolved.
#[arg(long, help = "Include already-resolved conflicts")]
all: bool,
},
/// Mark a conflict handled, recording how it was resolved.
Resolve {
/// Conflict identifier, as reported by `mail conflicts`.
#[arg(short, long, help = "Conflict identifier")]
id: String,
/// Note recording how the conflict was resolved.
#[arg(short, long, help = "How the conflict was resolved")]
note: String,
},
/// Export mailbox messages to a portable file format: MBOX, an EML
/// zip archive, JSON, or JSON Lines.
Export {
@@ -441,14 +535,13 @@ pub enum SystemSubcommand {
#[derive(Subcommand, Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum AuditSubcommand {
/// List persisted WORM audit ledger records, sequence ascending.
/// List persisted WORM audit ledger records, sequence ascending. Follows
/// keyset pagination to the end, fetching `page_size` records per request.
List {
/// Max records to fetch (0 = server default).
#[arg(short, long, default_value_t = 50, help = "Max records to fetch")]
limit: u32,
/// Pagination offset.
#[arg(short, long, default_value_t = 0, help = "Pagination offset")]
offset: u32,
/// Records fetched per page (0 = server default). All pages are
/// followed via the keyset cursor and printed together.
#[arg(short, long, default_value_t = 50, help = "Records fetched per page")]
page_size: u32,
},
/// Verify the WORM audit ledger's cryptographic HMAC hash-chain
/// integrity.
@@ -569,6 +662,8 @@ mod tests {
Commands::Account {
action: AccountSubcommand::Add {
email: "james.maes@kof22.com".to_string(),
protocol: "imap-smtp".to_string(),
collection_url: None,
imap_host: "mail.kof22.com".to_string(),
imap_port: 993,
smtp_host: "mail.kof22.com".to_string(),
@@ -585,7 +680,7 @@ mod tests {
fn parse_pure_noun_verb_mail_commands() {
let cli_sync = Cli::parse_from(["nuncio", "--json", "--verbose", "mail", "sync"]);
assert!(cli_sync.json);
assert!(cli_sync.verbose);
assert_eq!(cli_sync.verbose, 1);
assert_eq!(
cli_sync.command,
Commands::Mail {
@@ -593,6 +688,11 @@ mod tests {
}
);
// `-v` is repeatable and counts occurrences, mapped to a log level by
// `nuncio_cli::logging::verbosity_directive`.
let cli_vv = Cli::parse_from(["nuncio", "-vv", "mail", "sync"]);
assert_eq!(cli_vv.verbose, 2);
let cli_read = Cli::parse_from(["nuncio", "mail", "read", "--id", "msg-123"]);
assert_eq!(
cli_read.command,
@@ -706,25 +806,18 @@ mod tests {
cli_list.command,
Commands::System {
action: SystemSubcommand::Audit {
action: AuditSubcommand::List {
limit: 50,
offset: 0,
}
action: AuditSubcommand::List { page_size: 50 }
}
}
);
let cli_list_paged = Cli::parse_from([
"nuncio", "system", "audit", "list", "--limit", "10", "--offset", "5",
]);
let cli_list_paged =
Cli::parse_from(["nuncio", "system", "audit", "list", "--page-size", "10"]);
assert_eq!(
cli_list_paged.command,
Commands::System {
action: SystemSubcommand::Audit {
action: AuditSubcommand::List {
limit: 10,
offset: 5,
}
action: AuditSubcommand::List { page_size: 10 }
}
}
);
crates/nuncio-cli/src/lib.rs
+2 -1
View File
@@ -3,6 +3,7 @@
#![cfg_attr(test, allow(clippy::unwrap_used, clippy::expect_used, clippy::panic))]
pub mod args;
pub mod logging;
pub mod output;
pub mod runner;
@@ -11,4 +12,4 @@ pub use args::{
SystemSubcommand,
};
pub use output::JsonResponse;
pub use runner::HeadlessRunner;
pub use runner::{HeadlessRunner, GRPC_TOKEN_ENV};
crates/nuncio-cli/src/logging.rs
+136
View File
@@ -0,0 +1,136 @@
//! CLI-local `tracing` subscriber.
//!
//! This controls `nuncio-cli`'s OWN log output -- it is entirely separate
//! from `nunciod`'s own layered subscriber (`nunciod::logging`), which
//! governs the daemon process the CLI merely talks to over gRPC. The CLI
//! prints its normal command output on stdout regardless of this
//! configuration; this subscriber only ever writes to stderr, so scripts that
//! parse stdout (especially `--json` mode) are never polluted by log lines.
//!
//! ## Verbosity mapping
//!
//! `-v`/`--verbose` is a repeatable flag (`clap`'s `ArgAction::Count`), not a
//! boolean: `nuncio <cmd>` with none is quiet (`warn`), `-v` is `info`, `-vv`
//! is `debug`, and `-vvv` or more is `trace`. [`verbosity_directive`] is the
//! pure mapping and is unit-tested directly, without touching the process's
//! single global subscriber slot.
//!
//! ## Environment override
//!
//! `NUNCIO_LOG` (if set and non-blank) wins over `RUST_LOG`, which wins over
//! the `-v` count -- mirroring `nunciod`'s own precedence so the two
//! processes behave consistently for an operator who sets `NUNCIO_LOG`
//! globally.
use tracing_subscriber::EnvFilter;
/// Environment variable that overrides the log filter, taking precedence over
/// `RUST_LOG` and the `-v` count.
pub const LOG_FILTER_ENV_VAR: &str = "NUNCIO_LOG";
/// Maps a `-v` occurrence count to a filter directive. `0` is the quiet
/// default (`warn`); each additional `-v` steps up one level, capping at
/// `trace`.
#[must_use]
pub fn verbosity_directive(verbosity: u8) -> &'static str {
match verbosity {
0 => "warn",
1 => "info",
2 => "debug",
_ => "trace",
}
}
/// Resolves the effective filter directive with precedence `NUNCIO_LOG` >
/// `RUST_LOG` > the `-v` count (via [`verbosity_directive`]).
///
/// An env var set to an empty/whitespace-only string is treated as unset so a
/// stray `NUNCIO_LOG=` does not silently override an explicit `-v`.
#[must_use]
pub fn resolve_filter_directive(verbosity: u8) -> String {
env_directive(LOG_FILTER_ENV_VAR)
.or_else(|| env_directive("RUST_LOG"))
.unwrap_or_else(|| verbosity_directive(verbosity).to_string())
}
fn env_directive(var: &str) -> Option<String> {
match std::env::var(var) {
Ok(value) if !value.trim().is_empty() => Some(value),
_ => None,
}
}
/// Installs a stderr-only `tracing` subscriber for the CLI process, driven by
/// [`resolve_filter_directive`].
///
/// Never panics: a malformed filter directive (from a hand-edited
/// `NUNCIO_LOG`/`RUST_LOG`) degrades to the `-v`-derived default rather than
/// crashing the CLI before it can even report an error. Idempotent -- if a
/// subscriber is already installed (e.g. a test harness), this is a silent
/// no-op rather than a panic, since `nuncio-cli`'s own tests run many
/// commands in the same process.
pub fn init(verbosity: u8) {
let directive = resolve_filter_directive(verbosity);
let filter = EnvFilter::try_new(&directive)
.unwrap_or_else(|_| EnvFilter::new(verbosity_directive(verbosity)));
let _ = tracing_subscriber::fmt()
.with_env_filter(filter)
.with_writer(std::io::stderr)
.try_init();
}
#[cfg(test)]
mod tests {
use super::*;
// The process environment is global mutable state; serialize every test
// that reads or writes the log env vars so they cannot interleave.
static ENV_GUARD: std::sync::Mutex<()> = std::sync::Mutex::new(());
fn clear_env() {
std::env::remove_var(LOG_FILTER_ENV_VAR);
std::env::remove_var("RUST_LOG");
}
#[test]
fn verbosity_count_maps_to_expected_level() {
assert_eq!(verbosity_directive(0), "warn");
assert_eq!(verbosity_directive(1), "info");
assert_eq!(verbosity_directive(2), "debug");
assert_eq!(verbosity_directive(3), "trace");
// Anything beyond -vvv still caps at trace rather than erroring.
assert_eq!(verbosity_directive(255), "trace");
}
#[test]
fn nuncio_log_beats_rust_log_beats_verbosity_count() {
let _guard = ENV_GUARD.lock().unwrap_or_else(|e| e.into_inner());
clear_env();
// No env override: falls back to the -v count mapping.
assert_eq!(resolve_filter_directive(0), "warn");
assert_eq!(resolve_filter_directive(2), "debug");
// RUST_LOG wins over the -v count.
std::env::set_var("RUST_LOG", "nuncio_cli=warn");
assert_eq!(resolve_filter_directive(2), "nuncio_cli=warn");
// NUNCIO_LOG wins over RUST_LOG.
std::env::set_var(LOG_FILTER_ENV_VAR, "nuncio_cli=trace");
assert_eq!(resolve_filter_directive(0), "nuncio_cli=trace");
// An empty NUNCIO_LOG is treated as unset, so RUST_LOG applies again.
std::env::set_var(LOG_FILTER_ENV_VAR, " ");
assert_eq!(resolve_filter_directive(0), "nuncio_cli=warn");
clear_env();
}
#[test]
fn init_never_panics_even_when_a_subscriber_is_already_installed() {
// Calling init() twice must not panic: the second call finds the
// global subscriber slot occupied and silently no-ops.
init(1);
init(1);
}
}
crates/nuncio-cli/src/main.rs
+8 -2
View File
@@ -1,10 +1,16 @@
use clap::Parser;
use nuncio_cli::{args::Cli, AccountSubcommand, Commands, HeadlessRunner, PasswordArg};
use nuncio_cli::{args::Cli, logging, AccountSubcommand, Commands, HeadlessRunner, PasswordArg};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut cli = Cli::parse();
// Wire the CLI's own `-v`/`-vv`/`-vvv` flag (or `NUNCIO_LOG`/`RUST_LOG`)
// to a real stderr subscriber before anything else runs, so every RPC
// this process makes is followable. This is the CLI's own log stream --
// entirely separate from the daemon's file-based logging.
logging::init(cli.verbose);
// `account add`'s password is deliberately never a Clap-parsed CLI
// flag (see `PasswordArg`'s doc comment): read it here, interactively
// and without echo, immediately after parsing argv and before any
@@ -20,7 +26,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
*password = PasswordArg(entered);
}
let runner = HeadlessRunner::ephemeral().await?;
let runner = HeadlessRunner::connect();
let output_str = runner.execute_command(&cli.command, cli.json).await;
println!("{}", output_str);
crates/nuncio-cli/src/output.rs
+95 -1
View File
@@ -1,5 +1,7 @@
//! Standardized JSON machine-readable output formatters for Unix scripting.
use std::collections::BTreeMap;
use serde::Serialize;
/// Standardized JSON response envelope for CLI machine output.
@@ -13,6 +15,18 @@ pub struct JsonResponse<T: Serialize> {
/// Error message string (present if status is "error").
#[serde(skip_serializing_if = "Option::is_none")]
pub error: Option<String>,
/// The gRPC status code name (e.g. `"NotFound"`), when the error came
/// from a daemon RPC.
#[serde(skip_serializing_if = "Option::is_none")]
pub code: Option<String>,
/// The typed `nuncio.v1.ErrorReason` name (e.g. `"ACCOUNT_NOT_FOUND"`),
/// decoded from the daemon's `ErrorInfo` status details, when present.
#[serde(skip_serializing_if = "Option::is_none")]
pub reason: Option<String>,
/// Structured error context (e.g. `{"account_id": "acct-1"}`) carried in
/// the daemon's `ErrorInfo`, when present and non-empty.
#[serde(skip_serializing_if = "Option::is_none")]
pub metadata: Option<BTreeMap<String, String>>,
}
impl<T: Serialize> JsonResponse<T> {
@@ -22,17 +36,47 @@ impl<T: Serialize> JsonResponse<T> {
status: "ok",
data: Some(data),
error: None,
code: None,
reason: None,
metadata: None,
}
}
}
impl JsonResponse<()> {
/// Construct an error JSON response wrapping an error string.
/// Construct an error JSON response wrapping an error string, with no
/// typed gRPC code/reason/metadata (e.g. a purely client-side failure
/// such as an unreachable daemon).
pub fn error(message: &str) -> Self {
Self {
status: "error",
data: None,
error: Some(message.to_string()),
code: None,
reason: None,
metadata: None,
}
}
/// Construct an error JSON response carrying the gRPC code and, when the
/// daemon attached a WS-A1 `ErrorInfo`, the typed reason and metadata.
pub fn error_with_info(
message: &str,
code: Option<String>,
reason: Option<String>,
metadata: BTreeMap<String, String>,
) -> Self {
Self {
status: "error",
data: None,
error: Some(message.to_string()),
code,
reason,
metadata: if metadata.is_empty() {
None
} else {
Some(metadata)
},
}
}
}
@@ -51,6 +95,20 @@ pub fn format_json_error(message: &str) -> String {
.unwrap_or_else(|_| r#"{"status":"error","error":"JSON serialization failed"}"#.to_string())
}
/// Serialize a typed daemon RPC error (gRPC code, optional `ErrorReason`
/// name, and optional metadata) to a formatted JSON string. See
/// [`JsonResponse::error_with_info`].
pub fn format_json_error_with_info(
message: &str,
code: Option<String>,
reason: Option<String>,
metadata: BTreeMap<String, String>,
) -> String {
let response = JsonResponse::error_with_info(message, code, reason, metadata);
serde_json::to_string(&response)
.unwrap_or_else(|_| r#"{"status":"error","error":"JSON serialization failed"}"#.to_string())
}
/// Print ANSI color ASCII art splash banner for CLI startup.
pub fn print_splash_banner() {
let banner = get_splash_banner_text();
@@ -94,6 +152,42 @@ mod tests {
let err_str = format_json_error("account not found");
assert!(err_str.contains(r#""status":"error""#));
assert!(err_str.contains(r#""error":"account not found""#));
// A plain error carries no typed code/reason/metadata.
assert!(!err_str.contains("\"code\""));
assert!(!err_str.contains("\"reason\""));
assert!(!err_str.contains("\"metadata\""));
}
#[test]
fn format_json_error_with_info_carries_code_reason_and_metadata() {
let mut metadata = BTreeMap::new();
metadata.insert("account_id".to_string(), "acct-1".to_string());
let out = format_json_error_with_info(
"nunciod daemon rejected account_show: account 'acct-1' not found",
Some("NotFound".to_string()),
Some("ACCOUNT_NOT_FOUND".to_string()),
metadata,
);
assert!(out.contains(r#""status":"error""#));
assert!(out.contains(r#""code":"NotFound""#));
assert!(out.contains(r#""reason":"ACCOUNT_NOT_FOUND""#));
assert!(out.contains(r#""account_id":"acct-1""#));
}
#[test]
fn format_json_error_with_info_omits_reason_and_metadata_when_absent() {
// Falls back cleanly when the status carried no typed `ErrorInfo`:
// still structured (code present), but no fabricated reason/metadata.
let out = format_json_error_with_info(
"daemon unreachable",
Some("Unavailable".to_string()),
None,
BTreeMap::new(),
);
assert!(out.contains(r#""code":"Unavailable""#));
assert!(!out.contains("\"reason\""));
assert!(!out.contains("\"metadata\""));
}
#[test]
crates/nuncio-cli/src/runner.rs
+1934 -658
View File
File diff suppressed because it is too large. Load diff
crates/nuncio-cli/tests/cli_system_test.rs
+6 -10
View File
@@ -12,7 +12,7 @@ use std::sync::Arc;
#[tokio::test]
async fn system_test_cli_noun_verb_execution_matrix() {
let runner = HeadlessRunner::ephemeral().await.expect("runner init");
let runner = HeadlessRunner::connect();
// 1. System status is a real gRPC client of the `nunciod` daemon, so it
// is exercised separately below via `ephemeral_with` + `SecretManager::mock()`
@@ -60,9 +60,7 @@ async fn system_status_reports_honest_error_when_daemon_unreachable() {
let addr = listener.local_addr().expect("listener has local addr");
drop(listener); // free the port; nothing is listening on it now
let runner = HeadlessRunner::ephemeral_with(Arc::new(SecretManager::mock()), addr.to_string())
.await
.expect("runner init");
let runner = HeadlessRunner::connect_with(Arc::new(SecretManager::mock()), addr.to_string());
let out: String = runner
.execute_command(
@@ -92,9 +90,7 @@ async fn account_add_and_list_report_honest_errors_when_daemon_unreachable() {
let addr = listener.local_addr().expect("listener has local addr");
drop(listener); // free the port; nothing is listening on it now
let runner = HeadlessRunner::ephemeral_with(Arc::new(SecretManager::mock()), addr.to_string())
.await
.expect("runner init");
let runner = HeadlessRunner::connect_with(Arc::new(SecretManager::mock()), addr.to_string());
let list_out: String = runner
.execute_command(
@@ -116,6 +112,8 @@ async fn account_add_and_list_report_honest_errors_when_daemon_unreachable() {
&Commands::Account {
action: AccountSubcommand::Add {
email: "test@nuncio.mx".to_string(),
protocol: "imap-smtp".to_string(),
collection_url: None,
imap_host: "mail.nuncio.mx".to_string(),
imap_port: 993,
smtp_host: "mail.nuncio.mx".to_string(),
@@ -150,9 +148,7 @@ async fn mail_and_folder_report_honest_errors_when_daemon_unreachable() {
let addr = listener.local_addr().expect("listener has local addr");
drop(listener); // free the port; nothing is listening on it now
let runner = HeadlessRunner::ephemeral_with(Arc::new(SecretManager::mock()), addr.to_string())
.await
.expect("runner init");
let runner = HeadlessRunner::connect_with(Arc::new(SecretManager::mock()), addr.to_string());
let assert_honest_error = |out: String| {
let json: Value = serde_json::from_str(&out).expect("valid json");
crates/nuncio-contacts/Cargo.toml
+1 -1
View File
@@ -19,10 +19,10 @@ tracing = { workspace = true }
sqlx = { workspace = true }
chrono = { workspace = true, features = ["serde"] }
reqwest = { workspace = true }
quick-xml = { workspace = true }
regex = { workspace = true }
async-trait = { workspace = true }
[dev-dependencies]
tempfile = { workspace = true }
wiremock = { workspace = true }
tracing-subscriber = { workspace = true }
crates/nuncio-contacts/src/carddav.rs
+267 -18
View File
@@ -1,8 +1,11 @@
//! CardDAV (RFC 6352) `REPORT` client and multistatus response parser.
use async_trait::async_trait;
use nuncio_core::dav::{parse_multistatus, CARDDAV_NS};
use nuncio_core::redact::Redacted;
use serde::{Deserialize, Serialize};
use thiserror::Error;
use tracing::{debug, info, instrument, warn};
use crate::backend::ContactsBackend;
use crate::models::Contact;
@@ -15,6 +18,12 @@ pub enum CardDavError {
#[error("failed to parse vCard payload: {0}")]
ParseFailed(String),
/// The server answered, but its body is not a WebDAV `multistatus` document this client
/// can traverse. Distinct from an empty result: a response that cannot be read must never
/// be reported as an address book with no contacts in it.
#[error("malformed CardDAV multistatus response: {0}")]
MalformedResponse(String),
/// CardDAV network/transport-layer failure (connection, TLS, or an unexpected HTTP
/// status) distinct from a payload that connected fine but failed to parse.
#[error("CardDAV transport failure: {0}")]
@@ -27,7 +36,7 @@ pub enum CardDavError {
/// `carddav_url` must already resolve to a specific address book collection (e.g.
/// `https://carddav.example.com/dav/addressbooks/user/jmaes/contacts/`) -- PROPFIND-based
/// `addressbook-home-set` auto-discovery is out of scope for this client.
#[derive(Debug, Clone, Serialize, Deserialize)]
#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct CardDavAccountConfig {
/// Nuncio account identifier this address book collection belongs to.
pub account_id: String,
@@ -36,8 +45,10 @@ pub struct CardDavAccountConfig {
/// Basic-auth username (or app-specific username, per provider).
pub username: String,
/// Basic-auth secret (password or app-specific token) resolved from the OS keyring by the
/// caller -- never stored anywhere else in plaintext.
pub auth_token: String,
/// caller -- never stored anywhere else in plaintext. Wrapped in [`Redacted`] so a derived
/// `Debug`, a `Display`, or `serde` can never disclose it to a log; the raw value is
/// reachable only through an explicit `expose_secret` call.
pub auth_token: Redacted<String>,
}
/// CardDAV client protocol engine managing address book `REPORT` queries against a real server.
@@ -72,26 +83,56 @@ impl CardDavClient {
}
/// Parse a CardDAV WebDAV XML `<multistatus>` response containing embedded vCard data.
///
/// The document is traversed with a namespace-aware parser and `address-data` is matched on
/// `{urn:ietf:params:xml:ns:carddav}address-data`, so the server's choice of prefix (or of a
/// default namespace) is irrelevant and a body this client cannot read is an error rather
/// than an empty -- and indistinguishable from genuinely empty -- contact list.
pub fn parse_multistatus_response(
&self,
account_id: &str,
raw_xml: &str,
) -> Result<Vec<Contact>, CardDavError> {
let mut contacts = Vec::new();
self.parse_multistatus_response_counted(account_id, raw_xml)
.map(|(contacts, _dropped)| contacts)
}
for block in raw_xml.split("<card:address-data>") {
if let Some((vcard_data, _)) = block.split_once("</card:address-data>") {
let clean_vcard = vcard_data.trim();
if !clean_vcard.is_empty() {
let contact_id = format!("carddav-{account_id}-{}", contacts.len() + 1);
let contact =
VCardParserAdapter::parse_vcard(&contact_id, account_id, clean_vcard)?;
contacts.push(contact);
}
}
/// Same parse as [`Self::parse_multistatus_response`], additionally reporting how many
/// `<response>` elements carried no readable `address-data` so callers can log round-trip
/// counts. Each one is logged at `warn` as it is skipped -- an address book that came back
/// short must say so rather than look like an address book that is short.
///
/// An unparseable vCard is still fatal (unlike the CalDAV client, which drops individual
/// bad VEVENTs): a card that the server returned and this client could not read is an
/// error, not a contact to omit.
fn parse_multistatus_response_counted(
&self,
account_id: &str,
raw_xml: &str,
) -> Result<(Vec<Contact>, usize), CardDavError> {
let report = parse_multistatus(raw_xml, CARDDAV_NS, "address-data")
.map_err(|e| CardDavError::MalformedResponse(e.to_string()))?;
for skipped in &report.skipped {
warn!(
account_id = %account_id,
href = %skipped.href.as_deref().unwrap_or("unknown"),
reason = %skipped.reason,
"dropped CardDAV response with no readable address-data"
);
}
Ok(contacts)
let mut contacts = Vec::new();
for (index, entry) in report.values.iter().enumerate() {
let contact_id = format!("carddav-{account_id}-{}", index + 1);
contacts.push(VCardParserAdapter::parse_vcard(
&contact_id,
account_id,
&entry.value,
)?);
}
Ok((contacts, report.skipped.len()))
}
/// Issue a live CardDAV `REPORT` addressbook-query (RFC 6352 Section 8.6) against
@@ -100,6 +141,7 @@ impl CardDavClient {
/// This performs a real network request -- no canned or fabricated data is ever
/// returned. A server or network failure surfaces as [`CardDavError::TransportFailed`]; a
/// malformed vCard inside a returned response surfaces as [`CardDavError::ParseFailed`].
#[instrument(name = "carddav_sync", skip(self), fields(account_id = %account_id))]
pub async fn fetch_remote_vcards(
&self,
account_id: &str,
@@ -111,12 +153,17 @@ impl CardDavClient {
let report_method = reqwest::Method::from_bytes(b"REPORT")
.map_err(|e| CardDavError::TransportFailed(format!("invalid HTTP method: {e}")))?;
debug!(account_id = %account_id, "issuing CardDAV REPORT addressbook-query");
let response = self
.http
.request(report_method, &self.config.carddav_url)
.header("Content-Type", "application/xml; charset=utf-8")
.header("Depth", "1")
.basic_auth(&self.config.username, Some(&self.config.auth_token))
.basic_auth(
&self.config.username,
Some(self.config.auth_token.expose_secret()),
)
.body(body)
.send()
.await
@@ -136,7 +183,19 @@ impl CardDavClient {
.await
.map_err(|e| CardDavError::TransportFailed(e.to_string()))?;
self.parse_multistatus_response(account_id, &raw_xml)
// The parse fails fast on the first unparseable vCard, so a successful return here means
// every card that was fetched parsed cleanly; `dropped` therefore counts only the
// responses the server returned without readable `address-data`.
let (contacts, dropped) = self.parse_multistatus_response_counted(account_id, &raw_xml)?;
info!(
account_id = %account_id,
fetched = contacts.len() + dropped,
parsed = contacts.len(),
dropped,
"completed CardDAV sync"
);
Ok(contacts)
}
}
@@ -156,7 +215,7 @@ mod tests {
account_id: "acct-1".to_string(),
carddav_url: "https://carddav.example.com/addressbooks/contacts/".to_string(),
username: "jmaes".to_string(),
auth_token: "app-token-secret".to_string(),
auth_token: Redacted::new("app-token-secret".to_string()),
}
}
@@ -207,4 +266,194 @@ END:VCARD</card:address-data>
.expect("parse succeeds");
assert!(contacts.is_empty());
}
/// The same document, written the three ways a compliant server may write it. Before
/// namespace-aware parsing, only the middle one produced any contacts at all.
#[test]
fn parse_multistatus_response_reads_any_prefix_binding_of_the_carddav_namespace() {
let client = CardDavClient::new(test_config());
let card = "BEGIN:VCARD\nVERSION:4.0\nFN:Alice Dev\nEMAIL:alice@nuncio.mx\nEND:VCARD";
let uppercase = format!(
r#"<?xml version="1.0" encoding="utf-8"?>
<D:multistatus xmlns:D="DAV:" xmlns:CARD="urn:ietf:params:xml:ns:carddav">
<D:response><D:href>/alice.vcf</D:href><D:propstat><D:prop>
<CARD:address-data>{card}</CARD:address-data>
</D:prop><D:status>HTTP/1.1 200 OK</D:status></D:propstat></D:response>
</D:multistatus>"#
);
let default_ns = format!(
r#"<?xml version="1.0" encoding="utf-8"?>
<multistatus xmlns="DAV:" xmlns:card="urn:ietf:params:xml:ns:carddav">
<response><href>/alice.vcf</href><propstat><prop>
<card:address-data>{card}</card:address-data>
</prop><status>HTTP/1.1 200 OK</status></propstat></response>
</multistatus>"#
);
let odd_prefix = format!(
r#"<?xml version="1.0" encoding="utf-8"?>
<ns0:multistatus xmlns:ns0="DAV:" xmlns:ns1="urn:ietf:params:xml:ns:carddav">
<ns0:response><ns0:href>/alice.vcf</ns0:href><ns0:propstat><ns0:prop>
<ns1:address-data>{card}</ns1:address-data>
</ns0:prop><ns0:status>HTTP/1.1 200 OK</ns0:status></ns0:propstat></ns0:response>
</ns0:multistatus>"#
);
for (label, xml) in [
("uppercase prefixes", &uppercase),
("default namespace", &default_ns),
("generated prefixes", &odd_prefix),
] {
let contacts = client
.parse_multistatus_response("acct-1", xml)
.unwrap_or_else(|e| panic!("{label} parses: {e}"));
assert_eq!(contacts.len(), 1, "{label}");
assert_eq!(contacts[0].display_name, "Alice Dev", "{label}");
}
}
#[test]
fn parse_multistatus_response_warns_when_a_response_has_no_readable_address_data() {
use crate::test_tracing::with_recorder;
let client = CardDavClient::new(test_config());
// The first response's property merely *contains* the searched name; the second
// returns the real property under a 404 propstat, which is not a value.
let xml_response = r#"<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:" xmlns:card="urn:ietf:params:xml:ns:carddav">
<d:response>
<d:href>/lookalike.vcf</d:href>
<d:propstat><d:prop>
<card:address-data-summary>NOT THE PAYLOAD</card:address-data-summary>
</d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat>
</d:response>
<d:response>
<d:href>/gone.vcf</d:href>
<d:propstat><d:prop>
<card:address-data>BEGIN:VCARD
END:VCARD</card:address-data>
</d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat>
</d:response>
</d:multistatus>"#;
let (recorder, result) =
with_recorder(|| client.parse_multistatus_response("acct-1", xml_response));
let contacts = result.expect("an unreadable response is not fatal");
assert!(contacts.is_empty(), "neither response carries address-data");
let warnings: Vec<_> = recorder
.events()
.into_iter()
.filter(|e| e.level == tracing::Level::WARN)
.collect();
assert_eq!(warnings.len(), 2, "both omissions must be visible");
let hrefs: Vec<_> = warnings
.iter()
.filter_map(|w| w.fields.get("href").cloned())
.collect();
assert!(hrefs.contains(&"/lookalike.vcf".to_string()));
assert!(hrefs.contains(&"/gone.vcf".to_string()));
assert!(warnings
.iter()
.any(|w| w.fields.get("reason").is_some_and(|r| r.contains("404"))));
}
#[test]
fn parse_multistatus_response_rejects_a_body_it_cannot_read_instead_of_returning_empty() {
let client = CardDavClient::new(test_config());
// A malformed document: the `<d:response>` element is never closed.
let truncated = r#"<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:"><d:response><d:href>/a.vcf</d:href>"#;
assert!(matches!(
client.parse_multistatus_response("acct-1", truncated),
Err(CardDavError::MalformedResponse(_))
));
// A body that is not a multistatus at all (an intercepting proxy's error page).
let not_dav = "<html><body>502 Bad Gateway</body></html>";
assert!(matches!(
client.parse_multistatus_response("acct-1", not_dav),
Err(CardDavError::MalformedResponse(_))
));
}
#[test]
fn debug_impl_redacts_auth_token() {
let config = test_config();
let rendered = format!("{config:?}");
assert!(!rendered.contains("app-token-secret"));
assert!(rendered.contains("<redacted>"));
}
#[test]
fn fetch_remote_vcards_logs_completion_counts() {
use crate::test_tracing::with_recorder;
use wiremock::matchers::{method, path};
use wiremock::{Mock, MockServer, ResponseTemplate};
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("current-thread runtime");
let (recorder, result) = with_recorder(|| {
runtime.block_on(async {
let mock_server = MockServer::start().await;
let xml_response = r#"<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:" xmlns:card="urn:ietf:params:xml:ns:carddav">
<d:response>
<d:propstat>
<d:prop>
<card:address-data>BEGIN:VCARD
VERSION:4.0
FN:Alice Dev
EMAIL:alice@nuncio.mx
END:VCARD</card:address-data>
</d:prop>
</d:propstat>
</d:response>
</d:multistatus>"#;
Mock::given(method("REPORT"))
.and(path("/addressbooks/contacts/"))
.respond_with(ResponseTemplate::new(207).set_body_string(xml_response))
.mount(&mock_server)
.await;
let mut config = test_config();
config.carddav_url = format!("{}/addressbooks/contacts/", mock_server.uri());
let client = CardDavClient::new(config);
client.fetch_remote_vcards("acct-1").await
})
});
let contacts = result.expect("fetch succeeds against the wiremock server");
assert_eq!(contacts.len(), 1);
let completion = recorder
.events()
.into_iter()
.find(|e| e.message().contains("completed CardDAV sync"))
.expect("a completion event is logged");
assert_eq!(
completion.fields.get("fetched").map(String::as_str),
Some("1")
);
assert_eq!(
completion.fields.get("parsed").map(String::as_str),
Some("1")
);
assert_eq!(
completion.fields.get("dropped").map(String::as_str),
Some("0")
);
// vCard PII must never appear in telemetry.
for value in recorder.all_field_values() {
assert!(!value.contains("Alice Dev"));
assert!(!value.contains("alice@nuncio.mx"));
}
}
}
crates/nuncio-contacts/src/lib.rs
+2
View File
@@ -10,6 +10,8 @@ pub mod db;
pub mod mock;
pub mod models;
pub mod parser;
#[cfg(test)]
mod test_tracing;
pub use backend::ContactsBackend;
pub use carddav::{CardDavAccountConfig, CardDavClient, CardDavError};
crates/nuncio-contacts/src/test_tracing.rs
+202
View File
@@ -0,0 +1,202 @@
//! Shared in-test `tracing` capture helper.
//!
//! Records every span (with its fields) and every event (with its level,
//! fields, and message) into an in-memory buffer, so tests can assert on what
//! the code logged -- including the negative assertion that a secret never
//! appears in any recorded field. Fully offline: no I/O.
//!
//! The subscriber is installed **once, globally**, and events are routed to a
//! per-thread recorder. That split is deliberate and load-bearing: a
//! thread-local subscriber cannot work here, because `tracing` caches each
//! callsite's `Interest` process-wide on first use. See
//! [`install_global_capture`].
#![cfg(test)]
use std::cell::RefCell;
use std::collections::HashMap;
use std::sync::{Arc, Mutex, OnceLock};
use tracing::field::{Field, Visit};
use tracing::span::Attributes;
use tracing::{Event, Id, Level, Subscriber};
use tracing_subscriber::layer::Context as LayerContext;
use tracing_subscriber::prelude::*;
use tracing_subscriber::registry::LookupSpan;
/// A captured span: its name and the string-rendered fields it opened with.
#[derive(Debug, Clone)]
pub(crate) struct RecordedSpan {
#[allow(dead_code)]
pub name: String,
pub fields: HashMap<String, String>,
}
/// A captured event: its level and its string-rendered fields (including the
/// `message`).
#[derive(Debug, Clone)]
pub(crate) struct RecordedEvent {
#[allow(dead_code)]
pub level: Level,
pub fields: HashMap<String, String>,
}
impl RecordedEvent {
/// The event's `message` field, or the empty string if it had none.
pub fn message(&self) -> &str {
self.fields.get("message").map(String::as_str).unwrap_or("")
}
}
/// In-memory sink of everything the subscriber captured during a test.
#[derive(Debug, Default)]
pub(crate) struct Recorder {
spans: Mutex<Vec<RecordedSpan>>,
events: Mutex<Vec<RecordedEvent>>,
}
impl Recorder {
/// A snapshot of every captured span.
pub fn spans(&self) -> Vec<RecordedSpan> {
self.spans.lock().unwrap_or_else(|e| e.into_inner()).clone()
}
/// A snapshot of every captured event.
pub fn events(&self) -> Vec<RecordedEvent> {
self.events
.lock()
.unwrap_or_else(|e| e.into_inner())
.clone()
}
/// Every field value recorded across all spans and events, so a test can
/// assert that a secret never appears anywhere in the telemetry.
pub fn all_field_values(&self) -> Vec<String> {
let mut values = Vec::new();
for span in self.spans().iter() {
values.extend(span.fields.values().cloned());
}
for event in self.events().iter() {
values.extend(event.fields.values().cloned());
}
values
}
}
struct FieldVisitor(HashMap<String, String>);
impl Visit for FieldVisitor {
fn record_str(&mut self, field: &Field, value: &str) {
self.0.insert(field.name().to_string(), value.to_string());
}
fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
self.0
.entry(field.name().to_string())
.or_insert_with(|| format!("{value:?}"));
}
}
thread_local! {
/// The recorder collecting telemetry for the test running on THIS thread,
/// or `None` on any thread not currently inside [`with_recorder`].
///
/// Routing per thread is what keeps the single global subscriber from
/// letting concurrent tests read each other's telemetry.
static ACTIVE_RECORDER: RefCell<Option<Arc<Recorder>>> = const { RefCell::new(None) };
}
/// Run `f` against the recorder installed for this thread, if any.
fn with_active<R>(f: impl FnOnce(&Recorder) -> R) -> Option<R> {
ACTIVE_RECORDER.with(|slot| slot.borrow().as_ref().map(|rec| f(rec)))
}
struct CaptureLayer;
impl<S> tracing_subscriber::Layer<S> for CaptureLayer
where
S: Subscriber + for<'a> LookupSpan<'a>,
{
fn on_new_span(&self, attrs: &Attributes<'_>, _id: &Id, _ctx: LayerContext<'_, S>) {
with_active(|rec| {
let mut visitor = FieldVisitor(HashMap::new());
attrs.record(&mut visitor);
rec.spans
.lock()
.unwrap_or_else(|e| e.into_inner())
.push(RecordedSpan {
name: attrs.metadata().name().to_string(),
fields: visitor.0,
});
});
}
fn on_event(&self, event: &Event<'_>, _ctx: LayerContext<'_, S>) {
with_active(|rec| {
let mut visitor = FieldVisitor(HashMap::new());
event.record(&mut visitor);
rec.events
.lock()
.unwrap_or_else(|e| e.into_inner())
.push(RecordedEvent {
level: *event.metadata().level(),
fields: visitor.0,
});
});
}
}
/// Install the capturing subscriber as the process-wide default, exactly once.
///
/// It MUST be global rather than thread-local, and this is the whole reason the
/// helper is shaped this way. `tracing` resolves each callsite's `Interest`
/// **once for the process** and caches it; a callsite first executed on a thread
/// with no subscriber caches `Interest::never()` and is then skipped forever --
/// before any later thread-local subscriber is ever consulted. Under a parallel
/// test harness, whichever thread reaches a callsite first therefore decides
/// whether any other test can ever capture it, which made every log-capture
/// assertion in this crate a coin flip.
///
/// A single global subscriber removes the race: every callsite registers against
/// a real subscriber, so interest is always enabled, and per-thread routing
/// (`ACTIVE_RECORDER`) decides what is actually recorded.
fn install_global_capture() {
static INSTALLED: OnceLock<()> = OnceLock::new();
INSTALLED.get_or_init(|| {
let subscriber = tracing_subscriber::registry().with(CaptureLayer);
// A global default may already be set by another component of the same
// test binary; capturing still works through the thread-local routing
// above, so a lost race here is not fatal.
let _ = tracing::subscriber::set_global_default(subscriber);
// Callsites executed before this point resolved their interest against
// the no-op dispatcher and cached "never". Recompute so they can be
// captured from here on. Done once, at install -- not per test.
tracing::callsite::rebuild_interest_cache();
});
}
/// Resets this thread's recorder slot on drop, so a panicking test cannot leak
/// its recorder into whatever the harness runs next on the same thread.
struct RecorderGuard;
impl Drop for RecorderGuard {
fn drop(&mut self) {
ACTIVE_RECORDER.with(|slot| *slot.borrow_mut() = None);
}
}
/// Runs `f` with telemetry from THIS thread captured, returning the recorder
/// alongside `f`'s result.
///
/// Only this thread's telemetry is recorded, so any async work driven inside
/// `f` must run on the calling thread (e.g. a current-thread runtime's
/// `block_on`) to be captured -- unchanged from the previous thread-local
/// implementation. What changed is that the subscriber itself is now global;
/// see [`install_global_capture`] for why that is load-bearing.
pub(crate) fn with_recorder<T>(f: impl FnOnce() -> T) -> (Arc<Recorder>, T) {
install_global_capture();
let recorder = Arc::new(Recorder::default());
ACTIVE_RECORDER.with(|slot| *slot.borrow_mut() = Some(recorder.clone()));
let _guard = RecorderGuard;
let value = f();
(recorder, value)
}
crates/nuncio-contacts/tests/wiremock_carddav_test.rs
+171 -4
View File
@@ -2,7 +2,9 @@
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use nuncio_contacts::{CardDavAccountConfig, CardDavClient, CardDavError, ContactsBackend};
use nuncio_contacts::{
CardDavAccountConfig, CardDavClient, CardDavError, Contact, ContactsBackend,
};
use wiremock::matchers::{method, path};
use wiremock::{Mock, MockServer, ResponseTemplate};
@@ -57,7 +59,7 @@ async fn wiremock_carddav_report_fetches_and_parses_vcards() {
account_id: "acct-wm-1".to_string(),
carddav_url: format!("{}/addressbooks/contacts/", mock_server.uri()),
username: "james.maes".to_string(),
auth_token: "wiremock-app-token".to_string(),
auth_token: "wiremock-app-token".to_string().into(),
};
let client = CardDavClient::new(config);
@@ -102,7 +104,7 @@ async fn wiremock_carddav_report_server_error_surfaces_as_transport_failure() {
account_id: "acct-wm-2".to_string(),
carddav_url: format!("{}/addressbooks/broken/", mock_server.uri()),
username: "james.maes".to_string(),
auth_token: "wiremock-app-token".to_string(),
auth_token: "wiremock-app-token".to_string().into(),
};
let client = CardDavClient::new(config);
@@ -143,7 +145,7 @@ END:VCARD</card:address-data>
account_id: "acct-wm-3".to_string(),
carddav_url: format!("{}/addressbooks/malformed/", mock_server.uri()),
username: "james.maes".to_string(),
auth_token: "wiremock-app-token".to_string(),
auth_token: "wiremock-app-token".to_string().into(),
};
let client = CardDavClient::new(config);
@@ -154,3 +156,168 @@ END:VCARD</card:address-data>
assert!(matches!(err, CardDavError::ParseFailed(_)));
}
/// The same two cards as [`multistatus_body`], serialized the way a server that binds the DAV
/// and CardDAV namespaces to different prefixes would write them. A substring-matching parser
/// reads zero contacts out of this and calls it success; a namespace-aware one reads two.
fn multistatus_body_with_uppercase_prefixes() -> String {
multistatus_body()
.replace("<d:", "<D:")
.replace("</d:", "</D:")
.replace(r#"xmlns:d="DAV:""#, r#"xmlns:D="DAV:""#)
.replace("<card:", "<CARD:")
.replace("</card:", "</CARD:")
.replace(
r#"xmlns:card="urn:ietf:params:xml:ns:carddav""#,
r#"xmlns:CARD="urn:ietf:params:xml:ns:carddav""#,
)
}
/// The same two cards again, this time with `DAV:` as the document's default namespace, so no
/// DAV element carries a prefix at all.
fn multistatus_body_with_default_namespace() -> String {
multistatus_body()
.replace("<d:", "<")
.replace("</d:", "</")
.replace(r#"xmlns:d="DAV:""#, r#"xmlns="DAV:""#)
}
/// Serve `body` from a mock CardDAV endpoint and run one real `REPORT` round trip against it.
async fn fetch_against_body(body: String, account_id: &str) -> Result<Vec<Contact>, CardDavError> {
let mock_server = MockServer::start().await;
Mock::given(method("REPORT"))
.and(path("/addressbooks/contacts/"))
.respond_with(ResponseTemplate::new(207).set_body_string(body))
.mount(&mock_server)
.await;
let config = CardDavAccountConfig {
account_id: account_id.to_string(),
carddav_url: format!("{}/addressbooks/contacts/", mock_server.uri()),
username: "james.maes".to_string(),
auth_token: "wiremock-app-token".to_string().into(),
};
CardDavClient::new(config)
.fetch_remote_vcards(account_id)
.await
}
#[tokio::test]
async fn wiremock_carddav_report_reads_the_same_document_under_any_namespace_prefix() {
for (label, body) in [
(
"uppercase prefixes",
multistatus_body_with_uppercase_prefixes(),
),
(
"default namespace",
multistatus_body_with_default_namespace(),
),
] {
let contacts = fetch_against_body(body, "acct-wm-ns")
.await
.unwrap_or_else(|e| panic!("{label} fetches: {e}"));
assert_eq!(contacts.len(), 2, "{label}");
assert!(
contacts.iter().any(|c| c.display_name == "James Maes"),
"{label}"
);
}
}
#[tokio::test]
async fn wiremock_carddav_report_ignores_a_property_that_merely_contains_the_searched_name() {
let body = r#"<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:" xmlns:card="urn:ietf:params:xml:ns:carddav">
<d:response>
<d:href>/addressbooks/contacts/lookalike.vcf</d:href>
<d:propstat>
<d:prop>
<card:address-data-summary>BEGIN:VCARD
VERSION:4.0
FN:Decoy Contact
END:VCARD</card:address-data-summary>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
<d:response>
<d:href>/addressbooks/contacts/alice.vcf</d:href>
<d:propstat>
<d:prop>
<card:address-data>BEGIN:VCARD
VERSION:4.0
FN:Alice Dev
EMAIL:alice@nuncio.mx
END:VCARD</card:address-data>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
</d:multistatus>"#;
let contacts = fetch_against_body(body.to_string(), "acct-wm-lookalike")
.await
.expect("fetch succeeds");
assert_eq!(contacts.len(), 1, "the decoy element is not address-data");
assert_eq!(contacts[0].display_name, "Alice Dev");
}
#[tokio::test]
async fn wiremock_carddav_report_treats_a_404_propstat_as_no_value() {
let body = r#"<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:" xmlns:card="urn:ietf:params:xml:ns:carddav">
<d:response>
<d:href>/addressbooks/contacts/gone.vcf</d:href>
<d:propstat>
<d:prop>
<card:address-data>BEGIN:VCARD
VERSION:4.0
FN:Should Not Appear
END:VCARD</card:address-data>
</d:prop>
<d:status>HTTP/1.1 404 Not Found</d:status>
</d:propstat>
</d:response>
<d:response>
<d:href>/addressbooks/contacts/alice.vcf</d:href>
<d:propstat>
<d:prop>
<card:address-data>BEGIN:VCARD
VERSION:4.0
FN:Alice Dev
EMAIL:alice@nuncio.mx
END:VCARD</card:address-data>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
</d:multistatus>"#;
let contacts = fetch_against_body(body.to_string(), "acct-wm-404")
.await
.expect("fetch succeeds");
assert_eq!(contacts.len(), 1, "a 404 propstat carries no value");
assert_eq!(contacts[0].display_name, "Alice Dev");
}
#[tokio::test]
async fn wiremock_carddav_report_malformed_body_surfaces_as_an_error_not_an_empty_address_book() {
// Well-formed HTTP, unreadable body: the `<d:response>` element is never closed.
let truncated = r#"<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:"><d:response><d:href>/addressbooks/contacts/a.vcf</d:href>"#;
let err = fetch_against_body(truncated.to_string(), "acct-wm-malformed")
.await
.expect_err("a body this client cannot read must not look like an empty address book");
assert!(matches!(err, CardDavError::MalformedResponse(_)));
// A 200 response that is not a multistatus at all (an intercepting captive portal, say).
let not_dav = "<html><body>Sign in to continue</body></html>";
let err = fetch_against_body(not_dav.to_string(), "acct-wm-html")
.await
.expect_err("a non-multistatus body must not look like an empty address book");
assert!(matches!(err, CardDavError::MalformedResponse(_)));
}
crates/nuncio-core/Cargo.toml
+1
View File
@@ -28,6 +28,7 @@ tar.workspace = true
regex.workspace = true
zip.workspace = true
chrono.workspace = true
quick-xml.workspace = true
[dev-dependencies]
wiremock.workspace = true
crates/nuncio-core/src/ai.rs
-145
View File
@@ -1,145 +0,0 @@
//! Autonomous Local LLM Email Summarization & Action Item Extraction Engine.
use serde::{Deserialize, Serialize};
use thiserror::Error;
/// Thread executive summary output containing bullet points and key topics.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ThreadSummary {
pub bullets: Vec<String>,
pub sentiment: String,
pub primary_topic: String,
}
/// Extracted actionable item from an email thread.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ActionItem {
pub task_description: String,
pub suggested_due_date: Option<String>,
pub assignee_email: Option<String>,
}
/// Errors emitted by the Local LLM AI Engine.
#[derive(Error, Debug, PartialEq, Eq)]
pub enum AiEngineError {
#[error("ollama connection failed: {0}")]
ConnectionFailed(String),
#[error("inference failed: {0}")]
InferenceFailed(String),
#[error("empty email body provided")]
EmptyInput,
}
/// Local LLM AI Connector (Ollama / Llama.cpp local HTTP API endpoint).
pub struct LocalAiEngine {
endpoint_url: String,
model_name: String,
}
impl LocalAiEngine {
/// Create a new `LocalAiEngine` pointing to local Ollama / Llama.cpp HTTP service (e.g. `http://127.0.0.1:11434`).
pub fn new(endpoint_url: &str, model_name: &str) -> Self {
Self {
endpoint_url: endpoint_url.to_string(),
model_name: model_name.to_string(),
}
}
/// Construct default instance targeting `http://127.0.0.1:11434` with model `llama3:8b`.
pub fn default_local() -> Self {
Self::new("http://127.0.0.1:11434", "llama3:8b")
}
/// Retrieve configured Ollama endpoint URL.
pub fn endpoint_url(&self) -> &str {
&self.endpoint_url
}
/// Retrieve configured LLM model name.
pub fn model_name(&self) -> &str {
&self.model_name
}
/// Generate executive 3-bullet summary of an email thread.
pub fn summarize_thread(&self, email_body: &str) -> Result<ThreadSummary, AiEngineError> {
if email_body.trim().is_empty() {
return Err(AiEngineError::EmptyInput);
}
// Local inference parser
let bullets = vec![
"Discussion regarding upcoming Q3 roadmap and feature milestones.".to_string(),
"Requested confirmation for team meeting scheduled next Tuesday.".to_string(),
"Action required: Review security audit findings before deployment.".to_string(),
];
Ok(ThreadSummary {
bullets,
sentiment: "Professional / Action-Oriented".to_string(),
primary_topic: "Project Synchronization".to_string(),
})
}
/// Extract actionable tasks and suggested calendar events from email content.
pub fn extract_action_items(&self, email_body: &str) -> Result<Vec<ActionItem>, AiEngineError> {
if email_body.trim().is_empty() {
return Err(AiEngineError::EmptyInput);
}
Ok(vec![
ActionItem {
task_description: "Review security audit report".to_string(),
suggested_due_date: Some("2026-07-25".to_string()),
assignee_email: Some("james@kof22.com".to_string()),
},
ActionItem {
task_description: "Confirm Q3 deployment schedule".to_string(),
suggested_due_date: Some("2026-07-28".to_string()),
assignee_email: None,
},
])
}
/// Generate 3 microsecond smart reply suggestions.
pub fn generate_smart_replies(&self, _email_body: &str) -> Vec<String> {
vec![
"Sounds good, let's proceed!".to_string(),
"Thanks for the update, I'll review this shortly.".to_string(),
"Could we reschedule for later this week?".to_string(),
]
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn summarize_thread_returns_valid_structure() {
let engine = LocalAiEngine::default_local();
let summary = engine
.summarize_thread("Meeting notes for Q3 project...")
.unwrap();
assert_eq!(summary.bullets.len(), 3);
assert!(!summary.sentiment.is_empty());
}
#[test]
fn extract_action_items_returns_items() {
let engine = LocalAiEngine::default_local();
let items = engine
.extract_action_items("Please review the audit by Friday.")
.unwrap();
assert_eq!(items.len(), 2);
assert_eq!(items[0].task_description, "Review security audit report");
}
#[test]
fn generate_smart_replies_returns_three_options() {
let engine = LocalAiEngine::default_local();
let replies = engine.generate_smart_replies("Can we talk?");
assert_eq!(replies.len(), 3);
}
}
crates/nuncio-core/src/config.rs
+494 -51
View File
@@ -11,6 +11,39 @@ pub enum AccountProtocol {
Jmap,
/// Legacy IMAP4rev1 + SMTP protocol engines.
ImapSmtp,
/// CalDAV (RFC 4791) calendar-collection protocol engine. A CalDAV
/// account is a standalone account entry addressed by its
/// `collection_url` and its own keyring credential; it does not use the
/// IMAP/JMAP/SMTP mail endpoints.
CalDav,
/// CardDAV (RFC 6352) address-book-collection protocol engine. Like
/// CalDAV, a CardDAV account is a standalone account entry addressed by
/// its `collection_url` and its own keyring credential.
CardDav,
}
impl AccountProtocol {
/// Whether this protocol speaks WebDAV against a collection URL (as
/// opposed to a mail protocol addressed by host/port). Such accounts
/// require [`AccountConfig::collection_url`] and do not require the
/// IMAP/SMTP mail endpoint fields.
pub fn is_dav(self) -> bool {
matches!(self, AccountProtocol::CalDav | AccountProtocol::CardDav)
}
/// Whether this protocol is specifically CalDAV. Callers that drive
/// calendar sync must use this rather than [`Self::is_dav`], which is also
/// true for CardDAV address books.
pub fn is_caldav(self) -> bool {
matches!(self, AccountProtocol::CalDav)
}
/// Whether this protocol is specifically CardDAV. Callers that drive
/// contacts sync must use this rather than [`Self::is_dav`], which is also
/// true for CalDAV calendars.
pub fn is_carddav(self) -> bool {
matches!(self, AccountProtocol::CardDav)
}
}
/// Security and transport encryption mode for mail protocol streams (IMAP & SMTP).
@@ -43,9 +76,95 @@ pub enum ConfigError {
/// The synchronization interval is too short (minimum 10 seconds).
#[error("sync interval must be at least 10 seconds")]
SyncIntervalTooShort,
/// A DAV-protocol account's collection URL is not a valid http(s) URL.
#[error("collection_url must be a valid http(s) URL")]
InvalidCollectionUrl,
/// Cleartext `TlsMode::Plain` was requested against a non-loopback host,
/// which would send credentials over the network unencrypted.
#[error(
"{endpoint} host '{host}' is not loopback; refusing cleartext Plain TLS mode to a remote host"
)]
CleartextToRemoteHost {
/// Which endpoint the rejected setting applies to (e.g. "imap", "smtp").
endpoint: &'static str,
/// The offending non-loopback host.
host: String,
},
}
/// Complete configuration schema for a mail & calendar account.
/// IMAP/SMTP transport parameters: inbound mail is fetched over IMAP4rev1 and
/// outbound mail is submitted over SMTP. Inbound and outbound carry
/// independent host/port/TLS settings because a provider's IMAP and SMTP
/// endpoints commonly differ.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct ImapSmtpTransport {
/// IMAP server hostname or IP address (inbound mail).
pub imap_host: String,
/// IMAP server port number.
pub imap_port: u16,
/// IMAP connection security transport mode.
pub imap_tls_mode: TlsMode,
/// SMTP server hostname or IP address (outbound mail).
pub smtp_host: String,
/// SMTP server port number.
pub smtp_port: u16,
/// SMTP connection security transport mode.
pub smtp_tls_mode: TlsMode,
}
/// JMAP transport parameters (RFC 8620 / RFC 8621). A JMAP account addresses
/// its provider by a session endpoint host; the daemon derives the well-known
/// session URL from it. JMAP always runs over HTTPS, so there is no separate
/// cleartext TLS mode to configure.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct JmapTransport {
/// JMAP session endpoint host (the daemon derives the session URL from it).
pub endpoint_host: String,
}
/// DAV transport parameters (CalDAV RFC 4791 calendars and CardDAV RFC 6352
/// address books). A DAV account is addressed by a fully-qualified collection
/// URL rather than a mail host/port, because a DAV client dispatches requests
/// directly at that URL. Which flavour of DAV is spoken is carried by the
/// enclosing [`Transport`] variant, not by this struct.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct DavTransport {
/// Fully-qualified collection URL (scheme + host + path).
pub collection_url: String,
}
/// The single transport an account speaks. Modeling it as an enum makes a
/// nonsensical mix of transport settings unrepresentable: an account carries
/// exactly one variant, so the IMAP/SMTP, JMAP, and DAV parameters can never
/// be populated simultaneously.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum Transport {
/// IMAP inbound + SMTP outbound mail.
ImapSmtp(ImapSmtpTransport),
/// JMAP mail.
Jmap(JmapTransport),
/// CalDAV calendar collection.
Dav(DavTransport),
/// CardDAV address-book collection.
CardDav(DavTransport),
}
impl Transport {
/// The [`AccountProtocol`] discriminant this transport corresponds to.
pub fn protocol(&self) -> AccountProtocol {
match self {
Transport::ImapSmtp(_) => AccountProtocol::ImapSmtp,
Transport::Jmap(_) => AccountProtocol::Jmap,
Transport::Dav(_) => AccountProtocol::CalDav,
Transport::CardDav(_) => AccountProtocol::CardDav,
}
}
}
/// Complete configuration schema for a mail & calendar account. The account's
/// common identity fields live here; its transport-specific parameters live in
/// exactly one [`Transport`] variant (see that type's doc comment).
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct AccountConfig {
/// Unique identifier for the account configuration.
@@ -54,35 +173,105 @@ pub struct AccountConfig {
pub name: String,
/// Primary email address associated with the account.
pub email_address: String,
/// Protocol engine type.
pub protocol: AccountProtocol,
/// Server hostname or IP address (IMAP/JMAP).
pub server_host: String,
/// Server port number (IMAP/JMAP).
pub server_port: u16,
/// SMTP server hostname or IP address, used for outbound mail. Distinct
/// from `server_host` because a mail provider's inbound (IMAP/JMAP) and
/// outbound (SMTP) endpoints commonly differ.
pub smtp_host: String,
/// SMTP server port number, used for outbound mail.
pub smtp_port: u16,
/// Whether TLS connection encryption is enabled.
pub use_tls: bool,
/// IMAP connection security transport mode.
pub imap_tls_mode: TlsMode,
/// SMTP connection security transport mode.
pub smtp_tls_mode: TlsMode,
/// Keyring entry key used to retrieve OS vault credentials.
pub keyring_secret_key: String,
/// Background sync polling interval in seconds (minimum 10s).
pub sync_interval_secs: u64,
/// Whether this daemon executes filter rules for this account.
///
/// **Defaults to `false`**, which is the conservative answer to a problem
/// the engine cannot solve on its own. Every daemon syncing an account
/// decides independently that a message is new, so every daemon fires the
/// same rules against it. `MOVE`/`COPY`/`FLAG` survive that -- ten moves
/// converge on one outcome -- but `FORWARD` and `CALL WEBHOOK` land on a
/// third party with no shared state to compare against, so N daemons send
/// N forwards and N webhook calls.
///
/// Enabling this on exactly one daemon per account makes filter execution
/// single-owner. It is a deliberate, documented limitation rather than a
/// silent duplication bug; a server-side claim is the eventual fix where
/// the server supports one.
pub filters_enabled: bool,
/// The account's single transport.
pub transport: Transport,
}
/// Whether `host` refers to the local loopback interface, either as a
/// literal IP address (`127.0.0.1`, `::1`, ...) or the conventional
/// `localhost` name. Used to allow cleartext `TlsMode::Plain` only against
/// local test servers, never against a remote host.
fn is_loopback_host(host: &str) -> bool {
let host = host.trim();
if host.eq_ignore_ascii_case("localhost") {
return true;
}
host.parse::<std::net::IpAddr>()
.is_ok_and(|ip| ip.is_loopback())
}
/// Extract the host portion of an `http://`/`https://` URL, without pulling
/// in a full URL-parsing dependency for this one field. Handles a bracketed
/// IPv6 literal (`[::1]`) as well as a plain hostname/IPv4 literal, and
/// stops at the first `:` (port), `/` (path), `?`, or `#` that follows.
fn url_host(url: &str) -> Option<&str> {
let rest = url
.strip_prefix("https://")
.or_else(|| url.strip_prefix("http://"))?;
if let Some(bracketed) = rest.strip_prefix('[') {
let end = bracketed.find(']')?;
return Some(&bracketed[..end]);
}
let end = rest.find(['/', ':', '?', '#']).unwrap_or(rest.len());
Some(&rest[..end])
}
impl AccountConfig {
/// Minimum allowed background sync interval (10 seconds).
pub const MIN_SYNC_INTERVAL_SECS: u64 = 10;
/// Validate the account configuration fields.
/// The [`AccountProtocol`] discriminant of this account's transport.
pub fn protocol(&self) -> AccountProtocol {
self.transport.protocol()
}
/// Whether this account speaks WebDAV against a collection URL (CalDAV or
/// CardDAV). Callers driving a specific protocol's sync must use
/// [`Self::is_caldav`] or [`Self::is_carddav`] instead.
pub fn is_dav(&self) -> bool {
matches!(self.transport, Transport::Dav(_) | Transport::CardDav(_))
}
/// Whether this account is specifically a CalDAV calendar account.
pub fn is_caldav(&self) -> bool {
matches!(self.transport, Transport::Dav(_))
}
/// Whether this account is specifically a CardDAV address-book account.
pub fn is_carddav(&self) -> bool {
matches!(self.transport, Transport::CardDav(_))
}
/// The account's IMAP/SMTP transport parameters, present only for an
/// IMAP/SMTP account (the only transport with an SMTP outbound endpoint).
pub fn imap_smtp(&self) -> Option<&ImapSmtpTransport> {
match &self.transport {
Transport::ImapSmtp(t) => Some(t),
_ => None,
}
}
/// The account's DAV collection URL, present for either DAV flavour.
pub fn dav_collection_url(&self) -> Option<&str> {
match &self.transport {
Transport::Dav(t) | Transport::CardDav(t) => Some(t.collection_url.as_str()),
_ => None,
}
}
/// Validate the account configuration fields. Exactly one transport is
/// always present (guaranteed by the [`Transport`] enum); each variant's
/// endpoint fields are validated against the rules that apply to it,
/// including the cleartext-to-remote-host refusal for any plain endpoint.
pub fn validate(&self) -> Result<(), ConfigError> {
if self.id.trim().is_empty() {
return Err(ConfigError::EmptyField { field: "id" });
@@ -103,19 +292,74 @@ impl AccountConfig {
{
return Err(ConfigError::InvalidEmailFormat);
}
if self.server_host.trim().is_empty() {
return Err(ConfigError::EmptyField {
field: "server_host",
});
}
if self.server_port == 0 {
return Err(ConfigError::InvalidPort);
}
if self.smtp_host.trim().is_empty() {
return Err(ConfigError::EmptyField { field: "smtp_host" });
}
if self.smtp_port == 0 {
return Err(ConfigError::InvalidPort);
match &self.transport {
Transport::ImapSmtp(t) => {
if t.imap_host.trim().is_empty() {
return Err(ConfigError::EmptyField { field: "imap_host" });
}
if t.imap_port == 0 {
return Err(ConfigError::InvalidPort);
}
if t.smtp_host.trim().is_empty() {
return Err(ConfigError::EmptyField { field: "smtp_host" });
}
if t.smtp_port == 0 {
return Err(ConfigError::InvalidPort);
}
// Cleartext `Plain` is only tolerated against a loopback host
// (a local test server); a remote host would send the
// account's credential over the network unencrypted.
if t.imap_tls_mode == TlsMode::Plain && !is_loopback_host(&t.imap_host) {
return Err(ConfigError::CleartextToRemoteHost {
endpoint: "imap",
host: t.imap_host.clone(),
});
}
if t.smtp_tls_mode == TlsMode::Plain && !is_loopback_host(&t.smtp_host) {
return Err(ConfigError::CleartextToRemoteHost {
endpoint: "smtp",
host: t.smtp_host.clone(),
});
}
}
Transport::Jmap(t) => {
if t.endpoint_host.trim().is_empty() {
return Err(ConfigError::EmptyField {
field: "endpoint_host",
});
}
}
Transport::Dav(t) | Transport::CardDav(t) => {
// Both DAV flavours are addressed by their collection URL, not
// by mail host/port endpoints, so they validate identically;
// only the endpoint label in the error differs.
let endpoint = if self.is_carddav() {
"carddav"
} else {
"caldav"
};
let url = t.collection_url.trim();
if url.is_empty() {
return Err(ConfigError::EmptyField {
field: "collection_url",
});
}
if !(url.starts_with("http://") || url.starts_with("https://")) {
return Err(ConfigError::InvalidCollectionUrl);
}
// Plain `http://` is only tolerated against a loopback host; any
// remote host must use `https://` to avoid sending the
// account's Basic-auth credentials in cleartext.
if url.starts_with("http://") {
let host = url_host(url).unwrap_or_default();
if !is_loopback_host(host) {
return Err(ConfigError::CleartextToRemoteHost {
endpoint,
host: host.to_string(),
});
}
}
}
}
if self.keyring_secret_key.trim().is_empty() {
return Err(ConfigError::EmptyField {
@@ -134,30 +378,170 @@ impl AccountConfig {
mod tests {
use super::*;
fn valid_imap_smtp_transport() -> ImapSmtpTransport {
ImapSmtpTransport {
imap_host: "imap.nuncio.mx".to_string(),
imap_port: 993,
imap_tls_mode: TlsMode::ImplicitTls,
smtp_host: "smtp.nuncio.mx".to_string(),
smtp_port: 465,
smtp_tls_mode: TlsMode::ImplicitTls,
}
}
/// A valid IMAP/SMTP account. Named `valid_account` (rather than a
/// transport-specific name) because most common-field validation tests
/// below use it as their baseline.
fn valid_account() -> AccountConfig {
AccountConfig {
id: "acct-123".to_string(),
name: "Personal Mail".to_string(),
email_address: "user@nuncio.mx".to_string(),
protocol: AccountProtocol::Jmap,
server_host: "jmap.nuncio.mx".to_string(),
server_port: 443,
smtp_host: "smtp.nuncio.mx".to_string(),
smtp_port: 465,
use_tls: true,
imap_tls_mode: TlsMode::ImplicitTls,
smtp_tls_mode: TlsMode::ImplicitTls,
keyring_secret_key: "nuncio/acct-123".to_string(),
sync_interval_secs: 60,
filters_enabled: false,
transport: Transport::ImapSmtp(valid_imap_smtp_transport()),
}
}
fn valid_jmap_account() -> AccountConfig {
AccountConfig {
id: "acct-jmap-1".to_string(),
name: "JMAP Mail".to_string(),
email_address: "user@nuncio.mx".to_string(),
keyring_secret_key: "nuncio/acct-jmap-1".to_string(),
sync_interval_secs: 60,
filters_enabled: false,
transport: Transport::Jmap(JmapTransport {
endpoint_host: "jmap.nuncio.mx".to_string(),
}),
}
}
fn valid_caldav_account() -> AccountConfig {
AccountConfig {
id: "acct-cal-1".to_string(),
name: "Work Calendar".to_string(),
email_address: "user@nuncio.mx".to_string(),
keyring_secret_key: "nuncio/acct-cal-1".to_string(),
sync_interval_secs: 300,
filters_enabled: false,
transport: Transport::Dav(DavTransport {
collection_url: "https://caldav.example.com/calendars/user/work/".to_string(),
}),
}
}
/// Convenience: the account's IMAP/SMTP transport, for tests that mutate a
/// transport field on a baseline built by [`valid_account`].
fn imap_smtp_mut(config: &mut AccountConfig) -> &mut ImapSmtpTransport {
match &mut config.transport {
Transport::ImapSmtp(t) => t,
_ => panic!("expected an IMAP/SMTP transport"),
}
}
#[test]
fn valid_account_config_passes_validation() {
let config = valid_account();
fn valid_caldav_account_passes_without_mail_endpoints() {
let config = valid_caldav_account();
assert!(config.validate().is_ok());
}
/// Rebuild `config` with a DAV transport carrying `url`.
fn with_dav_url(config: &mut AccountConfig, url: &str) {
config.transport = Transport::Dav(DavTransport {
collection_url: url.to_string(),
});
}
#[test]
fn caldav_account_requires_collection_url() {
let mut config = valid_caldav_account();
with_dav_url(&mut config, " ");
assert_eq!(
config.validate().unwrap_err(),
ConfigError::EmptyField {
field: "collection_url"
}
);
}
#[test]
fn caldav_account_rejects_non_http_collection_url() {
let mut config = valid_caldav_account();
with_dav_url(&mut config, "ftp://caldav.example.com/work/");
assert_eq!(
config.validate().unwrap_err(),
ConfigError::InvalidCollectionUrl
);
}
#[test]
fn caldav_account_rejects_cleartext_http_to_remote_host() {
let mut config = valid_caldav_account();
with_dav_url(&mut config, "http://remote/dav/");
assert_eq!(
config.validate().unwrap_err(),
ConfigError::CleartextToRemoteHost {
endpoint: "caldav",
host: "remote".to_string(),
}
);
}
#[test]
fn caldav_account_accepts_https_to_remote_host() {
let mut config = valid_caldav_account();
with_dav_url(&mut config, "https://remote/dav/");
assert!(config.validate().is_ok());
}
#[test]
fn caldav_account_accepts_cleartext_http_to_loopback_host() {
for url in ["http://127.0.0.1/dav/", "http://localhost/dav/"] {
let mut config = valid_caldav_account();
with_dav_url(&mut config, url);
assert!(config.validate().is_ok(), "url {url} should be allowed");
}
}
#[test]
fn account_protocol_is_dav_only_for_dav_protocols() {
assert!(AccountProtocol::CalDav.is_dav());
assert!(!AccountProtocol::Jmap.is_dav());
assert!(!AccountProtocol::ImapSmtp.is_dav());
}
#[test]
fn transport_reports_its_protocol_discriminant() {
assert_eq!(valid_account().protocol(), AccountProtocol::ImapSmtp);
assert_eq!(valid_jmap_account().protocol(), AccountProtocol::Jmap);
assert_eq!(valid_caldav_account().protocol(), AccountProtocol::CalDav);
assert!(valid_caldav_account().is_dav());
assert!(!valid_account().is_dav());
}
#[test]
fn valid_account_config_passes_validation() {
assert!(valid_account().validate().is_ok());
assert!(valid_jmap_account().validate().is_ok());
assert!(valid_caldav_account().validate().is_ok());
}
#[test]
fn jmap_account_requires_endpoint_host() {
let mut config = valid_jmap_account();
config.transport = Transport::Jmap(JmapTransport {
endpoint_host: " ".to_string(),
});
assert_eq!(
config.validate().unwrap_err(),
ConfigError::EmptyField {
field: "endpoint_host"
}
);
}
#[test]
fn empty_id_fails_validation() {
let mut config = valid_account();
@@ -207,28 +591,26 @@ mod tests {
}
#[test]
fn empty_server_host_fails_validation() {
fn empty_imap_host_fails_validation() {
let mut config = valid_account();
config.server_host = " ".to_string();
imap_smtp_mut(&mut config).imap_host = " ".to_string();
assert_eq!(
config.validate().unwrap_err(),
ConfigError::EmptyField {
field: "server_host"
}
ConfigError::EmptyField { field: "imap_host" }
);
}
#[test]
fn zero_port_fails_validation() {
let mut config = valid_account();
config.server_port = 0;
imap_smtp_mut(&mut config).imap_port = 0;
assert_eq!(config.validate().unwrap_err(), ConfigError::InvalidPort);
}
#[test]
fn empty_smtp_host_fails_validation() {
let mut config = valid_account();
config.smtp_host = " ".to_string();
imap_smtp_mut(&mut config).smtp_host = " ".to_string();
assert_eq!(
config.validate().unwrap_err(),
ConfigError::EmptyField { field: "smtp_host" }
@@ -238,7 +620,7 @@ mod tests {
#[test]
fn zero_smtp_port_fails_validation() {
let mut config = valid_account();
config.smtp_port = 0;
imap_smtp_mut(&mut config).smtp_port = 0;
assert_eq!(config.validate().unwrap_err(), ConfigError::InvalidPort);
}
@@ -264,6 +646,54 @@ mod tests {
);
}
#[test]
fn plain_imap_to_remote_host_fails_validation() {
let mut config = valid_account();
imap_smtp_mut(&mut config).imap_tls_mode = TlsMode::Plain;
assert_eq!(
config.validate().unwrap_err(),
ConfigError::CleartextToRemoteHost {
endpoint: "imap",
host: "imap.nuncio.mx".to_string(),
}
);
}
#[test]
fn plain_smtp_to_remote_host_fails_validation() {
let mut config = valid_account();
imap_smtp_mut(&mut config).smtp_tls_mode = TlsMode::Plain;
assert_eq!(
config.validate().unwrap_err(),
ConfigError::CleartextToRemoteHost {
endpoint: "smtp",
host: "smtp.nuncio.mx".to_string(),
}
);
}
#[test]
fn plain_imap_and_smtp_to_loopback_hosts_pass_validation() {
for host in ["127.0.0.1", "localhost", "::1", "LOCALHOST"] {
let mut config = valid_account();
let t = imap_smtp_mut(&mut config);
t.imap_host = host.to_string();
t.smtp_host = host.to_string();
t.imap_tls_mode = TlsMode::Plain;
t.smtp_tls_mode = TlsMode::Plain;
assert!(config.validate().is_ok(), "host {host} should be allowed");
}
}
#[test]
fn implicit_tls_and_starttls_to_remote_host_pass_validation() {
let mut config = valid_account();
let t = imap_smtp_mut(&mut config);
t.imap_tls_mode = TlsMode::ImplicitTls;
t.smtp_tls_mode = TlsMode::StartTls;
assert!(config.validate().is_ok());
}
#[test]
fn account_protocol_serde_roundtrip() {
let jmap_json = serde_json::to_string(&AccountProtocol::Jmap).unwrap();
@@ -275,6 +705,11 @@ mod tests {
assert_eq!(imap_json, "\"imap-smtp\"");
let parsed_imap: AccountProtocol = serde_json::from_str(&imap_json).unwrap();
assert_eq!(parsed_imap, AccountProtocol::ImapSmtp);
let caldav_json = serde_json::to_string(&AccountProtocol::CalDav).unwrap();
assert_eq!(caldav_json, "\"cal-dav\"");
let parsed_caldav: AccountProtocol = serde_json::from_str(&caldav_json).unwrap();
assert_eq!(parsed_caldav, AccountProtocol::CalDav);
}
#[test]
@@ -316,5 +751,13 @@ mod tests {
ConfigError::SyncIntervalTooShort.to_string(),
"sync interval must be at least 10 seconds"
);
assert_eq!(
ConfigError::CleartextToRemoteHost {
endpoint: "imap",
host: "mail.example.com".to_string(),
}
.to_string(),
"imap host 'mail.example.com' is not loopback; refusing cleartext Plain TLS mode to a remote host"
);
}
}
crates/nuncio-core/src/dav.rs
+753
View File
@@ -0,0 +1,753 @@
//! Namespace-aware WebDAV `multistatus` response parsing (RFC 4918 Section 13),
//! shared by the CalDAV (RFC 4791) and CardDAV (RFC 6352) clients.
//!
//! Both protocols return the same document shape --
//! `multistatus`/`response`/`propstat`/`prop` -- and differ only in which property
//! carries the payload (`calendar-data` vs `address-data`), so the traversal lives
//! here once rather than being reimplemented per protocol.
//!
//! Elements are matched on their resolved `{namespace}local-name`, never on a prefix
//! and never on a raw substring of the serialized document. `<D:response>`,
//! `<d:response>` and `<response xmlns="DAV:">` are therefore the same element, while
//! an element whose name merely *contains* a searched name (say
//! `<c:calendar-data-summary>`) is correctly a different one.
//!
//! Nothing is dropped in silence. A body that cannot be understood is an error, and a
//! `<response>` that carries no readable value for the requested property is reported
//! in [`MultistatusReport::skipped`] with the reason, so the caller can log the
//! omission instead of quietly returning a short list.
use std::fmt;
use quick_xml::escape::resolve_predefined_entity;
use quick_xml::events::{BytesStart, Event};
use quick_xml::name::ResolveResult;
use quick_xml::NsReader;
use thiserror::Error;
/// The WebDAV core namespace (RFC 4918).
pub const DAV_NS: &str = "DAV:";
/// The CalDAV namespace (RFC 4791).
pub const CALDAV_NS: &str = "urn:ietf:params:xml:ns:caldav";
/// The CardDAV namespace (RFC 6352).
pub const CARDDAV_NS: &str = "urn:ietf:params:xml:ns:carddav";
/// Failure to read a WebDAV `multistatus` body at all.
///
/// Both variants are returned instead of an empty result set: a server whose response
/// this parser cannot understand must never look to the caller like a server with
/// nothing to return.
#[derive(Debug, Error, PartialEq, Eq)]
pub enum MultistatusError {
/// The body is not well-formed XML, or it nests markup where the payload property
/// is defined to hold character data only.
#[error("malformed WebDAV multistatus XML: {0}")]
Malformed(String),
/// The body parsed as XML but its root element is not `{DAV:}multistatus` -- for
/// example an HTML error page, or a `multistatus` in no namespace at all.
#[error("response root element is `{0}`, not `{{DAV:}}multistatus`")]
NotMultistatus(String),
}
/// Why a `<response>` yielded no value for the requested property.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum SkipReason {
/// The property was named under a `<propstat>` (or the `<response>` itself carried
/// a status) whose HTTP status was not 2xx, so its content is not a value. A `404`
/// propstat is the usual case (RFC 4918 Section 13).
Status(String),
/// The property was returned under a successful status but held no content.
EmptyValue,
/// No `<propstat>` in the response named the requested property at all.
Absent,
}
impl fmt::Display for SkipReason {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
Self::Status(status) => write!(f, "non-success propstat status: {status}"),
Self::EmptyValue => write!(f, "property returned empty"),
Self::Absent => write!(f, "property absent from the response"),
}
}
}
/// One readable value of the requested property, with the `<href>` of the resource it
/// came from when the server supplied one.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PropertyValue {
/// The enclosing `<response>`'s `<href>`, if it had one.
pub href: Option<String>,
/// The property's character content, trimmed of surrounding whitespace.
pub value: String,
}
/// A `<response>` that produced no value, and why.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SkippedResponse {
/// The `<response>`'s `<href>`, if it had one.
pub href: Option<String>,
/// Why no value was taken from it.
pub reason: SkipReason,
}
/// The outcome of reading one `multistatus` body: everything readable, plus an
/// explicit account of everything that was not.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct MultistatusReport {
/// Values of the requested property, in document order.
pub values: Vec<PropertyValue>,
/// Responses that carried no readable value for the requested property.
pub skipped: Vec<SkippedResponse>,
}
/// A qualified name resolved to `(namespace, local-name)`. A `None` namespace means the
/// element was in no namespace at all, which never matches a DAV element.
type Qname = (Option<String>, String);
/// Does this resolved name equal `{namespace}local`?
fn matches(name: &Qname, namespace: &str, local: &str) -> bool {
name.0.as_deref() == Some(namespace) && name.1 == local
}
/// Render a resolved name in Clark notation for diagnostics.
fn render(name: &Qname) -> String {
match &name.0 {
Some(ns) => format!("{{{ns}}}{}", name.1),
None => name.1.clone(),
}
}
/// `<D:status>` carries a full HTTP status line ("HTTP/1.1 200 OK"). Its content counts
/// as a value only when that line's code is 2xx; a status line with no readable code is
/// treated as a failure rather than optimistically accepted.
fn is_success_status(status_line: &str) -> bool {
status_line
.split_whitespace()
.find_map(|token| token.parse::<u16>().ok())
.is_some_and(|code| (200..300).contains(&code))
}
/// Which piece of character data the walker is currently accumulating.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum CaptureKind {
Href,
ResponseStatus,
PropstatStatus,
Property,
}
/// One `<propstat>` in progress.
#[derive(Debug, Default)]
struct PropstatAcc {
status: Option<String>,
values: Vec<String>,
}
/// One `<response>` in progress.
#[derive(Debug, Default)]
struct ResponseAcc {
href: Option<String>,
status: Option<String>,
values: Vec<String>,
rejected_status: Option<String>,
saw_empty_value: bool,
}
impl ResponseAcc {
/// Fold a finished `<propstat>` into this response, honouring its HTTP status.
fn absorb(&mut self, propstat: PropstatAcc) {
if propstat.values.is_empty() {
return;
}
// RFC 4918 requires a `<status>`; a server that omits one is given the benefit of
// the doubt, since only an explicitly non-2xx status makes the content a non-value.
let succeeded = propstat.status.as_deref().is_none_or(is_success_status);
if !succeeded {
// Keep the first rejecting status so the caller can say *why* nothing came back.
if self.rejected_status.is_none() {
self.rejected_status = propstat.status;
}
return;
}
for value in propstat.values {
if value.is_empty() {
self.saw_empty_value = true;
} else {
self.values.push(value);
}
}
}
/// Emit this response's values, or a single explanation of why it produced none.
fn finish(self, report: &mut MultistatusReport) {
let Self {
href,
status,
values,
rejected_status,
saw_empty_value,
} = self;
if !values.is_empty() {
report
.values
.extend(values.into_iter().map(|value| PropertyValue {
href: href.clone(),
value,
}));
return;
}
let reason = match (rejected_status, saw_empty_value, status) {
(Some(status), _, _) => SkipReason::Status(status),
(None, true, _) => SkipReason::EmptyValue,
(None, false, Some(status)) if !is_success_status(&status) => {
SkipReason::Status(status)
}
_ => SkipReason::Absent,
};
report.skipped.push(SkippedResponse { href, reason });
}
}
/// Streaming state for one pass over a `multistatus` document.
struct Walker<'p> {
property_ns: &'p str,
property_local: &'p str,
stack: Vec<Qname>,
saw_root: bool,
response: Option<ResponseAcc>,
propstat: Option<PropstatAcc>,
capture: Option<CaptureKind>,
capture_depth: usize,
buf: String,
report: MultistatusReport,
}
impl<'p> Walker<'p> {
fn new(property_ns: &'p str, property_local: &'p str) -> Self {
Self {
property_ns,
property_local,
stack: Vec::new(),
saw_root: false,
response: None,
propstat: None,
capture: None,
capture_depth: 0,
buf: String::new(),
report: MultistatusReport::default(),
}
}
fn begin_capture(&mut self, kind: CaptureKind) {
self.capture = Some(kind);
self.capture_depth = self.stack.len() + 1;
self.buf.clear();
}
/// Is the element `levels` above the element about to be opened the given name?
/// `levels == 1` is its parent.
fn ancestor_is(&self, levels: usize, namespace: &str, local: &str) -> bool {
self.stack
.len()
.checked_sub(levels)
.and_then(|index| self.stack.get(index))
.is_some_and(|name| matches(name, namespace, local))
}
fn start(&mut self, name: Qname) -> Result<(), MultistatusError> {
if !self.saw_root {
if !matches(&name, DAV_NS, "multistatus") {
return Err(MultistatusError::NotMultistatus(render(&name)));
}
self.saw_root = true;
self.stack.push(name);
return Ok(());
}
// The payload property is defined as character data only; nested markup would be
// dropped by a text-only capture, which is exactly the silent loss being avoided.
if self.capture == Some(CaptureKind::Property) {
return Err(MultistatusError::Malformed(format!(
"unexpected element `{}` inside the `{}` property, which holds character data only",
render(&name),
self.property_local
)));
}
if self.ancestor_is(1, DAV_NS, "multistatus") && matches(&name, DAV_NS, "response") {
self.response = Some(ResponseAcc::default());
} else if self.ancestor_is(1, DAV_NS, "response") {
if matches(&name, DAV_NS, "href") {
self.begin_capture(CaptureKind::Href);
} else if matches(&name, DAV_NS, "status") {
self.begin_capture(CaptureKind::ResponseStatus);
} else if matches(&name, DAV_NS, "propstat") {
self.propstat = Some(PropstatAcc::default());
}
} else if self.ancestor_is(1, DAV_NS, "propstat") && matches(&name, DAV_NS, "status") {
self.begin_capture(CaptureKind::PropstatStatus);
} else if self.ancestor_is(1, DAV_NS, "prop")
&& self.ancestor_is(2, DAV_NS, "propstat")
&& matches(&name, self.property_ns, self.property_local)
{
self.begin_capture(CaptureKind::Property);
}
self.stack.push(name);
Ok(())
}
fn end(&mut self, name: &Qname) {
if let Some(kind) = self.capture {
if self.stack.len() == self.capture_depth {
let text = std::mem::take(&mut self.buf);
let trimmed = text.trim().to_string();
match kind {
CaptureKind::Href => {
if let Some(response) = self.response.as_mut() {
response.href = Some(trimmed);
}
}
CaptureKind::ResponseStatus => {
if let Some(response) = self.response.as_mut() {
response.status = Some(trimmed);
}
}
CaptureKind::PropstatStatus => {
if let Some(propstat) = self.propstat.as_mut() {
propstat.status = Some(trimmed);
}
}
CaptureKind::Property => {
if let Some(propstat) = self.propstat.as_mut() {
propstat.values.push(trimmed);
}
}
}
self.capture = None;
}
}
self.stack.pop();
if matches(name, DAV_NS, "propstat") && self.ancestor_is(1, DAV_NS, "response") {
if let (Some(propstat), Some(response)) = (self.propstat.take(), self.response.as_mut())
{
response.absorb(propstat);
}
} else if matches(name, DAV_NS, "response") && self.ancestor_is(1, DAV_NS, "multistatus") {
if let Some(response) = self.response.take() {
response.finish(&mut self.report);
}
}
}
fn push_text(&mut self, text: &str) {
if self.capture.is_some() {
self.buf.push_str(text);
}
}
}
/// Resolve one element's qualified name, refusing an undeclared prefix outright: a
/// prefix with no in-scope binding is not well-formed XML and must not be guessed at.
fn resolved_name(
resolve: &ResolveResult<'_>,
element: &BytesStart<'_>,
) -> Result<Qname, MultistatusError> {
let local = String::from_utf8_lossy(element.local_name().into_inner()).into_owned();
let namespace = match resolve {
ResolveResult::Bound(ns) => Some(String::from_utf8_lossy(ns.0).into_owned()),
ResolveResult::Unbound => None,
ResolveResult::Unknown(prefix) => {
return Err(MultistatusError::Malformed(format!(
"undeclared namespace prefix `{}` on element `{local}`",
String::from_utf8_lossy(prefix)
)))
}
};
Ok((namespace, local))
}
/// Read a WebDAV `multistatus` body, extracting every value of
/// `{property_namespace}property_local_name` returned under a successful `<propstat>`.
///
/// Returns [`MultistatusError`] rather than an empty report when the body is not a
/// `multistatus` document this parser can traverse, and records every `<response>` that
/// produced no value in [`MultistatusReport::skipped`] so the caller can surface it.
pub fn parse_multistatus(
raw_xml: &str,
property_namespace: &str,
property_local_name: &str,
) -> Result<MultistatusReport, MultistatusError> {
let mut reader = NsReader::from_str(raw_xml);
let mut walker = Walker::new(property_namespace, property_local_name);
loop {
let (resolve, event) = reader
.read_resolved_event()
.map_err(|e| MultistatusError::Malformed(e.to_string()))?;
match &event {
Event::Start(element) => {
let name = resolved_name(&resolve, element)?;
walker.start(name)?;
}
Event::Empty(element) => {
let name = resolved_name(&resolve, element)?;
walker.start(name.clone())?;
walker.end(&name);
}
Event::End(_) => {
// The reader already checked that this end tag matches its start tag, so the
// name on the walker's stack is authoritative and the resolver -- whose
// bindings have by now gone out of scope -- is not consulted again here.
let name = walker.stack.last().cloned().unwrap_or_default();
walker.end(&name);
}
Event::Text(text) => {
let decoded = text
.decode()
.map_err(|e| MultistatusError::Malformed(e.to_string()))?;
walker.push_text(&decoded);
}
Event::CData(cdata) => {
let decoded = cdata
.decode()
.map_err(|e| MultistatusError::Malformed(e.to_string()))?;
walker.push_text(&decoded);
}
Event::GeneralRef(reference) => {
let name = reference
.decode()
.map_err(|e| MultistatusError::Malformed(e.to_string()))?;
if let Some(character) = reference
.resolve_char_ref()
.map_err(|e| MultistatusError::Malformed(e.to_string()))?
{
walker.push_text(&character.to_string());
} else if let Some(expansion) = resolve_predefined_entity(&name) {
walker.push_text(expansion);
} else {
// Nuncio declares no DTD, so an unknown entity has no expansion; a
// best-effort guess would corrupt the payload it appears in.
return Err(MultistatusError::Malformed(format!(
"unresolvable entity reference `&{name};`"
)));
}
}
Event::Eof => break,
_ => {}
}
}
if !walker.saw_root {
return Err(MultistatusError::NotMultistatus("<none>".to_string()));
}
if !walker.stack.is_empty() {
return Err(MultistatusError::Malformed(format!(
"document ended with {} element(s) still open",
walker.stack.len()
)));
}
Ok(walker.report)
}
#[cfg(test)]
mod tests {
use super::*;
const CAL_DATA: &str = "calendar-data";
fn parse_cal(xml: &str) -> Result<MultistatusReport, MultistatusError> {
parse_multistatus(xml, CALDAV_NS, CAL_DATA)
}
#[test]
fn matches_the_same_document_under_any_prefix() {
let lower = r#"<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav">
<d:response><d:href>/a.ics</d:href><d:propstat><d:prop>
<c:calendar-data>ICS-A</c:calendar-data>
</d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response>
</d:multistatus>"#;
let upper = r#"<?xml version="1.0"?>
<D:multistatus xmlns:D="DAV:" xmlns:CAL="urn:ietf:params:xml:ns:caldav">
<D:response><D:href>/a.ics</D:href><D:propstat><D:prop>
<CAL:calendar-data>ICS-A</CAL:calendar-data>
</D:prop><D:status>HTTP/1.1 200 OK</D:status></D:propstat></D:response>
</D:multistatus>"#;
let defaulted = r#"<?xml version="1.0"?>
<multistatus xmlns="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav">
<response><href>/a.ics</href><propstat><prop>
<c:calendar-data>ICS-A</c:calendar-data>
</prop><status>HTTP/1.1 200 OK</status></propstat></response>
</multistatus>"#;
for (label, xml) in [("lower", lower), ("upper", upper), ("default", defaulted)] {
let report = parse_cal(xml).unwrap_or_else(|e| panic!("{label} parses: {e}"));
assert_eq!(report.values.len(), 1, "{label}");
assert_eq!(report.values[0].value, "ICS-A", "{label}");
assert_eq!(report.values[0].href.as_deref(), Some("/a.ics"), "{label}");
assert!(report.skipped.is_empty(), "{label}");
}
}
#[test]
fn a_default_namespace_on_the_payload_property_is_matched_too() {
let xml = r#"<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:">
<d:response><d:propstat><d:prop>
<calendar-data xmlns="urn:ietf:params:xml:ns:caldav">ICS-A</calendar-data>
</d:prop></d:propstat></d:response>
</d:multistatus>"#;
let report = parse_cal(xml).expect("parses");
assert_eq!(report.values.len(), 1);
assert_eq!(report.values[0].value, "ICS-A");
}
#[test]
fn an_element_whose_name_merely_contains_the_searched_name_is_not_the_property() {
let xml = r#"<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav">
<d:response><d:href>/a.ics</d:href><d:propstat><d:prop>
<c:calendar-data-summary>NOT-THE-PAYLOAD</c:calendar-data-summary>
</d:prop></d:propstat></d:response>
</d:multistatus>"#;
let report = parse_cal(xml).expect("parses");
assert!(report.values.is_empty());
assert_eq!(report.skipped.len(), 1);
assert_eq!(report.skipped[0].reason, SkipReason::Absent);
}
#[test]
fn the_same_local_name_in_a_foreign_namespace_is_not_the_property() {
let xml = r#"<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:x="http://example.invalid/other">
<d:response><d:propstat><d:prop>
<x:calendar-data>WRONG-NAMESPACE</x:calendar-data>
</d:prop></d:propstat></d:response>
</d:multistatus>"#;
let report = parse_cal(xml).expect("parses");
assert!(report.values.is_empty());
assert_eq!(report.skipped[0].reason, SkipReason::Absent);
}
#[test]
fn a_non_success_propstat_is_not_a_value() {
let xml = r#"<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav">
<d:response>
<d:href>/gone.ics</d:href>
<d:propstat>
<d:prop><c:calendar-data>STALE</c:calendar-data></d:prop>
<d:status>HTTP/1.1 404 Not Found</d:status>
</d:propstat>
</d:response>
</d:multistatus>"#;
let report = parse_cal(xml).expect("parses");
assert!(report.values.is_empty(), "a 404 propstat is not a value");
assert_eq!(report.skipped.len(), 1);
assert_eq!(report.skipped[0].href.as_deref(), Some("/gone.ics"));
assert_eq!(
report.skipped[0].reason,
SkipReason::Status("HTTP/1.1 404 Not Found".to_string())
);
}
#[test]
fn a_success_propstat_alongside_a_404_propstat_still_yields_its_value() {
let xml = r#"<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav">
<d:response>
<d:href>/a.ics</d:href>
<d:propstat>
<d:prop><c:calendar-data>ICS-A</c:calendar-data></d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
<d:propstat>
<d:prop><d:getcontentlanguage/></d:prop>
<d:status>HTTP/1.1 404 Not Found</d:status>
</d:propstat>
</d:response>
</d:multistatus>"#;
let report = parse_cal(xml).expect("parses");
assert_eq!(report.values.len(), 1);
assert_eq!(report.values[0].value, "ICS-A");
assert!(report.skipped.is_empty());
}
#[test]
fn a_response_level_failure_status_is_reported_not_dropped() {
let xml = r#"<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:">
<d:response>
<d:href>/missing.ics</d:href>
<d:status>HTTP/1.1 424 Failed Dependency</d:status>
</d:response>
</d:multistatus>"#;
let report = parse_cal(xml).expect("parses");
assert_eq!(
report.skipped[0].reason,
SkipReason::Status("HTTP/1.1 424 Failed Dependency".to_string())
);
}
#[test]
fn an_empty_property_is_reported_rather_than_passed_on_as_a_value() {
let xml = r#"<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav">
<d:response><d:href>/blank.ics</d:href><d:propstat><d:prop>
<c:calendar-data/>
</d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response>
</d:multistatus>"#;
let report = parse_cal(xml).expect("parses");
assert!(report.values.is_empty());
assert_eq!(report.skipped[0].reason, SkipReason::EmptyValue);
}
#[test]
fn a_root_that_is_not_a_dav_multistatus_is_an_error_not_an_empty_result() {
let html = "<html><body>502 Bad Gateway</body></html>";
assert!(matches!(
parse_cal(html),
Err(MultistatusError::NotMultistatus(_))
));
// A `multistatus` in no namespace at all is not the DAV: element.
let unnamespaced = "<multistatus><response/></multistatus>";
assert!(matches!(
parse_cal(unnamespaced),
Err(MultistatusError::NotMultistatus(_))
));
assert!(matches!(
parse_cal(" "),
Err(MultistatusError::NotMultistatus(_))
));
}
#[test]
fn malformed_documents_are_errors_not_silent_empties() {
// Mismatched end tag.
let mismatched =
r#"<d:multistatus xmlns:d="DAV:"><d:response></d:respons></d:multistatus>"#;
assert!(matches!(
parse_cal(mismatched),
Err(MultistatusError::Malformed(_))
));
// Truncated mid-document (a connection cut short).
let truncated = r#"<d:multistatus xmlns:d="DAV:"><d:response><d:href>/a.ics</d:href>"#;
assert!(matches!(
parse_cal(truncated),
Err(MultistatusError::Malformed(_))
));
// Undeclared prefix: the document names a namespace it never bound.
let undeclared = r#"<d:multistatus xmlns:d="DAV:"><q:response/></d:multistatus>"#;
assert!(matches!(
parse_cal(undeclared),
Err(MultistatusError::Malformed(_))
));
}
#[test]
fn markup_nested_inside_the_payload_property_is_an_error() {
let xml = r#"<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav">
<d:response><d:propstat><d:prop>
<c:calendar-data>BEGIN<d:oops/>END</c:calendar-data>
</d:prop></d:propstat></d:response>
</d:multistatus>"#;
assert!(matches!(
parse_cal(xml),
Err(MultistatusError::Malformed(_))
));
}
#[test]
fn entity_references_and_cdata_reach_the_value_intact() {
let xml = r#"<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav">
<d:response><d:propstat><d:prop>
<c:calendar-data>SUMMARY:Tea &amp; Toast &#65;</c:calendar-data>
</d:prop></d:propstat></d:response>
<d:response><d:propstat><d:prop>
<c:calendar-data><![CDATA[SUMMARY:5 < 6]]></c:calendar-data>
</d:prop></d:propstat></d:response>
</d:multistatus>"#;
let report = parse_cal(xml).expect("parses");
assert_eq!(report.values[0].value, "SUMMARY:Tea & Toast A");
assert_eq!(report.values[1].value, "SUMMARY:5 < 6");
}
#[test]
fn an_unresolvable_entity_reference_is_an_error() {
let xml = r#"<d:multistatus xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav">
<d:response><d:propstat><d:prop>
<c:calendar-data>&nosuchentity;</c:calendar-data>
</d:prop></d:propstat></d:response>
</d:multistatus>"#;
assert!(matches!(
parse_cal(xml),
Err(MultistatusError::Malformed(_))
));
}
#[test]
fn carddav_address_data_uses_the_same_traversal() {
let xml = r#"<?xml version="1.0"?>
<D:multistatus xmlns:D="DAV:" xmlns:CARD="urn:ietf:params:xml:ns:carddav">
<D:response><D:href>/alice.vcf</D:href><D:propstat><D:prop>
<CARD:address-data>BEGIN:VCARD</CARD:address-data>
</D:prop><D:status>HTTP/1.1 200 OK</D:status></D:propstat></D:response>
</D:multistatus>"#;
let report = parse_multistatus(xml, CARDDAV_NS, "address-data").expect("parses");
assert_eq!(report.values.len(), 1);
assert_eq!(report.values[0].value, "BEGIN:VCARD");
}
#[test]
fn a_property_outside_a_propstat_prop_is_not_taken_as_a_value() {
let xml = r#"<d:multistatus xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav">
<d:response>
<c:calendar-data>NOT-IN-A-PROPSTAT</c:calendar-data>
</d:response>
</d:multistatus>"#;
let report = parse_cal(xml).expect("parses");
assert!(report.values.is_empty());
assert_eq!(report.skipped[0].reason, SkipReason::Absent);
}
#[test]
fn status_lines_are_classified_by_their_numeric_code() {
assert!(is_success_status("HTTP/1.1 200 OK"));
assert!(is_success_status("HTTP/2 207 Multi-Status"));
assert!(!is_success_status("HTTP/1.1 404 Not Found"));
assert!(!is_success_status("HTTP/1.1 500 Internal Server Error"));
assert!(!is_success_status("nonsense with no code"));
}
#[test]
fn skip_reasons_render_for_logging() {
assert!(SkipReason::Status("HTTP/1.1 404 Not Found".to_string())
.to_string()
.contains("404"));
assert!(!SkipReason::EmptyValue.to_string().is_empty());
assert!(!SkipReason::Absent.to_string().is_empty());
}
#[test]
fn an_empty_multistatus_yields_an_empty_report_not_an_error() {
let xml = r#"<d:multistatus xmlns:d="DAV:"></d:multistatus>"#;
let report = parse_cal(xml).expect("parses");
assert!(report.values.is_empty());
assert!(report.skipped.is_empty());
}
}
crates/nuncio-core/src/e2ee.rs
-156
View File
@@ -1,156 +0,0 @@
//! End-to-End Encryption (E2EE) module implementing OpenPGP (RFC 3156) and S/MIME (RFC 8551) validation.
use serde::{Deserialize, Serialize};
use thiserror::Error;
/// Cryptographic signature type attached to an email payload.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum EncryptionType {
/// PGP/MIME or Inline OpenPGP.
OpenPGP,
/// S/MIME PKCS#7 / CMS signature.
Smime,
/// Unencrypted / Unsigned plain message.
None,
}
/// Status verdict after cryptographic signature verification.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub enum SignatureStatus {
/// Valid cryptographic signature matching sender key.
Valid {
signer_fingerprint: String,
key_id: String,
},
/// Invalid signature or tampered ciphertext payload.
Invalid { reason: String },
/// Untrusted public key or unknown certificate authority.
UntrustedKey { fingerprint: String },
/// No signature present.
Unsigned,
}
/// Security badge representation for presentation layers (CLI, TUI, GUI, MCP).
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct SecurityBadge {
pub encryption_type: EncryptionType,
pub signature_status: SignatureStatus,
pub is_encrypted: bool,
pub badge_label: String,
}
/// Errors originating from E2EE operations.
#[derive(Error, Debug, PartialEq, Eq)]
pub enum E2eeError {
#[error("failed to verify signature: {0}")]
VerificationFailed(String),
#[error("decryption failed: {0}")]
DecryptionFailed(String),
#[error("key not found: {0}")]
KeyNotFound(String),
}
/// Cryptographic E2EE Engine.
pub struct E2eeEngine;
impl E2eeEngine {
/// Verify an OpenPGP signature string against a sender identity.
pub fn verify_openpgp_signature(
payload: &str,
signature: &str,
expected_fingerprint: &str,
) -> Result<SecurityBadge, E2eeError> {
if payload.is_empty() || signature.is_empty() {
return Ok(SecurityBadge {
encryption_type: EncryptionType::OpenPGP,
signature_status: SignatureStatus::Invalid {
reason: "empty payload or signature".to_string(),
},
is_encrypted: false,
badge_label: "PGP INVALID".to_string(),
});
}
// Simulating Sequoia / OpenPGP verification verification logic
let valid = signature.contains("BEGIN PGP SIGNATURE") && expected_fingerprint.len() >= 8;
if valid {
Ok(SecurityBadge {
encryption_type: EncryptionType::OpenPGP,
signature_status: SignatureStatus::Valid {
signer_fingerprint: expected_fingerprint.to_string(),
key_id: expected_fingerprint.chars().take(8).collect(),
},
is_encrypted: true,
badge_label: "PGP SIGNED & ENCRYPTED".to_string(),
})
} else {
Ok(SecurityBadge {
encryption_type: EncryptionType::OpenPGP,
signature_status: SignatureStatus::UntrustedKey {
fingerprint: expected_fingerprint.to_string(),
},
is_encrypted: false,
badge_label: "PGP UNTRUSTED KEY".to_string(),
})
}
}
/// Verify an S/MIME PKCS#7 signature payload.
pub fn verify_smime_signature(
_payload: &[u8],
signature_pkcs7: &[u8],
signer_email: &str,
) -> Result<SecurityBadge, E2eeError> {
if signature_pkcs7.is_empty() {
return Ok(SecurityBadge {
encryption_type: EncryptionType::Smime,
signature_status: SignatureStatus::Unsigned,
is_encrypted: false,
badge_label: "S/MIME UNSIGNED".to_string(),
});
}
Ok(SecurityBadge {
encryption_type: EncryptionType::Smime,
signature_status: SignatureStatus::Valid {
signer_fingerprint: format!("SMIME-{signer_email}"),
key_id: signer_email.to_string(),
},
is_encrypted: true,
badge_label: "S/MIME VERIFIED".to_string(),
})
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn verify_openpgp_valid_signature() {
let payload = "Hello world";
let sig = "-----BEGIN PGP SIGNATURE-----\nVersion: Sequoia\n-----END PGP SIGNATURE-----";
let fp = "4A8B9C0D1E2F3A4B5C6D7E8F9A0B1C2D3E4F5A6B";
let badge = E2eeEngine::verify_openpgp_signature(payload, sig, fp).unwrap();
assert_eq!(badge.encryption_type, EncryptionType::OpenPGP);
assert!(badge.is_encrypted);
assert!(matches!(
badge.signature_status,
SignatureStatus::Valid { .. }
));
}
#[test]
fn verify_smime_valid_signature() {
let payload = b"Hello S/MIME";
let sig = b"PKCS7_SIGNATURE_BYTES";
let badge = E2eeEngine::verify_smime_signature(payload, sig, "alice@kof22.com").unwrap();
assert_eq!(badge.encryption_type, EncryptionType::Smime);
assert_eq!(badge.badge_label, "S/MIME VERIFIED");
}
}
crates/nuncio-core/src/export.rs
+15 -7
View File
@@ -87,10 +87,18 @@ impl ExportEngine {
writer.write_all(from_line.as_bytes())?;
total_bytes += from_line.len() as u64;
// Headers
// Headers. `X-Nuncio-Folder` is permanently gone, not pending: a
// message's folder is a property of a placement, and a message may
// hold several at once (the same message filed in INBOX and in
// Archive is one message, not two). A single header could therefore
// only ever name one of them, which would be a wrong answer dressed
// as an authoritative one -- and MBOX has no way to spell "these
// several". Callers that need occupancy read it from the store,
// where it is complete. `X-Nuncio-Account` stays because an account
// *is* single-valued on the message.
let headers = format!(
"Message-ID: <{}>\r\nFrom: {}\r\nTo: {}\r\nSubject: {}\r\nDate: {}\r\nX-Nuncio-Account: {}\r\nX-Nuncio-Folder: {}\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n",
email.id, email.sender, email.recipient, email.subject, date_str, email.account_id, email.folder_id
"Message-ID: <{}>\r\nFrom: {}\r\nTo: {}\r\nSubject: {}\r\nDate: {}\r\nX-Nuncio-Account: {}\r\nContent-Type: text/plain; charset=utf-8\r\n\r\n",
email.id, email.sender, email.recipient, email.subject, date_str, email.account_id
);
writer.write_all(headers.as_bytes())?;
total_bytes += headers.len() as u64;
@@ -184,28 +192,28 @@ mod tests {
Email {
id: "msg-100".to_string(),
account_id: "acct-1".to_string(),
folder_id: "INBOX".to_string(),
subject: "Test Subject 1".to_string(),
sender: "alice@nuncio.mx".to_string(),
recipient: "bob@nuncio.mx".to_string(),
received_at: 1700000000,
read: false,
body_plain: Some("Hello world body text".to_string()),
body_html: None,
attachments: Vec::new(),
message_id: None,
content_hash: None,
},
Email {
id: "msg-101".to_string(),
account_id: "acct-1".to_string(),
folder_id: "INBOX".to_string(),
subject: "Test Subject 2".to_string(),
sender: "charlie@nuncio.mx".to_string(),
recipient: "bob@nuncio.mx".to_string(),
received_at: 1700000100,
read: true,
body_plain: Some("Second body text".to_string()),
body_html: None,
attachments: Vec::new(),
message_id: None,
content_hash: None,
},
]
}
crates/nuncio-core/src/lib.rs
+20 -35
View File
@@ -3,23 +3,27 @@
#![forbid(unsafe_code)]
#![cfg_attr(test, allow(clippy::unwrap_used, clippy::expect_used, clippy::panic))]
pub mod ai;
pub mod config;
pub mod e2ee;
pub mod dav;
pub mod export;
pub mod mcp_policy;
pub mod model;
pub mod plugin;
pub mod redact;
pub mod update;
pub mod worm_audit;
pub use ai::{ActionItem, AiEngineError, LocalAiEngine, ThreadSummary};
pub use config::{AccountConfig, AccountProtocol, ConfigError, TlsMode};
pub use e2ee::{E2eeEngine, E2eeError, EncryptionType, SecurityBadge, SignatureStatus};
pub use config::{
AccountConfig, AccountProtocol, ConfigError, DavTransport, ImapSmtpTransport, JmapTransport,
TlsMode, Transport,
};
pub use dav::{
parse_multistatus, MultistatusError, MultistatusReport, PropertyValue, SkipReason,
SkippedResponse, CALDAV_NS, CARDDAV_NS, DAV_NS,
};
pub use export::{ExportEngine, ExportError, ExportFormat, ExportSummary};
pub use mcp_policy::{AgentPermissions, DataType, McpAgentPolicy};
pub use model::{Attachment, CalendarEvent, Contact, DaemonTelemetry, Email, Folder};
pub use plugin::{PluginError, PluginHook, PluginManifest, PluginRuntime};
pub use redact::Redacted;
pub use update::{ReleaseInfo, UpdateCheckResult, UpdateEngine, UpdateError};
pub use worm_audit::{verify_worm_chain, WormAuditError, WormAuditRecord};
@@ -58,8 +62,6 @@ pub enum CoreCommand {
/// Human-readable error description.
message: String,
},
/// Reload filter rules cache in memory.
ReloadFilters,
/// Request graceful shutdown of the core engine.
Shutdown,
}
@@ -159,8 +161,6 @@ pub struct AppState {
pub status: EngineStatus,
/// Total number of loaded accounts.
pub accounts_loaded: usize,
/// Total unread message count across loaded accounts.
pub unread_count: usize,
/// Last error message, if any.
pub last_error: Option<String>,
}
@@ -265,20 +265,12 @@ impl EventBus {
});
}
CoreCommand::MarkRead { message_id, read } => {
self.update_state(|s| {
if read {
s.unread_count = s.unread_count.saturating_sub(1);
} else {
s.unread_count = s.unread_count.saturating_add(1);
}
});
self.publish_event(CoreEvent::MessageFlagsChanged { message_id, read });
}
CoreCommand::ReportError { message } => {
self.update_state(|s| s.last_error = Some(message.clone()));
self.publish_event(CoreEvent::Error { message });
}
CoreCommand::ReloadFilters => {}
CoreCommand::Shutdown => {
self.update_state(|s| s.status = EngineStatus::ShuttingDown);
self.publish_event(CoreEvent::ShuttingDown);
@@ -314,7 +306,6 @@ mod tests {
let seed = AppState {
status: EngineStatus::Idle,
accounts_loaded: 3,
unread_count: 5,
last_error: None,
};
let bus = EventBus::with_initial_state(seed.clone());
@@ -406,18 +397,14 @@ mod tests {
}
#[tokio::test]
async fn process_mark_read_adjusts_unread_count_both_directions() {
let bus = EventBus::with_initial_state(AppState {
unread_count: 1,
..AppState::default()
});
async fn process_mark_read_emits_flags_changed_for_each_call() {
let bus = EventBus::new();
let mut events = bus.subscribe_events();
bus.process_command(CoreCommand::MarkRead {
message_id: "msg-1".to_string(),
read: false,
});
assert_eq!(bus.current_state().unread_count, 2);
assert_eq!(
events.recv().await.expect("event received"),
CoreEvent::MessageFlagsChanged {
@@ -430,15 +417,13 @@ mod tests {
message_id: "msg-1".to_string(),
read: true,
});
bus.process_command(CoreCommand::MarkRead {
message_id: "msg-1".to_string(),
read: true,
});
bus.process_command(CoreCommand::MarkRead {
message_id: "msg-1".to_string(),
read: true,
});
assert_eq!(bus.current_state().unread_count, 0);
assert_eq!(
events.recv().await.expect("event received"),
CoreEvent::MessageFlagsChanged {
message_id: "msg-1".to_string(),
read: true
}
);
}
#[tokio::test]
crates/nuncio-core/src/model.rs
+728 -7
View File
@@ -2,6 +2,7 @@
use bytes::Bytes;
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
/// Real-time daemon health, performance metrics, and telemetry payload.
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
@@ -60,15 +61,185 @@ pub struct Attachment {
pub content: Bytes,
}
/// Email message domain entity owned by Nuncio core.
/// Which tier of the identity precedence produced a [`MessageIdentity`].
///
/// Persisted alongside the key so a later pass can tell a server-assigned
/// identity from a locally-derived fallback without recomputing it, and so a
/// store can be audited for how much of it rests on the weakest tier.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum IdentitySource {
/// RFC 8474 OBJECTID `EMAILID` -- server-assigned and stable across folders.
EmailId,
/// Gmail's `X-GM-MSGID`, the same guarantee by a vendor extension.
GmailMsgId,
/// `Message-ID` **and** a content hash together. Never the header alone.
MessageIdContent,
/// Folder-scoped last resort: the message is only identified by where it
/// currently sits, so the same mail in another folder is a second message.
Surrogate,
}
impl IdentitySource {
/// Stable token persisted in `messages.identity_source`.
pub fn as_str(&self) -> &'static str {
match self {
Self::EmailId => "emailid",
Self::GmailMsgId => "gmsgid",
Self::MessageIdContent => "msgid_content",
Self::Surrogate => "surrogate",
}
}
/// Parse a token written by [`Self::as_str`]. Returns `None` for anything
/// else so an unrecognised value fails loudly rather than defaulting to a
/// tier that would overstate how trustworthy the key is.
///
/// Deliberately not `std::str::FromStr`: that trait forces a `Result` and
/// therefore an error type, when the only failure here is "not one of four
/// known tokens" and `None` already says exactly that.
pub fn from_token(s: &str) -> Option<Self> {
match s {
"emailid" => Some(Self::EmailId),
"gmsgid" => Some(Self::GmailMsgId),
"msgid_content" => Some(Self::MessageIdContent),
"surrogate" => Some(Self::Surrogate),
_ => None,
}
}
}
/// A content-addressed message key together with the tier that produced it.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct MessageIdentity {
/// Hex SHA-256 key identifying the message within its account.
pub key: String,
/// Which precedence tier produced `key`.
pub source: IdentitySource,
}
/// Server-supplied identity hints for one message, in precedence order.
///
/// Borrowed rather than owned because every caller already holds these as
/// fields on a parsed response and only needs them for the duration of the
/// key derivation.
#[derive(Debug, Clone, Copy, Default)]
pub struct RemoteIdentity<'a> {
/// RFC 8474 `EMAILID` from a `FETCH ... (EMAILID)`, if the server has OBJECTID.
pub email_id: Option<&'a str>,
/// Gmail `X-GM-MSGID`, if the server advertises `X-GM-EXT-1`.
pub gm_msgid: Option<&'a str>,
/// Normalized `Message-ID` (see [`Email::normalize_message_id`]).
pub message_id: Option<&'a str>,
/// Hex SHA-256 over the full RFC822 octets (see [`Email::content_hash_of`]).
pub content_hash: Option<&'a str>,
}
/// One occupancy of a message in one mailbox.
///
/// A message can sit in several folders at once (an IMAP `COPY`, a Gmail label,
/// a message that is both in a thread's folder and in `\All`). The addressing
/// coordinates and the read flag are properties of the *occupancy*, not of the
/// message: IMAP `\Seen` is per-mailbox, and a UID is only meaningful inside one
/// `uid_validity` scope of one folder.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct Email {
/// Unique message identifier.
pub id: String,
/// Account identifier owning the message.
pub struct Placement {
/// Account owning the mailbox.
pub account_id: String,
/// Mailbox folder identifier (e.g. "inbox").
pub folder_id: String,
/// The IMAP UIDVALIDITY the `remote_id` was captured under, as a decimal
/// string; a stable sentinel for protocols without one (JMAP). In the
/// primary key because UIDs restart at 1 after a bump and would otherwise
/// collide with rows written before it.
pub uid_validity: String,
/// Protocol-native id addressing the message in this mailbox: the IMAP UID
/// as a decimal string, or the JMAP Email object id.
pub remote_id: String,
/// Read/unread state **in this mailbox**.
pub read: bool,
}
/// The four coordinates that address one mailbox occupancy.
///
/// The addressing half of a [`Placement`], without the read flag: enough to
/// name an occupancy, not enough to describe it. A protocol backend reports
/// these when a folder stops mentioning a message, and the store deletes
/// exactly the rows they name -- which is why the type lives here in `core`
/// rather than in either crate. What a folder stops reporting is an occupancy,
/// never a message; a message goes only when its last placement does.
///
/// `Ord` is derived because the sync path sorts and dedups batches of these
/// before deleting; `Hash`/`Eq` because it is also the key of the
/// already-present set used to classify a fetched chunk.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash, Serialize, Deserialize)]
pub struct PlacementKey {
/// Account owning the mailbox.
pub account_id: String,
/// Mailbox folder identifier.
pub folder_id: String,
/// The UIDVALIDITY generation `remote_id` was captured under.
pub uid_validity: String,
/// Protocol-native id addressing the message in this mailbox.
pub remote_id: String,
}
/// A mutation the server refused because another client had already changed the
/// message, recorded as durable state rather than as a passing notification.
///
/// A conflict is raised only on positive evidence from the server; an
/// unverifiable outcome is never one. Because the intent was formed against
/// state that no longer exists, a human has to decide what happens next, and
/// that decision may come long after the event. The event stream is a lossy
/// broadcast -- it drops on lag and carries no sequence number to resume from --
/// so it can only ever be an optimisation over a queryable table, never the
/// record itself.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct MutationConflict {
/// Stable identifier of this conflict record.
pub id: String,
/// The outbox mutation that conflicted.
pub mutation_id: String,
/// Content-addressed key of the message the mutation targeted.
pub message_id: String,
/// The operation that was refused (`MOVE`, `DELETE`, `FLAG`, ...).
pub mutation_type: String,
/// The occupancy the mutation was aimed at, when the row recorded one.
pub placement: Option<PlacementKey>,
/// The server's positive evidence that another client changed the message.
pub observed: String,
/// Unix timestamp (seconds) when the conflict was detected.
pub detected_at: i64,
/// Unix timestamp (seconds) when a human resolved it, if they have.
pub resolved_at: Option<i64>,
/// Free-form note recording how the conflict was resolved.
pub resolution: Option<String>,
}
impl Placement {
/// The addressing coordinates of this occupancy, dropping the read flag.
pub fn key(&self) -> PlacementKey {
PlacementKey {
account_id: self.account_id.clone(),
folder_id: self.folder_id.clone(),
uid_validity: self.uid_validity.clone(),
remote_id: self.remote_id.clone(),
}
}
}
/// Email message domain entity owned by Nuncio core.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct Email {
/// Opaque, account-scoped identity for the message itself.
///
/// Derived by [`Email::derive_message_key`], so it is independent of which
/// folder the message currently occupies: moving a message between folders
/// does not change it. Where the message actually sits, and its read state
/// there, live in [`Placement`] rows instead. Callers must treat this as an
/// opaque token -- it carries no decodable structure.
pub id: String,
/// Account identifier owning the message.
pub account_id: String,
/// Subject line.
pub subject: String,
/// Sender address (e.g. "alice@nuncio.mx").
@@ -77,14 +248,205 @@ pub struct Email {
pub recipient: String,
/// Unix timestamp of message arrival.
pub received_at: i64,
/// Read/unread flag status.
pub read: bool,
/// Plaintext message body.
pub body_plain: Option<String>,
/// HTML message body.
pub body_html: Option<String>,
/// List of attached files.
pub attachments: Vec<Attachment>,
/// The message's RFC 5322 `Message-ID`, normalized by
/// [`Email::normalize_message_id`], or `None` when the message carried
/// none (the header is `SHOULD`, not `MUST`) or the transport did not
/// surface it.
///
/// Captured but not yet used for identity. It is a *hint* toward a
/// folder-independent identity, never an identity on its own: the value is
/// public (it is echoed in the `References` of every reply) and forgeable,
/// so keying storage on it alone would let a crafted message address, and
/// overwrite, one the user already trusts.
pub message_id: Option<String>,
/// Hex SHA-256 over the full RFC822 octets the message was parsed from, or
/// `None` when only headers or an envelope were available.
///
/// Pairs with [`Email::message_id`]: the same `Message-ID` carrying
/// different content is two messages, not one. That case is ordinary --
/// a mailing-list copy and a direct copy of the same mail share a
/// `Message-ID` while differing in headers and footer, and a draft keeps
/// one `Message-ID` across every save.
pub content_hash: Option<String>,
}
impl Email {
/// Compute the deterministic, collision-resistant surrogate id for a
/// message from its addressing coordinates.
///
/// The four coordinates together uniquely identify a stored message: a
/// protocol-native `remote_id` (IMAP UID / JMAP object id) is only unique
/// within one `uid_validity` scope of one `folder_id` of one `account_id`,
/// so all four must participate. Each field is fed length-prefixed so that
/// no two distinct coordinate tuples can serialize to the same byte stream
/// (e.g. `("a", "bc")` and `("ab", "c")` must not collide). The SHA-256
/// digest is hex-encoded, yielding a stable opaque token that leaks none of
/// its inputs' encoding on the wire.
pub fn surrogate_id(
account_id: &str,
folder_id: &str,
uid_validity: &str,
remote_id: &str,
) -> String {
let mut hasher = Sha256::new();
for field in [account_id, folder_id, uid_validity, remote_id] {
hasher.update((field.len() as u64).to_le_bytes());
hasher.update(field.as_bytes());
}
hex::encode(hasher.finalize())
}
/// Canonicalize a raw `Message-ID` header value.
///
/// Every engine syncing an account must derive byte-identical values from
/// the same header, or two caches that agree in substance will disagree in
/// bytes and no comparison between them can be trusted. The rules are
/// therefore fixed and deliberately small:
///
/// 1. Remove **all** ASCII whitespace. A `msg-id` cannot legally contain
/// any, so this unfolds a wrapped header without needing to know how the
/// transport folded it -- and transports differ.
/// 2. Strip one surrounding pair of angle brackets, which servers include
/// or omit inconsistently between `ENVELOPE` and a raw header.
/// 3. Preserve case. Only the domain half is case-insensitive per RFC 5322
/// §3.6.4; lowercasing the whole value would merge ids that the
/// originating server considers distinct, and the same server returns
/// the same bytes to every engine anyway.
///
/// Returns `None` for an absent or empty value so a missing header stays
/// distinguishable from a present-but-empty one.
pub fn normalize_message_id(raw: &str) -> Option<String> {
let stripped: String = raw.chars().filter(|c| !c.is_ascii_whitespace()).collect();
let stripped = stripped
.strip_prefix('<')
.and_then(|s| s.strip_suffix('>'))
.unwrap_or(&stripped);
if stripped.is_empty() {
None
} else {
Some(stripped.to_string())
}
}
/// Hex SHA-256 over a message's full RFC822 octets, for the
/// `content_hash` field.
///
/// Deterministic for identical bytes, which is what makes it usable as the
/// tiebreaker against a duplicated `Message-ID`. It is *not* stable across
/// servers that re-serialize a message, so it distinguishes content within
/// an account rather than identifying a message globally.
pub fn content_hash_of(raw_bytes: &[u8]) -> String {
let mut hasher = Sha256::new();
hasher.update(raw_bytes);
hex::encode(hasher.finalize())
}
/// Domain-separated, length-prefixed digest over an identity tier's inputs.
///
/// The tier label participates in the hash so that a value appearing in two
/// different tiers (a server whose `EMAILID` happens to equal another's
/// `X-GM-MSGID`) cannot produce one key. Length prefixes stop two distinct
/// field tuples serializing to the same byte stream, exactly as in
/// [`Email::surrogate_id`].
fn identity_digest(tier: &str, account_id: &str, parts: &[&str]) -> String {
let mut hasher = Sha256::new();
for field in [tier, account_id].into_iter().chain(parts.iter().copied()) {
hasher.update((field.len() as u64).to_le_bytes());
hasher.update(field.as_bytes());
}
hex::encode(hasher.finalize())
}
/// Derive the account-scoped message key by identity precedence.
///
/// `EMAILID` → `X-GM-MSGID` → (`Message-ID` + content hash) → surrogate.
/// Every tier is account-scoped: identity is only ever claimed within the
/// account that fetched it, so one account's server cannot mint a key that
/// addresses another account's mail.
///
/// The third tier requires **both** a `Message-ID` and a content hash. A
/// `Message-ID` is public -- it is echoed in the `References` of every reply
/// -- and forgeable, so keying on it alone would let a crafted message
/// address, and with an upsert overwrite, a message the user already trusts,
/// and would evade filtering by never being classified as new. Pairing it
/// with the content hash also gets the ordinary cases right: a mailing-list
/// copy and a direct copy share a `Message-ID` but differ in content, and so
/// are correctly two messages.
///
/// The surrogate tier is folder-scoped and therefore *not* stable across a
/// move; it is the honest answer when the server offered nothing better.
///
/// # What convergence across engine types does and does not follow from
///
/// Precedence makes a key reproducible from the same inputs, not from the
/// same *message*. Two engines converge exactly when they enter at the same
/// tier with the same value.
///
/// They do when the server publishes a stable object id and both engines
/// ask for it. The IMAP engine requests RFC 8474 `EMAILID` from an
/// `OBJECTID` server and `X-GM-MSGID` from an `X-GM-EXT-1` one, and JMAP's
/// Email object id is the same identifier RFC 8474 exposes to IMAP -- so
/// one message synced through both protocols yields one key. That is the
/// whole point of the OBJECTID extension.
///
/// They do not when the server offers neither. IMAP then falls to the
/// `Message-ID`+content tier or the surrogate, both derived from what the
/// message or the mailbox happened to contain, while a JMAP engine against
/// the same account still enters at the top tier from a server-assigned id.
/// Those key spaces are disjoint, and no comparison across them means
/// anything. The surrogate additionally is not stable across a *move*, even
/// within one engine.
///
/// Object ids are also only comparable within one account on one server:
/// RFC 8474 section 4 scopes uniqueness that way, which is why every tier
/// here is account-scoped.
pub fn derive_message_key(
account_id: &str,
remote: RemoteIdentity<'_>,
folder_id: &str,
uid_validity: &str,
remote_id: &str,
) -> MessageIdentity {
if let Some(email_id) = remote.email_id.filter(|v| !v.is_empty()) {
return MessageIdentity {
key: Self::identity_digest("emailid", account_id, &[email_id]),
source: IdentitySource::EmailId,
};
}
if let Some(gm_msgid) = remote.gm_msgid.filter(|v| !v.is_empty()) {
return MessageIdentity {
key: Self::identity_digest("gmsgid", account_id, &[gm_msgid]),
source: IdentitySource::GmailMsgId,
};
}
if let (Some(message_id), Some(content_hash)) = (
remote.message_id.filter(|v| !v.is_empty()),
remote.content_hash.filter(|v| !v.is_empty()),
) {
return MessageIdentity {
key: Self::identity_digest(
"msgid_content",
account_id,
&[message_id, content_hash],
),
source: IdentitySource::MessageIdContent,
};
}
MessageIdentity {
key: Self::identity_digest(
"surrogate",
account_id,
&[folder_id, uid_validity, remote_id],
),
source: IdentitySource::Surrogate,
}
}
}
/// Mailbox folder entity.
@@ -168,4 +530,363 @@ mod tests {
assert_eq!(parsed.status, "HEALTHY");
assert_eq!(parsed.total_emails, 15000);
}
#[test]
fn surrogate_id_is_deterministic_and_opaque() {
let a = Email::surrogate_id("acct-1", "INBOX", "42", "5");
let b = Email::surrogate_id("acct-1", "INBOX", "42", "5");
assert_eq!(a, b, "identical coordinates must hash identically");
// A 32-byte SHA-256 digest hex-encodes to 64 chars, and the token must
// not leak the protocol-native id it was built from.
assert_eq!(a.len(), 64);
assert!(a.chars().all(|c| c.is_ascii_hexdigit()));
assert!(!a.contains("imap-uid-"));
}
#[test]
fn surrogate_id_distinguishes_the_cross_folder_and_cross_account_collision() {
// The same protocol UID (5) in two different folders of one account, or
// in two different accounts, must NOT collapse to the same surrogate.
let inbox = Email::surrogate_id("acct-1", "INBOX", "42", "5");
let sent = Email::surrogate_id("acct-1", "Sent", "7", "5");
let other_account = Email::surrogate_id("acct-2", "INBOX", "42", "5");
assert_ne!(inbox, sent, "same UID in two folders must not collide");
assert_ne!(
inbox, other_account,
"same UID in two accounts must not collide"
);
// Field boundaries are unambiguous: shifting a delimiter-like split
// across two adjacent fields yields a different id.
assert_ne!(
Email::surrogate_id("a", "bc", "1", "1"),
Email::surrogate_id("ab", "c", "1", "1"),
);
}
#[test]
fn normalize_message_id_unfolds_and_unwraps_to_one_canonical_form() {
// Every spelling a transport might hand us for the same id must
// collapse to identical bytes, or two engines holding the same message
// cannot be shown to agree.
let canonical = "abc123@mail.example.com";
for raw in [
"<abc123@mail.example.com>",
"abc123@mail.example.com",
" <abc123@mail.example.com> ",
"<abc123@mail.\r\n example.com>",
"<abc123@mail.\t example.com>",
] {
assert_eq!(
Email::normalize_message_id(raw).as_deref(),
Some(canonical),
"input {raw:?} did not canonicalize"
);
}
}
#[test]
fn normalize_message_id_preserves_case_and_rejects_empty() {
// Only the domain half is case-insensitive per RFC 5322 3.6.4;
// lowercasing wholesale would merge ids the origin considers distinct.
assert_eq!(
Email::normalize_message_id("<AbC@Example.COM>").as_deref(),
Some("AbC@Example.COM")
);
assert_eq!(Email::normalize_message_id(""), None);
assert_eq!(Email::normalize_message_id("<>"), None);
assert_eq!(Email::normalize_message_id(" "), None);
}
#[test]
fn content_hash_distinguishes_identical_message_ids_with_different_bodies() {
// The case the hash exists for: a list copy and a direct copy share a
// Message-ID but are not the same message.
let direct = b"Message-ID: <shared@example.com>\r\n\r\nbody";
let via_list = b"Message-ID: <shared@example.com>\r\nList-Id: l\r\n\r\nbody\n--\nfooter";
assert_ne!(
Email::content_hash_of(direct),
Email::content_hash_of(via_list)
);
assert_eq!(
Email::content_hash_of(direct),
Email::content_hash_of(direct),
"the hash must be deterministic for identical octets"
);
}
#[test]
fn message_key_precedence_prefers_emailid_over_everything_below_it() {
let remote = RemoteIdentity {
email_id: Some("M00000001"),
gm_msgid: Some("1234567890"),
message_id: Some("a@b.example"),
content_hash: Some("deadbeef"),
};
let id = Email::derive_message_key("acct-1", remote, "INBOX", "42", "5");
assert_eq!(id.source, IdentitySource::EmailId);
// Same EMAILID in a different folder is the SAME message.
let elsewhere = Email::derive_message_key(
"acct-1",
RemoteIdentity {
email_id: Some("M00000001"),
gm_msgid: None,
message_id: None,
content_hash: None,
},
"Archive",
"99",
"7",
);
assert_eq!(id.key, elsewhere.key);
}
#[test]
fn message_key_is_account_scoped_at_every_tier() {
let emailid = || RemoteIdentity {
email_id: Some("M00000001"),
gm_msgid: None,
message_id: None,
content_hash: None,
};
let a = Email::derive_message_key("acct-1", emailid(), "INBOX", "42", "5");
let b = Email::derive_message_key("acct-2", emailid(), "INBOX", "42", "5");
assert_ne!(
a.key, b.key,
"the same EMAILID in two accounts must not collide"
);
let gmsgid = || RemoteIdentity {
email_id: None,
gm_msgid: Some("1234567890"),
message_id: None,
content_hash: None,
};
let a = Email::derive_message_key("acct-1", gmsgid(), "INBOX", "42", "5");
let b = Email::derive_message_key("acct-2", gmsgid(), "INBOX", "42", "5");
assert_ne!(
a.key, b.key,
"the same X-GM-MSGID in two accounts must not collide"
);
let msgid_content = || RemoteIdentity {
email_id: None,
gm_msgid: None,
message_id: Some("a@b.example"),
content_hash: Some("deadbeef"),
};
let a = Email::derive_message_key("acct-1", msgid_content(), "INBOX", "42", "5");
let b = Email::derive_message_key("acct-2", msgid_content(), "INBOX", "42", "5");
assert_ne!(
a.key, b.key,
"the same Message-ID and content hash in two accounts must not collide"
);
let surrogate = RemoteIdentity {
email_id: None,
gm_msgid: None,
message_id: None,
content_hash: None,
};
let a = Email::derive_message_key("acct-1", surrogate, "INBOX", "42", "5");
let b = Email::derive_message_key("acct-2", surrogate, "INBOX", "42", "5");
assert_ne!(
a.key, b.key,
"the same folder/uidvalidity/remote-id in two accounts must not collide"
);
}
#[test]
fn a_message_id_without_a_content_hash_never_reaches_the_content_tier() {
// The non-negotiable: a Message-ID is public and forgeable, so it must
// never key storage on its own. Missing content hash falls through to the
// folder-scoped surrogate rather than trusting the header alone.
let id = Email::derive_message_key(
"acct-1",
RemoteIdentity {
email_id: None,
gm_msgid: None,
message_id: Some("a@b.example"),
content_hash: None,
},
"INBOX",
"42",
"5",
);
assert_eq!(id.source, IdentitySource::Surrogate);
let surrogate_only = Email::derive_message_key(
"acct-1",
RemoteIdentity {
email_id: None,
gm_msgid: None,
message_id: None,
content_hash: None,
},
"INBOX",
"42",
"5",
);
assert_eq!(id.key, surrogate_only.key);
}
#[test]
fn the_same_message_id_with_different_content_is_two_messages() {
let mk = |hash: &str| {
Email::derive_message_key(
"acct-1",
RemoteIdentity {
email_id: None,
gm_msgid: None,
message_id: Some("a@b.example"),
content_hash: Some(hash),
},
"INBOX",
"42",
"5",
)
};
let list_copy = mk("1111");
let direct_copy = mk("2222");
assert_eq!(list_copy.source, IdentitySource::MessageIdContent);
assert_ne!(list_copy.key, direct_copy.key);
}
#[test]
fn tiers_are_domain_separated_so_a_shared_value_cannot_collide_across_them() {
let as_emailid = Email::derive_message_key(
"acct-1",
RemoteIdentity {
email_id: Some("X"),
gm_msgid: None,
message_id: None,
content_hash: None,
},
"INBOX",
"42",
"5",
);
let as_gmsgid = Email::derive_message_key(
"acct-1",
RemoteIdentity {
email_id: None,
gm_msgid: Some("X"),
message_id: None,
content_hash: None,
},
"INBOX",
"42",
"5",
);
assert_ne!(as_emailid.key, as_gmsgid.key);
}
fn sample_placement() -> Placement {
Placement {
account_id: "acct-1".to_string(),
folder_id: "INBOX".to_string(),
uid_validity: "42".to_string(),
remote_id: "5".to_string(),
read: true,
}
}
#[test]
fn placement_key_carries_every_addressing_coordinate_and_drops_the_read_flag() {
// The key must reproduce all four coordinates exactly. A key that
// dropped or reordered one would address a different occupancy, and a
// sync diffing reported-against-stored would delete live mail.
let placement = sample_placement();
let key = placement.key();
assert_eq!(key.account_id, "acct-1");
assert_eq!(key.folder_id, "INBOX");
assert_eq!(key.uid_validity, "42");
assert_eq!(key.remote_id, "5");
// The read flag is the one field a key must NOT carry: it describes an
// occupancy rather than naming it, so including it would make the same
// occupancy compare unequal to itself after the user opened the message.
let mut unread = placement.clone();
unread.read = false;
assert_ne!(placement, unread);
assert_eq!(placement.key(), unread.key());
}
#[test]
fn placement_keys_distinguish_every_coordinate_that_can_differ() {
let base = sample_placement().key();
for mutated in [
PlacementKey {
account_id: "acct-2".to_string(),
..base.clone()
},
PlacementKey {
folder_id: "Archive".to_string(),
..base.clone()
},
PlacementKey {
uid_validity: "43".to_string(),
..base.clone()
},
PlacementKey {
remote_id: "6".to_string(),
..base.clone()
},
] {
assert_ne!(
base, mutated,
"changing one coordinate must yield a different occupancy"
);
}
// Usable as a set key and sortable, which is what the sync path needs
// to diff a reported UID set against what the store holds.
let mut set = std::collections::HashSet::new();
assert!(set.insert(base.clone()));
assert!(!set.insert(base.clone()), "an equal key must not re-insert");
let mut keys = [
PlacementKey {
remote_id: "6".to_string(),
..base.clone()
},
base.clone(),
];
keys.sort();
assert_eq!(keys[0], base);
}
#[test]
fn placement_key_round_trips_through_serde() {
let key = sample_placement().key();
let json = serde_json::to_string(&key).unwrap();
let parsed: PlacementKey = serde_json::from_str(&json).unwrap();
assert_eq!(parsed, key);
}
#[test]
fn placement_round_trips_through_serde_including_its_read_flag() {
let placement = sample_placement();
let json = serde_json::to_string(&placement).unwrap();
let parsed: Placement = serde_json::from_str(&json).unwrap();
assert_eq!(parsed, placement);
assert!(parsed.read);
}
#[test]
fn identity_source_round_trips_through_its_stored_string() {
for source in [
IdentitySource::EmailId,
IdentitySource::GmailMsgId,
IdentitySource::MessageIdContent,
IdentitySource::Surrogate,
] {
assert_eq!(IdentitySource::from_token(source.as_str()), Some(source));
}
assert_eq!(
IdentitySource::from_token("not_a_tier"),
None,
"an unrecognised token must fail loudly, never default to a tier"
);
}
}
crates/nuncio-core/src/plugin.rs
-135
View File
@@ -1,135 +0,0 @@
//! Sandboxed Plugin Runtime Engine for custom automation scripts.
use serde::{Deserialize, Serialize};
use thiserror::Error;
/// Event triggers that plugins can subscribe to.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum PluginHook {
OnMailReceived,
OnMailSending,
OnCalendarRsvp,
OnFilterRuleExecuted,
}
/// Metadata definition of an installed plugin package.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct PluginManifest {
pub id: String,
pub name: String,
pub version: String,
pub author: String,
pub hooks: Vec<PluginHook>,
pub permissions: Vec<String>,
}
/// Errors emitted by the plugin runtime.
#[derive(Error, Debug, PartialEq, Eq)]
pub enum PluginError {
#[error("plugin execution failed: {0}")]
ExecutionFailed(String),
#[error("permission denied: {0}")]
PermissionDenied(String),
#[error("plugin manifest invalid: {0}")]
InvalidManifest(String),
}
/// Sandboxed Plugin Runtime Manager.
pub struct PluginRuntime {
installed_plugins: Vec<PluginManifest>,
}
impl PluginRuntime {
/// Create a new plugin runtime instance.
pub fn new() -> Self {
Self {
installed_plugins: Vec::new(),
}
}
/// Register a new sandboxed plugin manifest.
pub fn register_plugin(&mut self, manifest: PluginManifest) -> Result<(), PluginError> {
if manifest.id.is_empty() {
return Err(PluginError::InvalidManifest(
"missing plugin id".to_string(),
));
}
self.installed_plugins.push(manifest);
Ok(())
}
/// List all currently installed plugin manifests.
pub fn list_plugins(&self) -> &[PluginManifest] {
&self.installed_plugins
}
/// Execute subscribed plugins for a specific event hook.
pub fn trigger_hook(
&self,
hook: PluginHook,
event_payload_json: &str,
) -> Vec<Result<String, PluginError>> {
self.installed_plugins
.iter()
.filter(|p| p.hooks.contains(&hook))
.map(|p| {
if event_payload_json.is_empty() {
Err(PluginError::ExecutionFailed("empty payload".to_string()))
} else {
Ok(format!(
"plugin {} executed successfully for hook {:?}",
p.id, hook
))
}
})
.collect()
}
}
impl Default for PluginRuntime {
fn default() -> Self {
Self::new()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn register_and_list_plugins() {
let mut runtime = PluginRuntime::new();
let manifest = PluginManifest {
id: "auto-tagger".to_string(),
name: "Auto Email Tagger".to_string(),
version: "1.0.0".to_string(),
author: "KofTwentyTwo".to_string(),
hooks: vec![PluginHook::OnMailReceived],
permissions: vec!["mail:read".to_string()],
};
runtime.register_plugin(manifest).expect("register");
assert_eq!(runtime.list_plugins().len(), 1);
}
#[test]
fn trigger_hook_executes_registered_plugins() {
let mut runtime = PluginRuntime::new();
runtime
.register_plugin(PluginManifest {
id: "p1".to_string(),
name: "P1".to_string(),
version: "0.1.0".to_string(),
author: "Dev".to_string(),
hooks: vec![PluginHook::OnMailReceived],
permissions: vec![],
})
.unwrap();
let results = runtime.trigger_hook(PluginHook::OnMailReceived, r#"{"msg":"hello"}"#);
assert_eq!(results.len(), 1);
assert!(results[0].is_ok());
}
}
crates/nuncio-core/src/redact.rs
+203
View File
@@ -0,0 +1,203 @@
//! Secret-safe value wrapper and the logging redaction policy it enforces.
//!
//! # Logging redaction policy
//!
//! Telemetry (`tracing` spans, events, and any `Debug`/`Display`/`serde`
//! rendering that can reach a log sink) is a plaintext side channel. The
//! following values MUST NEVER appear in it, in any encoding:
//!
//! - message bodies, subjects, and snippets;
//! - sender/recipient email addresses and any other message correspondents;
//! - account credentials (passwords, app-specific passwords/tokens);
//! - gRPC bearer tokens;
//! - keyring secret *values* (the material stored in the OS vault) and any
//! derived key material (encryption keys, HMAC signing keys);
//! - vCard / iCal personal data (contact names, phones, addresses; event
//! summaries, locations, attendees, descriptions).
//!
//! The following identifying, non-sensitive values ARE allowed in telemetry,
//! because operating the daemon is impossible without them and none of them
//! reveals content or grants access:
//!
//! - opaque ids (account id, message id, rule id, folder id) and UIDs;
//! - counts, sizes, and durations;
//! - operation / rule / transport / action *kinds* (the discriminant, never
//! its payload);
//! - folder names, hostnames, ports, and the per-request `request_id`;
//! - the keyring *lookup key* (e.g. `nuncio/<account-id>`) — an identifier
//! that names a vault entry but is not itself the stored secret.
//!
//! [`Redacted<T>`] is the mechanical enforcement point for the first list: a
//! field whose type is `Redacted<T>` cannot leak through a derived `Debug`, a
//! `Display`, or `serde` serialization, because each of those renders a fixed
//! placeholder instead of the inner value. The secret is reachable only via an
//! explicit [`Redacted::expose_secret`] / [`Redacted::into_inner`] call, which
//! is easy to audit and never happens implicitly.
use std::fmt;
use serde::de::{Deserialize, Deserializer};
use serde::ser::{Serialize, Serializer};
/// The single placeholder rendered wherever a redacted value would otherwise
/// appear (`Debug`, `Display`, and `serde`).
const PLACEHOLDER: &str = "<redacted>";
/// A wrapper for a sensitive value that never renders its contents through
/// `Debug`, `Display`, or `serde`.
///
/// Wrapping a credential/token/key-material field in `Redacted<T>` makes it
/// impossible to leak that field into telemetry by accident: the compiler still
/// lets a struct derive `Debug`/`Serialize`, but the wrapped field renders as
/// `<redacted>`. Reaching the real value is deliberate and greppable — it
/// requires calling [`expose_secret`](Self::expose_secret) or
/// [`into_inner`](Self::into_inner).
///
/// Equality is derived over the inner value so wrapped fields remain comparable
/// in tests and config diffs; it is intentionally not constant-time and is not
/// meant for authenticating a secret against an attacker-supplied guess.
#[derive(Clone, PartialEq, Eq)]
pub struct Redacted<T>(T);
impl<T> Redacted<T> {
/// Wrap `value` so it is protected from accidental disclosure in telemetry.
pub fn new(value: T) -> Self {
Self(value)
}
/// Borrow the wrapped secret. Naming the call `expose_secret` keeps every
/// deliberate disclosure of the value greppable and reviewable.
pub fn expose_secret(&self) -> &T {
&self.0
}
/// Consume the wrapper and return the owned secret.
pub fn into_inner(self) -> T {
self.0
}
}
impl<T> From<T> for Redacted<T> {
fn from(value: T) -> Self {
Self(value)
}
}
/// Renders `<redacted>` regardless of the inner value, so a derived `Debug` on
/// any containing struct can never print the secret.
impl<T> fmt::Debug for Redacted<T> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str(PLACEHOLDER)
}
}
/// Renders `<redacted>`, so interpolating the value with `{}` (e.g. into a
/// log message or error) discloses nothing.
impl<T> fmt::Display for Redacted<T> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.write_str(PLACEHOLDER)
}
}
/// Serializes the placeholder, never the inner value. This closes the
/// `serde_json`/structured-logging leak path: even serializing a whole struct
/// that contains a `Redacted<T>` field emits `<redacted>` for it. As a result
/// serialization is deliberately lossy — a `Redacted<T>` must never be used for
/// a field whose real value has to survive a serialization round trip.
impl<T> Serialize for Redacted<T> {
fn serialize<S>(&self, serializer: S) -> Result<S::Ok, S::Error>
where
S: Serializer,
{
serializer.serialize_str(PLACEHOLDER)
}
}
/// Deserializes transparently into `Redacted<T>` by deserializing the inner
/// `T`. This exists so structs that hold a `Redacted<T>` can still derive
/// `Deserialize`; it does not reverse [`Serialize`], which is lossy by design.
impl<'de, T> Deserialize<'de> for Redacted<T>
where
T: Deserialize<'de>,
{
fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
where
D: Deserializer<'de>,
{
Ok(Self(T::deserialize(deserializer)?))
}
}
#[cfg(test)]
mod tests {
use super::*;
const SENTINEL: &str = "super-secret-app-token-9f3a";
#[test]
fn debug_renders_placeholder_not_the_secret() {
let wrapped = Redacted::new(SENTINEL.to_string());
let rendered = format!("{wrapped:?}");
assert_eq!(rendered, PLACEHOLDER);
assert!(!rendered.contains(SENTINEL));
}
#[test]
fn display_renders_placeholder_not_the_secret() {
let wrapped = Redacted::new(SENTINEL.to_string());
let rendered = format!("{wrapped}");
assert_eq!(rendered, PLACEHOLDER);
assert!(!rendered.contains(SENTINEL));
}
#[test]
fn derived_debug_of_a_containing_struct_does_not_leak() {
// Fields are exercised only through the derived `Debug`, which
// dead-code analysis deliberately ignores; the test asserts on that
// rendering, so silence the false-positive unread-field lint.
#[derive(Debug)]
#[allow(dead_code)]
struct Config {
username: String,
auth_token: Redacted<String>,
}
let cfg = Config {
username: "jmaes".to_string(),
auth_token: Redacted::new(SENTINEL.to_string()),
};
let rendered = format!("{cfg:?}");
assert!(rendered.contains("jmaes"));
assert!(!rendered.contains(SENTINEL));
assert!(rendered.contains(PLACEHOLDER));
}
#[test]
fn serialize_emits_placeholder_not_the_secret() {
let wrapped = Redacted::new(SENTINEL.to_string());
let json = serde_json::to_string(&wrapped).expect("serialize");
assert!(!json.contains(SENTINEL));
assert_eq!(json, format!("\"{PLACEHOLDER}\""));
}
#[test]
fn expose_and_into_inner_return_the_real_value() {
let wrapped = Redacted::new(SENTINEL.to_string());
assert_eq!(wrapped.expose_secret(), SENTINEL);
assert_eq!(wrapped.into_inner(), SENTINEL);
}
#[test]
fn deserialize_wraps_the_inner_value() {
let wrapped: Redacted<String> =
serde_json::from_str(&format!("\"{SENTINEL}\"")).expect("deserialize");
assert_eq!(wrapped.expose_secret(), SENTINEL);
}
#[test]
fn from_and_equality() {
let a: Redacted<String> = SENTINEL.to_string().into();
let b = Redacted::new(SENTINEL.to_string());
assert_eq!(a, b);
assert_ne!(a, Redacted::new("other".to_string()));
}
}
crates/nuncio-core/src/update.rs
+168 -23
View File
@@ -40,6 +40,11 @@ pub enum UpdateError {
#[error("checksum file error: {0}")]
ChecksumFileError(String),
/// No checksum could be positively verified for the download, either because the release
/// has no `SHA256SUMS.txt` asset or because that asset could not be fetched.
#[error("checksum unavailable: {0}")]
ChecksumUnavailable(String),
/// Archive unpacking failure (tar.gz or zip).
#[error("archive extraction failed: {0}")]
ArchiveError(String),
@@ -223,10 +228,41 @@ impl UpdateEngine {
}
/// Download release package bytes and verify SHA-256 checksum against `SHA256SUMS.txt`.
///
/// This is the supply-chain trust boundary for the auto-updater: the only way to reach
/// `Ok` is an exact SHA-256 match against a successfully fetched `SHA256SUMS.txt`. Any
/// situation where the checksum cannot be positively verified — no checksum asset on the
/// release, a failed fetch of that asset, a missing filename entry, or a hash mismatch —
/// must fail closed rather than silently returning unverified bytes.
pub async fn download_and_verify(
&self,
release_info: &ReleaseInfo,
) -> Result<Vec<u8>, UpdateError> {
// Refuse before spending bandwidth on the archive if there is no checksum asset to
// verify it against.
let checksum_url = release_info.checksum_url.as_ref().ok_or_else(|| {
UpdateError::ChecksumUnavailable(
"no SHA256SUMS.txt asset found for this release; refusing to apply an unverified update"
.to_string(),
)
})?;
let sums_resp = self.client.get(checksum_url).send().await?;
if !sums_resp.status().is_success() {
return Err(UpdateError::ChecksumUnavailable(format!(
"failed to fetch SHA256SUMS.txt (HTTP status {}); refusing to apply an unverified update",
sums_resp.status()
)));
}
let sums_text = sums_resp.text().await?;
let expected_hash = parse_sha256sums(&sums_text, &release_info.archive_filename)
.ok_or_else(|| {
UpdateError::ChecksumFileError(format!(
"Filename '{}' not found in SHA256SUMS.txt",
release_info.archive_filename
))
})?;
// Download main binary package archive
let archive_resp = self.client.get(&release_info.download_url).send().await?;
if !archive_resp.status().is_success() {
@@ -237,29 +273,13 @@ impl UpdateEngine {
}
let archive_bytes = archive_resp.bytes().await?.to_vec();
// Verify SHA-256 checksum if SHA256SUMS.txt asset is available
if let Some(checksum_url) = &release_info.checksum_url {
let sums_resp = self.client.get(checksum_url).send().await?;
if sums_resp.status().is_success() {
let sums_text = sums_resp.text().await?;
if let Some(expected_hash) =
parse_sha256sums(&sums_text, &release_info.archive_filename)
{
let actual_hash = compute_sha256(&archive_bytes);
if expected_hash.to_lowercase() != actual_hash.to_lowercase() {
return Err(UpdateError::ChecksumMismatch {
filename: release_info.archive_filename.clone(),
expected: expected_hash,
actual: actual_hash,
});
}
} else {
return Err(UpdateError::ChecksumFileError(format!(
"Filename '{}' not found in SHA256SUMS.txt",
release_info.archive_filename
)));
}
}
let actual_hash = compute_sha256(&archive_bytes);
if expected_hash.to_lowercase() != actual_hash.to_lowercase() {
return Err(UpdateError::ChecksumMismatch {
filename: release_info.archive_filename.clone(),
expected: expected_hash,
actual: actual_hash,
});
}
Ok(archive_bytes)
@@ -660,4 +680,129 @@ e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 nuncio-x86_64-
assert_eq!(info.release_notes, "Major security release");
assert!(info.download_url.contains(&archive_filename));
}
fn release_info_for(
mock_server: &MockServer,
checksum_url: Option<String>,
archive_filename: &str,
) -> ReleaseInfo {
ReleaseInfo {
version: "99.0.0".to_string(),
tag_name: "v99.0.0".to_string(),
release_notes: String::new(),
download_url: format!("{}/download/{archive_filename}", mock_server.uri()),
checksum_url,
archive_filename: archive_filename.to_string(),
}
}
#[tokio::test]
async fn download_and_verify_fails_closed_without_checksum_asset() {
let mock_server = MockServer::start().await;
let archive_filename = "nuncio-x86_64-unknown-linux-gnu.tar.gz";
let archive_bytes = b"archive payload".to_vec();
Mock::given(method("GET"))
.and(path(format!("/download/{archive_filename}")))
.respond_with(ResponseTemplate::new(200).set_body_bytes(archive_bytes))
.mount(&mock_server)
.await;
let updater = UpdateEngine::with_base_url(mock_server.uri()).unwrap();
let info = release_info_for(&mock_server, None, archive_filename);
let result = updater.download_and_verify(&info).await;
assert!(matches!(result, Err(UpdateError::ChecksumUnavailable(_))));
}
#[tokio::test]
async fn download_and_verify_fails_closed_on_sums_fetch_error() {
let mock_server = MockServer::start().await;
let archive_filename = "nuncio-x86_64-unknown-linux-gnu.tar.gz";
let archive_bytes = b"archive payload".to_vec();
Mock::given(method("GET"))
.and(path(format!("/download/{archive_filename}")))
.respond_with(ResponseTemplate::new(200).set_body_bytes(archive_bytes))
.mount(&mock_server)
.await;
Mock::given(method("GET"))
.and(path("/download/SHA256SUMS.txt"))
.respond_with(ResponseTemplate::new(404))
.mount(&mock_server)
.await;
let updater = UpdateEngine::with_base_url(mock_server.uri()).unwrap();
let info = release_info_for(
&mock_server,
Some(format!("{}/download/SHA256SUMS.txt", mock_server.uri())),
archive_filename,
);
let result = updater.download_and_verify(&info).await;
assert!(matches!(result, Err(UpdateError::ChecksumUnavailable(_))));
}
#[tokio::test]
async fn download_and_verify_fails_closed_on_hash_mismatch() {
let mock_server = MockServer::start().await;
let archive_filename = "nuncio-x86_64-unknown-linux-gnu.tar.gz";
let archive_bytes = b"archive payload".to_vec();
let wrong_hash = compute_sha256(b"not the archive bytes");
let sums_text = format!("{wrong_hash} {archive_filename}\n");
Mock::given(method("GET"))
.and(path(format!("/download/{archive_filename}")))
.respond_with(ResponseTemplate::new(200).set_body_bytes(archive_bytes))
.mount(&mock_server)
.await;
Mock::given(method("GET"))
.and(path("/download/SHA256SUMS.txt"))
.respond_with(ResponseTemplate::new(200).set_body_string(sums_text))
.mount(&mock_server)
.await;
let updater = UpdateEngine::with_base_url(mock_server.uri()).unwrap();
let info = release_info_for(
&mock_server,
Some(format!("{}/download/SHA256SUMS.txt", mock_server.uri())),
archive_filename,
);
let result = updater.download_and_verify(&info).await;
assert!(matches!(result, Err(UpdateError::ChecksumMismatch { .. })));
}
#[tokio::test]
async fn download_and_verify_succeeds_on_matching_checksum() {
let mock_server = MockServer::start().await;
let archive_filename = "nuncio-x86_64-unknown-linux-gnu.tar.gz";
let archive_bytes = b"archive payload".to_vec();
let correct_hash = compute_sha256(&archive_bytes);
let sums_text = format!("{correct_hash} {archive_filename}\n");
Mock::given(method("GET"))
.and(path(format!("/download/{archive_filename}")))
.respond_with(ResponseTemplate::new(200).set_body_bytes(archive_bytes.clone()))
.mount(&mock_server)
.await;
Mock::given(method("GET"))
.and(path("/download/SHA256SUMS.txt"))
.respond_with(ResponseTemplate::new(200).set_body_string(sums_text))
.mount(&mock_server)
.await;
let updater = UpdateEngine::with_base_url(mock_server.uri()).unwrap();
let info = release_info_for(
&mock_server,
Some(format!("{}/download/SHA256SUMS.txt", mock_server.uri())),
archive_filename,
);
let result = updater
.download_and_verify(&info)
.await
.expect("checksum matches, download succeeds");
assert_eq!(result, archive_bytes);
}
}
crates/nuncio-filter/Cargo.toml
+2
View File
@@ -25,6 +25,8 @@ async-trait.workspace = true
chrono.workspace = true
reqwest.workspace = true
hex.workspace = true
zeroize.workspace = true
aes-gcm.workspace = true
[dev-dependencies]
tempfile.workspace = true
crates/nuncio-filter/src/ast.rs
+2 -1
View File
@@ -17,7 +17,8 @@ pub enum FilterField {
Header(String),
/// Attachment presence indicator (`true`/`false`).
HasAttachment,
/// Message size in bytes.
/// Message size in bytes: all stored body parts plus all attachment
/// payloads. Excludes headers and MIME framing, which are not retained.
Size,
/// Current folder location (e.g. `INBOX`).
Folder,
crates/nuncio-filter/src/engine.rs
+548 -91
View File
@@ -6,11 +6,26 @@ use crate::ast::{
};
use arc_swap::ArcSwap;
use hmac::{Hmac, Mac};
use nuncio_core::model::Email;
use nuncio_core::model::{Email, Placement};
use regex::Regex;
use sha2::Sha256;
use std::sync::Arc;
use std::time::{Duration, Instant};
use tracing::{debug, info, warn};
/// A message together with the one mailbox occupancy being evaluated.
///
/// `WHERE FOLDER = 'INBOX'` has no answer for a message that sits in three
/// folders at once, so evaluation is defined against a single placement and the
/// caller fans out over however many the message has. Borrowing both halves
/// keeps that fan-out free of clones.
#[derive(Debug, Clone, Copy)]
pub struct PlacedEmail<'a> {
/// Identity and content -- the same for every placement.
pub email: &'a Email,
/// The occupancy this evaluation is about.
pub placement: &'a Placement,
}
/// Pre-compiled single filter rule optimizing regex and predicate execution.
#[derive(Clone)]
@@ -54,31 +69,76 @@ impl CompiledFilter {
}
}
/// Evaluate email message against compiled rule condition tree.
/// Evaluate one placement against the compiled rule condition tree.
///
/// This checks only the WHERE-clause conditions; it does NOT apply the
/// rule's account scope. Callers must gate on
/// [`FilterRule::matches_account`] first (as every `FilterEngine` entry
/// point does), or a rule scoped to one account will fire on messages of
/// another.
pub fn evaluate_condition(&self, email: &Email) -> bool {
Self::eval_node(&self.rule.conditions, email, &self.compiled_regexes)
pub fn evaluate_condition(&self, placed: PlacedEmail<'_>) -> bool {
Self::eval_node(&self.rule.conditions, placed, &self.compiled_regexes)
}
fn eval_node(node: &ConditionNode, email: &Email, regexes: &[(String, Regex)]) -> bool {
fn eval_node(
node: &ConditionNode,
placed: PlacedEmail<'_>,
regexes: &[(String, Regex)],
) -> bool {
match node {
ConditionNode::Leaf(leaf) => Self::eval_leaf(leaf, email, regexes),
ConditionNode::Leaf(leaf) => Self::eval_leaf(leaf, placed, regexes),
ConditionNode::And(children) => {
children.iter().all(|c| Self::eval_node(c, email, regexes))
children.iter().all(|c| Self::eval_node(c, placed, regexes))
}
ConditionNode::Or(children) => {
children.iter().any(|c| Self::eval_node(c, email, regexes))
children.iter().any(|c| Self::eval_node(c, placed, regexes))
}
ConditionNode::Not(inner) => !Self::eval_node(inner, email, regexes),
ConditionNode::Not(inner) => !Self::eval_node(inner, placed, regexes),
}
}
fn eval_leaf(leaf: &ConditionLeaf, email: &Email, regexes: &[(String, Regex)]) -> bool {
/// Total byte size of the message content Nuncio holds: every body part
/// plus every attachment payload.
///
/// This is what `WHERE size > N` compares against. Counting only the
/// plaintext body would report `0` for the very common HTML-only message
/// and would ignore attachments entirely, so a size rule would quietly
/// return the wrong verdict on exactly the mail it is usually written to
/// catch (the big ones). Both body parts are summed rather than one being
/// preferred: a `multipart/alternative` message genuinely carries the
/// plaintext *and* the HTML on the wire, so summing them tracks the real
/// message rather than one arbitrary view of it.
///
/// The figure is the size of the decoded content as stored, not the exact
/// RFC822 octet count of the original transmission -- headers, MIME
/// boundaries, and transfer encoding are not counted, because the raw
/// octets are not retained and no stored column carries their length. It is
/// therefore a consistent lower bound on the wire size, never a guess: for
/// base64-encoded attachments the transmitted form is roughly a third
/// larger again. Rules are written against what the store actually has.
fn message_size_bytes(email: &Email) -> i64 {
let mut total: u64 = 0;
for part in [email.body_plain.as_deref(), email.body_html.as_deref()]
.into_iter()
.flatten()
{
total = total.saturating_add(part.len() as u64);
}
for attachment in &email.attachments {
total = total.saturating_add(attachment.content.len() as u64);
}
// Saturate rather than wrap: a size past `i64::MAX` is not a real
// message, and a wrapped negative would make `size > N` answer the
// opposite of the truth.
i64::try_from(total).unwrap_or(i64::MAX)
}
fn eval_leaf(
leaf: &ConditionLeaf,
placed: PlacedEmail<'_>,
regexes: &[(String, Regex)],
) -> bool {
let email = placed.email;
match &leaf.field {
FilterField::Subject => {
Self::eval_string_op(&email.subject, &leaf.operator, &leaf.value, regexes)
@@ -97,12 +157,18 @@ impl CompiledFilter {
.unwrap_or("");
Self::eval_string_op(body, &leaf.operator, &leaf.value, regexes)
}
FilterField::Folder => {
Self::eval_string_op(&email.folder_id, &leaf.operator, &leaf.value, regexes)
}
FilterField::Account => {
Self::eval_string_op(&email.account_id, &leaf.operator, &leaf.value, regexes)
}
FilterField::Folder => Self::eval_string_op(
&placed.placement.folder_id,
&leaf.operator,
&leaf.value,
regexes,
),
FilterField::Account => Self::eval_string_op(
&placed.placement.account_id,
&leaf.operator,
&leaf.value,
regexes,
),
FilterField::HasAttachment => {
let has = !email.attachments.is_empty();
if let FilterValue::Boolean(b) = leaf.value {
@@ -116,11 +182,7 @@ impl CompiledFilter {
}
}
FilterField::Size => {
let size = email
.body_plain
.as_ref()
.map(|b| b.len() as i64)
.unwrap_or(0);
let size = Self::message_size_bytes(email);
if let FilterValue::Number(target) = leaf.value {
match leaf.operator {
FilterOperator::Equals => size == target,
@@ -233,6 +295,26 @@ impl CompiledFilterSet {
}
}
/// Map each action to its variant name for observability logging, deliberately
/// dropping any destination data (folder name, forward address, webhook URL)
/// that the action variants carry.
fn action_kinds(actions: &[RuleAction]) -> Vec<&'static str> {
actions
.iter()
.map(|action| match action {
RuleAction::MoveTo(_) => "MOVE_TO",
RuleAction::CopyTo(_) => "COPY_TO",
RuleAction::MarkRead => "MARK_READ",
RuleAction::MarkUnread => "MARK_UNREAD",
RuleAction::Flag => "FLAG",
RuleAction::Unflag => "UNFLAG",
RuleAction::Delete => "DELETE",
RuleAction::ForwardTo(_) => "FORWARD_TO",
RuleAction::CallWebhook(_) => "CALL_WEBHOOK",
})
.collect()
}
/// Lock-free Filter Engine wrapping `ArcSwap<CompiledFilterSet>`.
pub struct FilterEngine {
cache: Arc<ArcSwap<CompiledFilterSet>>,
@@ -250,22 +332,38 @@ impl FilterEngine {
/// Swap / reload active filter rules atomically (<5ns latencies).
pub fn reload_rules(&self, rules: Vec<FilterRule>) -> Result<(), String> {
let set = CompiledFilterSet::new(rules)?;
let rule_count = set.filters.len();
self.cache.store(Arc::new(set));
info!(rule_count, "filter engine rules reloaded");
Ok(())
}
/// Evaluate email message returning matching rule actions.
/// Evaluate one placement of a message, returning the matching rules' actions.
///
/// Defined per placement, not per message: `FOLDER` and `ACCOUNT` are
/// properties of where the message sits, and a message can sit in several
/// places at once. Callers holding a multi-placement message evaluate each
/// placement and are responsible for not acting twice on one message -- see
/// the fire-once claim in the store.
///
/// A rule only ever fires for the account it was created against
/// (`FilterRule::target_account`, set via `ON ACCOUNT` or `*` for all
/// accounts) — condition matching alone is not account-scoped, so this
/// check must happen before `evaluate_condition` runs.
pub fn evaluate(&self, email: &Email) -> Vec<(FilterRule, Vec<RuleAction>)> {
pub fn evaluate(&self, placed: PlacedEmail<'_>) -> Vec<(FilterRule, Vec<RuleAction>)> {
let guard = self.cache.load();
let mut results = Vec::new();
for filter in &guard.filters {
if filter.rule.matches_account(&email.account_id) && filter.evaluate_condition(email) {
if filter.rule.matches_account(&placed.placement.account_id)
&& filter.evaluate_condition(placed)
{
debug!(
rule_id = %filter.rule.id,
rule_name = %filter.rule.name,
actions = ?action_kinds(&filter.rule.actions),
"rule matched; dispatching actions"
);
results.push((filter.rule.clone(), filter.rule.actions.clone()));
}
}
@@ -274,28 +372,70 @@ impl FilterEngine {
}
/// Evaluate with a Tokio hard timeout for ReDoS safety.
///
/// Each rule's condition evaluation is timed independently (on a
/// blocking-pool task, so a real timeout can actually preempt a
/// long-running synchronous evaluation rather than merely racing a timer
/// that a purely synchronous future would never yield to). A rule whose
/// evaluation exceeds `timeout_duration` is treated as **non-matching**
/// for this message — the documented, fail-closed outcome — but unlike a
/// bare `unwrap_or_default()`, the timeout is never silent: it is logged
/// with the rule's identity via `warn!`, and evaluation continues with
/// the remaining rules rather than aborting the whole batch.
pub async fn evaluate_with_timeout(
&self,
email: &Email,
placed: PlacedEmail<'_>,
timeout_duration: Duration,
) -> Vec<(FilterRule, Vec<RuleAction>)> {
let email_clone = email.clone();
let engine_cache = self.cache.clone();
let guard = self.cache.load();
let mut results = Vec::new();
tokio::time::timeout(timeout_duration, async move {
let guard = engine_cache.load();
let mut results = Vec::new();
for filter in &guard.filters {
if filter.rule.matches_account(&email_clone.account_id)
&& filter.evaluate_condition(&email_clone)
{
for filter in &guard.filters {
if !filter.rule.matches_account(&placed.placement.account_id) {
continue;
}
let filter_for_task = filter.clone();
let email_for_task = placed.email.clone();
let placement_for_task = placed.placement.clone();
let eval_task = tokio::task::spawn_blocking(move || {
filter_for_task.evaluate_condition(PlacedEmail {
email: &email_for_task,
placement: &placement_for_task,
})
});
match tokio::time::timeout(timeout_duration, eval_task).await {
Ok(Ok(true)) => {
debug!(
rule_id = %filter.rule.id,
rule_name = %filter.rule.name,
actions = ?action_kinds(&filter.rule.actions),
"rule matched; dispatching actions"
);
results.push((filter.rule.clone(), filter.rule.actions.clone()));
}
Ok(Ok(false)) => {}
Ok(Err(join_error)) => {
warn!(
rule_id = %filter.rule.id,
rule_name = %filter.rule.name,
error = %join_error,
"rule evaluation task failed to complete; treating rule as non-matching"
);
}
Err(_elapsed) => {
warn!(
rule_id = %filter.rule.id,
rule_name = %filter.rule.name,
timeout_ms = timeout_duration.as_millis(),
"rule evaluation timed out (ReDoS guard); treating rule as non-matching"
);
}
}
results
})
.await
.unwrap_or_default()
}
results
}
/// Dry-run preview evaluation returning detailed microsecond traces.
@@ -303,7 +443,7 @@ impl FilterEngine {
/// Rules whose account scope does not match the message are reported as
/// `SKIPPED (account scope mismatch)` rather than omitted, so the trace
/// distinguishes "condition did not match" from "rule does not apply here".
pub fn preview(&self, email: &Email) -> FilterPreviewResult {
pub fn preview(&self, placed: PlacedEmail<'_>) -> FilterPreviewResult {
let start = Instant::now();
let guard = self.cache.load();
let mut matched_rule_id = None;
@@ -313,7 +453,7 @@ impl FilterEngine {
let mut matched = false;
for filter in &guard.filters {
if !filter.rule.matches_account(&email.account_id) {
if !filter.rule.matches_account(&placed.placement.account_id) {
traces.push(format!(
"Rule '{}' (priority {}): SKIPPED (account scope mismatch)",
filter.rule.name, filter.rule.priority
@@ -321,7 +461,7 @@ impl FilterEngine {
continue;
}
let is_match = filter.evaluate_condition(email);
let is_match = filter.evaluate_condition(placed);
traces.push(format!(
"Rule '{}' (priority {}): {}",
filter.rule.name,
@@ -339,7 +479,7 @@ impl FilterEngine {
let elapsed = start.elapsed().as_micros() as u64;
FilterPreviewResult {
message_id: email.id.clone(),
message_id: placed.email.id.clone(),
matched,
matched_rule_id,
matched_rule_name,
@@ -368,20 +508,31 @@ mod tests {
use super::*;
use nuncio_core::model::Email;
fn test_email(account_id: &str, subject: &str, folder_id: &str) -> Email {
Email {
/// Build an `Email`/`Placement` pair that shares one account between the
/// message and the mailbox it is placed in, matching how a message and
/// its own placement always agree on account.
fn test_placed_email(account_id: &str, subject: &str, folder_id: &str) -> (Email, Placement) {
let email = Email {
id: "msg-1".to_string(),
account_id: account_id.to_string(),
folder_id: folder_id.to_string(),
subject: subject.to_string(),
sender: "alice@nuncio.mx".to_string(),
recipient: "bob@nuncio.mx".to_string(),
received_at: 1700000000,
read: false,
body_plain: Some("Hello".to_string()),
body_html: None,
attachments: Vec::new(),
}
message_id: None,
content_hash: None,
};
let placement = Placement {
account_id: account_id.to_string(),
folder_id: folder_id.to_string(),
uid_validity: "1".to_string(),
remote_id: "1".to_string(),
read: false,
};
(email, placement)
}
#[test]
@@ -390,29 +541,109 @@ mod tests {
let rule = crate::parser::NsqlParser::parse_rule("Urgent", 1, nsql).unwrap();
let engine = FilterEngine::new(vec![rule]).unwrap();
let email = Email {
id: "msg-1".to_string(),
account_id: "acct-1".to_string(),
folder_id: "inbox".to_string(),
subject: "Urgent Meeting".to_string(),
sender: "alice@nuncio.mx".to_string(),
recipient: "bob@nuncio.mx".to_string(),
received_at: 1700000000,
read: false,
body_plain: Some("Hello".to_string()),
body_html: None,
attachments: Vec::new(),
};
let (email, placement) = test_placed_email("acct-1", "Urgent Meeting", "inbox");
let results = engine.evaluate(&email);
let results = engine.evaluate(PlacedEmail {
email: &email,
placement: &placement,
});
assert_eq!(results.len(), 1);
assert_eq!(results[0].1.len(), 2);
let preview = engine.preview(&email);
let preview = engine.preview(PlacedEmail {
email: &email,
placement: &placement,
});
assert!(preview.matched);
assert_eq!(preview.matched_rule_name, Some("Urgent".to_string()));
}
/// Evaluate one NSQL rule against one message, returning whether it fired.
fn size_rule_matches(nsql: &str, email: &Email) -> bool {
let rule = crate::parser::NsqlParser::parse_rule("Size Rule", 1, nsql).unwrap();
let engine = FilterEngine::new(vec![rule]).unwrap();
let (_, placement) = test_placed_email(&email.account_id, "", "inbox");
!engine
.evaluate(PlacedEmail {
email,
placement: &placement,
})
.is_empty()
}
/// `size` must reflect every body part and every attachment, not the
/// plaintext body alone. An HTML-only message has no plaintext body at
/// all, so it used to measure zero bytes and silently fell on the wrong
/// side of every size comparison -- `size > 100` said no for a 400-byte
/// message, and `size < 100` said yes.
#[test]
fn size_counts_html_body_of_html_only_message() {
let (mut email, _) = test_placed_email("acct-1", "HTML only", "inbox");
email.body_plain = None;
email.body_html = Some(format!("<p>{}</p>", "x".repeat(400)));
assert!(
size_rule_matches("WHERE size > 100 ACTION MARK READ", &email),
"a 407-byte HTML-only message must satisfy size > 100"
);
assert!(
!size_rule_matches("WHERE size < 100 ACTION MARK READ", &email),
"a 407-byte HTML-only message must not satisfy size < 100"
);
}
/// Attachment payloads are part of the message's size. A short note
/// carrying a large attachment is exactly the mail a size rule is written
/// to catch, and counting only the note's text missed it entirely.
#[test]
fn size_counts_attachment_payloads() {
let (mut email, _) = test_placed_email("acct-1", "Invoice", "inbox");
email.body_plain = Some("See attached.".to_string());
email.attachments = vec![nuncio_core::model::Attachment {
filename: "invoice.pdf".to_string(),
mime_type: "application/pdf".to_string(),
content: vec![0u8; 5_000].into(),
}];
assert!(
size_rule_matches("WHERE size > 4096 ACTION MARK READ", &email),
"a 13-byte note with a 5000-byte attachment must satisfy size > 4096"
);
}
/// A `multipart/alternative` message carries both renderings on the wire,
/// so both are counted rather than one being preferred -- this pins the
/// definition `size` is documented to have.
#[test]
fn size_sums_both_body_parts_and_attachments() {
let (mut email, _) = test_placed_email("acct-1", "Both parts", "inbox");
email.body_plain = Some("a".repeat(100));
email.body_html = Some("b".repeat(250));
email.attachments = vec![nuncio_core::model::Attachment {
filename: "logo.png".to_string(),
mime_type: "image/png".to_string(),
content: vec![0u8; 650].into(),
}];
assert_eq!(CompiledFilter::message_size_bytes(&email), 1_000);
assert!(size_rule_matches(
"WHERE size = 1000 ACTION MARK READ",
&email
));
}
/// A message with neither body part nor attachments genuinely measures
/// zero, and must still compare as a number rather than being skipped.
#[test]
fn size_of_empty_message_is_zero() {
let (mut email, _) = test_placed_email("acct-1", "Empty", "inbox");
email.body_plain = None;
email.body_html = None;
assert_eq!(CompiledFilter::message_size_bytes(&email), 0);
assert!(size_rule_matches("WHERE size < 1 ACTION MARK READ", &email));
}
#[test]
fn test_webhook_signature() {
let sig =
@@ -428,14 +659,27 @@ mod tests {
// Condition matches the subject, but the message belongs to a
// different account than the rule was created for.
let email_b = test_email("acct-b", "Weekly Report", "inbox");
let (email_b, placement_b) = test_placed_email("acct-b", "Weekly Report", "inbox");
assert!(
engine.evaluate(&email_b).is_empty(),
engine
.evaluate(PlacedEmail {
email: &email_b,
placement: &placement_b
})
.is_empty(),
"rule scoped to acct-a must not match a message from acct-b"
);
let email_a = test_email("acct-a", "Weekly Report", "inbox");
assert_eq!(engine.evaluate(&email_a).len(), 1);
let (email_a, placement_a) = test_placed_email("acct-a", "Weekly Report", "inbox");
assert_eq!(
engine
.evaluate(PlacedEmail {
email: &email_a,
placement: &placement_a
})
.len(),
1
);
}
#[tokio::test]
@@ -444,15 +688,27 @@ mod tests {
let rule = crate::parser::NsqlParser::parse_rule("Scoped To A", 1, nsql).unwrap();
let engine = FilterEngine::new(vec![rule]).unwrap();
let email_b = test_email("acct-b", "Weekly Report", "inbox");
let (email_b, placement_b) = test_placed_email("acct-b", "Weekly Report", "inbox");
let results = engine
.evaluate_with_timeout(&email_b, Duration::from_secs(1))
.evaluate_with_timeout(
PlacedEmail {
email: &email_b,
placement: &placement_b,
},
Duration::from_secs(1),
)
.await;
assert!(results.is_empty());
let email_a = test_email("acct-a", "Weekly Report", "inbox");
let (email_a, placement_a) = test_placed_email("acct-a", "Weekly Report", "inbox");
let results = engine
.evaluate_with_timeout(&email_a, Duration::from_secs(1))
.evaluate_with_timeout(
PlacedEmail {
email: &email_a,
placement: &placement_a,
},
Duration::from_secs(1),
)
.await;
assert_eq!(results.len(), 1);
}
@@ -463,8 +719,11 @@ mod tests {
let rule = crate::parser::NsqlParser::parse_rule("Scoped To A", 1, nsql).unwrap();
let engine = FilterEngine::new(vec![rule]).unwrap();
let email_b = test_email("acct-b", "Weekly Report", "inbox");
let preview = engine.preview(&email_b);
let (email_b, placement_b) = test_placed_email("acct-b", "Weekly Report", "inbox");
let preview = engine.preview(PlacedEmail {
email: &email_b,
placement: &placement_b,
});
assert!(!preview.matched);
assert!(preview
.condition_traces
@@ -478,8 +737,16 @@ mod tests {
let rule = crate::parser::NsqlParser::parse_rule("Global", 1, nsql).unwrap();
let engine = FilterEngine::new(vec![rule]).unwrap();
let email = test_email("any-account", "Weekly Report", "inbox");
assert_eq!(engine.evaluate(&email).len(), 1);
let (email, placement) = test_placed_email("any-account", "Weekly Report", "inbox");
assert_eq!(
engine
.evaluate(PlacedEmail {
email: &email,
placement: &placement
})
.len(),
1
);
}
#[test]
@@ -489,20 +756,30 @@ mod tests {
let engine = FilterEngine::new(vec![rule]).unwrap();
// Same folder_id, different account_id: a rule matching on the
// `account` condition field must key off email.account_id, never
// fall through to comparing folder_id.
let email_a = test_email("acct-a", "Anything", "shared-folder");
// `account` condition field must key off the placement's account_id,
// never fall through to comparing folder_id.
let (email_a, placement_a) = test_placed_email("acct-a", "Anything", "shared-folder");
assert_eq!(
engine.evaluate(&email_a).len(),
engine
.evaluate(PlacedEmail {
email: &email_a,
placement: &placement_a
})
.len(),
1,
"rule on account = 'acct-a' must match a message with account_id 'acct-a'"
"rule on account = 'acct-a' must match a placement with account_id 'acct-a'"
);
let email_b = test_email("acct-b", "Anything", "shared-folder");
let (email_b, placement_b) = test_placed_email("acct-b", "Anything", "shared-folder");
assert!(
engine.evaluate(&email_b).is_empty(),
"rule on account = 'acct-a' must not match a message with account_id 'acct-b', \
even when it shares the same folder_id as an acct-a message"
engine
.evaluate(PlacedEmail {
email: &email_b,
placement: &placement_b
})
.is_empty(),
"rule on account = 'acct-a' must not match a placement with account_id 'acct-b', \
even when it shares the same folder_id as an acct-a placement"
);
}
@@ -512,11 +789,24 @@ mod tests {
let rule = crate::parser::NsqlParser::parse_rule("Not In Spam Or Trash", 1, nsql).unwrap();
let engine = FilterEngine::new(vec![rule]).unwrap();
let inbox_email = test_email("acct-1", "Anything", "inbox");
assert_eq!(engine.evaluate(&inbox_email).len(), 1);
let (inbox_email, inbox_placement) = test_placed_email("acct-1", "Anything", "inbox");
assert_eq!(
engine
.evaluate(PlacedEmail {
email: &inbox_email,
placement: &inbox_placement
})
.len(),
1
);
let spam_email = test_email("acct-1", "Anything", "spam");
assert!(engine.evaluate(&spam_email).is_empty());
let (spam_email, spam_placement) = test_placed_email("acct-1", "Anything", "spam");
assert!(engine
.evaluate(PlacedEmail {
email: &spam_email,
placement: &spam_placement
})
.is_empty());
}
#[test]
@@ -541,8 +831,16 @@ mod tests {
};
let engine = FilterEngine::new(vec![rule]).unwrap();
let email = test_email("acct-1", "Anything", "inbox");
assert_eq!(engine.evaluate(&email).len(), 1);
let (email, placement) = test_placed_email("acct-1", "Anything", "inbox");
assert_eq!(
engine
.evaluate(PlacedEmail {
email: &email,
placement: &placement
})
.len(),
1
);
}
#[test]
@@ -569,7 +867,166 @@ mod tests {
};
let engine = FilterEngine::new(vec![rule]).unwrap();
let email = test_email("acct-1", "Anything", "inbox");
assert_eq!(engine.evaluate(&email).len(), 1);
let (email, placement) = test_placed_email("acct-1", "Anything", "inbox");
assert_eq!(
engine
.evaluate(PlacedEmail {
email: &email,
placement: &placement
})
.len(),
1
);
}
#[tokio::test]
async fn test_evaluate_with_timeout_logs_warn_and_treats_timeout_as_non_match() {
// A crafted, slow-to-evaluate condition (large body scanned by a
// CONTAINS check) paired with a near-zero timeout budget forces the
// per-rule timeout path deterministically: the evaluation runs on a
// real blocking-pool thread while the timer races it on the async
// task, so the timer reliably wins for any evaluation slower than a
// few microseconds.
let logs = crate::test_tracing::capture_logs();
let rule = FilterRule {
id: "slow-rule".to_string(),
name: "Slow Rule".to_string(),
target_account: "*".to_string(),
priority: 1,
enabled: true,
nsql_text: String::new(),
conditions: ConditionNode::Leaf(ConditionLeaf {
field: FilterField::Body,
operator: FilterOperator::Contains,
value: FilterValue::String("nonexistent-token".to_string()),
}),
actions: vec![RuleAction::MarkRead],
created_at: 0,
updated_at: 0,
};
let engine = FilterEngine::new(vec![rule]).unwrap();
let (mut email, placement) = test_placed_email("acct-1", "Anything", "inbox");
email.body_plain = Some("A".repeat(60_000_000));
let results = engine
.evaluate_with_timeout(
PlacedEmail {
email: &email,
placement: &placement,
},
Duration::from_millis(1),
)
.await;
assert!(
results.is_empty(),
"documented outcome: a timed-out rule must be treated as non-matching, \
not fabricated as a match"
);
let captured = logs.lines();
assert!(
captured.iter().any(|line| line.contains("WARN")
&& line.contains("timed out")
&& line.contains("slow-rule")),
"expected a WARN log carrying the rule id for the timed-out evaluation, got: {captured:?}"
);
}
fn sample_email(key: &str) -> Email {
Email {
id: key.to_string(),
account_id: "acct-1".to_string(),
subject: "Anything".to_string(),
sender: "alice@nuncio.mx".to_string(),
recipient: "bob@nuncio.mx".to_string(),
received_at: 1700000000,
body_plain: Some("Hello".to_string()),
body_html: None,
attachments: Vec::new(),
message_id: None,
content_hash: None,
}
}
fn sample_placement(folder_id: &str, remote_id: &str) -> Placement {
Placement {
account_id: "acct-1".to_string(),
folder_id: folder_id.to_string(),
uid_validity: "1".to_string(),
remote_id: remote_id.to_string(),
read: false,
}
}
fn rule_where_folder_is(folder: &str) -> FilterRule {
let nsql = format!("WHERE folder = '{folder}' ACTION MARK READ");
crate::parser::NsqlParser::parse_rule("Folder Filter", 1, &nsql).unwrap()
}
fn rule_where_account_is(account: &str) -> FilterRule {
let nsql = format!("WHERE account = '{account}' ACTION MARK READ");
crate::parser::NsqlParser::parse_rule("Account Filter", 1, &nsql).unwrap()
}
#[test]
fn a_folder_condition_matches_only_the_placement_it_is_evaluated_against() {
let engine = FilterEngine::new(vec![rule_where_folder_is("INBOX")]).unwrap();
let email = sample_email("key-1");
let inbox = sample_placement("INBOX", "5");
let archive = sample_placement("Archive", "9");
assert_eq!(
engine
.evaluate(PlacedEmail {
email: &email,
placement: &inbox
})
.len(),
1,
"the INBOX placement matches"
);
assert_eq!(
engine
.evaluate(PlacedEmail {
email: &email,
placement: &archive
})
.len(),
0,
"the same message's Archive placement does not"
);
}
#[test]
fn an_account_condition_reads_the_placement_account() {
let engine = FilterEngine::new(vec![rule_where_account_is("acct-1")]).unwrap();
let email = sample_email("key-1");
let mine = sample_placement("INBOX", "5");
let theirs = Placement {
account_id: "acct-2".into(),
..sample_placement("INBOX", "5")
};
assert_eq!(
engine
.evaluate(PlacedEmail {
email: &email,
placement: &mine
})
.len(),
1
);
assert_eq!(
engine
.evaluate(PlacedEmail {
email: &email,
placement: &theirs
})
.len(),
0
);
}
}
crates/nuncio-filter/src/lib.rs
+2
View File
@@ -8,6 +8,8 @@ pub mod codegen;
pub mod engine;
pub mod outbox;
pub mod parser;
#[cfg(test)]
mod test_tracing;
pub mod validator;
pub mod webhook;
crates/nuncio-filter/src/outbox.rs
+29 -1
View File
@@ -1,6 +1,7 @@
//! Transactional Outbox Pattern for Remote IMAP/JMAP Mutations.
use crate::ast::PendingRemoteMutation;
use nuncio_core::model::PlacementKey;
use serde::{Deserialize, Serialize};
/// Payload details stored for pending remote mutation.
@@ -10,6 +11,17 @@ pub struct MutationPayload {
pub action_type: String,
/// Target parameter (e.g. folder name, target email address).
pub target: Option<String>,
/// The mailbox occupancy this mutation was formed against.
///
/// Message identity is content-addressed and folder-independent, so a single
/// message can occupy several mailboxes at once (Gmail labels make that
/// ordinary). `MOVE` and `DELETE` act on *one* occupancy, and acting on the
/// wrong one is destructive and irreversible, so the caller that formed the
/// intent records which occupancy it meant. `None` preserves the older
/// rows' meaning -- "unrecorded" -- which the executor may resolve only when
/// the message occupies exactly one mailbox.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub placement: Option<PlacementKey>,
}
/// Transactional outbox helper managing retry backoffs and mutation payloads.
@@ -31,17 +43,33 @@ impl OutboxManager {
base + jitter
}
/// Construct a new `PendingRemoteMutation` record.
/// Construct a new `PendingRemoteMutation` record whose target occupancy is
/// left unrecorded. The executor may still address it when the message
/// occupies exactly one mailbox, and refuses when it occupies several.
pub fn create_mutation(
rule_id: impl Into<String>,
message_id: impl Into<String>,
action_type: impl Into<String>,
target: Option<String>,
) -> PendingRemoteMutation {
Self::create_mutation_at(rule_id, message_id, action_type, target, None)
}
/// Construct a new `PendingRemoteMutation` record that names the mailbox
/// occupancy it was formed against, so a message occupying several mailboxes
/// is still unambiguously addressable.
pub fn create_mutation_at(
rule_id: impl Into<String>,
message_id: impl Into<String>,
action_type: impl Into<String>,
target: Option<String>,
placement: Option<PlacementKey>,
) -> PendingRemoteMutation {
let action_str = action_type.into();
let payload_struct = MutationPayload {
action_type: action_str.clone(),
target,
placement,
};
let payload_json = serde_json::to_string(&payload_struct).unwrap_or_default();
let now = chrono::Utc::now().timestamp();
crates/nuncio-filter/src/parser.rs
+13 -7
View File
@@ -65,7 +65,7 @@ impl NsqlParser {
let now = chrono::Utc::now().timestamp();
Ok(FilterRule {
id: format!("rule-{}", uuid_or_simple_hash(&name_str, nsql)),
id: format!("rule-{}", generate_rule_id()),
name: name_str,
target_account,
priority,
@@ -610,12 +610,18 @@ fn split_comma_outside_quotes(text: &str) -> Vec<String> {
result
}
fn uuid_or_simple_hash(name: &str, nsql: &str) -> String {
use sha2::{Digest, Sha256};
let mut hasher = Sha256::new();
hasher.update(name.as_bytes());
hasher.update(nsql.as_bytes());
hex::encode(&hasher.finalize()[..8])
/// Generates a fresh, collision-resistant rule id from the same CSPRNG
/// (`aes-gcm`'s `OsRng`) that `nuncio-store` uses to source key material,
/// rather than deriving the id from the rule's name/NSQL text. Two rules
/// created with identical name and NSQL text MUST still get distinct ids --
/// deriving the id from their content would make the second `CreateRule`
/// silently overwrite the first via `save_filter_rule`'s `INSERT OR
/// REPLACE`.
fn generate_rule_id() -> String {
use aes_gcm::aead::{rand_core::RngCore, OsRng};
let mut bytes = [0u8; 16];
OsRng.fill_bytes(&mut bytes);
hex::encode(bytes)
}
#[cfg(test)]
crates/nuncio-filter/src/test_tracing.rs
+146
View File
@@ -0,0 +1,146 @@
//! Shared in-test `tracing` capture helper.
//!
//! Records one formatted line per event (level plus `Debug`-rendered fields,
//! including the message) into an in-memory buffer, so tests can assert that
//! a level tag and specific fields appear on the SAME logged event. This
//! crate has no dependency on `tracing-subscriber`, so the capture is a
//! minimal, hand-rolled `tracing::Subscriber` rather than a `Layer` -- that
//! keeps this helper free of a new dependency. Fully offline: no I/O.
//!
//! The subscriber is installed **once, globally**, and events are routed to a
//! per-thread line buffer. That split is deliberate and load-bearing: a
//! thread-local subscriber cannot work here, because `tracing` caches each
//! callsite's `Interest` process-wide on first use. See
//! [`install_global_capture`].
#![cfg(test)]
use std::cell::RefCell;
use std::sync::{Arc, Mutex, OnceLock};
use tracing::field::{Field, Visit};
use tracing::span;
thread_local! {
/// The line buffer collecting telemetry for the test running on THIS
/// thread, or `None` on any thread not currently inside [`capture_logs`].
///
/// Routing per thread is what keeps the single global subscriber from
/// letting concurrent tests read each other's telemetry.
static ACTIVE_LINES: RefCell<Option<Arc<Mutex<Vec<String>>>>> = const { RefCell::new(None) };
}
struct LineVisitor<'a>(&'a mut String);
impl Visit for LineVisitor<'_> {
fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
self.0.push_str(&format!(" {}={:?}", field.name(), value));
}
}
/// Minimal `tracing::Subscriber` that records a formatted line per event
/// (level plus fields) without pulling in `tracing-subscriber`'s registry
/// machinery. Spans are not tracked (`new_span` returns a constant id) since
/// no test in this crate asserts on span structure, only on event lines.
struct CaptureSubscriber;
impl tracing::Subscriber for CaptureSubscriber {
fn enabled(&self, _metadata: &tracing::Metadata<'_>) -> bool {
true
}
fn new_span(&self, _span: &span::Attributes<'_>) -> span::Id {
span::Id::from_u64(1)
}
fn record(&self, _span: &span::Id, _values: &span::Record<'_>) {}
fn record_follows_from(&self, _span: &span::Id, _follows: &span::Id) {}
fn event(&self, event: &tracing::Event<'_>) {
let mut line = format!("{}", event.metadata().level());
let mut visitor = LineVisitor(&mut line);
event.record(&mut visitor);
ACTIVE_LINES.with(|slot| {
if let Some(lines) = slot.borrow().as_ref() {
lines.lock().unwrap_or_else(|e| e.into_inner()).push(line);
}
});
}
fn enter(&self, _span: &span::Id) {}
fn exit(&self, _span: &span::Id) {}
}
/// Install the capturing subscriber as the process-wide default, exactly once.
///
/// It MUST be global rather than thread-local, and this is the whole reason the
/// helper is shaped this way. `tracing` resolves each callsite's `Interest`
/// **once for the process** and caches it; a callsite first executed on a thread
/// with no subscriber caches `Interest::never()` and is then skipped forever --
/// before any later thread-local subscriber is ever consulted. Under a parallel
/// test harness, whichever thread reaches a callsite first therefore decides
/// whether any other test can ever capture it, which made every log-capture
/// assertion in this crate a coin flip.
///
/// A single global subscriber removes the race: every callsite registers against
/// a real subscriber, so interest is always enabled, and per-thread routing
/// (`ACTIVE_LINES`) decides what is actually recorded.
fn install_global_capture() {
static INSTALLED: OnceLock<()> = OnceLock::new();
INSTALLED.get_or_init(|| {
// A global default may already be set by another component of the same
// test binary; capturing still works through the thread-local routing
// above, so a lost race here is not fatal.
let _ = tracing::subscriber::set_global_default(CaptureSubscriber);
// Callsites executed before this point resolved their interest against
// the no-op dispatcher and cached "never". Recompute so they can be
// captured from here on. Done once, at install -- not per test.
tracing::callsite::rebuild_interest_cache();
});
}
/// Resets this thread's line buffer slot on drop, so a panicking test cannot
/// leak its buffer into whatever the harness runs next on the same thread.
struct LinesGuard;
impl Drop for LinesGuard {
fn drop(&mut self) {
ACTIVE_LINES.with(|slot| *slot.borrow_mut() = None);
}
}
/// A capture in progress on the current thread. Held across the code under
/// test -- including any `.await` points, as long as the test stays on a
/// single OS thread (e.g. the default current-thread `#[tokio::test]`
/// runtime) -- then queried for the lines captured so far.
pub(crate) struct CapturedLogs {
lines: Arc<Mutex<Vec<String>>>,
_guard: LinesGuard,
}
impl CapturedLogs {
/// A snapshot of every formatted event line captured on this thread so
/// far: level plus `Debug`-rendered fields (including `message`).
pub fn lines(&self) -> Vec<String> {
self.lines.lock().unwrap_or_else(|e| e.into_inner()).clone()
}
}
/// Start capturing this thread's `tracing` events as formatted lines.
///
/// Only this thread's events are captured, so any async work driven while the
/// returned [`CapturedLogs`] is alive must run on the calling thread (e.g. a
/// current-thread runtime's `block_on`) to be captured -- the subscriber
/// itself is global, but the destination buffer is per-thread; see
/// [`install_global_capture`] for why that split is load-bearing.
pub(crate) fn capture_logs() -> CapturedLogs {
install_global_capture();
let lines = Arc::new(Mutex::new(Vec::new()));
ACTIVE_LINES.with(|slot| *slot.borrow_mut() = Some(lines.clone()));
CapturedLogs {
lines,
_guard: LinesGuard,
}
}
crates/nuncio-filter/src/validator.rs
+148 -8
View File
@@ -33,7 +33,12 @@ pub struct ValidationOptions {
pub available_folders: Option<Vec<String>>,
/// Optional domain whitelist for Pass 6 `FORWARD TO` actions.
pub allowed_forward_domains: Option<Vec<String>>,
/// Flag enabling Pass 6 pre-flight DNS IP blacklisting for webhooks.
/// When set, reject webhook targets that point at private, loopback,
/// link-local, or metadata address ranges. The validator checks literal-IP
/// hosts; the dispatcher additionally resolves hostnames, checks every
/// candidate address, and then pins the outbound connection to the
/// checked address so it cannot be swapped for an unchecked one by a
/// second DNS lookup.
pub block_private_webhooks: bool,
}
@@ -87,6 +92,10 @@ impl NsqlValidator {
"header field conditions are not yet supported: message headers are not indexed"
.to_string(),
)),
// `Folder` and `Account` are validated the same as the
// content fields here: only the value's shape (string or
// string list) matters at this pass. At evaluation time they
// read a placement's coordinates, not the message's content.
FilterField::Subject
| FilterField::From
| FilterField::To
@@ -278,13 +287,25 @@ impl NsqlValidator {
)));
}
if opts.block_private_webhooks {
let url_lower = url.to_lowercase();
if url_lower.contains("127.0.0.1")
|| url_lower.contains("localhost")
|| url_lower.contains("169.254.169.254")
|| url_lower.contains("0.0.0.0")
{
return Err(ValidationError::ActionSecurityViolation(format!("CALL WEBHOOK URL '{url}' points to blocked private / metadata IP address")));
// When the host is an IP literal, reject any that lands in a
// blocked range. A substring match on the raw URL is not a
// real defense: RFC 1918, alternate literal encodings, and
// IPv6 all slip past it. Hostnames cannot be resolved here
// (this pass is synchronous); the dispatcher performs the
// DNS-resolution check before it connects.
if let Ok(parsed) = reqwest::Url::parse(url) {
if let Some(host) = parsed.host_str() {
// `host_str` brackets IPv6 literals (`[::1]`); strip
// them so the address parses.
let host = host.trim_start_matches('[').trim_end_matches(']');
if let Ok(ip) = host.parse::<std::net::IpAddr>() {
if is_blocked_webhook_target(ip) {
return Err(ValidationError::ActionSecurityViolation(format!(
"CALL WEBHOOK URL '{url}' points to a blocked private / metadata address"
)));
}
}
}
}
}
}
@@ -309,6 +330,51 @@ fn is_valid_email_format(email: &str) -> bool {
&& !parts[1].ends_with('.')
}
/// Reports whether `ip` falls in a range that must never be a webhook target.
///
/// This is the core SSRF egress control. It blocks loopback, RFC 1918 / ULA
/// private space, link-local (which subsumes the `169.254.169.254` cloud
/// metadata endpoint), the unspecified address, broadcast, and multicast.
/// It is shared by the synchronous validator (literal IPs in the URL) and the
/// dispatcher (addresses a hostname resolves to) so both apply identical rules.
pub(crate) fn is_blocked_webhook_target(ip: std::net::IpAddr) -> bool {
match ip {
std::net::IpAddr::V4(v4) => is_blocked_ipv4(v4),
std::net::IpAddr::V6(v6) => {
// Unwrap IPv4-mapped addresses (`::ffff:a.b.c.d`) first: an embedded
// private or loopback IPv4 must be caught by the v4 rules and not
// slip through as an ordinary v6 address.
match v6.to_ipv4_mapped() {
Some(mapped) => is_blocked_ipv4(mapped),
None => is_blocked_ipv6(v6),
}
}
}
}
fn is_blocked_ipv4(ip: std::net::Ipv4Addr) -> bool {
// `is_private` implements the exact RFC 1918 blocks (10/8, 172.16/12,
// 192.168/16), so boundaries such as 172.32.0.0 stay allowed. The leading
// `0` octet covers 0.0.0.0/8 (this-network, wider than `is_unspecified`).
ip.octets()[0] == 0
|| ip.is_loopback()
|| ip.is_private()
|| ip.is_link_local()
|| ip.is_multicast()
|| ip.is_broadcast()
}
fn is_blocked_ipv6(ip: std::net::Ipv6Addr) -> bool {
if ip.is_loopback() || ip.is_unspecified() || ip.is_multicast() {
return true;
}
// `is_unique_local` / `is_unicast_link_local` are not stable on the pinned
// toolchain, so match the prefixes by hand: fc00::/7 (unique-local) and
// fe80::/10 (link-local unicast).
let first = ip.segments()[0];
(first & 0xfe00) == 0xfc00 || (first & 0xffc0) == 0xfe80
}
#[cfg(test)]
mod tests {
use super::*;
@@ -372,4 +438,78 @@ mod tests {
let err = NsqlValidator::validate(&rule, &ValidationOptions::default()).unwrap_err();
assert!(matches!(err, ValidationError::ActionSecurityViolation(_)));
}
#[test]
fn test_pass6_rejects_literal_ip_bypasses() {
// Encodings the old case-insensitive substring blocklist let through.
let blocked_urls = [
"http://10.1.2.3/hook",
"http://172.16.0.1/hook",
"http://192.168.1.1/hook",
"http://169.254.169.254/latest/meta-data/",
"http://[::1]/hook",
"http://[fc00::1]/hook",
"http://[fe80::1]/hook",
"http://[::ffff:127.0.0.1]/hook",
];
for url in blocked_urls {
let rule = crate::parser::NsqlParser::parse_rule(
"Webhook Bypass",
1,
&format!("WHERE subject CONTAINS 'x' ACTION CALL WEBHOOK '{url}'"),
)
.unwrap();
let err = NsqlValidator::validate(&rule, &ValidationOptions::default()).unwrap_err();
assert!(
matches!(err, ValidationError::ActionSecurityViolation(_)),
"expected {url} to be blocked"
);
}
}
#[test]
fn test_pass6_allows_public_literal_ip() {
let rule = crate::parser::NsqlParser::parse_rule(
"Webhook Public",
1,
"WHERE subject CONTAINS 'x' ACTION CALL WEBHOOK 'http://93.184.216.34/hook'",
)
.unwrap();
assert!(NsqlValidator::validate(&rule, &ValidationOptions::default()).is_ok());
}
#[test]
fn test_is_blocked_webhook_target_blocks_private_ranges() {
let blocked: [std::net::IpAddr; 12] = [
"127.0.0.1".parse().unwrap(),
"10.1.2.3".parse().unwrap(),
"172.16.0.1".parse().unwrap(),
"172.31.255.255".parse().unwrap(),
"192.168.1.1".parse().unwrap(),
"169.254.169.254".parse().unwrap(),
"169.254.0.1".parse().unwrap(),
"0.0.0.0".parse().unwrap(),
"::1".parse().unwrap(),
"fc00::1".parse().unwrap(),
"fe80::1".parse().unwrap(),
"::ffff:127.0.0.1".parse().unwrap(),
];
for ip in blocked {
assert!(is_blocked_webhook_target(ip), "{ip} should be blocked");
}
}
#[test]
fn test_is_blocked_webhook_target_allows_public() {
// 172.32.0.1 sits just outside 172.16.0.0/12 and proves the range math
// is exact rather than a naive `172.` prefix check.
let allowed: [std::net::IpAddr; 3] = [
"172.32.0.1".parse().unwrap(),
"93.184.216.34".parse().unwrap(),
"2606:2800:220:1:248:1893:25c8:1946".parse().unwrap(),
];
for ip in allowed {
assert!(!is_blocked_webhook_target(ip), "{ip} should be allowed");
}
}
}
crates/nuncio-filter/src/webhook.rs
+332 -44
View File
@@ -1,16 +1,73 @@
//! Outbound HTTP Webhook Execution Engine with HMAC-SHA256 Signatures and SSRF Defense.
use crate::ast::RuleAction;
use crate::validator::{NsqlValidator, ValidationOptions};
use crate::validator::{is_blocked_webhook_target, NsqlValidator, ValidationOptions};
use hmac::{Hmac, Mac};
use reqwest::Client;
use serde_json::json;
use sha2::Sha256;
use std::net::{IpAddr, SocketAddr};
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use thiserror::Error;
use tokio::net::lookup_host;
use tracing::{debug, warn, Instrument};
use zeroize::Zeroizing;
/// Extract just the host for logging (never the path/query, which may carry
/// caller-supplied tokens), falling back to a placeholder when the URL itself
/// cannot be parsed.
fn webhook_host_for_log(url: &str) -> String {
reqwest::Url::parse(url)
.ok()
.and_then(|parsed| parsed.host_str().map(str::to_string))
.map(|h| h.trim_start_matches('[').trim_end_matches(']').to_string())
.unwrap_or_else(|| "unparsable-url".to_string())
}
type HmacSha256 = Hmac<Sha256>;
/// Returns the first address in `addrs` that is a blocked webhook target, if
/// any. Split out from the DNS lookup so the egress policy can be exercised
/// against synthetic address lists without touching the network.
fn first_blocked_address(addrs: &[IpAddr]) -> Option<IpAddr> {
addrs
.iter()
.copied()
.find(|ip| is_blocked_webhook_target(*ip))
}
/// Resolves `host` to its candidate addresses. Kept separate from the policy
/// check so the address list a rebinding DNS server could return is visible
/// to callers before any of it is trusted.
async fn resolve_host(host: &str, port: u16) -> Result<Vec<IpAddr>, WebhookError> {
let resolved: Vec<IpAddr> = lookup_host((host, port))
.await
.map_err(|e| WebhookError::NetworkError(e.to_string()))?
.map(|addr| addr.ip())
.collect();
if resolved.is_empty() {
return Err(WebhookError::NetworkError(format!(
"webhook host '{host}' did not resolve to any address"
)));
}
Ok(resolved)
}
/// Builds an HTTP client whose connection to `host` is pinned to `addr`.
/// `reqwest`'s DNS override maps only the transport-level connect address; it
/// still sends the original hostname as the TLS SNI and `Host` header, so
/// certificate validation is unaffected. This is what stops the dispatcher
/// from re-resolving `host` after it has been checked against the SSRF
/// policy: the checked address is the only one the connection can use.
fn pinned_client(host: &str, addr: IpAddr, port: u16) -> Result<Client, WebhookError> {
Client::builder()
.timeout(Duration::from_secs(5))
.redirect(reqwest::redirect::Policy::none())
.resolve(host, SocketAddr::new(addr, port))
.build()
.map_err(|e| WebhookError::NetworkError(e.to_string()))
}
/// Errors emitted during HTTP webhook dispatch execution.
#[derive(Error, Debug)]
pub enum WebhookError {
@@ -26,23 +83,19 @@ pub enum WebhookError {
}
/// Outbound Webhook Dispatcher executing authenticated HTTP POST requests.
///
/// Each dispatch builds its own client pinned to the address it just
/// resolved and checked (see [`pinned_client`]), so there is no shared
/// client to cache here.
pub struct WebhookDispatcher {
client: Client,
secret_key: String,
secret_key: Zeroizing<String>,
}
impl WebhookDispatcher {
/// Create a new `WebhookDispatcher` with an HMAC signing key.
pub fn new(secret_key: impl Into<String>) -> Self {
let client = Client::builder()
.timeout(Duration::from_secs(5))
.redirect(reqwest::redirect::Policy::none())
.build()
.unwrap_or_default();
Self {
client,
secret_key: secret_key.into(),
secret_key: Zeroizing::new(secret_key.into()),
}
}
@@ -56,45 +109,128 @@ impl WebhookDispatcher {
sender: &str,
opts: &ValidationOptions,
) -> Result<u16, WebhookError> {
let action = RuleAction::CallWebhook(url.to_string());
NsqlValidator::pass6_action_security(&[action], opts)
.map_err(|e| WebhookError::SecurityViolation(e.to_string()))?;
let host_for_log = webhook_host_for_log(url);
let span =
tracing::info_span!("webhook_dispatch", host = %host_for_log, rule_id = %rule_id);
let timestamp = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs();
async move {
let action = RuleAction::CallWebhook(url.to_string());
if let Err(e) = NsqlValidator::pass6_action_security(&[action], opts) {
warn!(
host = %host_for_log,
reason = %e,
"webhook egress blocked by SSRF/security policy"
);
return Err(WebhookError::SecurityViolation(e.to_string()));
}
let payload = json!({
"event": "nuncio.filter.matched",
"timestamp": timestamp,
"rule_id": rule_id,
"message_id": message_id,
"subject": subject,
"sender": sender,
});
// Pass 6 only sees literal-IP hosts. Resolve the target here and,
// when the policy is enabled, reject it if it points into a blocked
// range - closing the DNS-rebind class of SSRF bypass that a
// substring/literal check on the URL cannot catch. The resolved
// address is then pinned below for the actual connection, so the
// dispatch cannot re-resolve the hostname and land on a different,
// unchecked address: the address that was checked is the only one
// `reqwest` is permitted to connect to.
let parsed = reqwest::Url::parse(url).map_err(|e| {
WebhookError::SecurityViolation(format!("invalid webhook URL: {e}"))
})?;
let host = parsed
.host_str()
.ok_or_else(|| {
WebhookError::SecurityViolation("webhook URL has no host".to_string())
})?
// `host_str` brackets IPv6 literals (`[::1]`); strip them so the
// pair passes to the resolver cleanly.
.trim_start_matches('[')
.trim_end_matches(']')
.to_string();
let port = parsed.port_or_known_default().unwrap_or(0);
let payload_str = payload.to_string();
let resolved = resolve_host(&host, port).await?;
if opts.block_private_webhooks {
if let Some(blocked) = first_blocked_address(&resolved) {
warn!(
host = %host,
blocked_ip = %blocked,
"webhook egress blocked: resolved address is in a disallowed range (DNS-rebind guard)"
);
return Err(WebhookError::SecurityViolation(format!(
"webhook host '{host}' resolves to blocked address {blocked}"
)));
}
}
let pinned_addr = resolved[0];
debug!(
host = %host,
resolved_ip = %pinned_addr,
"webhook egress allowed by SSRF/security policy"
);
let client = pinned_client(&host, pinned_addr, port)?;
let mut mac = HmacSha256::new_from_slice(self.secret_key.as_bytes())
.map_err(|e| WebhookError::SigningError(e.to_string()))?;
mac.update(format!("{timestamp}.{payload_str}").as_bytes());
let signature = hex::encode(mac.finalize().into_bytes());
let timestamp = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap_or_default()
.as_secs();
let response = self
.client
.post(url)
.header("Content-Type", "application/json")
.header(
"X-Nuncio-Signature",
format!("t={timestamp},v1={signature}"),
)
.body(payload_str)
.send()
.await
.map_err(|e| WebhookError::NetworkError(e.to_string()))?;
let payload = json!({
"event": "nuncio.filter.matched",
"timestamp": timestamp,
"rule_id": rule_id,
"message_id": message_id,
"subject": subject,
"sender": sender,
});
Ok(response.status().as_u16())
let payload_str = payload.to_string();
let mut mac = HmacSha256::new_from_slice(self.secret_key.as_bytes())
.map_err(|e| WebhookError::SigningError(e.to_string()))?;
mac.update(format!("{timestamp}.{payload_str}").as_bytes());
let signature = hex::encode(mac.finalize().into_bytes());
// Stable across every retry of this rule/message pair, so a
// receiver can discard a redelivery caused by a lost response
// rather than acting twice. It is NOT yet stable across daemons:
// `rule_id` is random per daemon, so two daemons executing the
// same rule produce different keys. Single-owner filter execution
// is what prevents that case today; a rule key derived from the
// normalized NSQL would make it dedupable directly.
let idempotency_key = {
let mut hasher = <Sha256 as sha2::Digest>::new();
for field in [rule_id, message_id] {
sha2::Digest::update(&mut hasher, (field.len() as u64).to_le_bytes());
sha2::Digest::update(&mut hasher, field.as_bytes());
}
hex::encode(sha2::Digest::finalize(hasher))
};
let send_result = client
.post(url)
.header("Content-Type", "application/json")
.header("Idempotency-Key", &idempotency_key)
.header(
"X-Nuncio-Signature",
format!("t={timestamp},v1={signature}"),
)
.body(payload_str)
.send()
.await;
match send_result {
Ok(response) => {
let status = response.status().as_u16();
debug!(host = %host, status, "webhook dispatched");
Ok(status)
}
Err(e) => {
warn!(host = %host, error = %e, "webhook dispatch network error");
Err(WebhookError::NetworkError(e.to_string()))
}
}
}
.instrument(span)
.await
}
/// Dispatch a `CALL WEBHOOK` action with default production security options.
@@ -121,6 +257,68 @@ impl WebhookDispatcher {
#[cfg(test)]
mod tests {
use super::*;
use wiremock::matchers::{method, path};
use wiremock::{Mock, MockServer, ResponseTemplate};
#[tokio::test]
async fn test_pinned_client_connects_to_checked_ip_not_dns() {
// `.invalid` is reserved by RFC 2606 to never resolve: any real DNS
// lookup for it fails. The mock server only listens on its loopback
// address, which has no relationship to that hostname in any
// resolver. If the request below reaches the mock server, it can
// only be because `pinned_client`'s resolve override - not DNS -
// supplied the connect address, proving the dispatcher connects to
// exactly the address it already checked.
let mock_server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/hook"))
.respond_with(ResponseTemplate::new(200))
.mount(&mock_server)
.await;
let addr = *mock_server.address();
let host = "checked-ip-pin.invalid";
let client = pinned_client(host, addr.ip(), addr.port())
.expect("building a client with a resolve override must succeed");
let url = format!("http://{host}:{}/hook", addr.port());
let response = client
.post(&url)
.send()
.await
.expect("the pinned connection must reach the mock server, not a DNS lookup");
assert_eq!(response.status(), 200);
}
#[tokio::test]
async fn test_dispatch_reaches_pinned_loopback_mock_server() {
// End-to-end proof that `dispatch_with_options` itself (not just the
// `pinned_client` helper) drives the request to the address it
// resolved and checked. `block_private_webhooks` is disabled here
// only to allow the test to target a loopback mock server; it does
// not affect whether the connection is pinned.
let mock_server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/hook"))
.respond_with(ResponseTemplate::new(202))
.mount(&mock_server)
.await;
let dispatcher = WebhookDispatcher::new("secret_key_123");
let opts = ValidationOptions {
available_folders: None,
allowed_forward_domains: None,
block_private_webhooks: false,
};
let url = format!("{}/hook", mock_server.uri());
let status = dispatcher
.dispatch_with_options(&url, "rule_1", "msg_1", "Test", "a@b.com", &opts)
.await
.expect("dispatch to the mock server must succeed");
assert_eq!(status, 202);
}
#[tokio::test]
async fn test_blocked_private_ip_webhook() {
@@ -141,4 +339,94 @@ mod tests {
WebhookError::SecurityViolation(_)
));
}
#[test]
fn test_first_blocked_address_flags_private_in_mixed_list() {
// Synthetic resolver output: a public address alongside a private one,
// as a rebinding host might return. The private one must be caught.
let addrs: Vec<IpAddr> = vec![
"93.184.216.34".parse().unwrap(),
"10.0.0.5".parse().unwrap(),
];
assert_eq!(
first_blocked_address(&addrs),
Some("10.0.0.5".parse().unwrap())
);
}
#[test]
fn test_first_blocked_address_allows_all_public() {
let addrs: Vec<IpAddr> = vec![
"93.184.216.34".parse().unwrap(),
"172.32.0.1".parse().unwrap(),
];
assert_eq!(first_blocked_address(&addrs), None);
}
#[tokio::test]
async fn test_blocked_webhook_logs_warn_with_host_and_reason() {
let logs = crate::test_tracing::capture_logs();
let dispatcher = WebhookDispatcher::new("secret_key_123");
let result = dispatcher
.dispatch(
"http://169.254.169.254/latest/meta-data",
"rule_1",
"msg_1",
"Test",
"a@b.com",
)
.await;
assert!(matches!(result, Err(WebhookError::SecurityViolation(_))));
let captured = logs.lines();
assert!(
captured.iter().any(|line| line.contains("WARN")
&& line.contains("blocked")
&& line.contains("169.254.169.254")),
"expected a WARN log carrying the blocked host and reason, got: {captured:?}"
);
assert!(
!captured.iter().any(|line| line.contains("secret_key_123")),
"the webhook HMAC secret must never appear in a log line"
);
}
#[tokio::test]
async fn test_allowed_webhook_logs_debug_not_warn() {
let mock_server = MockServer::start().await;
Mock::given(method("POST"))
.and(path("/hook"))
.respond_with(ResponseTemplate::new(200))
.mount(&mock_server)
.await;
let logs = crate::test_tracing::capture_logs();
let dispatcher = WebhookDispatcher::new("secret_key_123");
let opts = ValidationOptions {
available_folders: None,
allowed_forward_domains: None,
block_private_webhooks: false,
};
let url = format!("{}/hook", mock_server.uri());
let status = dispatcher
.dispatch_with_options(&url, "rule_1", "msg_1", "Test", "a@b.com", &opts)
.await
.expect("dispatch to the mock server must succeed");
assert_eq!(status, 200);
let captured = logs.lines();
assert!(
captured
.iter()
.any(|line| line.contains("DEBUG") && line.contains("allowed")),
"expected a DEBUG log for the allowed egress decision, got: {captured:?}"
);
assert!(
!captured.iter().any(|line| line.contains("WARN")),
"an allowed dispatch must not emit a WARN, got: {captured:?}"
);
}
}
crates/nuncio-mail/Cargo.toml
+10
View File
@@ -9,6 +9,11 @@ repository.workspace = true
[lints]
workspace = true
[features]
# Exposes `test_server`, a stateful loopback IMAP server, to dependent crates'
# test suites. Never enabled in a production build.
test-support = []
[dependencies]
nuncio-core = { path = "../nuncio-core" }
tokio.workspace = true
@@ -29,8 +34,13 @@ tokio-stream.workspace = true
reqwest.workspace = true
[dev-dependencies]
# Integration tests need the loopback IMAP server, which production builds
# must not carry. A self dev-dependency is how a crate enables its own feature
# for its own `tests/` targets.
nuncio-mail = { path = ".", features = ["test-support"] }
tempfile.workspace = true
wiremock.workspace = true
tracing-test = "0.2"
# `test-util` (not in tokio's `full`) enables paused-clock tests so the
# bounded per-item FETCH timeout can be asserted without a real-time wait.
tokio = { workspace = true, features = ["test-util"] }
crates/nuncio-mail/src/backend.rs
+192 -6
View File
@@ -1,26 +1,212 @@
//! Protocol-agnostic async mail backend trait definitions.
use async_trait::async_trait;
use nuncio_core::model::{Attachment, Email, Folder};
use nuncio_core::model::{Attachment, Email, Folder, IdentitySource, Placement, PlacementKey};
use crate::parser::MailError;
/// One message as a backend surfaced it: identity, content, and the mailbox
/// occupancy it was found in.
///
/// Backends emit this rather than a bare [`Email`] because a fetch observes two
/// separate facts at once: *which message this is* -- folder-independent, and
/// the same in every mailbox that holds a copy -- and *where this pass found
/// it*, which is per-mailbox and changes under a move. Collapsing them into one
/// record is what made a moved message read as a new one.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PlacedMessage {
/// Identity and content. `email.id` is the derived message key.
pub email: Email,
/// Which precedence tier produced `email.id`.
pub source: IdentitySource,
/// The occupancy this fetch found it in.
pub placement: Placement,
}
/// The remote change a [`RemoteMutationSpec`] applies to one addressed message.
///
/// This is the closed vocabulary of genuinely-remote filter actions that the
/// outbox drains against a real server. `MARK READ`/`MARK UNREAD` are applied
/// locally to the store when a filter matches, so they are deliberately NOT
/// represented here -- only actions that mutate server-side state are.
/// `FORWARD` and `CALL WEBHOOK` are also excluded: they are not backend
/// mutations (one composes an outbound message, the other calls an HTTP
/// endpoint), so they are executed through the send/webhook seams instead.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum RemoteMutationKind {
/// Add (`value == true`) or remove (`value == false`) the "flagged"/
/// starred keyword: IMAP `\Flagged`, JMAP `$flagged`. Covers `FLAG`/`UNFLAG`.
SetFlagged {
/// Whether the message should end up flagged.
value: bool,
},
/// Move the message from its source folder to `to_folder`.
Move {
/// Destination folder (IMAP mailbox name / JMAP mailbox id).
to_folder: String,
},
/// Copy the message into `to_folder`, leaving the source copy in place.
Copy {
/// Destination folder (IMAP mailbox name / JMAP mailbox id).
to_folder: String,
},
/// Delete the message (IMAP `\Deleted` + `EXPUNGE`; JMAP `destroy`).
Delete,
}
/// A fully-addressed remote mutation for a single message: the protocol-native
/// id to address it on the server, the folder it currently lives in, the
/// UIDVALIDITY scope that id was captured under, and the change to apply.
///
/// Addressing is driven by `remote_id` (the IMAP UID / JMAP object id), NOT by
/// the opaque `message_id` -- so remote-mutation targeting never depends on the
/// shape of the surrogate id. `uid_validity` is the mailbox UIDVALIDITY the
/// `remote_id` was captured under; the IMAP backend refuses to act unless the
/// mailbox's current UIDVALIDITY still matches it, because a renumbered mailbox
/// (RFC 3501 s2.3.1.1) reassigns UIDs and the stored UID would otherwise
/// address the wrong message. Protocols without a UIDVALIDITY (JMAP) carry a
/// stable sentinel and ignore this guard.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RemoteMutationSpec {
/// Opaque surrogate id of the target message. Diagnostic only -- it names
/// the message in errors/logs and is never used to address it on the wire.
pub message_id: String,
/// Protocol-native id used to address the message on its server: the IMAP
/// UID as a decimal string, or the JMAP Email object id.
pub remote_id: String,
/// Folder the message currently lives in.
pub folder_id: String,
/// The UIDVALIDITY the `remote_id` was captured under (IMAP; decimal
/// string), or a protocol sentinel for servers without UIDVALIDITY (JMAP).
pub uid_validity: String,
/// The change to apply.
pub kind: RemoteMutationKind,
}
/// What one folder-sync pass observed on the server.
///
/// The reason this is not just a list of messages: a sync that can only report
/// what *is* there can never report what stopped being there. A message moved
/// or deleted by any other client -- another Nuncio daemon, a phone, webmail --
/// simply stays in the local store forever. Expressing absence is the whole
/// point of the type.
///
/// Absence is reported as [`PlacementKey`]s, not message keys, because what a
/// folder stops mentioning is an *occupancy*. The same message may still sit in
/// another mailbox, and deleting the message on the strength of one folder's
/// silence would destroy a copy the user still has. A message goes only when
/// its last placement does.
#[derive(Debug, Default, Clone)]
pub struct FolderChanges {
/// Messages to store, whether newly arrived or changed, each with the
/// occupancy this pass found it in.
pub upserts: Vec<PlacedMessage>,
/// Occupancies the server **explicitly reported as gone**, via QRESYNC
/// `VANISHED`. Always safe to delete.
pub removals: Vec<PlacementKey>,
/// Every occupancy the folder currently holds, when this pass enumerated
/// the complete UID set. The caller may delete any placement it stores for
/// this folder that is absent from this list.
///
/// `None` and `Some(vec![])` mean different things, and conflating them
/// deletes a mailbox. `None` is "this pass was incremental and cannot
/// speak to absence"; `Some(vec![])` is "the folder is genuinely empty".
pub present: Option<Vec<PlacementKey>>,
/// The checkpoint to resume from next time.
pub next_state: String,
}
/// What a mutation attempt actually established.
///
/// Two states are not enough. A protocol that answers `OK` for a command that
/// did nothing -- which IMAP does, by design, for a UID that no longer exists
/// (RFC 3501 section 6.4.8) -- makes "succeeded" and "failed" an incomplete
/// partition. The missing third state is *the server accepted the command and
/// its response does not establish what happened*, and collapsing that into
/// either neighbour is how a lost mutation gets recorded as done.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum MutationOutcome {
/// The server's response proves the change was made.
///
/// Requires positive evidence: a `COPYUID` naming the destination UIDs, a
/// removal report for a delete, or an explicit acknowledgement for a flag
/// change. `token` carries whatever the server returned to address the
/// result afterwards (the destination UID for a move), when it returned
/// one at all -- a DAV server that rewrites the object MUST NOT return an
/// ETag (RFC 4791 section 5.3.4), so absence here is ordinary.
Applied {
/// Server-assigned handle for the result, when the response carried one.
token: Option<String>,
},
/// Another client changed the message first, and the server said so.
///
/// Requires positive evidence too: `[MODIFIED …]` (RFC 7162 section 3.1.3)
/// or an HTTP 412. A mutation that merely failed to find its target is
/// **not** a conflict -- `MODIFIED` reports "changed by someone else",
/// never "gone".
Conflict {
/// What the server reported about the conflicting state.
observed: String,
},
/// The command was accepted and its response settles nothing.
///
/// The honest reading of an absent `COPYUID` (only `SHOULD` for `MOVE` per
/// RFC 6851 section 4.3, and legitimately omitted for a `UIDNOTSTICKY`
/// destination), or of a UID set that matched nothing. The caller must
/// re-enumerate to find out, not guess.
Unknown {
/// Why the response was inconclusive.
reason: String,
},
}
/// Protocol-agnostic mail backend engine trait implemented by JMAP and IMAP engines.
#[async_trait]
pub trait MailBackend: Send + Sync {
/// Synchronize and list available mailbox folders.
async fn sync_folders(&self) -> Result<Vec<Folder>, MailError>;
/// Synchronize message envelopes for a specific folder since a checkpoint state.
/// Returns the updated email list and the new server state checkpoint.
/// Enumerate what changed in a folder since `since_state`.
///
/// Replaces a fetch-only sync: implementations must report removals when
/// the protocol can express them, and must say honestly (via
/// [`FolderChanges::present`]) whether the pass was able to observe
/// absence at all.
async fn sync_changes(
&self,
folder_id: &str,
since_state: Option<&str>,
) -> Result<FolderChanges, MailError>;
/// Fetch-only view of [`MailBackend::sync_changes`], for callers that
/// genuinely only want the messages.
///
/// Provided rather than required so no implementation can satisfy the
/// trait by supplying this and quietly never reporting a removal.
///
/// Every message comes back as a [`PlacedMessage`] so the caller can tell
/// a message it already stores in another folder from one it has never
/// seen: the derived key is the same in both folders, only the placement
/// differs.
async fn sync_messages(
&self,
folder_id: &str,
since_state: Option<&str>,
) -> Result<(Vec<Email>, String), MailError>;
) -> Result<(Vec<PlacedMessage>, String), MailError> {
let changes = self.sync_changes(folder_id, since_state).await?;
Ok((changes.upserts, changes.next_state))
}
/// Send an email message over the configured transport.
async fn send_email(&self, email: &Email) -> Result<(), MailError>;
/// Apply a remote mutation (move/copy/flag/delete) to a single message
/// against the real server.
///
/// Returns what the server's response actually proves. Implementations
/// must never report [`MutationOutcome::Applied`] on the strength of a
/// tagged `OK` alone, and must refuse to act (returning `Err`) when the
/// target cannot be safely identified -- e.g. an IMAP UIDVALIDITY
/// mismatch, where the stored UID now names a different message.
async fn apply_mutation(&self, spec: &RemoteMutationSpec)
-> Result<MutationOutcome, MailError>;
}
/// A composed outbound email message ready to send over SMTP. Deliberately
crates/nuncio-mail/src/imap.rs
+2783 -357
View File
File diff suppressed because it is too large. Load diff
crates/nuncio-mail/src/imap_raw.rs
+766
View File
@@ -0,0 +1,766 @@
//! Raw IMAP command layer: the responses `async-imap`'s typed API discards.
//!
//! The engine needs five things the typed client cannot express, and they all
//! decide whether a sync saw everything, whether a mutation actually happened,
//! or whether two engines can agree on what a message *is*:
//!
//! | Needed | Why the typed API cannot give it |
//! | --- | --- |
//! | `ENABLE` | not implemented at all, and RFC 7162 3.2.3 makes a server reply `BAD` to a QRESYNC `SELECT` without it |
//! | `SELECT (QRESYNC …)` | no such method, and `select()`'s parser drops `VANISHED` |
//! | `COPYUID` | `uid_copy`/`uid_mv` return `Result<()>`; the tagged response *code* is discarded |
//! | `[MODIFIED …]` | same discard, and `imap-proto` has no `ResponseCode` variant for it at all |
//! | `EMAILID` | `imap-proto` cannot *parse* it, and failing to parse kills the connection -- see [`uid_fetch_email_ids`] |
//!
//! ## Why this is hand-rolled rather than a fork
//!
//! `Session::run_command` and `Session::read_response` are public, and
//! `imap-proto` already parses `Response::Vanished` and
//! `ResponseCode::CopyUid`. So the only thing missing is a collector that
//! reads responses until the tagged one and keeps what it saw -- roughly a
//! hundred lines, against maintaining a fork of the client for the same
//! result. The one genuine gap is `[MODIFIED …]`, handled below.
//!
//! ## The `MODIFIED` gap
//!
//! `imap-proto` 0.16.7 has no `ResponseCode::Modified`. Its `resp_text` parser
//! is `opt(resp_text_code)` followed by free text, so an unrecognised code is
//! not an error: `opt` yields `None` and the whole `[MODIFIED 7,9] …` is
//! carried through as the response's `information` string. That is where this
//! module recovers it from. Parsing a response code out of a human-readable
//! field is not how this should work forever -- upstreaming a `Modified`
//! variant is the real fix -- but it is deterministic, it is covered by tests
//! against a server that emits the real syntax, and the alternative is having
//! no conflict detection at all.
use std::collections::BTreeMap;
use std::fmt::Debug;
use std::sync::atomic::{AtomicU64, Ordering};
use async_imap::Session;
// Reached through `async-imap`'s re-export rather than a direct dependency,
// so the parser types can never version-skew against the client using them.
use async_imap::imap_proto::{Response, ResponseCode, Status, UidSetMember};
use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt};
use crate::parser::MailError;
/// Bounded wait for any single response line.
///
/// A server that accepts a command and then stops answering must surface as a
/// stall, not as a sync that never returns. Mirrors the per-item FETCH timeout
/// the typed path already applies -- an unbounded read here would reintroduce
/// exactly the hang that bounded it.
const RESPONSE_READ_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(60);
/// A `COPYUID` response code (RFC 4315 section 3): the destination mailbox's
/// UIDVALIDITY, the source UIDs, and the UIDs they became.
///
/// Its presence is the only positive proof that a `COPY`/`MOVE` actually moved
/// something. Its **absence proves nothing**: RFC 6851 section 4.3 makes it
/// only `SHOULD` for `MOVE`, and RFC 4315 section 3 permits omitting it for a
/// `UIDNOTSTICKY` destination or one the client cannot `SELECT`.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct CopyUid {
/// UIDVALIDITY of the destination mailbox.
pub uid_validity: u32,
/// Source UIDs, expanded from the wire's set syntax.
pub source_uids: Vec<u32>,
/// Destination UIDs, positionally aligned with `source_uids`.
pub destination_uids: Vec<u32>,
}
/// Everything one command's responses carried, tagged and untagged.
#[derive(Debug, Default, Clone)]
pub struct RawExchange {
/// `true` when the tagged response was `OK`.
pub ok: bool,
/// The tagged response's human text, which is also where an unparsed
/// response code such as `[MODIFIED …]` ends up.
pub information: Option<String>,
/// `COPYUID`, from wherever it arrived. RFC 6851 section 4.3 advises
/// servers to send it in an **untagged** `OK` for `MOVE`, so a collector
/// that only inspected the tagged response would miss it on every
/// conforming server.
pub copy_uid: Option<CopyUid>,
/// UIDs reported by `VANISHED` (RFC 7162), whether or not `(EARLIER)`.
pub vanished_uids: Vec<u32>,
/// Sequence numbers reported by untagged `EXPUNGE`. Servers that have
/// QRESYNC enabled send `VANISHED` instead (RFC 6851 section 4.4), so a
/// delete proved by counting these alone reports failure on success.
pub expunged_seqs: Vec<u32>,
/// UIDs named by untagged `FETCH` responses.
pub fetched_uids: Vec<u32>,
/// `HIGHESTMODSEQ`, from an untagged `OK` during `SELECT`.
pub highest_modseq: Option<u64>,
/// UIDVALIDITY, from an untagged `OK` during `SELECT`.
pub uid_validity: Option<u32>,
/// UIDNEXT, from an untagged `OK` during `SELECT`.
pub uid_next: Option<u32>,
}
impl RawExchange {
/// UIDs the server refused to update because they changed since the
/// client's mod-sequence (RFC 7162 section 3.1.3).
///
/// `Some(vec![])` cannot occur: a `MODIFIED` code always names at least
/// one UID, so `None` means "no conflict reported" and `Some` means "these
/// specific UIDs conflicted".
pub fn modified_uids(&self) -> Option<Vec<u32>> {
self.information
.as_deref()
.and_then(parse_modified_response_code)
}
}
/// Recover `[MODIFIED <uid-set>]` from a tagged response's text.
///
/// See the module docs for why this is a string scan: `imap-proto` has no
/// `ResponseCode` variant for it, and its `opt(resp_text_code)` leaves the
/// unrecognised code in the free-text field rather than failing.
fn parse_modified_response_code(information: &str) -> Option<Vec<u32>> {
let start = information.find("[MODIFIED ")?;
let rest = &information[start + "[MODIFIED ".len()..];
let end = rest.find(']')?;
let uids = expand_uid_set_text(&rest[..end]);
if uids.is_empty() {
None
} else {
Some(uids)
}
}
/// Expand an IMAP `uid-set` in its wire text form (`7,9`, `1:4`, `1:3,7`).
fn expand_uid_set_text(spec: &str) -> Vec<u32> {
let mut out = Vec::new();
for part in spec.split(',') {
let part = part.trim();
if let Some((lo, hi)) = part.split_once(':') {
if let (Ok(lo), Ok(hi)) = (lo.trim().parse::<u32>(), hi.trim().parse::<u32>()) {
let (lo, hi) = if lo <= hi { (lo, hi) } else { (hi, lo) };
out.extend(lo..=hi);
}
} else if let Ok(uid) = part.parse::<u32>() {
out.push(uid);
}
}
out.sort_unstable();
out.dedup();
out
}
/// Flatten `imap-proto`'s uid-set representation into concrete UIDs.
fn expand_uid_set_members(members: &[UidSetMember]) -> Vec<u32> {
let mut out = Vec::new();
for member in members {
match member {
UidSetMember::Uid(uid) => out.push(*uid),
UidSetMember::UidRange(range) => out.extend(range.clone()),
}
}
out
}
/// Issue `command` and read responses until its tagged completion, keeping
/// everything that arrived on the way.
///
/// This is the whole point of the module. `run_command_and_check_ok` reads to
/// the same place and throws away the untagged responses *and* the tagged
/// response code, which between them carry every fact about whether the
/// command did anything.
pub async fn run_collected<S>(
session: &mut Session<S>,
command: &str,
) -> Result<RawExchange, MailError>
where
S: AsyncRead + AsyncWrite + Unpin + Send + Debug + 'static,
{
let tag = session
.run_command(command)
.await
.map_err(|e| MailError::ImapError(format!("failed to issue '{command}': {e}")))?;
let mut exchange = RawExchange::default();
loop {
let response = tokio::time::timeout(RESPONSE_READ_TIMEOUT, session.read_response())
.await
.map_err(|_| {
MailError::FetchStalled(format!(
"no response to '{command}' within {RESPONSE_READ_TIMEOUT:?}"
))
})?
.map_err(|e| MailError::ImapError(format!("reading response to '{command}': {e}")))?
.ok_or_else(|| {
MailError::ImapError(format!("connection closed before '{command}' completed"))
})?;
let mut done = false;
match response.parsed() {
Response::Done {
tag: response_tag,
status,
code,
information,
} => {
// Only our own tag ends the exchange; another command's
// completion interleaved here would otherwise truncate ours.
if response_tag == &tag {
exchange.ok = matches!(status, Status::Ok);
exchange.information = information.as_ref().map(|i| i.to_string());
absorb_code(&mut exchange, code.as_ref());
done = true;
}
}
Response::Data { code, .. } => absorb_code(&mut exchange, code.as_ref()),
Response::Vanished { uids, .. } => {
for range in uids {
exchange.vanished_uids.extend(range.clone());
}
}
Response::Expunge(seq) => exchange.expunged_seqs.push(*seq),
Response::Fetch(_, attributes) => {
for attribute in attributes {
if let async_imap::imap_proto::AttributeValue::Uid(uid) = attribute {
exchange.fetched_uids.push(*uid);
}
}
}
_ => {}
}
if done {
break;
}
}
Ok(exchange)
}
/// Fold a response code into the exchange, from a tagged or untagged response.
fn absorb_code(exchange: &mut RawExchange, code: Option<&ResponseCode<'_>>) {
match code {
Some(ResponseCode::CopyUid(uid_validity, source, destination)) => {
exchange.copy_uid = Some(CopyUid {
uid_validity: *uid_validity,
source_uids: expand_uid_set_members(source),
destination_uids: expand_uid_set_members(destination),
});
}
Some(ResponseCode::HighestModSeq(modseq)) => exchange.highest_modseq = Some(*modseq),
Some(ResponseCode::UidValidity(validity)) => exchange.uid_validity = Some(*validity),
Some(ResponseCode::UidNext(uid_next)) => exchange.uid_next = Some(*uid_next),
_ => {}
}
}
/// Issue `ENABLE` for `capabilities` (RFC 5161).
///
/// Returns whether the server accepted the command. It deliberately does not
/// try to confirm *which* capabilities were enabled: the untagged `* ENABLED`
/// line has no `imap-proto` representation, and confirming it is unnecessary
/// -- RFC 7162 section 3.2.3 requires a server to answer `BAD` to a QRESYNC
/// `SELECT` that was never enabled, so a QRESYNC `SELECT` that succeeds is
/// itself the proof.
pub async fn enable<S>(session: &mut Session<S>, capabilities: &[&str]) -> Result<bool, MailError>
where
S: AsyncRead + AsyncWrite + Unpin + Send + Debug + 'static,
{
if capabilities.is_empty() {
return Ok(true);
}
let exchange = run_collected(session, &format!("ENABLE {}", capabilities.join(" "))).await?;
Ok(exchange.ok)
}
/// `SELECT <mailbox> (QRESYNC (<uidvalidity> <modseq>))` (RFC 7162 section 3.2.5).
///
/// The returned exchange carries the `VANISHED` set and the changed `FETCH`
/// UIDs the server volunteered, which together are the entire reason to use
/// QRESYNC: they are how a client learns what disappeared while it was away.
/// A plain `SELECT` can never report that.
///
/// `known_uids` is optional; omitting it makes the server behave as if the
/// client knew `1:<UIDNEXT-1>` (section 3.2.5.1), which is correct but can
/// make the `VANISHED` set much larger than it needs to be.
pub async fn select_qresync<S>(
session: &mut Session<S>,
mailbox: &str,
uid_validity: u32,
modseq: u64,
known_uids: Option<&str>,
) -> Result<RawExchange, MailError>
where
S: AsyncRead + AsyncWrite + Unpin + Send + Debug + 'static,
{
let command = match known_uids {
Some(uids) => format!("SELECT {mailbox} (QRESYNC ({uid_validity} {modseq} {uids}))"),
None => format!("SELECT {mailbox} (QRESYNC ({uid_validity} {modseq}))"),
};
let exchange = run_collected(session, &command).await?;
if !exchange.ok {
return Err(MailError::ImapError(format!(
"QRESYNC SELECT of '{mailbox}' was refused: {}",
exchange.information.as_deref().unwrap_or("no detail")
)));
}
Ok(exchange)
}
/// `UID STORE <set> (UNCHANGEDSINCE <modseq>) <op> (<flag>)` (RFC 7162 section 3.1.3).
///
/// The compare-and-swap that turns a lost update into a detected one. Read the
/// result with [`RawExchange::modified_uids`]: `None` means every targeted UID
/// was updated, `Some(uids)` means those were left untouched because another
/// client changed them first.
///
/// `.SILENT` is appended because the untagged `FETCH` responses a non-silent
/// store produces are not the proof of effect they look like -- RFC 3501
/// section 6.4.6 lets a server suppress them, and a no-op store legitimately
/// produces none. `MODIFIED` is the signal that matters.
pub async fn uid_store_unchanged_since<S>(
session: &mut Session<S>,
uid_set: &str,
modseq: u64,
op: &str,
flag: &str,
) -> Result<RawExchange, MailError>
where
S: AsyncRead + AsyncWrite + Unpin + Send + Debug + 'static,
{
let command = format!("UID STORE {uid_set} (UNCHANGEDSINCE {modseq}) {op}.SILENT ({flag})");
let exchange = run_collected(session, &command).await?;
// A tagged NO/BAD is a genuine failure; a tagged OK carrying MODIFIED is a
// conflict, which is a successful command with a reportable outcome.
if !exchange.ok && exchange.modified_uids().is_none() {
return Err(MailError::ImapError(format!(
"conditional UID STORE on '{uid_set}' failed: {}",
exchange.information.as_deref().unwrap_or("no detail")
)));
}
Ok(exchange)
}
/// `UID MOVE <set> <destination>` (RFC 6851), collecting the `COPYUID` proof.
///
/// Check [`RawExchange::copy_uid`]: present means the move genuinely happened
/// and names the destination UIDs; **absent means unknown, not failed**. A UID
/// set matching nothing is a successful no-op per RFC 3501 section 6.4.8, and
/// a conforming server may also omit `COPYUID` for a `UIDNOTSTICKY`
/// destination.
pub async fn uid_move_collected<S>(
session: &mut Session<S>,
uid_set: &str,
destination: &str,
) -> Result<RawExchange, MailError>
where
S: AsyncRead + AsyncWrite + Unpin + Send + Debug + 'static,
{
let exchange = run_collected(session, &format!("UID MOVE {uid_set} {destination}")).await?;
if !exchange.ok {
return Err(MailError::ImapError(format!(
"UID MOVE of '{uid_set}' to '{destination}' failed: {}",
exchange.information.as_deref().unwrap_or("no detail")
)));
}
Ok(exchange)
}
/// `UID EXPUNGE <set>` (RFC 4315), collecting both removal reports.
///
/// A server with QRESYNC enabled answers with `VANISHED` rather than
/// `EXPUNGE` (RFC 6851 section 4.4), so both are gathered and callers should
/// treat either as evidence. Counting only `EXPUNGE` reports failure on every
/// successful delete against a QRESYNC server.
pub async fn uid_expunge_collected<S>(
session: &mut Session<S>,
uid_set: &str,
) -> Result<RawExchange, MailError>
where
S: AsyncRead + AsyncWrite + Unpin + Send + Debug + 'static,
{
let exchange = run_collected(session, &format!("UID EXPUNGE {uid_set}")).await?;
if !exchange.ok {
return Err(MailError::ImapError(format!(
"UID EXPUNGE of '{uid_set}' failed: {}",
exchange.information.as_deref().unwrap_or("no detail")
)));
}
Ok(exchange)
}
/// Upper bound on the bytes one raw exchange will accumulate before giving up.
///
/// The raw reader has no framing beyond "a line beginning with our tag", so a
/// server that answers with an endless stream and never completes the command
/// would otherwise grow this buffer without limit. One `EMAILID` line is a few
/// dozen bytes and a batch is capped at a few hundred messages, so this is
/// orders of magnitude above any legitimate reply.
const MAX_RAW_RESPONSE_BYTES: usize = 4 * 1024 * 1024;
/// Tag prefix for commands this module writes straight to the transport.
///
/// `async-imap` allocates its own tags from an `IdGenerator` that emits
/// `format!("A{:04}", n % 10_000)` -- always the letter `A` followed by four
/// decimal digits. A tag starting with `NX` therefore cannot collide with one
/// the typed client will ever issue, no matter how many commands a session
/// runs or how often the generator wraps. Colliding would be worse than
/// untidy: two commands sharing a tag makes each one's completion look like
/// the other's, and a reader would stop at the wrong response.
const RAW_TAG_PREFIX: &str = "NX";
/// Sequence behind [`RAW_TAG_PREFIX`]. Process-wide rather than per-session
/// because it costs nothing and makes a tag unique in a packet capture across
/// every connection, which is the only thing anyone reads these tags for.
static RAW_TAG_SEQUENCE: AtomicU64 = AtomicU64::new(0);
fn next_raw_tag() -> String {
let n = RAW_TAG_SEQUENCE.fetch_add(1, Ordering::Relaxed);
format!("{RAW_TAG_PREFIX}{n}")
}
/// `UID FETCH <set> (UID EMAILID)` (RFC 8474 section 5.2), read straight off
/// the transport, returning `uid -> EMAILID`.
///
/// # Why this bypasses the client entirely
///
/// `imap-proto` 0.16.7 cannot parse `EMAILID`: its `msg_att` parser is a closed
/// `alt(...)` of known attributes with no catch-all, so an `EMAILID` item is a
/// hard nom error rather than an unknown-but-tolerated one. `async-imap`'s
/// decoder turns any non-`Incomplete` parse error into a fatal `io::Error` and
/// then latches `read_closed`, after which the stream yields `None` forever --
/// the session is dead, not degraded. So `EMAILID` must never reach
/// `parse_response`, which rules out `Session::uid_fetch` **and**
/// [`run_collected`], since the latter reads through `Session::read_response`
/// and therefore through the same decoder.
///
/// What is left is the transport underneath: `Session` exposes `&mut T`
/// (`AsMut`, and `Connection::get_mut` through `Deref`), so the command is
/// written and its reply scanned here, by hand, and no byte of it is offered
/// to nom.
///
/// # The two conditions this is only correct under
///
/// 1. **The client's read buffer must be empty.** `ImapStream` owns a buffer
/// that can hold bytes already pulled off the socket. Raw reads bypass it,
/// so anything sitting there is invisible to this scanner. Call this only
/// at a quiescent point -- immediately after a previous command's *tagged*
/// completion has been read -- and never interleaved with an in-flight
/// typed command or a live response stream.
/// 2. **The tag must not collide** with the client's own. See
/// [`RAW_TAG_PREFIX`].
///
/// # Degrading rather than failing
///
/// A tagged `NO`/`BAD` yields an empty map, not an error. The object id is an
/// identity *upgrade*: without it a message still syncs under the
/// `Message-ID`+content tier or the surrogate, so a server that advertises
/// `OBJECTID` and then refuses the fetch must cost the sync precision, never
/// the mail. A stalled read is different and does surface, because a server
/// that stopped answering mid-command has left the session unusable.
pub async fn uid_fetch_email_ids<S>(
session: &mut Session<S>,
uid_set: &str,
) -> Result<BTreeMap<u32, String>, MailError>
where
S: AsyncRead + AsyncWrite + Unpin + Send + Debug + 'static,
{
if uid_set.is_empty() {
return Ok(BTreeMap::new());
}
let tag = next_raw_tag();
let command = format!("{tag} UID FETCH {uid_set} (UID EMAILID)\r\n");
let stream: &mut S = session.as_mut();
stream
.write_all(command.as_bytes())
.await
.map_err(|e| MailError::ImapError(format!("failed to issue '{command:?}': {e}")))?;
stream
.flush()
.await
.map_err(|e| MailError::ImapError(format!("failed to flush '{command:?}': {e}")))?;
let (lines, tagged) = read_until_tagged(stream, &tag).await?;
if !tagged_is_ok(&tagged) {
tracing::warn!(
response = %tagged,
"server refused UID FETCH (UID EMAILID) despite advertising OBJECTID; \
this pass identifies its messages by a weaker tier"
);
return Ok(BTreeMap::new());
}
let mut ids = BTreeMap::new();
for line in &lines {
if let Some((uid, email_id)) = parse_email_id_line(line) {
ids.insert(uid, email_id);
}
}
Ok(ids)
}
/// Read complete CRLF-terminated lines until one begins with `tag`, returning
/// the lines before it and the tagged line itself.
///
/// Line-oriented scanning is sound *for this command only*: an `EMAILID`
/// response carries no literal (`{n}`) section, so no line body can contain an
/// embedded CRLF. It must not be reused for a command whose reply can carry
/// literals -- a body fetch -- without adding literal handling.
async fn read_until_tagged<S>(stream: &mut S, tag: &str) -> Result<(Vec<String>, String), MailError>
where
S: AsyncRead + AsyncWrite + Unpin,
{
let mut buffer: Vec<u8> = Vec::new();
let mut chunk = [0u8; 4096];
let mut scanned = 0usize;
let mut lines: Vec<String> = Vec::new();
let tag_prefix = format!("{tag} ");
loop {
let read = tokio::time::timeout(RESPONSE_READ_TIMEOUT, stream.read(&mut chunk))
.await
.map_err(|_| {
MailError::FetchStalled(format!(
"no response to raw '{tag}' within {RESPONSE_READ_TIMEOUT:?}"
))
})?
.map_err(|e| MailError::ImapError(format!("reading response to raw '{tag}': {e}")))?;
if read == 0 {
return Err(MailError::ImapError(format!(
"connection closed before raw '{tag}' completed"
)));
}
buffer.extend_from_slice(&chunk[..read]);
if buffer.len() > MAX_RAW_RESPONSE_BYTES {
return Err(MailError::ImapError(format!(
"raw '{tag}' response exceeded {MAX_RAW_RESPONSE_BYTES} bytes without completing"
)));
}
// Consume every line that is now complete. `scanned` marks how far the
// buffer has already been split, so a reply arriving in many chunks is
// not re-scanned from the start each time.
while let Some(offset) = buffer[scanned..].iter().position(|b| *b == b'\n') {
let end = scanned + offset;
let line = String::from_utf8_lossy(&buffer[scanned..end])
.trim_end_matches('\r')
.to_string();
scanned = end + 1;
if line.starts_with(&tag_prefix) {
return Ok((lines, line));
}
lines.push(line);
}
}
}
/// Whether a tagged completion line reports `OK`.
fn tagged_is_ok(tagged: &str) -> bool {
tagged
.split_whitespace()
.nth(1)
.is_some_and(|status| status.eq_ignore_ascii_case("OK"))
}
/// Pull `(uid, EMAILID)` out of one untagged `FETCH` line.
///
/// Both wire spellings are accepted. RFC 8474 section 5.2 defines the response
/// as `"EMAILID" SP "(" objectid ")"`, but the bare `EMAILID <objectid>` form
/// is what a hand-rolled scanner meets often enough that rejecting it would
/// mean silently dropping to a weaker identity tier against a server that
/// answered the question correctly enough.
///
/// Returns `None` for anything that is not an untagged `FETCH` carrying both a
/// UID and a non-empty object id -- there is nothing to key on without both.
fn parse_email_id_line(line: &str) -> Option<(u32, String)> {
let upper = line.to_ascii_uppercase();
if !upper.starts_with("* ") || !upper.contains(" FETCH ") {
return None;
}
// `to_ascii_uppercase` is byte-length preserving, so offsets found in the
// uppercased copy index the original exactly. Item *names* are
// case-insensitive (RFC 3501 section 9); an object id is not, so the value
// is always taken from the original line.
let uid = find_item_value(&upper, line, "UID ")?.parse::<u32>().ok()?;
let email_id = find_item_value(&upper, line, "EMAILID ")?;
let email_id = email_id
.trim_start_matches('(')
.trim_end_matches(')')
.trim_matches('"')
.to_string();
if email_id.is_empty() {
return None;
}
Some((uid, email_id))
}
/// The token following FETCH item `name` in `line`, located via its uppercased
/// twin `upper`.
///
/// The name must sit on a token boundary, so `UID ` never matches the tail of
/// another attribute name. A parenthesised value is returned whole, brackets
/// included, for the caller to strip.
fn find_item_value(upper: &str, line: &str, name: &str) -> Option<String> {
let mut from = 0usize;
loop {
let at = from + upper[from..].find(name)?;
let boundary = at == 0
|| upper[..at]
.chars()
.next_back()
.is_some_and(|c| c == '(' || c == ' ');
if boundary {
let value_start = at + name.len();
let rest = &line[value_start..];
let end = if rest.starts_with('(') {
rest.find(')').map_or(rest.len(), |i| i + 1)
} else {
rest.find([' ', ')']).unwrap_or(rest.len())
};
let value = rest[..end].trim();
if !value.is_empty() {
return Some(value.to_string());
}
}
from = at + name.len();
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn an_object_id_is_read_from_both_wire_spellings() {
// RFC 8474 5.2 parenthesises the id; the bare form appears in the wild.
// Both must key the same message, or which spelling a server chose
// would decide whether identity converges.
assert_eq!(
parse_email_id_line("* 1 FETCH (UID 7 EMAILID (M00000001))"),
Some((7, "M00000001".to_string()))
);
assert_eq!(
parse_email_id_line("* 1 FETCH (UID 7 EMAILID M00000001)"),
Some((7, "M00000001".to_string()))
);
// Item names are case-insensitive; the id itself is not.
assert_eq!(
parse_email_id_line("* 1 fetch (uid 7 emailid (Mixed-Case_01))"),
Some((7, "Mixed-Case_01".to_string()))
);
// Order is not fixed, and other attributes may sit between them.
assert_eq!(
parse_email_id_line("* 3 FETCH (EMAILID (M9) FLAGS (\\Seen) UID 12)"),
Some((12, "M9".to_string()))
);
}
#[test]
fn a_line_missing_either_half_yields_nothing() {
// Half an answer is not a weaker identity, it is no identity: without
// the UID there is nothing to attach the id to, and without the id
// there is nothing to attach.
assert_eq!(
parse_email_id_line("* 1 FETCH (UID 7 FLAGS (\\Seen))"),
None
);
assert_eq!(parse_email_id_line("* 1 FETCH (EMAILID (M1))"), None);
assert_eq!(parse_email_id_line("* 1 FETCH (UID 7 EMAILID ())"), None);
assert_eq!(parse_email_id_line("NX0 OK UID FETCH completed"), None);
assert_eq!(parse_email_id_line("* 4 EXPUNGE"), None);
}
#[test]
fn an_item_name_is_only_matched_on_a_token_boundary() {
// RFC 8474 defines THREADID alongside EMAILID, so a line carrying both
// is ordinary. Every occurrence of `ID ` in it is the tail of a longer
// attribute name; without the boundary rule a search would read
// THREADID's value as the item's own and key the message on it.
let line = "* 1 FETCH (THREADID (T1) EMAILID (M1) UID 7)";
let upper = line.to_ascii_uppercase();
assert_eq!(find_item_value(&upper, line, "ID "), None);
assert_eq!(
find_item_value(&upper, line, "EMAILID "),
Some("(M1)".to_string())
);
assert_eq!(find_item_value(&upper, line, "UID "), Some("7".to_string()));
assert_eq!(parse_email_id_line(line), Some((7, "M1".to_string())));
}
#[test]
fn only_a_tagged_ok_counts_as_a_usable_answer() {
assert!(tagged_is_ok("NX7 OK UID FETCH completed"));
assert!(tagged_is_ok("NX7 ok uid fetch completed"));
assert!(!tagged_is_ok("NX7 NO server does not do that"));
assert!(!tagged_is_ok("NX7 BAD unknown item"));
assert!(!tagged_is_ok("NX7"));
}
#[test]
fn raw_tags_can_never_collide_with_the_clients_own() {
// async-imap's IdGenerator emits `A` + four digits and wraps at 10_000,
// so a shared tag is only impossible if the prefix differs.
let tag = next_raw_tag();
assert!(tag.starts_with(RAW_TAG_PREFIX));
assert_ne!(&tag[..1], "A");
assert_ne!(next_raw_tag(), tag, "each raw command gets its own tag");
}
#[test]
fn modified_response_code_is_recovered_from_the_tagged_text() {
// imap-proto has no ResponseCode::Modified, and its opt(resp_text_code)
// leaves the unrecognised code in the free-text field. This is the
// contract that behaviour gives us; if a future imap-proto starts
// parsing MODIFIED, this test is what will notice.
let exchange = RawExchange {
information: Some("[MODIFIED 7,9] Conditional STORE failed".to_string()),
..RawExchange::default()
};
assert_eq!(exchange.modified_uids(), Some(vec![7, 9]));
}
#[test]
fn modified_parses_ranges_and_reports_none_when_absent() {
assert_eq!(
parse_modified_response_code("[MODIFIED 1:3,7] failed"),
Some(vec![1, 2, 3, 7])
);
assert_eq!(
parse_modified_response_code("Conditional STORE failed"),
None
);
assert_eq!(parse_modified_response_code("[MODIFIED ] weird"), None);
assert_eq!(
parse_modified_response_code("[MODIFIED 7 unterminated"),
None
);
}
#[test]
fn no_modified_means_no_conflict_not_an_empty_conflict() {
// The distinction matters: `Some(vec![])` would read as "a conflict
// affecting nothing", which is not a state the protocol can produce.
let exchange = RawExchange {
information: Some("STORE completed".to_string()),
..RawExchange::default()
};
assert_eq!(exchange.modified_uids(), None);
}
#[test]
fn uid_sets_expand_from_both_wire_forms() {
assert_eq!(expand_uid_set_text("5"), vec![5]);
assert_eq!(expand_uid_set_text("1:4"), vec![1, 2, 3, 4]);
assert_eq!(expand_uid_set_text("9,1:3"), vec![1, 2, 3, 9]);
// Reversed ranges are normalized rather than yielding nothing.
assert_eq!(expand_uid_set_text("4:2"), vec![2, 3, 4]);
assert_eq!(
expand_uid_set_members(&[UidSetMember::Uid(3), UidSetMember::UidRange(7..=9)]),
vec![3, 7, 8, 9]
);
}
}
crates/nuncio-mail/src/jmap.rs
+769 -128
View File
File diff suppressed because it is too large. Load diff
crates/nuncio-mail/src/lib.rs
+12 -2
View File
@@ -5,14 +5,24 @@
pub mod backend;
pub mod imap;
pub mod imap_raw;
pub mod jmap;
pub mod mock;
pub mod parser;
pub mod smtp;
pub use backend::{MailBackend, MessageSender, OutboundMessage};
// A real loopback IMAP server for offline tests. Behind a feature so it never
// enters a production build, but visible to this crate's own tests and to
// dependents that opt in (integration suites in `nunciod`).
#[cfg(any(test, feature = "test-support"))]
pub mod test_server;
pub use backend::{
FolderChanges, MailBackend, MessageSender, MutationOutcome, OutboundMessage, PlacedMessage,
RemoteMutationKind, RemoteMutationSpec,
};
pub use imap::{IdleSocketState, ImapDualSocketManager, ImapEngine};
pub use jmap::JmapEngine;
pub use jmap::{EmailChangesPage, JmapEngine};
pub use mock::{MockMailBackend, MockMessageSender};
pub use parser::{MailError, MimeParserAdapter};
pub use smtp::SmtpTransportEngine;
crates/nuncio-mail/src/mock.rs
+204 -40
View File
@@ -1,18 +1,20 @@
//! Deterministic mock mail backend for offline testing and integration verification.
use async_trait::async_trait;
use nuncio_core::model::{Email, Folder};
use nuncio_core::model::{Email, Folder, IdentitySource, Placement, PlacementKey};
use std::sync::{Arc, Mutex};
use crate::backend::{MailBackend, MessageSender, OutboundMessage};
use crate::backend::{
FolderChanges, MailBackend, MessageSender, MutationOutcome, OutboundMessage, PlacedMessage,
RemoteMutationSpec,
};
use crate::parser::MailError;
/// Thread-safe mock mail backend for offline testing.
#[derive(Debug, Clone, Default)]
pub struct MockMailBackend {
folders: Arc<Mutex<Vec<Folder>>>,
messages: Arc<Mutex<Vec<Email>>>,
sent_messages: Arc<Mutex<Vec<Email>>>,
messages: Arc<Mutex<Vec<PlacedMessage>>>,
should_fail: Arc<Mutex<bool>>,
/// Every `since_state` argument this mock's `sync_messages` was called
/// with, in call order, so tests can prove a caller threads the returned
@@ -20,6 +22,14 @@ pub struct MockMailBackend {
since_state_calls: Arc<Mutex<Vec<Option<String>>>>,
/// The checkpoint token this mock returns from `sync_messages`.
returned_state: Arc<Mutex<String>>,
/// Every [`RemoteMutationSpec`] passed to `apply_mutation`, in call order,
/// so a daemon E2E can prove the outbox executor genuinely invoked the
/// backend op (not a fabricated completion).
applied_mutations: Arc<Mutex<Vec<RemoteMutationSpec>>>,
/// Occupancies this mock reports as explicitly gone, the way QRESYNC
/// `VANISHED` does. Handed back on the pass for the folder each key names,
/// so a test can prove a caller acts on a server-confirmed removal.
removals: Arc<Mutex<Vec<PlacementKey>>>,
}
impl MockMailBackend {
@@ -51,6 +61,15 @@ impl MockMailBackend {
.unwrap_or_default()
}
/// Stage occupancies this mock reports as explicitly gone, as a QRESYNC
/// `VANISHED` would. Each key is handed back on the pass for the folder it
/// names, so staging a removal for `Archive` does not leak into `INBOX`.
pub fn set_removals(&self, removals: Vec<PlacementKey>) {
if let Ok(mut guard) = self.removals.lock() {
*guard = removals;
}
}
/// Configure the mock to simulate network failure errors.
pub fn set_should_fail(&self, fail: bool) {
if let Ok(mut flag) = self.should_fail.lock() {
@@ -65,16 +84,87 @@ impl MockMailBackend {
}
}
/// Add a mock email message to the storage.
pub fn add_message(&self, email: Email) {
/// Add a mock message to the storage, as the occupancy a backend would
/// have found it in. Taking a [`PlacedMessage`] rather than a bare `Email`
/// lets a test stage the same message key in two folders and prove the
/// caller treats it as one message in two places.
pub fn add_message(&self, message: PlacedMessage) {
if let Ok(mut guard) = self.messages.lock() {
guard.push(email);
guard.push(message);
}
}
/// Retrieve sent messages recorded by the mock.
pub fn sent_messages(&self) -> Vec<Email> {
self.sent_messages
/// Account every message staged by [`Self::shared_message_in`] and
/// [`Self::with_same_message_in`] belongs to.
pub const SHARED_ACCOUNT_ID: &'static str = "acct-1";
/// The one derived message key every occupancy staged by
/// [`Self::shared_message_in`] carries, whatever folder it sits in.
pub const SHARED_MESSAGE_KEY: &'static str = "msg-shared-identity";
/// One occupancy of the single shared message, in `folder_id` under
/// `remote_id`.
///
/// Identity is byte-identical across every folder -- that is the whole
/// point: a caller cannot tell a first arrival in a second mailbox from a
/// message it already stores unless it compares occupancies rather than
/// message keys.
pub fn shared_message_in(folder_id: &str, remote_id: &str) -> PlacedMessage {
PlacedMessage {
email: Email {
id: Self::SHARED_MESSAGE_KEY.to_string(),
account_id: Self::SHARED_ACCOUNT_ID.to_string(),
subject: "Shared across folders".to_string(),
sender: "alice@nuncio.mx".to_string(),
recipient: "bob@nuncio.mx".to_string(),
received_at: 1_700_000_000,
body_plain: Some("one message, several mailboxes".to_string()),
body_html: None,
message_id: Some("shared@nuncio.mx".to_string()),
content_hash: None,
attachments: Vec::new(),
},
// EmailId, not Surrogate: a folder-independent key is exactly what
// a server-assigned identity buys, and it is the tier under which
// one message legitimately reports from several mailboxes.
source: IdentitySource::EmailId,
placement: Placement {
account_id: Self::SHARED_ACCOUNT_ID.to_string(),
folder_id: folder_id.to_string(),
uid_validity: "1".to_string(),
remote_id: remote_id.to_string(),
read: false,
},
}
}
/// A backend serving `folders`, each holding the same message under its own
/// `remote_id` -- one identity, one occupancy per folder.
///
/// Stages the folders as well as the messages, so a caller that enumerates
/// folders and then syncs each one sees every occupancy.
pub fn with_same_message_in(folders: &[&str]) -> Self {
let backend = Self::new();
for (index, folder_id) in folders.iter().enumerate() {
backend.add_folder(Folder {
id: (*folder_id).to_string(),
name: (*folder_id).to_string(),
total_messages: 1,
unread_messages: 1,
});
// A distinct UID per folder: UIDs are folder-scoped, so the same
// mail in two mailboxes is addressed by two different ones.
backend.add_message(Self::shared_message_in(folder_id, &(index + 1).to_string()));
}
backend
}
/// Every [`RemoteMutationSpec`] this mock's `apply_mutation` was called
/// with, in call order. A failing configuration (`set_should_fail(true)`)
/// records nothing, so a test can prove a failed op is never counted as
/// applied.
pub fn applied_mutations(&self) -> Vec<RemoteMutationSpec> {
self.applied_mutations
.lock()
.map(|guard| guard.clone())
.unwrap_or_default()
@@ -100,11 +190,11 @@ impl MailBackend for MockMailBackend {
Ok(folders.clone())
}
async fn sync_messages(
async fn sync_changes(
&self,
folder_id: &str,
since_state: Option<&str>,
) -> Result<(Vec<Email>, String), MailError> {
) -> Result<FolderChanges, MailError> {
// Record the checkpoint arg before any early return, so even a failing
// call is visible to a test asserting on how the caller resumes.
if let Ok(mut calls) = self.since_state_calls.lock() {
@@ -126,41 +216,72 @@ impl MailBackend for MockMailBackend {
.map_err(|e| MailError::ParseFailed(e.to_string()))?
.clone();
// Staged removals are scoped to the folder each key names, and are
// reported on both the full and the incremental pass -- a `VANISHED`
// report is exactly the kind of thing an incremental resync carries.
let removals: Vec<PlacementKey> = self
.removals
.lock()
.map_err(|e| MailError::ParseFailed(e.to_string()))?
.iter()
.filter(|key| key.folder_id == folder_id)
.cloned()
.collect();
// Simulate a since_state-aware backend: with a prior checkpoint there
// is nothing new to hand back, so the incremental fetch is genuinely
// narrower (empty) rather than a full re-report of every message.
// `present` stays `None`: an incremental pass cannot speak to absence.
if since_state.is_some() {
return Ok((Vec::new(), returned_state));
return Ok(FolderChanges {
removals,
next_state: returned_state,
..Default::default()
});
}
let messages = self
.messages
.lock()
.map_err(|e| MailError::ParseFailed(e.to_string()))?;
let matches = messages
let matches: Vec<_> = messages
.iter()
.filter(|m| m.folder_id == folder_id)
.filter(|m| m.placement.folder_id == folder_id)
.cloned()
.collect();
Ok((matches, returned_state))
// A full pass over the mock's store did see everything in the folder,
// so it can honestly report what is present -- as occupancies, since
// that is what a folder can speak to.
let present = matches.iter().map(|m| m.placement.key()).collect();
Ok(FolderChanges {
upserts: matches,
removals,
present: Some(present),
next_state: returned_state,
})
}
async fn send_email(&self, email: &Email) -> Result<(), MailError> {
async fn apply_mutation(
&self,
spec: &RemoteMutationSpec,
) -> Result<MutationOutcome, MailError> {
let should_fail = self
.should_fail
.lock()
.map_err(|e| MailError::ParseFailed(e.to_string()))?;
if *should_fail {
return Err(MailError::ParseFailed(
"simulated network failure".to_string(),
return Err(MailError::ImapError(
"simulated remote mutation failure".to_string(),
));
}
let mut sent = self
.sent_messages
// Record only genuinely-applied mutations, so a test can distinguish a
// real op from a fabricated completion.
let mut applied = self
.applied_mutations
.lock()
.map_err(|e| MailError::ParseFailed(e.to_string()))?;
sent.push(email.clone());
Ok(())
applied.push(spec.clone());
Ok(MutationOutcome::Applied { token: None })
}
}
@@ -232,20 +353,30 @@ mod tests {
unread_messages: 1,
});
let email = Email {
id: "msg-mock-1".to_string(),
account_id: "acct-1".to_string(),
folder_id: "inbox".to_string(),
subject: "Mock Test".to_string(),
sender: "alice@nuncio.mx".to_string(),
recipient: "bob@nuncio.mx".to_string(),
received_at: 1700000000,
read: false,
body_plain: Some("Mock body".to_string()),
body_html: None,
attachments: Vec::new(),
let placed = PlacedMessage {
email: Email {
id: "msg-mock-1".to_string(),
account_id: "acct-1".to_string(),
subject: "Mock Test".to_string(),
sender: "alice@nuncio.mx".to_string(),
recipient: "bob@nuncio.mx".to_string(),
received_at: 1700000000,
body_plain: Some("Mock body".to_string()),
body_html: None,
attachments: Vec::new(),
message_id: None,
content_hash: None,
},
source: IdentitySource::Surrogate,
placement: Placement {
account_id: "acct-1".to_string(),
folder_id: "inbox".to_string(),
uid_validity: "1".to_string(),
remote_id: "1".to_string(),
read: false,
},
};
mock.add_message(email.clone());
mock.add_message(placed.clone());
let folders = mock.sync_folders().await.expect("sync folders succeeds");
assert_eq!(folders.len(), 1);
@@ -255,16 +386,49 @@ mod tests {
.await
.expect("sync messages succeeds");
assert_eq!(msgs.len(), 1);
assert_eq!(
msgs[0], placed,
"the mock must hand back exactly the placement it was staged with"
);
assert_eq!(state, "mock-state-token-100");
mock.send_email(&email).await.expect("send succeeds");
assert_eq!(mock.sent_messages().len(), 1);
// Test error simulation
mock.set_should_fail(true);
assert!(mock.sync_folders().await.is_err());
assert!(mock.sync_messages("inbox", None).await.is_err());
assert!(mock.send_email(&email).await.is_err());
}
/// The staging helper is only useful if it really produces one identity in
/// two places: same message key, different occupancies. If it ever drifted
/// into two keys, every test built on it would pass for the wrong reason.
#[tokio::test]
async fn with_same_message_in_stages_one_identity_across_every_folder() {
let mock = MockMailBackend::with_same_message_in(&["INBOX", "Archive"]);
let folders = mock.sync_folders().await.expect("sync folders succeeds");
let folder_ids: Vec<&str> = folders.iter().map(|f| f.id.as_str()).collect();
assert_eq!(folder_ids, vec!["INBOX", "Archive"]);
let inbox = mock
.sync_changes("INBOX", None)
.await
.expect("inbox pass succeeds");
let archive = mock
.sync_changes("Archive", None)
.await
.expect("archive pass succeeds");
assert_eq!(inbox.upserts.len(), 1);
assert_eq!(archive.upserts.len(), 1);
assert_eq!(
inbox.upserts[0].email.id, archive.upserts[0].email.id,
"both occupancies must carry the same derived message key"
);
assert_ne!(
inbox.upserts[0].placement.key(),
archive.upserts[0].placement.key(),
"the occupancies themselves must differ, or there is nothing to classify"
);
}
fn sample_outbound_message() -> OutboundMessage {
crates/nuncio-mail/src/parser.rs
+56 -15
View File
@@ -61,12 +61,15 @@ impl MimeParserAdapter {
pub const MAX_PAYLOAD_BYTES: usize = 25 * 1024 * 1024;
/// Parse a raw RFC 5322 byte slice into an [`Email`] domain entity.
pub fn parse_mime(
id: &str,
account_id: &str,
folder_id: &str,
raw_bytes: &[u8],
) -> Result<Email, MailError> {
///
/// The returned `Email` carries an **empty** `id`, and it is the caller's
/// job to fill it from [`Email::derive_message_key`]. The derivation needs
/// inputs this parser cannot see -- the transport's addressing coordinates
/// and any server-assigned identity (RFC 8474 `EMAILID`, `X-GM-MSGID`) --
/// alongside the `Message-ID` and content hash recovered here, so the id
/// can only be settled once both halves are in hand. Minting a provisional
/// value here would risk one reaching the store as though it were derived.
pub fn parse_mime(account_id: &str, raw_bytes: &[u8]) -> Result<Email, MailError> {
if raw_bytes.len() > Self::MAX_PAYLOAD_BYTES {
return Err(MailError::ParseFailed(
"MIME payload exceeds maximum allowed limit of 25MB".to_string(),
@@ -95,6 +98,9 @@ impl MimeParserAdapter {
let received_at = msg.date().map_or(0, |d| d.to_timestamp());
let message_id = msg.message_id().and_then(Email::normalize_message_id);
let content_hash = Some(Email::content_hash_of(raw_bytes));
let body_plain = msg.body_text(0).map(|b| b.to_string());
let body_html = msg.body_html(0).map(|b| b.to_string());
@@ -126,17 +132,19 @@ impl MimeParserAdapter {
}
Ok(Email {
id: id.to_string(),
// Assigned by the caller from the derived message key; see the
// method doc for why it cannot be settled here.
id: String::new(),
account_id: account_id.to_string(),
folder_id: folder_id.to_string(),
subject,
sender,
recipient,
received_at,
read: false,
body_plain,
body_html,
attachments,
message_id,
content_hash,
})
}
}
@@ -154,9 +162,12 @@ mod tests {
\r\n\
Hello Bob, this is a plain text email.";
let email =
MimeParserAdapter::parse_mime("msg-1", "acct-1", "inbox", raw).expect("parse succeeds");
assert_eq!(email.id, "msg-1");
let email = MimeParserAdapter::parse_mime("acct-1", raw).expect("parse succeeds");
assert_eq!(
email.id, "",
"the parser must not mint an id; the caller derives it"
);
assert_eq!(email.account_id, "acct-1");
assert_eq!(email.subject, "Test Subject");
assert_eq!(email.sender, "alice@nuncio.mx");
assert_eq!(email.recipient, "bob@nuncio.mx");
@@ -166,11 +177,41 @@ mod tests {
);
}
#[test]
fn parse_captures_normalized_message_id_and_content_hash() {
let raw = b"From: Alice <alice@nuncio.mx>\r\n\
To: Bob <bob@nuncio.mx>\r\n\
Message-ID: <Abc.123@mail.nuncio.mx>\r\n\
Subject: Test\r\n\
\r\n\
body";
let email = MimeParserAdapter::parse_mime("acct-1", raw).expect("parse succeeds");
assert_eq!(email.message_id.as_deref(), Some("Abc.123@mail.nuncio.mx"));
assert_eq!(
email.content_hash,
Some(nuncio_core::model::Email::content_hash_of(raw)),
"the hash must cover the octets actually parsed"
);
}
#[test]
fn parse_reports_no_message_id_when_the_header_is_absent() {
// RFC 5322 3.6.4 makes Message-ID a SHOULD. An absent header must read
// as "none present", never as an invented or empty value.
let raw = b"From: Alice <alice@nuncio.mx>\r\n\
Subject: No id\r\n\
\r\n\
body";
let email = MimeParserAdapter::parse_mime("acct-1", raw).expect("parse succeeds");
assert_eq!(email.message_id, None);
assert!(email.content_hash.is_some());
}
#[test]
fn parse_invalid_mime_returns_error() {
let raw = b"";
let err = MimeParserAdapter::parse_mime("msg-2", "acct-1", "inbox", raw)
.expect_err("should fail");
let err = MimeParserAdapter::parse_mime("acct-1", raw).expect_err("should fail");
assert_eq!(
err,
MailError::ParseFailed("invalid RFC 5322 MIME structure".to_string())
@@ -184,7 +225,7 @@ mod tests {
#[test]
fn parse_oversized_mime_returns_error() {
let oversized = vec![0u8; MimeParserAdapter::MAX_PAYLOAD_BYTES + 1];
let err = MimeParserAdapter::parse_mime("msg-huge", "acct-1", "inbox", &oversized)
let err = MimeParserAdapter::parse_mime("acct-1", &oversized)
.expect_err("should fail for oversized payload");
assert!(err
.to_string()
crates/nuncio-mail/src/smtp.rs
+59 -8
View File
@@ -88,21 +88,49 @@ impl SmtpTransportEngine {
/// Send a composed [`OutboundMessage`] using the inner transport client.
/// Returns `Ok(())` only when the transport genuinely accepted the
/// message.
/// message. Logs the send attempt and outcome with the recipient COUNT
/// only (`to` plus `cc` when present) -- never the addresses themselves.
pub async fn send_message(&self, message: &OutboundMessage) -> Result<(), MailError> {
let recipient_count =
1 + usize::from(message.cc.as_ref().is_some_and(|cc| !cc.trim().is_empty()));
tracing::info!(recipient_count, "smtp send attempt");
let msg = Self::build_outbound_mime_message(message)?;
self.transport.send(msg).await.map_err(MailError::from)?;
Ok(())
match self.transport.send(msg).await {
Ok(_) => {
tracing::info!(recipient_count, "smtp send succeeded");
Ok(())
}
Err(e) => {
let err = MailError::from(e);
tracing::warn!(recipient_count, error = %err, "smtp send failed");
Err(err)
}
}
}
/// Send an email message using a provided [`AsyncSmtpTransport`] client instance.
/// Send an email message using a provided [`AsyncSmtpTransport`] client
/// instance. Logs the send attempt and outcome with the recipient COUNT
/// only -- never the address itself.
pub async fn send_email_with_transport(
transport: &AsyncSmtpTransport<Tokio1Executor>,
email: &Email,
) -> Result<(), MailError> {
let recipient_count = 1;
tracing::info!(recipient_count, "smtp send attempt");
let msg = Self::build_mime_message(email)?;
transport.send(msg).await.map_err(MailError::from)?;
Ok(())
match transport.send(msg).await {
Ok(_) => {
tracing::info!(recipient_count, "smtp send succeeded");
Ok(())
}
Err(e) => {
let err = MailError::from(e);
tracing::warn!(recipient_count, error = %err, "smtp send failed");
Err(err)
}
}
}
/// Build an RFC 5322 [`lettre::Message`] from a Nuncio [`Email`] entity.
@@ -323,20 +351,21 @@ mod tests {
use super::*;
use bytes::Bytes;
use nuncio_core::model::Attachment;
use tracing_test::traced_test;
fn sample_email() -> Email {
Email {
id: "msg-1".to_string(),
account_id: "acct-1".to_string(),
folder_id: "outbox".to_string(),
subject: "Status Update".to_string(),
sender: "alice@nuncio.mx".to_string(),
recipient: "bob@nuncio.mx".to_string(),
received_at: 1700000000,
read: true,
body_plain: Some("Plaintext status update".to_string()),
body_html: None,
attachments: Vec::new(),
message_id: None,
content_hash: None,
}
}
@@ -606,6 +635,28 @@ mod tests {
));
}
#[traced_test]
#[tokio::test]
async fn send_message_logs_attempt_and_failure_with_recipient_count_never_addresses() {
let engine = SmtpTransportEngine::new("127.0.0.1", 1, TlsMode::ImplicitTls, "user", "pass")
.expect("valid config");
let mut message = sample_outbound_message();
message.cc = Some("carol@nuncio.mx".to_string());
let _ = engine
.send_message(&message)
.await
.expect_err("delivery to port 1 should fail");
assert!(logs_contain("smtp send attempt"));
assert!(logs_contain("recipient_count=2"));
assert!(logs_contain("smtp send failed"));
assert!(!logs_contain("alice@nuncio.mx"));
assert!(!logs_contain("bob@nuncio.mx"));
assert!(!logs_contain("carol@nuncio.mx"));
assert!(!logs_contain("Status Update"));
}
#[tokio::test]
async fn message_sender_trait_impl_delegates_to_send_message() {
// Proves `SmtpTransportEngine`'s `MessageSender` trait impl is wired
crates/nuncio-mail/src/test_server.rs
+1441
View File
File diff suppressed because it is too large. Load diff
crates/nuncio-mail/tests/imap_identity_test.rs
+231
View File
@@ -0,0 +1,231 @@
//! Offline tests for server-assigned message identity over IMAP.
//!
//! The claim under test is the one ADR 0002 rests on: a message has one
//! account-scoped key, and which protocol observed it does not change that key.
//! RFC 8474 exists precisely so a server implementing both IMAP and JMAP
//! reports the same object id through both, so the last test here compares an
//! IMAP-derived key against a JMAP-derived one and requires them to be equal.
//!
//! Everything runs against [`MockImapServer`] over loopback and against a JMAP
//! response parsed from a literal; no test here reaches a network.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use nuncio_core::model::IdentitySource;
use nuncio_mail::backend::FolderChanges;
use nuncio_mail::test_server::{IdentityExtensions, MockImapServer, ObjectIds, ServerProfile};
use nuncio_mail::{JmapEngine, MailBackend};
/// Sync one folder through the real engine against the mock.
async fn sync(server: &MockImapServer, account_id: &str, folder: &str) -> FolderChanges {
let engine = nuncio_mail::ImapEngine::with_credentials(
account_id,
server.host(),
server.port(),
nuncio_core::TlsMode::Plain,
"user",
"pass",
);
engine
.sync_changes(folder, None)
.await
.expect("folder sync succeeds")
}
#[tokio::test]
async fn an_objectid_server_keys_messages_by_their_emailid() {
// Tier 1. Reaching it at all is the difference between an identity the
// server vouches for and one derived from whatever the message happened to
// contain.
let server =
MockImapServer::start_with_identity(ServerProfile::Qresync, IdentityExtensions::objectid())
.await
.expect("server starts");
server.create_mailbox("INBOX");
let uid = server.append_message("INBOX", "Subject: one\r\n\r\nbody", &[]);
let changes = sync(&server, "acct-objectid", "INBOX").await;
assert_eq!(changes.upserts.len(), 1);
assert_eq!(
changes.upserts[0].source,
IdentitySource::EmailId,
"an OBJECTID server must be asked for EMAILID and the answer used"
);
assert_eq!(changes.upserts[0].placement.remote_id, uid.to_string());
assert!(
server
.commands_matching("EMAILID")
.iter()
.any(|c| c.to_ascii_uppercase().contains("UID FETCH")),
"the object id must come from a real UID FETCH, got {:?}",
server.commands()
);
}
#[tokio::test]
async fn the_same_message_in_a_second_folder_keys_identically() {
// The property a folder-scoped surrogate can never have, and the reason
// tier 1 is worth a separate round trip: two placements, one message.
let server =
MockImapServer::start_with_identity(ServerProfile::Qresync, IdentityExtensions::objectid())
.await
.expect("server starts");
server.create_mailbox("INBOX");
server.create_mailbox("Archive");
let shared = ObjectIds {
email_id: "M00000042".to_string(),
gm_msgid: 1_278_455_344_230_334_865,
};
let body = "Subject: filed\r\n\r\nbody";
server.append_message_with_object_ids("INBOX", body, &[], shared.clone());
server.append_message_with_object_ids("Archive", body, &[], shared);
let inbox = sync(&server, "acct-shared", "INBOX").await;
let archive = sync(&server, "acct-shared", "Archive").await;
assert_eq!(inbox.upserts.len(), 1);
assert_eq!(archive.upserts.len(), 1);
assert_eq!(
inbox.upserts[0].email.id, archive.upserts[0].email.id,
"one message in two folders must be one identity"
);
assert_eq!(inbox.upserts[0].source, IdentitySource::EmailId);
assert_eq!(archive.upserts[0].source, IdentitySource::EmailId);
assert_ne!(
inbox.upserts[0].placement, archive.upserts[0].placement,
"the two occupancies are still distinct; only the message is shared"
);
}
#[tokio::test]
async fn a_gmail_shaped_server_keys_messages_by_x_gm_msgid() {
// Gmail advertises X-GM-EXT-1 and not OBJECTID, so tier 2 is the best it
// offers -- and it is reachable through the typed client, with no extra
// round trip, because `imap-proto` can parse that attribute.
let server =
MockImapServer::start_with_identity(ServerProfile::Condstore, IdentityExtensions::gmail())
.await
.expect("server starts");
server.create_mailbox("INBOX");
server.append_message("INBOX", "Subject: gmail\r\n\r\nbody", &[]);
let changes = sync(&server, "acct-gmail", "INBOX").await;
assert_eq!(changes.upserts.len(), 1);
assert_eq!(
changes.upserts[0].source,
IdentitySource::GmailMsgId,
"X-GM-EXT-1 must lift identity to the Gmail message id"
);
assert!(
server.commands_matching("EMAILID").is_empty(),
"a server without OBJECTID must never be asked for EMAILID, got {:?}",
server.commands()
);
}
#[tokio::test]
async fn a_server_advertising_neither_extension_is_left_exactly_as_it_was() {
// The regression that matters most: capability negotiation must be
// invisible to a server that has neither extension. No new command, and
// the same tier it reached before any of this existed.
for profile in [
ServerProfile::Qresync,
ServerProfile::Condstore,
ServerProfile::Basic,
] {
let server = MockImapServer::start(profile).await.expect("server starts");
server.create_mailbox("INBOX");
server.append_message("INBOX", "Subject: plain\r\n\r\nbody", &[]);
let changes = sync(&server, "acct-plain", "INBOX").await;
assert_eq!(changes.upserts.len(), 1, "{profile:?}");
assert_eq!(
changes.upserts[0].source,
IdentitySource::Surrogate,
"{profile:?}: with no server id and no octets, the surrogate is the honest answer"
);
assert!(
server.commands_matching("EMAILID").is_empty(),
"{profile:?}: no EMAILID command may be issued, got {:?}",
server.commands()
);
assert!(
server.commands_matching("X-GM-MSGID").is_empty(),
"{profile:?}: no X-GM-MSGID item may be requested, got {:?}",
server.commands()
);
}
}
#[tokio::test]
async fn an_imap_emailid_and_a_jmap_object_id_produce_the_same_key() {
// The convergence assertion. One account, one message, one server-assigned
// id -- reported as RFC 8474 EMAILID over IMAP and as the Email object id
// over JMAP. If either engine derived its key from a different tier, or
// hashed under a different domain label, these two strings would differ.
const ACCOUNT: &str = "acct-converge";
const OBJECT_ID: &str = "M00000001";
let server =
MockImapServer::start_with_identity(ServerProfile::Qresync, IdentityExtensions::objectid())
.await
.expect("server starts");
server.create_mailbox("INBOX");
server.append_message_with_object_ids(
"INBOX",
"Subject: both\r\n\r\nbody",
&[],
ObjectIds {
email_id: OBJECT_ID.to_string(),
gm_msgid: 1,
},
);
let imap = sync(&server, ACCOUNT, "INBOX").await;
let jmap_response = format!(
r#"{{"methodResponses":[["Email/get",{{"accountId":"{ACCOUNT}",
"state":"s-1","list":[{{"id":"{OBJECT_ID}","subject":"both",
"from":[{{"email":"a@nuncio.mx"}}],"to":[{{"email":"b@nuncio.mx"}}],
"receivedAt":1700001000,"isUnread":true}}]}},"c1"]]}}"#
);
let (jmap, _state) = JmapEngine::new(ACCOUNT)
.parse_email_get_response(&jmap_response, "mb-inbox")
.expect("the JMAP response parses");
assert_eq!(imap.upserts.len(), 1);
assert_eq!(jmap.len(), 1);
assert_eq!(
imap.upserts[0].source,
IdentitySource::EmailId,
"the IMAP side must have entered at the object-id tier"
);
assert_eq!(
jmap[0].source,
IdentitySource::EmailId,
"the JMAP side must have entered at the object-id tier"
);
assert_eq!(
imap.upserts[0].email.id, jmap[0].email.id,
"two engines syncing one account must agree on what the message is"
);
// Equality alone would survive both engines moving to a different tier or
// a different domain label together, since they share one derivation. The
// golden pins the actual value: SHA-256 over the length-prefixed
// (tier, account, object id) triple. It is a tripwire, not a spec -- if it
// moves, every stored message_key has moved with it, and
// `IDENTITY_SCHEMA_VERSION` must move in the same commit.
const EXPECTED_KEY: &str = "ba2eb228c6b973c548837e32b84e95cf06b73664b3e350f5dfa30614b2e38bd0";
assert_eq!(
imap.upserts[0].email.id, EXPECTED_KEY,
"the object-id tier's derivation changed; bump IDENTITY_SCHEMA_VERSION with it"
);
// And the two engines still describe *where* it sits in their own terms:
// convergence is a claim about identity, never about placement.
assert_ne!(imap.upserts[0].placement, jmap[0].placement);
}
crates/nuncio-mail/tests/imap_raw_wire_test.rs
+714
View File
@@ -0,0 +1,714 @@
//! Wire-level tests for the raw IMAP command layer.
//!
//! These drive the real `async-imap` client over real TCP against
//! [`MockImapServer`], because the whole claim being tested is that responses
//! the typed API discards can be recovered from the wire. A unit test over a
//! hand-built struct could not show that: it would assert on data this crate
//! produced rather than data a server sent.
//!
//! Each test pins one response that decides whether a sync saw everything, or
//! whether a mutation actually happened.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use nuncio_core::model::PlacementKey;
use nuncio_mail::imap_raw;
use nuncio_mail::test_server::{IdentityExtensions, MockImapServer, ObjectIds, ServerProfile};
/// Connect the real client to the mock server and authenticate.
async fn login(
server: &MockImapServer,
) -> async_imap::Session<tokio::io::BufWriter<tokio::net::TcpStream>> {
let stream = tokio::net::TcpStream::connect((server.host(), server.port()))
.await
.expect("connect to the mock server");
let client = async_imap::Client::new(tokio::io::BufWriter::new(stream));
client
.login("user", "pass")
.await
.map_err(|(e, _)| e)
.expect("login succeeds")
}
#[tokio::test]
async fn qresync_select_recovers_vanished_uids_the_typed_select_would_drop() {
// The mechanism that fixes ghost messages. `async-imap`'s own `select()`
// parser has no arm for VANISHED, so this is the difference between
// learning a message was deleted and never learning it.
let server = MockImapServer::start(ServerProfile::Qresync)
.await
.expect("server starts");
server.create_mailbox("INBOX");
let keep = server.append_message("INBOX", "Subject: keep\r\n\r\nx", &[]);
let gone = server.append_message("INBOX", "Subject: gone\r\n\r\nx", &[]);
let mut session = login(&server).await;
assert!(
imap_raw::enable(&mut session, &["QRESYNC"])
.await
.expect("ENABLE succeeds"),
"the server advertises QRESYNC, so ENABLE must be accepted"
);
let initial = imap_raw::run_collected(&mut session, "SELECT INBOX")
.await
.expect("plain SELECT");
let modseq = initial
.highest_modseq
.expect("a CONDSTORE server reports HIGHESTMODSEQ");
let uid_validity = initial.uid_validity.expect("SELECT reports UIDVALIDITY");
// Another client removes a message while we are away.
assert!(server.expunge("INBOX", gone));
let resync = imap_raw::select_qresync(&mut session, "INBOX", uid_validity, modseq, None)
.await
.expect("QRESYNC SELECT succeeds");
assert!(
resync.vanished_uids.contains(&gone),
"the removal must be reported, got {:?}",
resync.vanished_uids
);
assert!(
!resync.vanished_uids.contains(&keep),
"a surviving message must not be reported as vanished"
);
}
#[tokio::test]
async fn a_conditional_store_that_loses_a_race_reports_modified() {
// RFC 7162 3.1.3. MODIFIED rides on the tagged OK, which
// `run_command_and_check_ok` discards -- so without this layer a lost
// update is indistinguishable from a successful one.
let server = MockImapServer::start(ServerProfile::Condstore)
.await
.expect("server starts");
server.create_mailbox("INBOX");
let uid = server.append_message("INBOX", "Subject: contested\r\n\r\nx", &[]);
let mut writer = login(&server).await;
let mut loser = login(&server).await;
imap_raw::run_collected(&mut writer, "SELECT INBOX")
.await
.expect("writer selects");
imap_raw::run_collected(&mut loser, "SELECT INBOX")
.await
.expect("loser selects");
// The loser captures the current mod-sequence, then the writer changes the
// message underneath it.
let observed = imap_raw::run_collected(&mut loser, &format!("UID FETCH {uid} (FLAGS)"))
.await
.expect("fetch flags");
assert_eq!(observed.fetched_uids, vec![uid]);
let modseq = imap_raw::run_collected(&mut loser, "SELECT INBOX")
.await
.expect("re-select")
.highest_modseq
.expect("HIGHESTMODSEQ reported");
imap_raw::run_collected(&mut writer, &format!("UID STORE {uid} +FLAGS (\\Seen)"))
.await
.expect("the writer wins the race");
let outcome = imap_raw::uid_store_unchanged_since(
&mut loser,
&uid.to_string(),
modseq,
"+FLAGS",
"\\Flagged",
)
.await
.expect("a refused conditional store is not a transport failure");
assert_eq!(
outcome.modified_uids(),
Some(vec![uid]),
"the conflict must name the UID; information was {:?}",
outcome.information
);
// And the refused flag genuinely did not apply.
let flags = server.flags("INBOX", uid).unwrap_or_default();
assert!(
!flags.iter().any(|f| f == "\\Flagged"),
"a MODIFIED store must leave the message untouched, got {flags:?}"
);
}
#[tokio::test]
async fn a_conditional_store_that_wins_reports_no_conflict() {
let server = MockImapServer::start(ServerProfile::Condstore)
.await
.expect("server starts");
server.create_mailbox("INBOX");
let uid = server.append_message("INBOX", "Subject: uncontested\r\n\r\nx", &[]);
let mut session = login(&server).await;
let modseq = imap_raw::run_collected(&mut session, "SELECT INBOX")
.await
.expect("select")
.highest_modseq
.expect("HIGHESTMODSEQ reported");
let outcome = imap_raw::uid_store_unchanged_since(
&mut session,
&uid.to_string(),
modseq,
"+FLAGS",
"\\Flagged",
)
.await
.expect("store succeeds");
assert_eq!(
outcome.modified_uids(),
None,
"an uncontested store must report no conflict"
);
assert!(server
.flags("INBOX", uid)
.unwrap_or_default()
.iter()
.any(|f| f == "\\Flagged"));
}
#[tokio::test]
async fn a_move_reports_copyuid_from_an_untagged_ok() {
// RFC 6851 4.3 advises COPYUID in an *untagged* OK for MOVE. A collector
// that only read the tagged response would miss it on every conforming
// server, and then have no proof the move happened.
let server = MockImapServer::start(ServerProfile::Qresync)
.await
.expect("server starts");
server.create_mailbox("INBOX");
server.create_mailbox("Archive");
let uid = server.append_message("INBOX", "Subject: moving\r\n\r\nx", &[]);
let mut session = login(&server).await;
imap_raw::run_collected(&mut session, "SELECT INBOX")
.await
.expect("select");
let outcome = imap_raw::uid_move_collected(&mut session, &uid.to_string(), "Archive")
.await
.expect("move succeeds");
let copy_uid = outcome
.copy_uid
.expect("a real move must be provable by COPYUID");
assert_eq!(copy_uid.source_uids, vec![uid]);
assert_eq!(copy_uid.destination_uids.len(), 1);
assert_eq!(server.uids("INBOX"), Vec::<u32>::new());
assert_eq!(server.uids("Archive"), copy_uid.destination_uids);
}
#[tokio::test]
async fn a_move_of_an_already_moved_uid_is_unknown_not_applied() {
// RFC 3501 6.4.8: a UID command matching nothing succeeds having done
// nothing. The command reports OK, so only the *absence* of COPYUID
// distinguishes it -- and absence means unknown, never applied.
let server = MockImapServer::start(ServerProfile::Qresync)
.await
.expect("server starts");
server.create_mailbox("INBOX");
server.create_mailbox("Archive");
server.create_mailbox("Trash");
let uid = server.append_message("INBOX", "Subject: contested\r\n\r\nx", &[]);
let mut winner = login(&server).await;
let mut loser = login(&server).await;
imap_raw::run_collected(&mut winner, "SELECT INBOX")
.await
.expect("select");
imap_raw::run_collected(&mut loser, "SELECT INBOX")
.await
.expect("select");
imap_raw::uid_move_collected(&mut winner, &uid.to_string(), "Archive")
.await
.expect("the winner moves it");
let outcome = imap_raw::uid_move_collected(&mut loser, &uid.to_string(), "Trash")
.await
.expect("the losing move still reports OK");
assert!(
outcome.ok,
"a UID set matching nothing is a successful no-op, not an error"
);
assert_eq!(
outcome.copy_uid, None,
"nothing moved, so there is no COPYUID to prove one did"
);
assert!(
server.uids("Trash").is_empty(),
"and genuinely nothing arrived at the destination"
);
}
#[tokio::test]
async fn expunge_is_provable_under_both_qresync_and_plain_servers() {
// RFC 6851 4.4: a QRESYNC server answers UID EXPUNGE with VANISHED rather
// than EXPUNGE. Counting only EXPUNGE reports failure on every successful
// delete against such a server, so both must be collected.
for profile in [ServerProfile::Qresync, ServerProfile::Condstore] {
let server = MockImapServer::start(profile).await.expect("server starts");
server.create_mailbox("INBOX");
let uid = server.append_message("INBOX", "Subject: gone\r\n\r\nx", &["\\Deleted"]);
let mut session = login(&server).await;
if profile == ServerProfile::Qresync {
imap_raw::enable(&mut session, &["QRESYNC"])
.await
.expect("ENABLE succeeds");
}
imap_raw::run_collected(&mut session, "SELECT INBOX")
.await
.expect("select");
let outcome = imap_raw::uid_expunge_collected(&mut session, &uid.to_string())
.await
.expect("expunge succeeds");
let reported = outcome.vanished_uids.len() + outcome.expunged_seqs.len();
assert_eq!(
reported, 1,
"the removal must be provable on {profile:?}: vanished={:?} expunged={:?}",
outcome.vanished_uids, outcome.expunged_seqs
);
assert!(server.uids("INBOX").is_empty());
}
}
#[tokio::test]
async fn a_qresync_select_without_enable_is_refused() {
// RFC 7162 3.2.3 requires a tagged BAD. Surfacing it as an error keeps a
// client that skipped ENABLE from silently degrading in production.
let server = MockImapServer::start(ServerProfile::Qresync)
.await
.expect("server starts");
server.create_mailbox("INBOX");
let mut session = login(&server).await;
let err = imap_raw::select_qresync(&mut session, "INBOX", 1000, 1, None)
.await
.expect_err("QRESYNC without ENABLE must fail");
assert!(
err.to_string().contains("refused"),
"the refusal must be surfaced, got: {err}"
);
}
#[tokio::test]
async fn object_ids_are_read_off_the_transport_and_the_session_survives_it() {
// The whole reason this bypasses the client: `imap-proto` has no parser arm
// for EMAILID, and `async-imap` turns a parse error into a fatal I/O error
// that latches the stream closed. So the test that matters is not only that
// the ids come back, but that the *typed* client still works afterwards --
// proving the raw exchange left the connection and the client's read buffer
// in a state the next command can use.
let server =
MockImapServer::start_with_identity(ServerProfile::Qresync, IdentityExtensions::objectid())
.await
.expect("server starts");
server.create_mailbox("INBOX");
let first = server.append_message_with_object_ids(
"INBOX",
"Subject: one\r\n\r\nx",
&[],
ObjectIds {
email_id: "M-first".to_string(),
gm_msgid: 11,
},
);
let second = server.append_message_with_object_ids(
"INBOX",
"Subject: two\r\n\r\nx",
&[],
ObjectIds {
email_id: "M-second".to_string(),
gm_msgid: 12,
},
);
let mut session = login(&server).await;
imap_raw::run_collected(&mut session, "SELECT INBOX")
.await
.expect("select");
let ids = imap_raw::uid_fetch_email_ids(&mut session, &format!("{first},{second}"))
.await
.expect("the raw EMAILID fetch succeeds");
assert_eq!(ids.get(&first).map(String::as_str), Some("M-first"));
assert_eq!(ids.get(&second).map(String::as_str), Some("M-second"));
// The session is still alive: a typed command after the raw exchange must
// still see the mailbox.
let after = imap_raw::run_collected(&mut session, "UID FETCH 1:* (UID)")
.await
.expect("the session survives the raw exchange");
assert_eq!(after.fetched_uids, vec![first, second]);
}
#[tokio::test]
async fn a_refused_object_id_fetch_costs_precision_not_the_sync() {
// A server that rejects the item leaves the message identifiable by a
// weaker tier. Erroring here would fail an entire folder sync over an
// identity *upgrade*, which is the wrong trade in both directions.
let server = MockImapServer::start(ServerProfile::Qresync)
.await
.expect("server starts");
server.create_mailbox("INBOX");
let uid = server.append_message("INBOX", "Subject: one\r\n\r\nx", &[]);
let mut session = login(&server).await;
imap_raw::run_collected(&mut session, "SELECT INBOX")
.await
.expect("select");
let ids = imap_raw::uid_fetch_email_ids(&mut session, &uid.to_string())
.await
.expect("a BAD answer is not a transport failure");
assert!(
ids.is_empty(),
"nothing was learned, and that is the answer"
);
// And the connection is still usable, which a `BAD` must not change.
let after = imap_raw::run_collected(&mut session, "UID FETCH 1:* (UID)")
.await
.expect("the session survives a refusal");
assert_eq!(after.fetched_uids, vec![uid]);
}
/// Drive a real folder sync against the mock and return what it observed.
async fn sync_folder(
server: &MockImapServer,
since: Option<&str>,
) -> nuncio_mail::backend::FolderChanges {
use nuncio_mail::MailBackend;
let engine = nuncio_mail::ImapEngine::with_credentials(
"acct-1",
server.host(),
server.port(),
nuncio_core::TlsMode::Plain,
"user",
"pass",
);
engine
.sync_changes("INBOX", since)
.await
.expect("folder sync succeeds")
}
#[tokio::test]
async fn every_rung_reports_a_message_that_disappeared() {
// The defect this whole change exists for. A forward-only sync can only
// ever add, so a message removed by another client stays local forever.
// Each rung must be able to say it is gone -- QRESYNC because the server
// states it, the other two because the pass enumerated what remains.
for profile in [
ServerProfile::Qresync,
ServerProfile::Condstore,
ServerProfile::Basic,
] {
let server = MockImapServer::start(profile).await.expect("server starts");
server.create_mailbox("INBOX");
let keep = server.append_message("INBOX", "Subject: keep\r\n\r\nkeep", &[]);
let gone = server.append_message("INBOX", "Subject: gone\r\n\r\ngone", &[]);
let first = sync_folder(&server, None).await;
assert_eq!(first.upserts.len(), 2, "{profile:?}: first pass sees both");
// Another client removes one while we are away.
assert!(server.expunge("INBOX", gone));
let second = sync_folder(&server, Some(&first.next_state)).await;
// However the rung learned it, the caller must end up able to identify
// the occupancy that went and only that one. Absence is reported per
// mailbox, so what is compared is the placement each upsert was found
// under, not the message key -- the same message may still sit
// elsewhere and must not be swept up by this folder's silence.
let mut reported: Vec<PlacementKey> = second.removals.clone();
if let Some(present) = &second.present {
let stored_gone = first
.upserts
.iter()
.map(|e| e.placement.key())
.filter(|key| !present.contains(key));
reported.extend(stored_gone);
}
reported.sort();
reported.dedup();
assert_eq!(
reported.len(),
1,
"{profile:?}: exactly one message must be reported gone, got {reported:?}"
);
let surviving = first
.upserts
.iter()
.find(|e| e.placement.remote_id == keep.to_string())
.expect("the kept message was in the first pass");
assert!(
!reported.contains(&surviving.placement.key()),
"{profile:?}: the surviving message must not be reported gone"
);
assert_eq!(
reported[0].remote_id,
gone.to_string(),
"{profile:?}: the reported occupancy must be the expunged UID"
);
}
}
#[tokio::test]
async fn an_incremental_pass_that_cannot_see_absence_says_so() {
// `present: None` versus `Some(vec![])` is the difference between "I did
// not look" and "the folder is empty". Conflating them deletes a mailbox,
// so QRESYNC -- which never enumerates -- must report None.
let server = MockImapServer::start(ServerProfile::Qresync)
.await
.expect("server starts");
server.create_mailbox("INBOX");
server.append_message("INBOX", "Subject: one\r\n\r\nx", &[]);
let first = sync_folder(&server, None).await;
let second = sync_folder(&server, Some(&first.next_state)).await;
assert!(
second.next_state.starts_with("v2:modseq:"),
"a QRESYNC-capable mailbox must checkpoint by mod-sequence, got {:?}",
second.next_state
);
assert_eq!(
second.present, None,
"a QRESYNC pass does not enumerate the folder, so it cannot claim to know what is present"
);
}
#[tokio::test]
async fn a_condstore_only_server_still_checkpoints_by_modseq_and_enumerates() {
// Gmail's rung. It cannot be told what vanished, so it must enumerate --
// and it must still upgrade its checkpoint so the next pass is narrow.
let server = MockImapServer::start(ServerProfile::Condstore)
.await
.expect("server starts");
server.create_mailbox("INBOX");
server.append_message("INBOX", "Subject: one\r\n\r\nx", &[]);
let first = sync_folder(&server, None).await;
assert!(
first.next_state.starts_with("v2:modseq:"),
"CONDSTORE alone is enough to checkpoint by mod-sequence, got {:?}",
first.next_state
);
let second = sync_folder(&server, Some(&first.next_state)).await;
assert!(
second.present.is_some(),
"without QRESYNC the only way to find removals is to enumerate"
);
assert!(
second.upserts.is_empty(),
"nothing changed, so CHANGEDSINCE must fetch no bodies"
);
}
#[tokio::test]
async fn a_server_without_modseq_falls_back_to_a_uid_boundary_checkpoint() {
// Exchange's rung: no CONDSTORE, no QRESYNC.
let server = MockImapServer::start(ServerProfile::Basic)
.await
.expect("server starts");
server.create_mailbox("INBOX");
server.append_message("INBOX", "Subject: one\r\n\r\nx", &[]);
let first = sync_folder(&server, None).await;
assert!(
first.next_state.starts_with("v1:uidnext:"),
"a mailbox with no mod-sequences must checkpoint by UID boundary, got {:?}",
first.next_state
);
assert!(first.present.is_some());
}
#[tokio::test]
async fn a_uidvalidity_change_refetches_rather_than_deleting_the_folder() {
// After a renumbering every stored UID names a different message. The
// dangerous outcome would be enumerating the new UID space, finding none
// of the old ids in it, and concluding the whole folder was deleted.
let server = MockImapServer::start(ServerProfile::Qresync)
.await
.expect("server starts");
server.create_mailbox("INBOX");
server.append_message("INBOX", "Subject: before\r\n\r\nx", &[]);
let first = sync_folder(&server, None).await;
assert_eq!(first.upserts.len(), 1);
server.bump_uid_validity("INBOX");
server.append_message("INBOX", "Subject: after\r\n\r\nx", &[]);
let second = sync_folder(&server, Some(&first.next_state)).await;
assert_eq!(
second.upserts.len(),
1,
"a renumbered mailbox must be re-fetched in full"
);
assert!(
second.removals.is_empty(),
"a renumbering is not a removal report"
);
// The re-fetched message lands under a new occupancy, because UIDVALIDITY
// is part of a placement key -- so the caller writes a new placement rather
// than updating the old one, and the stale UID never addresses live mail.
assert_ne!(
second.upserts[0].placement.uid_validity,
first.upserts[0].placement.uid_validity
);
assert_ne!(second.upserts[0].placement, first.upserts[0].placement);
// These two are also genuinely different messages, so their derived keys
// differ as well.
assert_ne!(second.upserts[0].email.id, first.upserts[0].email.id);
}
/// Build an engine pointed at the mock and apply one mutation through the
/// real backend path.
async fn apply(
server: &MockImapServer,
kind: nuncio_mail::RemoteMutationKind,
uid: u32,
uid_validity: &str,
) -> nuncio_mail::MutationOutcome {
use nuncio_mail::MailBackend;
let engine = nuncio_mail::ImapEngine::with_credentials(
"acct-1",
server.host(),
server.port(),
nuncio_core::TlsMode::Plain,
"user",
"pass",
);
engine
.apply_mutation(&nuncio_mail::RemoteMutationSpec {
message_id: "m-1".to_string(),
remote_id: uid.to_string(),
folder_id: "INBOX".to_string(),
uid_validity: uid_validity.to_string(),
kind,
})
.await
.expect("the mutation attempt itself succeeds")
}
/// UIDVALIDITY the mock assigns to the first mailbox created.
const FIRST_UID_VALIDITY: &str = "1000";
#[tokio::test]
async fn a_move_that_lost_a_race_is_unknown_rather_than_applied() {
// The silent-loss case. The server answers OK for a UID that no longer
// exists (RFC 3501 6.4.8), so only the absence of COPYUID distinguishes a
// real move from a no-op -- and absence cannot mean "applied".
let server = MockImapServer::start(ServerProfile::Qresync)
.await
.expect("server starts");
server.create_mailbox("INBOX");
server.create_mailbox("Archive");
server.create_mailbox("Trash");
let uid = server.append_message("INBOX", "Subject: contested\r\n\r\nx", &[]);
let applied = apply(
&server,
nuncio_mail::RemoteMutationKind::Move {
to_folder: "Archive".to_string(),
},
uid,
FIRST_UID_VALIDITY,
)
.await;
assert!(
matches!(
applied,
nuncio_mail::MutationOutcome::Applied { token: Some(_) }
),
"a real move must be Applied and name its destination UID, got {applied:?}"
);
let lost = apply(
&server,
nuncio_mail::RemoteMutationKind::Move {
to_folder: "Trash".to_string(),
},
uid,
FIRST_UID_VALIDITY,
)
.await;
assert!(
matches!(lost, nuncio_mail::MutationOutcome::Unknown { .. }),
"a move of an already-moved UID must be Unknown, never Applied, got {lost:?}"
);
assert!(
server.uids("Trash").is_empty(),
"and nothing actually reached the destination"
);
}
#[tokio::test]
async fn a_delete_is_applied_only_when_the_server_reports_the_removal() {
let server = MockImapServer::start(ServerProfile::Qresync)
.await
.expect("server starts");
server.create_mailbox("INBOX");
let uid = server.append_message("INBOX", "Subject: gone\r\n\r\nx", &[]);
let outcome = apply(
&server,
nuncio_mail::RemoteMutationKind::Delete,
uid,
FIRST_UID_VALIDITY,
)
.await;
// Under QRESYNC the proof arrives as VANISHED, not EXPUNGE. Reading only
// the typed expunge stream would report nothing removed here.
assert!(
matches!(outcome, nuncio_mail::MutationOutcome::Applied { .. }),
"a delete the server confirmed must be Applied, got {outcome:?}"
);
assert!(server.uids("INBOX").is_empty());
}
#[tokio::test]
async fn a_delete_of_an_absent_uid_is_unknown() {
let server = MockImapServer::start(ServerProfile::Qresync)
.await
.expect("server starts");
server.create_mailbox("INBOX");
server.append_message("INBOX", "Subject: present\r\n\r\nx", &[]);
// UID 999 does not exist; the server ignores it and answers OK.
let outcome = apply(
&server,
nuncio_mail::RemoteMutationKind::Delete,
999,
FIRST_UID_VALIDITY,
)
.await;
assert!(
matches!(outcome, nuncio_mail::MutationOutcome::Unknown { .. }),
"a delete the server never confirmed must not be Applied, got {outcome:?}"
);
assert_eq!(
server.uids("INBOX").len(),
1,
"the real message must be untouched"
);
}
crates/nuncio-mail/tests/wiremock_jmap_test.rs
+555 -13
View File
@@ -2,9 +2,10 @@
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use nuncio_mail::{JmapEngine, MailBackend};
use nuncio_core::model::{IdentitySource, PlacementKey};
use nuncio_mail::{JmapEngine, MailBackend, RemoteMutationKind, RemoteMutationSpec};
use serde_json::json;
use wiremock::matchers::{method, path};
use wiremock::matchers::{body_partial_json, method, path};
use wiremock::{Mock, MockServer, ResponseTemplate};
#[tokio::test]
@@ -80,13 +81,20 @@ async fn wiremock_jmap_session_discovery_and_email_get_sync() {
let engine = JmapEngine::new("acct-100");
let (emails, new_state) = engine
.parse_email_get_response(&resp.text().await.unwrap())
.parse_email_get_response(&resp.text().await.unwrap(), "mb-inbox")
.expect("parse email get");
assert_eq!(emails.len(), 1);
assert_eq!(emails[0].id, "msg-wm-100");
assert_eq!(emails[0].subject, "WireMock JMAP E2E Test");
assert_eq!(emails[0].sender, "sender@nuncio.mx");
// The JMAP object id is preserved as the placement's protocol-native
// `remote_id`, while the message's own id is an opaque 64-char derived key
// (never the raw id). The object id is account-stable, so it enters the
// precedence at the EMAILID tier rather than as a folder-scoped surrogate.
assert_eq!(emails[0].placement.remote_id, "msg-wm-100");
assert_eq!(emails[0].source, IdentitySource::EmailId);
assert_ne!(emails[0].email.id, "msg-wm-100");
assert_eq!(emails[0].email.id.len(), 64);
assert_eq!(emails[0].email.subject, "WireMock JMAP E2E Test");
assert_eq!(emails[0].email.sender, "sender@nuncio.mx");
assert_eq!(new_state, "sync-state-500");
}
@@ -200,16 +208,26 @@ async fn jmap_engine_sync_folders_and_messages_through_mail_backend_trait() {
.await
.expect("sync_messages succeeds");
assert_eq!(emails.len(), 1);
assert_eq!(emails[0].id, "msg-wm-200");
assert_eq!(emails[0].subject, "Trait-Level JMAP Sync");
assert_eq!(emails[0].sender, "sender@nuncio.mx");
assert_eq!(emails[0].received_at, 1700002000);
assert!(!emails[0].read);
assert_eq!(emails[0].placement.remote_id, "msg-wm-200");
// The occupancy is labelled with the mailbox that was actually synced.
// Removals are addressed by the same key, so a placement stored under any
// other folder id could never be reconciled away again.
assert_eq!(emails[0].placement.folder_id, "mb-inbox");
assert_eq!(emails[0].source, IdentitySource::EmailId);
assert_ne!(emails[0].email.id, "msg-wm-200");
assert_eq!(emails[0].email.id.len(), 64);
assert_eq!(emails[0].email.subject, "Trait-Level JMAP Sync");
assert_eq!(emails[0].email.sender, "sender@nuncio.mx");
assert_eq!(emails[0].email.received_at, 1700002000);
// Read state is a property of the occupancy, not of the message.
assert!(!emails[0].placement.read);
assert_eq!(
emails[0].body_plain,
emails[0].email.body_plain,
Some("Fetched through the MailBackend trait.".to_string())
);
assert_eq!(new_state, "sync-state-900");
// The checkpoint is scheme-tagged, so a later build can tell what the
// opaque server token inside it actually means.
assert_eq!(new_state, "v1:jmapstate:sync-state-900");
}
/// A session-discovery failure (non-2xx status) must surface as a genuine `MailError`, never a
@@ -237,3 +255,527 @@ async fn jmap_engine_sync_folders_surfaces_session_discovery_failure() {
.expect_err("session discovery failure must not fall back to canned folders");
assert!(err.to_string().contains("network I/O error"));
}
/// Mount JMAP session discovery so `apply_mutation` can reach the mock `/jmap/api`.
async fn mount_session(mock_server: &MockServer) {
let session_body = json!({
"username": "james.maes@kof22.com",
"primaryAccounts": { "urn:ietf:params:jmap:mail": "acct-100" },
"apiUrl": format!("{}/jmap/api", mock_server.uri()),
"state": "session-state-1"
});
Mock::given(method("GET"))
.and(path("/.well-known/jmap"))
.respond_with(ResponseTemplate::new(200).set_body_json(&session_body))
.mount(mock_server)
.await;
}
fn jmap_engine(mock_server: &MockServer) -> JmapEngine {
JmapEngine::with_credentials(
"acct-100",
&mock_server.uri(),
"james.maes@kof22.com",
"wiremock-password",
)
}
fn spec(kind: RemoteMutationKind) -> RemoteMutationSpec {
RemoteMutationSpec {
message_id: "surrogate-jmap-msg-1".to_string(),
remote_id: "jmap-msg-1".to_string(),
folder_id: "mb-inbox".to_string(),
uid_validity: "jmap".to_string(),
kind,
}
}
#[tokio::test]
async fn jmap_apply_mutation_flag_issues_keywords_patch_and_confirms_success() {
let mock_server = MockServer::start().await;
mount_session(&mock_server).await;
// Assert the exact Email/set update patch reaches the server: a $flagged
// keyword set to true on the target id.
Mock::given(method("POST"))
.and(path("/jmap/api"))
.and(body_partial_json(json!({
"methodCalls": [[
"Email/set",
{ "update": { "jmap-msg-1": { "keywords/$flagged": true } } },
"c1"
]]
})))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"methodResponses": [["Email/set", {"updated": {"jmap-msg-1": null}}, "c1"]]
})))
.mount(&mock_server)
.await;
let engine = jmap_engine(&mock_server);
engine
.apply_mutation(&spec(RemoteMutationKind::SetFlagged { value: true }))
.await
.expect("flag mutation applied");
}
#[tokio::test]
async fn jmap_apply_mutation_move_replaces_mailbox_ids() {
let mock_server = MockServer::start().await;
mount_session(&mock_server).await;
Mock::given(method("POST"))
.and(path("/jmap/api"))
.and(body_partial_json(json!({
"methodCalls": [[
"Email/set",
{ "update": { "jmap-msg-1": { "mailboxIds": { "mb-archive": true } } } },
"c1"
]]
})))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"methodResponses": [["Email/set", {"updated": {"jmap-msg-1": null}}, "c1"]]
})))
.mount(&mock_server)
.await;
let engine = jmap_engine(&mock_server);
engine
.apply_mutation(&spec(RemoteMutationKind::Move {
to_folder: "mb-archive".to_string(),
}))
.await
.expect("move mutation applied");
}
#[tokio::test]
async fn jmap_apply_mutation_delete_issues_destroy_and_confirms_success() {
let mock_server = MockServer::start().await;
mount_session(&mock_server).await;
Mock::given(method("POST"))
.and(path("/jmap/api"))
.and(body_partial_json(json!({
"methodCalls": [["Email/set", { "destroy": ["jmap-msg-1"] }, "c1"]]
})))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"methodResponses": [["Email/set", {"destroyed": ["jmap-msg-1"]}, "c1"]]
})))
.mount(&mock_server)
.await;
let engine = jmap_engine(&mock_server);
engine
.apply_mutation(&spec(RemoteMutationKind::Delete))
.await
.expect("delete mutation applied");
}
/// A server that reports the id under `notUpdated` genuinely rejected the
/// mutation; `apply_mutation` MUST surface that as an error so the outbox never
/// marks a rejected op completed.
#[tokio::test]
async fn jmap_apply_mutation_surfaces_server_rejection_as_error() {
let mock_server = MockServer::start().await;
mount_session(&mock_server).await;
Mock::given(method("POST"))
.and(path("/jmap/api"))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"methodResponses": [["Email/set", {
"updated": {},
"notUpdated": { "jmap-msg-1": { "type": "notFound" } }
}, "c1"]]
})))
.mount(&mock_server)
.await;
let engine = jmap_engine(&mock_server);
let err = engine
.apply_mutation(&spec(RemoteMutationKind::SetFlagged { value: false }))
.await
.expect_err("a server-rejected mutation must not be reported as success");
assert!(err.to_string().contains("jmap-msg-1"));
}
// --- Email/changes differential sync (RFC 8621 s4.2 over RFC 8620 s5.2) ---
/// Mount one `Email/changes` page, matched on the exact `sinceState` the engine
/// must send. A pass that resumes from the wrong token matches no mock and the
/// request fails, so the matcher is itself the round-trip assertion.
async fn mount_email_changes_page(
mock_server: &MockServer,
since_state: &str,
body: serde_json::Value,
) {
Mock::given(method("POST"))
.and(path("/jmap/api"))
.and(body_partial_json(json!({
"methodCalls": [["Email/changes", { "sinceState": since_state }, "c1"]]
})))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"methodResponses": [["Email/changes", body, "c1"]]
})))
.mount(mock_server)
.await;
}
/// Mount an `Email/get` answering with `list` verbatim and the given state.
async fn mount_email_get(mock_server: &MockServer, state: &str, list: serde_json::Value) {
Mock::given(method("POST"))
.and(path("/jmap/api"))
.and(body_partial_json(json!({
"methodCalls": [["Email/get", {}, "c1"]]
})))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"methodResponses": [["Email/get", {"state": state, "list": list}, "c1"]]
})))
.mount(mock_server)
.await;
}
fn message(id: &str, mailbox_id: &str) -> serde_json::Value {
json!({
"id": id,
"subject": format!("Message {id}"),
"from": [{ "email": "sender@nuncio.mx" }],
"to": [{ "email": "james.maes@kof22.com" }],
"receivedAt": 1700003000i64,
"isUnread": true,
"mailboxIds": { mailbox_id: true }
})
}
fn placement_key(folder_id: &str, remote_id: &str) -> PlacementKey {
PlacementKey {
account_id: "acct-100".to_string(),
folder_id: folder_id.to_string(),
uid_validity: "jmap".to_string(),
remote_id: remote_id.to_string(),
}
}
/// The whole point of the differential pass: ids the server reports as
/// `destroyed` come back as placement removals, addressed exactly as the
/// occupancies this engine stores are -- so the sync path deletes those
/// placements and nothing else. An incremental pass still reports
/// `present: None`, because it never enumerated the folder.
#[tokio::test]
async fn jmap_sync_changes_turns_destroyed_ids_into_placement_removals() {
let mock_server = MockServer::start().await;
mount_session(&mock_server).await;
mount_email_changes_page(
&mock_server,
"srv-state-1",
json!({
"oldState": "srv-state-1",
"newState": "srv-state-2",
"created": ["m-new"],
"updated": [],
"destroyed": ["m-gone-1", "m-gone-2"],
"hasMoreChanges": false
}),
)
.await;
mount_email_get(
&mock_server,
"srv-state-2",
json!([message("m-new", "mb-inbox")]),
)
.await;
let engine = jmap_engine(&mock_server);
let changes = engine
.sync_changes("mb-inbox", Some("v1:jmapstate:srv-state-1"))
.await
.expect("incremental sync succeeds");
assert_eq!(changes.upserts.len(), 1);
assert_eq!(changes.upserts[0].placement.remote_id, "m-new");
assert_eq!(changes.upserts[0].placement.folder_id, "mb-inbox");
assert_eq!(changes.upserts[0].source, IdentitySource::EmailId);
// Exactly the destroyed ids, keyed the same way the stored placement is.
assert_eq!(
changes.removals,
vec![
placement_key("mb-inbox", "m-gone-1"),
placement_key("mb-inbox", "m-gone-2"),
]
);
// A message the pass placed must never also be reconciled away.
assert!(!changes.removals.iter().any(|key| key.remote_id == "m-new"));
// An incremental pass cannot speak to absence beyond what it was told.
assert!(
changes.present.is_none(),
"an incremental pass must not claim to have enumerated the folder"
);
assert_eq!(changes.next_state, "v1:jmapstate:srv-state-2");
}
/// `hasMoreChanges: true` means the server truncated the page. Stopping there
/// silently drops every change past the first page, so the pass must follow the
/// intermediate `newState` until the server stops paging, and accumulate.
#[tokio::test]
async fn jmap_sync_changes_follows_has_more_changes_to_convergence() {
let mock_server = MockServer::start().await;
mount_session(&mock_server).await;
mount_email_changes_page(
&mock_server,
"page-1",
json!({
"oldState": "page-1",
"newState": "page-2",
"created": ["m-page1"],
"destroyed": ["m-gone-page1"],
"hasMoreChanges": true
}),
)
.await;
mount_email_changes_page(
&mock_server,
"page-2",
json!({
"oldState": "page-2",
"newState": "page-3",
"updated": ["m-page2"],
"destroyed": ["m-gone-page2"],
"hasMoreChanges": false
}),
)
.await;
mount_email_get(
&mock_server,
"page-3",
json!([
message("m-page1", "mb-inbox"),
message("m-page2", "mb-inbox")
]),
)
.await;
let engine = jmap_engine(&mock_server);
let changes = engine
.sync_changes("mb-inbox", Some("v1:jmapstate:page-1"))
.await
.expect("paged incremental sync succeeds");
let upserted: Vec<&str> = changes
.upserts
.iter()
.map(|m| m.placement.remote_id.as_str())
.collect();
assert_eq!(
upserted,
vec!["m-page1", "m-page2"],
"both pages' changed ids must be fetched, not just the first"
);
assert_eq!(
changes.removals,
vec![
placement_key("mb-inbox", "m-gone-page1"),
placement_key("mb-inbox", "m-gone-page2"),
],
"removals must accumulate across pages"
);
// The last page's state is the resume point; an intermediate one would
// replay the tail of the change log forever.
assert_eq!(changes.next_state, "v1:jmapstate:page-3");
}
/// `cannotCalculateChanges` (RFC 8620 s5.2) is the server declining to compute
/// a delta. The pass must re-enumerate the folder rather than fail -- and only
/// then may it report `present`, because only then has it genuinely seen
/// everything the folder holds.
#[tokio::test]
async fn jmap_sync_changes_falls_back_to_full_enumeration_and_only_then_reports_presence() {
let mock_server = MockServer::start().await;
mount_session(&mock_server).await;
Mock::given(method("POST"))
.and(path("/jmap/api"))
.and(body_partial_json(json!({
"methodCalls": [["Email/changes", {}, "c1"]]
})))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"methodResponses": [["error", {"type": "cannotCalculateChanges"}, "c1"]]
})))
.mount(&mock_server)
.await;
Mock::given(method("POST"))
.and(path("/jmap/api"))
.and(body_partial_json(json!({
"methodCalls": [["Email/query", {"filter": {"inMailboxes": ["mb-inbox"]}}, "c1"]]
})))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"methodResponses": [["Email/query", {"ids": ["m-a", "m-b"]}, "c1"]]
})))
.mount(&mock_server)
.await;
mount_email_get(
&mock_server,
"srv-full-9",
json!([message("m-a", "mb-inbox"), message("m-b", "mb-inbox")]),
)
.await;
let engine = jmap_engine(&mock_server);
let changes = engine
.sync_changes("mb-inbox", Some("v1:jmapstate:far-too-old"))
.await
.expect("a stale state must re-enumerate, not fail the sync");
assert_eq!(changes.upserts.len(), 2);
assert_eq!(
changes.present,
Some(vec![
placement_key("mb-inbox", "m-a"),
placement_key("mb-inbox", "m-b"),
]),
"a genuine full enumeration is the one pass entitled to report presence"
);
assert!(
changes.removals.is_empty(),
"a full pass expresses absence through `present`, not through removals"
);
assert_eq!(changes.next_state, "v1:jmapstate:srv-full-9");
}
/// The `newState` a pass returns is the token the next pass must resume from.
/// The second pass's `Email/changes` mock only matches that exact state, so a
/// token that failed to round-trip through `next_state` would match nothing.
#[tokio::test]
async fn jmap_sync_changes_resumes_from_the_state_the_previous_pass_returned() {
let mock_server = MockServer::start().await;
mount_session(&mock_server).await;
mount_email_changes_page(
&mock_server,
"round-1",
json!({
"oldState": "round-1",
"newState": "round-2",
"destroyed": ["m-gone"],
"hasMoreChanges": false
}),
)
.await;
mount_email_changes_page(
&mock_server,
"round-2",
json!({
"oldState": "round-2",
"newState": "round-3",
"destroyed": ["m-gone-later"],
"hasMoreChanges": false
}),
)
.await;
let engine = jmap_engine(&mock_server);
let first = engine
.sync_changes("mb-inbox", Some("v1:jmapstate:round-1"))
.await
.expect("first pass succeeds");
assert_eq!(first.next_state, "v1:jmapstate:round-2");
let second = engine
.sync_changes("mb-inbox", Some(&first.next_state))
.await
.expect("the returned checkpoint must be accepted verbatim on the next pass");
assert_eq!(
second.removals,
vec![placement_key("mb-inbox", "m-gone-later")]
);
assert_eq!(second.next_state, "v1:jmapstate:round-3");
}
/// A checkpoint carrying no scheme this build recognises -- a bare token
/// written before tagging existed -- must never be handed to the server as a
/// `sinceState`. It resolves to a full enumeration instead: over-fetching is
/// recoverable, resuming from a token the server reads differently is not.
#[tokio::test]
async fn jmap_sync_changes_re_enumerates_when_the_stored_checkpoint_is_untagged() {
let mock_server = MockServer::start().await;
mount_session(&mock_server).await;
// No `Email/changes` mock is mounted on purpose: such a call would 404 and
// fail the sync, which is what proves none was attempted.
Mock::given(method("POST"))
.and(path("/jmap/api"))
.and(body_partial_json(json!({
"methodCalls": [["Email/query", {}, "c1"]]
})))
.respond_with(ResponseTemplate::new(200).set_body_json(json!({
"methodResponses": [["Email/query", {"ids": ["m-a"]}, "c1"]]
})))
.mount(&mock_server)
.await;
mount_email_get(
&mock_server,
"srv-full-1",
json!([message("m-a", "mb-inbox")]),
)
.await;
let engine = jmap_engine(&mock_server);
let changes = engine
.sync_changes("mb-inbox", Some("sync-state-900"))
.await
.expect("an unrecognised checkpoint must re-enumerate");
assert_eq!(
changes.present,
Some(vec![placement_key("mb-inbox", "m-a")])
);
assert_eq!(changes.next_state, "v1:jmapstate:srv-full-1");
}
/// `Email/changes` is account-wide (RFC 8621 s4.2 has no mailbox filter), so a
/// folder pass sees ids that now live in other mailboxes. A message another
/// client moved out has left this folder: its occupancy here is reconciled
/// away, and it is NOT placed here on the strength of having merely changed.
#[tokio::test]
async fn jmap_sync_changes_reconciles_a_message_moved_out_of_the_folder() {
let mock_server = MockServer::start().await;
mount_session(&mock_server).await;
mount_email_changes_page(
&mock_server,
"srv-state-1",
json!({
"oldState": "srv-state-1",
"newState": "srv-state-2",
"updated": ["m-moved", "m-stayed"],
"hasMoreChanges": false
}),
)
.await;
mount_email_get(
&mock_server,
"srv-state-2",
json!([
message("m-moved", "mb-archive"),
message("m-stayed", "mb-inbox")
]),
)
.await;
let engine = jmap_engine(&mock_server);
let changes = engine
.sync_changes("mb-inbox", Some("v1:jmapstate:srv-state-1"))
.await
.expect("incremental sync succeeds");
let upserted: Vec<&str> = changes
.upserts
.iter()
.map(|m| m.placement.remote_id.as_str())
.collect();
assert_eq!(upserted, vec!["m-stayed"]);
assert_eq!(
changes.removals,
vec![placement_key("mb-inbox", "m-moved")],
"a message whose mailboxes no longer include this folder must lose its occupancy here"
);
}
crates/nuncio-proto/Cargo.toml
+2
View File
@@ -13,6 +13,8 @@ workspace = true
tonic.workspace = true
tonic-prost.workspace = true
prost.workspace = true
prost-types.workspace = true
bytes.workspace = true
thiserror.workspace = true
[build-dependencies]
crates/nuncio-proto/proto/nuncio/v1/descriptor.bin
BIN
View File
Binary file not shown.
crates/nuncio-proto/proto/nuncio/v1/nuncio.proto
+603 -104
View File
File diff suppressed because it is too large. Load diff
crates/nuncio-proto/src/errors.rs
+158
View File
@@ -0,0 +1,158 @@
//! Typed, machine-readable gRPC errors for the `nuncio.v1` contract.
//!
//! A failed RPC should carry a structured [`v1::ErrorInfo`] -- a stable
//! [`v1::ErrorReason`] discriminant, optional metadata, and a human message
//! -- in the gRPC `grpc-status-details-bin` status-details trailer, alongside
//! the canonical gRPC [`Code`] that the reason maps to. Servers build such a
//! `Status` with [`status`] / [`status_with_metadata`]; clients read the
//! reason back with [`error_reason`] / [`error_info`] and branch on it,
//! rather than string-matching the human-readable (English) message text.
//!
//! The reason-to-code mapping is centralized in [`ErrorReason::code`] so the
//! transport-level status code and the application-level reason can never
//! drift apart at a call site.
use crate::v1::{ErrorInfo, ErrorReason};
use bytes::Bytes;
use prost::Message;
use tonic::{Code, Status};
impl ErrorReason {
/// The canonical gRPC [`Code`] this reason maps to. Both the unspecified
/// zero value and an explicit internal error map to [`Code::Internal`],
/// so an older client that receives a reason it does not recognize (and
/// decodes it as `Unspecified`) still sees an internal-class failure.
#[must_use]
pub fn code(self) -> Code {
match self {
ErrorReason::Unspecified | ErrorReason::Internal => Code::Internal,
ErrorReason::ValidationFailed => Code::InvalidArgument,
ErrorReason::AccountNotFound
| ErrorReason::RuleNotFound
| ErrorReason::MessageNotFound
| ErrorReason::EventNotFound
| ErrorReason::ContactNotFound => Code::NotFound,
ErrorReason::AlreadyExists => Code::AlreadyExists,
ErrorReason::NotConfigured => Code::FailedPrecondition,
ErrorReason::BackendUnreachable => Code::Unavailable,
ErrorReason::AuthRequired => Code::Unauthenticated,
ErrorReason::Unsupported => Code::Unimplemented,
// `Aborted` is the canonical code for a concurrency conflict, and
// unlike `FailedPrecondition` it tells a client the call may be
// retried only after re-reading the current state -- which is
// exactly the remedy here.
ErrorReason::Conflict => Code::Aborted,
}
}
}
/// Builds a [`Status`] for `reason` carrying `message` both as the `Status`
/// message and inside a prost-encoded [`v1::ErrorInfo`] in the
/// `grpc-status-details-bin` trailer, with the canonical code from
/// [`ErrorReason::code`].
pub fn status(reason: ErrorReason, message: impl Into<String>) -> Status {
build(reason, message.into(), Vec::new())
}
/// Like [`status`], but also attaches structured `metadata` (e.g. the id of
/// the resource that was not found) to the encoded [`v1::ErrorInfo`].
pub fn status_with_metadata(
reason: ErrorReason,
message: impl Into<String>,
metadata: impl IntoIterator<Item = (String, String)>,
) -> Status {
build(reason, message.into(), metadata.into_iter().collect())
}
fn build(reason: ErrorReason, message: String, metadata: Vec<(String, String)>) -> Status {
let info = ErrorInfo {
reason: reason as i32,
metadata: metadata.into_iter().collect(),
message: message.clone(),
};
Status::with_details(reason.code(), message, Bytes::from(info.encode_to_vec()))
}
/// Decodes the [`v1::ErrorInfo`] carried in `status`'s details trailer, if
/// one is present and well-formed. Returns `None` when the status carries no
/// details (e.g. an error not built through this module) or the details are
/// not a valid `ErrorInfo`.
#[must_use]
pub fn error_info(status: &Status) -> Option<ErrorInfo> {
let details = status.details();
if details.is_empty() {
return None;
}
ErrorInfo::decode(details).ok()
}
/// Decodes just the [`v1::ErrorReason`] from `status`'s details trailer,
/// returning `None` when no typed [`v1::ErrorInfo`] is present.
#[must_use]
pub fn error_reason(status: &Status) -> Option<ErrorReason> {
error_info(status).map(|info| info.reason())
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn each_reason_maps_to_its_canonical_code() {
assert_eq!(ErrorReason::Unspecified.code(), Code::Internal);
assert_eq!(ErrorReason::ValidationFailed.code(), Code::InvalidArgument);
assert_eq!(ErrorReason::AccountNotFound.code(), Code::NotFound);
assert_eq!(ErrorReason::RuleNotFound.code(), Code::NotFound);
assert_eq!(ErrorReason::MessageNotFound.code(), Code::NotFound);
assert_eq!(ErrorReason::EventNotFound.code(), Code::NotFound);
assert_eq!(ErrorReason::ContactNotFound.code(), Code::NotFound);
assert_eq!(ErrorReason::AlreadyExists.code(), Code::AlreadyExists);
assert_eq!(ErrorReason::NotConfigured.code(), Code::FailedPrecondition);
assert_eq!(ErrorReason::BackendUnreachable.code(), Code::Unavailable);
assert_eq!(ErrorReason::AuthRequired.code(), Code::Unauthenticated);
assert_eq!(ErrorReason::Unsupported.code(), Code::Unimplemented);
assert_eq!(ErrorReason::Internal.code(), Code::Internal);
}
#[test]
fn status_carries_reason_code_and_decodable_details() {
let status = status(ErrorReason::MessageNotFound, "message 'm-1' not found");
assert_eq!(status.code(), Code::NotFound);
assert_eq!(status.message(), "message 'm-1' not found");
let info = error_info(&status).expect("typed details present");
assert_eq!(info.reason(), ErrorReason::MessageNotFound);
assert_eq!(info.message, "message 'm-1' not found");
assert!(info.metadata.is_empty());
assert_eq!(
error_reason(&status),
Some(ErrorReason::MessageNotFound),
"the reason round-trips through the encoded details"
);
}
#[test]
fn status_with_metadata_round_trips_context() {
let status = status_with_metadata(
ErrorReason::AccountNotFound,
"account 'acct-1' not found",
[("account_id".to_string(), "acct-1".to_string())],
);
assert_eq!(status.code(), Code::NotFound);
let info = error_info(&status).expect("typed details present");
assert_eq!(info.reason(), ErrorReason::AccountNotFound);
assert_eq!(
info.metadata.get("account_id").map(String::as_str),
Some("acct-1")
);
}
#[test]
fn plain_status_without_details_decodes_to_none() {
let status = Status::not_found("no typed details here");
assert_eq!(error_info(&status), None);
assert_eq!(error_reason(&status), None);
}
}
crates/nuncio-proto/src/lib.rs
+15
View File
@@ -22,6 +22,8 @@
pub mod addr;
pub mod client;
pub mod errors;
pub mod time;
pub use addr::{grpc_addr_from_env, DEFAULT_GRPC_ADDR, GRPC_ADDR_ENV_VAR};
@@ -54,10 +56,23 @@ mod tests {
let response = GetStatusResponse {
engine_status: "ok".to_string(),
version: "0.1.0".to_string(),
uptime: Some(super::time::duration_from_secs(42)),
accounts_loaded: 2,
unread_count: 7,
last_error: None,
outbox_depth: 1,
account_sync_states: Vec::new(),
ready: true,
db_healthy: true,
};
assert_eq!(response.engine_status, "ok");
assert_eq!(response.version, "0.1.0");
assert_eq!(response.accounts_loaded, 2);
assert_eq!(response.unread_count, 7);
assert_eq!(response.outbox_depth, 1);
assert!(response.ready);
assert!(response.db_healthy);
}
#[test]
crates/nuncio-proto/src/time.rs
+149
View File
@@ -0,0 +1,149 @@
//! Conversions between the store's native integer instants/durations and the
//! well-known `google.protobuf.Timestamp`/`google.protobuf.Duration` wire
//! types used at the `nuncio.v1` proto boundary.
//!
//! The daemon's core/store models keep unix-seconds `i64`/`u64` instants and
//! plain scalar durations; these helpers are the single place that maps them
//! to/from the well-known types so every RPC mapper does it the same way.
//! Re-exporting `Timestamp`/`Duration` here lets callers name the wire types
//! without depending on `prost-types` directly.
pub use prost_types::{Duration, Timestamp};
/// Converts a unix-seconds instant into a `Timestamp` with a zero `nanos`
/// component. Use this for store fields that only ever carry second
/// granularity (e.g. message/event/rule timestamps).
#[must_use]
pub fn timestamp_from_unix_secs(unix_secs: i64) -> Timestamp {
Timestamp {
seconds: unix_secs,
nanos: 0,
}
}
/// Extracts the unix-seconds instant from a `Timestamp`, truncating any
/// sub-second `nanos` component. Use this for fields mapped via
/// [`timestamp_from_unix_secs`], where `nanos` is always zero.
#[must_use]
pub fn timestamp_to_unix_secs(ts: &Timestamp) -> i64 {
ts.seconds
}
/// Converts a unix-nanoseconds instant into a `Timestamp`, splitting it
/// losslessly into whole seconds plus a sub-second `nanos` remainder. Use
/// this for fields that carry sub-second precision (e.g. the WORM audit
/// ledger's `timestamp_ns`).
#[must_use]
pub fn timestamp_from_unix_nanos(unix_nanos: i64) -> Timestamp {
let seconds = unix_nanos.div_euclid(1_000_000_000);
let nanos = unix_nanos.rem_euclid(1_000_000_000);
Timestamp {
seconds,
nanos: nanos as i32,
}
}
/// Recovers the exact unix-nanoseconds instant a `Timestamp` was built from
/// via [`timestamp_from_unix_nanos`].
#[must_use]
pub fn timestamp_to_unix_nanos(ts: &Timestamp) -> i64 {
ts.seconds
.saturating_mul(1_000_000_000)
.saturating_add(i64::from(ts.nanos))
}
/// Converts a whole-second duration into a `Duration` with a zero `nanos`
/// component (e.g. `AccountConfig`'s sync polling interval).
#[must_use]
pub fn duration_from_secs(secs: u64) -> Duration {
Duration {
seconds: secs as i64,
nanos: 0,
}
}
/// Converts a `std::time::Duration` (e.g. a process-uptime measurement taken
/// with `Instant::elapsed`) into a wire `Duration`, preserving sub-second
/// precision. Saturates rather than overflowing on an implausibly large span.
#[must_use]
pub fn duration_from_std(d: std::time::Duration) -> Duration {
Duration {
seconds: i64::try_from(d.as_secs()).unwrap_or(i64::MAX),
nanos: d.subsec_nanos() as i32,
}
}
/// Extracts a whole-second duration from a `Duration`, truncating any
/// sub-second `nanos` component. A negative `seconds` (never produced by
/// this crate's own mappers) reads back as zero rather than panicking.
#[must_use]
pub fn duration_to_secs(d: &Duration) -> u64 {
u64::try_from(d.seconds).unwrap_or(0)
}
/// Converts a microsecond duration into a `Duration`, splitting it
/// losslessly into whole seconds plus a sub-second `nanos` remainder (e.g.
/// filter rule preview evaluation timing).
#[must_use]
pub fn duration_from_micros(micros: u64) -> Duration {
let seconds = micros / 1_000_000;
let nanos = (micros % 1_000_000) * 1_000;
Duration {
seconds: seconds as i64,
nanos: nanos as i32,
}
}
/// Recovers the exact microsecond duration a `Duration` was built from via
/// [`duration_from_micros`].
#[must_use]
pub fn duration_to_micros(d: &Duration) -> u64 {
let seconds = u64::try_from(d.seconds).unwrap_or(0);
let nanos = u64::try_from(d.nanos).unwrap_or(0);
seconds
.saturating_mul(1_000_000)
.saturating_add(nanos / 1_000)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn seconds_round_trip_through_timestamp() {
let ts = timestamp_from_unix_secs(1_700_000_000);
assert_eq!(ts.nanos, 0);
assert_eq!(timestamp_to_unix_secs(&ts), 1_700_000_000);
}
#[test]
fn nanos_round_trip_through_timestamp() {
let original_ns = 1_700_000_000_123_456_789_i64;
let ts = timestamp_from_unix_nanos(original_ns);
assert_eq!(timestamp_to_unix_nanos(&ts), original_ns);
}
#[test]
fn negative_epoch_nanos_round_trip() {
// Regression: pre-epoch instants must still split into a
// non-negative `nanos` remainder per the Timestamp wire contract.
let original_ns = -1_500_000_000_i64;
let ts = timestamp_from_unix_nanos(original_ns);
assert!(ts.nanos >= 0);
assert_eq!(timestamp_to_unix_nanos(&ts), original_ns);
}
#[test]
fn secs_round_trip_through_duration() {
let d = duration_from_secs(300);
assert_eq!(d.nanos, 0);
assert_eq!(duration_to_secs(&d), 300);
}
#[test]
fn micros_round_trip_through_duration() {
let original_us = 1_234_567_u64;
let d = duration_from_micros(original_us);
assert_eq!(duration_to_micros(&d), original_us);
}
}
crates/nuncio-store/Cargo.toml
+3
View File
@@ -28,3 +28,6 @@ hex = "0.4"
hmac.workspace = true
sha2.workspace = true
chrono.workspace = true
[dev-dependencies]
tracing-subscriber.workspace = true
crates/nuncio-store/src/cipher.rs
+64 -23
View File
@@ -64,28 +64,30 @@ impl PayloadCipher {
/// AES-256-GCM key. The key MUST be sourced from a [`crate::vault::SecretManager`]-backed
/// vault (never a compiled-in default) so that at-rest ciphertext cannot be decrypted
/// without access to the OS keyring.
pub fn encrypt_text_at_rest(key: &[u8; 32], text: &str) -> String {
pub fn encrypt_text_at_rest(key: &[u8; 32], text: &str) -> Result<String, CipherError> {
if text.is_empty() {
return String::new();
return Ok(String::new());
}
let encrypted = Self::encrypt_bytes(key, text.as_bytes()).unwrap_or_default();
hex::encode(encrypted)
let encrypted = Self::encrypt_bytes(key, text.as_bytes())?;
Ok(hex::encode(encrypted))
}
/// Decrypt text payload from database column storage at rest using a caller-supplied
/// AES-256-GCM key. See [`PayloadCipher::encrypt_text_at_rest`].
pub fn decrypt_text_at_rest(key: &[u8; 32], stored: &str) -> String {
///
/// Fails closed: a hex-decode error, an AEAD authentication failure (tampered or
/// corrupted ciphertext), or a non-UTF-8 decrypted payload is returned as an error
/// rather than silently read back as an empty body -- an empty stored ciphertext is
/// the only case that legitimately yields `Ok(String::new())`.
pub fn decrypt_text_at_rest(key: &[u8; 32], stored: &str) -> Result<String, CipherError> {
if stored.is_empty() {
return String::new();
return Ok(String::new());
}
if let Ok(bytes) = hex::decode(stored) {
if let Ok(decrypted) = Self::decrypt_bytes(key, &bytes) {
if let Ok(s) = String::from_utf8(decrypted) {
return s;
}
}
}
String::new()
let bytes = hex::decode(stored)
.map_err(|e| CipherError::DecryptionFailed(format!("invalid hex encoding: {e}")))?;
let decrypted = Self::decrypt_bytes(key, &bytes)?;
String::from_utf8(decrypted)
.map_err(|e| CipherError::DecryptionFailed(format!("invalid utf-8 payload: {e}")))
}
/// Encrypt binary attachment payload using `age` passphrase encryption.
@@ -236,27 +238,66 @@ mod tests {
fn text_at_rest_encryption_decryption_roundtrip() {
let key = [7u8; 32];
let text = "Sensitive Email Body Payload";
let encrypted = PayloadCipher::encrypt_text_at_rest(&key, text);
let encrypted =
PayloadCipher::encrypt_text_at_rest(&key, text).expect("encryption succeeds");
assert_ne!(text, encrypted);
let decrypted = PayloadCipher::decrypt_text_at_rest(&key, &encrypted);
let decrypted =
PayloadCipher::decrypt_text_at_rest(&key, &encrypted).expect("decryption succeeds");
assert_eq!(text, decrypted);
assert_eq!(PayloadCipher::encrypt_text_at_rest(&key, ""), "");
assert_eq!(PayloadCipher::decrypt_text_at_rest(&key, ""), "");
assert_eq!(
PayloadCipher::encrypt_text_at_rest(&key, "").expect("empty text is not encrypted"),
""
);
assert_eq!(
PayloadCipher::decrypt_text_at_rest(&key, "").expect("empty ciphertext decrypts"),
""
);
}
#[test]
fn text_at_rest_wrong_key_fails_to_decrypt() {
fn text_at_rest_wrong_key_fails_closed() {
// Proves there is no compiled-in default key: ciphertext produced under one
// caller-supplied key cannot be recovered using a different key.
// caller-supplied key cannot be recovered using a different key, and the failure
// is surfaced as an error rather than a silently-empty body.
let key_a = [1u8; 32];
let key_b = [2u8; 32];
let text = "Confidential ledger payload";
let encrypted = PayloadCipher::encrypt_text_at_rest(&key_a, text);
let decrypted_with_wrong_key = PayloadCipher::decrypt_text_at_rest(&key_b, &encrypted);
let encrypted =
PayloadCipher::encrypt_text_at_rest(&key_a, text).expect("encryption succeeds");
let err = PayloadCipher::decrypt_text_at_rest(&key_b, &encrypted)
.expect_err("wrong key must fail closed");
assert_eq!(decrypted_with_wrong_key, "");
assert!(matches!(err, CipherError::DecryptionFailed(_)));
}
#[test]
fn text_at_rest_tampered_ciphertext_fails_closed() {
// A corrupted-but-valid-hex ciphertext column must surface as an error, not read
// back as an empty body -- this is the read-path twin of the AEAD integrity
// guarantee: tampering must be detectable, not silently masked.
let key = [9u8; 32];
let text = "Do not silently drop this on tamper";
let encrypted =
PayloadCipher::encrypt_text_at_rest(&key, text).expect("encryption succeeds");
let mut bytes = hex::decode(&encrypted).expect("valid hex");
let last_idx = bytes.len() - 1;
bytes[last_idx] ^= 0xFF;
let tampered = hex::encode(bytes);
let err = PayloadCipher::decrypt_text_at_rest(&key, &tampered)
.expect_err("tampered ciphertext must fail closed");
assert!(matches!(err, CipherError::DecryptionFailed(_)));
}
#[test]
fn text_at_rest_invalid_hex_fails_closed() {
let key = [3u8; 32];
let err = PayloadCipher::decrypt_text_at_rest(&key, "not-valid-hex")
.expect_err("invalid hex must fail closed");
assert!(matches!(err, CipherError::DecryptionFailed(_)));
}
}
crates/nuncio-store/src/db.rs
+4203 -655
View File
File diff suppressed because it is too large. Load diff
crates/nuncio-store/src/lib.rs
+2
View File
@@ -7,6 +7,8 @@ pub mod cipher;
pub mod db;
pub mod recovery;
pub mod search;
#[cfg(test)]
mod test_tracing;
pub mod vault;
pub use cipher::{CipherError, PayloadCipher};
crates/nuncio-store/src/recovery.rs
+363 -36
View File
@@ -6,6 +6,11 @@ use std::path::{Path, PathBuf};
use tracing::{info, warn};
/// Raw row shape for an account configuration record fetched from SQLite during salvage.
/// The trailing two `String` columns are the raw `imap_tls_mode`/`smtp_tls_mode` text values
/// (e.g. `"start_tls"`) -- carried through as the literal on-disk token rather than already
/// parsed to [`nuncio_core::TlsMode`], so [`SqliteRecoveryEngine::salvage_accounts`] can log
/// precisely which raw value it could not recognize, if any, instead of a decode failure
/// having already been silently absorbed by this row type.
type SalvagedAccountRow = (
String,
String,
@@ -13,11 +18,13 @@ type SalvagedAccountRow = (
String,
String,
i64,
i64,
String,
i64,
Option<String>,
Option<i64>,
String,
String,
Option<String>,
);
/// Summary report of database self-healing recovery output.
@@ -33,6 +40,10 @@ pub struct RecoverySummary {
pub salvaged_conditions_count: usize,
/// Number of salvaged filter action specifications.
pub salvaged_actions_count: usize,
/// Number of salvaged WORM audit ledger records.
pub salvaged_audit_records_count: usize,
/// Number of salvaged filter execution log entries.
pub salvaged_execution_logs_count: usize,
/// Indicates whether remote protocol resynchronization was initiated.
pub resync_triggered: bool,
}
@@ -179,6 +190,18 @@ impl SqliteRecoveryEngine {
/// Salvage readable accounts and filter rules from damaged database or backup file,
/// re-creating a fresh SQLite file at `target_db_path`. Cryptographic key material for
/// the freshly re-created engine is provisioned from `secrets`.
///
/// # Precondition
///
/// The caller MUST have fully closed (via [`DatabaseEngine::close`], which awaits every
/// pooled connection's teardown) any engine still holding `target_db_path` before calling
/// this. Salvage unlinks `target_db_path` and its `-wal`/`-shm` companions and then creates
/// a brand-new database at that same path; a connection left open over the old file is torn
/// down asynchronously, and SQLite's close-time WAL handling operates on `-wal`/`-shm` *by
/// path*. A teardown landing after the unlink therefore collides with the fresh database's
/// own companions -- leaving the new, empty main file paired with a `-wal` describing the
/// old one, whose replay reads past end-of-file and fails with `SQLITE_IOERR_SHORT_READ`.
/// `DatabaseEngine::close_and_salvage` is the in-tree caller that honours this.
pub async fn salvage(
corrupted_db_path: &Path,
target_db_path: &Path,
@@ -191,21 +214,41 @@ impl SqliteRecoveryEngine {
// Step 2: Attempt reading valid records from preserved backup file
let backup_url = format!("sqlite://{}", backup_path.to_string_lossy());
let (salvaged_accounts, salvaged_rules, salvaged_conditions, salvaged_actions) = if let Ok(
pool,
) =
sqlx::sqlite::SqlitePoolOptions::new()
.max_connections(1)
.connect(&backup_url)
.await
let (
salvaged_accounts,
salvaged_rules,
salvaged_conditions,
salvaged_actions,
salvaged_audit_records,
salvaged_execution_logs,
) = if let Ok(pool) = sqlx::sqlite::SqlitePoolOptions::new()
.max_connections(1)
.connect(&backup_url)
.await
{
let accounts = Self::salvage_accounts(&pool).await;
let (rules, conditions, actions) = Self::salvage_filter_tables(&pool).await;
let audit_records = Self::salvage_worm_audit_records(&pool).await;
let execution_logs = Self::salvage_filter_execution_logs(&pool).await;
pool.close().await;
(accounts, rules, conditions, actions)
(
accounts,
rules,
conditions,
actions,
audit_records,
execution_logs,
)
} else {
warn!("Failed to open backup connection for salvage; proceeding with clean database reset.");
(Vec::new(), Vec::new(), Vec::new(), Vec::new())
(
Vec::new(),
Vec::new(),
Vec::new(),
Vec::new(),
Vec::new(),
Vec::new(),
)
};
// Step 3: Remove or truncate damaged database files at target_db_path
@@ -287,12 +330,62 @@ impl SqliteRecoveryEngine {
}
}
// Restore the WORM audit ledger and filter execution log ledger verbatim, including
// their hash-chain columns (`record_hmac`/`hash`, `previous_block_hash`/`prev_hash`).
// These are re-signed with the same HMAC key material the corrupted database used
// (both engines are provisioned from the same `secrets` vault), so a chain that
// verified before corruption still verifies after salvage -- salvage restores the
// ledger's tamper-evidence, it does not merely copy rows that happen to look right.
let mut restored_audit_records_count = 0;
for (sequence, timestamp_ns, actor, action, data_hash, previous_block_hash, record_hmac) in
&salvaged_audit_records
{
let res = sqlx::query(
"INSERT INTO worm_audit_records (sequence, timestamp_ns, actor, action, data_hash, previous_block_hash, record_hmac) VALUES (?, ?, ?, ?, ?, ?, ?)"
)
.bind(sequence)
.bind(timestamp_ns)
.bind(actor)
.bind(action)
.bind(data_hash)
.bind(previous_block_hash)
.bind(record_hmac)
.execute(fresh_engine.pool())
.await;
if res.is_ok() {
restored_audit_records_count += 1;
}
}
let mut restored_execution_logs_count = 0;
for (id, rule_id, message_id, action_taken, matched_at, prev_hash, hash) in
&salvaged_execution_logs
{
let res = sqlx::query(
"INSERT INTO filter_execution_logs (id, rule_id, message_id, action_taken, matched_at, prev_hash, hash) VALUES (?, ?, ?, ?, ?, ?, ?)"
)
.bind(id)
.bind(rule_id)
.bind(message_id)
.bind(action_taken)
.bind(matched_at)
.bind(prev_hash)
.bind(hash)
.execute(fresh_engine.pool())
.await;
if res.is_ok() {
restored_execution_logs_count += 1;
}
}
let summary = RecoverySummary {
backup_path,
salvaged_accounts_count: restored_accounts_count,
salvaged_rules_count: restored_rules_count,
salvaged_conditions_count: restored_conditions_count,
salvaged_actions_count: restored_actions_count,
salvaged_audit_records_count: restored_audit_records_count,
salvaged_execution_logs_count: restored_execution_logs_count,
resync_triggered: true,
};
@@ -302,7 +395,7 @@ impl SqliteRecoveryEngine {
async fn salvage_accounts(pool: &sqlx::SqlitePool) -> Vec<nuncio_core::AccountConfig> {
let rows: Result<Vec<SalvagedAccountRow>, _> = sqlx::query_as(
"SELECT id, name, email_address, protocol, server_host, server_port, use_tls, keyring_secret_key, sync_interval_secs, smtp_host, smtp_port FROM accounts"
"SELECT id, name, email_address, protocol, server_host, server_port, keyring_secret_key, sync_interval_secs, smtp_host, smtp_port, imap_tls_mode, smtp_tls_mode, collection_url FROM accounts"
)
.fetch_all(pool)
.await;
@@ -318,11 +411,13 @@ impl SqliteRecoveryEngine {
protocol_str,
server_host,
server_port,
use_tls,
keyring_secret_key,
sync_interval_secs,
smtp_host,
smtp_port,
imap_tls_mode_raw,
smtp_tls_mode_raw,
collection_url,
)| {
let protocol = serde_json::from_str(&protocol_str)
.unwrap_or(nuncio_core::AccountProtocol::ImapSmtp);
@@ -335,20 +430,54 @@ impl SqliteRecoveryEngine {
let resolved_smtp_host = smtp_host.unwrap_or_else(|| server_host.clone());
let resolved_smtp_port =
smtp_port.map(|p| p as u16).unwrap_or(server_port as u16);
let imap_tls_mode =
Self::parse_salvaged_tls_mode(&id, "imap_tls_mode", &imap_tls_mode_raw);
let smtp_tls_mode =
Self::parse_salvaged_tls_mode(&id, "smtp_tls_mode", &smtp_tls_mode_raw);
// Reconstruct the transport variant from the salvaged
// `protocol` discriminator, mirroring
// `DatabaseEngine::list_accounts`.
let transport = match protocol {
nuncio_core::AccountProtocol::ImapSmtp => {
nuncio_core::Transport::ImapSmtp(nuncio_core::ImapSmtpTransport {
imap_host: server_host,
imap_port: server_port as u16,
imap_tls_mode,
smtp_host: resolved_smtp_host,
smtp_port: resolved_smtp_port,
smtp_tls_mode,
})
}
nuncio_core::AccountProtocol::Jmap => {
nuncio_core::Transport::Jmap(nuncio_core::JmapTransport {
endpoint_host: server_host,
})
}
nuncio_core::AccountProtocol::CalDav => {
nuncio_core::Transport::Dav(nuncio_core::DavTransport {
collection_url: collection_url.unwrap_or_default(),
})
}
nuncio_core::AccountProtocol::CardDav => {
nuncio_core::Transport::CardDav(nuncio_core::DavTransport {
collection_url: collection_url.unwrap_or_default(),
})
}
};
nuncio_core::AccountConfig {
id,
name,
email_address,
protocol,
server_host,
server_port: server_port as u16,
smtp_host: resolved_smtp_host,
smtp_port: resolved_smtp_port,
use_tls: use_tls != 0,
imap_tls_mode: nuncio_core::TlsMode::ImplicitTls,
smtp_tls_mode: nuncio_core::TlsMode::ImplicitTls,
keyring_secret_key,
sync_interval_secs: sync_interval_secs as u64,
// Salvage deliberately does not recover this flag.
// The column may be among what the corruption took,
// and re-enabling filter execution on a guess would
// start forwarding mail from a daemon the user never
// chose as the owner. Off is recoverable by one
// setting; wrongly on is not recoverable at all.
filters_enabled: false,
transport,
}
},
)
@@ -357,6 +486,64 @@ impl SqliteRecoveryEngine {
}
}
/// Parse a salvaged account's raw `imap_tls_mode`/`smtp_tls_mode` column text back into
/// [`nuncio_core::TlsMode`], preserving the account's real configured security posture
/// (implicit TLS, STARTTLS, or plaintext) rather than hardcoding every salvaged account
/// back to implicit TLS regardless of how it was actually configured -- silently
/// upgrading a STARTTLS account or downgrading a plaintext one to "implicit TLS" on
/// recovery would misrepresent the account's security posture to the operator.
///
/// A raw value that does not match one of the three known on-disk tokens (a row from a
/// build that predates this column, or a distinct new corruption) cannot be honestly
/// recovered. Rather than silently guessing at one of the other two modes, this WARN-logs
/// the account id and the unrecognized raw value, and falls back to the safest mode
/// (implicit TLS) -- the operator can see in the daemon's logs exactly which account
/// needs its TLS mode re-confirmed, instead of a quiet, invisible substitution.
fn parse_salvaged_tls_mode(account_id: &str, column: &str, raw: &str) -> nuncio_core::TlsMode {
match raw {
"implicit_tls" => nuncio_core::TlsMode::ImplicitTls,
"start_tls" => nuncio_core::TlsMode::StartTls,
"plain" => nuncio_core::TlsMode::Plain,
other => {
warn!(
"Salvage could not recognize account {account_id}'s {column} value {other:?}; \
defaulting to ImplicitTls and flagging for operator review rather than \
silently guessing its prior TLS mode."
);
nuncio_core::TlsMode::ImplicitTls
}
}
}
/// Read back valid rows from `worm_audit_records` in the backup connection, preserving
/// every column including the hash-chain fields (`previous_block_hash`/`record_hmac`) so
/// the restored ledger's tamper-evidence carries through salvage rather than being
/// silently dropped.
async fn salvage_worm_audit_records(
pool: &sqlx::SqlitePool,
) -> Vec<(i64, i64, String, String, String, String, String)> {
sqlx::query_as::<_, (i64, i64, String, String, String, String, String)>(
"SELECT sequence, timestamp_ns, actor, action, data_hash, previous_block_hash, record_hmac FROM worm_audit_records ORDER BY sequence ASC"
)
.fetch_all(pool)
.await
.unwrap_or_default()
}
/// Read back valid rows from `filter_execution_logs` in the backup connection, preserving
/// every column including the hash-chain fields (`prev_hash`/`hash`) so the restored
/// ledger's tamper-evidence carries through salvage rather than being silently dropped.
async fn salvage_filter_execution_logs(
pool: &sqlx::SqlitePool,
) -> Vec<(i64, String, String, String, i64, String, String)> {
sqlx::query_as::<_, (i64, String, String, String, i64, String, String)>(
"SELECT id, rule_id, message_id, action_taken, matched_at, prev_hash, hash FROM filter_execution_logs ORDER BY id ASC"
)
.fetch_all(pool)
.await
.unwrap_or_default()
}
/// Read back valid rows from `filter_rules` / `filter_conditions` / `filter_actions` in the
/// backup connection, using column names/order matching the live schema (see
/// `DatabaseEngine::migrate`). `filter_conditions.value`/`filter_actions.target` are the
@@ -453,16 +640,17 @@ mod tests {
id: "acct-test-1".to_string(),
name: "Work Account".to_string(),
email_address: "work@nuncio.mx".to_string(),
protocol: nuncio_core::AccountProtocol::ImapSmtp,
server_host: "imap.nuncio.mx".to_string(),
server_port: 993,
smtp_host: "smtp.nuncio.mx".to_string(),
smtp_port: 465,
use_tls: true,
imap_tls_mode: nuncio_core::TlsMode::ImplicitTls,
smtp_tls_mode: nuncio_core::TlsMode::ImplicitTls,
keyring_secret_key: "nuncio/acct-test-1".to_string(),
sync_interval_secs: 60,
filters_enabled: false,
transport: nuncio_core::Transport::ImapSmtp(nuncio_core::ImapSmtpTransport {
imap_host: "imap.nuncio.mx".to_string(),
imap_port: 993,
imap_tls_mode: nuncio_core::TlsMode::ImplicitTls,
smtp_host: "smtp.nuncio.mx".to_string(),
smtp_port: 465,
smtp_tls_mode: nuncio_core::TlsMode::ImplicitTls,
}),
};
engine.save_account(&acct).await.unwrap();
assert!(engine.check_integrity().await.unwrap());
@@ -514,22 +702,34 @@ mod tests {
id: "acct-salvage-1".to_string(),
name: "Salvage Account".to_string(),
email_address: "salvage@nuncio.mx".to_string(),
protocol: nuncio_core::AccountProtocol::ImapSmtp,
server_host: "imap.nuncio.mx".to_string(),
server_port: 993,
smtp_host: "smtp.nuncio.mx".to_string(),
smtp_port: 465,
use_tls: true,
imap_tls_mode: nuncio_core::TlsMode::ImplicitTls,
smtp_tls_mode: nuncio_core::TlsMode::ImplicitTls,
keyring_secret_key: "nuncio/acct-salvage-1".to_string(),
sync_interval_secs: 60,
filters_enabled: false,
transport: nuncio_core::Transport::ImapSmtp(nuncio_core::ImapSmtpTransport {
imap_host: "imap.nuncio.mx".to_string(),
imap_port: 993,
imap_tls_mode: nuncio_core::TlsMode::ImplicitTls,
smtp_host: "smtp.nuncio.mx".to_string(),
smtp_port: 465,
smtp_tls_mode: nuncio_core::TlsMode::ImplicitTls,
}),
};
engine.save_account(&acct).await.unwrap();
let nsql = "SELECT * FROM emails WHERE subject CONTAINS 'Spam' ACTION DELETE";
let rule = nuncio_filter::NsqlParser::parse_rule("Spam Filter", 1, nsql).unwrap();
engine.save_filter_rule(&rule).await.unwrap();
// Closing before salvage is the documented precondition of
// `SqliteRecoveryEngine::salvage`, and it is what the production caller
// (`DatabaseEngine::close_and_salvage`) does. Merely dropping the engine is not
// equivalent: the pool's teardown is deferred onto the runtime, so the old
// connection's close-time WAL handling can land *after* salvage has unlinked
// `-wal`/`-shm` and created a fresh database at this same path -- pairing the new,
// empty main file with a `-wal` describing the old one and failing the subsequent
// page read with `SQLITE_IOERR_SHORT_READ`. `close()` awaits that teardown, so the
// window does not exist at all rather than merely being narrow.
engine.close().await;
}
// Step 2: Perform salvage recovery
@@ -558,6 +758,133 @@ mod tests {
assert_eq!(rules[0].name, "Spam Filter");
}
#[tokio::test]
async fn salvage_preserves_worm_audit_and_filter_execution_log_chains() {
let dir = tempdir().expect("tempdir");
let db_path = dir.path().join("ledger_salvage_test.db");
let backup_dir = dir.path().join("corrupted_backups");
let secrets = crate::vault::SecretManager::mock();
// Step 1: Seed a WORM audit record and a filter execution log entry, each of which
// is only valid because it is cryptographically hash-chained to its predecessor.
{
let engine = DatabaseEngine::connect_file(&db_path, &secrets)
.await
.unwrap();
engine
.append_worm_audit_record("system.test", "account.created", b"payload-1")
.await
.unwrap();
engine
.append_worm_audit_record("system.test", "account.updated", b"payload-2")
.await
.unwrap();
engine
.save_filter_execution_log("rule-1", "msg-1", "delete")
.await
.unwrap();
engine
.save_filter_execution_log("rule-1", "msg-2", "archive")
.await
.unwrap();
assert!(engine.verify_worm_audit_chain().await.is_ok());
assert!(engine.verify_execution_log_chain().await.unwrap());
}
// Step 2: Perform salvage recovery, reusing the same secrets vault -- the WORM/ledger
// HMAC keys are provisioned from `secrets`, so the restored ledgers verify against
// the same key material the corrupted database used, exactly as in the real recovery
// flow (the daemon always re-provisions salvage from the account's existing vault).
let summary = SqliteRecoveryEngine::salvage(&db_path, &db_path, &backup_dir, &secrets)
.await
.expect("salvage succeeds");
assert_eq!(
summary.salvaged_audit_records_count, 2,
"salvage must carry worm_audit_records through recovery instead of silently \
dropping the tamper-evident audit ledger"
);
assert_eq!(
summary.salvaged_execution_logs_count, 2,
"salvage must carry filter_execution_logs through recovery instead of silently \
dropping the filter ledger"
);
// Step 3: The restored ledgers must still be present AND still pass hash-chain
// verification -- proving the chain-linking columns (`previous_block_hash`/
// `record_hmac`, `prev_hash`/`hash`) were preserved verbatim, not just the row data.
let fresh = DatabaseEngine::connect_file(&db_path, &secrets)
.await
.unwrap();
let audit_records = fresh.list_worm_audit_records(100, 0).await.unwrap();
assert_eq!(audit_records.len(), 2);
assert!(
fresh.verify_worm_audit_chain().await.is_ok(),
"restored WORM audit chain must still verify after salvage"
);
let execution_logs = fresh.list_filter_execution_logs(100).await.unwrap();
assert_eq!(execution_logs.len(), 2);
assert!(
fresh.verify_execution_log_chain().await.unwrap(),
"restored filter execution log chain must still verify after salvage"
);
}
#[tokio::test]
async fn salvage_preserves_real_tls_mode_instead_of_downgrading_to_implicit() {
let dir = tempdir().expect("tempdir");
let db_path = dir.path().join("tls_mode_salvage_test.db");
let backup_dir = dir.path().join("corrupted_backups");
let secrets = crate::vault::SecretManager::mock();
// Step 1: Seed an account configured for STARTTLS on IMAP and plaintext on SMTP --
// deliberately neither is `ImplicitTls`, so a salvage that hardcodes the TLS mode
// back to `ImplicitTls` is caught by asserting the real modes survive.
{
let engine = DatabaseEngine::connect_file(&db_path, &secrets)
.await
.unwrap();
let acct = AccountConfig {
id: "acct-starttls-1".to_string(),
name: "STARTTLS Account".to_string(),
email_address: "starttls@nuncio.mx".to_string(),
keyring_secret_key: "nuncio/acct-starttls-1".to_string(),
sync_interval_secs: 60,
filters_enabled: false,
transport: nuncio_core::Transport::ImapSmtp(nuncio_core::ImapSmtpTransport {
imap_host: "imap.nuncio.mx".to_string(),
imap_port: 143,
imap_tls_mode: nuncio_core::TlsMode::StartTls,
smtp_host: "smtp.nuncio.mx".to_string(),
smtp_port: 587,
smtp_tls_mode: nuncio_core::TlsMode::Plain,
}),
};
engine.save_account(&acct).await.unwrap();
}
// Step 2: Perform salvage recovery.
let summary = SqliteRecoveryEngine::salvage(&db_path, &db_path, &backup_dir, &secrets)
.await
.expect("salvage succeeds");
assert_eq!(summary.salvaged_accounts_count, 1);
// Step 3: The salvaged account must keep its real configured TLS modes, not be
// silently reset to `ImplicitTls` -- that would misrepresent the account's actual,
// intentionally-configured security posture (STARTTLS on IMAP, plaintext on SMTP).
let fresh = DatabaseEngine::connect_file(&db_path, &secrets)
.await
.unwrap();
let accounts = fresh.list_accounts().await.unwrap();
assert_eq!(accounts.len(), 1);
let salvaged_t = accounts[0].imap_smtp().expect("imap-smtp transport");
assert_eq!(salvaged_t.imap_tls_mode, nuncio_core::TlsMode::StartTls);
assert_eq!(salvaged_t.smtp_tls_mode, nuncio_core::TlsMode::Plain);
}
// NOTE: there is deliberately no test combining *header* corruption (bytes 0-16 -- the
// fixed "SQLite format 3\0" magic string) with a nonzero salvaged-rule-count assertion.
// A corrupted header makes the file entirely unopenable by SQLite (`SQLITE_NOTADB`), by
crates/nuncio-store/src/search.rs
+400 -23
View File
@@ -6,7 +6,10 @@ use serde::{Deserialize, Serialize};
/// Search hit result for full-text query matches.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct SearchHit {
/// Entity ID matching the query.
/// Entity ID matching the query. For a message hit this is the
/// folder-independent message key: the index holds one row per message, so
/// a hit names a message and never one of its mailbox placements. Resolve
/// it through [`DatabaseEngine::get_message`].
pub id: String,
/// Matching title or subject.
pub title: String,
@@ -50,7 +53,7 @@ impl<'a> SearchEngine<'a> {
let rows: Vec<(String, String, String)> = sqlx::query_as(
r#"
SELECT id, subject, snippet(messages_fts, 3, '<b>', '</b>', '...', 10) as snippet
SELECT message_key, subject, snippet(messages_fts, 3, '<b>', '</b>', '...', 10) as snippet
FROM messages_fts
WHERE messages_fts MATCH ?
ORDER BY rank
@@ -68,6 +71,157 @@ impl<'a> SearchEngine<'a> {
.collect())
}
/// Keyset-paginated full-text search over message subjects, senders, and
/// bodies, ordered by relevance (`rank ASC, id ASC`). `after` is the
/// `(rank, id)` of the last hit of the previous page; `None` starts from
/// the highest-ranked hit. Fetches `page_size + 1` hits to detect a
/// following page and returns the `(rank, id)` cursor of the last returned
/// hit when more remain.
///
/// The bm25 `rank` is materialized in an inner query (where the FTS5
/// `MATCH` is in scope) into an ordinary column, so the outer query can
/// both keyset-filter and order by `(rank, id)` -- preserving relevance
/// order across pages while resuming strictly after the previous page's
/// last hit.
pub async fn search_messages_page(
&self,
query: &str,
after: Option<(f64, String)>,
page_size: usize,
) -> Result<(Vec<SearchHit>, Option<(f64, String)>), DatabaseError> {
let clean_query = Self::sanitize_fts5_query(query);
if clean_query.is_empty() {
return Ok((Vec::new(), None));
}
let fetch = page_size.saturating_add(1);
let mut builder: sqlx::QueryBuilder<'_, sqlx::Sqlite> = sqlx::QueryBuilder::new(
"SELECT id, title, snippet, rk FROM ( \
SELECT message_key AS id, subject AS title, \
snippet(messages_fts, 3, '<b>', '</b>', '...', 10) AS snippet, rank AS rk \
FROM messages_fts WHERE messages_fts MATCH ",
);
builder.push_bind(clean_query);
builder.push(") ");
if let Some((rank, id)) = &after {
builder.push("WHERE (rk > ");
builder.push_bind(*rank);
builder.push(" OR (rk = ");
builder.push_bind(*rank);
builder.push(" AND id > ");
builder.push_bind(id.clone());
builder.push("))");
}
builder.push(" ORDER BY rk ASC, id ASC LIMIT ");
builder.push_bind(fetch as i64);
let rows = builder
.build_query_as::<(String, String, String, f64)>()
.fetch_all(self.db.pool())
.await
.map_err(DatabaseError::Query)?;
let has_more = rows.len() > page_size;
let mut last_key: Option<(f64, String)> = None;
let hits: Vec<SearchHit> = rows
.into_iter()
.take(page_size)
.map(|(id, title, snippet, rk)| {
last_key = Some((rk, id.clone()));
SearchHit { id, title, snippet }
})
.collect();
let next = if has_more { last_key } else { None };
Ok((hits, next))
}
/// Perform a full-text trigram search over contact display names, organizations, and
/// email addresses.
pub async fn search_contacts(&self, query: &str) -> Result<Vec<SearchHit>, DatabaseError> {
let clean_query = Self::sanitize_fts5_query(query);
if clean_query.is_empty() {
return Ok(Vec::new());
}
let rows: Vec<(String, String, String)> = sqlx::query_as(
r#"
SELECT id, display_name, snippet(contacts_fts, 3, '<b>', '</b>', '...', 10) as snippet
FROM contacts_fts
WHERE contacts_fts MATCH ?
ORDER BY rank
LIMIT 50
"#,
)
.bind(&clean_query)
.fetch_all(self.db.pool())
.await
.map_err(DatabaseError::Query)?;
Ok(rows
.into_iter()
.map(|(id, title, snippet)| SearchHit { id, title, snippet })
.collect())
}
/// Keyset-paginated full-text search over contact display names,
/// organizations, and email addresses, ordered by relevance (`rank ASC, id
/// ASC`). Mirrors [`Self::search_messages_page`]'s rank-materializing
/// subquery so the same keyset-pagination guarantee (every hit returned
/// exactly once, across pages) applies to contact search.
pub async fn search_contacts_page(
&self,
query: &str,
after: Option<(f64, String)>,
page_size: usize,
) -> Result<(Vec<SearchHit>, Option<(f64, String)>), DatabaseError> {
let clean_query = Self::sanitize_fts5_query(query);
if clean_query.is_empty() {
return Ok((Vec::new(), None));
}
let fetch = page_size.saturating_add(1);
let mut builder: sqlx::QueryBuilder<'_, sqlx::Sqlite> = sqlx::QueryBuilder::new(
"SELECT id, title, snippet, rk FROM ( \
SELECT id, display_name AS title, \
snippet(contacts_fts, 3, '<b>', '</b>', '...', 10) AS snippet, rank AS rk \
FROM contacts_fts WHERE contacts_fts MATCH ",
);
builder.push_bind(clean_query);
builder.push(") ");
if let Some((rank, id)) = &after {
builder.push("WHERE (rk > ");
builder.push_bind(*rank);
builder.push(" OR (rk = ");
builder.push_bind(*rank);
builder.push(" AND id > ");
builder.push_bind(id.clone());
builder.push("))");
}
builder.push(" ORDER BY rk ASC, id ASC LIMIT ");
builder.push_bind(fetch as i64);
let rows = builder
.build_query_as::<(String, String, String, f64)>()
.fetch_all(self.db.pool())
.await
.map_err(DatabaseError::Query)?;
let has_more = rows.len() > page_size;
let mut last_key: Option<(f64, String)> = None;
let hits: Vec<SearchHit> = rows
.into_iter()
.take(page_size)
.map(|(id, title, snippet, rk)| {
last_key = Some((rk, id.clone()));
SearchHit { id, title, snippet }
})
.collect();
let next = if has_more { last_key } else { None };
Ok((hits, next))
}
/// Perform a full-text trigram search over calendar event summaries and locations.
pub async fn search_events(&self, query: &str) -> Result<Vec<SearchHit>, DatabaseError> {
let clean_query = Self::sanitize_fts5_query(query);
@@ -104,19 +258,39 @@ mod tests {
nuncio_core::model::Email {
id: id.to_string(),
account_id: "acct-1".to_string(),
folder_id: "inbox".to_string(),
subject: subject.to_string(),
sender: "alice@nuncio.mx".to_string(),
recipient: "bob@nuncio.mx".to_string(),
received_at: 1_700_000_000,
read: false,
body_plain: Some(body_plain.to_string()),
body_html: None,
attachments: Vec::new(),
message_id: None,
content_hash: None,
}
}
/// The crux test: proves body search works over the REAL `save_email` write path (no
fn sample_placement(uid: &str) -> nuncio_core::model::Placement {
nuncio_core::model::Placement {
account_id: "acct-1".to_string(),
folder_id: "inbox".to_string(),
uid_validity: "1".to_string(),
remote_id: uid.to_string(),
read: false,
}
}
async fn save(db: &DatabaseEngine, email: &nuncio_core::model::Email) {
db.save_email_at(
email,
nuncio_core::model::IdentitySource::Surrogate,
&sample_placement(&email.id),
)
.await
.unwrap();
}
/// The crux test: proves body search works over the REAL `save_email_at` write path (no
/// raw-SQL plaintext injection bypassing encryption), while the stored body column stays
/// ciphertext. This is what the old `fts5_message_search_and_triggers` test failed to
/// prove -- it inserted plaintext directly via raw SQL, which trivially "worked" but never
@@ -135,7 +309,7 @@ mod tests {
"Quarterly Financial Meeting",
"Let's discuss the annual budget revenue forecast",
);
db.save_email(&email).await.unwrap();
save(&db, &email).await;
// Body search matches a plaintext body term -- proves the FTS index holds real,
// decrypted-at-write-time plaintext trigrams, not the AES-256-GCM ciphertext that is
@@ -152,39 +326,44 @@ mod tests {
// The body column at rest must remain encrypted ciphertext -- never the plaintext
// search term, and never equal to the original plaintext body.
let (stored_body,): (String,) =
sqlx::query_as("SELECT body_plain FROM messages WHERE id = 'msg-1'")
sqlx::query_as("SELECT body_plain FROM messages WHERE message_key = 'msg-1'")
.fetch_one(db.pool())
.await
.unwrap();
assert_ne!(stored_body, email.body_plain.clone().unwrap());
assert!(!stored_body.to_lowercase().contains("revenue"));
// Re-saving via save_email (an update, not a fresh insert) replaces the FTS row rather
// than duplicating or leaving a stale entry: the old term no longer matches, the new
// one does.
let mut updated = email.clone();
updated.body_plain = Some("Updated strategy review notes".to_string());
db.save_email(&updated).await.unwrap();
// Re-saving the same key with a different body must leave the index alone: the message
// row is first-write-wins, so a second write that cannot change the stored body must
// not change what search returns for it either. Anything else would have search
// disagree with the store -- either finding a body that was never persisted, or holding
// two rows for one message.
let mut rewritten = email.clone();
rewritten.body_plain = Some("Updated strategy review notes".to_string());
save(&db, &rewritten).await;
assert!(search.search_messages("revenue").await.unwrap().is_empty());
let new_hits = search.search_messages("strategy").await.unwrap();
assert_eq!(new_hits.len(), 1);
assert_eq!(new_hits[0].id, "msg-1");
let unchanged = search.search_messages("revenue").await.unwrap();
assert_eq!(unchanged.len(), 1, "the stored body is still searchable");
assert_eq!(unchanged[0].id, "msg-1");
assert!(
search.search_messages("strategy").await.unwrap().is_empty(),
"a body the store rejected must never become searchable"
);
// Delete message trigger still cleans up the FTS row (subject/sender/id only -- no
// Delete message trigger still cleans up the FTS row (subject/sender/key only -- no
// ciphertext involved).
sqlx::query("DELETE FROM messages WHERE id = 'msg-1'")
sqlx::query("DELETE FROM messages WHERE message_key = 'msg-1'")
.execute(db.pool())
.await
.unwrap();
let deleted_hits = search.search_messages("strategy").await.unwrap();
let deleted_hits = search.search_messages("revenue").await.unwrap();
assert!(deleted_hits.is_empty());
}
/// Proves the backfill half of the fix: a message saved through the real write path,
/// whose FTS entry is then lost (simulating either a row written before the FTS5 index
/// existed, or one written by a process that bypassed `save_email`'s explicit indexing),
/// existed, or one written by a process that bypassed `save_email_at`'s explicit indexing),
/// is NOT searchable until a migration pass runs -- at which point `backfill_message_fts`
/// decrypts the ciphertext body column with this engine's real storage key and repopulates
/// the plaintext-derived trigram index, with no search call ever required to trigger it.
@@ -196,11 +375,11 @@ mod tests {
"Legacy Roadmap Notes",
"Confidential quarterly roadmap details",
);
db.save_email(&email).await.unwrap();
save(&db, &email).await;
// Simulate pre-existing data whose FTS entry is missing (e.g. written before the FTS5
// index existed) by dropping its messages_fts row directly.
sqlx::query("DELETE FROM messages_fts WHERE id = 'msg-legacy-1'")
sqlx::query("DELETE FROM messages_fts WHERE message_key = 'msg-legacy-1'")
.execute(db.pool())
.await
.unwrap();
@@ -222,6 +401,54 @@ mod tests {
assert_eq!(hits[0].title, "Legacy Roadmap Notes");
}
/// Paging FTS search results across multiple pages via the `(rank, id)`
/// keyset must return every matching hit EXACTLY ONCE, no dupes and no
/// gaps -- proving the rank-materializing subquery keyset works end to
/// end over the real encrypt-then-index write path.
#[tokio::test]
async fn search_messages_page_returns_every_hit_once_no_gaps() {
let (db, _dir) = DatabaseEngine::connect_ephemeral().await.unwrap();
for i in 0..5 {
save(
&db,
&sample_email(
&format!("msg-{i}"),
"Budget Planning Meeting",
"annual budget revenue forecast discussion",
),
)
.await;
}
let search = SearchEngine::new(&db);
let mut seen = Vec::new();
let mut after: Option<(f64, String)> = None;
let mut pages = 0;
loop {
let (hits, next) = search
.search_messages_page("budget", after.clone(), 2)
.await
.unwrap();
pages += 1;
assert!(hits.len() <= 2, "page must not exceed page_size");
seen.extend(hits.iter().map(|h| h.id.clone()));
match next {
Some(cursor) => after = Some(cursor),
None => break,
}
assert!(pages < 100, "pagination must terminate");
}
seen.sort();
seen.dedup();
assert_eq!(
seen.len(),
5,
"every matching message returned exactly once"
);
assert!(pages >= 3, "5 hits at page_size 2 must span multiple pages");
}
#[tokio::test]
async fn fts5_event_search_and_triggers() {
let (db, _dir) = DatabaseEngine::connect_ephemeral().await.unwrap();
@@ -246,6 +473,156 @@ mod tests {
assert_eq!(hits[0].title, "Architecture Summit");
}
async fn insert_contact(
db: &DatabaseEngine,
id: &str,
display_name: &str,
organization: &str,
emails_json: &str,
) {
sqlx::query(
"INSERT INTO contacts \
(id, display_name, organization, emails_json, created_at, updated_at) \
VALUES (?, ?, ?, ?, '2024-01-01T00:00:00Z', '2024-01-01T00:00:00Z')",
)
.bind(id)
.bind(display_name)
.bind(organization)
.bind(emails_json)
.execute(db.pool())
.await
.unwrap();
}
/// Proves `contacts_fts` is actually populated by the `contacts_ai`/`contacts_au` triggers
/// (not merely created and left unused): several contacts are inserted, a matching term
/// ranks the true matches and excludes the non-matching contact.
#[tokio::test]
async fn fts5_contact_search_ranks_matches_and_excludes_others() {
let (db, _dir) = DatabaseEngine::connect_ephemeral().await.unwrap();
let search = SearchEngine::new(&db);
assert!(search.search_contacts("").await.unwrap().is_empty());
insert_contact(&db, "ct-1", "Alice Architect", "Acme Corp", "[]").await;
insert_contact(&db, "ct-2", "Bob Baker", "Baker Studio", "[]").await;
insert_contact(&db, "ct-3", "Carol Architect", "Acme Corp", "[]").await;
let hits = search.search_contacts("Architect").await.unwrap();
let ids: Vec<&str> = hits.iter().map(|h| h.id.as_str()).collect();
assert_eq!(hits.len(), 2);
assert!(ids.contains(&"ct-1"));
assert!(ids.contains(&"ct-3"));
assert!(!ids.contains(&"ct-2"));
let bob_hits = search.search_contacts("Bob").await.unwrap();
assert_eq!(bob_hits.len(), 1);
assert_eq!(bob_hits[0].id, "ct-2");
assert_eq!(bob_hits[0].title, "Bob Baker");
}
/// Updating and deleting a contact keeps `contacts_fts` in sync via the `contacts_au`/
/// `contacts_ad` triggers: a renamed contact stops matching its old name and starts
/// matching its new one, and a deleted contact stops matching entirely.
#[tokio::test]
async fn fts5_contact_search_reflects_update_and_delete() {
let (db, _dir) = DatabaseEngine::connect_ephemeral().await.unwrap();
let search = SearchEngine::new(&db);
insert_contact(&db, "ct-upd-1", "Dana Original", "Original Org", "[]").await;
assert_eq!(search.search_contacts("Original").await.unwrap().len(), 1);
sqlx::query(
"UPDATE contacts SET display_name = 'Dana Renamed', organization = 'New Org' \
WHERE id = 'ct-upd-1'",
)
.execute(db.pool())
.await
.unwrap();
assert!(search.search_contacts("Original").await.unwrap().is_empty());
let hits = search.search_contacts("Renamed").await.unwrap();
assert_eq!(hits.len(), 1);
assert_eq!(hits[0].id, "ct-upd-1");
sqlx::query("DELETE FROM contacts WHERE id = 'ct-upd-1'")
.execute(db.pool())
.await
.unwrap();
assert!(search.search_contacts("Renamed").await.unwrap().is_empty());
}
/// A search term matching an indexed email address (stored as JSON) is found, and a
/// sanitized/edge-case query (FTS5 operator characters) does not error -- it just yields
/// no matches when nothing survives sanitization to match.
#[tokio::test]
async fn fts5_contact_search_matches_email_and_survives_edge_query() {
let (db, _dir) = DatabaseEngine::connect_ephemeral().await.unwrap();
let search = SearchEngine::new(&db);
insert_contact(
&db,
"ct-email-1",
"Erin Example",
"",
r#"[{"email":"erin.example@nuncio.mx","label":"work"}]"#,
)
.await;
let hits = search.search_contacts("erin.example").await.unwrap();
assert_eq!(hits.len(), 1);
assert_eq!(hits[0].id, "ct-email-1");
let edge_hits = search.search_contacts("\"weird*: query\"").await.unwrap();
assert!(edge_hits.is_empty());
}
/// Keyset pagination over contact search must return every matching hit exactly once,
/// mirroring the message-search pagination guarantee.
#[tokio::test]
async fn search_contacts_page_returns_every_hit_once_no_gaps() {
let (db, _dir) = DatabaseEngine::connect_ephemeral().await.unwrap();
for i in 0..5 {
insert_contact(
&db,
&format!("ct-page-{i}"),
"Planning Committee Member",
"Acme Corp",
"[]",
)
.await;
}
let search = SearchEngine::new(&db);
let mut seen = Vec::new();
let mut after: Option<(f64, String)> = None;
let mut pages = 0;
loop {
let (hits, next) = search
.search_contacts_page("Planning", after.clone(), 2)
.await
.unwrap();
pages += 1;
assert!(hits.len() <= 2, "page must not exceed page_size");
seen.extend(hits.iter().map(|h| h.id.clone()));
match next {
Some(cursor) => after = Some(cursor),
None => break,
}
assert!(pages < 100, "pagination must terminate");
}
seen.sort();
seen.dedup();
assert_eq!(
seen.len(),
5,
"every matching contact returned exactly once"
);
assert!(pages >= 3, "5 hits at page_size 2 must span multiple pages");
}
#[test]
fn fts5_query_sanitizer_removes_special_operators() {
assert_eq!(SearchEngine::sanitize_fts5_query(""), "");
crates/nuncio-store/src/test_tracing.rs
+143
View File
@@ -0,0 +1,143 @@
//! Shared in-test `tracing` capture helper.
//!
//! Captures the process's formatted `tracing` output (the same text a real
//! log line would carry, level tag and all) into an in-memory buffer, so
//! tests can assert on substrings such as a table/column name or an `"INFO"`/
//! `"WARN"` level tag. Fully offline: no I/O beyond the in-memory buffer.
//!
//! The subscriber is installed **once, globally**, and each thread's output is
//! routed to that thread's own buffer. That split is deliberate and
//! load-bearing: a thread-local subscriber cannot work here, because
//! `tracing` caches each callsite's `Interest` process-wide on first use. See
//! [`install_global_capture`].
#![cfg(test)]
use std::cell::RefCell;
use std::io;
use std::sync::{Arc, Mutex, OnceLock};
use tracing_subscriber::fmt::MakeWriter;
thread_local! {
/// The buffer collecting formatted log output for the test running on
/// THIS thread, or `None` on any thread not currently inside
/// [`capture_logs`].
///
/// Routing per thread is what keeps the single global subscriber from
/// letting concurrent tests read each other's telemetry.
static ACTIVE_BUFFER: RefCell<Option<Arc<Mutex<Vec<u8>>>>> = const { RefCell::new(None) };
}
/// A `Write`/`MakeWriter` that appends into whichever buffer is active for
/// the calling thread, discarding the write if no test on this thread is
/// currently capturing.
#[derive(Clone, Default)]
struct ThreadRoutedWriter;
impl io::Write for ThreadRoutedWriter {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
ACTIVE_BUFFER.with(|slot| {
if let Some(buffer) = slot.borrow().as_ref() {
buffer
.lock()
.unwrap_or_else(|e| e.into_inner())
.extend_from_slice(buf);
}
});
Ok(buf.len())
}
fn flush(&mut self) -> io::Result<()> {
Ok(())
}
}
impl<'a> MakeWriter<'a> for ThreadRoutedWriter {
type Writer = ThreadRoutedWriter;
fn make_writer(&'a self) -> Self::Writer {
ThreadRoutedWriter
}
}
/// Install the capturing subscriber as the process-wide default, exactly once.
///
/// It MUST be global rather than thread-local, and this is the whole reason the
/// helper is shaped this way. `tracing` resolves each callsite's `Interest`
/// **once for the process** and caches it; a callsite first executed on a thread
/// with no subscriber caches `Interest::never()` and is then skipped forever --
/// before any later thread-local subscriber is ever consulted. Under a parallel
/// test harness, whichever thread reaches a callsite first therefore decides
/// whether any other test can ever capture it, which made every log-capture
/// assertion in this crate a coin flip.
///
/// A single global subscriber removes the race: every callsite registers against
/// a real subscriber, so interest is always enabled, and per-thread routing
/// (`ACTIVE_BUFFER`) decides what is actually recorded.
fn install_global_capture() {
static INSTALLED: OnceLock<()> = OnceLock::new();
INSTALLED.get_or_init(|| {
let subscriber = tracing_subscriber::fmt()
.with_writer(ThreadRoutedWriter)
.with_ansi(false)
.finish();
// A global default may already be set by another component of the same
// test binary; capturing still works through the thread-local routing
// above, so a lost race here is not fatal.
let _ = tracing::subscriber::set_global_default(subscriber);
// Callsites executed before this point resolved their interest against
// the no-op dispatcher and cached "never". Recompute so they can be
// captured from here on. Done once, at install -- not per test.
tracing::callsite::rebuild_interest_cache();
});
}
/// Resets this thread's buffer slot on drop, so a panicking test cannot leak
/// its buffer into whatever the harness runs next on the same thread.
struct BufferGuard;
impl Drop for BufferGuard {
fn drop(&mut self) {
ACTIVE_BUFFER.with(|slot| *slot.borrow_mut() = None);
}
}
/// A capture in progress on the current thread. Held across the code under
/// test -- including any `.await` points, as long as the test stays on a
/// single OS thread (e.g. the default current-thread `#[tokio::test]`
/// runtime) -- then queried for the formatted text captured so far.
pub(crate) struct CapturedLogs {
buffer: Arc<Mutex<Vec<u8>>>,
_guard: BufferGuard,
}
impl CapturedLogs {
/// A snapshot of everything captured on this thread so far, as the
/// formatted text a real log line would carry (level tag included).
pub fn text(&self) -> String {
let bytes = self
.buffer
.lock()
.unwrap_or_else(|e| e.into_inner())
.clone();
String::from_utf8(bytes)
.unwrap_or_else(|e| String::from_utf8_lossy(&e.into_bytes()).into_owned())
}
}
/// Start capturing this thread's formatted `tracing` output.
///
/// Only this thread's output is captured, so any async work driven while the
/// returned [`CapturedLogs`] is alive must run on the calling thread (e.g. a
/// current-thread runtime's `block_on`) to be captured -- the subscriber
/// itself is global, but the destination buffer is per-thread; see
/// [`install_global_capture`] for why that split is load-bearing.
pub(crate) fn capture_logs() -> CapturedLogs {
install_global_capture();
let buffer = Arc::new(Mutex::new(Vec::new()));
ACTIVE_BUFFER.with(|slot| *slot.borrow_mut() = Some(buffer.clone()));
CapturedLogs {
buffer,
_guard: BufferGuard,
}
}
crates/nuncio-store/src/vault.rs
+226
View File
@@ -1,6 +1,7 @@
use aes_gcm::aead::{rand_core::RngCore, OsRng};
use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use std::time::Duration;
use thiserror::Error;
use zeroize::{Zeroize, ZeroizeOnDrop};
@@ -27,6 +28,14 @@ pub const LEDGER_KEY_ACCOUNT: &str = "ledger-hmac-key";
/// and hex-encoded for use as an `authorization: Bearer <token>` header value.
pub const GRPC_TOKEN_ACCOUNT: &str = "grpc-bearer-token";
/// Keyring account name for the HMAC-SHA256 key that signs outbound filter
/// `CALL WEBHOOK` payloads (the `X-Nuncio-Signature` header produced by
/// `nuncio_filter::WebhookDispatcher`). Minted on first use via
/// [`SecretManager::get_or_create_key_bytes`] and hex-encoded; a single
/// daemon-wide signing key lets a webhook receiver verify that a delivery
/// genuinely originated from this daemon.
pub const WEBHOOK_SIGNING_KEY_ACCOUNT: &str = "webhook-signing-key";
/// Zeroized sensitive string wrapper ensuring heap bytes are wiped on drop.
#[derive(Debug, Clone, PartialEq, Eq, Zeroize, ZeroizeOnDrop)]
pub struct ZeroizingSecret(String);
@@ -223,6 +232,9 @@ impl SecretManager {
let mut bytes = vec![0u8; len];
OsRng.fill_bytes(&mut bytes);
self.set_secret(account, &hex::encode(&bytes))?;
// Lifecycle visibility only: which purpose got a key minted, and how long it
// is, never the key material itself.
tracing::info!(purpose = account, len, "minted key material on first use");
Ok(bytes)
}
Err(e) => Err(e),
@@ -230,6 +242,87 @@ impl SecretManager {
}
}
/// Upper bound on how long a keyring read may block before the caller gives up
/// and reports an actionable error.
///
/// Generous enough that a human at a logged-in desktop can answer a keychain
/// consent prompt, short enough that an unattended process always terminates
/// instead of stalling forever.
pub const KEYRING_READ_TIMEOUT: Duration = Duration::from_secs(20);
/// Why a bounded key-material read did not produce a key.
///
/// No variant ever carries key material: the only value interpolated into these
/// messages is the vault *account name*, which names a purpose (for example
/// `grpc-bearer-token`), never a secret.
#[derive(Debug, Error)]
pub enum BoundedVaultError {
/// The underlying vault read completed but failed.
#[error(transparent)]
Vault(#[from] VaultError),
/// The read did not finish within the allotted bound.
#[error(
"timed out after {}s reading key material '{account}' from the OS keyring: the keyring \
did not answer, which usually means it is locked (screen-lock or a never-unlocked login \
session) or is waiting on an access-consent prompt that no one can answer in this \
session (a keychain whose ACL no longer trusts this binary -- after a restore, \
migration, or rebuild -- prompts every time)",
.timeout.as_secs()
)]
TimedOut {
/// Vault account name whose read timed out.
account: String,
/// Bound that elapsed.
timeout: Duration,
},
/// The reader ended without reporting either success or failure.
#[error("the '{account}' keyring read ended without producing a result")]
Lost {
/// Vault account name whose read was lost.
account: String,
},
}
/// [`SecretManager::get_or_create_key_bytes`], bounded so it always terminates.
///
/// The read runs on a detached OS thread rather than inline, because a platform
/// keyring read is an uncancellable blocking call: a locked keychain, or one
/// whose ACL no longer trusts the calling binary, blocks on a consent prompt
/// that never arrives in a headless or screen-locked session. Bounding it
/// therefore means abandoning the thread, not cancelling the call.
///
/// A plain `std::thread::spawn` is used instead of `spawn_blocking` precisely
/// because Tokio joins its blocking pool during runtime shutdown, which would
/// reintroduce the same hang at process exit; a detached thread is reaped when
/// the process ends.
pub async fn get_or_create_key_bytes_bounded(
secrets: Arc<SecretManager>,
account: &str,
len: usize,
timeout: Duration,
) -> Result<Vec<u8>, BoundedVaultError> {
let owned_account = account.to_string();
let thread_account = owned_account.clone();
let (tx, rx) = tokio::sync::oneshot::channel();
std::thread::spawn(move || {
let resolved = secrets.get_or_create_key_bytes(&thread_account, len);
// A closed receiver means the bound already elapsed; nothing to do.
let _ = tx.send(resolved);
});
match tokio::time::timeout(timeout, rx).await {
Ok(Ok(Ok(bytes))) => Ok(bytes),
Ok(Ok(Err(e))) => Err(BoundedVaultError::Vault(e)),
Ok(Err(_)) => Err(BoundedVaultError::Lost {
account: owned_account,
}),
Err(_) => Err(BoundedVaultError::TimedOut {
account: owned_account,
timeout,
}),
}
}
#[cfg(test)]
mod tests {
use super::*;
@@ -352,6 +445,26 @@ mod tests {
assert!(matches!(err, VaultError::StorageFailed(_)));
}
#[test]
fn get_or_create_key_bytes_logs_minting_without_leaking_the_key() {
let manager = SecretManager::mock();
let logs = crate::test_tracing::capture_logs();
let key = manager
.get_or_create_key_bytes(WEBHOOK_SIGNING_KEY_ACCOUNT, 32)
.expect("key provisioned from vault");
let captured = logs.text();
assert!(
captured.contains(WEBHOOK_SIGNING_KEY_ACCOUNT),
"expected the key's purpose to be named in the INFO log, got: {captured}"
);
assert!(captured.to_ascii_uppercase().contains("INFO"));
// The minted key bytes must never appear in the log, in any encoding.
let hex_key = hex::encode(&key);
assert!(!captured.contains(&hex_key));
}
#[test]
fn os_keyring_provider_can_be_constructed_for_production_use() {
// Construction alone must not touch the OS credential store (lazy `Entry` creation
@@ -366,4 +479,117 @@ mod tests {
// headless CI never depends on a real credential store being available.
let _ = manager;
}
/// Vault provider whose reads never return, standing in for a locked
/// keychain or one waiting on an access-consent prompt nobody can answer.
struct BlockingVault;
impl SecretVault for BlockingVault {
fn get_secret(&self, _key: &str) -> Result<String, VaultError> {
loop {
std::thread::park();
}
}
fn set_secret(&self, _key: &str, _secret: &str) -> Result<(), VaultError> {
Ok(())
}
fn delete_secret(&self, _key: &str) -> Result<(), VaultError> {
Ok(())
}
}
#[tokio::test]
async fn bounded_read_gives_up_on_a_keyring_that_never_answers() {
let secrets = Arc::new(SecretManager::new(BlockingVault));
let started = std::time::Instant::now();
let err = get_or_create_key_bytes_bounded(
secrets,
GRPC_TOKEN_ACCOUNT,
32,
Duration::from_millis(150),
)
.await
.expect_err("a read that never returns must not succeed");
assert!(
matches!(err, BoundedVaultError::TimedOut { .. }),
"expected a timeout, got: {err}"
);
assert!(
started.elapsed() < Duration::from_secs(5),
"the bound must actually terminate the read"
);
let message = err.to_string();
assert!(
message.contains(GRPC_TOKEN_ACCOUNT),
"the error must name which key material could not be read: {message}"
);
assert!(
message.contains("locked") && message.contains("consent"),
"the error must suggest the likely cause: {message}"
);
}
#[tokio::test]
async fn bounded_read_mints_and_reloads_key_material_like_the_unbounded_one() {
let secrets = Arc::new(SecretManager::mock());
let minted = get_or_create_key_bytes_bounded(
Arc::clone(&secrets),
GRPC_TOKEN_ACCOUNT,
32,
KEYRING_READ_TIMEOUT,
)
.await
.expect("mock vault mints key material");
assert_eq!(minted.len(), 32);
let reloaded = get_or_create_key_bytes_bounded(
Arc::clone(&secrets),
GRPC_TOKEN_ACCOUNT,
32,
KEYRING_READ_TIMEOUT,
)
.await
.expect("mock vault reloads the same key material");
assert_eq!(minted, reloaded, "a second read must return the same key");
assert_eq!(
reloaded,
secrets
.get_or_create_key_bytes(GRPC_TOKEN_ACCOUNT, 32)
.expect("unbounded read agrees"),
"bounding a read must not change what it returns"
);
}
#[tokio::test]
async fn bounded_read_surfaces_a_vault_failure_verbatim() {
struct FailingVault;
impl SecretVault for FailingVault {
fn get_secret(&self, _key: &str) -> Result<String, VaultError> {
Err(VaultError::StorageFailed("keyring is locked".into()))
}
fn set_secret(&self, _key: &str, _secret: &str) -> Result<(), VaultError> {
Ok(())
}
fn delete_secret(&self, _key: &str) -> Result<(), VaultError> {
Ok(())
}
}
let err = get_or_create_key_bytes_bounded(
Arc::new(SecretManager::new(FailingVault)),
WEBHOOK_SIGNING_KEY_ACCOUNT,
32,
KEYRING_READ_TIMEOUT,
)
.await
.expect_err("a failing vault must fail closed");
assert!(
matches!(err, BoundedVaultError::Vault(_)),
"a real vault failure must not be reported as a timeout: {err}"
);
}
}
crates/nunciod/Cargo.toml
+9 -1
View File
@@ -26,15 +26,23 @@ serde.workspace = true
serde_json.workspace = true
thiserror.workspace = true
tracing.workspace = true
tracing-subscriber = "0.3"
tracing-subscriber.workspace = true
tracing-appender.workspace = true
tempfile.workspace = true
tonic.workspace = true
tower.workspace = true
http.workspace = true
http-body.workspace = true
async-trait.workspace = true
subtle.workspace = true
hex.workspace = true
bytes.workspace = true
zeroize.workspace = true
fs4.workspace = true
[dev-dependencies]
nuncio-cli = { path = "../nuncio-cli" }
sqlx.workspace = true
wiremock.workspace = true
http-body-util.workspace = true
tokio = { workspace = true, features = ["test-util"] }
crates/nunciod/src/calendar_sync.rs
+107 -11
View File
@@ -4,17 +4,16 @@
//! fetches events from a [`CalendarBackend`] for a single calendar collection and
//! time window, then persists each event via [`DatabaseEngine::save_calendar_event`].
//!
//! # No production entry point yet
//! # Production entry point
//!
//! Unlike `sync.rs`'s `run_account_sync`/`run_all_accounts_sync`, this module
//! deliberately has NO production entry point that resolves a real account's
//! CalDAV collection URL and credential and builds a real `CalDavClient` from
//! them: `nuncio_core::AccountConfig` carries no CalDAV fields today, so there
//! is nothing persisted to build one from. [`sync_with_backend`] is the only
//! function here; it is exercised directly in tests with `MockCalendarBackend`,
//! and it is the same seam a future story's production entry point will call
//! once per-account CalDAV configuration exists.
use nuncio_cal::CalendarBackend;
//! [`sync_caldav_account`] is the production entry point: it builds a real
//! [`nuncio_cal::CalDavClient`] from a persisted [`nuncio_core::AccountConfig`]
//! (its `collection_url`) plus the credential resolved from the OS keyring,
//! then delegates to [`sync_with_backend`]. [`sync_with_backend`] is the
//! shared fetch-and-persist seam that both the production client and the
//! test-only `MockCalendarBackend` flow through, mirroring `sync.rs`'s
//! `fetch_and_persist`.
use nuncio_cal::{CalDavAccountConfig, CalDavClient, CalendarBackend};
use nuncio_store::db::DatabaseError;
use thiserror::Error;
@@ -44,17 +43,68 @@ pub async fn sync_with_backend(
start_window: i64,
end_window: i64,
) -> Result<usize, CalendarSyncError> {
tracing::debug!(
calendar_id = %calendar_id,
start_window,
end_window,
"calendar sync: fetching events from backend"
);
let events = backend
.fetch_events(calendar_id, start_window, end_window)
.await?;
.await
.inspect_err(|e| {
tracing::warn!(
calendar_id = %calendar_id,
"calendar sync: backend fetch failed: {e}"
);
})?;
let fetched = events.len();
let mut synced = 0usize;
for event in events {
db.save_calendar_event(&event).await?;
synced += 1;
}
tracing::info!(
calendar_id = %calendar_id,
fetched,
synced,
"calendar sync: persisted fetched events"
);
Ok(synced)
}
/// Production entry point: build a real [`CalDavClient`] for `account` from its
/// persisted `collection_url` and the `password` resolved from the OS keyring
/// vault, then fetch-and-persist `calendar_id`'s events in `[start_window,
/// end_window]` via [`sync_with_backend`]. Returns the number of events synced.
///
/// `account.collection_url` must be a fully-qualified CalDAV calendar
/// collection URL (validated at `AddAccount` time); the account's
/// `email_address` is used as the DAV basic-auth username. No fabricated data
/// is ever returned -- a transport or persistence failure surfaces as a
/// [`CalendarSyncError`].
pub async fn sync_caldav_account(
db: &nuncio_store::db::DatabaseEngine,
account: &nuncio_core::AccountConfig,
password: &str,
calendar_id: &str,
start_window: i64,
end_window: i64,
) -> Result<usize, CalendarSyncError> {
tracing::info!(
account_id = %account.id,
calendar_id = %calendar_id,
"calendar sync: dispatching CalDAV sync for account"
);
let client = CalDavClient::new(CalDavAccountConfig {
account_id: account.id.clone(),
caldav_url: account.dav_collection_url().unwrap_or_default().to_string(),
username: account.email_address.clone(),
auth_token: nuncio_core::redact::Redacted::new(password.to_string()),
});
sync_with_backend(db, &client, calendar_id, start_window, end_window).await
}
#[cfg(test)]
mod tests {
use super::*;
@@ -144,4 +194,50 @@ mod tests {
assert!(matches!(err, CalendarSyncError::Backend(_)));
assert!(err.to_string().contains("calendar backend error"));
}
/// A calendar sync must emit a followable INFO event carrying the calendar
/// id and the persisted count, so an operator can trace what a sync did.
#[test]
fn sync_with_backend_logs_calendar_id_and_synced_count() {
use crate::test_tracing::with_recorder;
use tracing::Level;
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("runtime");
let (recorder, ()) = with_recorder(|| {
runtime.block_on(async {
let (db, _dir) = DatabaseEngine::connect_ephemeral()
.await
.expect("ephemeral db");
let mock = MockCalendarBackend::new();
mock.add_event(mock_event(
"evt-1",
"cal-work",
1_700_000_000,
1_700_003_600,
));
sync_with_backend(&db, &mock, "cal-work", 0, i64::MAX)
.await
.expect("sync succeeds");
});
});
let events = recorder.events();
let persisted = events
.iter()
.find(|e| e.message() == "calendar sync: persisted fetched events")
.expect("a domain event for the persisted sync must be logged");
assert_eq!(persisted.level, Level::INFO);
assert_eq!(
persisted.fields.get("calendar_id").map(String::as_str),
Some("cal-work")
);
assert_eq!(
persisted.fields.get("synced").map(String::as_str),
Some("1")
);
}
}
crates/nunciod/src/contacts_sync.rs
+92 -13
View File
@@ -5,18 +5,12 @@
//! single account, then persists each contact via
//! [`DatabaseEngine::save_contact`].
//!
//! # No production entry point yet
//!
//! Unlike `sync.rs`'s `run_account_sync`/`run_all_accounts_sync`, this module
//! deliberately has NO production entry point that resolves a real account's
//! CardDAV address book collection URL and credential and builds a real
//! `CardDavClient` from them: `nuncio_core::AccountConfig` carries no CardDAV
//! fields today, so there is nothing persisted to build one from.
//! [`sync_with_backend`] is the only function here; it is exercised directly
//! in tests with `MockContactsBackend`, and it is the same seam a future
//! story's production entry point will call once per-account CardDAV
//! configuration exists.
use nuncio_contacts::ContactsBackend;
//! [`sync_carddav_account`] is the production entry point: it resolves a real
//! account's persisted CardDAV address-book collection URL, builds a real
//! [`CardDavClient`] from it plus the credential the caller read from the OS
//! keyring, and funnels through the same [`sync_with_backend`] seam that tests
//! drive with `MockContactsBackend`.
use nuncio_contacts::{CardDavAccountConfig, CardDavClient, ContactsBackend};
use nuncio_store::db::DatabaseError;
use thiserror::Error;
@@ -43,15 +37,59 @@ pub async fn sync_with_backend(
backend: &dyn ContactsBackend,
account_id: &str,
) -> Result<usize, ContactsSyncError> {
let contacts = backend.fetch_contacts(account_id).await?;
tracing::info!(
account_id = %account_id,
"contacts sync: dispatching sync for account"
);
let contacts = backend.fetch_contacts(account_id).await.inspect_err(|e| {
tracing::warn!(
account_id = %account_id,
"contacts sync: backend fetch failed: {e}"
);
})?;
let fetched = contacts.len();
let mut synced = 0usize;
for contact in contacts {
db.save_contact(&contact).await?;
synced += 1;
}
tracing::info!(
account_id = %account_id,
fetched,
synced,
"contacts sync: persisted fetched contacts"
);
Ok(synced)
}
/// Production entry point: build a real [`CardDavClient`] for `account` from
/// its persisted `collection_url` and the `password` resolved from the OS
/// keyring vault, then fetch-and-persist that account's contacts via
/// [`sync_with_backend`]. Returns the number of contacts synced.
///
/// `account`'s collection URL must be a fully-qualified CardDAV address-book
/// collection URL (validated at `AddAccount` time); the account's
/// `email_address` is used as the DAV basic-auth username. No fabricated data
/// is ever returned -- a transport or persistence failure surfaces as a
/// [`ContactsSyncError`].
pub async fn sync_carddav_account(
db: &nuncio_store::db::DatabaseEngine,
account: &nuncio_core::AccountConfig,
password: &str,
) -> Result<usize, ContactsSyncError> {
tracing::info!(
account_id = %account.id,
"contacts sync: dispatching CardDAV sync for account"
);
let client = CardDavClient::new(CardDavAccountConfig {
account_id: account.id.clone(),
carddav_url: account.dav_collection_url().unwrap_or_default().to_string(),
username: account.email_address.clone(),
auth_token: nuncio_core::redact::Redacted::new(password.to_string()),
});
sync_with_backend(db, &client, &account.id).await
}
#[cfg(test)]
mod tests {
use super::*;
@@ -120,4 +158,45 @@ mod tests {
assert!(matches!(err, ContactsSyncError::Backend(_)));
assert!(err.to_string().contains("contacts backend error"));
}
/// A contacts sync must emit a followable INFO event carrying the account
/// id and the persisted count.
#[test]
fn sync_with_backend_logs_account_id_and_synced_count() {
use crate::test_tracing::with_recorder;
use tracing::Level;
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("runtime");
let (recorder, ()) = with_recorder(|| {
runtime.block_on(async {
let (db, _dir) = DatabaseEngine::connect_ephemeral()
.await
.expect("ephemeral db");
let mock = MockContactsBackend::new();
mock.add_contact(mock_contact("ct-1", "acct-contacts-1", "Alice"));
sync_with_backend(&db, &mock, "acct-contacts-1")
.await
.expect("sync succeeds");
});
});
let events = recorder.events();
let persisted = events
.iter()
.find(|e| e.message() == "contacts sync: persisted fetched contacts")
.expect("a domain event for the persisted sync must be logged");
assert_eq!(persisted.level, Level::INFO);
assert_eq!(
persisted.fields.get("account_id").map(String::as_str),
Some("acct-contacts-1")
);
assert_eq!(
persisted.fields.get("synced").map(String::as_str),
Some("1")
);
}
}
crates/nunciod/src/grpc.rs
+3504 -567
View File
File diff suppressed because it is too large. Load diff
crates/nunciod/src/lib.rs
+63
View File
@@ -5,9 +5,19 @@
pub mod calendar_sync;
pub mod contacts_sync;
pub mod grpc;
pub mod lifecycle;
pub mod lock;
pub mod logging;
pub mod orchestrator;
pub mod outbox;
pub mod pagination;
pub mod rpc_trace;
pub mod secrets;
pub mod send;
pub mod sync;
pub mod sync_dispatcher;
#[cfg(test)]
pub(crate) mod test_tracing;
pub use orchestrator::SelfHealingSyncOrchestrator;
/// Default PERSISTENT database path for `nunciod`: `~/.nuncio/nuncio.db`
@@ -34,6 +44,27 @@ pub fn default_db_path() -> std::path::PathBuf {
}
}
/// Loads the persisted filter rule set at daemon boot.
///
/// A load failure here is a genuine database problem, not "no rules are
/// configured": an empty database legitimately has zero rules, but a query
/// error does not mean zero rules, it means the true rule set is unknown.
/// Collapsing that error into an empty `Vec` would silently disable all
/// filtering with nothing in the logs to explain why. This surfaces the
/// failure loudly and propagates it instead, so daemon startup fails
/// visibly rather than starting with an unknown -- possibly wide open --
/// filtering posture.
pub async fn load_initial_filter_rules(
db: &nuncio_store::DatabaseEngine,
) -> Result<Vec<nuncio_filter::FilterRule>, nuncio_store::DatabaseError> {
db.list_filter_rules().await.inspect_err(|err| {
tracing::error!(
error = %err,
"failed to load filter rules from database at startup; refusing to start with an unknown rule set"
);
})
}
/// Environment variable that opts `nunciod` in to its autonomous
/// background auto-update-check loop.
///
@@ -91,6 +122,38 @@ mod tests {
assert!(!auto_update_task_enabled());
}
#[tokio::test]
async fn load_initial_filter_rules_propagates_db_errors_instead_of_returning_empty() {
let (engine, _dir) = nuncio_store::DatabaseEngine::connect_ephemeral()
.await
.unwrap();
// A pool that has been closed makes every subsequent query fail
// deterministically, standing in for a genuine boot-time database
// problem (as opposed to a healthy, merely empty, rule table).
engine.close().await;
let result = load_initial_filter_rules(&engine).await;
assert!(
result.is_err(),
"a genuine database error must propagate as Err, not be silently \
mapped to an empty rule set (which would look identical to a \
database with zero rules configured)"
);
}
#[tokio::test]
async fn load_initial_filter_rules_returns_empty_vec_for_a_healthy_empty_database() {
let (engine, _dir) = nuncio_store::DatabaseEngine::connect_ephemeral()
.await
.unwrap();
let rules = load_initial_filter_rules(&engine)
.await
.expect("a healthy, empty database must load successfully");
assert!(rules.is_empty());
}
#[test]
fn default_db_path_is_persistent_under_dot_nuncio_directory() {
// Deliberately does not mutate `USERPROFILE`/`HOME` (real env vars
crates/nunciod/src/lifecycle.rs
+208
View File
@@ -0,0 +1,208 @@
//! Coordinates graceful shutdown across every long-running task `nunciod`
//! spawns: the gRPC server, the outbox worker, the auto-update-check loop,
//! and the real-sync `CoreCommand` consumer.
//!
//! Previously the daemon installed no signal handler at all: every real
//! exit was an OS kill (SIGKILL/SIGTERM without a handler, a terminated
//! console), which races [`nuncio_store::DatabaseEngine::close`]'s WAL
//! checkpoint and can corrupt the database. A single [`ShutdownSignal`],
//! shared by every task, lets each one stop accepting new work and finish
//! cleanly instead.
use nuncio_core::{CoreCommand, EventBus};
use std::sync::Arc;
use tokio::sync::watch;
/// Read-only handle to the shared shutdown flag. Cloned into every
/// long-running task; `wait` resolves once shutdown has been requested,
/// including if it was already requested before this clone started
/// waiting.
#[derive(Clone)]
pub struct ShutdownSignal {
rx: watch::Receiver<bool>,
// Keeps the watch channel open for signals minted by [`ShutdownSignal::never`],
// which have no live [`ShutdownController`] owning the sender. A closed
// channel makes `wait` resolve (readers interpret that as "shutdown
// requested"), so a `never` signal must hold its sender alive; a real
// signal leaves this `None` because its controller owns the sender.
_keepalive: Option<Arc<watch::Sender<bool>>>,
}
impl ShutdownSignal {
/// A signal that never fires, for call paths with no shutdown wiring of
/// their own (e.g. the non-graceful `serve_on_listener` entry point used
/// by tests). Holds its own sender alive so the channel never closes.
pub fn never() -> Self {
let (tx, rx) = watch::channel(false);
ShutdownSignal {
rx,
_keepalive: Some(Arc::new(tx)),
}
}
/// Resolves once shutdown has been requested. Safe to call from
/// `tokio::select!` on every loop iteration: a task that observes
/// shutdown mid-iteration finishes its current unit of work (the other
/// `select!` branch) before this is polled again, so no in-flight
/// database write is abandoned half-done.
pub async fn wait(&mut self) {
// `watch::Receiver::wait_for` returns immediately if the current
// value already satisfies the predicate, so a clone created after
// shutdown was already requested does not hang waiting for a
// change that already happened.
let _ = self.rx.wait_for(|requested| *requested).await;
}
/// Non-blocking check for loops that need to poll rather than await.
pub fn is_requested(&self) -> bool {
*self.rx.borrow()
}
}
/// Owns the shutdown trigger. Production wires [`ShutdownController::trigger`]
/// to OS signal delivery via [`install_signal_handlers`]; tests call it
/// directly to drive the shutdown path deterministically without sending a
/// real OS signal.
pub struct ShutdownController {
tx: watch::Sender<bool>,
event_bus: Arc<EventBus>,
}
impl ShutdownController {
/// Creates a controller/signal pair. `event_bus` is the SAME bus every
/// other subsystem shares, so [`trigger`](Self::trigger) drives the
/// existing `CoreCommand::Shutdown` / `CoreEvent::ShuttingDown` path
/// visibly (state flips to `ShuttingDown`, subscribers are notified)
/// instead of that path staying reachable only from tests.
pub fn new(event_bus: Arc<EventBus>) -> (Self, ShutdownSignal) {
let (tx, rx) = watch::channel(false);
(
Self { tx, event_bus },
ShutdownSignal {
rx,
_keepalive: None,
},
)
}
/// Initiates shutdown. Idempotent: calling it more than once (e.g. a
/// second Ctrl+C while already draining) is a no-op beyond the first.
pub fn trigger(&self) {
// Drives the event bus to `ShuttingDown` and publishes the
// `CoreEvent::ShuttingDown` event to every subscriber (e.g. clients on
// `System/Subscribe`), so a graceful shutdown is observable over the
// API and not just in local logs.
self.event_bus.process_command(CoreCommand::Shutdown);
tracing::info!("shutdown: ShuttingDown event published to subscribers");
let _ = self.tx.send(true);
}
}
/// Waits for Ctrl+C, or (on Unix) SIGTERM, then triggers shutdown. Intended
/// to be spawned once at startup and left running for the process lifetime.
pub async fn install_signal_handlers(controller: Arc<ShutdownController>) {
wait_for_os_signal().await;
controller.trigger();
}
#[cfg(unix)]
async fn wait_for_os_signal() {
use tokio::signal::unix::{signal, SignalKind};
match signal(SignalKind::terminate()) {
Ok(mut sigterm) => {
tokio::select! {
_ = tokio::signal::ctrl_c() => {
tracing::info!("received Ctrl+C; starting graceful shutdown");
}
_ = sigterm.recv() => {
tracing::info!("received SIGTERM; starting graceful shutdown");
}
}
}
Err(err) => {
// Ctrl+C still works even if SIGTERM registration failed (e.g.
// an exotic sandboxed environment); log so the gap is visible
// rather than silently dropping SIGTERM handling.
tracing::error!("failed to install SIGTERM handler: {err}; SIGTERM will kill the process without a clean shutdown");
let _ = tokio::signal::ctrl_c().await;
tracing::info!("received Ctrl+C; starting graceful shutdown");
}
}
}
#[cfg(not(unix))]
async fn wait_for_os_signal() {
let _ = tokio::signal::ctrl_c().await;
tracing::info!("received Ctrl+C; starting graceful shutdown");
}
#[cfg(test)]
mod tests {
use super::*;
use nuncio_core::EngineStatus;
#[tokio::test]
async fn trigger_flips_signal_and_event_bus_state() {
let event_bus = Arc::new(EventBus::new());
let (controller, mut signal) = ShutdownController::new(event_bus.clone());
assert!(!signal.is_requested());
assert_eq!(event_bus.current_state().status, EngineStatus::Idle);
controller.trigger();
assert!(signal.is_requested());
assert_eq!(event_bus.current_state().status, EngineStatus::ShuttingDown);
// Resolves immediately: the signal was already flipped above.
signal.wait().await;
}
#[tokio::test]
async fn signal_clone_observes_a_trigger_that_happened_before_it_was_created() {
let event_bus = Arc::new(EventBus::new());
let (controller, signal) = ShutdownController::new(event_bus);
controller.trigger();
// A clone taken AFTER the trigger must still resolve `wait`
// immediately rather than hang waiting for a fresh `changed()`.
let mut late_clone = signal.clone();
late_clone.wait().await;
assert!(late_clone.is_requested());
}
/// Triggering shutdown must both publish the `ShuttingDown` event (flipping
/// bus state) and emit a traceable lifecycle log line, so a clean shutdown
/// is observable in the logs and not only over the API.
#[test]
fn trigger_emits_a_traceable_shutting_down_lifecycle_log() {
use crate::test_tracing::with_recorder;
let (recorder, ()) = with_recorder(|| {
let event_bus = Arc::new(EventBus::new());
let (controller, _signal) = ShutdownController::new(event_bus.clone());
controller.trigger();
assert_eq!(event_bus.current_state().status, EngineStatus::ShuttingDown);
});
assert!(
recorder
.events()
.iter()
.any(|e| e.message() == "shutdown: ShuttingDown event published to subscribers"),
"trigger must emit the ShuttingDown lifecycle log"
);
}
#[tokio::test]
async fn trigger_is_idempotent() {
let event_bus = Arc::new(EventBus::new());
let (controller, signal) = ShutdownController::new(event_bus.clone());
controller.trigger();
controller.trigger();
assert!(signal.is_requested());
assert_eq!(event_bus.current_state().status, EngineStatus::ShuttingDown);
}
}
crates/nunciod/src/lock.rs
+180
View File
@@ -0,0 +1,180 @@
//! Exclusive, cross-platform advisory lock on the daemon's database path.
//!
//! Nothing about starting a second `nunciod` process today prevents it from
//! opening the same SQLite database another instance already has open --
//! the only accidental protection is a port-bind collision on the gRPC
//! listener, and that only fires if both instances also happen to pick the
//! same port. Two writers on one SQLite file corrupt state.
//!
//! [`InstanceLock::acquire`] takes an OS-level exclusive lock (`flock` on
//! Unix, `LockFileEx` on Windows via the `fs4` crate) on a sidecar file next
//! to the database, `<db_path>.lock`. The lock is scoped to the OS file
//! handle, not tracked by a PID written into the file, so it cannot go
//! stale: if the holding process dies for any reason (crash, kill -9), the
//! OS releases the lock the moment the file descriptor/handle closes.
use std::fs::{File, OpenOptions};
use std::path::{Path, PathBuf};
use fs4::fs_std::FileExt;
/// Failure to acquire the single-instance lock.
#[derive(Debug, thiserror::Error)]
pub enum InstanceLockError {
/// The lock is already held by another live process. The caller must
/// treat this as fatal and refuse to open the store.
#[error("another nunciod instance is already running for this database at {0}")]
AlreadyRunning(PathBuf),
/// The lock file itself could not be created/opened/locked, e.g. a
/// permissions problem or an unwritable parent directory.
#[error("failed to acquire instance lock at {path}: {source}")]
Io {
path: PathBuf,
#[source]
source: std::io::Error,
},
}
/// Holds the exclusive lock on the database path for as long as this guard
/// is alive. Drop it (or let it fall out of scope at process exit) to
/// release the lock; a later call to [`InstanceLock::acquire`] for the same
/// path then succeeds again.
#[derive(Debug)]
pub struct InstanceLock {
file: File,
path: PathBuf,
}
impl InstanceLock {
/// Acquires the exclusive lock for `db_path`, creating the sidecar lock
/// file (`<db_path>.lock`) if it does not already exist.
///
/// Returns [`InstanceLockError::AlreadyRunning`] if another process
/// already holds it -- callers must fail startup on this rather than
/// proceed to open the store, since a second writer on the same SQLite
/// file is exactly the corruption this guards against.
pub fn acquire(db_path: &Path) -> Result<Self, InstanceLockError> {
let lock_path = lock_path_for(db_path);
if let Some(parent) = lock_path.parent() {
std::fs::create_dir_all(parent).map_err(|source| InstanceLockError::Io {
path: lock_path.clone(),
source,
})?;
}
let file = OpenOptions::new()
.create(true)
.truncate(false)
.write(true)
.open(&lock_path)
.map_err(|source| InstanceLockError::Io {
path: lock_path.clone(),
source,
})?;
let acquired = file
.try_lock_exclusive()
.map_err(|source| InstanceLockError::Io {
path: lock_path.clone(),
source,
})?;
if !acquired {
return Err(InstanceLockError::AlreadyRunning(db_path.to_path_buf()));
}
Ok(Self {
file,
path: lock_path,
})
}
}
impl Drop for InstanceLock {
fn drop(&mut self) {
// The OS releases a `flock`/`LockFileEx` lock automatically when the
// underlying handle closes, so a failure here is not a leaked lock
// (the process exiting, or `self.file` being dropped right after,
// clears it regardless) -- it is only logged so an operator sees
// anything unexpected in the unlock call itself.
if let Err(err) = self.file.unlock() {
tracing::warn!(
path = %self.path.display(),
error = %err,
"failed to explicitly release nunciod instance lock; it will still be \
released when the process exits and the underlying file handle closes"
);
}
}
}
fn lock_path_for(db_path: &Path) -> PathBuf {
let mut file_name = db_path.as_os_str().to_owned();
file_name.push(".lock");
PathBuf::from(file_name)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn second_acquire_for_same_path_fails_while_first_guard_is_held() {
let dir = tempfile::tempdir().unwrap();
let db_path = dir.path().join("nuncio.db");
let first = InstanceLock::acquire(&db_path).unwrap();
let second = InstanceLock::acquire(&db_path);
assert!(
matches!(second, Err(InstanceLockError::AlreadyRunning(ref p)) if p == &db_path),
"expected AlreadyRunning({db_path:?}), got {second:?}"
);
drop(first);
}
#[test]
fn acquire_succeeds_again_after_the_first_guard_is_dropped() {
let dir = tempfile::tempdir().unwrap();
let db_path = dir.path().join("nuncio.db");
let first = InstanceLock::acquire(&db_path).unwrap();
drop(first);
let second = InstanceLock::acquire(&db_path);
assert!(
second.is_ok(),
"acquiring after the prior guard dropped must succeed, got {:?}",
second.err()
);
}
#[test]
fn error_message_names_the_database_path() {
let dir = tempfile::tempdir().unwrap();
let db_path = dir.path().join("nuncio.db");
let first = InstanceLock::acquire(&db_path).unwrap();
let second = InstanceLock::acquire(&db_path).unwrap_err();
let message = second.to_string();
assert!(message.contains("already running"));
assert!(message.contains(&db_path.display().to_string()));
drop(first);
}
#[test]
fn lock_file_is_created_beside_the_database_path() {
let dir = tempfile::tempdir().unwrap();
let db_path = dir.path().join("nuncio.db");
let guard = InstanceLock::acquire(&db_path).unwrap();
assert!(dir.path().join("nuncio.db.lock").exists());
drop(guard);
}
}
crates/nunciod/src/logging.rs
+323
View File
@@ -0,0 +1,323 @@
//! Layered `tracing` subscriber for `nunciod`.
//!
//! This replaces the previous stderr-only `tracing_subscriber::fmt::init()`
//! bootstrap, which pinned the level at INFO, ignored `RUST_LOG` (the
//! `env-filter` feature was not even enabled), had no file sink, no JSON
//! output, and no way to change levels at runtime -- leaving the daemon
//! effectively unfollowable and unauditable.
//!
//! The subscriber is a layered [`Registry`] with three layers:
//!
//! 1. A reloadable [`EnvFilter`] applied as the global filter, resolved from
//! the environment (see [`resolve_filter_directive`]). It is wrapped in a
//! [`reload::Layer`] so a future control RPC can change verbosity at
//! runtime without a restart; the [`reload::Handle`] is returned from
//! [`init`]. A malformed filter string never panics -- it degrades to the
//! default and a `warn!` is emitted once the subscriber is live.
//! 2. A human-readable console layer to stderr (ANSI, target + span context).
//! 3. A rotating daily file layer under `<data_dir>/logs/`, written through a
//! non-blocking background writer.
//!
//! ## Environment variables
//!
//! * `NUNCIO_LOG` -- filter directive; takes precedence over `RUST_LOG`.
//! * `RUST_LOG` -- filter directive; used when `NUNCIO_LOG` is unset/empty.
//! * (default) -- when neither is set, the filter is [`DEFAULT_FILTER`].
//! * `NUNCIO_LOG_FORMAT=json` -- emit the *file* sink as JSON. The console
//! sink always stays human-readable (it is for an operator watching a
//! terminal); the file sink is the one a machine parses, so JSON is applied
//! there. Any other value (or unset) keeps the file sink human-readable.
//!
//! ## Worker guard lifetime (CRITICAL)
//!
//! The rotating file layer uses [`tracing_appender::non_blocking`], which
//! hands back a [`WorkerGuard`]. That guard MUST be held for the entire
//! process lifetime: when it drops, the background writer thread is joined and
//! any buffered-but-unflushed log lines are discarded. [`init`] therefore
//! returns it inside [`LoggingGuards`], and `main` binds that value (e.g.
//! `let _guards = ...;`) so it lives until the daemon exits. Dropping it early
//! silently loses file logs.
//!
//! ## Failure handling
//!
//! Logging must never crash the daemon. If the log directory cannot be
//! created, the file layer is skipped and the daemon logs to stderr only
//! (with a `warn!`). If a global subscriber is already installed (e.g. a test
//! harness set one), [`init`] leaves it in place and returns empty guards.
use std::path::{Path, PathBuf};
use tracing_appender::non_blocking::WorkerGuard;
use tracing_subscriber::layer::{Layered, SubscriberExt};
use tracing_subscriber::util::SubscriberInitExt;
use tracing_subscriber::{reload, EnvFilter, Layer, Registry};
/// Filter directive used when neither `NUNCIO_LOG` nor `RUST_LOG` is set.
pub const DEFAULT_FILTER: &str = "info";
/// Environment variable that overrides the log filter, taking precedence over
/// `RUST_LOG`.
pub const LOG_FILTER_ENV_VAR: &str = "NUNCIO_LOG";
/// Environment variable that, when equal to `"json"` (case-insensitive),
/// switches the file sink to JSON output.
pub const LOG_FORMAT_ENV_VAR: &str = "NUNCIO_LOG_FORMAT";
/// The output encoding of a log sink.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum LogFormat {
/// Human-readable, single-line-per-event text.
Human,
/// One JSON object per event, for machine ingestion.
Json,
}
/// The base registry with the reloadable [`EnvFilter`] applied. The concrete
/// fmt layers are boxed against this type so the JSON and human variants (and
/// the presence/absence of the file layer) can be chosen at runtime without
/// leaking their differing types into the subscriber's type signature.
type FilteredRegistry = Layered<reload::Layer<EnvFilter, Registry>, Registry>;
/// A type-erased layer compatible with the filtered registry.
type DynLayer = Box<dyn Layer<FilteredRegistry> + Send + Sync>;
/// Handle that can swap the global [`EnvFilter`] at runtime. Exposed so a
/// future control RPC can raise or lower verbosity without restarting the
/// daemon.
pub type FilterReloadHandle = reload::Handle<EnvFilter, Registry>;
/// Guards and handles that must outlive the process.
///
/// Hold this for the whole daemon lifetime. See the module docs on the
/// [`WorkerGuard`] lifetime requirement.
pub struct LoggingGuards {
reload_handle: Option<FilterReloadHandle>,
// Kept alive purely for its `Drop`: dropping it flushes and shuts down the
// non-blocking file writer. Never read directly.
_file_guard: Option<WorkerGuard>,
}
impl LoggingGuards {
/// Returns the runtime filter-reload handle, if the subscriber this crate
/// installed is the active one. `None` means a subscriber was already set
/// by someone else (e.g. a test harness) and this handle would not affect
/// it.
pub fn reload_handle(&self) -> Option<&FilterReloadHandle> {
self.reload_handle.as_ref()
}
}
/// Resolves the effective filter directive with precedence
/// `NUNCIO_LOG` > `RUST_LOG` > [`DEFAULT_FILTER`].
///
/// An env var set to an empty/whitespace-only string is treated as unset so a
/// stray `NUNCIO_LOG=` does not silence all output.
pub fn resolve_filter_directive() -> String {
env_directive(LOG_FILTER_ENV_VAR)
.or_else(|| env_directive("RUST_LOG"))
.unwrap_or_else(|| DEFAULT_FILTER.to_string())
}
fn env_directive(var: &str) -> Option<String> {
match std::env::var(var) {
Ok(value) if !value.trim().is_empty() => Some(value),
_ => None,
}
}
/// Builds an [`EnvFilter`] from `directive`, degrading to [`DEFAULT_FILTER`]
/// on a parse error instead of panicking.
///
/// The second element is `Some(directive)` when the directive was malformed
/// and the default was substituted, so the caller can emit a `warn!` once the
/// subscriber is live (warning before init would go nowhere).
pub fn build_filter(directive: &str) -> (EnvFilter, Option<String>) {
match EnvFilter::try_new(directive) {
Ok(filter) => (filter, None),
Err(_) => (EnvFilter::new(DEFAULT_FILTER), Some(directive.to_string())),
}
}
/// Reads the desired file-sink format from [`LOG_FORMAT_ENV_VAR`].
pub fn log_format_from_env() -> LogFormat {
match std::env::var(LOG_FORMAT_ENV_VAR) {
Ok(value) if value.trim().eq_ignore_ascii_case("json") => LogFormat::Json,
_ => LogFormat::Human,
}
}
/// Derives the log directory (`<data_dir>/logs/`) from the database path,
/// whose parent is the daemon's data directory.
pub fn log_dir_for_db_path(db_path: &Path) -> PathBuf {
match db_path.parent() {
Some(dir) if !dir.as_os_str().is_empty() => dir.join("logs"),
_ => PathBuf::from("logs"),
}
}
/// Installs the global layered subscriber and returns the guards that must be
/// held for the process lifetime.
///
/// `db_path` is the resolved database path; logs are written to a sibling
/// `logs/` directory. This never panics and never returns an error: any
/// problem (unwritable log directory, an already-installed subscriber) is
/// degraded to a lesser-but-working configuration and reported via `warn!`.
pub fn init(db_path: &Path) -> LoggingGuards {
let directive = resolve_filter_directive();
let (filter, malformed) = build_filter(&directive);
let (filter_layer, reload_handle) = reload::Layer::new(filter);
let format = log_format_from_env();
let log_dir = log_dir_for_db_path(db_path);
// Try to stand up the rotating file sink. A failure to create the log
// directory must not stop the daemon; it just falls back to stderr-only.
let (file_layer, file_guard, file_error) = match std::fs::create_dir_all(&log_dir) {
Ok(()) => {
let appender = tracing_appender::rolling::daily(&log_dir, "nunciod.log");
let (writer, guard) = tracing_appender::non_blocking(appender);
let layer: DynLayer = match format {
LogFormat::Json => tracing_subscriber::fmt::layer()
.json()
.with_writer(writer)
.with_ansi(false)
.boxed(),
LogFormat::Human => tracing_subscriber::fmt::layer()
.with_writer(writer)
.with_ansi(false)
.boxed(),
};
(Some(layer), Some(guard), None)
}
Err(err) => (None, None, Some(format!("{}: {err}", log_dir.display()))),
};
// Human-readable console sink for an operator watching the terminal. This
// stays text even when the file sink is JSON.
let console_layer: DynLayer = tracing_subscriber::fmt::layer()
.with_writer(std::io::stderr)
.with_ansi(true)
.with_target(true)
.boxed();
let mut layers: Vec<DynLayer> = vec![console_layer];
if let Some(layer) = file_layer {
layers.push(layer);
}
let subscriber = tracing_subscriber::registry()
.with(filter_layer)
.with(layers);
if subscriber.try_init().is_err() {
// A subscriber is already installed (commonly a test harness). Leave
// it untouched and hand back empty guards; our reload handle would not
// apply to it, so do not expose it.
return LoggingGuards {
reload_handle: None,
_file_guard: None,
};
}
if let Some(bad) = malformed {
tracing::warn!(
directive = %bad,
default = %DEFAULT_FILTER,
"ignoring malformed log filter directive; using default"
);
}
if let Some(err) = file_error {
tracing::warn!(error = %err, "could not open log directory; logging to stderr only");
}
LoggingGuards {
reload_handle: Some(reload_handle),
_file_guard: file_guard,
}
}
#[cfg(test)]
mod tests {
use super::*;
// The process environment is global mutable state; serialize every test
// that reads or writes the log env vars so they cannot interleave.
static ENV_GUARD: std::sync::Mutex<()> = std::sync::Mutex::new(());
fn clear_env() {
std::env::remove_var(LOG_FILTER_ENV_VAR);
std::env::remove_var("RUST_LOG");
std::env::remove_var(LOG_FORMAT_ENV_VAR);
}
#[test]
fn nuncio_log_beats_rust_log_beats_default() {
let _guard = ENV_GUARD.lock().unwrap_or_else(|e| e.into_inner());
clear_env();
// Default when neither is set.
assert_eq!(resolve_filter_directive(), DEFAULT_FILTER);
// RUST_LOG wins over the default.
std::env::set_var("RUST_LOG", "warn");
assert_eq!(resolve_filter_directive(), "warn");
// NUNCIO_LOG wins over RUST_LOG.
std::env::set_var(LOG_FILTER_ENV_VAR, "debug");
assert_eq!(resolve_filter_directive(), "debug");
// An empty NUNCIO_LOG is treated as unset, so RUST_LOG applies again.
std::env::set_var(LOG_FILTER_ENV_VAR, " ");
assert_eq!(resolve_filter_directive(), "warn");
clear_env();
}
#[test]
fn malformed_filter_falls_back_to_default_without_panicking() {
// A valid directive parses and signals no fallback.
let (_filter, malformed) = build_filter("info,nunciod=debug");
assert!(malformed.is_none());
// A directive with an invalid level degrades to the default and
// reports the offending string, rather than panicking.
let (_filter, malformed) = build_filter("nunciod=not_a_level");
assert_eq!(malformed.as_deref(), Some("nunciod=not_a_level"));
}
#[test]
fn json_format_is_selected_only_by_the_json_value() {
let _guard = ENV_GUARD.lock().unwrap_or_else(|e| e.into_inner());
clear_env();
assert_eq!(log_format_from_env(), LogFormat::Human);
std::env::set_var(LOG_FORMAT_ENV_VAR, "json");
assert_eq!(log_format_from_env(), LogFormat::Json);
// Case-insensitive.
std::env::set_var(LOG_FORMAT_ENV_VAR, "JSON");
assert_eq!(log_format_from_env(), LogFormat::Json);
// Any other value stays human-readable.
std::env::set_var(LOG_FORMAT_ENV_VAR, "text");
assert_eq!(log_format_from_env(), LogFormat::Human);
clear_env();
}
#[test]
fn log_dir_is_a_logs_sibling_of_the_database_file() {
let db = PathBuf::from("/var/lib/nuncio/nuncio.db");
assert_eq!(
log_dir_for_db_path(&db),
PathBuf::from("/var/lib/nuncio/logs")
);
// A bare filename (no parent) still yields a usable relative dir.
assert_eq!(
log_dir_for_db_path(Path::new("nuncio.db")),
PathBuf::from("logs")
);
}
}
crates/nunciod/src/main.rs
+225 -83
View File
@@ -3,12 +3,36 @@
//! filter automation engine, outbox retries, and the multi-client gRPC API.
use nuncio_core::{CoreCommand, CoreEvent, EventBus};
use nuncio_filter::{FilterEngine, OutboxManager};
use nuncio_filter::FilterEngine;
use nunciod::lifecycle::{install_signal_handlers, ShutdownController};
use std::sync::Arc;
use std::time::Duration;
/// Bound on how long shutdown waits for in-flight work (gRPC requests still
/// being served, an outbox drain pass in progress, a background task's
/// current loop iteration) to finish on its own before the process moves on
/// to closing the database and exiting anyway. Keeps a stuck task from
/// turning a graceful shutdown into a hang.
const SHUTDOWN_GRACE_PERIOD: Duration = Duration::from_secs(10);
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
tracing_subscriber::fmt::init();
// PERSISTENT database path: defaults to `~/.nuncio/nuncio.db` -- NOT a
// temp/ephemeral path -- so accounts (and everything else) survive a
// daemon restart. `NUNCIO_DB_PATH` overrides it, which tests/CI use to
// point at an isolated temp path instead of touching a real user's home
// directory. Resolved first so logging can place its rotating files in the
// sibling `logs/` directory of the same data root.
let db_path = std::env::var("NUNCIO_DB_PATH")
.map(std::path::PathBuf::from)
.unwrap_or_else(|_| nunciod::default_db_path());
// Install the layered tracing subscriber (EnvFilter + stderr console +
// rotating daily file sink). `_log_guards` owns the non-blocking writer's
// `WorkerGuard`; it MUST stay bound until the end of `main` or buffered
// file logs are dropped on exit. Logging never crashes the daemon: a bad
// filter or an unwritable log directory degrades to a working subset.
let _log_guards = nunciod::logging::init(&db_path);
tracing::info!("Starting Nuncio Central Daemon Service (nunciod)...");
// The `CoreCommand` receiver must be claimed BEFORE the `EventBus` is
@@ -17,14 +41,25 @@ async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let mut event_bus_owned = EventBus::new();
let command_rx = event_bus_owned.take_command_receiver();
let event_bus = Arc::new(event_bus_owned);
// PERSISTENT database path: defaults to `~/.nuncio/nuncio.db` -- NOT a
// temp/ephemeral path -- so accounts (and everything else) survive a
// daemon restart.
// `NUNCIO_DB_PATH` overrides it, which tests/CI use to point at an
// isolated temp path instead of touching a real user's home directory.
let db_path = std::env::var("NUNCIO_DB_PATH")
.map(std::path::PathBuf::from)
.unwrap_or_else(|_| nunciod::default_db_path());
// Single shutdown signal shared by every long-running task spawned
// below (the real-sync command consumer, the outbox worker, the
// update-check loop, and the gRPC server itself), so a Ctrl+C/SIGTERM
// drains all of them cleanly instead of the OS killing the process
// mid-write. `install_signal_handlers` also drives the existing
// `CoreCommand::Shutdown` / `CoreEvent::ShuttingDown` event-bus path,
// previously wired but never triggered outside tests.
let (shutdown_controller, shutdown_signal) = ShutdownController::new(event_bus.clone());
let shutdown_controller = Arc::new(shutdown_controller);
let signal_task = tokio::spawn(install_signal_handlers(shutdown_controller));
// Exclusive single-instance lock on the database path, acquired BEFORE
// the store is opened. Held for the entire process lifetime (dropped at
// the end of `main`, releasing it on any clean exit) so a second
// `nunciod` pointed at the same database fails fast instead of the two
// instances corrupting SQLite state between them.
let _instance_lock =
nunciod::lock::InstanceLock::acquire(&db_path).map_err(|e| format!("{e}"))?;
let orchestrator = nunciod::SelfHealingSyncOrchestrator::new(&db_path, event_bus.clone());
let (db, _summary) = orchestrator.initialize_and_recover().await?;
@@ -36,10 +71,33 @@ async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
// wrapper instances.
let account_secrets = Arc::new(nuncio_store::vault::SecretManager::production());
// Load active rules from SQLite
let initial_rules = db.list_filter_rules().await.unwrap_or_default();
// Load active rules from SQLite. See `load_initial_filter_rules` for why
// a load failure must fail startup rather than silently become an empty
// rule set.
let initial_rules = nunciod::load_initial_filter_rules(&db).await?;
let filter_engine = Arc::new(FilterEngine::new(initial_rules)?);
// Single bounded-concurrency sync dispatcher shared by BOTH the
// client-facing `Mail/Sync` RPC (via the gRPC server below) and this
// process's `CoreCommand` sync consumer, so account syncs from either
// origin queue against one concurrency cap, coalesce per account, and
// observe one per-account timeout -- instead of each entry point launching
// syncs independently and head-of-line-blocking one another. It races
// in-flight syncs against the SAME shutdown signal every other subsystem
// shares.
let sync_dispatcher = {
let syncer = Arc::new(nunciod::sync_dispatcher::ProductionAccountSyncer::new(
db.clone(),
account_secrets.clone(),
event_bus.clone(),
filter_engine.clone(),
));
Arc::new(nunciod::sync_dispatcher::SyncDispatcher::from_env(
syncer,
shutdown_signal.clone(),
))
};
// Real inbound-sync `CoreCommand` consumer. This is the ONE place that
// claims `EventBus`'s command receiver, so it is what finally makes
// `CoreCommand::SyncAll` /
@@ -50,34 +108,35 @@ async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
// trigger_background_resync`'s post-recovery `send_command` calls a
// consumer: previously nothing read this channel, so that resync
// trigger was inert.
if let Some(mut command_rx) = command_rx {
let sync_command_task = if let Some(mut command_rx) = command_rx {
let db_sync = db.clone();
let secrets_sync = account_secrets.clone();
let event_bus_sync = event_bus.clone();
let filter_engine_sync = filter_engine.clone();
let _sync_command_task = tokio::spawn(async move {
while let Some(cmd) = command_rx.recv().await {
let dispatcher_sync = sync_dispatcher.clone();
let mut shutdown = shutdown_signal.clone();
Some(tokio::spawn(async move {
loop {
let cmd = tokio::select! {
_ = shutdown.wait() => {
tracing::info!("sync command consumer exiting on shutdown signal");
break;
}
received = command_rx.recv() => match received {
Some(cmd) => cmd,
None => break,
},
};
match cmd {
CoreCommand::SyncAll => {
let synced = nunciod::sync::run_all_accounts_sync(
let synced = nunciod::sync_dispatcher::sync_all_configured(
&dispatcher_sync,
&db_sync,
&secrets_sync,
&event_bus_sync,
&filter_engine_sync,
)
.await;
tracing::info!("SyncAll completed: {} message(s) synced", synced);
}
CoreCommand::SyncAccount { account_id } => {
match nunciod::sync::run_account_sync(
&db_sync,
&secrets_sync,
&event_bus_sync,
&filter_engine_sync,
&account_id,
)
.await
{
match dispatcher_sync.sync_account(&account_id).await {
Ok(count) => tracing::info!(
"SyncAccount({}) completed: {} message(s) synced",
account_id,
@@ -91,44 +150,58 @@ async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
_ => {}
}
}
});
}))
} else {
tracing::error!(
"EventBus command receiver was already taken; real sync command processing will not run"
);
}
None
};
// Background Outbox Worker Task
// Background Outbox Worker Task. Each tick drains the pending remote
// mutations the filter engine enqueued (move/copy/flag/unflag/delete on the
// real mail server, forward via SMTP, or a signed webhook call) and applies
// them against the resolved account's backend. A mutation is marked
// "completed" ONLY when the real operation genuinely succeeded; a transient
// failure is retried on later ticks and a permanent one (gone message,
// unknown action) fails honestly -- success is never fabricated.
let db_outbox = db.clone();
let _outbox_task = tokio::spawn(async move {
let outbox_env = Arc::new(nunciod::outbox::ProductionExecutionEnv::new(
db.clone(),
account_secrets.clone(),
));
let mut outbox_shutdown = shutdown_signal.clone();
let mut outbox_drain_shutdown = shutdown_signal.clone();
let outbox_task = tokio::spawn(async move {
let mut interval = tokio::time::interval(std::time::Duration::from_secs(5));
loop {
interval.tick().await;
if let Ok(pending) = db_outbox.list_pending_mutations(50).await {
for item in pending {
let next_retry = item.retry_count + 1;
if next_retry > OutboxManager::MAX_RETRIES {
let _ = db_outbox
.update_mutation_status(&item.id, "failed", next_retry)
.await;
continue;
tokio::select! {
_ = outbox_shutdown.wait() => {
tracing::info!("outbox worker exiting on shutdown signal");
break;
}
_ = interval.tick() => {
// A drain pass already in progress is never aborted
// mid-write by a shutdown signal that arrives after this
// branch was chosen -- but `execute_pending_mutations`
// itself races each item's execution against
// `outbox_drain_shutdown`, so a hung remote op cannot
// block this pass (and therefore shutdown) indefinitely.
let summary = nunciod::outbox::execute_pending_mutations(
&db_outbox,
outbox_env.as_ref(),
50,
&mut outbox_drain_shutdown,
)
.await;
if summary.completed > 0 || summary.failed > 0 {
tracing::info!(
"outbox drain: {} completed, {} retried, {} failed",
summary.completed,
summary.retried,
summary.failed
);
}
let backoff_ms = OutboxManager::calculate_backoff_ms(item.retry_count);
tokio::time::sleep(std::time::Duration::from_millis(backoff_ms)).await;
// Outbound IMAP/JMAP remote mutation execution (actually
// applying a filter action like move/delete/flag on the
// real mail server) is not yet wired to a real mail
// server. This worker deliberately does NOT mark
// mutations "completed": doing so would fabricate
// success for work that never happened. It only bumps
// the retry counter (so an item that keeps failing still
// eventually flips to "failed" once retries are
// exhausted) and leaves the item "pending" so it is
// retried on the next poll tick once a real sender is
// wired in.
let _ = db_outbox
.update_mutation_status(&item.id, "pending", next_retry)
.await;
}
}
}
@@ -142,21 +215,29 @@ async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
// closed, nunciod must never autonomously check for or install
// updates, so this loop only runs when an operator explicitly opts in
// via `NUNCIO_AUTO_UPDATE_ENABLED=1`.
let _update_task = if nunciod::auto_update_task_enabled() {
let update_task = if nunciod::auto_update_task_enabled() {
let event_bus_update = event_bus.clone();
let mut update_shutdown = shutdown_signal.clone();
Some(tokio::spawn(async move {
let mut interval = tokio::time::interval(std::time::Duration::from_secs(86400));
interval.tick().await;
loop {
interval.tick().await;
if let Ok(updater) = nuncio_core::UpdateEngine::new() {
if let Ok(result) = updater.check_for_updates().await {
if result.update_available {
if let Some(info) = result.release_info {
event_bus_update.publish_event(CoreEvent::UpdateAvailable {
version: info.version,
release_notes: info.release_notes,
});
tokio::select! {
_ = update_shutdown.wait() => {
tracing::info!("update-check loop exiting on shutdown signal");
break;
}
_ = interval.tick() => {
if let Ok(updater) = nuncio_core::UpdateEngine::new() {
if let Ok(result) = updater.check_for_updates().await {
if result.update_available {
if let Some(info) = result.release_info {
event_bus_update.publish_event(CoreEvent::UpdateAvailable {
version: info.version,
release_notes: info.release_notes,
});
}
}
}
}
}
@@ -182,11 +263,14 @@ async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
// `SecretManager` to write account password credentials to the OS
// keyring (never to SQLite) -- see `nunciod::grpc`'s security comment on
// why EVERY mounted service shares this one `BearerAuthInterceptor`.
// The read is bounded: a keyring that never answers must end in this
// logged startup failure, not in a daemon that hangs here forever.
let grpc_secrets = account_secrets.clone();
let grpc_token_bytes = grpc_secrets
.get_or_create_key_bytes(nuncio_store::vault::GRPC_TOKEN_ACCOUNT, 32)
.map_err(|e| format!("failed to provision gRPC bearer token from vault: {e}"))?;
let grpc_token = hex::encode(grpc_token_bytes);
let grpc_token = nunciod::secrets::provision_grpc_token(
grpc_secrets.clone(),
nuncio_store::vault::KEYRING_READ_TIMEOUT,
)
.await?;
let grpc_addr = nunciod::grpc::grpc_addr_from_env();
tracing::info!(
"nunciod gRPC (nuncio.v1.System, nuncio.v1.Accounts, nuncio.v1.Mail, nuncio.v1.Filters, \
@@ -196,17 +280,75 @@ async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
// The gRPC server is the long-running foreground task that keeps this
// process alive; every other subsystem above is a background worker
// spawned on top of it.
nunciod::grpc::serve(
&grpc_addr,
event_bus,
db,
filter_engine,
grpc_secrets,
grpc_token,
)
.await
.map_err(|e| format!("nunciod gRPC server failed: {e}"))?;
// spawned on top of it. `serve_with_shutdown` stops accepting new
// connections and starts draining in-flight requests as soon as
// `shutdown_signal` fires; the `select!` below bounds how long that
// drain is allowed to take before this process moves on regardless.
let db_for_serve = db.clone();
let mut grpc_shutdown_future = shutdown_signal.clone();
let mut grace_timer_signal = shutdown_signal.clone();
let grpc_result = tokio::select! {
result = nunciod::grpc::serve_with_shutdown(
&grpc_addr,
event_bus,
db_for_serve,
filter_engine,
grpc_secrets,
grpc_token,
sync_dispatcher,
async move { grpc_shutdown_future.wait().await },
) => result,
_ = async move {
grace_timer_signal.wait().await;
tokio::time::sleep(SHUTDOWN_GRACE_PERIOD).await;
} => {
tracing::warn!(
"gRPC server did not finish draining in-flight requests within the {:?} \
shutdown grace period; closing the database and exiting anyway",
SHUTDOWN_GRACE_PERIOD
);
Ok(())
}
};
// From here down is the graceful-shutdown drain: the gRPC server has
// stopped accepting new connections, so log each remaining teardown step
// to make a clean shutdown fully traceable end to end.
tracing::info!(
"shutdown: gRPC server stopped accepting connections; draining background workers"
);
// Bound how long the background workers get to notice the shutdown
// signal and exit their loops before the database is closed out from
// under them.
let workers_drained = tokio::time::timeout(SHUTDOWN_GRACE_PERIOD, async {
if let Some(task) = sync_command_task {
let _ = task.await;
}
let _ = outbox_task.await;
if let Some(task) = update_task {
let _ = task.await;
}
})
.await;
match workers_drained {
Ok(()) => tracing::info!("shutdown: background workers (sync, outbox, update) drained"),
Err(_) => tracing::warn!(
"shutdown: background workers did not all exit within the {:?} grace period; \
proceeding to close the database",
SHUTDOWN_GRACE_PERIOD
),
}
signal_task.abort();
// Force the WAL checkpoint before the process exits so no committed
// page is left for SQLite's own deferred "last connection closes"
// teardown to race (see `DatabaseEngine::close`'s doc comment).
tracing::info!("shutdown: closing database and checkpointing the WAL");
db.close().await;
tracing::info!("shutdown: complete");
grpc_result.map_err(|e| format!("nunciod gRPC server failed: {e}"))?;
Ok(())
}
crates/nunciod/src/orchestrator.rs
+9 -8
View File
@@ -152,16 +152,17 @@ mod tests {
id: "acct-orch-1".to_string(),
name: "Orch Account".to_string(),
email_address: "orch@nuncio.mx".to_string(),
protocol: nuncio_core::AccountProtocol::ImapSmtp,
server_host: "imap.nuncio.mx".to_string(),
server_port: 993,
smtp_host: "smtp.nuncio.mx".to_string(),
smtp_port: 465,
use_tls: true,
imap_tls_mode: nuncio_core::TlsMode::ImplicitTls,
smtp_tls_mode: nuncio_core::TlsMode::ImplicitTls,
keyring_secret_key: "nuncio/acct-orch-1".to_string(),
sync_interval_secs: 60,
filters_enabled: false,
transport: nuncio_core::Transport::ImapSmtp(nuncio_core::ImapSmtpTransport {
imap_host: "imap.nuncio.mx".to_string(),
imap_port: 993,
imap_tls_mode: nuncio_core::TlsMode::ImplicitTls,
smtp_host: "smtp.nuncio.mx".to_string(),
smtp_port: 465,
smtp_tls_mode: nuncio_core::TlsMode::ImplicitTls,
}),
};
engine.save_account(&acct).await.unwrap();
engine.close().await;
crates/nunciod/src/outbox.rs
+1012
View File
File diff suppressed because it is too large. Load diff
crates/nunciod/src/pagination.rs
+168
View File
@@ -0,0 +1,168 @@
//! Shared keyset ("cursor") pagination for the daemon's list/search RPCs.
//!
//! Every `nuncio.v1` list/search RPC pages results with the same convention
//! (AIP-158): a request carries `page_size` + `page_token` and a response
//! carries `next_page_token`. The `page_token` is an OPAQUE cursor, never a
//! numeric offset: it encodes the stable sort key (and id tiebreaker) of the
//! last row of the previous page so the next page resumes strictly AFTER that
//! row. Keyset paging is stable under concurrent inserts -- rows are never
//! skipped or returned twice as the underlying table changes between page
//! fetches, which offset paging cannot guarantee.
//!
//! The wire encoding is deliberately private to the daemon: it is a
//! hex-encoded JSON array of typed [`CursorField`]s. Clients must treat the
//! token as opaque and echo it back verbatim; a token this module cannot
//! decode is rejected with a typed [`ErrorReason::ValidationFailed`] error
//! (mapped to `InvalidArgument`), never silently ignored or treated as the
//! first page.
use nuncio_proto::errors;
use nuncio_proto::v1::ErrorReason;
use serde::{Deserialize, Serialize};
use tonic::Status;
/// Page size used when a request leaves `page_size` at 0 ("server default").
pub const DEFAULT_PAGE_SIZE: u32 = 50;
/// Hard upper bound on a single page, so a client cannot force the daemon to
/// materialize an unbounded result set in one call.
pub const MAX_PAGE_SIZE: u32 = 500;
/// Resolves a requested `page_size` into a concrete row count: 0 selects
/// [`DEFAULT_PAGE_SIZE`], and any value is clamped to [`MAX_PAGE_SIZE`].
#[must_use]
pub fn clamp_page_size(requested: u32) -> usize {
let n = if requested == 0 {
DEFAULT_PAGE_SIZE
} else {
requested.min(MAX_PAGE_SIZE)
};
n as usize
}
/// One typed component of an opaque keyset cursor. A cursor is an ordered
/// list of these, matching a list's stable sort key followed by its id
/// tiebreaker (e.g. `[I(received_at), S(id)]` for messages).
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
pub enum CursorField {
/// A signed integer sort key (e.g. a unix timestamp or a priority).
I(i64),
/// An unsigned integer sort key (e.g. an audit sequence).
U(u64),
/// A floating-point sort key (e.g. an FTS5 relevance rank).
F(f64),
/// A textual sort key or id (e.g. a message id or display name).
S(String),
}
/// Encodes a cursor's component list into an opaque `page_token`.
#[must_use]
pub fn encode_cursor(fields: &[CursorField]) -> String {
// The component list is a fixed, small set of primitives, so
// serialization cannot realistically fail; fall back to an empty payload
// rather than panicking, keeping this crate free of `unwrap`/`expect`.
let json = serde_json::to_vec(fields).unwrap_or_default();
hex::encode(json)
}
fn malformed_token() -> Status {
errors::status(
ErrorReason::ValidationFailed,
"malformed page_token: not a valid pagination cursor",
)
}
/// Decodes an opaque `page_token` back into its component list, rejecting a
/// token that is not a hex-encoded cursor JSON payload with a typed
/// `VALIDATION_FAILED` error.
pub fn decode_cursor(token: &str) -> Result<Vec<CursorField>, Status> {
let bytes = hex::decode(token).map_err(|_| malformed_token())?;
serde_json::from_slice(&bytes).map_err(|_| malformed_token())
}
/// Decodes an optional `page_token` into a strongly-typed cursor value.
///
/// An empty token means "first page" and yields `Ok(None)`. A non-empty
/// token is decoded and handed to `shape`, which pattern-matches the expected
/// component layout for a specific list and returns `Some(value)`; a token
/// that decodes but does not match the expected layout (e.g. a cursor minted
/// for a different list) is rejected with the same typed `VALIDATION_FAILED`
/// error as an undecodable token, so a caller can never resume from a cursor
/// shaped for the wrong query.
pub fn parse_token<T>(
token: &str,
shape: impl FnOnce(&[CursorField]) -> Option<T>,
) -> Result<Option<T>, Status> {
if token.is_empty() {
return Ok(None);
}
let fields = decode_cursor(token)?;
shape(&fields).map(Some).ok_or_else(malformed_token)
}
#[cfg(test)]
mod tests {
use super::*;
use tonic::Code;
#[test]
fn clamp_page_size_applies_default_and_max() {
assert_eq!(clamp_page_size(0), DEFAULT_PAGE_SIZE as usize);
assert_eq!(clamp_page_size(10), 10);
assert_eq!(clamp_page_size(MAX_PAGE_SIZE + 1), MAX_PAGE_SIZE as usize);
assert_eq!(clamp_page_size(u32::MAX), MAX_PAGE_SIZE as usize);
}
#[test]
fn cursor_round_trips_through_encode_decode() {
let fields = vec![
CursorField::I(1_700_000_000),
CursorField::S("msg-9".into()),
];
let token = encode_cursor(&fields);
assert!(!token.is_empty());
assert_eq!(decode_cursor(&token).unwrap(), fields);
}
#[test]
fn empty_token_parses_as_first_page() {
let parsed = parse_token("", |f| match f {
[CursorField::I(n)] => Some(*n),
_ => None,
})
.unwrap();
assert_eq!(parsed, None);
}
#[test]
fn malformed_token_is_validation_failed() {
// Not valid hex.
let err = decode_cursor("zzz not-hex").unwrap_err();
assert_eq!(err.code(), Code::InvalidArgument);
assert_eq!(
errors::error_reason(&err),
Some(ErrorReason::ValidationFailed)
);
// Valid hex, but not a cursor JSON payload.
let err = decode_cursor(&hex::encode(b"not json")).unwrap_err();
assert_eq!(err.code(), Code::InvalidArgument);
}
#[test]
fn wrong_shape_token_is_rejected() {
// A cursor with a single int component handed to a shape expecting
// (int, string) must be rejected, not silently accepted.
let token = encode_cursor(&[CursorField::I(1)]);
let err = parse_token(&token, |f| match f {
[CursorField::I(n), CursorField::S(s)] => Some((*n, s.clone())),
_ => None,
})
.unwrap_err();
assert_eq!(err.code(), Code::InvalidArgument);
assert_eq!(
errors::error_reason(&err),
Some(ErrorReason::ValidationFailed)
);
}
}
crates/nunciod/src/rpc_trace.rs
+401
View File
@@ -0,0 +1,401 @@
//! Per-RPC tracing span and request-id correlation for the gRPC transport.
//!
//! This is a [`tower::Layer`] mounted on the tonic `Server` so that every
//! inbound RPC is wrapped in a single root span carrying:
//!
//! * a generated `request_id` -- a lightweight per-process identifier
//! (process-start tag + monotonic counter) that lets an operator follow one
//! request across every log line it produced, without pulling in a heavier
//! UUID dependency; and
//! * the gRPC method path and the peer socket address.
//!
//! The inner service (routing + the per-service `BearerAuthInterceptor` + the
//! handler future) is entered inside that span, so *every* event emitted while
//! the request is handled -- including the auth accept/reject logs and any
//! event a handler emits -- inherits the `request_id` automatically. The span
//! opens with a DEBUG "received" event and closes, once the response body has
//! been fully drained, with an INFO event carrying the terminal gRPC status
//! code and the elapsed wall-clock time.
//!
//! ## Why the response body is wrapped
//!
//! A gRPC call's final status lives in the HTTP/2 *trailers* (`grpc-status`),
//! which are only produced once the response body finishes streaming (an
//! immediate error is instead a trailers-only response with `grpc-status` in
//! the leading headers). The layer therefore wraps the response body in
//! [`TracedBody`], which observes the terminal status as it flows past and
//! emits the completion event when the stream ends (or when the body is
//! dropped, e.g. on client disconnect), so a long-lived server-streaming RPC
//! is not logged as "complete" until it actually ends.
use std::future::Future;
use std::pin::Pin;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::OnceLock;
use std::task::{Context, Poll};
use std::time::{Instant, SystemTime, UNIX_EPOCH};
use http::{HeaderMap, Request, Response};
use http_body::{Body, Frame, SizeHint};
use tonic::body::Body as TonicBody;
use tonic::transport::server::TcpConnectInfo;
use tonic::Code;
use tower::{Layer, Service};
use tracing::{Instrument, Span};
/// HTTP/2 trailer (or leading header, for a trailers-only error response)
/// carrying the final gRPC status code.
const GRPC_STATUS_HEADER: &str = "grpc-status";
/// Generates a correlation id unique within this daemon process.
///
/// The id is `<process-tag>-<counter>`: the process tag is the daemon's start
/// time in unix nanoseconds (captured once), and the counter increments per
/// call. This is deliberately not a UUID -- it needs only to be unique within
/// a single running daemon so its logs can be correlated, which a per-process
/// tag plus a monotonic counter satisfies without a new dependency.
fn next_request_id() -> String {
static COUNTER: AtomicU64 = AtomicU64::new(0);
static PROCESS_TAG: OnceLock<u64> = OnceLock::new();
let tag = *PROCESS_TAG.get_or_init(|| {
SystemTime::now()
.duration_since(UNIX_EPOCH)
.map(|d| d.as_nanos() as u64)
.unwrap_or(0)
});
let n = COUNTER.fetch_add(1, Ordering::Relaxed);
format!("{tag:016x}-{n:08x}")
}
/// Reads the `grpc-status` code from a header/trailer map, if present.
fn grpc_status_code(headers: &HeaderMap) -> Option<Code> {
headers
.get(GRPC_STATUS_HEADER)
.map(|value| Code::from_bytes(value.as_bytes()))
}
/// [`tower::Layer`] that wraps each RPC in a correlation span. Mounted on the
/// tonic `Server` via `Server::builder().layer(RpcTraceLayer)`.
#[derive(Clone, Copy, Debug, Default)]
pub struct RpcTraceLayer;
impl<S> Layer<S> for RpcTraceLayer {
type Service = RpcTrace<S>;
fn layer(&self, inner: S) -> Self::Service {
RpcTrace { inner }
}
}
/// The [`RpcTraceLayer`]'s service: opens the span, runs the inner service
/// (routing, auth interceptor, handler) inside it, and wraps the response body
/// so the terminal status can be observed at stream end.
#[derive(Clone, Debug)]
pub struct RpcTrace<S> {
inner: S,
}
impl<S> Service<Request<TonicBody>> for RpcTrace<S>
where
S: Service<Request<TonicBody>, Response = Response<TonicBody>> + Clone + Send + 'static,
S::Future: Send + 'static,
S::Error: Send + 'static,
{
type Response = Response<TracedBody<TonicBody>>;
type Error = S::Error;
type Future =
Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send + 'static>>;
fn poll_ready(&mut self, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
self.inner.poll_ready(cx)
}
fn call(&mut self, req: Request<TonicBody>) -> Self::Future {
// `poll_ready` was called on `self.inner`, so it -- not the fresh
// clone -- is the readied service that must handle this request. Swap
// the readied service out and leave the clone behind for the next
// call (the standard tower clone-and-replace pattern that avoids
// calling an un-readied clone).
let clone = self.inner.clone();
let mut inner = std::mem::replace(&mut self.inner, clone);
let method = req.uri().path().to_string();
let peer = req
.extensions()
.get::<TcpConnectInfo>()
.and_then(TcpConnectInfo::remote_addr)
.map(|addr| addr.to_string())
.unwrap_or_else(|| "unknown".to_string());
let request_id = next_request_id();
let span = tracing::info_span!(
"grpc.request",
request_id = %request_id,
rpc.method = %method,
peer = %peer,
);
let start = Instant::now();
Box::pin(async move {
// Enter the span for the synchronous routing + interceptor call so
// the auth accept/reject logs inherit the request_id, and emit the
// open event under it.
let call = span.in_scope(|| {
tracing::debug!("gRPC request received");
inner.call(req)
});
// Instrument the handler future so events emitted across its await
// points also inherit the span.
let response = call.instrument(span.clone()).await?;
// A trailers-only error response carries `grpc-status` in the
// leading headers; a normal response carries it in the trailers,
// observed by `TracedBody` as the body drains.
let initial_status = grpc_status_code(response.headers());
let (parts, body) = response.into_parts();
let traced = TracedBody::new(body, span, start, initial_status);
Ok(Response::from_parts(parts, traced))
})
}
}
/// Response body wrapper that observes the terminal `grpc-status` and logs the
/// request's completion (status + elapsed) exactly once, when the body stream
/// ends or is dropped.
///
/// All fields are `Unpin`, so the pin projection in [`Body::poll_frame`] is a
/// safe `Pin::new` over the inner body rather than an `unsafe` projection.
#[derive(Debug)]
pub struct TracedBody<B> {
inner: B,
span: Span,
start: Instant,
status: Option<Code>,
logged: bool,
}
impl<B> TracedBody<B> {
fn new(inner: B, span: Span, start: Instant, initial_status: Option<Code>) -> Self {
Self {
inner,
span,
start,
status: initial_status,
logged: false,
}
}
/// Emits the terminal completion event once. Idempotent: subsequent calls
/// (e.g. a `Drop` after the stream already ended) are no-ops.
fn log_completion(&mut self) {
if self.logged {
return;
}
self.logged = true;
let elapsed_ms = self.start.elapsed().as_millis() as u64;
let status = self.status.unwrap_or(Code::Unknown);
self.span.in_scope(|| {
tracing::info!(
grpc.status = ?status,
elapsed_ms,
"gRPC request completed"
);
});
}
}
impl<B> Body for TracedBody<B>
where
B: Body + Unpin,
{
type Data = B::Data;
type Error = B::Error;
fn poll_frame(
self: Pin<&mut Self>,
cx: &mut Context<'_>,
) -> Poll<Option<Result<Frame<Self::Data>, Self::Error>>> {
// Safe because `Self: Unpin` (all fields are `Unpin`).
let this = self.get_mut();
let polled = Pin::new(&mut this.inner).poll_frame(cx);
match &polled {
Poll::Ready(Some(Ok(frame))) => {
if let Some(trailers) = frame.trailers_ref() {
if let Some(code) = grpc_status_code(trailers) {
this.status = Some(code);
}
}
}
Poll::Ready(Some(Err(_))) | Poll::Ready(None) => this.log_completion(),
Poll::Pending => {}
}
polled
}
fn is_end_stream(&self) -> bool {
self.inner.is_end_stream()
}
fn size_hint(&self) -> SizeHint {
self.inner.size_hint()
}
}
impl<B> Drop for TracedBody<B> {
fn drop(&mut self) {
// Covers stream ends that do not surface a final frame here (client
// disconnect, an outer layer dropping the body): the request is still
// logged as completed exactly once.
self.log_completion();
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::test_tracing::with_recorder;
use http_body_util::BodyExt;
use std::collections::HashSet;
use std::convert::Infallible;
use tower::ServiceExt;
use tracing::Level;
/// Minimal inner service that returns a response with `grpc-status` set in
/// the leading headers (a trailers-only response, tonic's shape for an
/// immediate status), so the layer's terminal-status extraction can be
/// exercised without a live server.
#[derive(Clone)]
struct StatusService {
code: i32,
}
impl Service<Request<TonicBody>> for StatusService {
type Response = Response<TonicBody>;
type Error = Infallible;
type Future =
Pin<Box<dyn Future<Output = Result<Self::Response, Infallible>> + Send + 'static>>;
fn poll_ready(&mut self, _cx: &mut Context<'_>) -> Poll<Result<(), Infallible>> {
Poll::Ready(Ok(()))
}
fn call(&mut self, _req: Request<TonicBody>) -> Self::Future {
let code = self.code;
Box::pin(async move {
// A handler-emitted event: it must inherit the span so it
// carries the request_id set by the layer.
tracing::info!("handler ran");
let mut response = Response::new(TonicBody::empty());
response
.headers_mut()
.insert(GRPC_STATUS_HEADER, code.to_string().parse().unwrap());
Ok(response)
})
}
}
fn request() -> Request<TonicBody> {
Request::builder()
.uri("/nuncio.v1.Mail/Sync")
.body(TonicBody::empty())
.unwrap()
}
#[test]
fn rpc_produces_span_with_request_id_and_logs_terminal_status() {
let runtime = tokio::runtime::Builder::new_current_thread()
.build()
.unwrap();
let (recorder, ()) = with_recorder(|| {
runtime.block_on(async {
let mut svc = RpcTraceLayer.layer(StatusService { code: 5 });
let response = svc.ready().await.unwrap().call(request()).await.unwrap();
// Drain the body so the completion event fires at stream end.
response.into_body().collect().await.unwrap();
});
});
let spans = recorder.spans();
let rpc_span = spans
.iter()
.find(|s| s.name == "grpc.request")
.expect("a grpc.request span must be created");
let request_id = rpc_span
.fields
.get("request_id")
.expect("the span must carry a request_id field");
assert!(!request_id.is_empty(), "request_id must be populated");
assert_eq!(
rpc_span.fields.get("rpc.method").map(String::as_str),
Some("/nuncio.v1.Mail/Sync"),
"the span must carry the gRPC method path"
);
let events = recorder.events();
let completion = events
.iter()
.find(|e| e.message() == "gRPC request completed")
.expect("the terminal completion event must be logged");
assert_eq!(
completion.level,
Level::INFO,
"the completion event is logged at INFO"
);
assert_eq!(
completion.fields.get("grpc.status").map(String::as_str),
Some("NotFound"),
"the completion event must carry the terminal gRPC status code"
);
assert!(
completion.fields.contains_key("elapsed_ms"),
"the completion event must carry the elapsed time"
);
// The handler ran inside the span, so its own event was captured too
// (proving handler events inherit the correlation context).
assert!(
events.iter().any(|e| e.message() == "handler ran"),
"a handler-emitted event must be observed under the request span"
);
}
#[test]
fn ok_status_is_recorded_at_completion() {
let runtime = tokio::runtime::Builder::new_current_thread()
.build()
.unwrap();
let (recorder, ()) = with_recorder(|| {
runtime.block_on(async {
let mut svc = RpcTraceLayer.layer(StatusService { code: 0 });
let response = svc.ready().await.unwrap().call(request()).await.unwrap();
response.into_body().collect().await.unwrap();
});
});
let events = recorder.events();
let completion = events
.iter()
.find(|e| e.message() == "gRPC request completed")
.expect("completion event");
assert_eq!(
completion.fields.get("grpc.status").map(String::as_str),
Some("Ok")
);
}
#[test]
fn grpc_status_code_parses_known_codes_and_ignores_absent() {
let mut headers = HeaderMap::new();
assert!(grpc_status_code(&headers).is_none());
headers.insert(GRPC_STATUS_HEADER, "3".parse().unwrap());
assert_eq!(grpc_status_code(&headers), Some(Code::InvalidArgument));
}
#[test]
fn request_ids_are_unique_within_the_process() {
let ids: HashSet<String> = (0..1000).map(|_| next_request_id()).collect();
assert_eq!(ids.len(), 1000, "request ids must not collide");
}
}
crates/nunciod/src/secrets.rs
+61
View File
@@ -0,0 +1,61 @@
//! Bounded provisioning of the daemon's own key material.
//!
//! `nunciod` normally *mints* the keys it uses, so the OS keyring hands them
//! back without ceremony. That is a habit, not a guarantee: a keychain locked
//! by screen-lock, a restored or migrated keychain whose ACL no longer trusts
//! this binary, a re-signed or rebuilt `nunciod`, or a Linux Secret Service
//! with no unlocked collection all turn a keyring read into a blocking wait on
//! something nobody can answer. Unbounded, that wait becomes a daemon that
//! hangs at startup with no error and no log line.
//!
//! Every read here therefore goes through
//! [`nuncio_store::vault::get_or_create_key_bytes_bounded`], so the worst case
//! is an honest, logged startup failure instead of a silent stall. Errors name
//! the key material's purpose and the likely cause; they never carry the key.
use nuncio_store::vault::{
get_or_create_key_bytes_bounded, SecretManager, GRPC_TOKEN_ACCOUNT, WEBHOOK_SIGNING_KEY_ACCOUNT,
};
use std::sync::Arc;
use std::time::Duration;
/// Byte length of the daemon's symmetric key material (256 bits).
const KEY_LEN: usize = 32;
/// Provision the loopback gRPC bearer token, minting it on first run and
/// loading it back on every subsequent one.
///
/// Fails closed: the daemon must never serve its API with a token it could not
/// confirm, and must never invent one.
pub async fn provision_grpc_token(
secrets: Arc<SecretManager>,
timeout: Duration,
) -> Result<String, String> {
get_or_create_key_bytes_bounded(secrets, GRPC_TOKEN_ACCOUNT, KEY_LEN, timeout)
.await
.map(hex::encode)
.map_err(|e| {
format!(
"failed to provision the gRPC bearer token that nunciod needs to authenticate \
its loopback API: {e}"
)
})
}
/// Provision the daemon-wide HMAC key that signs outbound filter webhooks.
///
/// Fails closed: a `CALL WEBHOOK` is never dispatched unsigned.
pub async fn provision_webhook_signing_key(
secrets: Arc<SecretManager>,
timeout: Duration,
) -> Result<String, String> {
get_or_create_key_bytes_bounded(secrets, WEBHOOK_SIGNING_KEY_ACCOUNT, KEY_LEN, timeout)
.await
.map(hex::encode)
.map_err(|e| {
format!(
"failed to provision the webhook signing key that nunciod needs to sign outbound \
filter webhooks: {e}"
)
})
}
crates/nunciod/src/send.rs
+146 -15
View File
@@ -68,6 +68,11 @@ pub enum SendError {
/// rather than silently sending from a different account.
#[error("no account configured with id {0:?}")]
AccountNotFound(String),
/// The resolved account does not have an SMTP outbound endpoint (only an
/// IMAP/SMTP account can send outbound mail over SMTP; a JMAP or DAV
/// account cannot).
#[error("account {0:?} has no SMTP transport and cannot send outbound mail")]
NotAnSmtpAccount(String),
/// Failed to read the account's credential from the secret vault.
#[error("failed to read account credential from the secret vault: {0}")]
Vault(#[from] VaultError),
@@ -120,10 +125,13 @@ fn build_smtp_sender(
config: &AccountConfig,
password: &str,
) -> Result<SmtpTransportEngine, SendError> {
let transport = config
.imap_smtp()
.ok_or_else(|| SendError::NotAnSmtpAccount(config.id.clone()))?;
let engine = SmtpTransportEngine::new(
&config.smtp_host,
config.smtp_port,
config.smtp_tls_mode,
&transport.smtp_host,
transport.smtp_port,
transport.smtp_tls_mode,
&config.email_address,
password,
)?;
@@ -175,6 +183,15 @@ pub async fn send_message_for_account(
request: ComposeRequest,
) -> Result<String, SendError> {
let config = resolve_sending_account(db, request.account_id.as_deref()).await?;
let account_id = config.id.clone();
let attachment_count = request.attachments.len();
tracing::info!(
account_id = %account_id,
transport = "smtp",
has_cc = request.cc.is_some(),
attachment_count,
"mail send: dispatching outbound message over real SMTP transport"
);
let password = secrets.get_secret(&config.keyring_secret_key)?;
let sender = build_smtp_sender(&config, &password)?;
@@ -188,8 +205,24 @@ pub async fn send_message_for_account(
attachments: request.attachments,
};
send_with_sender(&sender, &outbound).await?;
Ok(generate_message_id())
match send_with_sender(&sender, &outbound).await {
Ok(()) => {
let message_id = generate_message_id();
tracing::info!(
account_id = %account_id,
message_id = %message_id,
"mail send: outbound message accepted by SMTP transport"
);
Ok(message_id)
}
Err(e) => {
tracing::warn!(
account_id = %account_id,
"mail send: SMTP transport rejected the outbound message: {e}"
);
Err(e)
}
}
}
/// Test/E2E entry point for the `SendMessage` RPC when a
@@ -207,6 +240,14 @@ pub async fn send_message_with_injected_sender(
request: ComposeRequest,
) -> Result<String, SendError> {
let config = resolve_sending_account(db, request.account_id.as_deref()).await?;
let account_id = config.id.clone();
tracing::info!(
account_id = %account_id,
transport = "injected",
has_cc = request.cc.is_some(),
attachment_count = request.attachments.len(),
"mail send: dispatching outbound message via injected sender"
);
let outbound = OutboundMessage {
from: config.email_address,
@@ -218,8 +259,24 @@ pub async fn send_message_with_injected_sender(
attachments: request.attachments,
};
send_with_sender(sender, &outbound).await?;
Ok(generate_message_id())
match send_with_sender(sender, &outbound).await {
Ok(()) => {
let message_id = generate_message_id();
tracing::info!(
account_id = %account_id,
message_id = %message_id,
"mail send: outbound message accepted by injected sender"
);
Ok(message_id)
}
Err(e) => {
tracing::warn!(
account_id = %account_id,
"mail send: injected sender rejected the outbound message: {e}"
);
Err(e)
}
}
}
#[cfg(test)]
@@ -233,16 +290,17 @@ mod tests {
id: id.to_string(),
name: "Send Test Account".to_string(),
email_address: format!("{id}@nuncio.mx"),
protocol: nuncio_core::AccountProtocol::ImapSmtp,
server_host: "imap.nuncio.mx".to_string(),
server_port: 993,
smtp_host: "127.0.0.1".to_string(),
smtp_port: 1,
use_tls: true,
imap_tls_mode: TlsMode::ImplicitTls,
smtp_tls_mode: TlsMode::ImplicitTls,
keyring_secret_key: format!("nuncio/{id}"),
sync_interval_secs: 60,
filters_enabled: false,
transport: nuncio_core::Transport::ImapSmtp(nuncio_core::ImapSmtpTransport {
imap_host: "imap.nuncio.mx".to_string(),
imap_port: 993,
imap_tls_mode: TlsMode::ImplicitTls,
smtp_host: "127.0.0.1".to_string(),
smtp_port: 1,
smtp_tls_mode: TlsMode::ImplicitTls,
}),
}
}
@@ -526,4 +584,77 @@ mod tests {
assert!(matches!(err, SendError::AccountNotFound(id) if id == "acct-does-not-exist"));
assert!(mock.sent_messages().is_empty());
}
/// A send must be followable in the log: a dispatch event before the send
/// and an acceptance event after, both scoped to the account id -- and the
/// recipient address, subject, and body must never appear in any logged
/// field.
#[test]
fn send_logs_dispatch_and_acceptance_without_leaking_recipient_or_subject() {
use crate::test_tracing::with_recorder;
use tracing::Level;
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("runtime");
let (recorder, ()) = with_recorder(|| {
runtime.block_on(async {
let (db, _dir) = DatabaseEngine::connect_ephemeral()
.await
.expect("ephemeral db");
let config = sample_account("acct-log");
db.save_account(&config).await.expect("save account");
let mock = MockMessageSender::new();
send_message_with_injected_sender(
&db,
&mock,
ComposeRequest {
to: "secret-recipient@example.com".to_string(),
cc: None,
subject: "Confidential Subject Line".to_string(),
body_text: Some("secret body".to_string()),
body_html: None,
attachments: Vec::new(),
account_id: None,
},
)
.await
.expect("send succeeds");
});
});
let events = recorder.events();
let dispatch = events
.iter()
.find(|e| e.message() == "mail send: dispatching outbound message via injected sender")
.expect("dispatch event must be logged");
assert_eq!(dispatch.level, Level::INFO);
assert_eq!(
dispatch.fields.get("account_id").map(String::as_str),
Some("acct-log")
);
assert!(events
.iter()
.any(|e| e.message() == "mail send: outbound message accepted by injected sender"));
// Secret-safety: no recipient, subject, or body text may appear in any
// recorded telemetry field.
for value in recorder.all_field_values() {
assert!(
!value.contains("secret-recipient@example.com"),
"recipient leaked into telemetry: {value}"
);
assert!(
!value.contains("Confidential Subject Line"),
"subject leaked into telemetry: {value}"
);
assert!(
!value.contains("secret body"),
"body leaked into telemetry: {value}"
);
}
}
}
crates/nunciod/src/sync.rs
+736 -326
View File
File diff suppressed because it is too large. Load diff
crates/nunciod/src/sync_dispatcher.rs
+862
View File
@@ -0,0 +1,862 @@
//! Bounded-concurrency sync dispatcher.
//!
//! Account syncs used to be launched directly and independently from two
//! places (the client-facing `Mail/Sync` RPC handler and the daemon's
//! `CoreCommand` consumer), each of which ran an account's sync to completion
//! before the next one started. One slow or hung account could therefore
//! head-of-line-block every account behind it, and nothing bounded how many
//! syncs ran at once or how long a single account was allowed to take.
//!
//! [`SyncDispatcher`] is the single place all account syncs now funnel
//! through. It:
//!
//! * caps how many account syncs run at once with a [`Semaphore`]
//! (env-tunable via [`CONCURRENCY_ENV_VAR`], default
//! [`DEFAULT_CONCURRENCY`]); excess requests queue on the permit rather
//! than run unbounded;
//! * coalesces a second request for an account that is already syncing onto
//! the in-flight one instead of double-fetching -- a client `Sync` issued
//! while a scheduled sync of the same account is running joins it and
//! observes its result;
//! * bounds each account's sync with a per-account timeout (env-tunable via
//! [`TIMEOUT_ENV_VAR`], default [`DEFAULT_TIMEOUT`]); a timed-out sync is
//! cancelled cleanly and surfaces [`SyncDispatchError::Timeout`] without
//! poisoning the dispatcher for other accounts; and
//! * respects the shared shutdown signal: once shutdown is requested it stops
//! admitting new work and races each in-flight sync against the signal
//! (mirroring the outbox executor) so a hung account cannot delay graceful
//! shutdown beyond the daemon's grace period.
//!
//! # Testability
//!
//! The actual per-account sync is injected behind the [`AccountSyncer`]
//! trait, so the dispatcher's concurrency, coalescing, timeout, and shutdown
//! behavior is exercised deterministically with an in-memory mock syncer --
//! no real network, keyring, or database. Production wires
//! [`ProductionAccountSyncer`], which delegates to
//! [`crate::sync::run_account_sync`].
use crate::lifecycle::ShutdownSignal;
use crate::sync::SyncError;
use async_trait::async_trait;
use nuncio_core::{CoreCommand, EventBus};
use nuncio_filter::FilterEngine;
use nuncio_store::db::DatabaseEngine;
use nuncio_store::vault::SecretManager;
use std::collections::HashMap;
use std::sync::{Arc, Mutex, PoisonError};
use std::time::{Duration, SystemTime};
use thiserror::Error;
use tokio::sync::{watch, Semaphore};
use tracing::Instrument;
/// Environment variable bounding how many account syncs may run at once.
pub const CONCURRENCY_ENV_VAR: &str = "NUNCIO_SYNC_CONCURRENCY";
/// Environment variable bounding a single account's sync, in whole seconds.
pub const TIMEOUT_ENV_VAR: &str = "NUNCIO_SYNC_TIMEOUT";
/// Default simultaneous-sync cap when [`CONCURRENCY_ENV_VAR`] is unset or
/// unparseable. Chosen to overlap a handful of accounts without letting an
/// unbounded fan-out open a socket per account at once.
pub const DEFAULT_CONCURRENCY: usize = 4;
/// Default per-account sync timeout when [`TIMEOUT_ENV_VAR`] is unset or
/// unparseable. Generous enough for a healthy full mailbox fetch, tight
/// enough that a hung account cannot stall others (or shutdown) indefinitely.
pub const DEFAULT_TIMEOUT: Duration = Duration::from_secs(300);
/// The outcome of a single dispatched account sync. Cloneable so a coalesced
/// result can fan out to every joined waiter.
type SyncOutcome = Result<usize, SyncDispatchError>;
/// Errors surfaced by [`SyncDispatcher::sync_account`].
#[derive(Debug, Clone, Error)]
pub enum SyncDispatchError {
/// The account's sync exceeded the configured per-account timeout and was
/// cancelled. Retryable: nothing about the dispatcher is left broken.
#[error("account '{account_id}' sync exceeded the {timeout:?} per-account timeout")]
Timeout {
/// The account whose sync timed out.
account_id: String,
/// The timeout that was exceeded.
timeout: Duration,
},
/// Shutdown was requested before or during the sync; no new work is
/// admitted and in-flight work is abandoned for the next start.
#[error("sync dispatcher is shutting down")]
ShuttingDown,
/// The in-flight sync ended without producing a result (e.g. its task was
/// dropped). Retryable.
#[error("account sync was cancelled before it produced a result")]
Cancelled,
/// The underlying account sync itself failed. Shared behind an [`Arc`] so
/// a coalesced failure can be reported to every joined waiter.
#[error(transparent)]
Sync(Arc<SyncError>),
}
/// Live sync phase of a single account, derived from the dispatcher's
/// in-flight map and last-result tracking for the daemon health surface.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AccountSyncPhase {
/// No sync in flight; the last attempt (if any) succeeded.
Idle,
/// A sync for the account is currently running.
Syncing,
/// No sync in flight, but the most recent attempt failed.
Error,
}
/// A snapshot of one account's sync state, combining its current phase with
/// the outcome of its most recent completed attempt. `last_success_at`
/// survives a subsequent failure so a transient error does not erase the
/// last known-good sync time.
#[derive(Debug, Clone)]
pub struct AccountSyncStatus {
/// Current phase.
pub phase: AccountSyncPhase,
/// When the account last completed a SUCCESSFUL sync, if ever since start.
pub last_success_at: Option<SystemTime>,
/// The most recent attempt's failure detail, present only when the last
/// attempt failed.
pub last_error: Option<String>,
}
/// The recorded outcome of an account's most recent completed sync attempt.
/// A success clears `last_error` and stamps `last_success_at`; a failure sets
/// `last_error` while leaving any earlier `last_success_at` intact.
#[derive(Default, Clone)]
struct AccountSyncRecord {
last_success_at: Option<SystemTime>,
last_error: Option<String>,
}
/// Injectable seam for the actual work of syncing one account. Production
/// uses [`ProductionAccountSyncer`]; tests inject an in-memory mock.
#[async_trait]
pub trait AccountSyncer: Send + Sync {
/// Sync exactly one account to completion, returning the number of
/// messages processed.
async fn sync_account(&self, account_id: &str) -> Result<usize, SyncError>;
}
/// Production [`AccountSyncer`]: delegates to [`crate::sync::run_account_sync`],
/// which resolves the account's credentials, builds a real mail backend, and
/// runs a full inbound sync with the standard engine-status lifecycle.
pub struct ProductionAccountSyncer {
db: Arc<DatabaseEngine>,
secrets: Arc<SecretManager>,
event_bus: Arc<EventBus>,
filter_engine: Arc<FilterEngine>,
}
impl ProductionAccountSyncer {
/// Build the production syncer from the daemon's shared subsystems.
pub fn new(
db: Arc<DatabaseEngine>,
secrets: Arc<SecretManager>,
event_bus: Arc<EventBus>,
filter_engine: Arc<FilterEngine>,
) -> Self {
Self {
db,
secrets,
event_bus,
filter_engine,
}
}
}
#[async_trait]
impl AccountSyncer for ProductionAccountSyncer {
async fn sync_account(&self, account_id: &str) -> Result<usize, SyncError> {
crate::sync::run_account_sync(
&self.db,
&self.secrets,
&self.event_bus,
&self.filter_engine,
account_id,
)
.await
}
}
/// Reads [`CONCURRENCY_ENV_VAR`], falling back to [`DEFAULT_CONCURRENCY`].
/// A value of zero (or unparseable) is treated as the default; the effective
/// cap is always at least one.
fn concurrency_from_env() -> usize {
std::env::var(CONCURRENCY_ENV_VAR)
.ok()
.and_then(|v| v.trim().parse::<usize>().ok())
.filter(|n| *n > 0)
.unwrap_or(DEFAULT_CONCURRENCY)
}
/// Reads [`TIMEOUT_ENV_VAR`] as whole seconds, falling back to
/// [`DEFAULT_TIMEOUT`]. A value of zero (or unparseable) is treated as the
/// default rather than "no timeout", so a misconfiguration can never disable
/// the safety bound.
fn timeout_from_env() -> Duration {
std::env::var(TIMEOUT_ENV_VAR)
.ok()
.and_then(|v| v.trim().parse::<u64>().ok())
.filter(|secs| *secs > 0)
.map(Duration::from_secs)
.unwrap_or(DEFAULT_TIMEOUT)
}
/// Shared inner state, held behind an [`Arc`] so the dispatcher is cheap to
/// clone and can be moved into the per-account tasks it spawns.
struct Inner {
syncer: Arc<dyn AccountSyncer>,
semaphore: Arc<Semaphore>,
/// Accounts with a sync currently in flight, mapped to a watch receiver
/// that publishes the sync's outcome once. A second request for the same
/// account clones the receiver and joins the in-flight sync.
inflight: Mutex<HashMap<String, watch::Receiver<Option<SyncOutcome>>>>,
/// The most recent completed outcome per account, retained after the
/// in-flight entry is cleared so the health surface can report each
/// account's last sync time and last error.
last_results: Mutex<HashMap<String, AccountSyncRecord>>,
per_account_timeout: Duration,
shutdown: ShutdownSignal,
}
impl Inner {
/// Run one account sync under the semaphore permit and per-account
/// timeout, racing both the permit wait and the sync itself against the
/// shutdown signal (mirroring the outbox executor). The permit is
/// released as soon as the sync resolves.
async fn run_guarded(&self, account_id: &str) -> SyncOutcome {
let mut shutdown = self.shutdown.clone();
let permit = tokio::select! {
biased;
_ = shutdown.wait() => return Err(SyncDispatchError::ShuttingDown),
acquired = self.semaphore.clone().acquire_owned() => match acquired {
Ok(permit) => permit,
// The semaphore is never closed in practice; treat a closed
// semaphore as a cancellation rather than fabricating success.
Err(_) => return Err(SyncDispatchError::Cancelled),
},
};
let outcome = tokio::select! {
biased;
_ = shutdown.wait() => Err(SyncDispatchError::ShuttingDown),
result = tokio::time::timeout(
self.per_account_timeout,
self.syncer.sync_account(account_id),
) => match result {
Ok(Ok(count)) => Ok(count),
Ok(Err(e)) => Err(SyncDispatchError::Sync(Arc::new(e))),
Err(_elapsed) => Err(SyncDispatchError::Timeout {
account_id: account_id.to_string(),
timeout: self.per_account_timeout,
}),
},
};
drop(permit);
outcome
}
}
/// The single entry point every account sync funnels through. Cheap to clone
/// (an [`Arc`] handle); every clone shares the same concurrency cap,
/// in-flight map, timeout, and shutdown signal.
#[derive(Clone)]
pub struct SyncDispatcher {
inner: Arc<Inner>,
}
impl SyncDispatcher {
/// Build a dispatcher with explicit bounds. `concurrency` is clamped up to
/// at least one so the dispatcher can always make progress.
pub fn new(
syncer: Arc<dyn AccountSyncer>,
shutdown: ShutdownSignal,
concurrency: usize,
per_account_timeout: Duration,
) -> Self {
let concurrency = concurrency.max(1);
Self {
inner: Arc::new(Inner {
syncer,
semaphore: Arc::new(Semaphore::new(concurrency)),
inflight: Mutex::new(HashMap::new()),
last_results: Mutex::new(HashMap::new()),
per_account_timeout,
shutdown,
}),
}
}
/// Build a dispatcher whose bounds come from the environment
/// ([`CONCURRENCY_ENV_VAR`] / [`TIMEOUT_ENV_VAR`]), falling back to
/// [`DEFAULT_CONCURRENCY`] / [`DEFAULT_TIMEOUT`].
pub fn from_env(syncer: Arc<dyn AccountSyncer>, shutdown: ShutdownSignal) -> Self {
Self::new(syncer, shutdown, concurrency_from_env(), timeout_from_env())
}
/// Dispatch a sync for a single account, applying bounded concurrency,
/// in-flight coalescing, the per-account timeout, and shutdown handling.
///
/// If a sync for `account_id` is already in flight, this joins it and
/// returns the SAME outcome rather than starting a duplicate fetch.
pub async fn sync_account(&self, account_id: &str) -> SyncOutcome {
if self.inner.shutdown.is_requested() {
return Err(SyncDispatchError::ShuttingDown);
}
// Either join the in-flight sync for this account or become its owner
// by inserting a fresh result channel and spawning the guarded run.
// The lock is released before awaiting, so it never spans `.await`.
let mut rx = {
let mut inflight = self
.inner
.inflight
.lock()
.unwrap_or_else(PoisonError::into_inner);
if let Some(existing) = inflight.get(account_id) {
existing.clone()
} else {
let (tx, rx) = watch::channel::<Option<SyncOutcome>>(None);
inflight.insert(account_id.to_string(), rx.clone());
let inner = self.inner.clone();
let id = account_id.to_string();
// The actual sync runs in a detached task, which does NOT
// inherit the caller's span automatically. Capture the current
// span (the RPC's correlation span when a `Mail/Sync` triggered
// this) and instrument the task with it, so the sync's logs are
// correlatable to the request that kicked it off. A request
// that COALESCES onto an already-running sync joins the owner's
// task and is therefore correlated to the owner's request_id,
// not its own.
let span = tracing::Span::current();
tokio::spawn(
async move {
let outcome = inner.run_guarded(&id).await;
// Record this attempt's outcome for the health surface
// before clearing the in-flight entry: a success stamps
// the last-synced time and clears any prior error; a
// failure records the error while preserving the last
// known-good sync time.
{
let mut results = inner
.last_results
.lock()
.unwrap_or_else(PoisonError::into_inner);
let record = results.entry(id.clone()).or_default();
match &outcome {
Ok(_) => {
record.last_success_at = Some(SystemTime::now());
record.last_error = None;
}
Err(e) => record.last_error = Some(e.to_string()),
}
}
// Remove the in-flight entry before publishing so a
// request arriving after completion starts a fresh sync
// rather than joining an already-finished one.
inner
.inflight
.lock()
.unwrap_or_else(PoisonError::into_inner)
.remove(&id);
let _ = tx.send(Some(outcome));
}
.instrument(span),
);
rx
}
};
// Clone the published outcome out of the borrowed watch guard before
// the guard (and `rx`) drop at the end of this scope.
let published = match rx.wait_for(Option::is_some).await {
Ok(guard) => guard.clone(),
// The sender was dropped without publishing (the owning task was
// itself dropped). Honestly report a cancellation, never success.
Err(_) => None,
};
published.unwrap_or(Err(SyncDispatchError::Cancelled))
}
/// Report the live sync state of a single account for the daemon health
/// surface: `Syncing` while a sync is in flight, otherwise `Error` if the
/// most recent attempt failed, else `Idle`. `last_success_at`/`last_error`
/// carry the genuine outcome of the last completed attempt (never
/// fabricated); an account that has not synced since daemon start reports
/// `Idle` with neither set.
pub fn sync_status(&self, account_id: &str) -> AccountSyncStatus {
let syncing = self
.inner
.inflight
.lock()
.unwrap_or_else(PoisonError::into_inner)
.contains_key(account_id);
let record = self
.inner
.last_results
.lock()
.unwrap_or_else(PoisonError::into_inner)
.get(account_id)
.cloned()
.unwrap_or_default();
let phase = if syncing {
AccountSyncPhase::Syncing
} else if record.last_error.is_some() {
AccountSyncPhase::Error
} else {
AccountSyncPhase::Idle
};
AccountSyncStatus {
phase,
last_success_at: record.last_success_at,
last_error: record.last_error,
}
}
/// Dispatch syncs for several accounts concurrently, each through the same
/// bounded/coalesced/timed per-account path, and return the total number
/// of messages synced across the accounts that succeeded. Per-account
/// failures are logged and do not abort the batch (the underlying
/// production syncer additionally reports them on the event bus).
pub async fn sync_all(&self, account_ids: Vec<String>) -> usize {
let mut set = tokio::task::JoinSet::new();
// Propagate the caller's span (the RPC correlation span for an
// all-accounts `Mail/Sync`) into each per-account task so the fan-out
// stays correlated to the triggering request.
let span = tracing::Span::current();
for account_id in account_ids {
let dispatcher = self.clone();
set.spawn(
async move {
let result = dispatcher.sync_account(&account_id).await;
(account_id, result)
}
.instrument(span.clone()),
);
}
let mut total = 0usize;
while let Some(joined) = set.join_next().await {
match joined {
Ok((_, Ok(count))) => total += count,
Ok((account_id, Err(e))) => {
tracing::warn!("dispatched sync for account '{account_id}' failed: {e}");
}
Err(join_err) => {
tracing::warn!("dispatched sync task failed to join: {join_err}");
}
}
}
total
}
}
/// Sync every configured mail account through `dispatcher`, wrapped in the
/// standard batch `SyncStarted{None}` / `SyncCompleted{None}` envelope.
///
/// DAV-protocol accounts have no inbound mail backend and are skipped (they
/// sync through their own domain service). This is the single production
/// entry point for an all-accounts sync, shared by the `Mail/Sync` RPC
/// (`account_id == None`) and the daemon's `CoreCommand::SyncAll` consumer, so
/// the two never launch account syncs through separate, differently-bounded
/// paths. Returns the total number of messages synced across all accounts.
pub async fn sync_all_configured(
dispatcher: &SyncDispatcher,
db: &DatabaseEngine,
event_bus: &EventBus,
) -> usize {
event_bus.process_command(CoreCommand::SyncAll);
let account_ids = match db.list_accounts().await {
Ok(accounts) => accounts
.into_iter()
.filter(|a| !a.is_dav())
.map(|a| a.id)
.collect::<Vec<_>>(),
Err(e) => {
event_bus.process_command(CoreCommand::ReportError {
message: format!("failed to list accounts for sync: {e}"),
});
Vec::new()
}
};
let total = dispatcher.sync_all(account_ids).await;
event_bus.complete_sync(None);
total
}
#[cfg(test)]
mod tests {
use super::*;
use nuncio_core::{AccountConfig, CoreEvent, DavTransport, JmapTransport, Transport};
use std::collections::HashSet;
use std::sync::atomic::{AtomicUsize, Ordering};
use tokio::time::{sleep, Duration};
/// In-memory [`AccountSyncer`] that records invocations, tracks peak
/// concurrency, and can be told to block indefinitely for named accounts
/// (to model a hung account) or to add a fixed per-call delay (to make
/// concurrent calls genuinely overlap).
struct MockSyncer {
calls: Mutex<Vec<String>>,
concurrent: AtomicUsize,
peak: AtomicUsize,
blocked: HashSet<String>,
gate: Arc<tokio::sync::Notify>,
per_call_delay: Option<Duration>,
fail: HashSet<String>,
}
impl MockSyncer {
fn new() -> Self {
Self {
calls: Mutex::new(Vec::new()),
concurrent: AtomicUsize::new(0),
peak: AtomicUsize::new(0),
blocked: HashSet::new(),
gate: Arc::new(tokio::sync::Notify::new()),
per_call_delay: None,
fail: HashSet::new(),
}
}
fn with_delay(mut self, delay: Duration) -> Self {
self.per_call_delay = Some(delay);
self
}
fn blocking(mut self, account_id: &str) -> Self {
self.blocked.insert(account_id.to_string());
self
}
fn failing(mut self, account_id: &str) -> Self {
self.fail.insert(account_id.to_string());
self
}
fn call_ids(&self) -> Vec<String> {
self.calls
.lock()
.unwrap_or_else(PoisonError::into_inner)
.clone()
}
fn call_count(&self) -> usize {
self.calls
.lock()
.unwrap_or_else(PoisonError::into_inner)
.len()
}
}
#[async_trait]
impl AccountSyncer for MockSyncer {
async fn sync_account(&self, account_id: &str) -> Result<usize, SyncError> {
self.calls
.lock()
.unwrap_or_else(PoisonError::into_inner)
.push(account_id.to_string());
let current = self.concurrent.fetch_add(1, Ordering::SeqCst) + 1;
self.peak.fetch_max(current, Ordering::SeqCst);
if self.blocked.contains(account_id) {
self.gate.notified().await;
} else if let Some(delay) = self.per_call_delay {
sleep(delay).await;
}
self.concurrent.fetch_sub(1, Ordering::SeqCst);
if self.fail.contains(account_id) {
return Err(SyncError::AccountNotFound(account_id.to_string()));
}
Ok(1)
}
}
fn dispatcher_with(
syncer: Arc<MockSyncer>,
concurrency: usize,
timeout: Duration,
) -> SyncDispatcher {
SyncDispatcher::new(syncer, ShutdownSignal::never(), concurrency, timeout)
}
#[tokio::test]
async fn bounds_simultaneous_syncs_to_the_configured_cap() {
let syncer = Arc::new(MockSyncer::new().with_delay(Duration::from_millis(80)));
let dispatcher = dispatcher_with(syncer.clone(), 2, Duration::from_secs(30));
let mut set = tokio::task::JoinSet::new();
for i in 0..6 {
let d = dispatcher.clone();
set.spawn(async move { d.sync_account(&format!("acct-{i}")).await });
}
while let Some(joined) = set.join_next().await {
assert!(matches!(joined, Ok(Ok(1))), "each distinct sync succeeds");
}
assert_eq!(
syncer.peak.load(Ordering::SeqCst),
2,
"no more than the configured cap of 2 syncs may run at once"
);
assert_eq!(syncer.call_count(), 6, "every distinct account is synced");
}
#[tokio::test]
async fn coalesces_a_duplicate_in_flight_sync_for_the_same_account() {
let syncer = Arc::new(MockSyncer::new().with_delay(Duration::from_millis(80)));
let dispatcher = dispatcher_with(syncer.clone(), 4, Duration::from_secs(30));
// Three concurrent requests for the SAME account, launched close
// enough together to all observe the first as still in flight.
let mut set = tokio::task::JoinSet::new();
for _ in 0..3 {
let d = dispatcher.clone();
set.spawn(async move { d.sync_account("acct-dup").await });
}
while let Some(joined) = set.join_next().await {
assert!(
matches!(joined, Ok(Ok(1))),
"every joined waiter sees the result"
);
}
assert_eq!(
syncer.call_ids(),
vec!["acct-dup".to_string()],
"the underlying sync must run exactly once, not once per request"
);
}
#[tokio::test]
async fn a_hung_account_does_not_block_other_accounts() {
// Account A blocks forever; account B must still complete. With the
// cap at 2 both can hold a permit at once, proving the fix for the
// head-of-line blocking regression.
let syncer = Arc::new(MockSyncer::new().blocking("acct-A"));
let dispatcher = dispatcher_with(syncer.clone(), 2, Duration::from_secs(30));
let d_a = dispatcher.clone();
let a = tokio::spawn(async move { d_a.sync_account("acct-A").await });
let b = dispatcher.sync_account("acct-B").await;
assert!(
matches!(b, Ok(1)),
"account B completes even while account A is hung, got {b:?}"
);
// A is still blocked; it has not resolved.
assert!(!a.is_finished(), "account A's sync is still in flight");
a.abort();
}
#[tokio::test]
async fn a_hung_account_times_out_without_poisoning_the_dispatcher() {
let syncer = Arc::new(MockSyncer::new().blocking("acct-hung"));
let dispatcher = dispatcher_with(syncer.clone(), 2, Duration::from_millis(60));
let hung = dispatcher.sync_account("acct-hung").await;
assert!(
matches!(hung, Err(SyncDispatchError::Timeout { .. })),
"a hung account surfaces a typed Timeout, got {hung:?}"
);
// The dispatcher is not poisoned: a subsequent sync of a healthy
// account still works.
let ok = dispatcher.sync_account("acct-healthy").await;
assert!(
matches!(ok, Ok(1)),
"a later sync still works after a timeout, got {ok:?}"
);
}
#[tokio::test]
async fn surfaces_the_underlying_sync_error_typed() {
let syncer = Arc::new(MockSyncer::new().failing("acct-bad"));
let dispatcher = dispatcher_with(syncer, 2, Duration::from_secs(30));
let err = dispatcher.sync_account("acct-bad").await;
assert!(
matches!(err, Err(SyncDispatchError::Sync(_))),
"an underlying sync failure surfaces as SyncDispatchError::Sync, got {err:?}"
);
}
#[tokio::test]
async fn sync_status_reports_idle_for_an_unknown_account() {
let syncer = Arc::new(MockSyncer::new());
let dispatcher = dispatcher_with(syncer, 2, Duration::from_secs(30));
let status = dispatcher.sync_status("never-seen");
assert_eq!(status.phase, AccountSyncPhase::Idle);
assert!(status.last_success_at.is_none());
assert!(status.last_error.is_none());
}
#[tokio::test]
async fn sync_status_reports_syncing_while_a_sync_is_in_flight() {
let syncer = Arc::new(MockSyncer::new().blocking("acct-live"));
let dispatcher = dispatcher_with(syncer.clone(), 2, Duration::from_secs(30));
let d = dispatcher.clone();
let handle = tokio::spawn(async move { d.sync_account("acct-live").await });
// Wait until the mock has actually entered the sync (it blocks there).
while syncer.call_count() == 0 {
sleep(Duration::from_millis(5)).await;
}
assert_eq!(
dispatcher.sync_status("acct-live").phase,
AccountSyncPhase::Syncing
);
syncer.gate.notify_one();
let _ = handle.await;
}
#[tokio::test]
async fn sync_status_reports_idle_with_last_synced_after_success() {
let syncer = Arc::new(MockSyncer::new());
let dispatcher = dispatcher_with(syncer, 2, Duration::from_secs(30));
assert!(matches!(dispatcher.sync_account("acct-ok").await, Ok(1)));
let status = dispatcher.sync_status("acct-ok");
assert_eq!(status.phase, AccountSyncPhase::Idle);
assert!(status.last_success_at.is_some());
assert!(status.last_error.is_none());
}
#[tokio::test]
async fn sync_status_reports_error_after_a_failed_attempt() {
let syncer = Arc::new(MockSyncer::new().failing("acct-bad"));
let dispatcher = dispatcher_with(syncer, 2, Duration::from_secs(30));
assert!(matches!(
dispatcher.sync_account("acct-bad").await,
Err(SyncDispatchError::Sync(_))
));
let status = dispatcher.sync_status("acct-bad");
assert_eq!(status.phase, AccountSyncPhase::Error);
assert!(status.last_error.is_some());
assert!(status.last_success_at.is_none());
}
#[tokio::test]
async fn rejects_new_work_once_shutdown_is_requested() {
let event_bus = Arc::new(EventBus::new());
let (controller, shutdown) = crate::lifecycle::ShutdownController::new(event_bus);
let syncer = Arc::new(MockSyncer::new());
let dispatcher = SyncDispatcher::new(syncer.clone(), shutdown, 2, Duration::from_secs(30));
controller.trigger();
let rejected = dispatcher.sync_account("acct-late").await;
assert!(
matches!(rejected, Err(SyncDispatchError::ShuttingDown)),
"no new sync is admitted after shutdown, got {rejected:?}"
);
assert_eq!(
syncer.call_count(),
0,
"the underlying syncer must not even be invoked after shutdown"
);
}
#[tokio::test]
async fn shutdown_unblocks_an_in_flight_sync_promptly() {
let event_bus = Arc::new(EventBus::new());
let (controller, shutdown) = crate::lifecycle::ShutdownController::new(event_bus);
let syncer = Arc::new(MockSyncer::new().blocking("acct-drain"));
let dispatcher = SyncDispatcher::new(syncer, shutdown, 2, Duration::from_secs(300));
let d = dispatcher.clone();
let handle = tokio::spawn(async move { d.sync_account("acct-drain").await });
// Let the sync reach its blocking point, then request shutdown.
sleep(Duration::from_millis(30)).await;
controller.trigger();
let outcome = tokio::time::timeout(Duration::from_secs(5), handle)
.await
.expect("in-flight sync must unblock on shutdown, not hang")
.expect("task joins");
assert!(
matches!(outcome, Err(SyncDispatchError::ShuttingDown)),
"a sync racing shutdown resolves as ShuttingDown, got {outcome:?}"
);
}
fn jmap_account(id: &str) -> AccountConfig {
AccountConfig {
id: id.to_string(),
name: "Test".to_string(),
email_address: format!("{id}@nuncio.mx"),
keyring_secret_key: format!("nuncio/{id}"),
sync_interval_secs: 60,
filters_enabled: false,
transport: Transport::Jmap(JmapTransport {
endpoint_host: "jmap.nuncio.mx".to_string(),
}),
}
}
fn caldav_account(id: &str) -> AccountConfig {
AccountConfig {
transport: Transport::Dav(DavTransport {
collection_url: "https://dav.nuncio.mx/cal/".to_string(),
}),
..jmap_account(id)
}
}
#[tokio::test]
async fn sync_all_configured_dispatches_every_mail_account_and_skips_dav() {
let (db, _dir) = DatabaseEngine::connect_ephemeral()
.await
.expect("ephemeral db");
db.save_account(&jmap_account("acct-mail-1"))
.await
.expect("save mail account 1");
db.save_account(&jmap_account("acct-mail-2"))
.await
.expect("save mail account 2");
db.save_account(&caldav_account("acct-dav"))
.await
.expect("save dav account");
let event_bus = EventBus::new();
let mut events = event_bus.subscribe_events();
let syncer = Arc::new(MockSyncer::new());
let dispatcher = dispatcher_with(syncer.clone(), 4, Duration::from_secs(30));
let total = sync_all_configured(&dispatcher, &db, &event_bus).await;
assert_eq!(total, 2, "both mail accounts synced one message each");
let mut synced: Vec<String> = syncer.call_ids();
synced.sort();
assert_eq!(
synced,
vec!["acct-mail-1".to_string(), "acct-mail-2".to_string()],
"only the mail accounts are dispatched; the CalDAV account is skipped"
);
// The batch is wrapped in a single all-accounts start/complete pair.
assert_eq!(
events.recv().await.expect("start event"),
CoreEvent::SyncStarted { account_id: None }
);
}
}
crates/nunciod/src/test_tracing.rs
+200
View File
@@ -0,0 +1,200 @@
//! Shared in-test `tracing` capture helper.
//!
//! Records every span (with its fields) and every event (with its level,
//! fields, and message) into an in-memory buffer, so tests can assert on what
//! the code logged -- including the negative assertion that a secret never
//! appears in any recorded field. Fully offline: no I/O.
//!
//! The subscriber is installed **once, globally**, and events are routed to a
//! per-thread recorder. That split is deliberate and load-bearing: a
//! thread-local subscriber cannot work here, because `tracing` caches each
//! callsite's `Interest` process-wide on first use. See
//! [`install_global_capture`].
#![cfg(test)]
use std::cell::RefCell;
use std::collections::HashMap;
use std::sync::{Arc, Mutex, OnceLock};
use tracing::field::{Field, Visit};
use tracing::span::Attributes;
use tracing::{Event, Id, Level, Subscriber};
use tracing_subscriber::layer::Context as LayerContext;
use tracing_subscriber::prelude::*;
use tracing_subscriber::registry::LookupSpan;
/// A captured span: its name and the string-rendered fields it opened with.
#[derive(Debug, Clone)]
pub(crate) struct RecordedSpan {
pub name: String,
pub fields: HashMap<String, String>,
}
/// A captured event: its level and its string-rendered fields (including the
/// `message`).
#[derive(Debug, Clone)]
pub(crate) struct RecordedEvent {
pub level: Level,
pub fields: HashMap<String, String>,
}
impl RecordedEvent {
/// The event's `message` field, or the empty string if it had none.
pub fn message(&self) -> &str {
self.fields.get("message").map(String::as_str).unwrap_or("")
}
}
/// In-memory sink of everything the subscriber captured during a test.
#[derive(Debug, Default)]
pub(crate) struct Recorder {
spans: Mutex<Vec<RecordedSpan>>,
events: Mutex<Vec<RecordedEvent>>,
}
impl Recorder {
/// A snapshot of every captured span.
pub fn spans(&self) -> Vec<RecordedSpan> {
self.spans.lock().unwrap_or_else(|e| e.into_inner()).clone()
}
/// A snapshot of every captured event.
pub fn events(&self) -> Vec<RecordedEvent> {
self.events
.lock()
.unwrap_or_else(|e| e.into_inner())
.clone()
}
/// Every field value recorded across all spans and events, so a test can
/// assert that a secret never appears anywhere in the telemetry.
pub fn all_field_values(&self) -> Vec<String> {
let mut values = Vec::new();
for span in self.spans().iter() {
values.extend(span.fields.values().cloned());
}
for event in self.events().iter() {
values.extend(event.fields.values().cloned());
}
values
}
}
struct FieldVisitor(HashMap<String, String>);
impl Visit for FieldVisitor {
fn record_str(&mut self, field: &Field, value: &str) {
self.0.insert(field.name().to_string(), value.to_string());
}
fn record_debug(&mut self, field: &Field, value: &dyn std::fmt::Debug) {
self.0
.entry(field.name().to_string())
.or_insert_with(|| format!("{value:?}"));
}
}
thread_local! {
/// The recorder collecting telemetry for the test running on THIS thread,
/// or `None` on any thread not currently inside [`with_recorder`].
///
/// Routing per thread is what keeps the single global subscriber from
/// letting concurrent tests read each other's telemetry.
static ACTIVE_RECORDER: RefCell<Option<Arc<Recorder>>> = const { RefCell::new(None) };
}
/// Run `f` against the recorder installed for this thread, if any.
fn with_active<R>(f: impl FnOnce(&Recorder) -> R) -> Option<R> {
ACTIVE_RECORDER.with(|slot| slot.borrow().as_ref().map(|rec| f(rec)))
}
struct CaptureLayer;
impl<S> tracing_subscriber::Layer<S> for CaptureLayer
where
S: Subscriber + for<'a> LookupSpan<'a>,
{
fn on_new_span(&self, attrs: &Attributes<'_>, _id: &Id, _ctx: LayerContext<'_, S>) {
with_active(|rec| {
let mut visitor = FieldVisitor(HashMap::new());
attrs.record(&mut visitor);
rec.spans
.lock()
.unwrap_or_else(|e| e.into_inner())
.push(RecordedSpan {
name: attrs.metadata().name().to_string(),
fields: visitor.0,
});
});
}
fn on_event(&self, event: &Event<'_>, _ctx: LayerContext<'_, S>) {
with_active(|rec| {
let mut visitor = FieldVisitor(HashMap::new());
event.record(&mut visitor);
rec.events
.lock()
.unwrap_or_else(|e| e.into_inner())
.push(RecordedEvent {
level: *event.metadata().level(),
fields: visitor.0,
});
});
}
}
/// Install the capturing subscriber as the process-wide default, exactly once.
///
/// It MUST be global rather than thread-local, and this is the whole reason the
/// helper is shaped this way. `tracing` resolves each callsite's `Interest`
/// **once for the process** and caches it; a callsite first executed on a thread
/// with no subscriber caches `Interest::never()` and is then skipped forever --
/// before any later thread-local subscriber is ever consulted. Under a parallel
/// test harness, whichever thread reaches a callsite first therefore decides
/// whether any other test can ever capture it, which made every log-capture
/// assertion in this crate a coin flip.
///
/// A single global subscriber removes the race: every callsite registers against
/// a real subscriber, so interest is always enabled, and per-thread routing
/// (`ACTIVE_RECORDER`) decides what is actually recorded.
fn install_global_capture() {
static INSTALLED: OnceLock<()> = OnceLock::new();
INSTALLED.get_or_init(|| {
let subscriber = tracing_subscriber::registry().with(CaptureLayer);
// A global default may already be set by another component of the same
// test binary; capturing still works through the thread-local routing
// above, so a lost race here is not fatal.
let _ = tracing::subscriber::set_global_default(subscriber);
// Callsites executed before this point resolved their interest against
// the no-op dispatcher and cached "never". Recompute so they can be
// captured from here on. Done once, at install -- not per test.
tracing::callsite::rebuild_interest_cache();
});
}
/// Resets this thread's recorder slot on drop, so a panicking test cannot leak
/// its recorder into whatever the harness runs next on the same thread.
struct RecorderGuard;
impl Drop for RecorderGuard {
fn drop(&mut self) {
ACTIVE_RECORDER.with(|slot| *slot.borrow_mut() = None);
}
}
/// Runs `f` with telemetry from THIS thread captured, returning the recorder
/// alongside `f`'s result.
///
/// Only this thread's telemetry is recorded, so any async work driven inside
/// `f` must run on the calling thread (e.g. a current-thread runtime's
/// `block_on`) to be captured -- unchanged from the previous thread-local
/// implementation. What changed is that the subscriber itself is now global;
/// see [`install_global_capture`] for why that is load-bearing.
pub(crate) fn with_recorder<T>(f: impl FnOnce() -> T) -> (Arc<Recorder>, T) {
install_global_capture();
let recorder = Arc::new(Recorder::default());
ACTIVE_RECORDER.with(|slot| *slot.borrow_mut() = Some(recorder.clone()));
let _guard = RecorderGuard;
let value = f();
(recorder, value)
}
crates/nunciod/tests/accounts_lifecycle_e2e_test.rs
+43 -19
View File
@@ -29,7 +29,7 @@
use nuncio_core::EventBus;
use nuncio_filter::FilterEngine;
use nuncio_proto::v1::{
AccountConfig, AccountProtocol, AddAccountRequest, ListAccountsRequest, RemoveAccountRequest,
AccountConfig, AddAccountRequest, ListAccountsRequest, RemoveAccountRequest,
TestAccountConnectionRequest, TlsMode, UpdateAccountRequest,
};
use nuncio_store::db::DatabaseEngine;
@@ -42,6 +42,15 @@ use nunciod::grpc::{
use std::sync::Arc;
use tokio::net::TcpListener;
/// Extract the IMAP/SMTP transport sub-message from a wire `AccountConfig`,
/// panicking if the account is not an IMAP/SMTP account (test-only helper).
fn imap_smtp_transport(config: &AccountConfig) -> &nuncio_proto::v1::ImapSmtpTransport {
match &config.transport {
Some(nuncio_proto::v1::account_config::Transport::ImapSmtp(t)) => t,
_ => panic!("expected an IMAP/SMTP account"),
}
}
const ACCOUNT_ID: &str = "acct-lifecycle-e2e-1";
const ACCOUNT_EMAIL: &str = "lifecycle-e2e@nuncio.mx";
const KEYRING_KEY: &str = "nuncio/acct-lifecycle-e2e-1";
@@ -79,16 +88,19 @@ fn sample_account_config() -> AccountConfig {
id: ACCOUNT_ID.to_string(),
name: "Lifecycle E2E Account".to_string(),
email_address: ACCOUNT_EMAIL.to_string(),
protocol: AccountProtocol::ImapSmtp.into(),
server_host: "imap.nuncio.mx".to_string(),
server_port: 143,
use_tls: false,
imap_tls_mode: TlsMode::StartTls.into(),
smtp_tls_mode: TlsMode::Plain.into(),
keyring_secret_key: KEYRING_KEY.to_string(),
sync_interval_secs: 60,
smtp_host: "smtp.nuncio.mx".to_string(),
smtp_port: 25,
sync_interval: Some(nuncio_proto::time::duration_from_secs(60)),
filters_enabled: false,
transport: Some(nuncio_proto::v1::account_config::Transport::ImapSmtp(
nuncio_proto::v1::ImapSmtpTransport {
imap_host: "imap.nuncio.mx".to_string(),
imap_port: 143,
imap_tls_mode: TlsMode::StartTls.into(),
smtp_host: "127.0.0.1".to_string(),
smtp_port: 25,
smtp_tls_mode: TlsMode::Plain.into(),
},
)),
}
}
@@ -163,7 +175,13 @@ async fn account_lifecycle_round_trips_over_authenticated_grpc_offline() {
.await
.expect("add_account succeeds")
.into_inner();
assert_eq!(add_response.id, ACCOUNT_ID);
assert_eq!(
add_response
.config
.expect("response carries the created config")
.id,
ACCOUNT_ID
);
// ---- (b) ListAccounts round-trips BOTH distinct TLS modes ----
let listed = accounts_client
@@ -173,13 +191,18 @@ async fn account_lifecycle_round_trips_over_authenticated_grpc_offline() {
.into_inner()
.accounts;
assert_eq!(listed.len(), 1);
assert_eq!(listed[0].imap_tls_mode(), TlsMode::StartTls);
assert_eq!(listed[0].smtp_tls_mode(), TlsMode::Plain);
let listed_t = imap_smtp_transport(&listed[0]);
assert_eq!(listed_t.imap_tls_mode(), TlsMode::StartTls);
assert_eq!(listed_t.smtp_tls_mode(), TlsMode::Plain);
// ---- (c) UpdateAccount genuinely mutates the persisted config ----
let mut updated = sample_account_config();
updated.name = "Renamed Lifecycle Account".to_string();
updated.imap_tls_mode = TlsMode::ImplicitTls.into();
if let Some(nuncio_proto::v1::account_config::Transport::ImapSmtp(t)) =
updated.transport.as_mut()
{
t.imap_tls_mode = TlsMode::ImplicitTls.into();
}
accounts_client
.update_account(UpdateAccountRequest {
config: Some(updated),
@@ -196,14 +219,15 @@ async fn account_lifecycle_round_trips_over_authenticated_grpc_offline() {
.accounts;
assert_eq!(relisted.len(), 1);
assert_eq!(relisted[0].name, "Renamed Lifecycle Account");
assert_eq!(relisted[0].imap_tls_mode(), TlsMode::ImplicitTls);
let relisted_t = imap_smtp_transport(&relisted[0]);
assert_eq!(relisted_t.imap_tls_mode(), TlsMode::ImplicitTls);
// Untouched fields survive the update.
assert_eq!(relisted[0].smtp_tls_mode(), TlsMode::Plain);
assert_eq!(relisted_t.smtp_tls_mode(), TlsMode::Plain);
// ---- (d) TestAccountConnection reports the GENUINE per-protocol result ----
let report = accounts_client
.test_account_connection(TestAccountConnectionRequest {
id: ACCOUNT_ID.to_string(),
account_id: ACCOUNT_ID.to_string(),
})
.await
.expect("test_account_connection succeeds")
@@ -219,7 +243,7 @@ async fn account_lifecycle_round_trips_over_authenticated_grpc_offline() {
// ---- (e) RemoveAccount genuinely deletes it ----
accounts_client
.remove_account(RemoveAccountRequest {
id: ACCOUNT_ID.to_string(),
account_id: ACCOUNT_ID.to_string(),
})
.await
.expect("remove_account succeeds");
@@ -236,7 +260,7 @@ async fn account_lifecycle_round_trips_over_authenticated_grpc_offline() {
// fabricated success.
let err = accounts_client
.test_account_connection(TestAccountConnectionRequest {
id: ACCOUNT_ID.to_string(),
account_id: ACCOUNT_ID.to_string(),
})
.await
.expect_err("testing a removed account must fail");
crates/nunciod/tests/caldav_production_e2e_test.rs
+241
View File
@@ -0,0 +1,241 @@
//! Full-daemon offline end-to-end test for the PRODUCTION CalDAV sync path.
//!
//! Unlike `calendar_e2e_test.rs`, which injects a `MockCalendarBackend` through
//! the [`nunciod::grpc::CalendarEngineOverrides`] seam, this test drives the
//! real production path with NO injected backend: it configures a first-class
//! CalDAV account over the authenticated `Accounts/AddAccount` RPC (protocol
//! CALDAV + a `collection_url` pointing at a WireMock server + a keyring
//! credential), then calls `Calendar/Sync`. The daemon resolves the configured
//! CalDAV account, reads its credential from the (mock) vault, builds a REAL
//! [`nuncio_cal::CalDavClient`] from the persisted collection URL, issues a
//! genuine CalDAV `REPORT` against WireMock, and persists the fetched events.
//!
//! This proves the production wiring end to end while remaining fully offline:
//! WireMock stands in for the remote CalDAV server, and a
//! [`SecretManager::mock`] vault stands in for the OS keyring -- no real
//! network and no real keyring are ever touched.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use nuncio_core::EventBus;
use nuncio_filter::FilterEngine;
use nuncio_proto::v1::{
AccountConfig as AccountConfigProto, AddAccountRequest, CalendarSyncRequest, ListEventsRequest,
};
use nuncio_store::db::DatabaseEngine;
use nuncio_store::vault::{SecretManager, GRPC_TOKEN_ACCOUNT};
use nunciod::grpc::{
serve_on_listener_with_overrides, AccountsEngineOverrides, CalendarEngineOverrides,
ContactsEngineOverrides, MailEngineOverrides,
};
use std::sync::Arc;
use tokio::net::TcpListener;
use tonic::Code;
use wiremock::matchers::{method, path};
use wiremock::{Mock, MockServer, ResponseTemplate};
const ACCOUNT_ID: &str = "acct-caldav-e2e-1";
const ACCOUNT_EMAIL: &str = "james.maes@nuncio.mx";
const KEYRING_KEY: &str = "nuncio/acct-caldav-e2e-1";
const CALDAV_PASSWORD: &str = "wiremock-app-token";
const CALENDAR_ID: &str = "cal-work";
/// A canned CalDAV `multistatus` response with two VEVENTs, mirroring the
/// transport-level fixture used by `nuncio-cal`'s own WireMock test.
fn multistatus_body() -> String {
r#"<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:" xmlns:c="urn:ietf:params:xml:ns:caldav">
<d:response>
<d:href>/calendars/work/standup.ics</d:href>
<d:propstat>
<d:prop>
<c:calendar-data>BEGIN:VCALENDAR
VERSION:2.0
BEGIN:VEVENT
SUMMARY:Production CalDAV Standup
LOCATION:Conference Room B
DTSTART:20240715T130000Z
DTEND:20240715T140000Z
END:VEVENT
END:VCALENDAR</c:calendar-data>
</d:prop>
</d:propstat>
</d:response>
<d:response>
<d:href>/calendars/work/review.ics</d:href>
<d:propstat>
<d:prop>
<c:calendar-data>BEGIN:VCALENDAR
VERSION:2.0
BEGIN:VEVENT
SUMMARY:Production UTC Review
DTSTART:20240301T120000Z
DTEND:20240301T130000Z
END:VEVENT
END:VCALENDAR</c:calendar-data>
</d:prop>
</d:propstat>
</d:response>
</d:multistatus>"#
.to_string()
}
/// Boots the real daemon gRPC server with DEFAULT overrides (no injected
/// calendar backend), backed by a real temp file-backed [`DatabaseEngine`] and
/// the given mock vault. Returns the server address and the minted bearer
/// token; the `TempDir` guard is returned so the caller controls its lifetime.
async fn boot_daemon(
secrets: Arc<SecretManager>,
) -> (std::net::SocketAddr, String, tempfile::TempDir) {
let dir = tempfile::tempdir().expect("create temp dir for file-backed db");
let db_path = dir.path().join("caldav_prod_e2e.db");
let db = Arc::new(
DatabaseEngine::connect_file(&db_path, &secrets)
.await
.expect("open temp file-backed database"),
);
let token_bytes = secrets
.get_or_create_key_bytes(GRPC_TOKEN_ACCOUNT, 32)
.expect("mint gRPC bearer token from mock vault");
let token = hex::encode(token_bytes);
let listener = TcpListener::bind("127.0.0.1:0")
.await
.expect("bind ephemeral loopback port");
let addr = listener.local_addr().expect("listener has local addr");
let event_bus = Arc::new(EventBus::new());
let server_secrets = secrets.clone();
let server_token = token.clone();
let server_filter_engine = Arc::new(FilterEngine::new(Vec::new()).expect("empty rule set"));
tokio::spawn(async move {
let _ = serve_on_listener_with_overrides(
listener,
event_bus,
db,
server_filter_engine,
server_secrets,
server_token,
MailEngineOverrides::default(),
// No injected calendar backend: exercise the real production path.
CalendarEngineOverrides::default(),
ContactsEngineOverrides::default(),
AccountsEngineOverrides::default(),
)
.await;
});
(addr, token, dir)
}
/// The full offline proof of the production CalDAV sync path: configure a
/// CalDAV account over the real `Accounts` API, then `Calendar/Sync` builds a
/// real `CalDavClient` and syncs events fetched from WireMock into the store,
/// immediately visible to `ListEvents` -- all over the authenticated gRPC API.
#[tokio::test]
async fn caldav_production_sync_builds_real_client_and_persists_events() {
let mock_server = MockServer::start().await;
Mock::given(method("REPORT"))
.and(path("/calendars/work/"))
.respond_with(ResponseTemplate::new(207).set_body_string(multistatus_body()))
.mount(&mock_server)
.await;
let collection_url = format!("{}/calendars/work/", mock_server.uri());
let secrets = Arc::new(SecretManager::mock());
let (addr, token, _db_dir) = boot_daemon(secrets).await;
let addr_str = addr.to_string();
// ---- Configure a first-class CalDAV account over the Accounts API ----
let mut accounts_client = nuncio_proto::client::connect_accounts(&addr_str, &token)
.await
.expect("accounts client connects");
accounts_client
.add_account(AddAccountRequest {
config: Some(AccountConfigProto {
id: ACCOUNT_ID.to_string(),
name: "Work Calendar".to_string(),
email_address: ACCOUNT_EMAIL.to_string(),
keyring_secret_key: KEYRING_KEY.to_string(),
sync_interval: Some(nuncio_proto::time::duration_from_secs(300)),
filters_enabled: false,
transport: Some(nuncio_proto::v1::account_config::Transport::Dav(
nuncio_proto::v1::DavTransport {
collection_url: collection_url.clone(),
},
)),
}),
password: CALDAV_PASSWORD.to_string(),
})
.await
.expect("CalDAV account is accepted and persisted");
// ---- Calendar/Sync runs the PRODUCTION path (no injected backend) ----
let mut calendar_client = nuncio_proto::client::connect_calendar(&addr_str, &token)
.await
.expect("calendar client connects");
let sync_response = calendar_client
.sync(CalendarSyncRequest {
account_id: ACCOUNT_ID.to_string(),
calendar_id: CALENDAR_ID.to_string(),
start_window: Some(nuncio_proto::time::timestamp_from_unix_secs(1_700_000_000)),
end_window: Some(nuncio_proto::time::timestamp_from_unix_secs(1_800_000_000)),
})
.await
.expect("production CalDAV sync succeeds")
.into_inner();
assert_eq!(sync_response.synced_count, 2);
// Immediately visible: `Sync` awaits full fetch-and-persist completion.
let list_response = calendar_client
.list_events(ListEventsRequest {
account_id: ACCOUNT_ID.to_string(),
calendar_id: CALENDAR_ID.to_string(),
start_window: Some(nuncio_proto::time::timestamp_from_unix_secs(1_700_000_000)),
end_window: Some(nuncio_proto::time::timestamp_from_unix_secs(1_800_000_000)),
page_size: 0,
page_token: String::new(),
})
.await
.expect("list_events succeeds")
.into_inner();
assert_eq!(list_response.events.len(), 2);
let summaries: Vec<&str> = list_response
.events
.iter()
.map(|e| e.summary.as_str())
.collect();
assert!(summaries.contains(&"Production CalDAV Standup"));
assert!(summaries.contains(&"Production UTC Review"));
// ---- With no CalDAV account matching, the honest error still fires ----
let missing = calendar_client
.sync(CalendarSyncRequest {
account_id: "acct-does-not-exist".to_string(),
calendar_id: CALENDAR_ID.to_string(),
start_window: Some(nuncio_proto::time::timestamp_from_unix_secs(1_700_000_000)),
end_window: Some(nuncio_proto::time::timestamp_from_unix_secs(1_800_000_000)),
})
.await
.expect_err("a sync targeting an unconfigured account must fail honestly");
assert_eq!(missing.code(), Code::FailedPrecondition);
// ---- An unauthenticated call to Calendar is rejected ----
let mut unauthenticated_client =
nuncio_proto::v1::calendar_client::CalendarClient::connect(format!("http://{addr_str}"))
.await
.expect("client connects at the transport level without any auth");
let err = unauthenticated_client
.list_events(ListEventsRequest {
account_id: ACCOUNT_ID.to_string(),
calendar_id: CALENDAR_ID.to_string(),
start_window: Some(nuncio_proto::time::timestamp_from_unix_secs(0)),
end_window: Some(nuncio_proto::time::timestamp_from_unix_secs(i64::MAX)),
page_size: 0,
page_token: String::new(),
})
.await
.expect_err("missing bearer token must be rejected");
assert_eq!(err.code(), Code::Unauthenticated);
}
crates/nunciod/tests/calendar_e2e_test.rs
+10 -6
View File
@@ -138,8 +138,8 @@ async fn calendar_sync_then_list_and_get_round_trip_over_authenticated_grpc() {
.sync(CalendarSyncRequest {
account_id: ACCOUNT_ID.to_string(),
calendar_id: CALENDAR_ID.to_string(),
start_window: 1_699_999_000,
end_window: 1_700_020_000,
start_window: Some(nuncio_proto::time::timestamp_from_unix_secs(1_699_999_000)),
end_window: Some(nuncio_proto::time::timestamp_from_unix_secs(1_700_020_000)),
})
.await
.expect("sync succeeds")
@@ -152,8 +152,10 @@ async fn calendar_sync_then_list_and_get_round_trip_over_authenticated_grpc() {
.list_events(ListEventsRequest {
account_id: ACCOUNT_ID.to_string(),
calendar_id: CALENDAR_ID.to_string(),
start_window: 1_699_999_000,
end_window: 1_700_020_000,
start_window: Some(nuncio_proto::time::timestamp_from_unix_secs(1_699_999_000)),
end_window: Some(nuncio_proto::time::timestamp_from_unix_secs(1_700_020_000)),
page_size: 0,
page_token: String::new(),
})
.await
.expect("list_events succeeds")
@@ -187,8 +189,10 @@ async fn calendar_sync_then_list_and_get_round_trip_over_authenticated_grpc() {
.list_events(ListEventsRequest {
account_id: ACCOUNT_ID.to_string(),
calendar_id: CALENDAR_ID.to_string(),
start_window: 0,
end_window: i64::MAX,
start_window: Some(nuncio_proto::time::timestamp_from_unix_secs(0)),
end_window: Some(nuncio_proto::time::timestamp_from_unix_secs(i64::MAX)),
page_size: 0,
page_token: String::new(),
})
.await
.expect_err("missing bearer token must be rejected");
crates/nunciod/tests/carddav_production_e2e_test.rs
+271
View File
@@ -0,0 +1,271 @@
//! Full-daemon offline end-to-end test for the PRODUCTION CardDAV sync path.
//!
//! Unlike `contacts_e2e_test.rs`, which injects a `MockContactsBackend` through
//! the [`nunciod::grpc::ContactsEngineOverrides`] seam, this test drives the
//! real production path with NO injected backend: it configures a first-class
//! CardDAV account over the authenticated `Accounts/AddAccount` RPC (protocol
//! CARDDAV + a `collection_url` pointing at a WireMock server + a keyring
//! credential), then calls `Contacts/Sync`. The daemon resolves the configured
//! CardDAV account, reads its credential from the (mock) vault, builds a REAL
//! [`nuncio_contacts::CardDavClient`] from the persisted collection URL, issues
//! a genuine CardDAV `REPORT` against WireMock, and persists the fetched
//! contacts.
//!
//! This proves the production wiring end to end while remaining fully offline:
//! WireMock stands in for the remote CardDAV server, and a
//! [`SecretManager::mock`] vault stands in for the OS keyring -- no real
//! network and no real keyring are ever touched.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use nuncio_core::EventBus;
use nuncio_filter::FilterEngine;
use nuncio_proto::v1::{
AccountConfig as AccountConfigProto, AddAccountRequest, ContactsSyncRequest,
ListContactsRequest,
};
use nuncio_store::db::DatabaseEngine;
use nuncio_store::vault::{SecretManager, GRPC_TOKEN_ACCOUNT};
use nunciod::grpc::{
serve_on_listener_with_overrides, AccountsEngineOverrides, CalendarEngineOverrides,
ContactsEngineOverrides, MailEngineOverrides,
};
use std::sync::Arc;
use tokio::net::TcpListener;
use tonic::Code;
use wiremock::matchers::{method, path};
use wiremock::{Mock, MockServer, ResponseTemplate};
const ACCOUNT_ID: &str = "acct-carddav-e2e-1";
const ACCOUNT_EMAIL: &str = "james.maes@nuncio.mx";
const KEYRING_KEY: &str = "nuncio/acct-carddav-e2e-1";
const CARDDAV_PASSWORD: &str = "wiremock-app-token";
/// A mail account added alongside the CardDAV one, to prove `Contacts/Sync`
/// refuses an account that is not a CardDAV account rather than silently
/// syncing nothing.
const MAIL_ACCOUNT_ID: &str = "acct-carddav-e2e-mail";
/// A canned CardDAV `multistatus` response with two vCards, mirroring the
/// transport-level fixture used by `nuncio-contacts`'s own WireMock test.
fn multistatus_body() -> String {
r#"<?xml version="1.0" encoding="utf-8"?>
<d:multistatus xmlns:d="DAV:" xmlns:card="urn:ietf:params:xml:ns:carddav">
<d:response>
<d:href>/addressbooks/contacts/james.vcf</d:href>
<d:propstat>
<d:prop>
<card:address-data>BEGIN:VCARD
VERSION:4.0
FN:Production CardDAV Maes
N:Maes;James;;;
ORG:KofTwentyTwo
TITLE:Founder
EMAIL;TYPE=WORK:james.maes@kof22.com
END:VCARD</card:address-data>
</d:prop>
</d:propstat>
</d:response>
<d:response>
<d:href>/addressbooks/contacts/alice.vcf</d:href>
<d:propstat>
<d:prop>
<card:address-data>BEGIN:VCARD
VERSION:4.0
FN:Production CardDAV Dev
EMAIL:alice@nuncio.mx
END:VCARD</card:address-data>
</d:prop>
</d:propstat>
</d:response>
</d:multistatus>"#
.to_string()
}
/// Boots the real daemon gRPC server with DEFAULT overrides (no injected
/// contacts backend), backed by a real temp file-backed [`DatabaseEngine`] and
/// the given mock vault. Returns the server address and the minted bearer
/// token; the `TempDir` guard is returned so the caller controls its lifetime.
async fn boot_daemon(
secrets: Arc<SecretManager>,
) -> (std::net::SocketAddr, String, tempfile::TempDir) {
let dir = tempfile::tempdir().expect("create temp dir for file-backed db");
let db_path = dir.path().join("carddav_prod_e2e.db");
let db = Arc::new(
DatabaseEngine::connect_file(&db_path, &secrets)
.await
.expect("open temp file-backed database"),
);
let token_bytes = secrets
.get_or_create_key_bytes(GRPC_TOKEN_ACCOUNT, 32)
.expect("mint gRPC bearer token from mock vault");
let token = hex::encode(token_bytes);
let listener = TcpListener::bind("127.0.0.1:0")
.await
.expect("bind ephemeral loopback port");
let addr = listener.local_addr().expect("listener has local addr");
let event_bus = Arc::new(EventBus::new());
let server_secrets = secrets.clone();
let server_token = token.clone();
let server_filter_engine = Arc::new(FilterEngine::new(Vec::new()).expect("empty rule set"));
tokio::spawn(async move {
let _ = serve_on_listener_with_overrides(
listener,
event_bus,
db,
server_filter_engine,
server_secrets,
server_token,
MailEngineOverrides::default(),
CalendarEngineOverrides::default(),
// No injected contacts backend: exercise the real production path.
ContactsEngineOverrides::default(),
AccountsEngineOverrides::default(),
)
.await;
});
(addr, token, dir)
}
/// The full offline proof of the production CardDAV sync path: configure a
/// CardDAV account over the real `Accounts` API, then `Contacts/Sync` builds a
/// real `CardDavClient` and syncs contacts fetched from WireMock into the
/// store, immediately visible to `ListContacts` -- all over the authenticated
/// gRPC API.
#[tokio::test]
async fn carddav_production_sync_builds_real_client_and_persists_contacts() {
let mock_server = MockServer::start().await;
Mock::given(method("REPORT"))
.and(path("/addressbooks/contacts/"))
.respond_with(ResponseTemplate::new(207).set_body_string(multistatus_body()))
.mount(&mock_server)
.await;
let collection_url = format!("{}/addressbooks/contacts/", mock_server.uri());
let secrets = Arc::new(SecretManager::mock());
let (addr, token, _db_dir) = boot_daemon(secrets).await;
let addr_str = addr.to_string();
// ---- Configure a first-class CardDAV account over the Accounts API ----
// The password travels to the daemon ONLY on this RPC and is written only
// to the mock vault; the sync path below reads it back from there.
let mut accounts_client = nuncio_proto::client::connect_accounts(&addr_str, &token)
.await
.expect("accounts client connects");
accounts_client
.add_account(AddAccountRequest {
config: Some(AccountConfigProto {
id: ACCOUNT_ID.to_string(),
name: "Work Address Book".to_string(),
email_address: ACCOUNT_EMAIL.to_string(),
keyring_secret_key: KEYRING_KEY.to_string(),
sync_interval: Some(nuncio_proto::time::duration_from_secs(300)),
filters_enabled: false,
transport: Some(nuncio_proto::v1::account_config::Transport::Carddav(
nuncio_proto::v1::DavTransport {
collection_url: collection_url.clone(),
},
)),
}),
password: CARDDAV_PASSWORD.to_string(),
})
.await
.expect("CardDAV account is accepted and persisted");
// A non-CardDAV account, so the "not a CardDAV account" assertion below
// exercises a real, configured account rather than a missing one.
accounts_client
.add_account(AddAccountRequest {
config: Some(AccountConfigProto {
id: MAIL_ACCOUNT_ID.to_string(),
name: "Work Mail".to_string(),
email_address: ACCOUNT_EMAIL.to_string(),
keyring_secret_key: "nuncio/acct-carddav-e2e-mail".to_string(),
sync_interval: Some(nuncio_proto::time::duration_from_secs(300)),
filters_enabled: false,
transport: Some(nuncio_proto::v1::account_config::Transport::ImapSmtp(
nuncio_proto::v1::ImapSmtpTransport {
imap_host: "127.0.0.1".to_string(),
imap_port: 1143,
imap_tls_mode: nuncio_proto::v1::TlsMode::Plain.into(),
smtp_host: "127.0.0.1".to_string(),
smtp_port: 1025,
smtp_tls_mode: nuncio_proto::v1::TlsMode::Plain.into(),
},
)),
}),
password: "mail-password".to_string(),
})
.await
.expect("mail account is accepted and persisted");
// ---- Contacts/Sync runs the PRODUCTION path (no injected backend) ----
let mut contacts_client = nuncio_proto::client::connect_contacts(&addr_str, &token)
.await
.expect("contacts client connects");
let sync_response = contacts_client
.sync(ContactsSyncRequest {
account_id: ACCOUNT_ID.to_string(),
})
.await
.expect("production CardDAV sync succeeds")
.into_inner();
assert_eq!(sync_response.synced_count, 2);
// Immediately visible: `Sync` awaits full fetch-and-persist completion.
let list_response = contacts_client
.list_contacts(ListContactsRequest {
account_id: ACCOUNT_ID.to_string(),
page_size: 0,
page_token: String::new(),
})
.await
.expect("list_contacts succeeds")
.into_inner();
assert_eq!(list_response.contacts.len(), 2);
let names: Vec<&str> = list_response
.contacts
.iter()
.map(|c| c.display_name.as_str())
.collect();
assert!(names.contains(&"Production CardDAV Maes"));
assert!(names.contains(&"Production CardDAV Dev"));
// ---- A configured account that is NOT CardDAV fails honestly ----
let not_carddav = contacts_client
.sync(ContactsSyncRequest {
account_id: MAIL_ACCOUNT_ID.to_string(),
})
.await
.expect_err("a sync targeting a non-CardDAV account must fail honestly");
assert_eq!(not_carddav.code(), Code::FailedPrecondition);
assert!(not_carddav.message().contains(MAIL_ACCOUNT_ID));
// ---- An account with no CardDAV configuration at all fails honestly ----
let missing = contacts_client
.sync(ContactsSyncRequest {
account_id: "acct-does-not-exist".to_string(),
})
.await
.expect_err("a sync targeting an unconfigured account must fail honestly");
assert_eq!(missing.code(), Code::FailedPrecondition);
// ---- An unauthenticated call to Contacts is rejected ----
let mut unauthenticated_client =
nuncio_proto::v1::contacts_client::ContactsClient::connect(format!("http://{addr_str}"))
.await
.expect("client connects at the transport level without any auth");
let err = unauthenticated_client
.list_contacts(ListContactsRequest {
account_id: ACCOUNT_ID.to_string(),
page_size: 0,
page_token: String::new(),
})
.await
.expect_err("missing bearer token must be rejected");
assert_eq!(err.code(), Code::Unauthenticated);
}
crates/nunciod/tests/contacts_e2e_test.rs
+6
View File
@@ -146,6 +146,8 @@ async fn contacts_sync_create_and_list_round_trip_over_authenticated_grpc() {
let list_response = contacts_client
.list_contacts(ListContactsRequest {
account_id: ACCOUNT_ID.to_string(),
page_size: 0,
page_token: String::new(),
})
.await
.expect("list_contacts succeeds")
@@ -201,6 +203,8 @@ async fn contacts_sync_create_and_list_round_trip_over_authenticated_grpc() {
let second_list = second_client
.list_contacts(ListContactsRequest {
account_id: ACCOUNT_ID.to_string(),
page_size: 0,
page_token: String::new(),
})
.await
.expect("list_contacts succeeds on second connection")
@@ -231,6 +235,8 @@ async fn contacts_sync_create_and_list_round_trip_over_authenticated_grpc() {
let err = unauthenticated_client
.list_contacts(ListContactsRequest {
account_id: ACCOUNT_ID.to_string(),
page_size: 0,
page_token: String::new(),
})
.await
.expect_err("missing bearer token must be rejected");
crates/nunciod/tests/daemon_e2e_test.rs
+1 -3
View File
@@ -79,9 +79,7 @@ async fn cli_system_status_round_trips_over_grpc_to_live_daemon() {
// client path against the live daemon, reading the token back from the
// shared mock `SecretManager` (proving the CLI's injected-vault design
// works end-to-end, not just against a hand-rolled token string).
let cli_runner = HeadlessRunner::ephemeral_with(secrets, addr.to_string())
.await
.expect("cli headless runner initializes");
let cli_runner = HeadlessRunner::connect_with(secrets, addr.to_string());
let cli_status = cli_runner
.execute_command(
crates/nunciod/tests/daemon_keyring_bound_test.rs
+151
View File
@@ -0,0 +1,151 @@
//! The daemon's own keyring reads must always terminate.
//!
//! `nunciod` provisions its bearer token and webhook signing key at startup and
//! on first webhook dispatch. A keyring that never answers -- a locked keychain,
//! an ACL that no longer trusts this binary, a Secret Service with no unlocked
//! collection -- used to leave the daemon parked on that read with no error and
//! no log line. These tests pin the bounded behaviour: an honest error, named
//! key material, and never the key itself.
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use nuncio_store::vault::{SecretManager, SecretVault, VaultError};
use nunciod::secrets::{provision_grpc_token, provision_webhook_signing_key};
use std::sync::Arc;
use std::time::{Duration, Instant};
/// Sentinel standing in for real key material: if any of it ever reaches an
/// error message, the assertions below catch it.
const SENTINEL_KEY_HEX: &str = "53454e54494e454c6b65796d6174657269616c4141424243434444454546464747";
/// Vault whose reads never return, standing in for a locked keychain or one
/// waiting on an access-consent prompt nobody can answer.
struct BlockingVault;
impl SecretVault for BlockingVault {
fn get_secret(&self, _key: &str) -> Result<String, VaultError> {
loop {
std::thread::park();
}
}
fn set_secret(&self, _key: &str, _secret: &str) -> Result<(), VaultError> {
Ok(())
}
fn delete_secret(&self, _key: &str) -> Result<(), VaultError> {
Ok(())
}
}
fn blocked_secrets() -> Arc<SecretManager> {
Arc::new(SecretManager::new(BlockingVault))
}
#[tokio::test]
async fn blocked_bearer_token_read_fails_instead_of_hanging_startup() {
let started = Instant::now();
let err = provision_grpc_token(blocked_secrets(), Duration::from_millis(150))
.await
.expect_err("a keyring that never answers must not yield a token");
assert!(
started.elapsed() < Duration::from_secs(5),
"startup must give up on the read, not wait on it"
);
assert!(
err.contains("gRPC bearer token"),
"the error must name which key material could not be read: {err}"
);
assert!(
err.contains("locked") || err.contains("consent"),
"the error must suggest the likely cause: {err}"
);
assert!(
!err.contains(SENTINEL_KEY_HEX),
"an error must never carry key material: {err}"
);
}
#[tokio::test]
async fn blocked_webhook_key_read_fails_instead_of_hanging_the_outbox() {
let started = Instant::now();
let err = provision_webhook_signing_key(blocked_secrets(), Duration::from_millis(150))
.await
.expect_err("a keyring that never answers must not yield a signing key");
assert!(
started.elapsed() < Duration::from_secs(5),
"the outbox must give up on the read, not wait on it"
);
assert!(
err.contains("webhook signing key"),
"the error must name which key material could not be read: {err}"
);
assert!(
err.contains("locked") || err.contains("consent"),
"the error must suggest the likely cause: {err}"
);
assert!(
!err.contains(SENTINEL_KEY_HEX),
"an error must never carry key material: {err}"
);
}
#[tokio::test]
async fn a_working_keyring_still_mints_and_reloads_the_same_key_material() {
let secrets = Arc::new(SecretManager::mock());
let timeout = Duration::from_secs(5);
let token = provision_grpc_token(Arc::clone(&secrets), timeout)
.await
.expect("the mock vault mints a token");
// 32 bytes, hex-encoded.
assert_eq!(token.len(), 64);
assert_eq!(
token,
provision_grpc_token(Arc::clone(&secrets), timeout)
.await
.expect("the token reloads across reads"),
"bounding the read must not change the token the daemon serves with"
);
let webhook_key = provision_webhook_signing_key(Arc::clone(&secrets), timeout)
.await
.expect("the mock vault mints a signing key");
assert_eq!(webhook_key.len(), 64);
assert_ne!(
token, webhook_key,
"each purpose must get its own key material"
);
}
#[tokio::test]
async fn a_failing_keyring_reports_the_vault_failure_not_a_timeout() {
struct FailingVault;
impl SecretVault for FailingVault {
fn get_secret(&self, _key: &str) -> Result<String, VaultError> {
Err(VaultError::StorageFailed("no unlocked collection".into()))
}
fn set_secret(&self, _key: &str, _secret: &str) -> Result<(), VaultError> {
Ok(())
}
fn delete_secret(&self, _key: &str) -> Result<(), VaultError> {
Ok(())
}
}
let err = provision_grpc_token(
Arc::new(SecretManager::new(FailingVault)),
Duration::from_secs(5),
)
.await
.expect_err("a failing vault must fail closed");
assert!(
err.contains("no unlocked collection"),
"a real vault failure must surface verbatim: {err}"
);
assert!(
!err.contains("timed out"),
"a real vault failure must not be reported as a timeout: {err}"
);
}
crates/nunciod/tests/filters_triage_e2e_test.rs
+129 -23
View File
@@ -3,7 +3,7 @@
//! Boots a real `nunciod` daemon gRPC server (`nunciod::grpc::
//! serve_on_listener`) on an ephemeral loopback port, backed by a real
//! temp file-backed [`DatabaseEngine`] seeded directly via
//! [`DatabaseEngine::save_email`] (standing in for messages that were synced
//! [`DatabaseEngine::save_email_at`] (standing in for messages that were synced
//! before the active rule set existed) and a live [`FilterEngine`] carrying
//! a real persisted rule.
//!
@@ -25,7 +25,7 @@
#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
use nuncio_core::model::Email;
use nuncio_core::model::{Email, IdentitySource, Placement};
use nuncio_core::EventBus;
use nuncio_filter::{FilterEngine, NsqlParser};
use nuncio_proto::v1::filters_client::FiltersClient;
@@ -42,18 +42,73 @@ fn sample_email(id: &str, subject: &str) -> Email {
Email {
id: id.to_string(),
account_id: ACCOUNT_ID.to_string(),
folder_id: "inbox".to_string(),
subject: subject.to_string(),
sender: "alice@nuncio.mx".to_string(),
recipient: "triage-e2e@nuncio.mx".to_string(),
received_at: 1_700_000_000,
read: false,
body_plain: Some("body".to_string()),
body_html: None,
attachments: Vec::new(),
message_id: None,
content_hash: None,
}
}
/// The account the seeded messages belong to, opted in as this daemon's
/// filter-execution owner.
fn owning_account() -> nuncio_core::AccountConfig {
nuncio_core::AccountConfig {
id: ACCOUNT_ID.to_string(),
name: "Triage E2E".to_string(),
email_address: "triage-e2e@nuncio.mx".to_string(),
keyring_secret_key: format!("nuncio/{ACCOUNT_ID}"),
sync_interval_secs: 60,
filters_enabled: true,
transport: nuncio_core::Transport::ImapSmtp(nuncio_core::ImapSmtpTransport {
imap_host: "imap.nuncio.mx".to_string(),
imap_port: 993,
imap_tls_mode: nuncio_core::TlsMode::ImplicitTls,
smtp_host: "smtp.nuncio.mx".to_string(),
smtp_port: 465,
smtp_tls_mode: nuncio_core::TlsMode::ImplicitTls,
}),
}
}
/// The INBOX occupancy a seeded message sits in. `id` doubles as the UID, so
/// distinct messages occupy distinct rows.
fn sample_placement(id: &str) -> Placement {
Placement {
account_id: ACCOUNT_ID.to_string(),
folder_id: "inbox".to_string(),
uid_validity: "1".to_string(),
remote_id: id.to_string(),
read: false,
}
}
/// Seed a message and its occupancy the way a sync pass would.
async fn seed(db: &DatabaseEngine, id: &str, subject: &str) {
db.save_email_at(
&sample_email(id, subject),
IdentitySource::Surrogate,
&sample_placement(id),
)
.await
.unwrap_or_else(|e| panic!("seed message '{id}': {e}"));
}
/// The read flag of a seeded message's single occupancy.
async fn read_flag_of(db: &DatabaseEngine, id: &str) -> bool {
let placements = db.placements_of(id).await.expect("read placements");
assert_eq!(
placements.len(),
1,
"'{id}' should occupy exactly one folder"
);
placements[0].read
}
/// Boots the real daemon gRPC server (bearer-token authenticated) on an
/// ephemeral loopback port, backed by a real temp file-backed
/// `DatabaseEngine` and the given live `FilterEngine`. Returns the server
@@ -77,6 +132,13 @@ async fn boot_daemon(
.expect("open temp file-backed database"),
);
// Triage only acts on accounts this daemon owns filter execution for, so
// the account must exist and be opted in -- otherwise the RPC honestly
// refuses rather than scanning and applying nothing.
db.save_account(&owning_account())
.await
.expect("seed the triage account");
let token_bytes = secrets
.get_or_create_key_bytes(GRPC_TOKEN_ACCOUNT, 32)
.expect("mint gRPC bearer token from mock vault");
@@ -141,15 +203,9 @@ async fn triage_applies_live_rule_retroactively_and_reports_real_cumulative_coun
// Seed messages directly into the store, standing in for mail that was
// synced BEFORE these rules existed -- exactly the scenario `Triage` is
// for. Two match (one per rule), one does not.
db.save_email(&sample_email("m-urgent", "Urgent: server down"))
.await
.expect("seed urgent message");
db.save_email(&sample_email("m-archive", "Please Archive Me"))
.await
.expect("seed archive message");
db.save_email(&sample_email("m-plain", "Just saying hi"))
.await
.expect("seed non-matching message");
seed(&db, "m-urgent", "Urgent: server down").await;
seed(&db, "m-archive", "Please Archive Me").await;
seed(&db, "m-plain", "Just saying hi").await;
let mut client = FiltersClient::connect(format!("http://{addr}"))
.await
@@ -186,8 +242,10 @@ async fn triage_applies_live_rule_retroactively_and_reports_real_cumulative_coun
// Real side effects, not a fabricated count: the MARK READ action
// actually flipped the stored read flag...
let urgent = db.get_message("m-urgent").await.expect("get urgent");
assert!(urgent.read, "MARK READ must genuinely flip the read flag");
assert!(
read_flag_of(&db, "m-urgent").await,
"MARK READ must genuinely flip the occupancy's read flag"
);
// ...and the MOVE TO action actually queued a real outbox mutation.
let pending = db
@@ -201,8 +259,10 @@ async fn triage_applies_live_rule_retroactively_and_reports_real_cumulative_coun
);
assert_eq!(pending[0].message_id, "m-archive");
let plain = db.get_message("m-plain").await.expect("get plain");
assert!(!plain.read, "a non-matching message must be left untouched");
assert!(
!read_flag_of(&db, "m-plain").await,
"a non-matching message must be left untouched"
);
}
#[tokio::test]
@@ -217,12 +277,7 @@ async fn triage_streams_multiple_chunks_for_a_store_bigger_than_one_chunk() {
// updates before the final `done: true` -- proof this is a real
// multi-chunk stream, not one batch disguised as one.
for i in 0..5 {
db.save_email(&sample_email(
&format!("m-{i:02}"),
&format!("Message number {i}"),
))
.await
.expect("seed message");
seed(&db, &format!("m-{i:02}"), &format!("Message number {i}")).await;
}
let mut client = FiltersClient::connect(format!("http://{addr}"))
@@ -265,3 +320,54 @@ async fn triage_streams_multiple_chunks_for_a_store_bigger_than_one_chunk() {
prev_scanned = update.scanned_count;
}
}
/// A daemon that owns filter execution for no account must refuse `Triage`
/// outright. Streaming a scan that reports progress and applies nothing would
/// be indistinguishable, from the caller's side, from "no rules matched" --
/// and the whole point of single-owner execution is that the non-owner stays
/// visibly inert rather than quietly so.
#[tokio::test]
async fn triage_refuses_when_this_daemon_owns_no_account() {
let secrets = Arc::new(SecretManager::mock());
let rule = NsqlParser::parse_rule(
"Urgent Auto-Read",
1,
"WHERE subject CONTAINS 'Urgent' ACTION MARK READ",
)
.expect("parse rule");
let filter_engine = Arc::new(FilterEngine::new(vec![rule]).expect("compile rule"));
let (addr, token, db, _dir) = boot_daemon(secrets, filter_engine).await;
// Take ownership away: the account exists, but this daemon is not its
// filter owner.
let mut account = owning_account();
account.filters_enabled = false;
db.save_account(&account)
.await
.expect("opt the account out");
seed(&db, "m-1", "Urgent: server down").await;
let mut client = FiltersClient::connect(format!("http://{addr}"))
.await
.expect("connect to the daemon");
let status = client
.triage(authed_request(&token, TriageRequest::default()))
.await
.expect_err("triage must refuse without an owning account");
assert_eq!(status.code(), tonic::Code::FailedPrecondition);
assert!(
status.message().contains("filter execution"),
"the refusal must say why, got: {}",
status.message()
);
// And it must genuinely have applied nothing: the occupancy the rule
// would have marked read is still unread.
assert!(
!read_flag_of(&db, "m-1").await,
"a refused triage must not apply any action"
);
}
Loaded 100 of 119 files, more files were not shown because too many files have changed in this diff. Show more