nunciod mints the gRPC bearer token into the OS keyring, so the keychain
item's ACL trusts nunciod. nuncio-cli is a different binary, so reading
that item raises an interactive consent prompt. Nothing can answer that
prompt from CI, ssh, or a background shell, and the blocking keyring read
never returns -- the CLI hung with no output and no error.
Add an opt-in NUNCIO_GRPC_TOKEN override: when set and non-empty the CLI
uses it verbatim and never touches the keyring. The keyring remains the
default and is otherwise unchanged.
The keyring read itself can no longer stall forever either. It now runs on
a detached OS thread behind a bounded timeout that reports what is blocking
and how to bypass it. A detached thread rather than spawn_blocking, because
Tokio joins its blocking pool during runtime shutdown, which would just move
the hang to process exit.
Collapse the eight copies of the token-resolution block onto that one
resolver so no call site can drift.
Refs #419
IMAP hardcoded no object id into the message key, so it could only ever
reach the Message-ID+content tier or the folder-scoped surrogate, while
JMAP entered at the top tier from the server's Email id. One account
synced through both protocols therefore minted disjoint key spaces.
RFC 8474 EMAILID is the same identifier JMAP exposes, so requesting it
makes the two keys equal. Negotiate per capability: EMAILID only where
the server advertises OBJECTID, X-GM-MSGID only where it advertises
X-GM-EXT-1. A server offering neither behaves exactly as before.
EMAILID cannot go through the typed client: imap-proto's msg_att parser
is a closed alt with no catch-all, and async-imap turns the resulting
nom error into a fatal io::Error that closes the session. The raw layer
reads it off the transport beneath ImapStream instead, so an attribute
the crate cannot parse can never reach its decoder.
Lifting stored rows to a stronger tier changes every message_key, so
IDENTITY_SCHEMA_VERSION moves to 3 and the existing reconciliation
rebuilds the cached store rather than repointing rows one at a time.
The engine is ahead of its contract: it executes verified mutations but the
Mail service has no RPC to ask for one, so a human cannot move, delete or
flag anything. By this repo's definition of done the placement work has an
engine and nothing else.
Records the three steps that buy manual usability, what works and what does
not if you boot it today, the 81-issue backlog behind them, and the two test
traps (tracing callsite Interest, tokio::time::pause over real I/O) that are
invisible at default parallelism.
existing_placements binds four parameters per key into one row-value
IN (...) list, and its caller hands it an entire folder: FolderChanges
accumulates every batch into one Vec, so the Full rung -- an ordinary
first sync, and the unrecognised-checkpoint fallback -- passes the whole
mailbox. SQLITE_MAX_VARIABLE_NUMBER is 32766, so past 8191 placements
the statement fails, the folder sync aborts before writing a checkpoint,
and every retry fails identically. A first sync of any large mailbox was
permanently broken.
Split the batch internally at 4000 keys and union the results, so the
limit is unreachable from outside rather than one forgetful caller away.
existing_message_ids had the same shape at a 32766 ceiling and is
chunked the same way; it stays because it is public API drawing the
identity-vs-occupancy distinction, not because it has callers.
Also correct three comments that documented traps as free:
- The IMAP batching doc claimed the working set stays bounded regardless
of mailbox size. Batching bounds each request, not memory; the parsed
messages accumulate across batches.
- Adding the EMAILID/X-GM-MSGID FETCH items is not free: it repoints
every stored message's key, orphaning the old messages row and
stranding its decrypted body in messages_fts. It needs the repoint
path to reap first, or it leaks plaintext.
- IMAP never requests the tier-1/2 identifiers, so an IMAP-derived key
and a JMAP-derived key for the same message differ. Recorded on
derive_message_key and on the IMAP place() helper.
De-link ADR 0002 in the roadmap: the file exists on no merge target, so
the markdown now matches the rendered page, which already used plain
bold.
WS-B, WS-C and WS-E grew when ADR 0002 broke the multi-engine sync model into
#386-#397, and the 2026-08-07 snapshot never caught up. Counts now match the
live milestones. Nothing is marked closed: the identity split and its chain are
green on a branch, and the numbers move when it merges.
Refs #394
Five defects found in review:
- existing_placements never pinned that a never-saved key is absent, and
its negative case was vacuous (no Archive row existed to over-report).
Over-reporting makes genuinely new mail look already-seen and be
skipped by the sync classifier. Both are now real assertions, backed
by a count so a dropped IN filter fails.
- save_email_at and delete_placements both read then write in one
deferred transaction. In WAL mode that aborts with SQLITE_BUSY, which
busy_timeout does not retry; twenty racing saves reproducibly failed.
Both now use BEGIN IMMEDIATE, matching the WORM and ledger appenders,
with a concurrency regression test.
- list_messages_page claimed the folder join was one-to-one. It is not:
the placements key permits one message in one folder twice, and the
message-only cursor made the second row unreachable across a page
boundary. The keyset now carries the full placement address.
- list_messages_for_export matched a folder id without its account,
reintroducing the cross-account collision this model removes.
The first draft reasoned like a shipped product: preserve every stored row,
never change behaviour without a path, keep old and new daemons interoperable.
Nothing is released, so none of that is owed -- and paying for it distorted the
model in one place that matters.
Message identity cannot be derived for rows already stored, because Message-ID
was never persisted and the octets are discarded after parse. Honouring those
rows meant keeping them under their old surrogate forever: permanently
un-deduped, invisible to every cross-folder guarantee the design exists to
provide, and leaving the store two-tier for as long as the install survives.
Resetting the message store and re-syncing from the server -- which is
authoritative regardless -- costs a developer one sync and buys a single
coherent model.
The audit ledger follows the same reasoning rather than gaining a permanent
namespace boundary: a hash chain whose entries reference ids that no longer
resolve has no audit value, so it is reinitialised with the store it describes.
This also removes the dependency that shaped the build order. Placements waited
on captured identity accumulating only because history had to be preserved;
now capture need only land one merge ahead of the model consuming it. The order
is recorded explicitly, along with the note that change enumeration still ships
alone and still fixes ghost messages for a single engine.
The posture section is written to be deleted at the first real release, when
every one of these questions becomes a genuine obligation.
The engine syncs as though it is the only client of an account: a forward-only
UID high-water mark that cannot observe deletion, and fire-and-forget mutations
that report success when a UID command silently did nothing. A second client --
another machine, a phone, or webmail -- makes both assumptions false, so the
local view diverges and nothing detects it.
ADR 0002 decides how a daemon behaves as one of several clients. The core
choice is to build no coordination layer: IMAP and WebDAV already coordinate
concurrent clients, so the work is finishing the sync model rather than adding
multi-engine support, and correctness at ten engines follows from correctness
at one. The exception is side-effecting filter actions, which land on third
parties with no shared resource to compare against; those get a single owner.
A draft was reviewed independently for protocol correctness, codebase fit, and
adversarially. All three falsified load-bearing claims, and the corrections are
material enough to record rather than summarise: Gmail has no QRESYNC and
Exchange has neither QRESYNC nor CONDSTORE, so the ladder needs four rungs
selected per mailbox; an absent COPYUID is conforming on success and cannot mean
conflict; keying identity on a Message-ID hash would have let a forged message
overwrite a trusted body; and the proposed convergence test could not have
detected the design's largest hole. The review evidence lands alongside the ADR
because several findings cost real effort to establish and would otherwise be
rediscovered expensively.
Documentation only. No code or contract changes.
In-repo dev instrument (crates/nuncio-monitor) plus the three engine
changes it needs: System.Shutdown, pending_remote_mutations.account_id,
and System.GetHealth (implements #296).
Roadmap and its rendered page carried no mention of M6.5, the WS-A..WS-F
workstreams, or the completed OBS epic. AGENTS.md becomes a pointer to
CLAUDE.md instead of a near-verbatim copy.
Add a Redacted<T> newtype in nuncio-core whose Debug, Display, and serde
render a fixed placeholder, so a wrapped credential cannot leak through a
derived Debug or serde_json; the value is reachable only via an explicit
expose_secret(). Migrate the CalDAV/CardDAV account auth tokens to it and
drop their hand-written redacting Debug impls.
Document the logging redaction policy (what must never be logged vs what
is allowed) in docs/LOGGING.md and the redact module doc.
Add a cross-crate no-secrets canary that installs a capturing subscriber,
drives auth reject, account add, mock and JMAP-wiremock sync, mark-read,
send, a filter rule-fire, a webhook dispatch, and CalDAV/CardDAV sync with
sentinel secrets/PII, and asserts none appear in captured telemetry.
Refs #363
Co-authored-by: James Maes <jmaes@dmdbrands.com>
Wire -v/-vv/-vvv to a real stderr tracing subscriber (previously parsed
but never used), with NUNCIO_LOG/RUST_LOG override precedence matching
the daemon. Decode the WS-A1 ErrorInfo from a failed RPC's status
details and render the gRPC code plus the typed reason alongside the
message, in both text and --json output, instead of dropping the code
or dumping a raw Status. Document daemon log levels, the rotating log
file location, JSON log output, and request_id correlation in
docs/RUNNING.md.
Refs #365
Co-authored-by: James Maes <jmaes@dmdbrands.com>
* docs: mark M5 complete (feature surface done), set M6 as next
* docs: correct HANDOFF service count to eight
---------
Co-authored-by: James Maes <jmaes@dmdbrands.com>
* docs: mark M1-M4 complete, set M5 as next milestone
Phase 3 feature completeness (Calendar, Contacts, JMAP, live Filters)
is done and merged to dev; update ROADMAP.md, the rendered roadmap
page, and HANDOFF.md to reflect it and point at M5 next.
* docs: correct service count to eight (Calendar+Contacts added)
---------
Co-authored-by: James Maes <jmaes@dmdbrands.com>
Cargo aliases are a single command argv and cannot chain subcommands, so
`cargo verify = ["fmt --all -- --check", "check-all", "test-all"]` tried to
run a literal subcommand named "fmt --all -- --check" and always errored.
The pre-commit hook already runs the three checks as separate commands, so
it was unaffected. Drop the broken alias, drop the matching RustRover run
configuration, and reword every doc that told readers to run `cargo verify`
to instead run fmt/check-all/test-all as three commands.
Sweep every README/doc for accuracy and remove fabricated or stale content:
- README/CLAUDE: add the live roadmap-page link and a rule to keep the rendered
page in sync with docs/ROADMAP.md; fix the stale cov alias and the
already-moved-shells note.
- .github/CONTRIBUTING+SECURITY: drop the fabricated nuncio.mx site link and the
aspirational release/security claims; state honest pre-alpha status.
- docs/RUNNING, DEVELOPING, ARCHITECTURE, BACKLOG, Home, NSQL spec, corruption
notes, proto/README: correct crate count/list, service list, CLI flags, phase
status, and remove claims of unshipped/removed things (IPC-as-current,
hardcoded-keys, unwired engines); mark superseded api-surface-draft historical.
- Remove the fabricated benchmark baked into nsql_pipeline.svg and delete the
orphaned ipc_sequence.svg (depicted the removed IPC with a fake timing).
Add docs/STORY-WORKFLOW.md (the canonical per-story execution/gate/review
protocol every roadmap issue links to) and docs/HANDOFF.md (how a new machine or
LLM picks up the work: current state, branches, the M1-M7 story index, and
coordination rules). Include the incomplete calendar API-vertical WIP as a
reference patch under docs/handoff/ (prior art for the M1 calendar story;
does not compile, apply-to-inspect only).
Rewrite ROADMAP.md as the authoritative plan from today's state (Phases 0-2
complete; Phase 3 in progress) to a frozen, versioned gRPC API that front-end
repos can consume. Records true status, folds in gaps found via dogfooding and
audit, and defines seven milestones (M1 calendar, M2 contacts, M3 JMAP, M4 live
filters, M5 mail lifecycle + incremental sync, M6 security hardening, M7 API
freeze + client-readiness gate) each with concrete exit criteria.
Update CLAUDE.md, ARCHITECTURE.md, and the RustRover guide: the daemon serves
only the gRPC API on 127.0.0.1:9420; the old JSON-RPC IPC (:9422) is removed. Also
corrected the stale 'shells still in the workspace' note (they moved to _reference).
Add docs/DEVELOPING-IN-RUSTROVER.md (setup, running the daemon + CLI, running and
debugging tests, quality gates, the dev loop, gotchas) and link it from CLAUDE.md.
Add one-click run configs: Run nuncio-cli (mail sync / account list / mail list).
docs/RUNNING.md: verified build/run/use steps for the daemon + CLI (real gRPC +
OS keyring confirmed working). Add a 'cargo build-release' alias for the two
runnable binaries.
Catalogue of 40 candidate RPCs across 8 services (25 real / 10 partial / 5 stub)
with reality tags, parity gaps, and open Phase 2 decisions. Input to the
proto/nuncio/v1 contract; sync RPCs to be drafted from Phase 1's real spine,
not today's status-flag stubs.
Refs #144
Remove fabricated PLAN-*/EXECUTIVE-*/TODO/TICKET/spec docs; add authoritative
ROADMAP, ADR-0001 (engine-first gRPC), and Phase 0/1 BACKLOG; rewrite README,
CLAUDE.md, ARCHITECTURE, and wiki Home to honest reality; add status banners to
the NSQL and self-healing specs; make wiki-sync mirror-with-delete.
james.maes
James Maes