feat(mail): capture Message-ID and a content hash at ingest #398

2026-08-10T19:48:55Z

KofTwentyTwo commented

2026-08-10 19:48:55 +00:00

(Migrated from github.com)

Summary of Changes

Captures a normalized Message-ID and a content hash at every ingest path, as nullable columns nothing reads yet. Prerequisite for #394 (the identity/placements split), which cannot start until real identity data has accrued.

Why this has to ship on its own, early

Identity cannot be reconstructed for messages already stored. MimeParserAdapter::parse_mime never calls message_id(), Email has no field for it, messages has no column, and build_email_from_fetch discards the BODY[] bytes after parsing. So an in-place migration to a folder-independent identity has nothing to derive from — the only alternatives are re-downloading every mailbox or inventing values. Landing capture now starts the clock.

Normalization is fixed and shared

Defined once in nuncio-core rather than per call site. Engines that normalize differently would hold the same message under different bytes, and no comparison between their caches would mean anything.

Strip all ASCII whitespace — a msg-id cannot legally contain any, so this unfolds a wrapped header without needing to know how the transport folded it, and transports differ.
Strip one pair of angle brackets — servers include or omit them inconsistently between ENVELOPE and a raw header.
Preserve case — only the domain half is case-insensitive per RFC 5322 §3.6.4. Lowercasing wholesale would merge ids the originating server considers distinct, and the same server returns identical bytes to every engine anyway.

All four ingest paths, including the two that don't parse a full message

Path	`message_id`	`content_hash`
`parse_mime` (full body)	from header	over the parsed octets
IMAP envelope-only	from `ENVELOPE` (RFC 3501 §7.4.2)	`None` — no octets
IMAP headers-only (oversized)	from headers	`None` — deliberately cleared
JMAP `Email/get`	`messageId[0]`, newly requested	`None` — returns a snippet, not octets

Two details worth review attention:

The envelope-only path was flagged in #386 as a candidate for removal because identity "is not computable there." It is computable — ENVELOPE carries message-id — so capture was added rather than the fetch reshaped. That keeps this change off the FETCH/batching logic.
The headers-only path discards the hash parse_mime computed, because that hash covered only the header section. content_hash is defined over a message's full octets; publishing a header-only digest under that name would quietly mean something different. The Message-ID recovered from those headers is correct and is kept.

Why a content hash at all

Message-ID is public, echoed in every References: header, forgeable, and legitimately reused — a mailing-list copy and a direct copy share one, and a draft keeps one across every save. The pair distinguishes messages that the id alone would merge, and it is the reason ADR 0002 refuses to key storage on a Message-ID hash.

Migration

Additive, nullable, no backfill. Legacy rows stay NULL, which reads as not captured rather than this message has no Message-ID — a distinction any future consumer must respect. Inventing a value would be worse than admitting the gap.

Type of Change

Bug fix (non-breaking change fixing an issue)
New feature (non-breaking change adding functionality)
Breaking change (fix or feature causing existing functionality to change)
Documentation update

No behavior change: the columns are written and round-tripped, and nothing reads them yet.

Related Issues

Refs #386. Prerequisite for #394. Design: ADR 0002 (PR #385).

Quality Verification Checklist

I have verified code formatting (cargo fmt --all -- --check)
I have verified clippy lints (cargo clippy --all-targets --workspace -- -D warnings)
I have executed the full test suite (cargo test --workspace) and all tests pass (604 passed, 0 failed)
Business logic is implemented in headless engine crates, keeping UI presentation shells skinny
No secrets, keys, or credentials are added to tracked files

Notes for review

8 new tests: normalization canonical-form and case/empty handling; content hash distinguishing a shared Message-ID; parser capture with and without the header; JMAP capture; store round-trip through get_message and list_messages; migration idempotence with a legacy row reading back as not-captured.
Coverage gap, stated plainly: the IMAP envelope-only path has no direct test on this branch. Constructing an async_imap::types::Fetch requires a scripted server, and the stateful harness that makes that practical is in PR #384. Worth adding once that merges.
The ~30 remaining edits are mechanical message_id: None, content_hash: None additions to test fixtures and one synthetic PreviewRule sample that is never persisted.
No .proto changes — the descriptor golden is untouched and the one-proto-story slot stays free.
This branch is cut from dev, so it still carries the flaky webhook test fixed in #384. It passed on this run; #384 is the fix.

## Summary of Changes Captures a normalized `Message-ID` and a content hash at every ingest path, as **nullable columns nothing reads yet**. Prerequisite for #394 (the identity/placements split), which cannot start until real identity data has accrued. ### Why this has to ship on its own, early Identity cannot be reconstructed for messages already stored. `MimeParserAdapter::parse_mime` never calls `message_id()`, `Email` has no field for it, `messages` has no column, and `build_email_from_fetch` discards the `BODY[]` bytes after parsing. So an in-place migration to a folder-independent identity has nothing to derive from — the only alternatives are re-downloading every mailbox or inventing values. Landing capture now starts the clock. ### Normalization is fixed and shared Defined once in `nuncio-core` rather than per call site. Engines that normalize differently would hold the same message under different bytes, and no comparison between their caches would mean anything. 1. **Strip all ASCII whitespace** — a `msg-id` cannot legally contain any, so this unfolds a wrapped header without needing to know how the transport folded it, and transports differ. 2. **Strip one pair of angle brackets** — servers include or omit them inconsistently between `ENVELOPE` and a raw header. 3. **Preserve case** — only the domain half is case-insensitive per RFC 5322 §3.6.4. Lowercasing wholesale would merge ids the originating server considers distinct, and the same server returns identical bytes to every engine anyway. ### All four ingest paths, including the two that don't parse a full message | Path | `message_id` | `content_hash` | | --- | --- | --- | | `parse_mime` (full body) | from header | over the parsed octets | | IMAP envelope-only | **from `ENVELOPE`** (RFC 3501 §7.4.2) | `None` — no octets | | IMAP headers-only (oversized) | from headers | **`None` — deliberately cleared** | | JMAP `Email/get` | `messageId[0]`, newly requested | `None` — returns a snippet, not octets | Two details worth review attention: - The envelope-only path was flagged in #386 as a candidate for removal because identity "is not computable there." It **is** computable — `ENVELOPE` carries `message-id` — so capture was added rather than the fetch reshaped. That keeps this change off the FETCH/batching logic. - The headers-only path **discards** the hash `parse_mime` computed, because that hash covered only the header section. `content_hash` is defined over a message's *full* octets; publishing a header-only digest under that name would quietly mean something different. The `Message-ID` recovered from those headers is correct and is kept. ### Why a content hash at all `Message-ID` is public, echoed in every `References:` header, forgeable, and legitimately reused — a mailing-list copy and a direct copy share one, and a draft keeps one across every save. The pair distinguishes messages that the id alone would merge, and it is the reason ADR 0002 refuses to key storage on a Message-ID hash. ### Migration Additive, nullable, **no backfill**. Legacy rows stay `NULL`, which reads as *not captured* rather than *this message has no Message-ID* — a distinction any future consumer must respect. Inventing a value would be worse than admitting the gap. ## Type of Change - [ ] Bug fix (non-breaking change fixing an issue) - [x] New feature (non-breaking change adding functionality) - [ ] Breaking change (fix or feature causing existing functionality to change) - [ ] Documentation update No behavior change: the columns are written and round-tripped, and nothing reads them yet. ## Related Issues Refs #386. Prerequisite for #394. Design: [ADR 0002](docs/adr/0002-multi-engine-sync-model.md) (PR #385). ## Quality Verification Checklist - [x] I have verified code formatting (`cargo fmt --all -- --check`) - [x] I have verified clippy lints (`cargo clippy --all-targets --workspace -- -D warnings`) - [x] I have executed the full test suite (`cargo test --workspace`) and all tests pass (**604 passed, 0 failed**) - [x] Business logic is implemented in headless engine crates, keeping UI presentation shells skinny - [x] No secrets, keys, or credentials are added to tracked files ## Notes for review - **8 new tests**: normalization canonical-form and case/empty handling; content hash distinguishing a shared `Message-ID`; parser capture with and without the header; JMAP capture; store round-trip through `get_message` and `list_messages`; migration idempotence with a legacy row reading back as not-captured. - **Coverage gap, stated plainly:** the IMAP envelope-only path has no direct test on this branch. Constructing an `async_imap::types::Fetch` requires a scripted server, and the stateful harness that makes that practical is in PR #384. Worth adding once that merges. - The ~30 remaining edits are mechanical `message_id: None, content_hash: None` additions to test fixtures and one synthetic `PreviewRule` sample that is never persisted. - No `.proto` changes — the descriptor golden is untouched and the one-proto-story slot stays free. - This branch is cut from `dev`, so it still carries the flaky webhook test fixed in #384. It passed on this run; #384 is the fix.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/nuncio#398