feat(mail): bound IMAP fetch by batch count and per-message size #352

2026-07-31T21:53:19Z

KofTwentyTwo commented

2026-07-31 21:53:19 +00:00

(Migrated from github.com)

Closes #301 (WS-D5).

Problem

sync_folder_messages* issued one unbounded UID FETCH <range> (... BODY.PEEK[] ...) and accumulated every full body into an in-memory Vec. RFC822.SIZE was fetched but unused. A malicious or huge mailbox could OOM the daemon.

What changed (engine-only, `crates/nuncio-mail/src/imap.rs`)

Two-phase, bounded fetch:

Enumeration pass — UID FETCH <range> (UID RFC822.SIZE) with NO body, so discovering the mailbox stays cheap even when huge.
Bounded body batches — normal messages are fetched in explicit UID-set chunks of at most MAX_MESSAGES_PER_FETCH_BATCH (200). The transient set of full bodies in memory never exceeds one batch, regardless of mailbox size. No single 1:* body fetch is ever issued.

Per-message size cap — a message whose RFC822.SIZE exceeds MAX_MESSAGE_BODY_BYTES (25 MiB, aligned with the MIME parser's own MAX_PAYLOAD_BYTES) is not pulled with its full body. It is fetched headers-only (BODY.PEEK[HEADER]), recorded with honest metadata (subject/sender/date from the parsed header) and an empty body, plus a WARN naming the uid + size. Never buffered whole, never silently dropped, never a fabricated body.

Integrates cleanly with the existing incremental-sync UID range ({uidnext}:*), the per-item FETCH timeout (surfaced as FetchStalled via a shared drive_fetch_stream helper), and the UIDVALIDITY checkpoint. The checkpoint derivation and fall-back logging are unchanged, so incremental checkpointing and the daemon's fire-once-per-new-message guard are preserved.

Oversized-message policy

Headers-only fetch → Email with parsed header metadata + empty body_plain/body_html/attachments + WARN(uid, size). Documented on build_headers_only_email / MAX_MESSAGE_BODY_BYTES.

Tests (offline, scripted `tokio::io::duplex` IMAP session — no live server)

a_large_folder_is_fetched_across_multiple_bounded_batches — 5 messages, batch size 2 → asserts 3 bounded body fetches (each ≤ 2 UIDs, never :*), not a single 1:*.
an_oversized_message_is_fetched_headers_only_not_buffered_whole — the oversized UID never appears in a full-body fetch; it is fetched BODY.PEEK[HEADER] and recorded with an empty body + honest subject; the two normal messages get full bodies.
batched_sync_preserves_incremental_checkpointing — first sync records 1:4; second sync with that checkpoint enumerates only the narrowed 4:* range (never 1:*) and keeps the checkpoint stable.

All existing imap tests remain green.

Gate (three commands, all green)

cargo fmt --all -- --check — clean
cargo check-all (clippy --all-targets --workspace -D warnings) — 0 warnings
cargo test-all — 521 passed; 0 failed; 0 ignored

No unsafe/unwrap/expect/panic/todo in non-test code. No proto change. Commit is SSH-signed.

Closes #301 (WS-D5). ## Problem `sync_folder_messages*` issued one unbounded `UID FETCH <range> (... BODY.PEEK[] ...)` and accumulated every full body into an in-memory `Vec`. `RFC822.SIZE` was fetched but unused. A malicious or huge mailbox could OOM the daemon. ## What changed (engine-only, `crates/nuncio-mail/src/imap.rs`) Two-phase, bounded fetch: 1. **Enumeration pass** — `UID FETCH <range> (UID RFC822.SIZE)` with NO body, so discovering the mailbox stays cheap even when huge. 2. **Bounded body batches** — normal messages are fetched in explicit UID-set chunks of at most `MAX_MESSAGES_PER_FETCH_BATCH` (**200**). The transient set of full bodies in memory never exceeds one batch, regardless of mailbox size. No single `1:*` body fetch is ever issued. **Per-message size cap** — a message whose `RFC822.SIZE` exceeds `MAX_MESSAGE_BODY_BYTES` (**25 MiB**, aligned with the MIME parser's own `MAX_PAYLOAD_BYTES`) is **not** pulled with its full body. It is fetched headers-only (`BODY.PEEK[HEADER]`), recorded with honest metadata (subject/sender/date from the parsed header) and an **empty body**, plus a `WARN` naming the uid + size. Never buffered whole, never silently dropped, never a fabricated body. Integrates cleanly with the existing incremental-sync UID range (`{uidnext}:*`), the per-item FETCH timeout (surfaced as `FetchStalled` via a shared `drive_fetch_stream` helper), and the UIDVALIDITY checkpoint. The checkpoint derivation and fall-back logging are unchanged, so incremental checkpointing and the daemon's fire-once-per-new-message guard are preserved. ## Oversized-message policy Headers-only fetch → `Email` with parsed header metadata + empty `body_plain`/`body_html`/`attachments` + `WARN(uid, size)`. Documented on `build_headers_only_email` / `MAX_MESSAGE_BODY_BYTES`. ## Tests (offline, scripted `tokio::io::duplex` IMAP session — no live server) - `a_large_folder_is_fetched_across_multiple_bounded_batches` — 5 messages, batch size 2 → asserts 3 bounded body fetches (each ≤ 2 UIDs, never `:*`), not a single `1:*`. - `an_oversized_message_is_fetched_headers_only_not_buffered_whole` — the oversized UID never appears in a full-body fetch; it is fetched `BODY.PEEK[HEADER]` and recorded with an empty body + honest subject; the two normal messages get full bodies. - `batched_sync_preserves_incremental_checkpointing` — first sync records `1:4`; second sync with that checkpoint enumerates only the narrowed `4:*` range (never `1:*`) and keeps the checkpoint stable. All existing imap tests remain green. ## Gate (three commands, all green) - `cargo fmt --all -- --check` — clean - `cargo check-all` (clippy `--all-targets --workspace -D warnings`) — 0 warnings - `cargo test-all` — **521 passed; 0 failed; 0 ignored** No `unsafe`/`unwrap`/`expect`/`panic`/`todo` in non-test code. No proto change. Commit is SSH-signed.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/nuncio#352