feat(nunciod): bound outbox per-item execution with a timeout #349

2026-07-31T21:29:43Z

KofTwentyTwo commented

2026-07-31 21:29:43 +00:00

(Migrated from github.com)

Summary

Wrap each outbox mutation's execution in a bounded tokio::time::timeout (PER_ITEM_EXECUTION_TIMEOUT, 30s) raced against the WS-C1 shutdown signal, so one hung remote op (a stuck IMAP/JMAP call or webhook) can no longer freeze delivery for every other pending mutation across every account.
A timeout is dispositioned exactly like any other transient failure: RETRYABLE, bumping retry_count, never completed -- same "no fabricated success" invariant as the existing executor.
On shutdown, the drain pass stops immediately rather than waiting out the timeout, leaving the in-flight (and any queued) item untouched and pending.

Test plan

cargo fmt --all -- --check
cargo check-all
cargo test-all (0 failed, 0 ignored)
New offline tests in crates/nunciod/tests/outbox_executor_e2e_test.rs:
- a_hung_mutation_times_out_and_is_retried_never_completed -- a hung mail-backend call is bounded by the timeout, never marked completed, retry bumped.
- a_timed_out_item_does_not_block_a_later_item_in_a_subsequent_pass -- proves a timed-out item doesn't jam the queue for a later drain pass.
- shutdown_signal_interrupts_a_hung_item_instead_of_waiting_out_the_timeout -- shutdown observed mid-hang, well before the timeout would elapse.
Existing outbox E2E tests still pass unmodified in behavior (signature updated to thread the shutdown signal through).

Refs #290

## Summary - Wrap each outbox mutation's execution in a bounded `tokio::time::timeout` (`PER_ITEM_EXECUTION_TIMEOUT`, 30s) raced against the WS-C1 shutdown signal, so one hung remote op (a stuck IMAP/JMAP call or webhook) can no longer freeze delivery for every other pending mutation across every account. - A timeout is dispositioned exactly like any other transient failure: RETRYABLE, bumping `retry_count`, never `completed` -- same "no fabricated success" invariant as the existing executor. - On shutdown, the drain pass stops immediately rather than waiting out the timeout, leaving the in-flight (and any queued) item untouched and `pending`. ## Test plan - [x] `cargo fmt --all -- --check` - [x] `cargo check-all` - [x] `cargo test-all` (0 failed, 0 ignored) - [x] New offline tests in `crates/nunciod/tests/outbox_executor_e2e_test.rs`: - `a_hung_mutation_times_out_and_is_retried_never_completed` -- a hung mail-backend call is bounded by the timeout, never marked completed, retry bumped. - `a_timed_out_item_does_not_block_a_later_item_in_a_subsequent_pass` -- proves a timed-out item doesn't jam the queue for a later drain pass. - `shutdown_signal_interrupts_a_hung_item_instead_of_waiting_out_the_timeout` -- shutdown observed mid-hang, well before the timeout would elapse. - [x] Existing outbox E2E tests still pass unmodified in behavior (signature updated to thread the shutdown signal through). Refs #290

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/nuncio#349