feat(nunciod): graceful shutdown with WAL-checkpointing close #340

2026-07-31T18:53:25Z

KofTwentyTwo commented

2026-07-31 18:53:25 +00:00

(Migrated from github.com)

Summary

Closes the gap in #289: nunciod previously installed no signal handler and never called DatabaseEngine::close(), so every real exit was an OS kill racing the WAL checkpoint that close() forces. This adds:

A nunciod::lifecycle module: ShutdownController/ShutdownSignal built on tokio::sync::watch (no new dependency; tokio_util is not a direct workspace dependency). install_signal_handlers awaits Ctrl+C, and on Unix also SIGTERM via tokio::signal::unix, then calls ShutdownController::trigger, which flips the shared signal AND calls EventBus::process_command(CoreCommand::Shutdown) -- finally exercising the existing CoreCommand::Shutdown/CoreEvent::ShuttingDown path in production instead of only from tests.
nunciod::grpc::serve_with_shutdown / serve_on_listener_with_shutdown / serve_on_listener_with_overrides_and_shutdown: new functions that call tonic's serve_with_incoming_shutdown. The existing serve/serve_on_listener/serve_on_listener_with_overrides signatures are untouched -- they now delegate to the shutdown-aware variant with a std::future::pending() shutdown future, so no existing test call site needed to change.
main.rs: one ShutdownSignal clone passed to the real-sync command consumer, the outbox worker, and the update-check loop, each rewritten as tokio::select! between the signal and its unit of work (a select! branch runs its body -- an in-flight DB write -- to completion once chosen, so nothing is aborted mid-write). After the gRPC server drains (bounded by a 10s grace period via tokio::select! against a timer that starts once shutdown fires), background tasks get the same bounded grace period to join, then DatabaseEngine::close() runs unconditionally before the process exits.

Test plan

New crates/nunciod/tests/graceful_shutdown_test.rs (2 tests, deterministic -- shutdown is triggered directly via ShutdownController::trigger(), never a real OS signal):

shutdown_signal_stops_grpc_server_exits_background_worker_and_closes_the_database: proves (a) the gRPC server accepts a call, then stops accepting connections once shutdown fires (bounded join, then a follow-up dial fails); (b) a select!-based worker ticks at least once, then exits its loop on the same signal (bounded join, not aborted); (c) DatabaseEngine::close() runs and a subsequent query fails with a pool-closed error.
controller_trigger_is_observed_by_every_clone_of_the_signal: multiple ShutdownSignal clones (taken before AND after the trigger) all observe it.

Plus 3 new unit tests in crates/nunciod/src/lifecycle.rs covering trigger/event-bus wiring, late-clone observation, and idempotence.

cargo fmt --all -- --check
cargo check-all (clippy --all-targets --workspace -D warnings) -- 0 warnings
cargo test-all -- 0 failed, 0 ignored (nunciod unit: 126 passed; nunciod integration suites: 21 passed across 9 files including the 2 new graceful-shutdown tests; full workspace green)

Refs #289

## Summary Closes the gap in #289: `nunciod` previously installed no signal handler and never called `DatabaseEngine::close()`, so every real exit was an OS kill racing the WAL checkpoint that `close()` forces. This adds: - A `nunciod::lifecycle` module: `ShutdownController`/`ShutdownSignal` built on `tokio::sync::watch` (no new dependency; `tokio_util` is not a direct workspace dependency). `install_signal_handlers` awaits Ctrl+C, and on Unix also SIGTERM via `tokio::signal::unix`, then calls `ShutdownController::trigger`, which flips the shared signal AND calls `EventBus::process_command(CoreCommand::Shutdown)` -- finally exercising the existing `CoreCommand::Shutdown`/`CoreEvent::ShuttingDown` path in production instead of only from tests. - `nunciod::grpc::serve_with_shutdown` / `serve_on_listener_with_shutdown` / `serve_on_listener_with_overrides_and_shutdown`: new functions that call tonic's `serve_with_incoming_shutdown`. The existing `serve`/`serve_on_listener`/`serve_on_listener_with_overrides` signatures are untouched -- they now delegate to the shutdown-aware variant with a `std::future::pending()` shutdown future, so no existing test call site needed to change. - `main.rs`: one `ShutdownSignal` clone passed to the real-sync command consumer, the outbox worker, and the update-check loop, each rewritten as `tokio::select!` between the signal and its unit of work (a `select!` branch runs its body -- an in-flight DB write -- to completion once chosen, so nothing is aborted mid-write). After the gRPC server drains (bounded by a 10s grace period via `tokio::select!` against a timer that starts once shutdown fires), background tasks get the same bounded grace period to join, then `DatabaseEngine::close()` runs unconditionally before the process exits. ## Test plan New `crates/nunciod/tests/graceful_shutdown_test.rs` (2 tests, deterministic -- shutdown is triggered directly via `ShutdownController::trigger()`, never a real OS signal): - `shutdown_signal_stops_grpc_server_exits_background_worker_and_closes_the_database`: proves (a) the gRPC server accepts a call, then stops accepting connections once shutdown fires (bounded join, then a follow-up dial fails); (b) a `select!`-based worker ticks at least once, then exits its loop on the same signal (bounded join, not aborted); (c) `DatabaseEngine::close()` runs and a subsequent query fails with a pool-closed error. - `controller_trigger_is_observed_by_every_clone_of_the_signal`: multiple `ShutdownSignal` clones (taken before AND after the trigger) all observe it. Plus 3 new unit tests in `crates/nunciod/src/lifecycle.rs` covering trigger/event-bus wiring, late-clone observation, and idempotence. - [x] `cargo fmt --all -- --check` - [x] `cargo check-all` (clippy `--all-targets --workspace -D warnings`) -- 0 warnings - [x] `cargo test-all` -- 0 failed, 0 ignored (nunciod unit: 126 passed; nunciod integration suites: 21 passed across 9 files including the 2 new graceful-shutdown tests; full workspace green) Refs #289

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/nuncio#340