feat(filter): add streaming Filters.Triage RPC for retroactive rescan #226

2026-07-30T00:50:38Z

KofTwentyTwo commented

2026-07-30 00:50:38 +00:00

(Migrated from github.com)

Summary

Closes #193 (M4 [PROTO], the last M4 story).

Adds a Filters.Triage server-streaming RPC that retroactively applies the daemon's currently-active rule set to every stored message — for mail that was synced before those rules existed.

Proto (crates/nuncio-proto/proto/nuncio/v1/nuncio.proto): additive rpc Triage(TriageRequest) returns (stream TriageProgress) on service Filters, plus TriageRequest/TriageProgress messages. The committed golden (descriptor.bin) was regenerated per STORY-WORKFLOW.md §5 and contract_stability::descriptor_matches_committed_golden passes.
Daemon (crates/nunciod/src/grpc.rs, FiltersGrpcService::triage): this is a genuinely new pattern versus System::subscribe (which wraps an already-live broadcast) — a tokio::spawned worker owns the scan loop, walks DatabaseEngine::get_message_chunk in keyset pages, applies the SAME apply_filter_actions helper #190 introduced (no second copy of action-routing) to each message, and pushes Ok(TriageProgress)/Err(Status) onto an mpsc::channel wrapped in ReceiverStream as the returned stream. A get_message_chunk failure sends one Err(Status::internal) and stops — never a fabricated done: true. The receiver dropping (caller disconnects) also stops the worker.
CLI: nuncio filter triage [--rule-id <id>] [--chunk-size <n>] — the CLI's first real streaming-RPC consumer (handle_filter_triage drains tonic::Streaming<TriageProgress> via repeated .message().await, printing/accumulating progress, respecting --json).
Tests: new crates/nunciod/tests/filters_triage_e2e_test.rs (offline, mock-only) proves (a) correct final cumulative matched_count/actions_applied_count against a mixed matching/non-matching store, (b) the SAME real side effects as live sync (MARK READ flips the flag; MOVE TO queues a real outbox row), and (c) genuine multi-chunk streaming (5 messages / chunk_size=2 yields >1 non-final progress message). grpc_auth_test.rs gained rejects_triage_calls_without_a_bearer_token (unauthenticated → Code::Unauthenticated, no new mounting needed — Triage lives on the already-guarded Filters service). nuncio-cli's stub-daemon test exercises filter triage end to end against a stub streaming server.

Decisions to confirm

rule_id: accepted on the wire for forward compatibility but NOT currently honored. apply_filter_actions evaluates against the whole live FilterEngine rule set (exactly what a real sync pass does); scoping to one rule would require a second, divergent evaluation path, which isn't worth it yet. This is stated explicitly in the proto comment and in FiltersGrpcService::triage's doc comment — a client setting rule_id today gets a full rescan, never a silently-ignored no-op passed off as scoped.
matched_count definition: a message counts as matched if apply_filter_actions applied/enqueued at least one action for it (i.e. any rule fired on it), not "number of rules that matched." Documented in the proto comment on TriageProgress.matched_count.

Gate output

cargo fmt --all -- --check   -> clean, no diff
cargo check-all              -> clean, 0 warnings (clippy --all-targets --workspace -D warnings)
cargo test-all                -> 398 passed; 0 failed; 0 ignored (workspace unit + integration + doc tests)
cargo test -p nuncio-proto contract_stability
   -> test contract_stability::descriptor_matches_committed_golden ... ok

The pre-commit hook re-ran the identical fmt/clippy/test gate on commit and passed cleanly ("All Local Pre-Commit Security Checks Passed Cleanly").

Test plan

cargo test -p nunciod --test filters_triage_e2e_test — multi-chunk streaming + real side effects
cargo test -p nunciod --test grpc_auth_test — unauthenticated Triage rejected
cargo test -p nuncio-cli filter — CLI filter triage round-trips against a stub streaming server
cargo test -p nuncio-proto contract_stability — golden matches regenerated descriptor
Full cargo test-all green, 0 ignored

## Summary Closes #193 (M4 [PROTO], the last M4 story). Adds a `Filters.Triage` server-streaming RPC that retroactively applies the daemon's currently-active rule set to every stored message — for mail that was synced before those rules existed. - **Proto** (`crates/nuncio-proto/proto/nuncio/v1/nuncio.proto`): additive `rpc Triage(TriageRequest) returns (stream TriageProgress)` on `service Filters`, plus `TriageRequest`/`TriageProgress` messages. The committed golden (`descriptor.bin`) was regenerated per STORY-WORKFLOW.md §5 and `contract_stability::descriptor_matches_committed_golden` passes. - **Daemon** (`crates/nunciod/src/grpc.rs`, `FiltersGrpcService::triage`): this is a genuinely new pattern versus `System::subscribe` (which wraps an already-live broadcast) — a `tokio::spawn`ed worker owns the scan loop, walks `DatabaseEngine::get_message_chunk` in keyset pages, applies the SAME `apply_filter_actions` helper #190 introduced (no second copy of action-routing) to each message, and pushes `Ok(TriageProgress)`/`Err(Status)` onto an `mpsc::channel` wrapped in `ReceiverStream` as the returned stream. A `get_message_chunk` failure sends one `Err(Status::internal)` and stops — never a fabricated `done: true`. The receiver dropping (caller disconnects) also stops the worker. - **CLI**: `nuncio filter triage [--rule-id <id>] [--chunk-size <n>]` — the CLI's first real streaming-RPC consumer (`handle_filter_triage` drains `tonic::Streaming<TriageProgress>` via repeated `.message().await`, printing/accumulating progress, respecting `--json`). - **Tests**: new `crates/nunciod/tests/filters_triage_e2e_test.rs` (offline, mock-only) proves (a) correct final cumulative `matched_count`/`actions_applied_count` against a mixed matching/non-matching store, (b) the SAME real side effects as live sync (MARK READ flips the flag; MOVE TO queues a real outbox row), and (c) genuine multi-chunk streaming (5 messages / chunk_size=2 yields >1 non-final progress message). `grpc_auth_test.rs` gained `rejects_triage_calls_without_a_bearer_token` (unauthenticated → `Code::Unauthenticated`, no new mounting needed — `Triage` lives on the already-guarded `Filters` service). `nuncio-cli`'s stub-daemon test exercises `filter triage` end to end against a stub streaming server. ## Decisions to confirm - **`rule_id`**: accepted on the wire for forward compatibility but **NOT currently honored**. `apply_filter_actions` evaluates against the whole live `FilterEngine` rule set (exactly what a real sync pass does); scoping to one rule would require a second, divergent evaluation path, which isn't worth it yet. This is stated explicitly in the proto comment and in `FiltersGrpcService::triage`'s doc comment — a client setting `rule_id` today gets a full rescan, never a silently-ignored no-op passed off as scoped. - **`matched_count` definition**: a message counts as matched if `apply_filter_actions` applied/enqueued **at least one action** for it (i.e. any rule fired on it), not "number of rules that matched." Documented in the proto comment on `TriageProgress.matched_count`. ## Gate output ``` cargo fmt --all -- --check -> clean, no diff cargo check-all -> clean, 0 warnings (clippy --all-targets --workspace -D warnings) cargo test-all -> 398 passed; 0 failed; 0 ignored (workspace unit + integration + doc tests) cargo test -p nuncio-proto contract_stability -> test contract_stability::descriptor_matches_committed_golden ... ok ``` The pre-commit hook re-ran the identical fmt/clippy/test gate on commit and passed cleanly ("All Local Pre-Commit Security Checks Passed Cleanly"). ## Test plan - [x] `cargo test -p nunciod --test filters_triage_e2e_test` — multi-chunk streaming + real side effects - [x] `cargo test -p nunciod --test grpc_auth_test` — unauthenticated `Triage` rejected - [x] `cargo test -p nuncio-cli filter` — CLI `filter triage` round-trips against a stub streaming server - [x] `cargo test -p nuncio-proto contract_stability` — golden matches regenerated descriptor - [x] Full `cargo test-all` green, 0 ignored

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/nuncio#226