M7: Replace the informational coverage job with an engine-crate gate + a required E2E CI job #208

New Issue

2026-07-28T14:04:59Z

KofTwentyTwo commented

2026-07-28 14:04:59 +00:00

(Migrated from github.com)

Context

Story for milestone M7 (reconciles #143). Read docs/STORY-WORKFLOW.md first — it defines the process, gates, and patterns; this issue defines only the specific work.

The old --fail-under-lines 100 coverage gate is already gone from .cargo/config.toml — the cov alias there today is llvm-cov --workspace --ignore-filename-regex main\.rs --summary-only (no threshold at all), and .github/workflows/ci.yml's coverage job (lines 57-76) runs the equivalent and is explicitly commented as "informational for now... A realistic threshold plus a required mock-server E2E gate lands with Phase 1.C.6 (GH #143)". So the gamed 100%-line gate is already retired; what's missing is the replacement the comment promises: a real, non-gamed threshold and a required E2E job. Separately, CLAUDE.md (root) still documents cargo cov as llvm-cov --workspace --fail-under-lines 100 — stale text that no longer matches .cargo/config.toml and needs to be corrected as part of this story so the docs stop contradicting the actual gate.

Goal

Replace the purely-informational coverage job with two real gates: (1) a coverage floor scoped to the headless engine crates only (nuncio-core, nuncio-store, nuncio-mail, nuncio-cal, nuncio-contacts, nuncio-filter — the crates CLAUDE.md's "Library-first boundary" section identifies as holding business logic), excluding nunciod/nuncio-cli/nuncio-proto (integration wiring and generated code, which the M6/M7 E2E suites cover instead), set from real measured numbers rather than guessed; and (2) a CI job that runs the offline mock-server E2E suites (spine_e2e_test, daemon_e2e_test, grpc_auth_test, mcp_system_test/ipc_system_test if still present, and the M7-S3 RPC-coverage matrix) as its own named, must-pass job distinct from the general cargo test --workspace job — so E2E regressions are visible as their own CI status, not buried inside a generic "test" pass. Must not fabricate a coverage number: the threshold is derived from an actual cargo llvm-cov run against the current tree, not picked arbitrarily.

Files to touch / patterns to mirror

.cargo/config.toml — cov alias (line 5); change its scope from --workspace to the six named engine crates (-p nuncio-core -p nuncio-store -p nuncio-mail -p nuncio-cal -p nuncio-contacts -p nuncio-filter) and add --fail-under-lines <N>.
.github/workflows/ci.yml — coverage job (lines 57-76): update its command to match the new scoped-and-thresholded cargo cov alias, drop the "informational for now" comment (it's no longer informational), and add a new job (e.g. e2e) alongside fmt/clippy/test/coverage that runs the specific E2E test binaries; add e2e to build-matrix's needs: list (line 80) alongside the existing jobs so a broken E2E suite blocks the cross-platform build stage exactly like a failing test/clippy job does today.
CLAUDE.md (root, "Build, test, and quality gates" section) — fix the stale cargo cov alias documentation (currently says --fail-under-lines 100) to describe the new engine-crate-scoped, realistically-thresholded gate; update the "Gate status" note if it becomes inaccurate as a result.
docs/DEVELOPING-IN-RUSTROVER.md — the shared Cargo Coverage run configuration is currently described as "informational"; update that description to match once the gate is real.

Implementation steps

Run cargo llvm-cov -p nuncio-core -p nuncio-store -p nuncio-mail -p nuncio-cal -p nuncio-contacts -p nuncio-filter --summary-only locally (or in a scratch CI run) against the current dev tip to get real, current line-coverage numbers per crate and in aggregate. Do not guess a number.
Pick the threshold as the aggregate figure from step 1, rounded down to the nearest 5%, so the gate starts as a floor under what's already true today (a real regression-prevention gate) rather than an aspirational target nobody currently meets. Record the raw measured numbers in the PR body.
Update .cargo/config.toml's cov alias to the scoped, thresholded form.
Update .github/workflows/ci.yml's coverage job to run that same alias (so CI and local cargo cov are identical, matching the repo's existing "green locally == green in CI" principle) and to actually fail the job below the threshold (remove the "informational" framing/comment).
Add the new e2e job to ci.yml: run the specific E2E test targets (e.g. cargo test -p nunciod --test spine_e2e_test --test daemon_e2e_test --test grpc_auth_test plus any others identified during M7-S3) as a separately-named, separately-visible status check; wire it into build-matrix's needs:.
Fix CLAUDE.md's coverage documentation and docs/DEVELOPING-IN-RUSTROVER.md's Cargo Coverage description to match the new reality.
Run cargo verify plus the new cargo cov alias locally to confirm both gates pass as configured.

Acceptance criteria

.cargo/config.toml's cov alias is scoped to the six named engine crates and includes a --fail-under-lines <N> threshold, where N is a real number measured in step 1 of the implementation steps (pasted in the PR body alongside the raw cargo llvm-cov output that produced it) — not an invented figure.
.github/workflows/ci.yml's coverage job runs the same scoped/thresholded command and is a real, failing-when-below-threshold gate (no "informational" framing left in its comment or behavior).
A new e2e CI job runs the offline mock-server E2E suites as its own named status check and is added to build-matrix's needs: list.
CLAUDE.md and docs/DEVELOPING-IN-RUSTROVER.md no longer describe coverage as a 100%-line gate or as purely informational; both match .cargo/config.toml's actual alias.
cargo verify green (fmt + clippy -D warnings all-targets + test-all), 0 ignored; cargo cov (the new alias) passes locally against the current tree.

Definition of done & process

Follow docs/STORY-WORKFLOW.md: no fabricated data — this explicitly includes not fabricating the coverage threshold; comment convention (intent-only, no story/GH/phase refs in .rs/.proto; CI YAML is not covered by that restriction but keep the #143 reference in ci.yml's comment out once this story lands, since the gap it describes will be closed). Branch story/m7-s4-coverage-gate off dev; PR to dev with the gate output (both cargo verify and the new cargo cov) pasted as evidence; the PO reviews before the issue is closed.

Dependencies / notes

Not a [PROTO] story. Benefits from M7-S3 landing first (its RPC-coverage-matrix test is a natural addition to the new e2e job), but can proceed independently and just add that test to the job afterward if ordering doesn't allow it. CLAUDE.md's existing note that "CI itself is paused until the GitHub Actions minutes reset" may mean this story's CI changes cannot be observed running on GitHub Actions until minutes are available again — verify locally (act, or by reasoning through the YAML) and say so honestly in the PR if a live CI run could not be confirmed. Size M.

## Context Story for milestone M7 (reconciles #143). **Read [`docs/STORY-WORKFLOW.md`](https://github.com/KofTwentyTwo/nuncio/blob/dev/docs/STORY-WORKFLOW.md) first** — it defines the process, gates, and patterns; this issue defines only the specific work. The old `--fail-under-lines 100` coverage gate is already gone from `.cargo/config.toml` — the `cov` alias there today is `llvm-cov --workspace --ignore-filename-regex main\.rs --summary-only` (no threshold at all), and `.github/workflows/ci.yml`'s `coverage` job (lines 57-76) runs the equivalent and is explicitly commented as "informational for now... A realistic threshold plus a required mock-server E2E gate lands with Phase 1.C.6 (GH #143)". So the gamed 100%-line gate is already retired; what's missing is the *replacement* the comment promises: a real, non-gamed threshold and a required E2E job. Separately, `CLAUDE.md` (root) still documents `cargo cov` as `llvm-cov --workspace --fail-under-lines 100` — stale text that no longer matches `.cargo/config.toml` and needs to be corrected as part of this story so the docs stop contradicting the actual gate. ## Goal Replace the purely-informational coverage job with two real gates: (1) a coverage *floor* scoped to the headless engine crates only (`nuncio-core`, `nuncio-store`, `nuncio-mail`, `nuncio-cal`, `nuncio-contacts`, `nuncio-filter` — the crates `CLAUDE.md`'s "Library-first boundary" section identifies as holding business logic), excluding `nunciod`/`nuncio-cli`/`nuncio-proto` (integration wiring and generated code, which the M6/M7 E2E suites cover instead), set from real measured numbers rather than guessed; and (2) a CI job that runs the offline mock-server E2E suites (`spine_e2e_test`, `daemon_e2e_test`, `grpc_auth_test`, `mcp_system_test`/`ipc_system_test` if still present, and the M7-S3 RPC-coverage matrix) as its own named, must-pass job distinct from the general `cargo test --workspace` job — so E2E regressions are visible as their own CI status, not buried inside a generic "test" pass. Must not fabricate a coverage number: the threshold is derived from an actual `cargo llvm-cov` run against the current tree, not picked arbitrarily. ## Files to touch / patterns to mirror - `.cargo/config.toml` — `cov` alias (line 5); change its scope from `--workspace` to the six named engine crates (`-p nuncio-core -p nuncio-store -p nuncio-mail -p nuncio-cal -p nuncio-contacts -p nuncio-filter`) and add `--fail-under-lines <N>`. - `.github/workflows/ci.yml` — `coverage` job (lines 57-76): update its command to match the new scoped-and-thresholded `cargo cov` alias, drop the "informational for now" comment (it's no longer informational), and add a new job (e.g. `e2e`) alongside `fmt`/`clippy`/`test`/`coverage` that runs the specific E2E test binaries; add `e2e` to `build-matrix`'s `needs:` list (line 80) alongside the existing jobs so a broken E2E suite blocks the cross-platform build stage exactly like a failing `test`/`clippy` job does today. - `CLAUDE.md` (root, "Build, test, and quality gates" section) — fix the stale `cargo cov` alias documentation (currently says `--fail-under-lines 100`) to describe the new engine-crate-scoped, realistically-thresholded gate; update the "Gate status" note if it becomes inaccurate as a result. - `docs/DEVELOPING-IN-RUSTROVER.md` — the shared *Cargo Coverage* run configuration is currently described as "informational"; update that description to match once the gate is real. ## Implementation steps 1. Run `cargo llvm-cov -p nuncio-core -p nuncio-store -p nuncio-mail -p nuncio-cal -p nuncio-contacts -p nuncio-filter --summary-only` locally (or in a scratch CI run) against the current `dev` tip to get real, current line-coverage numbers per crate and in aggregate. Do not guess a number. 2. Pick the threshold as the aggregate figure from step 1, rounded **down** to the nearest 5%, so the gate starts as a floor under what's already true today (a real regression-prevention gate) rather than an aspirational target nobody currently meets. Record the raw measured numbers in the PR body. 3. Update `.cargo/config.toml`'s `cov` alias to the scoped, thresholded form. 4. Update `.github/workflows/ci.yml`'s `coverage` job to run that same alias (so CI and local `cargo cov` are identical, matching the repo's existing "green locally == green in CI" principle) and to actually fail the job below the threshold (remove the "informational" framing/comment). 5. Add the new `e2e` job to `ci.yml`: run the specific E2E test targets (e.g. `cargo test -p nunciod --test spine_e2e_test --test daemon_e2e_test --test grpc_auth_test` plus any others identified during M7-S3) as a separately-named, separately-visible status check; wire it into `build-matrix`'s `needs:`. 6. Fix `CLAUDE.md`'s coverage documentation and `docs/DEVELOPING-IN-RUSTROVER.md`'s Cargo Coverage description to match the new reality. 7. Run `cargo verify` plus the new `cargo cov` alias locally to confirm both gates pass as configured. ## Acceptance criteria - [ ] `.cargo/config.toml`'s `cov` alias is scoped to the six named engine crates and includes a `--fail-under-lines <N>` threshold, where `N` is a real number measured in step 1 of the implementation steps (pasted in the PR body alongside the raw `cargo llvm-cov` output that produced it) — not an invented figure. - [ ] `.github/workflows/ci.yml`'s `coverage` job runs the same scoped/thresholded command and is a real, failing-when-below-threshold gate (no "informational" framing left in its comment or behavior). - [ ] A new `e2e` CI job runs the offline mock-server E2E suites as its own named status check and is added to `build-matrix`'s `needs:` list. - [ ] `CLAUDE.md` and `docs/DEVELOPING-IN-RUSTROVER.md` no longer describe coverage as a 100%-line gate or as purely informational; both match `.cargo/config.toml`'s actual alias. - [ ] `cargo verify` green (fmt + clippy `-D warnings` all-targets + test-all), 0 ignored; `cargo cov` (the new alias) passes locally against the current tree. ## Definition of done & process Follow [`docs/STORY-WORKFLOW.md`](https://github.com/KofTwentyTwo/nuncio/blob/dev/docs/STORY-WORKFLOW.md): no fabricated data — this explicitly includes not fabricating the coverage threshold; comment convention (intent-only, no story/GH/phase refs in `.rs`/`.proto`; CI YAML is not covered by that restriction but keep the `#143` reference in `ci.yml`'s comment out once this story lands, since the gap it describes will be closed). Branch `story/m7-s4-coverage-gate` off `dev`; PR to `dev` with the gate output (both `cargo verify` and the new `cargo cov`) pasted as evidence; the PO reviews before the issue is closed. ## Dependencies / notes Not a `[PROTO]` story. Benefits from M7-S3 landing first (its RPC-coverage-matrix test is a natural addition to the new `e2e` job), but can proceed independently and just add that test to the job afterward if ordering doesn't allow it. `CLAUDE.md`'s existing note that "CI itself is paused until the GitHub Actions minutes reset" may mean this story's CI changes cannot be *observed* running on GitHub Actions until minutes are available again — verify locally (`act`, or by reasoning through the YAML) and say so honestly in the PR if a live CI run could not be confirmed. Size M.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/nuncio#208