feat(evaluation): write regressions, budget checks, site changes and comparisons (#8) #10

Merged
KofTwentyTwo merged 2 commits from feature/regressions-budgets into develop 2026-10-14 16:03:58 +00:00
KofTwentyTwo commented 2026-10-14 16:01:45 +00:00 (Migrated from github.com)

Part 1 of #8; part 2 (dropdowns and Tested Pages insert) was #9. The evaluation tables existed with list screens, but nothing wrote them.

Follows the web performance design (QRun-IO/qqq docs/specs/2026-10-12-web-performance-design.md):

  • Site changes: a new build id from the DeployProbe, or an event from a ChangeEventSource, becomes a durable change_event row (unique event_key, last_seen_at) and queues the affected pages. Lab runs record their deploymentId / change_event_id and cdnStage.
  • Budgets: enabled budgets matching a page (scope, tier, strategy, metric, comparator, threshold) are evaluated after each lab run and field import (evaluated_at checkpoints, so nothing is evaluated twice). A breach opens or refreshes a regression at the budget's severity.
  • Regressions: each new result is compared with the page, strategy and metric baseline using the design's rule. A regression is confirmed before it opens (one open per page, strategy and metric), gets linked to the likely change event, and resolves on recovery. A Lighthouse major-version change is recorded as a site change and resets the baseline (open lab regressions close). open_regression_count is now real, replacing the hard-coded 0.
  • Comparisons: before/after rows per change event, page, source, strategy, metric and segment.
  • Actions: Dismiss and Mark resolved are standard streamed-ETL record actions on regressions (Review and Result lines, permission-gated).
  • Alerts: optional webperf.regression alerts through core QAlerts (the logging sink when the Alerts qbit isn't installed), resolved when the regression closes.
  • Schema: changeset 11 adds the progress and identity columns plus unique indexes; existing history is kept.
  • Build: the QQQ BOM moves from 4.1.0-RC.4 to the released RC.6, which has core alerts.

Not done (SPI limits): RUM drift and country or window comparisons need capabilities the current RumSource SPI doesn't have. README notes it.

Tests: mvnw -o -B verify green after merging develop (#9). It covers 41 tests at 93.6% instruction coverage, and the guardrails pass. WebPerfRegressionTest uses fake runners and probes, with no network, to cover change events, budget breaches, confirmation, recovery, comparisons, the actions and idempotent re-runs.

🤖 Generated with Claude Code

Part 1 of #8; part 2 (dropdowns and Tested Pages insert) was #9. The evaluation tables existed with list screens, but nothing wrote them. Follows the web performance design (QRun-IO/qqq `docs/specs/2026-10-12-web-performance-design.md`): - **Site changes:** a new build id from the `DeployProbe`, or an event from a `ChangeEventSource`, becomes a durable `change_event` row (unique `event_key`, `last_seen_at`) and queues the affected pages. Lab runs record their `deploymentId` / `change_event_id` and `cdnStage`. - **Budgets:** enabled budgets matching a page (scope, tier, strategy, metric, comparator, threshold) are evaluated after each lab run and field import (`evaluated_at` checkpoints, so nothing is evaluated twice). A breach opens or refreshes a regression at the budget's severity. - **Regressions:** each new result is compared with the page, strategy and metric baseline using the design's rule. A regression is confirmed before it opens (one open per page, strategy and metric), gets linked to the likely change event, and resolves on recovery. A Lighthouse major-version change is recorded as a site change and resets the baseline (open lab regressions close). `open_regression_count` is now real, replacing the hard-coded 0. - **Comparisons:** before/after rows per change event, page, source, strategy, metric and segment. - **Actions:** **Dismiss** and **Mark resolved** are standard streamed-ETL record actions on regressions (Review and Result lines, permission-gated). - **Alerts:** optional `webperf.regression` alerts through core `QAlerts` (the logging sink when the Alerts qbit isn't installed), resolved when the regression closes. - **Schema:** changeset 11 adds the progress and identity columns plus unique indexes; existing history is kept. - **Build:** the QQQ BOM moves from 4.1.0-RC.4 to the released RC.6, which has core alerts. **Not done (SPI limits):** RUM drift and country or window comparisons need capabilities the current `RumSource` SPI doesn't have. README notes it. **Tests:** `mvnw -o -B verify` green after merging develop (#9). It covers 41 tests at 93.6% instruction coverage, and the guardrails pass. `WebPerfRegressionTest` uses fake runners and probes, with no network, to cover change events, budget breaches, confirmation, recovery, comparisons, the actions and idempotent re-runs. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign in to join this conversation.