From 0745f331f41273f72cd9489c8af87c2fe797f735 Mon Sep 17 00:00:00 2001 From: James Maes Date: Wed, 14 Oct 2026 11:00:41 -0500 Subject: [PATCH] feat(evaluation): write regressions, budget checks, site changes and comparisons The regression, budget, change event and comparison tables had list screens but nothing wrote them; the open regression count was a hard-coded 0 and budgets were never evaluated. - Change events: new build ids from the DeployProbe and ChangeEventSource events become durable, idempotent change_event rows (event_key); lab runs record the deployment (change event) and CDN stage. - Budgets are evaluated after lab runs and field imports; breaches open or refresh regressions. - Regression detection compares new results with the baseline per page, strategy and metric as the design describes, links the likely change, confirms before opening, and resolves on recovery. The open regression count is real. - Before/after comparisons are written per change event. - Dismiss and Mark resolved are standard record actions on regressions. - Optional alerts through core QAlerts (logging sink without the Alerts qbit). QQQ BOM moves to the released 4.1.0-RC.6 for core alerts. - Changeset 11 adds the progress and identity columns. Refs #8 Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 12 + README.md | 18 +- pom.xml | 4 +- .../qbits/webperf/lab/LabResultParser.java | 47 + .../webperf/metadata/WebPerfMetaData.java | 53 +- .../webperf/processes/RegressionSteps.java | 276 ++++++ .../webperf/processes/RetryRunsSteps.java | 19 +- .../webperf/processes/ScheduledSteps.java | 25 + .../webperf/processes/WebPerfProcesses.java | 28 + .../webperf/service/WebPerfChangeService.java | 246 ++++++ .../service/WebPerfComparisonService.java | 188 ++++ .../service/WebPerfEvaluationService.java | 611 +++++++++++++ .../qbits/webperf/service/WebPerfService.java | 72 +- .../qbits/webperf/store/WebPerfStore.java | 88 +- .../web_performance.changelog.sql | 19 + .../qbits/webperf/WebPerfP1Test.java | 2 + .../qbits/webperf/WebPerfRegressionTest.java | 828 ++++++++++++++++++ 17 files changed, 2501 insertions(+), 35 deletions(-) create mode 100644 src/main/java/com/kingsrook/qbits/webperf/processes/RegressionSteps.java create mode 100644 src/main/java/com/kingsrook/qbits/webperf/service/WebPerfChangeService.java create mode 100644 src/main/java/com/kingsrook/qbits/webperf/service/WebPerfComparisonService.java create mode 100644 src/main/java/com/kingsrook/qbits/webperf/service/WebPerfEvaluationService.java create mode 100644 src/test/java/com/kingsrook/qbits/webperf/WebPerfRegressionTest.java diff --git a/CHANGELOG.md b/CHANGELOG.md index 9d8b84d..e6d4e91 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,4 +5,16 @@ All notable changes to this project are documented here. The format follows Keep ## [0.1.0] - Unreleased ### Added +- Record deploy and external site changes once, queue settled retests, and attribute lab runs to the observed build and CDN stage. +- Evaluate enabled lab and URL CrUX budgets; retain breaches, confirmation evidence, automatic recovery and accurate open counts in the standard regression records. +- Confirm statistical lab regressions with a median of three, reset baselines on Lighthouse major changes, and detect deterministic category/audit failures. +- Add permission-gated Dismiss, Mark resolved and Compare before and after record actions with standard Review/Result summaries. +- Store before/after lab medians, sample counts and windows, including measured TTFB; schedule CDN samples before launch and at one and 24 hours after launch. +- Raise and resolve optional alerts through QQQ core, with logging fallback when no Alerts QBit is installed. + +### Changed +- Use QQQ 4.1.0-RC.6 for the core alerts API; add migration fields for durable evaluation and source-event identity. +- Store complete run results atomically and retain confirmation/comparison identities when retrying samples. + +### Previously added - First implementation; see README.md for the tables, permissions and configuration. diff --git a/README.md b/README.md index f981f6a..7fe5c35 100644 --- a/README.md +++ b/README.md @@ -13,15 +13,19 @@ Design and owner decisions: QRun-IO/qqq `docs/specs/2026-10-12-web-performance-d ## Status -This is a **skeleton (0.1.0-SNAPSHOT)**. It is local only, and the GitHub repository does not exist yet. It contains: +The lab and Google field-data runners, queue, schedules, record actions and performance widgets are implemented. This change adds durable site changes, budget evaluation, confirmed lab regressions and before/after lab comparisons. -- the producer and config, with validation and secret redaction -- the nine `web_performance_` tables, their lookups and the Performance app -- the host-included changelog, including the seeded budgets (D12) -- the SPIs: `UrlSource`, `ContentResolver`, `LabRunner`, `DeployProbe`, `RumSource`, `ChangeEventSource` -- two implementations that need no Google API key: `SitemapUrlSource` and `HttpBuildIdProbe`, which reads Website-Frontend's `/api/build` +- Deploy probes run at startup through the scheduler and every five minutes. New build ids are recorded once per component, and KEY pages are queued after `deploySettleSeconds`. External `ChangeEventSource` events are deduplicated by source registration and their complete event tuple. Keep source registration order stable across restarts. Content events queue enabled pages after `publishSettleSeconds`, because the current SPI does not identify affected target ids. +- Each run records the latest observed frontend build, hostname CDN stage and preceding site change when it starts. A rollback to an already recorded build updates its last observation without duplicating its change row. +- Enabled LAB budgets match site, tier, target, content table or literal path prefix, plus strategy and metric. Comparators describe acceptable values. Breaches are recorded in Regressions with the budget id; there is no separate budget-check table in the design. The strongest matching severity wins. CRUX budgets evaluate URL p75 snapshots; NO_DATA, errors and origin fallback are never treated as page measurements. +- Statistical lab baselines require five successful SCHEDULE, CONFIRM or SITE_AUDIT runs in the preceding 30 days on the same Lighthouse major version. Performance drops of 10 points, LCP increases of both 25% and 500 ms, CLS increases of 0.05, TBT increases of 200 ms, and budget breaches create suspected regressions. Two independent confirmation runs provide a median of three. Category drops of 5 points and newly failing Accessibility, Best Practices or SEO audits confirm immediately. Recovery requires a median of three later runs. Lighthouse major changes record a marker and reset incompatible baselines. +- The page's open count includes suspected and confirmed regressions. **Dismiss** and **Mark resolved** require a note and `web_performance.manage.hasAccess`, use standard Review/Result summary lines, and preserve history. +- **Compare before and after** uses three successful same-version lab samples on each side of a change, with page, sample counts and actual time windows. It stores Performance, LCP, FCP and measured TTFB when available. Results are also updated after attributed runs complete. CDN HEADERS_ONLY queues three samples per strategy; LIVE queues three at +1 hour and three at +24 hours. Insufficient history produces no invented comparison. +- Confirmed regressions raise keyed `webperf.regression` alerts through QQQ core `QAlerts`; recovery, dismissal and manual resolution clear them. `ALERTS` can be disabled. Without an Alerts QBit, core uses its logging sink. No Alerts QBit dependency is required. -Not built yet: the PSI and CrUX runners (they need the key), the queue worker, the schedules, the actions, the widgets, and the Alerts qbit producer integration. +Evaluation progress and related writes commit together. Retrying an evaluation does not duplicate incidents, and retrying a failed confirmation preserves its sample reference. The additive changelog must be applied before running the new worker. This version requires QQQ **4.1.0-RC.6** for the core alerts API. + +RUM drift and RUM/country before-and-after comparisons remain separate work: the existing `RumSource` exposes fixed daily/28-day aggregates, not arbitrary before/after windows. Enum field metadata and Tested Pages insertion are also outside this change. ## Install diff --git a/pom.xml b/pom.xml index 37769ed..07efbfb 100644 --- a/pom.xml +++ b/pom.xml @@ -143,14 +143,14 @@ - + qqq-snapshot true - 4.1.0-RC.4 + 4.1.0-RC.6 diff --git a/src/main/java/com/kingsrook/qbits/webperf/lab/LabResultParser.java b/src/main/java/com/kingsrook/qbits/webperf/lab/LabResultParser.java index 0ae6a35..2aa3ace 100644 --- a/src/main/java/com/kingsrook/qbits/webperf/lab/LabResultParser.java +++ b/src/main/java/com/kingsrook/qbits/webperf/lab/LabResultParser.java @@ -139,6 +139,53 @@ public final class LabResultParser + /******************************************************************************* + ** Read a reported TTFB subpart from Lighthouse's LCP insight, never savings. + *******************************************************************************/ + public static Integer ttfb(String json) + { + JSONObject root = new JSONObject(json); + JSONObject lhr = root.optJSONObject("lighthouseResult"); + JSONObject audits = (lhr == null ? root : lhr).optJSONObject("audits"); + if(audits == null) + { + return (null); + } + for(String name : List.of("lcp-phases-insight", "lcp-breakdown-insight")) + { + JSONObject audit = audits.optJSONObject(name); + JSONObject details = audit == null ? null : audit.optJSONObject("details"); + JSONArray items = details == null ? null : details.optJSONArray("items"); + if(items != null) + { + for(int i = 0; i < items.length(); i++) + { + JSONObject item = items.optJSONObject(i); + if(item == null) + { + continue; + } + JSONArray subparts = item.optJSONArray("items"); + if(subparts == null) + { + subparts = new JSONArray().put(item); + } + for(int j = 0; j < subparts.length(); j++) + { + JSONObject part = subparts.optJSONObject(j); + if(part != null && "timeToFirstByte".equals(part.optString("subpart")) && part.opt("duration") instanceof Number duration) + { + return ((int) Math.round(duration.doubleValue())); + } + } + } + } + } + return (null); + } + + + /******************************************************************************* ** Parser only. *******************************************************************************/ diff --git a/src/main/java/com/kingsrook/qbits/webperf/metadata/WebPerfMetaData.java b/src/main/java/com/kingsrook/qbits/webperf/metadata/WebPerfMetaData.java index a002525..cd4946f 100644 --- a/src/main/java/com/kingsrook/qbits/webperf/metadata/WebPerfMetaData.java +++ b/src/main/java/com/kingsrook/qbits/webperf/metadata/WebPerfMetaData.java @@ -26,20 +26,24 @@ import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import java.util.Set; +import com.kingsrook.qbits.webperf.WebPerfFeature; import com.kingsrook.qbits.webperf.WebPerfQBitConfig; import com.kingsrook.qbits.webperf.WebPerfRuntime; import com.kingsrook.qbits.webperf.WebPerfSchema; import com.kingsrook.qbits.webperf.processes.WebPerfProcesses; +import com.kingsrook.qbits.webperf.service.WebPerfEvaluationService; import com.kingsrook.qbits.webperf.widgets.WebPerfWidgets; import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer; -import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType; -import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; -import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData; -import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData; +import com.kingsrook.qqq.backend.core.alerts.AlertSeverity; +import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData; +import com.kingsrook.qqq.backend.core.alerts.QAlertTypeMetaData; import com.kingsrook.qqq.backend.core.instances.QInstanceEnricher; +import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType; import com.kingsrook.qqq.backend.core.model.metadata.QInstance; import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel; import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData; import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData; @@ -48,6 +52,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon; import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel; import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules; import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData; import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability; import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection; import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; @@ -98,6 +103,16 @@ public final class WebPerfMetaData *******************************************************************************/ public void addTo(QInstance qInstance) { + if(config.isOn(WebPerfFeature.ALERTS)) + { + AlertsInstanceMetaData.ofOrWithNew(qInstance).withAlertType( + new QAlertTypeMetaData() + .withName(WebPerfEvaluationService.ALERT_TYPE) + .withLabel("Web performance regression").withSourceQBitName("web-performance") + .withDefaultSeverity(AlertSeverity.WARNING) + .withSubjectTableName(config.applyPrefix(WebPerfSchema.REGRESSION)) + .withVisibilityPermission(config.getPermissionBaseName() + ".read")); + } for(QTableMetaData table : tables().values()) { qInstance.addTable(table); @@ -277,7 +292,10 @@ public final class WebPerfMetaData f("lcpElement", QFieldType.STRING, "LCP Element").withMaxLength(300), f("runWarnings", QFieldType.STRING, "Warnings").withMaxLength(2000), f("deploymentId", QFieldType.STRING, "Frontend Build").withMaxLength(64), - f("cdnStage", QFieldType.STRING, "CDN Stage").withMaxLength(20))); + f("cdnStage", QFieldType.STRING, "CDN Stage").withMaxLength(20), + f("ttfbMs", QFieldType.INTEGER, "TTFB (ms)"), + f("evaluatedAt", QFieldType.DATE_TIME, "Evaluated"), + f("changeEventId", QFieldType.LONG, "Site Change").withPossibleValueSourceName(config.applyPrefix(WebPerfSchema.CHANGE_EVENT)))); return (table(WebPerfSchema.LAB_RUN, "Lab Runs", "speed", fields, false, Set.of()) .withRecordLabelFormatAndFields("%s %s", "strategy", "url") @@ -286,7 +304,7 @@ public final class WebPerfMetaData .withSection(section("metrics", "Lab Metrics", "timer", Tier.T2, "fcpMs", "lcpMs", "tbtMs", "cls", "speedIndexMs", "lcpElement")) .withSection(section("details", "Details", "info", Tier.T2, "targetId", "finalUrl", "runner", "triggerRef", "batchId", "notBefore", "queuedAt", "startedAt", "attempts", "httpStatus", "errorMessage", "lighthouseVersion", "chromeUserAgent", "benchmarkIndex", - "runWarnings", "deploymentId", "cdnStage")) + "runWarnings", "deploymentId", "cdnStage", "ttfbMs", "evaluatedAt", "changeEventId")) .withSection(dates())); } @@ -363,11 +381,12 @@ public final class WebPerfMetaData f("p75FcpMs", QFieldType.INTEGER, "FCP p75 (ms)"), f("p75TtfbMs", QFieldType.INTEGER, "TTFB p75 (ms)"), f("distributions", QFieldType.TEXT, "Distributions"), - f("lastCheckedAt", QFieldType.DATE_TIME, "Last Checked"))); + f("lastCheckedAt", QFieldType.DATE_TIME, "Last Checked"), + f("evaluatedAt", QFieldType.DATE_TIME, "Evaluated"))); return (table(WebPerfSchema.FIELD_SNAPSHOT, "Google Field Data", "public", fields, false, Set.of()) .withRecordLabelFormatAndFields("%s %s", "scopeKey", "formFactor") - .withSection(section("snapshot", "Snapshot", "public", Tier.T1, "id", "scopeKey", "scope", "formFactor", "status", "lastCheckedAt")) + .withSection(section("snapshot", "Snapshot", "public", Tier.T1, "id", "scopeKey", "scope", "formFactor", "status", "lastCheckedAt", "evaluatedAt")) .withSection(section("p75", "p75 (28 days)", "insights", Tier.T2, "p75LcpMs", "p75InpMs", "p75Cls", "p75FcpMs", "p75TtfbMs", "periodStart", "periodEnd", "source", "distributions")) .withSection(dates())); @@ -427,14 +446,18 @@ public final class WebPerfMetaData f("detectedAt", QFieldType.DATE_TIME, "Detected"), f("confirmedAt", QFieldType.DATE_TIME, "Confirmed"), f("resolvedAt", QFieldType.DATE_TIME, "Resolved"), - f("resolutionNote", QFieldType.STRING, "Resolution Note").withMaxLength(2000))); + f("resolutionNote", QFieldType.STRING, "Resolution Note").withMaxLength(2000), + f("firstRunId", QFieldType.LONG, "First Run").withPossibleValueSourceName(config.applyPrefix(WebPerfSchema.LAB_RUN)), + f("checkpointRunId", QFieldType.LONG, "Confirmation Run").withPossibleValueSourceName(config.applyPrefix(WebPerfSchema.LAB_RUN)), + f("lighthouseMajor", QFieldType.STRING, "Lighthouse Major").withMaxLength(20), + f("failedAudits", QFieldType.STRING, "Newly Failing Audits").withMaxLength(2000))); return (table(WebPerfSchema.REGRESSION, "Regressions", "trending_down", fields, true, Set.of()) .withRecordLabelFormatAndFields("%s %s", "metric", "status") .withSection(section("regression", "Regression", "trending_down", Tier.T1, "id", "targetId", "metric", "strategy", "baselineValue", "currentValue", "delta", "severity", "status")) .withSection(section("details", "Details", "info", Tier.T2, "source", "contentTable", "contentId", "baselineRef", "currentRef", - "budgetId", "causeChangeEventId", "detectedAt", "confirmedAt", "resolvedAt", "resolutionNote")) + "budgetId", "causeChangeEventId", "detectedAt", "confirmedAt", "resolvedAt", "resolutionNote", "firstRunId", "checkpointRunId", "lighthouseMajor", "failedAudits")) .withSection(dates())); } @@ -452,12 +475,15 @@ public final class WebPerfMetaData f("toValue", QFieldType.STRING, "To").withMaxLength(200), f("observedAt", QFieldType.DATE_TIME, "Observed"), f("source", QFieldType.STRING, "Source").withMaxLength(20), - f("note", QFieldType.STRING, "Note").withMaxLength(2000))); + f("note", QFieldType.STRING, "Note").withMaxLength(2000), + f("lastSeenAt", QFieldType.DATE_TIME, "Last Observed"), + f("eventKey", QFieldType.STRING, "Source Event Key").withMaxLength(64), + f("siteKey", QFieldType.STRING, "Site").withMaxLength(200))); return (table(WebPerfSchema.CHANGE_EVENT, "Site Changes", "change_circle", fields, false, Set.of()) .withRecordLabelFormatAndFields("%s %s", "kind", "toValue") .withSection(section("change", "Change", "change_circle", Tier.T1, "id", "kind", "component", "fromValue", "toValue", "observedAt", - "source", "note")) + "source", "note", "lastSeenAt", "eventKey", "siteKey")) .withSection(dates())); } @@ -470,6 +496,7 @@ public final class WebPerfMetaData { List fields = withCommon(List.of( f("changeEventId", QFieldType.LONG, "Change").withPossibleValueSourceName(config.applyPrefix(WebPerfSchema.CHANGE_EVENT)), + f("targetId", QFieldType.LONG, "Page").withPossibleValueSourceName(config.applyPrefix(WebPerfSchema.TARGET)), f("scope", QFieldType.STRING, "Scope").withMaxLength(20), f("source", QFieldType.STRING, "Source").withMaxLength(10), f("strategy", QFieldType.STRING, "Strategy or Form Factor").withMaxLength(10), @@ -486,7 +513,7 @@ public final class WebPerfMetaData return (table(WebPerfSchema.COMPARISON, "Before and After", "compare_arrows", fields, false, Set.of()) .withRecordLabelFormatAndFields("%s %s", "metric", "segment") - .withSection(section("comparison", "Comparison", "compare_arrows", Tier.T1, "id", "changeEventId", "metric", "segment", "beforeValue", + .withSection(section("comparison", "Comparison", "compare_arrows", Tier.T1, "id", "changeEventId", "targetId", "metric", "segment", "beforeValue", "afterValue", "delta", "deltaPct")) .withSection(section("details", "Details", "info", Tier.T2, "scope", "source", "strategy", "beforeN", "beforeWindow", "afterN", "afterWindow")) diff --git a/src/main/java/com/kingsrook/qbits/webperf/processes/RegressionSteps.java b/src/main/java/com/kingsrook/qbits/webperf/processes/RegressionSteps.java new file mode 100644 index 0000000..528272e --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/webperf/processes/RegressionSteps.java @@ -0,0 +1,276 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.webperf.processes; + + +import java.sql.SQLException; +import java.time.Instant; +import java.util.ArrayList; +import java.util.List; +import com.kingsrook.qbits.webperf.WebPerfRuntime; +import com.kingsrook.qbits.webperf.service.WebPerfComparisonService; +import com.kingsrook.qbits.webperf.service.WebPerfEvaluationService; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine; +import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput; +import com.kingsrook.qqq.backend.core.model.actions.processes.Status; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractLoadStep; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractTransformStep; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ProcessSummaryProviderInterface; + + +/******************************************************************************* + ** Standard streamed Review/Result actions for regression status and comparisons. + *******************************************************************************/ +public final class RegressionSteps +{ + public static final String NOTE = "resolutionNote"; + + + + /******************************************************************************* + ** Steps only. + *******************************************************************************/ + private RegressionSteps() + { + } + + + + /******************************************************************************* + ** Review open records; closed records are explicitly skipped. + *******************************************************************************/ + public static class Transform extends AbstractTransformStep implements ProcessSummaryProviderInterface + { + private ProcessSummaryLine change; + private final ProcessSummaryLine skip = Lines.line(Status.INFO, "closed regression will be skipped", "closed regressions will be skipped", + "closed regression was skipped", "closed regressions were skipped"); + + + + /******************************************************************************* + ** Validate the note and classify this page without writing anything. + *******************************************************************************/ + @Override + public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + String status = input.getValueString(WebPerfProcesses.TRIGGER_FIELD); + String note = input.getValueString(NOTE); + if(note == null || note.isBlank() || note.length() > 2000) + { + throw (new QException("Enter a resolution note of 1 to 2000 characters")); + } + if(change == null) + { + String verb = "DISMISSED".equals(status) ? "dismissed" : "marked resolved"; + change = Lines.line(Status.OK, "regression will be " + verb, "regressions will be " + verb, + "regression was " + verb, "regressions were " + verb); + } + for(QRecord record : input.getRecords()) + { + if(List.of("SUSPECTED", "CONFIRMED").contains(record.getValueString("status"))) + { + change.incrementCountAndAddPrimaryKey(record.getValue("id")); + output.addRecord(record); + } + else + { + skip.incrementCountAndAddPrimaryKey(record.getValue("id")); + } + } + } + + + + /******************************************************************************* + ** Review summary. + *******************************************************************************/ + @Override + public ArrayList getProcessSummary(RunBackendStepOutput output, boolean result) + { + ArrayList lines = new ArrayList<>(); + if(change != null) + { + Lines.addAll(lines, change); + } + Lines.addAll(lines, skip); + return (lines); + } + } + + + + /******************************************************************************* + ** Apply the requested state under a database lock, rechecking eligibility. + *******************************************************************************/ + public static class Load extends AbstractLoadStep implements ProcessSummaryProviderInterface + { + private ProcessSummaryLine changed; + private final ProcessSummaryLine skipped = Lines.line(Status.INFO, "regression will be skipped", "regressions will be skipped", + "regression was already closed", "regressions were already closed"); + + + + /******************************************************************************* + ** Apply one page and report only the rows actually changed. + *******************************************************************************/ + @Override + public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + String status = input.getValueString(WebPerfProcesses.TRIGGER_FIELD); + if(changed == null) + { + String verb = "DISMISSED".equals(status) ? "dismissed" : "marked resolved"; + changed = Lines.line(Status.OK, "regression will be " + verb, "regressions will be " + verb, + "regression was " + verb, "regressions were " + verb); + } + WebPerfEvaluationService service = new WebPerfEvaluationService(WebPerfRuntime.forInput(input).store()); + try + { + for(QRecord record : input.getRecords()) + { + boolean updated = service.finishRegression(record.getValueLong("id"), status, input.getValueString(NOTE), Instant.now()); + (updated ? changed : skipped).incrementCountAndAddPrimaryKey(record.getValue("id")); + } + } + catch(SQLException | IllegalArgumentException e) + { + throw (new QException("Could not update the regressions", e)); + } + } + + + + /******************************************************************************* + ** Result summary. + *******************************************************************************/ + @Override + public ArrayList getProcessSummary(RunBackendStepOutput output, boolean result) + { + ArrayList lines = new ArrayList<>(); + if(changed != null) + { + Lines.addAll(lines, changed); + } + Lines.addAll(lines, skipped); + if(getTransformStep() instanceof Transform transform) + { + Lines.addAll(lines, transform.skip); + } + return (lines); + } + } + + + + /******************************************************************************* + ** Review comparisons using recorded lab history. + *******************************************************************************/ + public static class CompareTransform extends AbstractTransformStep implements ProcessSummaryProviderInterface + { + private final ProcessSummaryLine changes = Lines.line(Status.OK, "site change will be compared using recorded lab samples", + "site changes will be compared using recorded lab samples", "site change was compared", "site changes were compared"); + + + + /******************************************************************************* + ** Pass through a bounded page of selected changes. + *******************************************************************************/ + @Override + public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + for(QRecord record : input.getRecords()) + { + changes.incrementCountAndAddPrimaryKey(record.getValue("id")); + output.addRecord(record); + } + } + + + + /******************************************************************************* + ** Review summary. + *******************************************************************************/ + @Override + public ArrayList getProcessSummary(RunBackendStepOutput output, boolean result) + { + ArrayList lines = new ArrayList<>(); + Lines.addAll(lines, changes); + return (lines); + } + } + + + + /******************************************************************************* + ** Write comparisons when both sides have three comparable measurements. + *******************************************************************************/ + public static class CompareLoad extends AbstractLoadStep implements ProcessSummaryProviderInterface + { + private final ProcessSummaryLine compared = Lines.line(Status.OK, "comparison will be recorded", "comparisons will be recorded", + "comparison was recorded", "comparisons were recorded"); + private final ProcessSummaryLine insufficient = Lines.line(Status.INFO, "change needs more samples", "changes need more samples", + "change had fewer than three comparable samples on each side", "changes had fewer than three comparable samples on each side"); + + + + /******************************************************************************* + ** Compute and persist this page of selected changes. + *******************************************************************************/ + @Override + public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + try + { + WebPerfComparisonService service = new WebPerfComparisonService(WebPerfRuntime.forInput(input).store()); + for(QRecord record : input.getRecords()) + { + int count = service.compare(record.getValueLong("id")); + compared.incrementCount(count); + if(count == 0) + { + insufficient.incrementCountAndAddPrimaryKey(record.getValue("id")); + } + } + } + catch(SQLException e) + { + throw (new QException("Could not compare the site changes", e)); + } + } + + + + /******************************************************************************* + ** Result summary, including an honest insufficient-history line. + *******************************************************************************/ + @Override + public ArrayList getProcessSummary(RunBackendStepOutput output, boolean result) + { + ArrayList lines = new ArrayList<>(); + Lines.addAll(lines, compared, insufficient); + return (lines); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/webperf/processes/RetryRunsSteps.java b/src/main/java/com/kingsrook/qbits/webperf/processes/RetryRunsSteps.java index 7f51bbc..ad4e386 100644 --- a/src/main/java/com/kingsrook/qbits/webperf/processes/RetryRunsSteps.java +++ b/src/main/java/com/kingsrook/qbits/webperf/processes/RetryRunsSteps.java @@ -144,8 +144,23 @@ public final class RetryRunsSteps String batch = java.util.UUID.randomUUID().toString(); for(QRecord record : input.getRecords()) { - requests.add(new WebPerfService.RunRequest(record.getValueLong("targetId"), record.getValueString("url"), record.getValueString("strategy"), - "MANUAL", "retry of run " + record.getValue("id"), batch, Instant.now())); + if(List.of("CONFIRM", "CHANGE_COMPARE").contains(record.getValueString("triggerType"))) + { + try + { + String status = service.retrySample(record.getValueLong("id"), Instant.now()); + ("QUEUED".equals(status) ? queued : "MERGED".equals(status) ? merged : capped).incrementCount(1); + } + catch(SQLException e) + { + throw (new QException("Could not retry the measurement sample", e)); + } + } + else + { + requests.add(new WebPerfService.RunRequest(record.getValueLong("targetId"), record.getValueString("url"), record.getValueString("strategy"), + "MANUAL", "retry of run " + record.getValue("id"), batch, Instant.now())); + } } try { diff --git a/src/main/java/com/kingsrook/qbits/webperf/processes/ScheduledSteps.java b/src/main/java/com/kingsrook/qbits/webperf/processes/ScheduledSteps.java index d31072f..9109c08 100644 --- a/src/main/java/com/kingsrook/qbits/webperf/processes/ScheduledSteps.java +++ b/src/main/java/com/kingsrook/qbits/webperf/processes/ScheduledSteps.java @@ -25,6 +25,7 @@ import java.time.Instant; import java.util.List; import java.util.Map; import com.kingsrook.qbits.webperf.WebPerfRuntime; +import com.kingsrook.qbits.webperf.service.WebPerfChangeService; import com.kingsrook.qbits.webperf.service.WebPerfService; import com.kingsrook.qqq.backend.core.actions.processes.BackendStep; import com.kingsrook.qqq.backend.core.exceptions.QException; @@ -54,6 +55,30 @@ public final class ScheduledSteps + /******************************************************************************* + ** Poll deploy probes and external change sources on the existing scheduler. + *******************************************************************************/ + public static class ProbeChanges implements BackendStep + { + /******************************************************************************* + ** Store changes and queue the affected pages. + *******************************************************************************/ + @Override + public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + try + { + new WebPerfChangeService(WebPerfRuntime.forInput(input).store()).poll(Instant.now()); + } + catch(Exception e) + { + throw (new QException("Checking site changes failed", e)); + } + } + } + + + /******************************************************************************* ** Every minute: run due queued tests. *******************************************************************************/ diff --git a/src/main/java/com/kingsrook/qbits/webperf/processes/WebPerfProcesses.java b/src/main/java/com/kingsrook/qbits/webperf/processes/WebPerfProcesses.java index 323b312..c6df699 100644 --- a/src/main/java/com/kingsrook/qbits/webperf/processes/WebPerfProcesses.java +++ b/src/main/java/com/kingsrook/qbits/webperf/processes/WebPerfProcesses.java @@ -37,6 +37,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules; import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData; import com.kingsrook.qqq.backend.core.model.metadata.processes.QFunctionInputMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData; import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData; import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.QScheduleMetaData; import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractExtractStep; @@ -63,6 +64,10 @@ import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwith *******************************************************************************/ public final class WebPerfProcesses { + public static final String DISMISS_REGRESSIONS = "DismissRegressions"; + public static final String RESOLVE_REGRESSIONS = "ResolveRegressions"; + public static final String COMPARE_CHANGE = "CompareChange"; + public static final String PROBE_DEPLOYS = "ProbeDeploys"; public static final String TEST_PAGE_SPEED = "TestPageSpeed"; public static final String RUN_SITE_AUDIT = "RunSiteAudit"; public static final String SYNC_TARGETS = "SyncPages"; @@ -146,6 +151,29 @@ public final class WebPerfProcesses processes.add(scheduled(config, SCHEDULED_FIELD, "Refresh Google field data (scheduled)", ScheduledSteps.FieldData.class, new QScheduleMetaData().withCronExpression(FIELD_CRON).withCronTimeZoneId("UTC"))); } + String regressions = config.applyPrefix(WebPerfSchema.REGRESSION); + for(String status : List.of("DISMISSED", "RESOLVED")) + { + boolean dismiss = "DISMISSED".equals(status); + QProcessMetaData process = etl(config, dismiss ? DISMISS_REGRESSIONS : RESOLVE_REGRESSIONS, dismiss ? "Dismiss" : "Mark resolved", "done", + regressions, regressions, ExtractViaQueryStep.class, RegressionSteps.Transform.class, RegressionSteps.Load.class, status, "manage", 1, + "Open regressions will be updated. Closed regressions will be skipped."); + process.withStep(0, new QFrontendStepMetaData().withName("note").withLabel("Resolution note") + .withFormField(new QFieldMetaData(RegressionSteps.NOTE, QFieldType.STRING).withLabel("Resolution note").withIsRequired(true).withMaxLength(2000))); + processes.add(process); + } + if(config.isOn(WebPerfFeature.CDN_COMPARE)) + { + String changes = config.applyPrefix(WebPerfSchema.CHANGE_EVENT); + processes.add(etl(config, COMPARE_CHANGE, "Compare before and after", "compare", changes, config.applyPrefix(WebPerfSchema.COMPARISON), + ExtractViaQueryStep.class, RegressionSteps.CompareTransform.class, RegressionSteps.CompareLoad.class, null, "view", 1, + "Lab comparisons require three successful samples on each side, on the same Lighthouse major version.")); + } + if(config.isOn(WebPerfFeature.DEPLOY_PROBES) || !config.getChangeEventSources().isEmpty()) + { + processes.add(scheduled(config, PROBE_DEPLOYS, "Check site changes", ScheduledSteps.ProbeChanges.class, + new QScheduleMetaData().withRepeatSeconds(300).withInitialDelaySeconds(1))); + } return (processes); } diff --git a/src/main/java/com/kingsrook/qbits/webperf/service/WebPerfChangeService.java b/src/main/java/com/kingsrook/qbits/webperf/service/WebPerfChangeService.java new file mode 100644 index 0000000..94f0a45 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/webperf/service/WebPerfChangeService.java @@ -0,0 +1,246 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.webperf.service; + + +import java.net.URI; +import java.nio.charset.StandardCharsets; +import java.sql.SQLException; +import java.time.Instant; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import com.kingsrook.qbits.webperf.WebPerfFeature; +import com.kingsrook.qbits.webperf.WebPerfQBitConfig; +import com.kingsrook.qbits.webperf.WebPerfRuntime; +import com.kingsrook.qbits.webperf.WebPerfSchema; +import com.kingsrook.qbits.webperf.spi.ChangeEventSource; +import com.kingsrook.qbits.webperf.spi.DeployProbe; +import com.kingsrook.qbits.webperf.store.WebPerfStore; +import org.json.JSONArray; + + +/******************************************************************************* + ** Records source changes and attributes runs to the state observed at start. + ** All identities and queue writes are durable; polling can safely repeat. + *******************************************************************************/ +public class WebPerfChangeService +{ + private final WebPerfStore store; + private final WebPerfQBitConfig config; + + + + /******************************************************************************* + ** Changes for one installation. + *******************************************************************************/ + public WebPerfChangeService(WebPerfStore store) + { + this.store = store; + this.config = store.getConfig(); + } + + + + /******************************************************************************* + ** Poll configured sources. Failed probes propagate so the schedule retries. + *******************************************************************************/ + public void poll(Instant now) throws Exception + { + if(config.isOn(WebPerfFeature.DEPLOY_PROBES)) + { + for(DeployProbe probe : config.getDeployProbes()) + { + String build = probe.currentBuildId(); + if(build != null && !build.isBlank()) + { + String kind = "BACKEND".equalsIgnoreCase(probe.component()) ? "BACKEND_DEPLOY" : "FRONTEND_DEPLOY"; + Map previous = latest(kind, probe.component(), now); + record(new ChangeEventSource.Change(kind, probe.component(), previous == null ? null : (String) previous.get("to_value"), build, now), + "PROBE", probe.component() + ":" + build, true); + } + } + } + for(int i = 0; i < config.getChangeEventSources().size(); i++) + { + String sourceKey = "source:" + i; + List> last = store.query("SELECT MAX(observed_at) AS observed_at FROM {change_event} WHERE site_key = ? AND note = ?", + siteKey(), sourceKey); + Instant since = (Instant) last.get(0).get("observed_at"); + for(ChangeEventSource.Change change : config.getChangeEventSources().get(i).changesSince(since == null ? Instant.EPOCH : since)) + { + if(!"CDN_STAGE".equals(change.kind()) || config.isOn(WebPerfFeature.CDN_COMPARE)) + { + record(change, "CDN_STAGE".equals(change.kind()) ? "QBIT_CDN" : "WEBHOOK", sourceKey, true); + } + } + } + } + + + + /******************************************************************************* + ** Insert once per build or source event and queue affected pages atomically. + ** The source SPI has no event id, so its complete tuple is its identity. + *******************************************************************************/ + public long record(ChangeEventSource.Change change, String source, String sourceKey, boolean queue) throws SQLException + { + String identity = new JSONArray(List.of(siteKey(), source, sourceKey, change.kind(), change.component(), change.toValue(), + "PROBE".equals(source) ? "" : change.observedAt().toString(), "PROBE".equals(source) ? "" : String.valueOf(change.fromValue()))).toString(); + String key = WebPerfService.sha256(identity.getBytes(StandardCharsets.UTF_8)); + try + { + return (store.transaction(tx -> + { + List> existing = tx.query("SELECT id FROM {change_event} WHERE event_key = ?", key); + if(!existing.isEmpty()) + { + if("PROBE".equals(source)) + { + tx.update("UPDATE {change_event} SET last_seen_at = ? WHERE id = ? AND (last_seen_at IS NULL OR last_seen_at < ?)", + change.observedAt(), existing.get(0).get("id"), change.observedAt()); + } + return (WebPerfStore.asLong(existing.get(0).get("id"))); + } + Map values = new LinkedHashMap<>(); + values.put("kind", change.kind()); + values.put("component", change.component()); + values.put("from_value", change.fromValue()); + values.put("to_value", change.toValue()); + values.put("observed_at", change.observedAt()); + values.put("last_seen_at", change.observedAt()); + values.put("source", source); + values.put("note", sourceKey); + values.put("event_key", key); + values.put("site_key", siteKey()); + long id = tx.insert(WebPerfSchema.CHANGE_EVENT, values); + if(queue && config.isOn(WebPerfFeature.LAB)) + { + queueChange(tx, id, change); + } + return (id); + })); + } + catch(SQLException e) + { + ///////////////////////////////////////////////////////////////////// + // A competing poll may have committed this exact identity first. // + ///////////////////////////////////////////////////////////////////// + if("23505".equals(e.getSQLState())) + { + List> existing = store.query("SELECT id FROM {change_event} WHERE event_key = ?", key); + if(!existing.isEmpty()) + { + return (WebPerfStore.asLong(existing.get(0).get("id"))); + } + } + throw (e); + } + } + + + + /******************************************************************************* + ** Page through targets; CDN sampling bypasses the ordinary queue merge. + *******************************************************************************/ + private void queueChange(WebPerfStore tx, long eventId, ChangeEventSource.Change change) throws SQLException + { + WebPerfService service = new WebPerfService(new WebPerfRuntime(config), tx); + boolean cdn = "CDN_STAGE".equals(change.kind()); + boolean deploy = change.kind().endsWith("DEPLOY"); + long after = 0; + while(true) + { + List> pages = tx.query("SELECT id, url FROM {target} WHERE site_key = ? AND enabled = TRUE AND id > ?" + + (cdn || deploy ? " AND tier = 'KEY'" : "") + " ORDER BY id FETCH FIRST 100 ROWS ONLY", siteKey(), after); + if(pages.isEmpty()) + { + return; + } + for(Map page : pages) + { + after = WebPerfStore.asLong(page.get("id")); + for(String strategy : config.getStrategies()) + { + if(cdn && ("HEADERS_ONLY".equals(change.toValue()) || "LIVE".equals(change.toValue()))) + { + for(int sample = 0; sample < 3; sample++) + { + service.enqueueSample(page, strategy, "CHANGE_COMPARE", "change:" + eventId + ":1:" + sample, + change.observedAt().plusSeconds("LIVE".equals(change.toValue()) ? 3600 : 0)); + if("LIVE".equals(change.toValue())) + { + service.enqueueSample(page, strategy, "CHANGE_COMPARE", "change:" + eventId + ":24:" + sample, change.observedAt().plusSeconds(86400)); + } + } + } + else + { + Instant due = change.observedAt().plusSeconds(deploy ? config.getDeploySettleSeconds() : config.getPublishSettleSeconds()); + service.enqueue(List.of(new WebPerfService.RunRequest(after, (String) page.get("url"), strategy, + deploy ? "DEPLOY" : "PUBLISH", "change:" + eventId, null, due)), change.observedAt()); + } + } + } + } + } + + + + /******************************************************************************* + ** Latest observed state at a time, optionally for a specific component. + *******************************************************************************/ + public Map latest(String kind, String component, Instant at) throws SQLException + { + List> rows = component == null + ? store.query("SELECT * FROM {change_event} WHERE site_key = ? AND kind = ? AND observed_at <= ? ORDER BY last_seen_at DESC, id DESC FETCH FIRST 1 ROWS ONLY", + siteKey(), kind, at) + : store.query("SELECT * FROM {change_event} WHERE site_key = ? AND kind = ? AND component = ? AND observed_at <= ? ORDER BY last_seen_at DESC, id DESC FETCH FIRST 1 ROWS ONLY", + siteKey(), kind, component, at); + return (rows.isEmpty() ? null : rows.get(0)); + } + + + + /******************************************************************************* + ** Attribute at execution, not queue time (a deployment may happen meanwhile). + *******************************************************************************/ + public void attribute(long runId, Instant now) throws SQLException + { + Map build = latest("FRONTEND_DEPLOY", null, now); + Map cdn = latest("CDN_STAGE", URI.create(config.getSiteBaseUrl()).getHost(), now); + List> cause = store.query("SELECT id FROM {change_event} WHERE site_key = ? AND observed_at <= ? ORDER BY observed_at DESC, id DESC FETCH FIRST 1 ROWS ONLY", + siteKey(), now); + store.update("UPDATE {lab_run} SET deployment_id = ?, cdn_stage = ?, change_event_id = ? WHERE id = ?", + build == null ? null : build.get("to_value"), cdn == null ? null : cdn.get("to_value"), cause.isEmpty() ? null : cause.get(0).get("id"), runId); + } + + + + /******************************************************************************* + ** Same site identity as the tested pages. + *******************************************************************************/ + String siteKey() + { + return ("host:" + URI.create(config.getSiteBaseUrl()).getHost().toLowerCase(Locale.ROOT)); + } +} diff --git a/src/main/java/com/kingsrook/qbits/webperf/service/WebPerfComparisonService.java b/src/main/java/com/kingsrook/qbits/webperf/service/WebPerfComparisonService.java new file mode 100644 index 0000000..6fd6355 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/webperf/service/WebPerfComparisonService.java @@ -0,0 +1,188 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.webperf.service; + + +import java.sql.SQLException; +import java.time.Duration; +import java.time.Instant; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.webperf.WebPerfSchema; +import com.kingsrook.qbits.webperf.store.WebPerfStore; + + +/******************************************************************************* + ** Reproducible before/after lab medians, with counts and actual sample windows. + ** Three successful samples on each side and one Lighthouse major are required. + *******************************************************************************/ +public class WebPerfComparisonService +{ + private final WebPerfStore store; + + + + /******************************************************************************* + ** Comparisons for one installation. + *******************************************************************************/ + public WebPerfComparisonService(WebPerfStore store) + { + this.store = store; + } + + + + /******************************************************************************* + ** Recompute the run's attributed change after its results have been stored. + *******************************************************************************/ + public void compareForRun(Map run) throws SQLException + { + if(run.get("change_event_id") != null) + { + List> events = store.query("SELECT * FROM {change_event} WHERE id = ?", run.get("change_event_id")); + if(!events.isEmpty()) + { + compareTarget(events.get(0), WebPerfStore.asLong(run.get("target_id")), (String) run.get("strategy")); + } + } + } + + + + /******************************************************************************* + ** Record action: page through enabled targets and compare existing evidence. + ** Sparse history stays absent instead of being filled with estimated values. + *******************************************************************************/ + public int compare(long eventId) throws SQLException + { + return (store.transaction(tx -> + { + Map event = tx.query("SELECT * FROM {change_event} WHERE id = ? FOR UPDATE", eventId).get(0); + WebPerfComparisonService service = new WebPerfComparisonService(tx); + int count = 0; + long after = 0; + while(true) + { + List> pages = tx.query("SELECT id FROM {target} WHERE enabled = TRUE AND site_key = ? AND id > ? ORDER BY id FETCH FIRST 100 ROWS ONLY", + event.get("site_key"), after); + if(pages.isEmpty()) + { + return (count); + } + for(Map page : pages) + { + after = WebPerfStore.asLong(page.get("id")); + for(String strategy : store.getConfig().getStrategies()) + { + count += service.compareTarget(event, after, strategy); + } + } + } + })); + } + + + + /******************************************************************************* + ** Serialize comparison upserts on the change row, shared by all callers. + *******************************************************************************/ + private int compareTarget(Map event, long targetId, String strategy) throws SQLException + { + store.query("SELECT id FROM {change_event} WHERE id = ? FOR UPDATE", event.get("id")); + Instant at = (Instant) event.get("observed_at"); + List> after = store.query("SELECT * FROM {lab_run} WHERE target_id = ? AND strategy = ? AND status = 'SUCCEEDED'" + + " AND started_at >= ? AND change_event_id = ? ORDER BY completed_at DESC, id DESC FETCH FIRST 3 ROWS ONLY", targetId, strategy, at, event.get("id")); + if(after.size() != 3) + { + return (0); + } + String major = WebPerfEvaluationService.major(after.get(0)); + if(after.stream().anyMatch(row -> !major.equals(WebPerfEvaluationService.major(row)))) + { + return (0); + } + List> before = store.query("SELECT * FROM {lab_run} WHERE target_id = ? AND strategy = ? AND status = 'SUCCEEDED'" + + " AND completed_at < ? AND completed_at >= ? AND (lighthouse_version = ? OR lighthouse_version LIKE ?)" + + " ORDER BY completed_at DESC, id DESC FETCH FIRST 3 ROWS ONLY", targetId, strategy, at, at.minus(Duration.ofDays(30)), major, major + ".%"); + if(before.size() != 3) + { + return (0); + } + int count = 0; + for(String metric : List.of("PERFORMANCE_SCORE", "LCP", "FCP", "TTFB")) + { + String column = WebPerfEvaluationService.LAB_METRICS.get(metric); + if(before.stream().anyMatch(row -> row.get(column) == null) || after.stream().anyMatch(row -> row.get(column) == null)) + { + continue; + } + double beforeValue = WebPerfEvaluationService.median(before, column); + double afterValue = WebPerfEvaluationService.median(after, column); + double delta = afterValue - beforeValue; + Double percentage = beforeValue == 0 ? null : delta * 100 / beforeValue; + if(percentage != null && Math.abs(percentage) >= 1000000) + { + percentage = null; + } + List> existing = store.query("SELECT id FROM {comparison} WHERE change_event_id = ? AND target_id = ? AND source = 'LAB'" + + " AND strategy = ? AND metric = ? AND segment = 'ALL'", event.get("id"), targetId, strategy, metric); + if(existing.isEmpty()) + { + Map row = new LinkedHashMap<>(); + row.put("change_event_id", event.get("id")); + row.put("target_id", targetId); + row.put("scope", "TARGET"); + row.put("source", "LAB"); + row.put("strategy", strategy); + row.put("metric", metric); + row.put("segment", "ALL"); + row.put("before_value", beforeValue); + row.put("before_n", 3); + row.put("before_window", window(before)); + row.put("after_value", afterValue); + row.put("after_n", 3); + row.put("after_window", window(after)); + row.put("delta", delta); + row.put("delta_pct", percentage); + store.insert(WebPerfSchema.COMPARISON, row); + } + else + { + store.update("UPDATE {comparison} SET before_value = ?, before_n = 3, before_window = ?, after_value = ?, after_n = 3, after_window = ?," + + " delta = ?, delta_pct = ?, modify_date = ? WHERE id = ?", beforeValue, window(before), afterValue, window(after), delta, percentage, + Instant.now(), existing.get(0).get("id")); + } + count++; + } + return (count); + } + + + + /******************************************************************************* + ** The actual sample times, not an implied continuous measurement window. + *******************************************************************************/ + private static String window(List> samples) + { + return (samples.get(samples.size() - 1).get("completed_at") + " / " + samples.get(0).get("completed_at")); + } +} diff --git a/src/main/java/com/kingsrook/qbits/webperf/service/WebPerfEvaluationService.java b/src/main/java/com/kingsrook/qbits/webperf/service/WebPerfEvaluationService.java new file mode 100644 index 0000000..f49bf73 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/webperf/service/WebPerfEvaluationService.java @@ -0,0 +1,611 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.webperf.service; + + +import java.sql.SQLException; +import java.time.Duration; +import java.time.Instant; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Objects; +import com.kingsrook.qbits.webperf.WebPerfFeature; +import com.kingsrook.qbits.webperf.WebPerfRuntime; +import com.kingsrook.qbits.webperf.WebPerfSchema; +import com.kingsrook.qbits.webperf.spi.ChangeEventSource; +import com.kingsrook.qbits.webperf.store.WebPerfStore; +import com.kingsrook.qqq.backend.core.alerts.AlertRecordRef; +import com.kingsrook.qqq.backend.core.alerts.AlertSeverity; +import com.kingsrook.qqq.backend.core.alerts.AlertSignal; +import com.kingsrook.qqq.backend.core.alerts.QAlerts; + + +/******************************************************************************* + ** Budget breaches and confirmed lab regressions, backed by ordinary records. + ** Target row locks serialize evaluation across workers. Baselines and samples + ** are bounded, and evaluation progress commits with the resulting records. + *******************************************************************************/ +public class WebPerfEvaluationService +{ + public static final String ALERT_TYPE = "webperf.regression"; + public static final Map LAB_METRICS = Map.of( + "PERFORMANCE_SCORE", "score_performance", "ACCESSIBILITY_SCORE", "score_accessibility", + "BEST_PRACTICES_SCORE", "score_best_practices", "SEO_SCORE", "score_seo", + "LCP", "lcp_ms", "FCP", "fcp_ms", "CLS", "cls", "TBT", "tbt_ms", "TTFB", "ttfb_ms"); + private final WebPerfStore store; + + + + /******************************************************************************* + ** Evaluation for one installation. + *******************************************************************************/ + public WebPerfEvaluationService(WebPerfStore store) + { + this.store = store; + } + + + + /******************************************************************************* + ** Bounded work per worker tick, including results left by an interrupted tick. + *******************************************************************************/ + public void evaluatePending() throws SQLException + { + for(Map run : store.query("SELECT id FROM {lab_run} WHERE status = 'SUCCEEDED' AND evaluated_at IS NULL ORDER BY completed_at, id FETCH FIRST 100 ROWS ONLY")) + { + evaluateLab(WebPerfStore.asLong(run.get("id"))); + } + for(Map field : store.query("SELECT id FROM {field_snapshot} WHERE evaluated_at IS NULL ORDER BY id FETCH FIRST 100 ROWS ONLY")) + { + evaluateField(WebPerfStore.asLong(field.get("id"))); + } + } + + + + /******************************************************************************* + ** Evaluate a stored successful result once; publish alerts after commit. + *******************************************************************************/ + public void evaluateLab(long runId) throws SQLException + { + List changed = store.transaction(tx -> + { + List> runs = tx.query("SELECT * FROM {lab_run} WHERE id = ? AND status = 'SUCCEEDED'", runId); + if(runs.isEmpty()) + { + return (List.of()); + } + Map run = runs.get(0); + List> targets = tx.query("SELECT * FROM {target} WHERE id = ? FOR UPDATE", run.get("target_id")); + if(targets.isEmpty() || !tx.query("SELECT id FROM {lab_run} WHERE id = ? AND evaluated_at IS NOT NULL", runId).isEmpty()) + { + return (List.of()); + } + Map target = targets.get(0); + WebPerfEvaluationService evaluator = new WebPerfEvaluationService(tx); + List transitions = new ArrayList<>(); + evaluator.versionChange(run, transitions); + List> baseline = evaluator.baseline(run); + for(Map.Entry metric : LAB_METRICS.entrySet()) + { + if(run.get(metric.getValue()) != null) + { + evaluator.evaluateMetric(target, run, metric.getKey(), metric.getValue(), baseline, transitions); + } + } + evaluator.updateCount(target.get("id")); + if(store.getConfig().isOn(WebPerfFeature.CDN_COMPARE)) + { + new WebPerfComparisonService(tx).compareForRun(run); + } + tx.update("UPDATE {lab_run} SET evaluated_at = ? WHERE id = ?", Instant.now(), runId); + return (transitions); + }); + changed.forEach(this::sendAlert); + } + + + + /******************************************************************************* + ** CrUX URL budgets use real p75 values, never NO_DATA, errors or origin + ** fallback as page measurements. A new collection period can resolve a breach. + *******************************************************************************/ + public void evaluateField(long snapshotId) throws SQLException + { + List changed = store.transaction(tx -> + { + Map field = tx.query("SELECT * FROM {field_snapshot} WHERE id = ? FOR UPDATE", snapshotId).get(0); + List transitions = new ArrayList<>(); + if(field.get("evaluated_at") != null) + { + return (transitions); + } + if("OK".equals(field.get("status")) && "URL".equals(field.get("scope"))) + { + List> targets = tx.query("SELECT * FROM {target} WHERE site_key = ? AND url = ? FOR UPDATE", + new WebPerfChangeService(tx).siteKey(), field.get("scope_key")); + WebPerfEvaluationService evaluator = new WebPerfEvaluationService(tx); + for(Map target : targets) + { + for(String metric : List.of("LCP", "INP", "CLS", "FCP", "TTFB")) + { + String column = "p75_" + metric.toLowerCase(Locale.ROOT) + ("CLS".equals(metric) ? "" : "_ms"); + Double value = WebPerfStore.asDouble(field.get(column)); + if(value == null) + { + continue; + } + String strategy = "PHONE".equals(field.get("form_factor")) ? "MOBILE" : (String) field.get("form_factor"); + Map budget = evaluator.breachedBudget(target, "CRUX", strategy, metric, value); + List> open = tx.query("SELECT * FROM {regression} WHERE target_id = ? AND source = 'CRUX' AND strategy = ? AND metric = ?" + + " AND status = 'CONFIRMED' FETCH FIRST 1 ROWS ONLY", target.get("id"), field.get("form_factor"), metric); + Instant at = (Instant) field.get("last_checked_at"); + if(budget != null && open.isEmpty()) + { + Map row = new LinkedHashMap<>(); + row.put("target_id", target.get("id")); + row.put("content_table", target.get("content_table")); + row.put("content_id", target.get("content_id")); + row.put("source", "CRUX"); + row.put("strategy", field.get("form_factor")); + row.put("metric", metric); + row.put("current_value", value); + row.put("current_ref", "field:" + snapshotId); + row.put("budget_id", budget.get("id")); + row.put("severity", budget.get("severity")); + row.put("status", "CONFIRMED"); + row.put("detected_at", at); + row.put("confirmed_at", at); + transitions.add(tx.insert(WebPerfSchema.REGRESSION, row)); + } + else if(!open.isEmpty()) + { + Map regression = open.get(0); + tx.update("UPDATE {regression} SET current_value = ?, current_ref = ?, modify_date = ? WHERE id = ?", + value, "field:" + snapshotId, at, regression.get("id")); + if(budget == null) + { + evaluator.close(regression, "RESOLVED", "Google field data is within its budget", at); + transitions.add(WebPerfStore.asLong(regression.get("id"))); + } + } + } + evaluator.updateCount(target.get("id")); + } + } + tx.update("UPDATE {field_snapshot} SET evaluated_at = ? WHERE id = ?", Instant.now(), snapshotId); + return (transitions); + }); + changed.forEach(this::sendAlert); + } + + + + /******************************************************************************* + ** Five eligible results in the preceding 30 days, excluding the new result. + *******************************************************************************/ + private List> baseline(Map run) throws SQLException + { + return (store.query("SELECT * FROM {lab_run} WHERE target_id = ? AND strategy = ? AND status = 'SUCCEEDED'" + + " AND trigger_type IN ('SCHEDULE', 'CONFIRM', 'SITE_AUDIT') AND completed_at >= ?" + + " AND (completed_at < ? OR (completed_at = ? AND id < ?)) AND (lighthouse_version = ? OR lighthouse_version LIKE ?)" + + " ORDER BY completed_at DESC, id DESC FETCH FIRST 5 ROWS ONLY", run.get("target_id"), run.get("strategy"), + ((Instant) run.get("completed_at")).minus(Duration.ofDays(30)), run.get("completed_at"), run.get("completed_at"), run.get("id"), + major(run), major(run) + ".%")); + } + + + + /******************************************************************************* + ** Major-version changes are markers and retire incompatible open baselines. + *******************************************************************************/ + private void versionChange(Map run, List changed) throws SQLException + { + List> previous = store.query("SELECT lighthouse_version FROM {lab_run} WHERE target_id = ? AND strategy = ?" + + " AND status = 'SUCCEEDED' AND id < ? ORDER BY id DESC FETCH FIRST 1 ROWS ONLY", run.get("target_id"), run.get("strategy"), run.get("id")); + if(!previous.isEmpty() && !major(previous.get(0)).equals(major(run))) + { + new WebPerfChangeService(store).record(new ChangeEventSource.Change("LIGHTHOUSE_VERSION", "LIGHTHOUSE", major(previous.get(0)), major(run), + (Instant) run.get("completed_at")), "PROBE", "lighthouse:" + major(run), false); + } + for(Map open : store.query("SELECT * FROM {regression} WHERE target_id = ? AND strategy = ? AND source = 'LAB'" + + " AND status IN ('SUSPECTED', 'CONFIRMED') AND lighthouse_major <> ?", run.get("target_id"), run.get("strategy"), major(run))) + { + close(open, "CONFIRMED".equals(open.get("status")) ? "RESOLVED" : "NOT_CONFIRMED", "Lighthouse major version changed; baseline reset", + (Instant) run.get("completed_at")); + changed.add(WebPerfStore.asLong(open.get("id"))); + } + } + + + + /******************************************************************************* + ** Keep a single open row per page, source, strategy and metric. Its baseline + ** stays fixed through confirmation and recovery rather than drifting. + *******************************************************************************/ + private void evaluateMetric(Map target, Map run, String metric, String column, + List> baseline, List changed) throws SQLException + { + List> existing = store.query("SELECT * FROM {regression} WHERE target_id = ? AND strategy = ? AND source = 'LAB' AND metric = ?" + + " AND status IN ('SUSPECTED', 'CONFIRMED') FETCH FIRST 1 ROWS ONLY", target.get("id"), run.get("strategy"), metric); + if(!existing.isEmpty()) + { + advance(target, run, metric, column, existing.get(0), changed); + return; + } + Double value = WebPerfStore.asDouble(run.get(column)); + Double reference = baseline.size() == 5 && baseline.stream().allMatch(row -> row.get(column) != null) ? median(baseline, column) : null; + Map budget = breachedBudget(target, "LAB", (String) run.get("strategy"), metric, value); + List failedAudits = newFailures(run, metric); + if(!drift(metric, reference, value) && budget == null && failedAudits.isEmpty()) + { + return; + } + boolean deterministic = metric.endsWith("_SCORE") && !"PERFORMANCE_SCORE".equals(metric); + Instant at = (Instant) run.get("completed_at"); + Map row = new LinkedHashMap<>(); + row.put("target_id", target.get("id")); + row.put("content_table", target.get("content_table")); + row.put("content_id", target.get("content_id")); + row.put("source", "LAB"); + row.put("strategy", run.get("strategy")); + row.put("metric", metric); + row.put("baseline_value", reference); + row.put("baseline_ref", reference == null ? null : refs(baseline)); + row.put("current_value", value); + row.put("current_ref", run.get("id").toString()); + row.put("delta", reference == null ? null : value - reference); + row.put("budget_id", budget == null ? null : budget.get("id")); + row.put("severity", budget == null ? "WARNING" : budget.get("severity")); + row.put("status", deterministic ? "CONFIRMED" : "SUSPECTED"); + row.put("detected_at", at); + row.put("confirmed_at", deterministic ? at : null); + row.put("cause_change_event_id", run.get("change_event_id")); + row.put("first_run_id", run.get("id")); + row.put("checkpoint_run_id", run.get("id")); + row.put("lighthouse_major", major(run)); + row.put("failed_audits", String.join(",", failedAudits)); + long id = store.insert(WebPerfSchema.REGRESSION, row); + if(deterministic) + { + changed.add(id); + } + else + { + WebPerfService service = new WebPerfService(new WebPerfRuntime(store.getConfig()), store); + for(int sample = 0; sample < 2; sample++) + { + service.enqueueSample(target, (String) run.get("strategy"), "CONFIRM", "confirm:" + run.get("id") + ":" + sample, at); + } + } + } + + + + /******************************************************************************* + ** Confirm with the original plus two explicit samples. Recover only after + ** three later successful runs on the same major version. + *******************************************************************************/ + private void advance(Map target, Map run, String metric, String column, + Map open, List changed) throws SQLException + { + boolean suspected = "SUSPECTED".equals(open.get("status")); + List> samples = suspected + ? store.query("SELECT * FROM {lab_run} WHERE target_id = ? AND strategy = ? AND status = 'SUCCEEDED' AND id <= ?" + + " AND (id = ? OR trigger_ref IN (?, ?)) AND (lighthouse_version = ? OR lighthouse_version LIKE ?) ORDER BY id FETCH FIRST 3 ROWS ONLY", + target.get("id"), run.get("strategy"), run.get("id"), open.get("first_run_id"), "confirm:" + open.get("first_run_id") + ":0", + "confirm:" + open.get("first_run_id") + ":1", major(run), major(run) + ".%") + : store.query("SELECT * FROM {lab_run} WHERE target_id = ? AND strategy = ? AND status = 'SUCCEEDED' AND id > ? AND id <= ?" + + " AND (lighthouse_version = ? OR lighthouse_version LIKE ?) ORDER BY id DESC FETCH FIRST 3 ROWS ONLY", + target.get("id"), run.get("strategy"), open.get("checkpoint_run_id"), run.get("id"), major(run), major(run) + ".%"); + if(samples.size() < 3 || samples.stream().anyMatch(row -> row.get(column) == null)) + { + return; + } + double value = median(samples, column); + Double reference = WebPerfStore.asDouble(open.get("baseline_value")); + Map budget = breachedBudget(target, "LAB", (String) run.get("strategy"), metric, value); + boolean breached = drift(metric, reference, value) || budget != null || auditsStillFail(open, samples); + Instant at = (Instant) run.get("completed_at"); + store.update("UPDATE {regression} SET current_value = ?, current_ref = ?, delta = ?, budget_id = ?, severity = ?, modify_date = ? WHERE id = ?", + value, refs(samples), reference == null ? null : value - reference, budget == null ? null : budget.get("id"), + budget == null ? "WARNING" : budget.get("severity"), at, open.get("id")); + if(suspected && breached) + { + store.update("UPDATE {regression} SET status = 'CONFIRMED', confirmed_at = ?, checkpoint_run_id = ? WHERE id = ?", at, run.get("id"), open.get("id")); + changed.add(WebPerfStore.asLong(open.get("id"))); + } + else if(!breached) + { + close(open, suspected ? "NOT_CONFIRMED" : "RESOLVED", suspected ? "Median of three did not confirm the regression" : "Recovered: median of three later runs", at); + changed.add(WebPerfStore.asLong(open.get("id"))); + } + } + + + + /******************************************************************************* + ** Match enabled budgets by source, strategy and every supported scope. + ** The comparator describes acceptable values; a breach is its inverse. + *******************************************************************************/ + public Map breachedBudget(Map target, String source, String strategy, String metric, double value) throws SQLException + { + Map winner = null; + long after = 0; + while(true) + { + List> budgets = store.query("SELECT * FROM {budget} WHERE enabled = TRUE AND source = ? AND metric = ? AND id > ?" + + " AND (strategy IS NULL OR strategy = '' OR strategy = ?) ORDER BY id FETCH FIRST 100 ROWS ONLY", source, metric, after, strategy); + if(budgets.isEmpty()) + { + return (winner); + } + for(Map budget : budgets) + { + after = WebPerfStore.asLong(budget.get("id")); + if(matches(target, budget) && budget.get("threshold") != null && !acceptable(value, (String) budget.get("comparator"), WebPerfStore.asDouble(budget.get("threshold")))) + { + if(winner == null || "CRITICAL".equals(budget.get("severity"))) + { + winner = budget; + } + } + } + } + } + + + + /******************************************************************************* + ** Scope values are literal values, never SQL or wildcard expressions. + *******************************************************************************/ + private static boolean matches(Map target, Map budget) + { + String value = (String) budget.get("scope_value"); + return (switch(String.valueOf(budget.get("scope"))) + { + case "SITE" -> value == null || value.isBlank() || value.equals(target.get("site_key")); + case "TIER" -> Objects.equals(value, target.get("tier")); + case "TARGET" -> Objects.equals(value, String.valueOf(target.get("id"))); + case "CONTENT_TABLE" -> Objects.equals(value, target.get("content_table")); + case "PATH_PREFIX" -> value != null && target.get("path") instanceof String path && path.startsWith(value); + default -> false; + }); + } + + + + /******************************************************************************* + ** Supported budget comparators. Invalid records fail visibly on evaluation. + *******************************************************************************/ + static boolean acceptable(double value, String comparator, double threshold) + { + return (switch(String.valueOf(comparator)) + { + case "<=", "LE" -> value <= threshold; + case "<", "LT" -> value < threshold; + case ">=", "GE" -> value >= threshold; + case ">", "GT" -> value > threshold; + case "=", "==", "EQ" -> value == threshold; + case "!=", "<>", "NE" -> value != threshold; + default -> throw new IllegalArgumentException("Unsupported performance budget comparator: " + comparator); + }); + } + + + + /******************************************************************************* + ** Design section 6.4: relative and absolute LCP limits must both be met. + *******************************************************************************/ + static boolean drift(String metric, Double baseline, double value) + { + if(baseline == null) + { + return (false); + } + double delta = value - baseline; + return (switch(metric) + { + case "PERFORMANCE_SCORE" -> delta <= -10; + case "ACCESSIBILITY_SCORE", "BEST_PRACTICES_SCORE", "SEO_SCORE" -> delta <= -5; + case "LCP" -> delta >= 500 && value >= baseline * 1.25; + case "CLS" -> delta >= 0.05 - 0.0000001; + case "TBT" -> delta >= 200; + default -> false; + }); + } + + + + /******************************************************************************* + ** Newly failing deterministic audits, compared only within the same major. + *******************************************************************************/ + private List newFailures(Map run, String metric) throws SQLException + { + if(!metric.endsWith("_SCORE") || "PERFORMANCE_SCORE".equals(metric)) + { + return (List.of()); + } + String category = metric.replace("_SCORE", ""); + List> previous = store.query("SELECT id FROM {lab_run} WHERE target_id = ? AND strategy = ? AND id < ?" + + " AND status = 'SUCCEEDED' AND (lighthouse_version = ? OR lighthouse_version LIKE ?) ORDER BY id DESC FETCH FIRST 1 ROWS ONLY", + run.get("target_id"), run.get("strategy"), run.get("id"), major(run), major(run) + ".%"); + if(previous.isEmpty()) + { + return (List.of()); + } + return (store.query("SELECT audit_id FROM {lab_finding} f WHERE run_id = ? AND category = ? AND score < 1" + + " AND NOT EXISTS (SELECT 1 FROM {lab_finding} old WHERE old.run_id = ? AND old.audit_id = f.audit_id AND old.score < 1)" + + " ORDER BY audit_id FETCH FIRST 19 ROWS ONLY", run.get("id"), category, previous.get(0).get("id")) + .stream().map(row -> (String) row.get("audit_id")).toList()); + } + + + + /******************************************************************************* + ** An audit-caused regression recovers when a majority no longer fail it. + *******************************************************************************/ + private boolean auditsStillFail(Map open, List> samples) throws SQLException + { + String audits = (String) open.get("failed_audits"); + if(audits == null || audits.isBlank()) + { + return (false); + } + for(String audit : audits.split(",")) + { + int failed = 0; + for(Map sample : samples) + { + if(!store.query("SELECT id FROM {lab_finding} WHERE run_id = ? AND audit_id = ? AND score < 1 FETCH FIRST 1 ROWS ONLY", sample.get("id"), audit).isEmpty()) + { + failed++; + } + } + if(failed >= 2) + { + return (true); + } + } + return (false); + } + + + + /******************************************************************************* + ** Record the terminal state without discarding evidence. + *******************************************************************************/ + private void close(Map regression, String status, String note, Instant at) throws SQLException + { + store.update("UPDATE {regression} SET status = ?, resolved_at = ?, resolution_note = ?, modify_date = ? WHERE id = ?", + status, at, note, at, regression.get("id")); + } + + + + /******************************************************************************* + ** Record action; rechecks state under the same target lock as evaluation. + *******************************************************************************/ + public boolean finishRegression(long id, String status, String note, Instant at) throws SQLException + { + if(!List.of("DISMISSED", "RESOLVED").contains(status) || note == null || note.isBlank() || note.length() > 2000) + { + throw (new IllegalArgumentException("A resolution note of 1 to 2000 characters is required")); + } + boolean changed = store.transaction(tx -> + { + List> rows = tx.query("SELECT * FROM {regression} WHERE id = ?", id); + if(rows.isEmpty()) + { + return (false); + } + Map row = rows.get(0); + tx.query("SELECT id FROM {target} WHERE id = ? FOR UPDATE", row.get("target_id")); + int updated = tx.update("UPDATE {regression} SET status = ?, resolved_at = ?, resolution_note = ?, modify_date = ? WHERE id = ? AND status IN ('SUSPECTED', 'CONFIRMED')", + status, at, note, at, id); + new WebPerfEvaluationService(tx).updateCount(row.get("target_id")); + return (updated == 1); + }); + if(changed) + { + sendAlert(id); + } + return (changed); + } + + + + /******************************************************************************* + ** The list count includes suspected and confirmed open regressions. + *******************************************************************************/ + private void updateCount(Object targetId) throws SQLException + { + store.update("UPDATE {target} SET open_regression_count = (SELECT COUNT(*) FROM {regression} WHERE target_id = ? AND status IN ('SUSPECTED', 'CONFIRMED')) WHERE id = ?", + targetId, targetId); + } + + + + /******************************************************************************* + ** Optional core producer API; without an Alerts QBit core uses its log sink. + *******************************************************************************/ + private void sendAlert(long id) + { + if(!store.getConfig().isOn(WebPerfFeature.ALERTS)) + { + return; + } + try + { + Map row = store.query("SELECT * FROM {regression} WHERE id = ?", id).get(0); + String key = store.getConfig().getTableNamePrefix() + ":" + row.get("target_id") + ":" + row.get("source") + ":" + row.get("strategy") + ":" + row.get("metric"); + if("CONFIRMED".equals(row.get("status"))) + { + QAlerts.raise(new AlertSignal(ALERT_TYPE).withDedupKey(key).withSeverity(AlertSeverity.valueOf((String) row.get("severity"))) + .withTitle("Confirmed web performance regression").withSummary(row.get("strategy") + " " + row.get("metric")) + .withRecordRef(AlertRecordRef.of(store.getConfig().applyPrefix(WebPerfSchema.REGRESSION), id))); + } + else if(List.of("RESOLVED", "DISMISSED").contains(row.get("status"))) + { + QAlerts.resolve(ALERT_TYPE, key, "Performance regression " + row.get("status").toString().toLowerCase(Locale.ROOT)); + } + } + catch(SQLException e) + { + throw (new IllegalStateException("Reading a committed performance regression failed", e)); + } + } + + + + /******************************************************************************* + ** Median of bounded, non-null measurement values. + *******************************************************************************/ + static Double median(List> rows, String column) + { + List values = rows.stream().map(row -> WebPerfStore.asDouble(row.get(column))).filter(Objects::nonNull).sorted().toList(); + int size = values.size(); + return (size == 0 ? null : size % 2 == 1 ? values.get(size / 2) : (values.get(size / 2 - 1) + values.get(size / 2)) / 2); + } + + + + /******************************************************************************* + ** Stored ids make a statistical result reproducible. + *******************************************************************************/ + static String refs(List> rows) + { + return (String.join(",", rows.stream().map(row -> row.get("id").toString()).toList())); + } + + + + /******************************************************************************* + ** Unknown versions never mix with a known Lighthouse baseline. + *******************************************************************************/ + static String major(Map run) + { + return (String.valueOf(run.get("lighthouse_version")).split("\\.")[0]); + } +} diff --git a/src/main/java/com/kingsrook/qbits/webperf/service/WebPerfService.java b/src/main/java/com/kingsrook/qbits/webperf/service/WebPerfService.java index d0889a2..08e3786 100644 --- a/src/main/java/com/kingsrook/qbits/webperf/service/WebPerfService.java +++ b/src/main/java/com/kingsrook/qbits/webperf/service/WebPerfService.java @@ -205,7 +205,6 @@ public class WebPerfService values.put("enabled", true); values.put("in_sitemap", true); values.put("last_seen_in_sitemap_at", now); - values.put("open_regression_count", 0); store.insert(WebPerfSchema.TARGET, values); } for(Long id : plan.seenIds()) @@ -303,10 +302,6 @@ public class WebPerfService public QueuePlan planQueue(List requests, Instant now) throws SQLException { Set open = new HashSet<>(); - for(Map row : store.query("SELECT target_id, strategy FROM {lab_run} WHERE status IN ('QUEUED', 'RUNNING')")) - { - open.add(row.get("target_id") + "|" + row.get("strategy")); - } int runsToday = runsSince(startOfToday(now)); List queue = new ArrayList<>(); @@ -315,7 +310,8 @@ public class WebPerfService for(RunRequest request : requests) { String key = request.targetId() + "|" + request.strategy(); - if(!open.add(key)) + if(!open.add(key) || !store.query("SELECT id FROM {lab_run} WHERE target_id = ? AND strategy = ? AND status IN ('QUEUED', 'RUNNING') FETCH FIRST 1 ROWS ONLY", + request.targetId(), request.strategy()).isEmpty()) { merged.add(request); } @@ -359,6 +355,48 @@ public class WebPerfService + /******************************************************************************* + ** Queue an independent confirmation or comparison sample. Its reference is + ** stable across retries; deliberate samples must not merge with each other. + *******************************************************************************/ + public String enqueueSample(Map target, String strategy, String trigger, String reference, Instant due) throws SQLException + { + if(!store.query("SELECT id FROM {lab_run} WHERE target_id = ? AND strategy = ? AND trigger_ref = ? AND status NOT IN ('FAILED', 'RUNTIME_ERROR', 'SKIPPED_QUOTA') FETCH FIRST 1 ROWS ONLY", + target.get("id"), strategy, reference).isEmpty()) + { + return ("MERGED"); + } + Instant now = Instant.now(); + boolean capped = runsSince(startOfToday(now)) >= config.getMaxRunsPerDay(); + RunRequest request = new RunRequest(WebPerfStore.asLong(target.get("id")), (String) target.get("url"), strategy, trigger, reference, null, due); + store.insert(WebPerfSchema.LAB_RUN, runValues(request, capped ? "SKIPPED_QUOTA" : "QUEUED", now, + capped ? "Daily run cap reached; use Retry to collect this sample" : null)); + return (capped ? "SKIPPED_QUOTA" : "QUEUED"); + } + + + + /******************************************************************************* + ** Retry a deliberate sample without losing its confirmation/comparison slot. + ** Keep the failed row as history; simultaneous retries share the target lock. + *******************************************************************************/ + public String retrySample(long runId, Instant now) throws SQLException + { + return (store.transaction(tx -> + { + Map run = tx.query("SELECT * FROM {lab_run} WHERE id = ?", runId).get(0); + Map target = tx.query("SELECT id, url FROM {target} WHERE id = ? FOR UPDATE", run.get("target_id")).get(0); + if(!List.of("FAILED", "RUNTIME_ERROR", "SKIPPED_QUOTA").contains(run.get("status"))) + { + return ("MERGED"); + } + return (new WebPerfService(runtime, tx).enqueueSample(target, (String) run.get("strategy"), (String) run.get("trigger_type"), + (String) run.get("trigger_ref"), now)); + })); + } + + + /******************************************************************************* ** The column values of a new run. *******************************************************************************/ @@ -460,6 +498,7 @@ public class WebPerfService *******************************************************************************/ public int runDue(Instant now) throws Exception { + new WebPerfEvaluationService(store).evaluatePending(); LabRunner runner = runtime.labRunner(); if(runner == null) { @@ -467,7 +506,7 @@ public class WebPerfService return (0); } - List> due = store.query("SELECT id FROM {lab_run} WHERE status = 'QUEUED' AND not_before <= ? ORDER BY not_before, id", now); + List> due = store.query("SELECT id FROM {lab_run} WHERE status = 'QUEUED' AND not_before <= ? ORDER BY not_before, id FETCH FIRST " + config.getConcurrency() + " ROWS ONLY", now); List claimed = new ArrayList<>(); for(Map row : due) { @@ -513,6 +552,7 @@ public class WebPerfService { executor.shutdownNow(); } + new WebPerfEvaluationService(store).evaluatePending(); return (claimed.size()); } @@ -529,6 +569,7 @@ public class WebPerfService { Map run = store.query("SELECT id, target_id, url, strategy, attempts FROM {lab_run} WHERE id = ?", runId).get(0); int attempts = WebPerfStore.asInt(run.get("attempts")) + 1; + new WebPerfChangeService(store).attribute(runId, Instant.now()); LabRunner.LabResponse response; try { @@ -572,6 +613,21 @@ public class WebPerfService ** the compressed report, and the page's latest results. *******************************************************************************/ private void store(long runId, Map run, int attempts, LabResultParser.ParsedLab parsed, LabRunner.LabResponse response) throws SQLException + { + store.transaction(tx -> + { + new WebPerfService(runtime, tx).storeResult(runId, run, attempts, parsed, response); + return (null); + }); + } + + + + /******************************************************************************* + ** Commit the result, findings, report and latest values as one unit, so an + ** evaluator never sees a successful run with only some of its evidence. + *******************************************************************************/ + private void storeResult(long runId, Map run, int attempts, LabResultParser.ParsedLab parsed, LabRunner.LabResponse response) throws SQLException { Instant completed = response.completedAt(); String status = parsed.runtimeError() == null ? "SUCCEEDED" : "RUNTIME_ERROR"; @@ -585,6 +641,7 @@ public class WebPerfService parsed.score("SEO"), parsed.fcpMs(), parsed.lcpMs(), parsed.tbtMs(), parsed.cls(), parsed.speedIndexMs(), parsed.lcpElement(), parsed.runWarnings(), completed, runId); + store.update("UPDATE {lab_run} SET ttfb_ms = ? WHERE id = ?", LabResultParser.ttfb(response.rawJson()), runId); Long targetId = WebPerfStore.asLong(run.get("target_id")); for(LabResultParser.Finding finding : parsed.findings()) { @@ -718,6 +775,7 @@ public class WebPerfService } } } + new WebPerfEvaluationService(store).evaluatePending(); return (counts); } diff --git a/src/main/java/com/kingsrook/qbits/webperf/store/WebPerfStore.java b/src/main/java/com/kingsrook/qbits/webperf/store/WebPerfStore.java index dcd874a..788ae8f 100644 --- a/src/main/java/com/kingsrook/qbits/webperf/store/WebPerfStore.java +++ b/src/main/java/com/kingsrook/qbits/webperf/store/WebPerfStore.java @@ -74,6 +74,86 @@ public class WebPerfStore private final WebPerfQBitConfig config; private final ConnectionSupplier connections; + private Connection transactionConnection; + + + + /******************************************************************************* + ** Work sharing one database transaction. + *******************************************************************************/ + @FunctionalInterface + public interface TransactionWork + { + /******************************************************************************* + ** Execute against the transaction's store. + *******************************************************************************/ + T run(WebPerfStore store) throws SQLException; + } + + + + /******************************************************************************* + ** Commit related writes together; roll back every write on failure. + *******************************************************************************/ + public T transaction(TransactionWork work) throws SQLException + { + if(transactionConnection != null) + { + return (work.run(this)); + } + try(Connection connection = connections.get()) + { + boolean autoCommit = connection.getAutoCommit(); + connection.setAutoCommit(false); + WebPerfStore transactional = new WebPerfStore(config, connections); + transactional.transactionConnection = connection; + try + { + T result = work.run(transactional); + connection.commit(); + return (result); + } + catch(SQLException | RuntimeException e) + { + connection.rollback(); + throw (e); + } + finally + { + connection.setAutoCommit(autoCommit); + } + } + } + + + + /******************************************************************************* + ** Own a connection only outside an enclosing transaction. + *******************************************************************************/ + private ConnectionLease lease() throws SQLException + { + return (new ConnectionLease(transactionConnection == null ? connections.get() : transactionConnection, transactionConnection == null)); + } + + + + /******************************************************************************* + ** A statement's connection lifetime. + *******************************************************************************/ + private record ConnectionLease(Connection connection, boolean owned) implements AutoCloseable + { + /******************************************************************************* + ** Leave an enclosing transaction's connection open. + *******************************************************************************/ + @Override + public void close() throws SQLException + { + if(owned) + { + connection.close(); + } + } + } @@ -119,7 +199,7 @@ public class WebPerfStore *******************************************************************************/ public int update(String template, Object... params) throws SQLException { - try(Connection connection = connections.get(); PreparedStatement statement = connection.prepareStatement(sql(template))) + try(ConnectionLease lease = lease(); PreparedStatement statement = lease.connection().prepareStatement(sql(template))) { bind(statement, params); return (statement.executeUpdate()); @@ -145,8 +225,8 @@ public class WebPerfStore String columns = String.join(", ", row.keySet()); String placeholders = String.join(", ", java.util.Collections.nCopies(row.size(), "?")); - try(Connection connection = connections.get(); - PreparedStatement statement = connection.prepareStatement("INSERT INTO " + config.applyPrefix(table) + " (" + columns + ") VALUES (" + placeholders + ")", + try(ConnectionLease lease = lease(); + PreparedStatement statement = lease.connection().prepareStatement("INSERT INTO " + config.applyPrefix(table) + " (" + columns + ") VALUES (" + placeholders + ")", Statement.RETURN_GENERATED_KEYS)) { bind(statement, row.values().toArray()); @@ -171,7 +251,7 @@ public class WebPerfStore *******************************************************************************/ public List> query(String template, Object... params) throws SQLException { - try(Connection connection = connections.get(); PreparedStatement statement = connection.prepareStatement(sql(template))) + try(ConnectionLease lease = lease(); PreparedStatement statement = lease.connection().prepareStatement(sql(template))) { bind(statement, params); List> rows = new ArrayList<>(); diff --git a/src/main/resources/db/liquibase/qbit-web-performance/web_performance.changelog.sql b/src/main/resources/db/liquibase/qbit-web-performance/web_performance.changelog.sql index b8acf4b..3c08e3d 100644 --- a/src/main/resources/db/liquibase/qbit-web-performance/web_performance.changelog.sql +++ b/src/main/resources/db/liquibase/qbit-web-performance/web_performance.changelog.sql @@ -261,3 +261,22 @@ SELECT name, scope, scope_value, source, strategy, metric, comparator, threshold UNION ALL SELECT 'Performance at least 50 (KEY pages)', 'TIER', 'KEY', 'LAB', NULL, 'PERFORMANCE_SCORE', '>=', 50, 'CRITICAL' ) seeds WHERE NOT EXISTS (SELECT 1 FROM ${web_performance.tablePrefix}budget WHERE seeded = TRUE); + + +-- changeset qbit-web-performance:11 +-- comment: Durable evaluation progress, event identity and comparison page identity. Existing history is preserved. +ALTER TABLE ${web_performance.tablePrefix}lab_run ADD COLUMN evaluated_at TIMESTAMP WITHOUT TIME ZONE; +ALTER TABLE ${web_performance.tablePrefix}lab_run ADD COLUMN change_event_id BIGINT; +ALTER TABLE ${web_performance.tablePrefix}regression ADD COLUMN first_run_id BIGINT; +ALTER TABLE ${web_performance.tablePrefix}regression ADD COLUMN checkpoint_run_id BIGINT; +ALTER TABLE ${web_performance.tablePrefix}regression ADD COLUMN lighthouse_major VARCHAR(20); +ALTER TABLE ${web_performance.tablePrefix}change_event ADD COLUMN event_key VARCHAR(64); +ALTER TABLE ${web_performance.tablePrefix}change_event ADD COLUMN site_key VARCHAR(200); +ALTER TABLE ${web_performance.tablePrefix}comparison ADD COLUMN target_id BIGINT; +CREATE UNIQUE INDEX uq_${web_performance.tablePrefix}change_event_key ON ${web_performance.tablePrefix}change_event (event_key); +CREATE UNIQUE INDEX uq_${web_performance.tablePrefix}comparison_page ON ${web_performance.tablePrefix}comparison (change_event_id, target_id, source, strategy, metric, segment); +CREATE INDEX idx_${web_performance.tablePrefix}lab_evaluation ON ${web_performance.tablePrefix}lab_run (evaluated_at, status, id); +ALTER TABLE ${web_performance.tablePrefix}regression ADD COLUMN failed_audits VARCHAR(2000); +ALTER TABLE ${web_performance.tablePrefix}lab_run ADD COLUMN ttfb_ms INTEGER; +ALTER TABLE ${web_performance.tablePrefix}field_snapshot ADD COLUMN evaluated_at TIMESTAMP WITHOUT TIME ZONE; +ALTER TABLE ${web_performance.tablePrefix}change_event ADD COLUMN last_seen_at TIMESTAMP WITHOUT TIME ZONE; diff --git a/src/test/java/com/kingsrook/qbits/webperf/WebPerfP1Test.java b/src/test/java/com/kingsrook/qbits/webperf/WebPerfP1Test.java index f2a675c..ab503a0 100644 --- a/src/test/java/com/kingsrook/qbits/webperf/WebPerfP1Test.java +++ b/src/test/java/com/kingsrook/qbits/webperf/WebPerfP1Test.java @@ -213,6 +213,7 @@ class WebPerfP1Test @Test void testWorkerSuccess() throws Exception { + store().update("UPDATE {budget} SET enabled = FALSE"); google.respond(new FixtureServer.Response(200, FixtureServer.fixture("psi-kof22-home-mobile.json"))); WebPerfService service = service(); service.applySync(service.planSync(List.of(SITE + "/")), Instant.now()); @@ -319,6 +320,7 @@ class WebPerfP1Test @Test void testActions() throws Exception { + store().update("UPDATE {budget} SET enabled = FALSE"); RunProcessOutput sync = run(WebPerfProcesses.SYNC_TARGETS); assertThat(summary(sync)).anyMatch(l -> l.contains("3") && l.contains("added")); assertEquals(3, store().query("SELECT id FROM {target}").size()); diff --git a/src/test/java/com/kingsrook/qbits/webperf/WebPerfRegressionTest.java b/src/test/java/com/kingsrook/qbits/webperf/WebPerfRegressionTest.java new file mode 100644 index 0000000..4123547 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/webperf/WebPerfRegressionTest.java @@ -0,0 +1,828 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.webperf; + + +import java.io.Serializable; +import java.time.Instant; +import java.time.LocalDate; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.Executors; +import com.kingsrook.qbits.webperf.lab.LabResultParser; +import com.kingsrook.qbits.webperf.processes.RegressionSteps; +import com.kingsrook.qbits.webperf.processes.WebPerfProcesses; +import com.kingsrook.qbits.webperf.service.WebPerfChangeService; +import com.kingsrook.qbits.webperf.service.WebPerfComparisonService; +import com.kingsrook.qbits.webperf.service.WebPerfEvaluationService; +import com.kingsrook.qbits.webperf.service.WebPerfService; +import com.kingsrook.qbits.webperf.spi.ChangeEventSource; +import com.kingsrook.qbits.webperf.spi.DeployProbe; +import com.kingsrook.qbits.webperf.spi.LabRunner; +import com.kingsrook.qbits.webperf.store.WebPerfStore; +import com.kingsrook.qqq.backend.core.actions.processes.QProcessCallback; +import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction; +import com.kingsrook.qqq.backend.core.alerts.AlertResolution; +import com.kingsrook.qqq.backend.core.alerts.AlertSignal; +import com.kingsrook.qqq.backend.core.alerts.AlertSinkInterface; +import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData; +import com.kingsrook.qqq.backend.core.alerts.QAlerts; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess; +import org.json.JSONObject; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; +import static org.junit.jupiter.api.Assertions.assertThrows; + + +/******************************************************************************* + ** P2 behavior on H2, with fake probes and runner and real QQQ record actions. + *******************************************************************************/ +public class WebPerfRegressionTest +{ + private static final Instant START = Instant.parse("2026-10-01T12:00:00Z"); + private QInstance instance; + private WebPerfQBitConfig config; + private WebPerfStore store; + private WebPerfEvaluationService evaluator; + private long target; + private int sequence; + + + + /******************************************************************************* + ** Fresh tables; disable reference budgets to isolate statistical rules. + *******************************************************************************/ + @BeforeEach + void setUp() throws Exception + { + instance = WebPerfTestSupport.newInstance(); + config = WebPerfTestSupport.config().withStrategies(List.of("MOBILE")).withMinSpacingMillis(0).withConcurrency(2); + new WebPerfQBitProducer().withQBitConfig(config).produce(instance); + WebPerfTestSupport.applyChangelog(instance, config.getTableNamePrefix()); + WebPerfTestSupport.login(instance, "web_performance.read", "web_performance.write", "web_performance.manage.hasAccess", + "web_performance.view.hasAccess", "web_performance.admin.hasAccess", "web_performance.test.hasAccess"); + store = new WebPerfRuntime(config).store(instance); + evaluator = new WebPerfEvaluationService(store); + store.update("UPDATE {budget} SET enabled = FALSE"); + WebPerfService service = new WebPerfRuntime(config).service(); + service.applySync(service.planSync(List.of("https://www.example.com/")), START); + target = WebPerfStore.asLong(store.query("SELECT id FROM {target}").get(0).get("id")); + store.update("UPDATE {target} SET tier = 'KEY', content_table = 'page', content_id = 7 WHERE id = ?", target); + RecordingSink.signals.clear(); + RecordingSink.resolutions.clear(); + AlertsInstanceMetaData.ofOrWithNew(instance).withSink(new QCodeReference(RecordingSink.class)); + } + + + + /******************************************************************************* + ** Release this test's context. + *******************************************************************************/ + @AfterEach + void tearDown() + { + QContext.clear(); + } + + + + /******************************************************************************* + ** Suspect, two confirmations, one alert, then three recovery samples. + *******************************************************************************/ + @Test + void confirmsAndRecoversWithoutDuplicates() throws Exception + { + baseline(); + long change = change("FRONTEND_DEPLOY", "FRONTEND", "build-2", START.plusSeconds(6)); + long first = measurement(70, "SCHEDULE", "13.1", Map.of("change_event_id", change)); + evaluator.evaluateLab(first); + assertThat(regression().get("status")).isEqualTo("SUSPECTED"); + assertThat(store.query("SELECT * FROM {lab_run} WHERE trigger_type = 'CONFIRM'")).hasSize(2); + assertThat(RecordingSink.signals).isEmpty(); + assertThat(openCount()).isEqualTo(1); + completeConfirmations(first, 72, 71); + assertThat(regression().get("status")).isEqualTo("CONFIRMED"); + assertThat(WebPerfStore.asDouble(regression().get("current_value"))).isEqualTo(71); + assertThat(regression().get("baseline_ref")).isEqualTo("5,4,3,2,1"); + assertThat(WebPerfStore.asLong(regression().get("cause_change_event_id"))).isEqualTo(change); + evaluator.evaluateLab(first); + assertThat(store.query("SELECT * FROM {regression}")).hasSize(1); + assertThat(RecordingSink.signals).hasSize(1); + evaluate(96); + evaluate(95); + assertThat(regression().get("status")).isEqualTo("CONFIRMED"); + evaluate(94); + assertThat(regression().get("status")).isEqualTo("RESOLVED"); + assertThat(openCount()).isZero(); + assertThat(RecordingSink.resolutions).hasSize(1); + assertThat(RecordingSink.signals.get(0).getDedupKey()).isEqualTo(RecordingSink.resolutions.get(0).getDedupKey()); + } + + + + /******************************************************************************* + ** Lab noise stays visible as not confirmed, with no alert. + *******************************************************************************/ + @Test + void rejectsNoise() throws Exception + { + baseline(); + long first = evaluate(70); + completeConfirmations(first, 96, 95); + assertThat(regression().get("status")).isEqualTo("NOT_CONFIRMED"); + assertThat(openCount()).isZero(); + assertThat(RecordingSink.signals).isEmpty(); + } + + + + /******************************************************************************* + ** Baselines require five eligible recent results on the same major version. + *******************************************************************************/ + @Test + void boundsBaselineHistory() throws Exception + { + for(int i = 0; i < 4; i++) + { + measurement(96, "SCHEDULE", "13.0", Map.of()); + } + measurement(96, "MANUAL", "13.0", Map.of()); + measurement(96, "SCHEDULE", "12.9", Map.of()); + long old = measurement(96, "SCHEDULE", "13.0", Map.of()); + store.update("UPDATE {lab_run} SET completed_at = ? WHERE id = ?", START.minusSeconds(31 * 86400L), old); + evaluate(70); + assertThat(store.query("SELECT * FROM {regression}")).isEmpty(); + measurement(96, "SITE_AUDIT", "13.1", Map.of()); + evaluate(60); + assertThat(regression().get("status")).isEqualTo("SUSPECTED"); + } + + + + /******************************************************************************* + ** A major-version change records a marker and retires incompatible baselines. + *******************************************************************************/ + @Test + void resetsAtMajorVersionChange() throws Exception + { + baseline(); + long first = evaluate(70); + completeConfirmations(first, 71, 72); + evaluator.evaluateLab(measurement(60, "SCHEDULE", "14.0", Map.of())); + assertThat(regression().get("status")).isEqualTo("RESOLVED"); + assertThat(regression().get("resolution_note")).isEqualTo("Lighthouse major version changed; baseline reset"); + assertThat(store.query("SELECT * FROM {change_event} WHERE kind = 'LIGHTHOUSE_VERSION'")).hasSize(1); + assertThat(store.query("SELECT * FROM {regression}")).hasSize(1); + } + + + + /******************************************************************************* + ** Relative and absolute thresholds; deterministic categories confirm now. + *******************************************************************************/ + @Test + void appliesMetricThresholds() throws Exception + { + baseline(); + evaluator.evaluateLab(measurement(95, "MANUAL", "13.0", Map.of("lcp_ms", 2499, "cls", 0.049, "tbt_ms", 199))); + assertThat(store.query("SELECT * FROM {regression}")).isEmpty(); + evaluator.evaluateLab(measurement(95, "MANUAL", "13.0", Map.of("lcp_ms", 2500, "cls", 0.05, "tbt_ms", 200, + "score_accessibility", 95, "score_best_practices", 95, "score_seo", 95))); + assertThat(store.query("SELECT * FROM {regression} WHERE status = 'SUSPECTED'")).hasSize(3); + assertThat(store.query("SELECT * FROM {regression} WHERE status = 'CONFIRMED'")).hasSize(3); + assertThat(store.query("SELECT * FROM {lab_run} WHERE trigger_type = 'CONFIRM'")).hasSize(2); + assertThat(RecordingSink.signals).hasSize(3); + } + + + + /******************************************************************************* + ** A newly failing audit is actionable even if the rounded score is unchanged. + *******************************************************************************/ + @Test + void newlyFailingAuditConfirms() throws Exception + { + baseline(); + long run = measurement(96, "MANUAL", "13.0", Map.of()); + store.insert(WebPerfSchema.LAB_FINDING, Map.of("run_id", run, "target_id", target, "strategy", "MOBILE", + "category", "SEO", "audit_id", "robots-txt", "kind", "FAILED_AUDIT", "score", 0)); + evaluator.evaluateLab(run); + assertThat(regression().get("status")).isEqualTo("CONFIRMED"); + assertThat(regression().get("failed_audits")).isEqualTo("robots-txt"); + evaluate(96); + evaluate(96); + evaluate(96); + assertThat(regression().get("status")).isEqualTo("RESOLVED"); + } + + + + /******************************************************************************* + ** Every scope and comparator, with source, strategy and severity matching. + *******************************************************************************/ + @Test + void matchesBudgetScopesComparatorsAndSeverity() throws Exception + { + Map page = store.query("SELECT * FROM {target}").get(0); + for(String scope : List.of("SITE", "TIER", "TARGET", "CONTENT_TABLE", "PATH_PREFIX")) + { + String value = switch(scope) + { + case "SITE" -> "host:www.example.com"; + case "TIER" -> "KEY"; + case "TARGET" -> String.valueOf(target); + case "CONTENT_TABLE" -> "page"; + default -> "/"; + }; + long id = budget(scope, value, "LAB", "MOBILE", "LCP", "<=", 2500, "WARNING"); + assertThat(evaluator.breachedBudget(page, "LAB", "MOBILE", "LCP", 2500)).isNull(); + assertThat(evaluator.breachedBudget(page, "LAB", "MOBILE", "LCP", 2501)).isNotNull(); + assertThat(evaluator.breachedBudget(page, "LAB", "DESKTOP", "LCP", 2501)).isNull(); + assertThat(evaluator.breachedBudget(page, "CRUX", "MOBILE", "LCP", 2501)).isNull(); + store.update("UPDATE {budget} SET scope_value = 'unmatched' WHERE id = ?", id); + assertThat(evaluator.breachedBudget(page, "LAB", "MOBILE", "LCP", 2501)).isNull(); + store.update("UPDATE {budget} SET enabled = FALSE WHERE id = ?", id); + } + for(String comparator : List.of("<", ">", ">=", "=", "!=", "<=", "==", "<>", "LE", "GE", "LT", "GT", "EQ", "NE")) + { + long id = budget("SITE", "", "LAB", null, "LCP", comparator, 10, "WARNING"); + for(double value : List.of(9.0, 10.0, 11.0)) + { + boolean acceptable = switch(comparator) + { + case "<", "LT" -> value < 10; + case ">", "GT" -> value > 10; + case ">=", "GE" -> value >= 10; + case "<=", "LE" -> value <= 10; + case "=", "==", "EQ" -> value == 10; + default -> value != 10; + }; + assertThat(evaluator.breachedBudget(page, "LAB", "MOBILE", "LCP", value) == null).isEqualTo(acceptable); + } + store.update("UPDATE {budget} SET enabled = FALSE WHERE id = ?", id); + } + budget("TIER", "KEY", "LAB", null, "PERFORMANCE_SCORE", ">=", 90, "WARNING"); + long critical = budget("TIER", "KEY", "LAB", null, "PERFORMANCE_SCORE", ">=", 50, "CRITICAL"); + long first = evaluate(40); + assertThat(WebPerfStore.asLong(regression().get("budget_id"))).isEqualTo(critical); + completeConfirmations(first, 40, 41); + assertThat(regression().get("severity")).isEqualTo("CRITICAL"); + } + + + + /******************************************************************************* + ** CrUX URL budgets use real values, never absent data or origin fallbacks. + *******************************************************************************/ + @Test + void evaluatesCruxUrlBudgets() throws Exception + { + budget("SITE", "", "CRUX", "MOBILE", "LCP", "<=", 2500, "WARNING"); + long first = field("URL", "OK", 3000, 1); + evaluator.evaluateField(first); + evaluator.evaluateField(first); + assertThat(regression().get("source")).isEqualTo("CRUX"); + assertThat(regression().get("status")).isEqualTo("CONFIRMED"); + evaluator.evaluateField(field("URL", "NO_DATA", null, 2)); + evaluator.evaluateField(field("ORIGIN", "OK", 1000, 3)); + assertThat(regression().get("status")).isEqualTo("CONFIRMED"); + evaluator.evaluateField(field("URL", "OK", 2000, 4)); + assertThat(regression().get("status")).isEqualTo("RESOLVED"); + assertThat(openCount()).isZero(); + assertThat(RecordingSink.signals).hasSize(1); + } + + + + /******************************************************************************* + ** Poll replay, settle delays and attribution through the real queue worker. + *******************************************************************************/ + @Test + void pollsAndAttributesWorkerResults() throws Exception + { + MutableProbe probe = new MutableProbe(); + config.withDeployProbe(probe).withLabRunner(new FakeRunner()); + ChangeEventSource.Change cdn = new ChangeEventSource.Change("CDN_STAGE", "www.example.com", "OFF", "HEADERS_ONLY", START); + ChangeEventSource.Change content = new ChangeEventSource.Change("CONTENT", "page:7", "old", "new", START.plusSeconds(1)); + config.withChangeEventSource(since -> List.of(cdn, content)); + WebPerfChangeService changes = new WebPerfChangeService(store); + changes.poll(START.plusSeconds(2)); + changes.poll(START.plusSeconds(3)); + assertThat(store.query("SELECT * FROM {change_event}")).hasSize(3); + assertThat(store.query("SELECT * FROM {lab_run} WHERE trigger_type = 'CHANGE_COMPARE'")).hasSize(3); + assertThat(store.query("SELECT * FROM {lab_run} WHERE trigger_type = 'DEPLOY'").get(0).get("not_before")).isEqualTo(START.plusSeconds(302)); + new WebPerfRuntime(config).service().runDue(Instant.now()); + List> done = store.query("SELECT * FROM {lab_run} WHERE status = 'SUCCEEDED'"); + assertThat(done).hasSize(2); + assertThat(done).allSatisfy(row -> + { + assertThat(row.get("deployment_id")).isEqualTo("build-1"); + assertThat(row.get("cdn_stage")).isEqualTo("HEADERS_ONLY"); + assertThat(row.get("evaluated_at")).isNotNull(); + }); + probe.build = "build-2"; + changes.poll(Instant.now()); + assertThat(store.query("SELECT * FROM {change_event} WHERE kind = 'FRONTEND_DEPLOY'")).hasSize(2); + probe.build = null; + changes.poll(Instant.now()); + assertThat(store.query("SELECT * FROM {change_event} WHERE kind = 'FRONTEND_DEPLOY'")).hasSize(2); + } + + + + /******************************************************************************* + ** Three samples on each side create reproducible, idempotent comparisons. + *******************************************************************************/ + @Test + void recordsComparisonsWithCountsAndWindows() throws Exception + { + for(int i = 0; i < 3; i++) + { + measurement(90, "CHANGE_COMPARE", "13.0", Map.of("lcp_ms", 3000, "fcp_ms", 2000, "ttfb_ms", 900)); + } + long event = change("CDN_STAGE", "www.example.com", "LIVE", START.plusSeconds(4)); + sequence = 4; + for(int i = 0; i < 3; i++) + { + long id = measurement(96, "CHANGE_COMPARE", "13.1", Map.of("change_event_id", event, "lcp_ms", 2000, "fcp_ms", 1500, "ttfb_ms", 600)); + evaluator.evaluateLab(id); + assertThat(store.query("SELECT * FROM {comparison}")).hasSize(i == 2 ? 4 : 0); + } + assertThat(new WebPerfComparisonService(store).compare(event)).isEqualTo(4); + assertThat(store.query("SELECT * FROM {comparison}")).hasSize(4); + Map lcp = store.query("SELECT * FROM {comparison} WHERE metric = 'LCP'").get(0); + assertThat(WebPerfStore.asDouble(lcp.get("delta"))).isEqualTo(-1000); + assertThat(WebPerfStore.asInt(lcp.get("before_n"))).isEqualTo(3); + assertThat(WebPerfStore.asInt(lcp.get("after_n"))).isEqualTo(3); + assertThat(lcp.get("before_window").toString()).contains(START.plusSeconds(1).toString()); + assertThat(runAction(WebPerfProcesses.COMPARE_CHANGE, Map.of()).getException()).isEmpty(); + } + + + + /******************************************************************************* + ** Real ETL actions require a note and manage permission, and retain history. + *******************************************************************************/ + @Test + void actionsRequireNotesAndPermissions() throws Exception + { + budget("SITE", "", "LAB", null, "SEO_SCORE", ">=", 100, "WARNING"); + evaluator.evaluateLab(measurement(96, "MANUAL", "13.0", Map.of("score_seo", 90))); + long id = WebPerfStore.asLong(regression().get("id")); + assertThrows(com.kingsrook.qqq.backend.core.exceptions.QException.class, () -> runAction(WebPerfProcesses.DISMISS_REGRESSIONS, Map.of())); + assertThat(regression().get("status")).isEqualTo("CONFIRMED"); + WebPerfTestSupport.login(instance, "web_performance.read", "web_performance.write"); + assertThrows(com.kingsrook.qqq.backend.core.exceptions.QException.class, () -> runAction(WebPerfProcesses.DISMISS_REGRESSIONS, Map.of(RegressionSteps.NOTE, "Expected change"))); + assertThat(regression().get("status")).isEqualTo("CONFIRMED"); + WebPerfTestSupport.login(instance, "web_performance.read", "web_performance.write", "web_performance.manage.hasAccess"); + RunProcessOutput result = runAction(WebPerfProcesses.DISMISS_REGRESSIONS, Map.of(RegressionSteps.NOTE, "Expected change")); + assertThat(result.getException()).isEmpty(); + assertThat(result.getValues().get(StreamedETLWithFrontendProcess.FIELD_PROCESS_SUMMARY)).isNotNull(); + assertThat(regression().get("status")).isEqualTo("DISMISSED"); + assertThat(regression().get("resolution_note")).isEqualTo("Expected change"); + assertThat(openCount()).isZero(); + assertThat(evaluator.finishRegression(id, "RESOLVED", "Already closed", START)).isFalse(); + evaluator.evaluateLab(measurement(96, "MANUAL", "13.0", Map.of("score_seo", 90))); + assertThat(runAction(WebPerfProcesses.RESOLVE_REGRESSIONS, Map.of(RegressionSteps.NOTE, "Fixed externally")).getException()).isEmpty(); + assertThat(store.query("SELECT * FROM {regression} WHERE status = 'RESOLVED'")).hasSize(1); + } + + + + /******************************************************************************* + ** Target locking prevents concurrent duplicate incidents; failed writes roll back. + *******************************************************************************/ + @Test + void serializesEvaluationAndRollsBack() throws Exception + { + baseline(); + long id = measurement(70, "SCHEDULE", "13.0", Map.of()); + try(var executor = Executors.newFixedThreadPool(2)) + { + var first = executor.submit(() -> + { + evaluator.evaluateLab(id); + return (null); + }); + var second = executor.submit(() -> + { + evaluator.evaluateLab(id); + return (null); + }); + first.get(); + second.get(); + } + assertThat(store.query("SELECT * FROM {regression}")).hasSize(1); + assertThat(store.query("SELECT * FROM {lab_run} WHERE trigger_type = 'CONFIRM'")).hasSize(2); + assertThrows(java.sql.SQLException.class, () -> store.transaction(tx -> + { + tx.update("UPDATE {target} SET label = 'must roll back' WHERE id = ?", target); + tx.update("UPDATE no_such_table SET id = 1"); + return (null); + })); + assertThat(store.query("SELECT label FROM {target}").get(0).get("label")).isNotEqualTo("must roll back"); + } + + + + /******************************************************************************* + ** Core falls back to logging without an Alerts QBit; feature gating works. + *******************************************************************************/ + @Test + void alertsAreOptional() throws Exception + { + AlertsInstanceMetaData.ofOrWithNew(instance).withSink(null); + long failures = QAlerts.getProducerFailureCount(); + budget("SITE", "", "LAB", null, "SEO_SCORE", ">=", 100, "WARNING"); + evaluator.evaluateLab(measurement(96, "MANUAL", "13.0", Map.of("score_seo", 90))); + assertThat(QAlerts.getProducerFailureCount()).isEqualTo(failures); + config.getFeatures().remove(WebPerfFeature.ALERTS); + evaluator.finishRegression(WebPerfStore.asLong(regression().get("id")), "RESOLVED", "Done", START); + assertThat(RecordingSink.signals).isEmpty(); + } + + + + /******************************************************************************* + ** TTFB comes from a measured subpart and is null when not reported. + *******************************************************************************/ + @Test + void parsesMeasuredTtfbOnly() + { + assertThat(LabResultParser.ttfb("{\"audits\":{}}")).isNull(); + assertThat(LabResultParser.ttfb("{\"audits\":{\"lcp-phases-insight\":{\"details\":{\"items\":[{\"subpart\":\"timeToFirstByte\",\"duration\":321.4}]}}}}")).isEqualTo(321); + } + + + + /******************************************************************************* + ** Quota skips remain visible, and retry preserves each deliberate sample slot. + *******************************************************************************/ + @Test + void retriesCappedConfirmationSamples() throws Exception + { + baseline(); + config.withMaxRunsPerDay(1); + long first = evaluate(70); + assertThat(store.query("SELECT * FROM {lab_run} WHERE status = 'SKIPPED_QUOTA'")).hasSize(1); + config.withMaxRunsPerDay(500); + assertThat(runAction(WebPerfProcesses.RETRY_RUNS, Map.of()).getException()).isEmpty(); + assertThat(store.query("SELECT * FROM {lab_run} WHERE status = 'QUEUED' AND trigger_type = 'CONFIRM'")).hasSize(2); + assertThat(runAction(WebPerfProcesses.RETRY_RUNS, Map.of()).getException()).isEmpty(); + assertThat(store.query("SELECT * FROM {lab_run} WHERE status = 'QUEUED' AND trigger_type = 'CONFIRM'")).hasSize(2); + store.update("UPDATE {lab_run} SET status = 'SUCCEEDED', completed_at = ?, lighthouse_version = '13.0', score_performance = 70 WHERE status = 'QUEUED'", + START.plusSeconds(10)); + evaluator.evaluatePending(); + assertThat(regression().get("status")).isEqualTo("CONFIRMED"); + assertThat(WebPerfStore.asLong(regression().get("first_run_id"))).isEqualTo(first); + } + + + + /******************************************************************************* + ** Existing seeded budgets are evaluated even before enough baseline exists. + *******************************************************************************/ + @Test + void seededBudgetsWorkWithoutBaseline() throws Exception + { + store.update("UPDATE {budget} SET enabled = TRUE WHERE seeded = TRUE"); + long first = evaluate(40); + assertThat(regression().get("status")).isEqualTo("SUSPECTED"); + assertThat(regression().get("baseline_value")).isNull(); + assertThat(regression().get("severity")).isEqualTo("CRITICAL"); + completeConfirmations(first, 40, 40); + assertThat(regression().get("status")).isEqualTo("CONFIRMED"); + } + + + + /******************************************************************************* + ** Build replays are idempotent even after a rollback to an earlier build. + *******************************************************************************/ + @Test + void attributesRollbackAndQueuesBothCdnWindows() throws Exception + { + MutableProbe probe = new MutableProbe(); + config.withDeployProbe(probe); + WebPerfChangeService changes = new WebPerfChangeService(store); + changes.poll(START); + probe.build = "build-2"; + changes.poll(START.plusSeconds(1)); + probe.build = "build-1"; + changes.poll(START.plusSeconds(2)); + long run = measurement(96, "MANUAL", "13.0", Map.of()); + changes.attribute(run, START.plusSeconds(3)); + assertThat(store.query("SELECT deployment_id FROM {lab_run} WHERE id = ?", run).get(0).get("deployment_id")).isEqualTo("build-1"); + assertThat(store.query("SELECT * FROM {change_event}")).hasSize(2); + ChangeEventSource.Change cdn = new ChangeEventSource.Change("CDN_STAGE", "www.example.com", "HEADERS_ONLY", "LIVE", START.plusSeconds(4)); + config.withChangeEventSource(since -> List.of(cdn)); + changes.poll(START.plusSeconds(4)); + changes.poll(START.plusSeconds(5)); + assertThat(store.query("SELECT * FROM {lab_run} WHERE trigger_type = 'CHANGE_COMPARE'")).hasSize(6); + assertThat(store.query("SELECT * FROM {lab_run} WHERE trigger_type = 'CHANGE_COMPARE' AND not_before = ?", START.plusSeconds(3604))).hasSize(3); + assertThat(store.query("SELECT * FROM {lab_run} WHERE trigger_type = 'CHANGE_COMPARE' AND not_before = ?", START.plusSeconds(86404))).hasSize(3); + assertThat(runAction(WebPerfProcesses.PROBE_DEPLOYS, Map.of()).getException()).isEmpty(); + } + + + + /******************************************************************************* + ** Five clean baseline measurements. + *******************************************************************************/ + private void baseline() throws Exception + { + for(int i = 0; i < 5; i++) + { + measurement(96, "SCHEDULE", "13.0", Map.of()); + } + } + + + + /******************************************************************************* + ** Store a deterministic successful measurement, with no network access. + *******************************************************************************/ + private long measurement(int score, String trigger, String version, Map extra) throws Exception + { + Instant time = START.plusSeconds(++sequence); + Map row = new LinkedHashMap<>(); + row.put("target_id", target); + row.put("url", "https://www.example.com/"); + row.put("strategy", "MOBILE"); + row.put("status", "SUCCEEDED"); + row.put("trigger_type", trigger); + row.put("lighthouse_version", version); + row.put("started_at", time); + row.put("completed_at", time); + row.put("queued_at", time); + row.put("score_performance", score); + row.put("score_accessibility", 100); + row.put("score_best_practices", 100); + row.put("score_seo", 100); + row.put("lcp_ms", 2000); + row.put("fcp_ms", 1000); + row.put("tbt_ms", 0); + row.put("cls", 0); + row.putAll(extra); + return (store.insert(WebPerfSchema.LAB_RUN, row)); + } + + + + /******************************************************************************* + ** Store and evaluate the next performance measurement. + *******************************************************************************/ + private long evaluate(int score) throws Exception + { + long id = measurement(score, "SCHEDULE", "13.0", Map.of()); + evaluator.evaluateLab(id); + return (id); + } + + + + /******************************************************************************* + ** Complete the independently queued confirmation samples. + *******************************************************************************/ + private void completeConfirmations(long first, int score1, int score2) throws Exception + { + int index = 0; + for(Map row : store.query("SELECT id FROM {lab_run} WHERE trigger_ref IN (?, ?) ORDER BY id", "confirm:" + first + ":0", "confirm:" + first + ":1")) + { + store.update("UPDATE {lab_run} SET status = 'SUCCEEDED', completed_at = ?, lighthouse_version = '13.0', score_performance = ? WHERE id = ?", + START.plusSeconds(++sequence), index++ == 0 ? score1 : score2, row.get("id")); + evaluator.evaluateLab(WebPerfStore.asLong(row.get("id"))); + } + } + + + + /******************************************************************************* + ** One editable budget. + *******************************************************************************/ + private long budget(String scope, String scopeValue, String source, String strategy, String metric, String comparator, double threshold, String severity) throws Exception + { + Map row = new LinkedHashMap<>(); + row.put("name", "Test budget"); + row.put("scope", scope); + row.put("scope_value", scopeValue); + row.put("source", source); + row.put("strategy", strategy); + row.put("metric", metric); + row.put("comparator", comparator); + row.put("threshold", threshold); + row.put("severity", severity); + row.put("enabled", true); + return (store.insert(WebPerfSchema.BUDGET, row)); + } + + + + /******************************************************************************* + ** A source event without automatic queueing for fixtures. + *******************************************************************************/ + private long change(String kind, String component, String value, Instant time) throws Exception + { + return (new WebPerfChangeService(store).record(new ChangeEventSource.Change(kind, component, "before", value, time), "WEBHOOK", "test", false)); + } + + + + /******************************************************************************* + ** A real or unavailable CrUX collection period. + *******************************************************************************/ + private long field(String scope, String status, Integer lcp, int day) throws Exception + { + Map row = new LinkedHashMap<>(); + row.put("source", "CRUX_API"); + row.put("scope", scope); + row.put("scope_key", "https://www.example.com/"); + row.put("form_factor", "PHONE"); + row.put("status", status); + row.put("p75_lcp_ms", lcp); + row.put("period_end", LocalDate.of(2026, 10, day)); + row.put("last_checked_at", START.plusSeconds(day)); + return (store.insert(WebPerfSchema.FIELD_SNAPSHOT, row)); + } + + + + /******************************************************************************* + ** The first incident, retained through status changes. + *******************************************************************************/ + private Map regression() throws Exception + { + return (store.query("SELECT * FROM {regression} ORDER BY id").get(0)); + } + + + + /******************************************************************************* + ** The count shown in Tested Pages. + *******************************************************************************/ + private int openCount() throws Exception + { + return (WebPerfStore.asInt(store.query("SELECT open_regression_count FROM {target} WHERE id = ?", target).get(0).get("open_regression_count"))); + } + + + + /******************************************************************************* + ** Run the actual ETL process, following the existing frontend-skip convention. + *******************************************************************************/ + private RunProcessOutput runAction(String name, Map values) throws Exception + { + RunProcessInput input = new RunProcessInput(); + input.setPermissionEnforcement(com.kingsrook.qqq.backend.core.actions.permissions.PermissionEnforcement.ENFORCE); + input.setProcessName(config.qualify(name)); + input.setValues(new HashMap<>(values)); + input.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.SKIP); + input.setCallback(new QProcessCallback() + { + /******************************************************************************* + ** Select this test's records. + *******************************************************************************/ + @Override + public QQueryFilter getQueryFilter() + { + return (new QQueryFilter()); + } + }); + return (new RunProcessAction().execute(input)); + } + + + + /******************************************************************************* + ** Core alert sink, without any Alerts QBit dependency. + *******************************************************************************/ + public static class RecordingSink implements AlertSinkInterface + { + static final List signals = new ArrayList<>(); + static final List resolutions = new ArrayList<>(); + + + + /******************************************************************************* + ** Capture a raise. + *******************************************************************************/ + @Override + public void raise(AlertSignal signal) + { + signals.add(signal); + } + + + + /******************************************************************************* + ** Capture a resolution. + *******************************************************************************/ + @Override + public void resolve(AlertResolution resolution) + { + resolutions.add(resolution); + } + + + + /******************************************************************************* + ** This producer does not emit heartbeats. + *******************************************************************************/ + @Override + public void heartbeat(String key, Instant at) + { + } + } + + + + /******************************************************************************* + ** A probe controlled by the test. + *******************************************************************************/ + private static class MutableProbe implements DeployProbe + { + String build = "build-1"; + + + + /******************************************************************************* + ** Frontend component. + *******************************************************************************/ + @Override + public String component() + { + return ("FRONTEND"); + } + + + + /******************************************************************************* + ** Build id without transport. + *******************************************************************************/ + @Override + public String currentBuildId() + { + return (build); + } + } + + + + /******************************************************************************* + ** A minimal Lighthouse runner without transport. + *******************************************************************************/ + private static class FakeRunner implements LabRunner + { + /******************************************************************************* + ** Stored runner name. + *******************************************************************************/ + @Override + public String name() + { + return ("TEST"); + } + + + + /******************************************************************************* + ** Return a successful response. + *******************************************************************************/ + @Override + public LabResponse run(LabRequest request) + { + JSONObject body = new JSONObject().put("lighthouseVersion", "13.0").put("finalUrl", request.url()) + .put("categories", new JSONObject().put("performance", new JSONObject().put("score", 0.96))) + .put("audits", new JSONObject()); + return (new LabResponse(200, body.toString(), Instant.now())); + } + } +} -- 2.54.0