QRun-IO/qbit-web-performance
3
0 Fork 0
Code Issues 2 Pull Requests 2 Actions Packages Projects Releases 7 Wiki Activity

feat(evaluation): write regressions, budget checks, site changes and comparisons (#8) #10

Merged
KofTwentyTwo merged 2 commits from feature/regressions-budgets into develop 2026-10-14 16:03:58 +00:00
Conversation 0 Commits 2 Files Changed 17
+2501 -35
17 changed files with 2501 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

CHANGELOG.md
+12
View File
@@ -9,4 +9,16 @@ All notable changes to this project are documented here. The format follows Keep
- Allow manual Tested Pages inserts with a URL. Validate absolute HTTPS URLs and fill in the site, path and Manual source automatically; validate URL edits too.
### Added
- Record deploy and external site changes once, queue settled retests, and attribute lab runs to the observed build and CDN stage.
- Evaluate enabled lab and URL CrUX budgets; retain breaches, confirmation evidence, automatic recovery and accurate open counts in the standard regression records.
- Confirm statistical lab regressions with a median of three, reset baselines on Lighthouse major changes, and detect deterministic category/audit failures.
- Add permission-gated Dismiss, Mark resolved and Compare before and after record actions with standard Review/Result summaries.
- Store before/after lab medians, sample counts and windows, including measured TTFB; schedule CDN samples before launch and at one and 24 hours after launch.
- Raise and resolve optional alerts through QQQ core, with logging fallback when no Alerts QBit is installed.
### Changed
- Use QQQ 4.1.0-RC.6 for the core alerts API; add migration fields for durable evaluation and source-event identity.
- Store complete run results atomically and retain confirmation/comparison identities when retrying samples.
### Previously added
- First implementation; see README.md for the tables, permissions and configuration.
README.md
+11 -7
View File
@@ -13,15 +13,19 @@ Design and owner decisions: QRun-IO/qqq `docs/specs/2026-10-12-web-performance-d
## Status
This is a **skeleton (0.1.0-SNAPSHOT)**. It is local only, and the GitHub repository does not exist yet. It contains:
The lab and Google field-data runners, queue, schedules, record actions and performance widgets are implemented. This change adds durable site changes, budget evaluation, confirmed lab regressions and before/after lab comparisons.
- the producer and config, with validation and secret redaction
- the nine `web_performance_` tables, their lookups and the Performance app
- the host-included changelog, including the seeded budgets (D12)
- the SPIs: `UrlSource`, `ContentResolver`, `LabRunner`, `DeployProbe`, `RumSource`, `ChangeEventSource`
- two implementations that need no Google API key: `SitemapUrlSource` and `HttpBuildIdProbe`, which reads Website-Frontend's `/api/build`
- Deploy probes run at startup through the scheduler and every five minutes. New build ids are recorded once per component, and KEY pages are queued after `deploySettleSeconds`. External `ChangeEventSource` events are deduplicated by source registration and their complete event tuple. Keep source registration order stable across restarts. Content events queue enabled pages after `publishSettleSeconds`, because the current SPI does not identify affected target ids.
- Each run records the latest observed frontend build, hostname CDN stage and preceding site change when it starts. A rollback to an already recorded build updates its last observation without duplicating its change row.
- Enabled LAB budgets match site, tier, target, content table or literal path prefix, plus strategy and metric. Comparators describe acceptable values. Breaches are recorded in Regressions with the budget id; there is no separate budget-check table in the design. The strongest matching severity wins. CRUX budgets evaluate URL p75 snapshots; NO_DATA, errors and origin fallback are never treated as page measurements.
- Statistical lab baselines require five successful SCHEDULE, CONFIRM or SITE_AUDIT runs in the preceding 30 days on the same Lighthouse major version. Performance drops of 10 points, LCP increases of both 25% and 500 ms, CLS increases of 0.05, TBT increases of 200 ms, and budget breaches create suspected regressions. Two independent confirmation runs provide a median of three. Category drops of 5 points and newly failing Accessibility, Best Practices or SEO audits confirm immediately. Recovery requires a median of three later runs. Lighthouse major changes record a marker and reset incompatible baselines.
- The page's open count includes suspected and confirmed regressions. **Dismiss** and **Mark resolved** require a note and `web_performance.manage.hasAccess`, use standard Review/Result summary lines, and preserve history.
- **Compare before and after** uses three successful same-version lab samples on each side of a change, with page, sample counts and actual time windows. It stores Performance, LCP, FCP and measured TTFB when available. Results are also updated after attributed runs complete. CDN HEADERS_ONLY queues three samples per strategy; LIVE queues three at +1 hour and three at +24 hours. Insufficient history produces no invented comparison.
- Confirmed regressions raise keyed `webperf.regression` alerts through QQQ core `QAlerts`; recovery, dismissal and manual resolution clear them. `ALERTS` can be disabled. Without an Alerts QBit, core uses its logging sink. No Alerts QBit dependency is required.
Not built yet: the PSI and CrUX runners (they need the key), the queue worker, the schedules, the actions, the widgets, and the Alerts qbit producer integration.
Evaluation progress and related writes commit together. Retrying an evaluation does not duplicate incidents, and retrying a failed confirmation preserves its sample reference. The additive changelog must be applied before running the new worker. This version requires QQQ **4.1.0-RC.6** for the core alerts API.
RUM drift and RUM/country before-and-after comparisons remain separate work: the existing `RumSource` exposes fixed daily/28-day aggregates, not arbitrary before/after windows. Enum field metadata and Tested Pages insertion are also outside this change.
## Install
pom.xml
+2 -2
View File
@@ -143,14 +143,14 @@
<profiles>
<!-- Built against qqq 4.1 (QBitMetaDataProducer). Until qbit-build-parent -->
<!-- imports the released 4.1 BOM, this profile is active by default and -->
<!-- imports qqq-bom-pom 4.1.0-RC.4 ahead of the parent's 4.0.0 BOM. -->
<!-- imports qqq-bom-pom 4.1.0-RC.6 ahead of the parent's 4.0.0 BOM. -->
<profile>
<id>qqq-snapshot</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<qqq.snapshot.version>4.1.0-RC.4</qqq.snapshot.version>
<qqq.snapshot.version>4.1.0-RC.6</qqq.snapshot.version>
</properties>
<dependencyManagement>
<dependencies>
src/main/java/com/kingsrook/qbits/webperf/lab/LabResultParser.java
+47
View File
@@ -139,6 +139,53 @@ public final class LabResultParser
/*******************************************************************************
** Read a reported TTFB subpart from Lighthouse's LCP insight, never savings.
*******************************************************************************/
public static Integer ttfb(String json)
{
JSONObject root = new JSONObject(json);
JSONObject lhr = root.optJSONObject("lighthouseResult");
JSONObject audits = (lhr == null ? root : lhr).optJSONObject("audits");
if(audits == null)
{
return (null);
}
for(String name : List.of("lcp-phases-insight", "lcp-breakdown-insight"))
{
JSONObject audit = audits.optJSONObject(name);
JSONObject details = audit == null ? null : audit.optJSONObject("details");
JSONArray items = details == null ? null : details.optJSONArray("items");
if(items != null)
{
for(int i = 0; i < items.length(); i++)
{
JSONObject item = items.optJSONObject(i);
if(item == null)
{
continue;
}
JSONArray subparts = item.optJSONArray("items");
if(subparts == null)
{
subparts = new JSONArray().put(item);
}
for(int j = 0; j < subparts.length(); j++)
{
JSONObject part = subparts.optJSONObject(j);
if(part != null && "timeToFirstByte".equals(part.optString("subpart")) && part.opt("duration") instanceof Number duration)
{
return ((int) Math.round(duration.doubleValue()));
}
}
}
}
}
return (null);
}
/*******************************************************************************
** Parser only.
*******************************************************************************/
src/main/java/com/kingsrook/qbits/webperf/metadata/WebPerfMetaData.java
+40 -13
View File
@@ -26,21 +26,25 @@ import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import com.kingsrook.qbits.webperf.WebPerfFeature;
import com.kingsrook.qbits.webperf.WebPerfQBitConfig;
import com.kingsrook.qbits.webperf.WebPerfRuntime;
import com.kingsrook.qbits.webperf.WebPerfSchema;
import com.kingsrook.qbits.webperf.processes.WebPerfProcesses;
import com.kingsrook.qbits.webperf.service.WebPerfEvaluationService;
import com.kingsrook.qbits.webperf.widgets.WebPerfWidgets;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
import com.kingsrook.qqq.backend.core.alerts.QAlertTypeMetaData;
import com.kingsrook.qqq.backend.core.instances.QInstanceEnricher;
import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel;
import com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
@@ -50,6 +54,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValue;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection;
@@ -101,6 +106,16 @@ public final class WebPerfMetaData
*******************************************************************************/
public void addTo(QInstance qInstance)
{
if(config.isOn(WebPerfFeature.ALERTS))
{
AlertsInstanceMetaData.ofOrWithNew(qInstance).withAlertType(
new QAlertTypeMetaData()
.withName(WebPerfEvaluationService.ALERT_TYPE)
.withLabel("Web performance regression").withSourceQBitName("web-performance")
.withDefaultSeverity(AlertSeverity.WARNING)
.withSubjectTableName(config.applyPrefix(WebPerfSchema.REGRESSION))
.withVisibilityPermission(config.getPermissionBaseName() + ".read"));
}
for(QTableMetaData table : tables().values())
{
qInstance.addTable(table);
@@ -365,7 +380,10 @@ public final class WebPerfMetaData
f("lcpElement", QFieldType.STRING, "LCP Element").withMaxLength(300),
f("runWarnings", QFieldType.STRING, "Warnings").withMaxLength(2000),
f("deploymentId", QFieldType.STRING, "Frontend Build").withMaxLength(64),
f("cdnStage", QFieldType.STRING, "CDN Stage").withMaxLength(20)));
f("cdnStage", QFieldType.STRING, "CDN Stage").withMaxLength(20),
f("ttfbMs", QFieldType.INTEGER, "TTFB (ms)"),
f("evaluatedAt", QFieldType.DATE_TIME, "Evaluated"),
f("changeEventId", QFieldType.LONG, "Site Change").withPossibleValueSourceName(config.applyPrefix(WebPerfSchema.CHANGE_EVENT))));
return (table(WebPerfSchema.LAB_RUN, "Lab Runs", "speed", fields, false, Set.of())
.withRecordLabelFormatAndFields("%s %s", "strategy", "url")
@@ -374,7 +392,7 @@ public final class WebPerfMetaData
.withSection(section("metrics", "Lab Metrics", "timer", Tier.T2, "fcpMs", "lcpMs", "tbtMs", "cls", "speedIndexMs", "lcpElement"))
.withSection(section("details", "Details", "info", Tier.T2, "targetId", "finalUrl", "runner", "triggerRef", "batchId", "notBefore",
"queuedAt", "startedAt", "attempts", "httpStatus", "errorMessage", "lighthouseVersion", "chromeUserAgent", "benchmarkIndex",
"runWarnings", "deploymentId", "cdnStage"))
"runWarnings", "deploymentId", "cdnStage", "ttfbMs", "evaluatedAt", "changeEventId"))
.withSection(dates()));
}
@@ -451,11 +469,12 @@ public final class WebPerfMetaData
f("p75FcpMs", QFieldType.INTEGER, "FCP p75 (ms)"),
f("p75TtfbMs", QFieldType.INTEGER, "TTFB p75 (ms)"),
f("distributions", QFieldType.TEXT, "Distributions"),
f("lastCheckedAt", QFieldType.DATE_TIME, "Last Checked")));
f("lastCheckedAt", QFieldType.DATE_TIME, "Last Checked"),
f("evaluatedAt", QFieldType.DATE_TIME, "Evaluated")));
return (table(WebPerfSchema.FIELD_SNAPSHOT, "Google Field Data", "public", fields, false, Set.of())
.withRecordLabelFormatAndFields("%s %s", "scopeKey", "formFactor")
.withSection(section("snapshot", "Snapshot", "public", Tier.T1, "id", "scopeKey", "scope", "formFactor", "status", "lastCheckedAt"))
.withSection(section("snapshot", "Snapshot", "public", Tier.T1, "id", "scopeKey", "scope", "formFactor", "status", "lastCheckedAt", "evaluatedAt"))
.withSection(section("p75", "p75 (28 days)", "insights", Tier.T2, "p75LcpMs", "p75InpMs", "p75Cls", "p75FcpMs", "p75TtfbMs",
"periodStart", "periodEnd", "source", "distributions"))
.withSection(dates()));
@@ -515,14 +534,18 @@ public final class WebPerfMetaData
f("detectedAt", QFieldType.DATE_TIME, "Detected"),
f("confirmedAt", QFieldType.DATE_TIME, "Confirmed"),
f("resolvedAt", QFieldType.DATE_TIME, "Resolved"),
f("resolutionNote", QFieldType.STRING, "Resolution Note").withMaxLength(2000)));
f("resolutionNote", QFieldType.STRING, "Resolution Note").withMaxLength(2000),
f("firstRunId", QFieldType.LONG, "First Run").withPossibleValueSourceName(config.applyPrefix(WebPerfSchema.LAB_RUN)),
f("checkpointRunId", QFieldType.LONG, "Confirmation Run").withPossibleValueSourceName(config.applyPrefix(WebPerfSchema.LAB_RUN)),
f("lighthouseMajor", QFieldType.STRING, "Lighthouse Major").withMaxLength(20),
f("failedAudits", QFieldType.STRING, "Newly Failing Audits").withMaxLength(2000)));
return (table(WebPerfSchema.REGRESSION, "Regressions", "trending_down", fields, true, Set.of())
.withRecordLabelFormatAndFields("%s %s", "metric", "status")
.withSection(section("regression", "Regression", "trending_down", Tier.T1, "id", "targetId", "metric", "strategy", "baselineValue",
"currentValue", "delta", "severity", "status"))
.withSection(section("details", "Details", "info", Tier.T2, "source", "contentTable", "contentId", "baselineRef", "currentRef",
"budgetId", "causeChangeEventId", "detectedAt", "confirmedAt", "resolvedAt", "resolutionNote"))
"budgetId", "causeChangeEventId", "detectedAt", "confirmedAt", "resolvedAt", "resolutionNote", "firstRunId", "checkpointRunId", "lighthouseMajor", "failedAudits"))
.withSection(dates()));
}
@@ -540,12 +563,15 @@ public final class WebPerfMetaData
f("toValue", QFieldType.STRING, "To").withMaxLength(200),
f("observedAt", QFieldType.DATE_TIME, "Observed"),
f("source", QFieldType.STRING, "Source").withPossibleValueSourceName(config.applyPrefix("change_source")).withMaxLength(20),
f("note", QFieldType.STRING, "Note").withMaxLength(2000)));
f("note", QFieldType.STRING, "Note").withMaxLength(2000),
f("lastSeenAt", QFieldType.DATE_TIME, "Last Observed"),
f("eventKey", QFieldType.STRING, "Source Event Key").withMaxLength(64),
f("siteKey", QFieldType.STRING, "Site").withMaxLength(200)));
return (table(WebPerfSchema.CHANGE_EVENT, "Site Changes", "change_circle", fields, false, Set.of())
.withRecordLabelFormatAndFields("%s %s", "kind", "toValue")
.withSection(section("change", "Change", "change_circle", Tier.T1, "id", "kind", "component", "fromValue", "toValue", "observedAt",
"source", "note"))
"source", "note", "lastSeenAt", "eventKey", "siteKey"))
.withSection(dates()));
}
@@ -558,6 +584,7 @@ public final class WebPerfMetaData
{
List<QFieldMetaData> fields = withCommon(List.of(
f("changeEventId", QFieldType.LONG, "Change").withPossibleValueSourceName(config.applyPrefix(WebPerfSchema.CHANGE_EVENT)),
f("targetId", QFieldType.LONG, "Page").withPossibleValueSourceName(config.applyPrefix(WebPerfSchema.TARGET)),
f("scope", QFieldType.STRING, "Scope").withPossibleValueSourceName(config.applyPrefix("comparison_scope")).withMaxLength(20),
f("source", QFieldType.STRING, "Source").withPossibleValueSourceName(config.applyPrefix("comparison_source")).withMaxLength(10),
f("strategy", QFieldType.STRING, "Strategy or Form Factor").withPossibleValueSourceName(config.applyPrefix("strategy_or_form_factor")).withMaxLength(10),
@@ -574,7 +601,7 @@ public final class WebPerfMetaData
return (table(WebPerfSchema.COMPARISON, "Before and After", "compare_arrows", fields, false, Set.of())
.withRecordLabelFormatAndFields("%s %s", "metric", "segment")
.withSection(section("comparison", "Comparison", "compare_arrows", Tier.T1, "id", "changeEventId", "metric", "segment", "beforeValue",
.withSection(section("comparison", "Comparison", "compare_arrows", Tier.T1, "id", "changeEventId", "targetId", "metric", "segment", "beforeValue",
"afterValue", "delta", "deltaPct"))
.withSection(section("details", "Details", "info", Tier.T2, "scope", "source", "strategy", "beforeN", "beforeWindow", "afterN",
"afterWindow"))
src/main/java/com/kingsrook/qbits/webperf/processes/RegressionSteps.java
+276
View File
@@ -0,0 +1,276 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.webperf.processes;
import java.sql.SQLException;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qbits.webperf.WebPerfRuntime;
import com.kingsrook.qbits.webperf.service.WebPerfComparisonService;
import com.kingsrook.qbits.webperf.service.WebPerfEvaluationService;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.actions.processes.Status;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractLoadStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractTransformStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ProcessSummaryProviderInterface;
/*******************************************************************************
** Standard streamed Review/Result actions for regression status and comparisons.
*******************************************************************************/
public final class RegressionSteps
{
public static final String NOTE = "resolutionNote";
/*******************************************************************************
** Steps only.
*******************************************************************************/
private RegressionSteps()
{
}
/*******************************************************************************
** Review open records; closed records are explicitly skipped.
*******************************************************************************/
public static class Transform extends AbstractTransformStep implements ProcessSummaryProviderInterface
{
private ProcessSummaryLine change;
private final ProcessSummaryLine skip = Lines.line(Status.INFO, "closed regression will be skipped", "closed regressions will be skipped",
"closed regression was skipped", "closed regressions were skipped");
/*******************************************************************************
** Validate the note and classify this page without writing anything.
*******************************************************************************/
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
String status = input.getValueString(WebPerfProcesses.TRIGGER_FIELD);
String note = input.getValueString(NOTE);
if(note == null || note.isBlank() || note.length() > 2000)
{
throw (new QException("Enter a resolution note of 1 to 2000 characters"));
}
if(change == null)
{
String verb = "DISMISSED".equals(status) ? "dismissed" : "marked resolved";
change = Lines.line(Status.OK, "regression will be " + verb, "regressions will be " + verb,
"regression was " + verb, "regressions were " + verb);
}
for(QRecord record : input.getRecords())
{
if(List.of("SUSPECTED", "CONFIRMED").contains(record.getValueString("status")))
{
change.incrementCountAndAddPrimaryKey(record.getValue("id"));
output.addRecord(record);
}
else
{
skip.incrementCountAndAddPrimaryKey(record.getValue("id"));
}
}
}
/*******************************************************************************
** Review summary.
*******************************************************************************/
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean result)
{
ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
if(change != null)
{
Lines.addAll(lines, change);
}
Lines.addAll(lines, skip);
return (lines);
}
}
/*******************************************************************************
** Apply the requested state under a database lock, rechecking eligibility.
*******************************************************************************/
public static class Load extends AbstractLoadStep implements ProcessSummaryProviderInterface
{
private ProcessSummaryLine changed;
private final ProcessSummaryLine skipped = Lines.line(Status.INFO, "regression will be skipped", "regressions will be skipped",
"regression was already closed", "regressions were already closed");
/*******************************************************************************
** Apply one page and report only the rows actually changed.
*******************************************************************************/
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
String status = input.getValueString(WebPerfProcesses.TRIGGER_FIELD);
if(changed == null)
{
String verb = "DISMISSED".equals(status) ? "dismissed" : "marked resolved";
changed = Lines.line(Status.OK, "regression will be " + verb, "regressions will be " + verb,
"regression was " + verb, "regressions were " + verb);
}
WebPerfEvaluationService service = new WebPerfEvaluationService(WebPerfRuntime.forInput(input).store());
try
{
for(QRecord record : input.getRecords())
{
boolean updated = service.finishRegression(record.getValueLong("id"), status, input.getValueString(NOTE), Instant.now());
(updated ? changed : skipped).incrementCountAndAddPrimaryKey(record.getValue("id"));
}
}
catch(SQLException | IllegalArgumentException e)
{
throw (new QException("Could not update the regressions", e));
}
}
/*******************************************************************************
** Result summary.
*******************************************************************************/
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean result)
{
ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
if(changed != null)
{
Lines.addAll(lines, changed);
}
Lines.addAll(lines, skipped);
if(getTransformStep() instanceof Transform transform)
{
Lines.addAll(lines, transform.skip);
}
return (lines);
}
}
/*******************************************************************************
** Review comparisons using recorded lab history.
*******************************************************************************/
public static class CompareTransform extends AbstractTransformStep implements ProcessSummaryProviderInterface
{
private final ProcessSummaryLine changes = Lines.line(Status.OK, "site change will be compared using recorded lab samples",
"site changes will be compared using recorded lab samples", "site change was compared", "site changes were compared");
/*******************************************************************************
** Pass through a bounded page of selected changes.
*******************************************************************************/
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
for(QRecord record : input.getRecords())
{
changes.incrementCountAndAddPrimaryKey(record.getValue("id"));
output.addRecord(record);
}
}
/*******************************************************************************
** Review summary.
*******************************************************************************/
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean result)
{
ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
Lines.addAll(lines, changes);
return (lines);
}
}
/*******************************************************************************
** Write comparisons when both sides have three comparable measurements.
*******************************************************************************/
public static class CompareLoad extends AbstractLoadStep implements ProcessSummaryProviderInterface
{
private final ProcessSummaryLine compared = Lines.line(Status.OK, "comparison will be recorded", "comparisons will be recorded",
"comparison was recorded", "comparisons were recorded");
private final ProcessSummaryLine insufficient = Lines.line(Status.INFO, "change needs more samples", "changes need more samples",
"change had fewer than three comparable samples on each side", "changes had fewer than three comparable samples on each side");
/*******************************************************************************
** Compute and persist this page of selected changes.
*******************************************************************************/
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
try
{
WebPerfComparisonService service = new WebPerfComparisonService(WebPerfRuntime.forInput(input).store());
for(QRecord record : input.getRecords())
{
int count = service.compare(record.getValueLong("id"));
compared.incrementCount(count);
if(count == 0)
{
insufficient.incrementCountAndAddPrimaryKey(record.getValue("id"));
}
}
}
catch(SQLException e)
{
throw (new QException("Could not compare the site changes", e));
}
}
/*******************************************************************************
** Result summary, including an honest insufficient-history line.
*******************************************************************************/
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean result)
{
ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
Lines.addAll(lines, compared, insufficient);
return (lines);
}
}
}
src/main/java/com/kingsrook/qbits/webperf/processes/RetryRunsSteps.java
+17 -2
View File
@@ -144,8 +144,23 @@ public final class RetryRunsSteps
String batch = java.util.UUID.randomUUID().toString();
for(QRecord record : input.getRecords())
{
requests.add(new WebPerfService.RunRequest(record.getValueLong("targetId"), record.getValueString("url"), record.getValueString("strategy"),
"MANUAL", "retry of run " + record.getValue("id"), batch, Instant.now()));
if(List.of("CONFIRM", "CHANGE_COMPARE").contains(record.getValueString("triggerType")))
{
try
{
String status = service.retrySample(record.getValueLong("id"), Instant.now());
("QUEUED".equals(status) ? queued : "MERGED".equals(status) ? merged : capped).incrementCount(1);
}
catch(SQLException e)
{
throw (new QException("Could not retry the measurement sample", e));
}
}
else
{
requests.add(new WebPerfService.RunRequest(record.getValueLong("targetId"), record.getValueString("url"), record.getValueString("strategy"),
"MANUAL", "retry of run " + record.getValue("id"), batch, Instant.now()));
}
}
try
{
src/main/java/com/kingsrook/qbits/webperf/processes/ScheduledSteps.java
+25
View File
@@ -25,6 +25,7 @@ import java.time.Instant;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.webperf.WebPerfRuntime;
import com.kingsrook.qbits.webperf.service.WebPerfChangeService;
import com.kingsrook.qbits.webperf.service.WebPerfService;
import com.kingsrook.qqq.backend.core.actions.processes.BackendStep;
import com.kingsrook.qqq.backend.core.exceptions.QException;
@@ -54,6 +55,30 @@ public final class ScheduledSteps
/*******************************************************************************
** Poll deploy probes and external change sources on the existing scheduler.
*******************************************************************************/
public static class ProbeChanges implements BackendStep
{
/*******************************************************************************
** Store changes and queue the affected pages.
*******************************************************************************/
@Override
public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
try
{
new WebPerfChangeService(WebPerfRuntime.forInput(input).store()).poll(Instant.now());
}
catch(Exception e)
{
throw (new QException("Checking site changes failed", e));
}
}
}
/*******************************************************************************
** Every minute: run due queued tests.
*******************************************************************************/
src/main/java/com/kingsrook/qbits/webperf/processes/WebPerfProcesses.java
+28
View File
@@ -37,6 +37,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QFunctionInputMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.QScheduleMetaData;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractExtractStep;
@@ -63,6 +64,10 @@ import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwith
*******************************************************************************/
public final class WebPerfProcesses
{
public static final String DISMISS_REGRESSIONS = "DismissRegressions";
public static final String RESOLVE_REGRESSIONS = "ResolveRegressions";
public static final String COMPARE_CHANGE = "CompareChange";
public static final String PROBE_DEPLOYS = "ProbeDeploys";
public static final String TEST_PAGE_SPEED = "TestPageSpeed";
public static final String RUN_SITE_AUDIT = "RunSiteAudit";
public static final String SYNC_TARGETS = "SyncPages";
@@ -146,6 +151,29 @@ public final class WebPerfProcesses
processes.add(scheduled(config, SCHEDULED_FIELD, "Refresh Google field data (scheduled)", ScheduledSteps.FieldData.class,
new QScheduleMetaData().withCronExpression(FIELD_CRON).withCronTimeZoneId("UTC")));
}
String regressions = config.applyPrefix(WebPerfSchema.REGRESSION);
for(String status : List.of("DISMISSED", "RESOLVED"))
{
boolean dismiss = "DISMISSED".equals(status);
QProcessMetaData process = etl(config, dismiss ? DISMISS_REGRESSIONS : RESOLVE_REGRESSIONS, dismiss ? "Dismiss" : "Mark resolved", "done",
regressions, regressions, ExtractViaQueryStep.class, RegressionSteps.Transform.class, RegressionSteps.Load.class, status, "manage", 1,
"Open regressions will be updated. Closed regressions will be skipped.");
process.withStep(0, new QFrontendStepMetaData().withName("note").withLabel("Resolution note")
.withFormField(new QFieldMetaData(RegressionSteps.NOTE, QFieldType.STRING).withLabel("Resolution note").withIsRequired(true).withMaxLength(2000)));
processes.add(process);
}
if(config.isOn(WebPerfFeature.CDN_COMPARE))
{
String changes = config.applyPrefix(WebPerfSchema.CHANGE_EVENT);
processes.add(etl(config, COMPARE_CHANGE, "Compare before and after", "compare", changes, config.applyPrefix(WebPerfSchema.COMPARISON),
ExtractViaQueryStep.class, RegressionSteps.CompareTransform.class, RegressionSteps.CompareLoad.class, null, "view", 1,
"Lab comparisons require three successful samples on each side, on the same Lighthouse major version."));
}
if(config.isOn(WebPerfFeature.DEPLOY_PROBES) || !config.getChangeEventSources().isEmpty())
{
processes.add(scheduled(config, PROBE_DEPLOYS, "Check site changes", ScheduledSteps.ProbeChanges.class,
new QScheduleMetaData().withRepeatSeconds(300).withInitialDelaySeconds(1)));
}
return (processes);
}
src/main/java/com/kingsrook/qbits/webperf/service/WebPerfChangeService.java
+246
View File
@@ -0,0 +1,246 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.webperf.service;
import java.net.URI;
import java.nio.charset.StandardCharsets;
import java.sql.SQLException;
import java.time.Instant;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import com.kingsrook.qbits.webperf.WebPerfFeature;
import com.kingsrook.qbits.webperf.WebPerfQBitConfig;
import com.kingsrook.qbits.webperf.WebPerfRuntime;
import com.kingsrook.qbits.webperf.WebPerfSchema;
import com.kingsrook.qbits.webperf.spi.ChangeEventSource;
import com.kingsrook.qbits.webperf.spi.DeployProbe;
import com.kingsrook.qbits.webperf.store.WebPerfStore;
import org.json.JSONArray;
/*******************************************************************************
** Records source changes and attributes runs to the state observed at start.
** All identities and queue writes are durable; polling can safely repeat.
*******************************************************************************/
public class WebPerfChangeService
{
private final WebPerfStore store;
private final WebPerfQBitConfig config;
/*******************************************************************************
** Changes for one installation.
*******************************************************************************/
public WebPerfChangeService(WebPerfStore store)
{
this.store = store;
this.config = store.getConfig();
}
/*******************************************************************************
** Poll configured sources. Failed probes propagate so the schedule retries.
*******************************************************************************/
public void poll(Instant now) throws Exception
{
if(config.isOn(WebPerfFeature.DEPLOY_PROBES))
{
for(DeployProbe probe : config.getDeployProbes())
{
String build = probe.currentBuildId();
if(build != null && !build.isBlank())
{
String kind = "BACKEND".equalsIgnoreCase(probe.component()) ? "BACKEND_DEPLOY" : "FRONTEND_DEPLOY";
Map<String, Object> previous = latest(kind, probe.component(), now);
record(new ChangeEventSource.Change(kind, probe.component(), previous == null ? null : (String) previous.get("to_value"), build, now),
"PROBE", probe.component() + ":" + build, true);
}
}
}
for(int i = 0; i < config.getChangeEventSources().size(); i++)
{
String sourceKey = "source:" + i;
List<Map<String, Object>> last = store.query("SELECT MAX(observed_at) AS observed_at FROM {change_event} WHERE site_key = ? AND note = ?",
siteKey(), sourceKey);
Instant since = (Instant) last.get(0).get("observed_at");
for(ChangeEventSource.Change change : config.getChangeEventSources().get(i).changesSince(since == null ? Instant.EPOCH : since))
{
if(!"CDN_STAGE".equals(change.kind()) || config.isOn(WebPerfFeature.CDN_COMPARE))
{
record(change, "CDN_STAGE".equals(change.kind()) ? "QBIT_CDN" : "WEBHOOK", sourceKey, true);
}
}
}
}
/*******************************************************************************
** Insert once per build or source event and queue affected pages atomically.
** The source SPI has no event id, so its complete tuple is its identity.
*******************************************************************************/
public long record(ChangeEventSource.Change change, String source, String sourceKey, boolean queue) throws SQLException
{
String identity = new JSONArray(List.of(siteKey(), source, sourceKey, change.kind(), change.component(), change.toValue(),
"PROBE".equals(source) ? "" : change.observedAt().toString(), "PROBE".equals(source) ? "" : String.valueOf(change.fromValue()))).toString();
String key = WebPerfService.sha256(identity.getBytes(StandardCharsets.UTF_8));
try
{
return (store.transaction(tx ->
{
List<Map<String, Object>> existing = tx.query("SELECT id FROM {change_event} WHERE event_key = ?", key);
if(!existing.isEmpty())
{
if("PROBE".equals(source))
{
tx.update("UPDATE {change_event} SET last_seen_at = ? WHERE id = ? AND (last_seen_at IS NULL OR last_seen_at < ?)",
change.observedAt(), existing.get(0).get("id"), change.observedAt());
}
return (WebPerfStore.asLong(existing.get(0).get("id")));
}
Map<String, Object> values = new LinkedHashMap<>();
values.put("kind", change.kind());
values.put("component", change.component());
values.put("from_value", change.fromValue());
values.put("to_value", change.toValue());
values.put("observed_at", change.observedAt());
values.put("last_seen_at", change.observedAt());
values.put("source", source);
values.put("note", sourceKey);
values.put("event_key", key);
values.put("site_key", siteKey());
long id = tx.insert(WebPerfSchema.CHANGE_EVENT, values);
if(queue && config.isOn(WebPerfFeature.LAB))
{
queueChange(tx, id, change);
}
return (id);
}));
}
catch(SQLException e)
{
/////////////////////////////////////////////////////////////////////
// A competing poll may have committed this exact identity first. //
/////////////////////////////////////////////////////////////////////
if("23505".equals(e.getSQLState()))
{
List<Map<String, Object>> existing = store.query("SELECT id FROM {change_event} WHERE event_key = ?", key);
if(!existing.isEmpty())
{
return (WebPerfStore.asLong(existing.get(0).get("id")));
}
}
throw (e);
}
}
/*******************************************************************************
** Page through targets; CDN sampling bypasses the ordinary queue merge.
*******************************************************************************/
private void queueChange(WebPerfStore tx, long eventId, ChangeEventSource.Change change) throws SQLException
{
WebPerfService service = new WebPerfService(new WebPerfRuntime(config), tx);
boolean cdn = "CDN_STAGE".equals(change.kind());
boolean deploy = change.kind().endsWith("DEPLOY");
long after = 0;
while(true)
{
List<Map<String, Object>> pages = tx.query("SELECT id, url FROM {target} WHERE site_key = ? AND enabled = TRUE AND id > ?"
+ (cdn || deploy ? " AND tier = 'KEY'" : "") + " ORDER BY id FETCH FIRST 100 ROWS ONLY", siteKey(), after);
if(pages.isEmpty())
{
return;
}
for(Map<String, Object> page : pages)
{
after = WebPerfStore.asLong(page.get("id"));
for(String strategy : config.getStrategies())
{
if(cdn && ("HEADERS_ONLY".equals(change.toValue()) || "LIVE".equals(change.toValue())))
{
for(int sample = 0; sample < 3; sample++)
{
service.enqueueSample(page, strategy, "CHANGE_COMPARE", "change:" + eventId + ":1:" + sample,
change.observedAt().plusSeconds("LIVE".equals(change.toValue()) ? 3600 : 0));
if("LIVE".equals(change.toValue()))
{
service.enqueueSample(page, strategy, "CHANGE_COMPARE", "change:" + eventId + ":24:" + sample, change.observedAt().plusSeconds(86400));
}
}
}
else
{
Instant due = change.observedAt().plusSeconds(deploy ? config.getDeploySettleSeconds() : config.getPublishSettleSeconds());
service.enqueue(List.of(new WebPerfService.RunRequest(after, (String) page.get("url"), strategy,
deploy ? "DEPLOY" : "PUBLISH", "change:" + eventId, null, due)), change.observedAt());
}
}
}
}
}
/*******************************************************************************
** Latest observed state at a time, optionally for a specific component.
*******************************************************************************/
public Map<String, Object> latest(String kind, String component, Instant at) throws SQLException
{
List<Map<String, Object>> rows = component == null
? store.query("SELECT * FROM {change_event} WHERE site_key = ? AND kind = ? AND observed_at <= ? ORDER BY last_seen_at DESC, id DESC FETCH FIRST 1 ROWS ONLY",
siteKey(), kind, at)
: store.query("SELECT * FROM {change_event} WHERE site_key = ? AND kind = ? AND component = ? AND observed_at <= ? ORDER BY last_seen_at DESC, id DESC FETCH FIRST 1 ROWS ONLY",
siteKey(), kind, component, at);
return (rows.isEmpty() ? null : rows.get(0));
}
/*******************************************************************************
** Attribute at execution, not queue time (a deployment may happen meanwhile).
*******************************************************************************/
public void attribute(long runId, Instant now) throws SQLException
{
Map<String, Object> build = latest("FRONTEND_DEPLOY", null, now);
Map<String, Object> cdn = latest("CDN_STAGE", URI.create(config.getSiteBaseUrl()).getHost(), now);
List<Map<String, Object>> cause = store.query("SELECT id FROM {change_event} WHERE site_key = ? AND observed_at <= ? ORDER BY observed_at DESC, id DESC FETCH FIRST 1 ROWS ONLY",
siteKey(), now);
store.update("UPDATE {lab_run} SET deployment_id = ?, cdn_stage = ?, change_event_id = ? WHERE id = ?",
build == null ? null : build.get("to_value"), cdn == null ? null : cdn.get("to_value"), cause.isEmpty() ? null : cause.get(0).get("id"), runId);
}
/*******************************************************************************
** Same site identity as the tested pages.
*******************************************************************************/
String siteKey()
{
return ("host:" + URI.create(config.getSiteBaseUrl()).getHost().toLowerCase(Locale.ROOT));
}
}
src/main/java/com/kingsrook/qbits/webperf/service/WebPerfComparisonService.java
+188
View File
@@ -0,0 +1,188 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.webperf.service;
import java.sql.SQLException;
import java.time.Duration;
import java.time.Instant;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.webperf.WebPerfSchema;
import com.kingsrook.qbits.webperf.store.WebPerfStore;
/*******************************************************************************
** Reproducible before/after lab medians, with counts and actual sample windows.
** Three successful samples on each side and one Lighthouse major are required.
*******************************************************************************/
public class WebPerfComparisonService
{
private final WebPerfStore store;
/*******************************************************************************
** Comparisons for one installation.
*******************************************************************************/
public WebPerfComparisonService(WebPerfStore store)
{
this.store = store;
}
/*******************************************************************************
** Recompute the run's attributed change after its results have been stored.
*******************************************************************************/
public void compareForRun(Map<String, Object> run) throws SQLException
{
if(run.get("change_event_id") != null)
{
List<Map<String, Object>> events = store.query("SELECT * FROM {change_event} WHERE id = ?", run.get("change_event_id"));
if(!events.isEmpty())
{
compareTarget(events.get(0), WebPerfStore.asLong(run.get("target_id")), (String) run.get("strategy"));
}
}
}
/*******************************************************************************
** Record action: page through enabled targets and compare existing evidence.
** Sparse history stays absent instead of being filled with estimated values.
*******************************************************************************/
public int compare(long eventId) throws SQLException
{
return (store.transaction(tx ->
{
Map<String, Object> event = tx.query("SELECT * FROM {change_event} WHERE id = ? FOR UPDATE", eventId).get(0);
WebPerfComparisonService service = new WebPerfComparisonService(tx);
int count = 0;
long after = 0;
while(true)
{
List<Map<String, Object>> pages = tx.query("SELECT id FROM {target} WHERE enabled = TRUE AND site_key = ? AND id > ? ORDER BY id FETCH FIRST 100 ROWS ONLY",
event.get("site_key"), after);
if(pages.isEmpty())
{
return (count);
}
for(Map<String, Object> page : pages)
{
after = WebPerfStore.asLong(page.get("id"));
for(String strategy : store.getConfig().getStrategies())
{
count += service.compareTarget(event, after, strategy);
}
}
}
}));
}
/*******************************************************************************
** Serialize comparison upserts on the change row, shared by all callers.
*******************************************************************************/
private int compareTarget(Map<String, Object> event, long targetId, String strategy) throws SQLException
{
store.query("SELECT id FROM {change_event} WHERE id = ? FOR UPDATE", event.get("id"));
Instant at = (Instant) event.get("observed_at");
List<Map<String, Object>> after = store.query("SELECT * FROM {lab_run} WHERE target_id = ? AND strategy = ? AND status = 'SUCCEEDED'"
+ " AND started_at >= ? AND change_event_id = ? ORDER BY completed_at DESC, id DESC FETCH FIRST 3 ROWS ONLY", targetId, strategy, at, event.get("id"));
if(after.size() != 3)
{
return (0);
}
String major = WebPerfEvaluationService.major(after.get(0));
if(after.stream().anyMatch(row -> !major.equals(WebPerfEvaluationService.major(row))))
{
return (0);
}
List<Map<String, Object>> before = store.query("SELECT * FROM {lab_run} WHERE target_id = ? AND strategy = ? AND status = 'SUCCEEDED'"
+ " AND completed_at < ? AND completed_at >= ? AND (lighthouse_version = ? OR lighthouse_version LIKE ?)"
+ " ORDER BY completed_at DESC, id DESC FETCH FIRST 3 ROWS ONLY", targetId, strategy, at, at.minus(Duration.ofDays(30)), major, major + ".%");
if(before.size() != 3)
{
return (0);
}
int count = 0;
for(String metric : List.of("PERFORMANCE_SCORE", "LCP", "FCP", "TTFB"))
{
String column = WebPerfEvaluationService.LAB_METRICS.get(metric);
if(before.stream().anyMatch(row -> row.get(column) == null) || after.stream().anyMatch(row -> row.get(column) == null))
{
continue;
}
double beforeValue = WebPerfEvaluationService.median(before, column);
double afterValue = WebPerfEvaluationService.median(after, column);
double delta = afterValue - beforeValue;
Double percentage = beforeValue == 0 ? null : delta * 100 / beforeValue;
if(percentage != null && Math.abs(percentage) >= 1000000)
{
percentage = null;
}
List<Map<String, Object>> existing = store.query("SELECT id FROM {comparison} WHERE change_event_id = ? AND target_id = ? AND source = 'LAB'"
+ " AND strategy = ? AND metric = ? AND segment = 'ALL'", event.get("id"), targetId, strategy, metric);
if(existing.isEmpty())
{
Map<String, Object> row = new LinkedHashMap<>();
row.put("change_event_id", event.get("id"));
row.put("target_id", targetId);
row.put("scope", "TARGET");
row.put("source", "LAB");
row.put("strategy", strategy);
row.put("metric", metric);
row.put("segment", "ALL");
row.put("before_value", beforeValue);
row.put("before_n", 3);
row.put("before_window", window(before));
row.put("after_value", afterValue);
row.put("after_n", 3);
row.put("after_window", window(after));
row.put("delta", delta);
row.put("delta_pct", percentage);
store.insert(WebPerfSchema.COMPARISON, row);
}
else
{
store.update("UPDATE {comparison} SET before_value = ?, before_n = 3, before_window = ?, after_value = ?, after_n = 3, after_window = ?,"
+ " delta = ?, delta_pct = ?, modify_date = ? WHERE id = ?", beforeValue, window(before), afterValue, window(after), delta, percentage,
Instant.now(), existing.get(0).get("id"));
}
count++;
}
return (count);
}
/*******************************************************************************
** The actual sample times, not an implied continuous measurement window.
*******************************************************************************/
private static String window(List<Map<String, Object>> samples)
{
return (samples.get(samples.size() - 1).get("completed_at") + " / " + samples.get(0).get("completed_at"));
}
}
src/main/java/com/kingsrook/qbits/webperf/service/WebPerfEvaluationService.java
+611
View File
@@ -0,0 +1,611 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.webperf.service;
import java.sql.SQLException;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Objects;
import com.kingsrook.qbits.webperf.WebPerfFeature;
import com.kingsrook.qbits.webperf.WebPerfRuntime;
import com.kingsrook.qbits.webperf.WebPerfSchema;
import com.kingsrook.qbits.webperf.spi.ChangeEventSource;
import com.kingsrook.qbits.webperf.store.WebPerfStore;
import com.kingsrook.qqq.backend.core.alerts.AlertRecordRef;
import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
/*******************************************************************************
** Budget breaches and confirmed lab regressions, backed by ordinary records.
** Target row locks serialize evaluation across workers. Baselines and samples
** are bounded, and evaluation progress commits with the resulting records.
*******************************************************************************/
public class WebPerfEvaluationService
{
public static final String ALERT_TYPE = "webperf.regression";
public static final Map<String, String> LAB_METRICS = Map.of(
"PERFORMANCE_SCORE", "score_performance", "ACCESSIBILITY_SCORE", "score_accessibility",
"BEST_PRACTICES_SCORE", "score_best_practices", "SEO_SCORE", "score_seo",
"LCP", "lcp_ms", "FCP", "fcp_ms", "CLS", "cls", "TBT", "tbt_ms", "TTFB", "ttfb_ms");
private final WebPerfStore store;
/*******************************************************************************
** Evaluation for one installation.
*******************************************************************************/
public WebPerfEvaluationService(WebPerfStore store)
{
this.store = store;
}
/*******************************************************************************
** Bounded work per worker tick, including results left by an interrupted tick.
*******************************************************************************/
public void evaluatePending() throws SQLException
{
for(Map<String, Object> run : store.query("SELECT id FROM {lab_run} WHERE status = 'SUCCEEDED' AND evaluated_at IS NULL ORDER BY completed_at, id FETCH FIRST 100 ROWS ONLY"))
{
evaluateLab(WebPerfStore.asLong(run.get("id")));
}
for(Map<String, Object> field : store.query("SELECT id FROM {field_snapshot} WHERE evaluated_at IS NULL ORDER BY id FETCH FIRST 100 ROWS ONLY"))
{
evaluateField(WebPerfStore.asLong(field.get("id")));
}
}
/*******************************************************************************
** Evaluate a stored successful result once; publish alerts after commit.
*******************************************************************************/
public void evaluateLab(long runId) throws SQLException
{
List<Long> changed = store.transaction(tx ->
{
List<Map<String, Object>> runs = tx.query("SELECT * FROM {lab_run} WHERE id = ? AND status = 'SUCCEEDED'", runId);
if(runs.isEmpty())
{
return (List.of());
}
Map<String, Object> run = runs.get(0);
List<Map<String, Object>> targets = tx.query("SELECT * FROM {target} WHERE id = ? FOR UPDATE", run.get("target_id"));
if(targets.isEmpty() || !tx.query("SELECT id FROM {lab_run} WHERE id = ? AND evaluated_at IS NOT NULL", runId).isEmpty())
{
return (List.of());
}
Map<String, Object> target = targets.get(0);
WebPerfEvaluationService evaluator = new WebPerfEvaluationService(tx);
List<Long> transitions = new ArrayList<>();
evaluator.versionChange(run, transitions);
List<Map<String, Object>> baseline = evaluator.baseline(run);
for(Map.Entry<String, String> metric : LAB_METRICS.entrySet())
{
if(run.get(metric.getValue()) != null)
{
evaluator.evaluateMetric(target, run, metric.getKey(), metric.getValue(), baseline, transitions);
}
}
evaluator.updateCount(target.get("id"));
if(store.getConfig().isOn(WebPerfFeature.CDN_COMPARE))
{
new WebPerfComparisonService(tx).compareForRun(run);
}
tx.update("UPDATE {lab_run} SET evaluated_at = ? WHERE id = ?", Instant.now(), runId);
return (transitions);
});
changed.forEach(this::sendAlert);
}
/*******************************************************************************
** CrUX URL budgets use real p75 values, never NO_DATA, errors or origin
** fallback as page measurements. A new collection period can resolve a breach.
*******************************************************************************/
public void evaluateField(long snapshotId) throws SQLException
{
List<Long> changed = store.transaction(tx ->
{
Map<String, Object> field = tx.query("SELECT * FROM {field_snapshot} WHERE id = ? FOR UPDATE", snapshotId).get(0);
List<Long> transitions = new ArrayList<>();
if(field.get("evaluated_at") != null)
{
return (transitions);
}
if("OK".equals(field.get("status")) && "URL".equals(field.get("scope")))
{
List<Map<String, Object>> targets = tx.query("SELECT * FROM {target} WHERE site_key = ? AND url = ? FOR UPDATE",
new WebPerfChangeService(tx).siteKey(), field.get("scope_key"));
WebPerfEvaluationService evaluator = new WebPerfEvaluationService(tx);
for(Map<String, Object> target : targets)
{
for(String metric : List.of("LCP", "INP", "CLS", "FCP", "TTFB"))
{
String column = "p75_" + metric.toLowerCase(Locale.ROOT) + ("CLS".equals(metric) ? "" : "_ms");
Double value = WebPerfStore.asDouble(field.get(column));
if(value == null)
{
continue;
}
String strategy = "PHONE".equals(field.get("form_factor")) ? "MOBILE" : (String) field.get("form_factor");
Map<String, Object> budget = evaluator.breachedBudget(target, "CRUX", strategy, metric, value);
List<Map<String, Object>> open = tx.query("SELECT * FROM {regression} WHERE target_id = ? AND source = 'CRUX' AND strategy = ? AND metric = ?"
+ " AND status = 'CONFIRMED' FETCH FIRST 1 ROWS ONLY", target.get("id"), field.get("form_factor"), metric);
Instant at = (Instant) field.get("last_checked_at");
if(budget != null && open.isEmpty())
{
Map<String, Object> row = new LinkedHashMap<>();
row.put("target_id", target.get("id"));
row.put("content_table", target.get("content_table"));
row.put("content_id", target.get("content_id"));
row.put("source", "CRUX");
row.put("strategy", field.get("form_factor"));
row.put("metric", metric);
row.put("current_value", value);
row.put("current_ref", "field:" + snapshotId);
row.put("budget_id", budget.get("id"));
row.put("severity", budget.get("severity"));
row.put("status", "CONFIRMED");
row.put("detected_at", at);
row.put("confirmed_at", at);
transitions.add(tx.insert(WebPerfSchema.REGRESSION, row));
}
else if(!open.isEmpty())
{
Map<String, Object> regression = open.get(0);
tx.update("UPDATE {regression} SET current_value = ?, current_ref = ?, modify_date = ? WHERE id = ?",
value, "field:" + snapshotId, at, regression.get("id"));
if(budget == null)
{
evaluator.close(regression, "RESOLVED", "Google field data is within its budget", at);
transitions.add(WebPerfStore.asLong(regression.get("id")));
}
}
}
evaluator.updateCount(target.get("id"));
}
}
tx.update("UPDATE {field_snapshot} SET evaluated_at = ? WHERE id = ?", Instant.now(), snapshotId);
return (transitions);
});
changed.forEach(this::sendAlert);
}
/*******************************************************************************
** Five eligible results in the preceding 30 days, excluding the new result.
*******************************************************************************/
private List<Map<String, Object>> baseline(Map<String, Object> run) throws SQLException
{
return (store.query("SELECT * FROM {lab_run} WHERE target_id = ? AND strategy = ? AND status = 'SUCCEEDED'"
+ " AND trigger_type IN ('SCHEDULE', 'CONFIRM', 'SITE_AUDIT') AND completed_at >= ?"
+ " AND (completed_at < ? OR (completed_at = ? AND id < ?)) AND (lighthouse_version = ? OR lighthouse_version LIKE ?)"
+ " ORDER BY completed_at DESC, id DESC FETCH FIRST 5 ROWS ONLY", run.get("target_id"), run.get("strategy"),
((Instant) run.get("completed_at")).minus(Duration.ofDays(30)), run.get("completed_at"), run.get("completed_at"), run.get("id"),
major(run), major(run) + ".%"));
}
/*******************************************************************************
** Major-version changes are markers and retire incompatible open baselines.
*******************************************************************************/
private void versionChange(Map<String, Object> run, List<Long> changed) throws SQLException
{
List<Map<String, Object>> previous = store.query("SELECT lighthouse_version FROM {lab_run} WHERE target_id = ? AND strategy = ?"
+ " AND status = 'SUCCEEDED' AND id < ? ORDER BY id DESC FETCH FIRST 1 ROWS ONLY", run.get("target_id"), run.get("strategy"), run.get("id"));
if(!previous.isEmpty() && !major(previous.get(0)).equals(major(run)))
{
new WebPerfChangeService(store).record(new ChangeEventSource.Change("LIGHTHOUSE_VERSION", "LIGHTHOUSE", major(previous.get(0)), major(run),
(Instant) run.get("completed_at")), "PROBE", "lighthouse:" + major(run), false);
}
for(Map<String, Object> open : store.query("SELECT * FROM {regression} WHERE target_id = ? AND strategy = ? AND source = 'LAB'"
+ " AND status IN ('SUSPECTED', 'CONFIRMED') AND lighthouse_major <> ?", run.get("target_id"), run.get("strategy"), major(run)))
{
close(open, "CONFIRMED".equals(open.get("status")) ? "RESOLVED" : "NOT_CONFIRMED", "Lighthouse major version changed; baseline reset",
(Instant) run.get("completed_at"));
changed.add(WebPerfStore.asLong(open.get("id")));
}
}
/*******************************************************************************
** Keep a single open row per page, source, strategy and metric. Its baseline
** stays fixed through confirmation and recovery rather than drifting.
*******************************************************************************/
private void evaluateMetric(Map<String, Object> target, Map<String, Object> run, String metric, String column,
List<Map<String, Object>> baseline, List<Long> changed) throws SQLException
{
List<Map<String, Object>> existing = store.query("SELECT * FROM {regression} WHERE target_id = ? AND strategy = ? AND source = 'LAB' AND metric = ?"
+ " AND status IN ('SUSPECTED', 'CONFIRMED') FETCH FIRST 1 ROWS ONLY", target.get("id"), run.get("strategy"), metric);
if(!existing.isEmpty())
{
advance(target, run, metric, column, existing.get(0), changed);
return;
}
Double value = WebPerfStore.asDouble(run.get(column));
Double reference = baseline.size() == 5 && baseline.stream().allMatch(row -> row.get(column) != null) ? median(baseline, column) : null;
Map<String, Object> budget = breachedBudget(target, "LAB", (String) run.get("strategy"), metric, value);
List<String> failedAudits = newFailures(run, metric);
if(!drift(metric, reference, value) && budget == null && failedAudits.isEmpty())
{
return;
}
boolean deterministic = metric.endsWith("_SCORE") && !"PERFORMANCE_SCORE".equals(metric);
Instant at = (Instant) run.get("completed_at");
Map<String, Object> row = new LinkedHashMap<>();
row.put("target_id", target.get("id"));
row.put("content_table", target.get("content_table"));
row.put("content_id", target.get("content_id"));
row.put("source", "LAB");
row.put("strategy", run.get("strategy"));
row.put("metric", metric);
row.put("baseline_value", reference);
row.put("baseline_ref", reference == null ? null : refs(baseline));
row.put("current_value", value);
row.put("current_ref", run.get("id").toString());
row.put("delta", reference == null ? null : value - reference);
row.put("budget_id", budget == null ? null : budget.get("id"));
row.put("severity", budget == null ? "WARNING" : budget.get("severity"));
row.put("status", deterministic ? "CONFIRMED" : "SUSPECTED");
row.put("detected_at", at);
row.put("confirmed_at", deterministic ? at : null);
row.put("cause_change_event_id", run.get("change_event_id"));
row.put("first_run_id", run.get("id"));
row.put("checkpoint_run_id", run.get("id"));
row.put("lighthouse_major", major(run));
row.put("failed_audits", String.join(",", failedAudits));
long id = store.insert(WebPerfSchema.REGRESSION, row);
if(deterministic)
{
changed.add(id);
}
else
{
WebPerfService service = new WebPerfService(new WebPerfRuntime(store.getConfig()), store);
for(int sample = 0; sample < 2; sample++)
{
service.enqueueSample(target, (String) run.get("strategy"), "CONFIRM", "confirm:" + run.get("id") + ":" + sample, at);
}
}
}
/*******************************************************************************
** Confirm with the original plus two explicit samples. Recover only after
** three later successful runs on the same major version.
*******************************************************************************/
private void advance(Map<String, Object> target, Map<String, Object> run, String metric, String column,
Map<String, Object> open, List<Long> changed) throws SQLException
{
boolean suspected = "SUSPECTED".equals(open.get("status"));
List<Map<String, Object>> samples = suspected
? store.query("SELECT * FROM {lab_run} WHERE target_id = ? AND strategy = ? AND status = 'SUCCEEDED' AND id <= ?"
+ " AND (id = ? OR trigger_ref IN (?, ?)) AND (lighthouse_version = ? OR lighthouse_version LIKE ?) ORDER BY id FETCH FIRST 3 ROWS ONLY",
target.get("id"), run.get("strategy"), run.get("id"), open.get("first_run_id"), "confirm:" + open.get("first_run_id") + ":0",
"confirm:" + open.get("first_run_id") + ":1", major(run), major(run) + ".%")
: store.query("SELECT * FROM {lab_run} WHERE target_id = ? AND strategy = ? AND status = 'SUCCEEDED' AND id > ? AND id <= ?"
+ " AND (lighthouse_version = ? OR lighthouse_version LIKE ?) ORDER BY id DESC FETCH FIRST 3 ROWS ONLY",
target.get("id"), run.get("strategy"), open.get("checkpoint_run_id"), run.get("id"), major(run), major(run) + ".%");
if(samples.size() < 3 || samples.stream().anyMatch(row -> row.get(column) == null))
{
return;
}
double value = median(samples, column);
Double reference = WebPerfStore.asDouble(open.get("baseline_value"));
Map<String, Object> budget = breachedBudget(target, "LAB", (String) run.get("strategy"), metric, value);
boolean breached = drift(metric, reference, value) || budget != null || auditsStillFail(open, samples);
Instant at = (Instant) run.get("completed_at");
store.update("UPDATE {regression} SET current_value = ?, current_ref = ?, delta = ?, budget_id = ?, severity = ?, modify_date = ? WHERE id = ?",
value, refs(samples), reference == null ? null : value - reference, budget == null ? null : budget.get("id"),
budget == null ? "WARNING" : budget.get("severity"), at, open.get("id"));
if(suspected && breached)
{
store.update("UPDATE {regression} SET status = 'CONFIRMED', confirmed_at = ?, checkpoint_run_id = ? WHERE id = ?", at, run.get("id"), open.get("id"));
changed.add(WebPerfStore.asLong(open.get("id")));
}
else if(!breached)
{
close(open, suspected ? "NOT_CONFIRMED" : "RESOLVED", suspected ? "Median of three did not confirm the regression" : "Recovered: median of three later runs", at);
changed.add(WebPerfStore.asLong(open.get("id")));
}
}
/*******************************************************************************
** Match enabled budgets by source, strategy and every supported scope.
** The comparator describes acceptable values; a breach is its inverse.
*******************************************************************************/
public Map<String, Object> breachedBudget(Map<String, Object> target, String source, String strategy, String metric, double value) throws SQLException
{
Map<String, Object> winner = null;
long after = 0;
while(true)
{
List<Map<String, Object>> budgets = store.query("SELECT * FROM {budget} WHERE enabled = TRUE AND source = ? AND metric = ? AND id > ?"
+ " AND (strategy IS NULL OR strategy = '' OR strategy = ?) ORDER BY id FETCH FIRST 100 ROWS ONLY", source, metric, after, strategy);
if(budgets.isEmpty())
{
return (winner);
}
for(Map<String, Object> budget : budgets)
{
after = WebPerfStore.asLong(budget.get("id"));
if(matches(target, budget) && budget.get("threshold") != null && !acceptable(value, (String) budget.get("comparator"), WebPerfStore.asDouble(budget.get("threshold"))))
{
if(winner == null || "CRITICAL".equals(budget.get("severity")))
{
winner = budget;
}
}
}
}
}
/*******************************************************************************
** Scope values are literal values, never SQL or wildcard expressions.
*******************************************************************************/
private static boolean matches(Map<String, Object> target, Map<String, Object> budget)
{
String value = (String) budget.get("scope_value");
return (switch(String.valueOf(budget.get("scope")))
{
case "SITE" -> value == null || value.isBlank() || value.equals(target.get("site_key"));
case "TIER" -> Objects.equals(value, target.get("tier"));
case "TARGET" -> Objects.equals(value, String.valueOf(target.get("id")));
case "CONTENT_TABLE" -> Objects.equals(value, target.get("content_table"));
case "PATH_PREFIX" -> value != null && target.get("path") instanceof String path && path.startsWith(value);
default -> false;
});
}
/*******************************************************************************
** Supported budget comparators. Invalid records fail visibly on evaluation.
*******************************************************************************/
static boolean acceptable(double value, String comparator, double threshold)
{
return (switch(String.valueOf(comparator))
{
case "<=", "LE" -> value <= threshold;
case "<", "LT" -> value < threshold;
case ">=", "GE" -> value >= threshold;
case ">", "GT" -> value > threshold;
case "=", "==", "EQ" -> value == threshold;
case "!=", "<>", "NE" -> value != threshold;
default -> throw new IllegalArgumentException("Unsupported performance budget comparator: " + comparator);
});
}
/*******************************************************************************
** Design section 6.4: relative and absolute LCP limits must both be met.
*******************************************************************************/
static boolean drift(String metric, Double baseline, double value)
{
if(baseline == null)
{
return (false);
}
double delta = value - baseline;
return (switch(metric)
{
case "PERFORMANCE_SCORE" -> delta <= -10;
case "ACCESSIBILITY_SCORE", "BEST_PRACTICES_SCORE", "SEO_SCORE" -> delta <= -5;
case "LCP" -> delta >= 500 && value >= baseline * 1.25;
case "CLS" -> delta >= 0.05 - 0.0000001;
case "TBT" -> delta >= 200;
default -> false;
});
}
/*******************************************************************************
** Newly failing deterministic audits, compared only within the same major.
*******************************************************************************/
private List<String> newFailures(Map<String, Object> run, String metric) throws SQLException
{
if(!metric.endsWith("_SCORE") || "PERFORMANCE_SCORE".equals(metric))
{
return (List.of());
}
String category = metric.replace("_SCORE", "");
List<Map<String, Object>> previous = store.query("SELECT id FROM {lab_run} WHERE target_id = ? AND strategy = ? AND id < ?"
+ " AND status = 'SUCCEEDED' AND (lighthouse_version = ? OR lighthouse_version LIKE ?) ORDER BY id DESC FETCH FIRST 1 ROWS ONLY",
run.get("target_id"), run.get("strategy"), run.get("id"), major(run), major(run) + ".%");
if(previous.isEmpty())
{
return (List.of());
}
return (store.query("SELECT audit_id FROM {lab_finding} f WHERE run_id = ? AND category = ? AND score < 1"
+ " AND NOT EXISTS (SELECT 1 FROM {lab_finding} old WHERE old.run_id = ? AND old.audit_id = f.audit_id AND old.score < 1)"
+ " ORDER BY audit_id FETCH FIRST 19 ROWS ONLY", run.get("id"), category, previous.get(0).get("id"))
.stream().map(row -> (String) row.get("audit_id")).toList());
}
/*******************************************************************************
** An audit-caused regression recovers when a majority no longer fail it.
*******************************************************************************/
private boolean auditsStillFail(Map<String, Object> open, List<Map<String, Object>> samples) throws SQLException
{
String audits = (String) open.get("failed_audits");
if(audits == null || audits.isBlank())
{
return (false);
}
for(String audit : audits.split(","))
{
int failed = 0;
for(Map<String, Object> sample : samples)
{
if(!store.query("SELECT id FROM {lab_finding} WHERE run_id = ? AND audit_id = ? AND score < 1 FETCH FIRST 1 ROWS ONLY", sample.get("id"), audit).isEmpty())
{
failed++;
}
}
if(failed >= 2)
{
return (true);
}
}
return (false);
}
/*******************************************************************************
** Record the terminal state without discarding evidence.
*******************************************************************************/
private void close(Map<String, Object> regression, String status, String note, Instant at) throws SQLException
{
store.update("UPDATE {regression} SET status = ?, resolved_at = ?, resolution_note = ?, modify_date = ? WHERE id = ?",
status, at, note, at, regression.get("id"));
}
/*******************************************************************************
** Record action; rechecks state under the same target lock as evaluation.
*******************************************************************************/
public boolean finishRegression(long id, String status, String note, Instant at) throws SQLException
{
if(!List.of("DISMISSED", "RESOLVED").contains(status) || note == null || note.isBlank() || note.length() > 2000)
{
throw (new IllegalArgumentException("A resolution note of 1 to 2000 characters is required"));
}
boolean changed = store.transaction(tx ->
{
List<Map<String, Object>> rows = tx.query("SELECT * FROM {regression} WHERE id = ?", id);
if(rows.isEmpty())
{
return (false);
}
Map<String, Object> row = rows.get(0);
tx.query("SELECT id FROM {target} WHERE id = ? FOR UPDATE", row.get("target_id"));
int updated = tx.update("UPDATE {regression} SET status = ?, resolved_at = ?, resolution_note = ?, modify_date = ? WHERE id = ? AND status IN ('SUSPECTED', 'CONFIRMED')",
status, at, note, at, id);
new WebPerfEvaluationService(tx).updateCount(row.get("target_id"));
return (updated == 1);
});
if(changed)
{
sendAlert(id);
}
return (changed);
}
/*******************************************************************************
** The list count includes suspected and confirmed open regressions.
*******************************************************************************/
private void updateCount(Object targetId) throws SQLException
{
store.update("UPDATE {target} SET open_regression_count = (SELECT COUNT(*) FROM {regression} WHERE target_id = ? AND status IN ('SUSPECTED', 'CONFIRMED')) WHERE id = ?",
targetId, targetId);
}
/*******************************************************************************
** Optional core producer API; without an Alerts QBit core uses its log sink.
*******************************************************************************/
private void sendAlert(long id)
{
if(!store.getConfig().isOn(WebPerfFeature.ALERTS))
{
return;
}
try
{
Map<String, Object> row = store.query("SELECT * FROM {regression} WHERE id = ?", id).get(0);
String key = store.getConfig().getTableNamePrefix() + ":" + row.get("target_id") + ":" + row.get("source") + ":" + row.get("strategy") + ":" + row.get("metric");
if("CONFIRMED".equals(row.get("status")))
{
QAlerts.raise(new AlertSignal(ALERT_TYPE).withDedupKey(key).withSeverity(AlertSeverity.valueOf((String) row.get("severity")))
.withTitle("Confirmed web performance regression").withSummary(row.get("strategy") + " " + row.get("metric"))
.withRecordRef(AlertRecordRef.of(store.getConfig().applyPrefix(WebPerfSchema.REGRESSION), id)));
}
else if(List.of("RESOLVED", "DISMISSED").contains(row.get("status")))
{
QAlerts.resolve(ALERT_TYPE, key, "Performance regression " + row.get("status").toString().toLowerCase(Locale.ROOT));
}
}
catch(SQLException e)
{
throw (new IllegalStateException("Reading a committed performance regression failed", e));
}
}
/*******************************************************************************
** Median of bounded, non-null measurement values.
*******************************************************************************/
static Double median(List<Map<String, Object>> rows, String column)
{
List<Double> values = rows.stream().map(row -> WebPerfStore.asDouble(row.get(column))).filter(Objects::nonNull).sorted().toList();
int size = values.size();
return (size == 0 ? null : size % 2 == 1 ? values.get(size / 2) : (values.get(size / 2 - 1) + values.get(size / 2)) / 2);
}
/*******************************************************************************
** Stored ids make a statistical result reproducible.
*******************************************************************************/
static String refs(List<Map<String, Object>> rows)
{
return (String.join(",", rows.stream().map(row -> row.get("id").toString()).toList()));
}
/*******************************************************************************
** Unknown versions never mix with a known Lighthouse baseline.
*******************************************************************************/
static String major(Map<String, Object> run)
{
return (String.valueOf(run.get("lighthouse_version")).split("\\.")[0]);
}
}
src/main/java/com/kingsrook/qbits/webperf/service/WebPerfService.java
+65 -7
View File
@@ -205,7 +205,6 @@ public class WebPerfService
values.put("enabled", true);
values.put("in_sitemap", true);
values.put("last_seen_in_sitemap_at", now);
values.put("open_regression_count", 0);
store.insert(WebPerfSchema.TARGET, values);
}
for(Long id : plan.seenIds())
@@ -303,10 +302,6 @@ public class WebPerfService
public QueuePlan planQueue(List<RunRequest> requests, Instant now) throws SQLException
{
Set<String> open = new HashSet<>();
for(Map<String, Object> row : store.query("SELECT target_id, strategy FROM {lab_run} WHERE status IN ('QUEUED', 'RUNNING')"))
{
open.add(row.get("target_id") + "|" + row.get("strategy"));
}
int runsToday = runsSince(startOfToday(now));
List<RunRequest> queue = new ArrayList<>();
@@ -315,7 +310,8 @@ public class WebPerfService
for(RunRequest request : requests)
{
String key = request.targetId() + "|" + request.strategy();
if(!open.add(key))
if(!open.add(key) || !store.query("SELECT id FROM {lab_run} WHERE target_id = ? AND strategy = ? AND status IN ('QUEUED', 'RUNNING') FETCH FIRST 1 ROWS ONLY",
request.targetId(), request.strategy()).isEmpty())
{
merged.add(request);
}
@@ -359,6 +355,48 @@ public class WebPerfService
/*******************************************************************************
** Queue an independent confirmation or comparison sample. Its reference is
** stable across retries; deliberate samples must not merge with each other.
*******************************************************************************/
public String enqueueSample(Map<String, Object> target, String strategy, String trigger, String reference, Instant due) throws SQLException
{
if(!store.query("SELECT id FROM {lab_run} WHERE target_id = ? AND strategy = ? AND trigger_ref = ? AND status NOT IN ('FAILED', 'RUNTIME_ERROR', 'SKIPPED_QUOTA') FETCH FIRST 1 ROWS ONLY",
target.get("id"), strategy, reference).isEmpty())
{
return ("MERGED");
}
Instant now = Instant.now();
boolean capped = runsSince(startOfToday(now)) >= config.getMaxRunsPerDay();
RunRequest request = new RunRequest(WebPerfStore.asLong(target.get("id")), (String) target.get("url"), strategy, trigger, reference, null, due);
store.insert(WebPerfSchema.LAB_RUN, runValues(request, capped ? "SKIPPED_QUOTA" : "QUEUED", now,
capped ? "Daily run cap reached; use Retry to collect this sample" : null));
return (capped ? "SKIPPED_QUOTA" : "QUEUED");
}
/*******************************************************************************
** Retry a deliberate sample without losing its confirmation/comparison slot.
** Keep the failed row as history; simultaneous retries share the target lock.
*******************************************************************************/
public String retrySample(long runId, Instant now) throws SQLException
{
return (store.transaction(tx ->
{
Map<String, Object> run = tx.query("SELECT * FROM {lab_run} WHERE id = ?", runId).get(0);
Map<String, Object> target = tx.query("SELECT id, url FROM {target} WHERE id = ? FOR UPDATE", run.get("target_id")).get(0);
if(!List.of("FAILED", "RUNTIME_ERROR", "SKIPPED_QUOTA").contains(run.get("status")))
{
return ("MERGED");
}
return (new WebPerfService(runtime, tx).enqueueSample(target, (String) run.get("strategy"), (String) run.get("trigger_type"),
(String) run.get("trigger_ref"), now));
}));
}
/*******************************************************************************
** The column values of a new run.
*******************************************************************************/
@@ -460,6 +498,7 @@ public class WebPerfService
*******************************************************************************/
public int runDue(Instant now) throws Exception
{
new WebPerfEvaluationService(store).evaluatePending();
LabRunner runner = runtime.labRunner();
if(runner == null)
{
@@ -467,7 +506,7 @@ public class WebPerfService
return (0);
}
List<Map<String, Object>> due = store.query("SELECT id FROM {lab_run} WHERE status = 'QUEUED' AND not_before <= ? ORDER BY not_before, id", now);
List<Map<String, Object>> due = store.query("SELECT id FROM {lab_run} WHERE status = 'QUEUED' AND not_before <= ? ORDER BY not_before, id FETCH FIRST " + config.getConcurrency() + " ROWS ONLY", now);
List<Long> claimed = new ArrayList<>();
for(Map<String, Object> row : due)
{
@@ -513,6 +552,7 @@ public class WebPerfService
{
executor.shutdownNow();
}
new WebPerfEvaluationService(store).evaluatePending();
return (claimed.size());
}
@@ -529,6 +569,7 @@ public class WebPerfService
{
Map<String, Object> run = store.query("SELECT id, target_id, url, strategy, attempts FROM {lab_run} WHERE id = ?", runId).get(0);
int attempts = WebPerfStore.asInt(run.get("attempts")) + 1;
new WebPerfChangeService(store).attribute(runId, Instant.now());
LabRunner.LabResponse response;
try
{
@@ -572,6 +613,21 @@ public class WebPerfService
** the compressed report, and the page's latest results.
*******************************************************************************/
private void store(long runId, Map<String, Object> run, int attempts, LabResultParser.ParsedLab parsed, LabRunner.LabResponse response) throws SQLException
{
store.transaction(tx ->
{
new WebPerfService(runtime, tx).storeResult(runId, run, attempts, parsed, response);
return (null);
});
}
/*******************************************************************************
** Commit the result, findings, report and latest values as one unit, so an
** evaluator never sees a successful run with only some of its evidence.
*******************************************************************************/
private void storeResult(long runId, Map<String, Object> run, int attempts, LabResultParser.ParsedLab parsed, LabRunner.LabResponse response) throws SQLException
{
Instant completed = response.completedAt();
String status = parsed.runtimeError() == null ? "SUCCEEDED" : "RUNTIME_ERROR";
@@ -585,6 +641,7 @@ public class WebPerfService
parsed.score("SEO"), parsed.fcpMs(), parsed.lcpMs(), parsed.tbtMs(), parsed.cls(), parsed.speedIndexMs(), parsed.lcpElement(), parsed.runWarnings(),
completed, runId);
store.update("UPDATE {lab_run} SET ttfb_ms = ? WHERE id = ?", LabResultParser.ttfb(response.rawJson()), runId);
Long targetId = WebPerfStore.asLong(run.get("target_id"));
for(LabResultParser.Finding finding : parsed.findings())
{
@@ -718,6 +775,7 @@ public class WebPerfService
}
}
}
new WebPerfEvaluationService(store).evaluatePending();
return (counts);
}
src/main/java/com/kingsrook/qbits/webperf/store/WebPerfStore.java
+84 -4
View File
@@ -74,6 +74,86 @@ public class WebPerfStore
private final WebPerfQBitConfig config;
private final ConnectionSupplier connections;
private Connection transactionConnection;
/*******************************************************************************
** Work sharing one database transaction.
*******************************************************************************/
@FunctionalInterface
public interface TransactionWork<T>
{
/*******************************************************************************
** Execute against the transaction's store.
*******************************************************************************/
T run(WebPerfStore store) throws SQLException;
}
/*******************************************************************************
** Commit related writes together; roll back every write on failure.
*******************************************************************************/
public <T> T transaction(TransactionWork<T> work) throws SQLException
{
if(transactionConnection != null)
{
return (work.run(this));
}
try(Connection connection = connections.get())
{
boolean autoCommit = connection.getAutoCommit();
connection.setAutoCommit(false);
WebPerfStore transactional = new WebPerfStore(config, connections);
transactional.transactionConnection = connection;
try
{
T result = work.run(transactional);
connection.commit();
return (result);
}
catch(SQLException | RuntimeException e)
{
connection.rollback();
throw (e);
}
finally
{
connection.setAutoCommit(autoCommit);
}
}
}
/*******************************************************************************
** Own a connection only outside an enclosing transaction.
*******************************************************************************/
private ConnectionLease lease() throws SQLException
{
return (new ConnectionLease(transactionConnection == null ? connections.get() : transactionConnection, transactionConnection == null));
}
/*******************************************************************************
** A statement's connection lifetime.
*******************************************************************************/
private record ConnectionLease(Connection connection, boolean owned) implements AutoCloseable
{
/*******************************************************************************
** Leave an enclosing transaction's connection open.
*******************************************************************************/
@Override
public void close() throws SQLException
{
if(owned)
{
connection.close();
}
}
}
@@ -119,7 +199,7 @@ public class WebPerfStore
*******************************************************************************/
public int update(String template, Object... params) throws SQLException
{
try(Connection connection = connections.get(); PreparedStatement statement = connection.prepareStatement(sql(template)))
try(ConnectionLease lease = lease(); PreparedStatement statement = lease.connection().prepareStatement(sql(template)))
{
bind(statement, params);
return (statement.executeUpdate());
@@ -145,8 +225,8 @@ public class WebPerfStore
String columns = String.join(", ", row.keySet());
String placeholders = String.join(", ", java.util.Collections.nCopies(row.size(), "?"));
try(Connection connection = connections.get();
PreparedStatement statement = connection.prepareStatement("INSERT INTO " + config.applyPrefix(table) + " (" + columns + ") VALUES (" + placeholders + ")",
try(ConnectionLease lease = lease();
PreparedStatement statement = lease.connection().prepareStatement("INSERT INTO " + config.applyPrefix(table) + " (" + columns + ") VALUES (" + placeholders + ")",
Statement.RETURN_GENERATED_KEYS))
{
bind(statement, row.values().toArray());
@@ -171,7 +251,7 @@ public class WebPerfStore
*******************************************************************************/
public List<Map<String, Object>> query(String template, Object... params) throws SQLException
{
try(Connection connection = connections.get(); PreparedStatement statement = connection.prepareStatement(sql(template)))
try(ConnectionLease lease = lease(); PreparedStatement statement = lease.connection().prepareStatement(sql(template)))
{
bind(statement, params);
List<Map<String, Object>> rows = new ArrayList<>();
src/main/resources/db/liquibase/qbit-web-performance/web_performance.changelog.sql
+19
View File
@@ -261,3 +261,22 @@ SELECT name, scope, scope_value, source, strategy, metric, comparator, threshold
UNION ALL SELECT 'Performance at least 50 (KEY pages)', 'TIER', 'KEY', 'LAB', NULL, 'PERFORMANCE_SCORE', '>=', 50, 'CRITICAL'
) seeds
WHERE NOT EXISTS (SELECT 1 FROM ${web_performance.tablePrefix}budget WHERE seeded = TRUE);
-- changeset qbit-web-performance:11
-- comment: Durable evaluation progress, event identity and comparison page identity. Existing history is preserved.
ALTER TABLE ${web_performance.tablePrefix}lab_run ADD COLUMN evaluated_at TIMESTAMP WITHOUT TIME ZONE;
ALTER TABLE ${web_performance.tablePrefix}lab_run ADD COLUMN change_event_id BIGINT;
ALTER TABLE ${web_performance.tablePrefix}regression ADD COLUMN first_run_id BIGINT;
ALTER TABLE ${web_performance.tablePrefix}regression ADD COLUMN checkpoint_run_id BIGINT;
ALTER TABLE ${web_performance.tablePrefix}regression ADD COLUMN lighthouse_major VARCHAR(20);
ALTER TABLE ${web_performance.tablePrefix}change_event ADD COLUMN event_key VARCHAR(64);
ALTER TABLE ${web_performance.tablePrefix}change_event ADD COLUMN site_key VARCHAR(200);
ALTER TABLE ${web_performance.tablePrefix}comparison ADD COLUMN target_id BIGINT;
CREATE UNIQUE INDEX uq_${web_performance.tablePrefix}change_event_key ON ${web_performance.tablePrefix}change_event (event_key);
CREATE UNIQUE INDEX uq_${web_performance.tablePrefix}comparison_page ON ${web_performance.tablePrefix}comparison (change_event_id, target_id, source, strategy, metric, segment);
CREATE INDEX idx_${web_performance.tablePrefix}lab_evaluation ON ${web_performance.tablePrefix}lab_run (evaluated_at, status, id);
ALTER TABLE ${web_performance.tablePrefix}regression ADD COLUMN failed_audits VARCHAR(2000);
ALTER TABLE ${web_performance.tablePrefix}lab_run ADD COLUMN ttfb_ms INTEGER;
ALTER TABLE ${web_performance.tablePrefix}field_snapshot ADD COLUMN evaluated_at TIMESTAMP WITHOUT TIME ZONE;
ALTER TABLE ${web_performance.tablePrefix}change_event ADD COLUMN last_seen_at TIMESTAMP WITHOUT TIME ZONE;
src/test/java/com/kingsrook/qbits/webperf/WebPerfP1Test.java
+2
View File
@@ -213,6 +213,7 @@ class WebPerfP1Test
@Test
void testWorkerSuccess() throws Exception
{
store().update("UPDATE {budget} SET enabled = FALSE");
google.respond(new FixtureServer.Response(200, FixtureServer.fixture("psi-kof22-home-mobile.json")));
WebPerfService service = service();
service.applySync(service.planSync(List.of(SITE + "/")), Instant.now());
@@ -319,6 +320,7 @@ class WebPerfP1Test
@Test
void testActions() throws Exception
{
store().update("UPDATE {budget} SET enabled = FALSE");
RunProcessOutput sync = run(WebPerfProcesses.SYNC_TARGETS);
assertThat(summary(sync)).anyMatch(l -> l.contains("3") && l.contains("added"));
assertEquals(3, store().query("SELECT id FROM {target}").size());
src/test/java/com/kingsrook/qbits/webperf/WebPerfRegressionTest.java
+828
View File
@@ -0,0 +1,828 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.webperf;
import java.io.Serializable;
import java.time.Instant;
import java.time.LocalDate;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.Executors;
import com.kingsrook.qbits.webperf.lab.LabResultParser;
import com.kingsrook.qbits.webperf.processes.RegressionSteps;
import com.kingsrook.qbits.webperf.processes.WebPerfProcesses;
import com.kingsrook.qbits.webperf.service.WebPerfChangeService;
import com.kingsrook.qbits.webperf.service.WebPerfComparisonService;
import com.kingsrook.qbits.webperf.service.WebPerfEvaluationService;
import com.kingsrook.qbits.webperf.service.WebPerfService;
import com.kingsrook.qbits.webperf.spi.ChangeEventSource;
import com.kingsrook.qbits.webperf.spi.DeployProbe;
import com.kingsrook.qbits.webperf.spi.LabRunner;
import com.kingsrook.qbits.webperf.store.WebPerfStore;
import com.kingsrook.qqq.backend.core.actions.processes.QProcessCallback;
import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction;
import com.kingsrook.qqq.backend.core.alerts.AlertResolution;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.AlertSinkInterface;
import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
import org.json.JSONObject;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
import static org.junit.jupiter.api.Assertions.assertThrows;
/*******************************************************************************
** P2 behavior on H2, with fake probes and runner and real QQQ record actions.
*******************************************************************************/
public class WebPerfRegressionTest
{
private static final Instant START = Instant.parse("2026-10-01T12:00:00Z");
private QInstance instance;
private WebPerfQBitConfig config;
private WebPerfStore store;
private WebPerfEvaluationService evaluator;
private long target;
private int sequence;
/*******************************************************************************
** Fresh tables; disable reference budgets to isolate statistical rules.
*******************************************************************************/
@BeforeEach
void setUp() throws Exception
{
instance = WebPerfTestSupport.newInstance();
config = WebPerfTestSupport.config().withStrategies(List.of("MOBILE")).withMinSpacingMillis(0).withConcurrency(2);
new WebPerfQBitProducer().withQBitConfig(config).produce(instance);
WebPerfTestSupport.applyChangelog(instance, config.getTableNamePrefix());
WebPerfTestSupport.login(instance, "web_performance.read", "web_performance.write", "web_performance.manage.hasAccess",
"web_performance.view.hasAccess", "web_performance.admin.hasAccess", "web_performance.test.hasAccess");
store = new WebPerfRuntime(config).store(instance);
evaluator = new WebPerfEvaluationService(store);
store.update("UPDATE {budget} SET enabled = FALSE");
WebPerfService service = new WebPerfRuntime(config).service();
service.applySync(service.planSync(List.of("https://www.example.com/")), START);
target = WebPerfStore.asLong(store.query("SELECT id FROM {target}").get(0).get("id"));
store.update("UPDATE {target} SET tier = 'KEY', content_table = 'page', content_id = 7 WHERE id = ?", target);
RecordingSink.signals.clear();
RecordingSink.resolutions.clear();
AlertsInstanceMetaData.ofOrWithNew(instance).withSink(new QCodeReference(RecordingSink.class));
}
/*******************************************************************************
** Release this test's context.
*******************************************************************************/
@AfterEach
void tearDown()
{
QContext.clear();
}
/*******************************************************************************
** Suspect, two confirmations, one alert, then three recovery samples.
*******************************************************************************/
@Test
void confirmsAndRecoversWithoutDuplicates() throws Exception
{
baseline();
long change = change("FRONTEND_DEPLOY", "FRONTEND", "build-2", START.plusSeconds(6));
long first = measurement(70, "SCHEDULE", "13.1", Map.of("change_event_id", change));
evaluator.evaluateLab(first);
assertThat(regression().get("status")).isEqualTo("SUSPECTED");
assertThat(store.query("SELECT * FROM {lab_run} WHERE trigger_type = 'CONFIRM'")).hasSize(2);
assertThat(RecordingSink.signals).isEmpty();
assertThat(openCount()).isEqualTo(1);
completeConfirmations(first, 72, 71);
assertThat(regression().get("status")).isEqualTo("CONFIRMED");
assertThat(WebPerfStore.asDouble(regression().get("current_value"))).isEqualTo(71);
assertThat(regression().get("baseline_ref")).isEqualTo("5,4,3,2,1");
assertThat(WebPerfStore.asLong(regression().get("cause_change_event_id"))).isEqualTo(change);
evaluator.evaluateLab(first);
assertThat(store.query("SELECT * FROM {regression}")).hasSize(1);
assertThat(RecordingSink.signals).hasSize(1);
evaluate(96);
evaluate(95);
assertThat(regression().get("status")).isEqualTo("CONFIRMED");
evaluate(94);
assertThat(regression().get("status")).isEqualTo("RESOLVED");
assertThat(openCount()).isZero();
assertThat(RecordingSink.resolutions).hasSize(1);
assertThat(RecordingSink.signals.get(0).getDedupKey()).isEqualTo(RecordingSink.resolutions.get(0).getDedupKey());
}
/*******************************************************************************
** Lab noise stays visible as not confirmed, with no alert.
*******************************************************************************/
@Test
void rejectsNoise() throws Exception
{
baseline();
long first = evaluate(70);
completeConfirmations(first, 96, 95);
assertThat(regression().get("status")).isEqualTo("NOT_CONFIRMED");
assertThat(openCount()).isZero();
assertThat(RecordingSink.signals).isEmpty();
}
/*******************************************************************************
** Baselines require five eligible recent results on the same major version.
*******************************************************************************/
@Test
void boundsBaselineHistory() throws Exception
{
for(int i = 0; i < 4; i++)
{
measurement(96, "SCHEDULE", "13.0", Map.of());
}
measurement(96, "MANUAL", "13.0", Map.of());
measurement(96, "SCHEDULE", "12.9", Map.of());
long old = measurement(96, "SCHEDULE", "13.0", Map.of());
store.update("UPDATE {lab_run} SET completed_at = ? WHERE id = ?", START.minusSeconds(31 * 86400L), old);
evaluate(70);
assertThat(store.query("SELECT * FROM {regression}")).isEmpty();
measurement(96, "SITE_AUDIT", "13.1", Map.of());
evaluate(60);
assertThat(regression().get("status")).isEqualTo("SUSPECTED");
}
/*******************************************************************************
** A major-version change records a marker and retires incompatible baselines.
*******************************************************************************/
@Test
void resetsAtMajorVersionChange() throws Exception
{
baseline();
long first = evaluate(70);
completeConfirmations(first, 71, 72);
evaluator.evaluateLab(measurement(60, "SCHEDULE", "14.0", Map.of()));
assertThat(regression().get("status")).isEqualTo("RESOLVED");
assertThat(regression().get("resolution_note")).isEqualTo("Lighthouse major version changed; baseline reset");
assertThat(store.query("SELECT * FROM {change_event} WHERE kind = 'LIGHTHOUSE_VERSION'")).hasSize(1);
assertThat(store.query("SELECT * FROM {regression}")).hasSize(1);
}
/*******************************************************************************
** Relative and absolute thresholds; deterministic categories confirm now.
*******************************************************************************/
@Test
void appliesMetricThresholds() throws Exception
{
baseline();
evaluator.evaluateLab(measurement(95, "MANUAL", "13.0", Map.of("lcp_ms", 2499, "cls", 0.049, "tbt_ms", 199)));
assertThat(store.query("SELECT * FROM {regression}")).isEmpty();
evaluator.evaluateLab(measurement(95, "MANUAL", "13.0", Map.of("lcp_ms", 2500, "cls", 0.05, "tbt_ms", 200,
"score_accessibility", 95, "score_best_practices", 95, "score_seo", 95)));
assertThat(store.query("SELECT * FROM {regression} WHERE status = 'SUSPECTED'")).hasSize(3);
assertThat(store.query("SELECT * FROM {regression} WHERE status = 'CONFIRMED'")).hasSize(3);
assertThat(store.query("SELECT * FROM {lab_run} WHERE trigger_type = 'CONFIRM'")).hasSize(2);
assertThat(RecordingSink.signals).hasSize(3);
}
/*******************************************************************************
** A newly failing audit is actionable even if the rounded score is unchanged.
*******************************************************************************/
@Test
void newlyFailingAuditConfirms() throws Exception
{
baseline();
long run = measurement(96, "MANUAL", "13.0", Map.of());
store.insert(WebPerfSchema.LAB_FINDING, Map.of("run_id", run, "target_id", target, "strategy", "MOBILE",
"category", "SEO", "audit_id", "robots-txt", "kind", "FAILED_AUDIT", "score", 0));
evaluator.evaluateLab(run);
assertThat(regression().get("status")).isEqualTo("CONFIRMED");
assertThat(regression().get("failed_audits")).isEqualTo("robots-txt");
evaluate(96);
evaluate(96);
evaluate(96);
assertThat(regression().get("status")).isEqualTo("RESOLVED");
}
/*******************************************************************************
** Every scope and comparator, with source, strategy and severity matching.
*******************************************************************************/
@Test
void matchesBudgetScopesComparatorsAndSeverity() throws Exception
{
Map<String, Object> page = store.query("SELECT * FROM {target}").get(0);
for(String scope : List.of("SITE", "TIER", "TARGET", "CONTENT_TABLE", "PATH_PREFIX"))
{
String value = switch(scope)
{
case "SITE" -> "host:www.example.com";
case "TIER" -> "KEY";
case "TARGET" -> String.valueOf(target);
case "CONTENT_TABLE" -> "page";
default -> "/";
};
long id = budget(scope, value, "LAB", "MOBILE", "LCP", "<=", 2500, "WARNING");
assertThat(evaluator.breachedBudget(page, "LAB", "MOBILE", "LCP", 2500)).isNull();
assertThat(evaluator.breachedBudget(page, "LAB", "MOBILE", "LCP", 2501)).isNotNull();
assertThat(evaluator.breachedBudget(page, "LAB", "DESKTOP", "LCP", 2501)).isNull();
assertThat(evaluator.breachedBudget(page, "CRUX", "MOBILE", "LCP", 2501)).isNull();
store.update("UPDATE {budget} SET scope_value = 'unmatched' WHERE id = ?", id);
assertThat(evaluator.breachedBudget(page, "LAB", "MOBILE", "LCP", 2501)).isNull();
store.update("UPDATE {budget} SET enabled = FALSE WHERE id = ?", id);
}
for(String comparator : List.of("<", ">", ">=", "=", "!=", "<=", "==", "<>", "LE", "GE", "LT", "GT", "EQ", "NE"))
{
long id = budget("SITE", "", "LAB", null, "LCP", comparator, 10, "WARNING");
for(double value : List.of(9.0, 10.0, 11.0))
{
boolean acceptable = switch(comparator)
{
case "<", "LT" -> value < 10;
case ">", "GT" -> value > 10;
case ">=", "GE" -> value >= 10;
case "<=", "LE" -> value <= 10;
case "=", "==", "EQ" -> value == 10;
default -> value != 10;
};
assertThat(evaluator.breachedBudget(page, "LAB", "MOBILE", "LCP", value) == null).isEqualTo(acceptable);
}
store.update("UPDATE {budget} SET enabled = FALSE WHERE id = ?", id);
}
budget("TIER", "KEY", "LAB", null, "PERFORMANCE_SCORE", ">=", 90, "WARNING");
long critical = budget("TIER", "KEY", "LAB", null, "PERFORMANCE_SCORE", ">=", 50, "CRITICAL");
long first = evaluate(40);
assertThat(WebPerfStore.asLong(regression().get("budget_id"))).isEqualTo(critical);
completeConfirmations(first, 40, 41);
assertThat(regression().get("severity")).isEqualTo("CRITICAL");
}
/*******************************************************************************
** CrUX URL budgets use real values, never absent data or origin fallbacks.
*******************************************************************************/
@Test
void evaluatesCruxUrlBudgets() throws Exception
{
budget("SITE", "", "CRUX", "MOBILE", "LCP", "<=", 2500, "WARNING");
long first = field("URL", "OK", 3000, 1);
evaluator.evaluateField(first);
evaluator.evaluateField(first);
assertThat(regression().get("source")).isEqualTo("CRUX");
assertThat(regression().get("status")).isEqualTo("CONFIRMED");
evaluator.evaluateField(field("URL", "NO_DATA", null, 2));
evaluator.evaluateField(field("ORIGIN", "OK", 1000, 3));
assertThat(regression().get("status")).isEqualTo("CONFIRMED");
evaluator.evaluateField(field("URL", "OK", 2000, 4));
assertThat(regression().get("status")).isEqualTo("RESOLVED");
assertThat(openCount()).isZero();
assertThat(RecordingSink.signals).hasSize(1);
}
/*******************************************************************************
** Poll replay, settle delays and attribution through the real queue worker.
*******************************************************************************/
@Test
void pollsAndAttributesWorkerResults() throws Exception
{
MutableProbe probe = new MutableProbe();
config.withDeployProbe(probe).withLabRunner(new FakeRunner());
ChangeEventSource.Change cdn = new ChangeEventSource.Change("CDN_STAGE", "www.example.com", "OFF", "HEADERS_ONLY", START);
ChangeEventSource.Change content = new ChangeEventSource.Change("CONTENT", "page:7", "old", "new", START.plusSeconds(1));
config.withChangeEventSource(since -> List.of(cdn, content));
WebPerfChangeService changes = new WebPerfChangeService(store);
changes.poll(START.plusSeconds(2));
changes.poll(START.plusSeconds(3));
assertThat(store.query("SELECT * FROM {change_event}")).hasSize(3);
assertThat(store.query("SELECT * FROM {lab_run} WHERE trigger_type = 'CHANGE_COMPARE'")).hasSize(3);
assertThat(store.query("SELECT * FROM {lab_run} WHERE trigger_type = 'DEPLOY'").get(0).get("not_before")).isEqualTo(START.plusSeconds(302));
new WebPerfRuntime(config).service().runDue(Instant.now());
List<Map<String, Object>> done = store.query("SELECT * FROM {lab_run} WHERE status = 'SUCCEEDED'");
assertThat(done).hasSize(2);
assertThat(done).allSatisfy(row ->
{
assertThat(row.get("deployment_id")).isEqualTo("build-1");
assertThat(row.get("cdn_stage")).isEqualTo("HEADERS_ONLY");
assertThat(row.get("evaluated_at")).isNotNull();
});
probe.build = "build-2";
changes.poll(Instant.now());
assertThat(store.query("SELECT * FROM {change_event} WHERE kind = 'FRONTEND_DEPLOY'")).hasSize(2);
probe.build = null;
changes.poll(Instant.now());
assertThat(store.query("SELECT * FROM {change_event} WHERE kind = 'FRONTEND_DEPLOY'")).hasSize(2);
}
/*******************************************************************************
** Three samples on each side create reproducible, idempotent comparisons.
*******************************************************************************/
@Test
void recordsComparisonsWithCountsAndWindows() throws Exception
{
for(int i = 0; i < 3; i++)
{
measurement(90, "CHANGE_COMPARE", "13.0", Map.of("lcp_ms", 3000, "fcp_ms", 2000, "ttfb_ms", 900));
}
long event = change("CDN_STAGE", "www.example.com", "LIVE", START.plusSeconds(4));
sequence = 4;
for(int i = 0; i < 3; i++)
{
long id = measurement(96, "CHANGE_COMPARE", "13.1", Map.of("change_event_id", event, "lcp_ms", 2000, "fcp_ms", 1500, "ttfb_ms", 600));
evaluator.evaluateLab(id);
assertThat(store.query("SELECT * FROM {comparison}")).hasSize(i == 2 ? 4 : 0);
}
assertThat(new WebPerfComparisonService(store).compare(event)).isEqualTo(4);
assertThat(store.query("SELECT * FROM {comparison}")).hasSize(4);
Map<String, Object> lcp = store.query("SELECT * FROM {comparison} WHERE metric = 'LCP'").get(0);
assertThat(WebPerfStore.asDouble(lcp.get("delta"))).isEqualTo(-1000);
assertThat(WebPerfStore.asInt(lcp.get("before_n"))).isEqualTo(3);
assertThat(WebPerfStore.asInt(lcp.get("after_n"))).isEqualTo(3);
assertThat(lcp.get("before_window").toString()).contains(START.plusSeconds(1).toString());
assertThat(runAction(WebPerfProcesses.COMPARE_CHANGE, Map.of()).getException()).isEmpty();
}
/*******************************************************************************
** Real ETL actions require a note and manage permission, and retain history.
*******************************************************************************/
@Test
void actionsRequireNotesAndPermissions() throws Exception
{
budget("SITE", "", "LAB", null, "SEO_SCORE", ">=", 100, "WARNING");
evaluator.evaluateLab(measurement(96, "MANUAL", "13.0", Map.of("score_seo", 90)));
long id = WebPerfStore.asLong(regression().get("id"));
assertThrows(com.kingsrook.qqq.backend.core.exceptions.QException.class, () -> runAction(WebPerfProcesses.DISMISS_REGRESSIONS, Map.of()));
assertThat(regression().get("status")).isEqualTo("CONFIRMED");
WebPerfTestSupport.login(instance, "web_performance.read", "web_performance.write");
assertThrows(com.kingsrook.qqq.backend.core.exceptions.QException.class, () -> runAction(WebPerfProcesses.DISMISS_REGRESSIONS, Map.of(RegressionSteps.NOTE, "Expected change")));
assertThat(regression().get("status")).isEqualTo("CONFIRMED");
WebPerfTestSupport.login(instance, "web_performance.read", "web_performance.write", "web_performance.manage.hasAccess");
RunProcessOutput result = runAction(WebPerfProcesses.DISMISS_REGRESSIONS, Map.of(RegressionSteps.NOTE, "Expected change"));
assertThat(result.getException()).isEmpty();
assertThat(result.getValues().get(StreamedETLWithFrontendProcess.FIELD_PROCESS_SUMMARY)).isNotNull();
assertThat(regression().get("status")).isEqualTo("DISMISSED");
assertThat(regression().get("resolution_note")).isEqualTo("Expected change");
assertThat(openCount()).isZero();
assertThat(evaluator.finishRegression(id, "RESOLVED", "Already closed", START)).isFalse();
evaluator.evaluateLab(measurement(96, "MANUAL", "13.0", Map.of("score_seo", 90)));
assertThat(runAction(WebPerfProcesses.RESOLVE_REGRESSIONS, Map.of(RegressionSteps.NOTE, "Fixed externally")).getException()).isEmpty();
assertThat(store.query("SELECT * FROM {regression} WHERE status = 'RESOLVED'")).hasSize(1);
}
/*******************************************************************************
** Target locking prevents concurrent duplicate incidents; failed writes roll back.
*******************************************************************************/
@Test
void serializesEvaluationAndRollsBack() throws Exception
{
baseline();
long id = measurement(70, "SCHEDULE", "13.0", Map.of());
try(var executor = Executors.newFixedThreadPool(2))
{
var first = executor.submit(() ->
{
evaluator.evaluateLab(id);
return (null);
});
var second = executor.submit(() ->
{
evaluator.evaluateLab(id);
return (null);
});
first.get();
second.get();
}
assertThat(store.query("SELECT * FROM {regression}")).hasSize(1);
assertThat(store.query("SELECT * FROM {lab_run} WHERE trigger_type = 'CONFIRM'")).hasSize(2);
assertThrows(java.sql.SQLException.class, () -> store.transaction(tx ->
{
tx.update("UPDATE {target} SET label = 'must roll back' WHERE id = ?", target);
tx.update("UPDATE no_such_table SET id = 1");
return (null);
}));
assertThat(store.query("SELECT label FROM {target}").get(0).get("label")).isNotEqualTo("must roll back");
}
/*******************************************************************************
** Core falls back to logging without an Alerts QBit; feature gating works.
*******************************************************************************/
@Test
void alertsAreOptional() throws Exception
{
AlertsInstanceMetaData.ofOrWithNew(instance).withSink(null);
long failures = QAlerts.getProducerFailureCount();
budget("SITE", "", "LAB", null, "SEO_SCORE", ">=", 100, "WARNING");
evaluator.evaluateLab(measurement(96, "MANUAL", "13.0", Map.of("score_seo", 90)));
assertThat(QAlerts.getProducerFailureCount()).isEqualTo(failures);
config.getFeatures().remove(WebPerfFeature.ALERTS);
evaluator.finishRegression(WebPerfStore.asLong(regression().get("id")), "RESOLVED", "Done", START);
assertThat(RecordingSink.signals).isEmpty();
}
/*******************************************************************************
** TTFB comes from a measured subpart and is null when not reported.
*******************************************************************************/
@Test
void parsesMeasuredTtfbOnly()
{
assertThat(LabResultParser.ttfb("{\"audits\":{}}")).isNull();
assertThat(LabResultParser.ttfb("{\"audits\":{\"lcp-phases-insight\":{\"details\":{\"items\":[{\"subpart\":\"timeToFirstByte\",\"duration\":321.4}]}}}}")).isEqualTo(321);
}
/*******************************************************************************
** Quota skips remain visible, and retry preserves each deliberate sample slot.
*******************************************************************************/
@Test
void retriesCappedConfirmationSamples() throws Exception
{
baseline();
config.withMaxRunsPerDay(1);
long first = evaluate(70);
assertThat(store.query("SELECT * FROM {lab_run} WHERE status = 'SKIPPED_QUOTA'")).hasSize(1);
config.withMaxRunsPerDay(500);
assertThat(runAction(WebPerfProcesses.RETRY_RUNS, Map.of()).getException()).isEmpty();
assertThat(store.query("SELECT * FROM {lab_run} WHERE status = 'QUEUED' AND trigger_type = 'CONFIRM'")).hasSize(2);
assertThat(runAction(WebPerfProcesses.RETRY_RUNS, Map.of()).getException()).isEmpty();
assertThat(store.query("SELECT * FROM {lab_run} WHERE status = 'QUEUED' AND trigger_type = 'CONFIRM'")).hasSize(2);
store.update("UPDATE {lab_run} SET status = 'SUCCEEDED', completed_at = ?, lighthouse_version = '13.0', score_performance = 70 WHERE status = 'QUEUED'",
START.plusSeconds(10));
evaluator.evaluatePending();
assertThat(regression().get("status")).isEqualTo("CONFIRMED");
assertThat(WebPerfStore.asLong(regression().get("first_run_id"))).isEqualTo(first);
}
/*******************************************************************************
** Existing seeded budgets are evaluated even before enough baseline exists.
*******************************************************************************/
@Test
void seededBudgetsWorkWithoutBaseline() throws Exception
{
store.update("UPDATE {budget} SET enabled = TRUE WHERE seeded = TRUE");
long first = evaluate(40);
assertThat(regression().get("status")).isEqualTo("SUSPECTED");
assertThat(regression().get("baseline_value")).isNull();
assertThat(regression().get("severity")).isEqualTo("CRITICAL");
completeConfirmations(first, 40, 40);
assertThat(regression().get("status")).isEqualTo("CONFIRMED");
}
/*******************************************************************************
** Build replays are idempotent even after a rollback to an earlier build.
*******************************************************************************/
@Test
void attributesRollbackAndQueuesBothCdnWindows() throws Exception
{
MutableProbe probe = new MutableProbe();
config.withDeployProbe(probe);
WebPerfChangeService changes = new WebPerfChangeService(store);
changes.poll(START);
probe.build = "build-2";
changes.poll(START.plusSeconds(1));
probe.build = "build-1";
changes.poll(START.plusSeconds(2));
long run = measurement(96, "MANUAL", "13.0", Map.of());
changes.attribute(run, START.plusSeconds(3));
assertThat(store.query("SELECT deployment_id FROM {lab_run} WHERE id = ?", run).get(0).get("deployment_id")).isEqualTo("build-1");
assertThat(store.query("SELECT * FROM {change_event}")).hasSize(2);
ChangeEventSource.Change cdn = new ChangeEventSource.Change("CDN_STAGE", "www.example.com", "HEADERS_ONLY", "LIVE", START.plusSeconds(4));
config.withChangeEventSource(since -> List.of(cdn));
changes.poll(START.plusSeconds(4));
changes.poll(START.plusSeconds(5));
assertThat(store.query("SELECT * FROM {lab_run} WHERE trigger_type = 'CHANGE_COMPARE'")).hasSize(6);
assertThat(store.query("SELECT * FROM {lab_run} WHERE trigger_type = 'CHANGE_COMPARE' AND not_before = ?", START.plusSeconds(3604))).hasSize(3);
assertThat(store.query("SELECT * FROM {lab_run} WHERE trigger_type = 'CHANGE_COMPARE' AND not_before = ?", START.plusSeconds(86404))).hasSize(3);
assertThat(runAction(WebPerfProcesses.PROBE_DEPLOYS, Map.of()).getException()).isEmpty();
}
/*******************************************************************************
** Five clean baseline measurements.
*******************************************************************************/
private void baseline() throws Exception
{
for(int i = 0; i < 5; i++)
{
measurement(96, "SCHEDULE", "13.0", Map.of());
}
}
/*******************************************************************************
** Store a deterministic successful measurement, with no network access.
*******************************************************************************/
private long measurement(int score, String trigger, String version, Map<String, Object> extra) throws Exception
{
Instant time = START.plusSeconds(++sequence);
Map<String, Object> row = new LinkedHashMap<>();
row.put("target_id", target);
row.put("url", "https://www.example.com/");
row.put("strategy", "MOBILE");
row.put("status", "SUCCEEDED");
row.put("trigger_type", trigger);
row.put("lighthouse_version", version);
row.put("started_at", time);
row.put("completed_at", time);
row.put("queued_at", time);
row.put("score_performance", score);
row.put("score_accessibility", 100);
row.put("score_best_practices", 100);
row.put("score_seo", 100);
row.put("lcp_ms", 2000);
row.put("fcp_ms", 1000);
row.put("tbt_ms", 0);
row.put("cls", 0);
row.putAll(extra);
return (store.insert(WebPerfSchema.LAB_RUN, row));
}
/*******************************************************************************
** Store and evaluate the next performance measurement.
*******************************************************************************/
private long evaluate(int score) throws Exception
{
long id = measurement(score, "SCHEDULE", "13.0", Map.of());
evaluator.evaluateLab(id);
return (id);
}
/*******************************************************************************
** Complete the independently queued confirmation samples.
*******************************************************************************/
private void completeConfirmations(long first, int score1, int score2) throws Exception
{
int index = 0;
for(Map<String, Object> row : store.query("SELECT id FROM {lab_run} WHERE trigger_ref IN (?, ?) ORDER BY id", "confirm:" + first + ":0", "confirm:" + first + ":1"))
{
store.update("UPDATE {lab_run} SET status = 'SUCCEEDED', completed_at = ?, lighthouse_version = '13.0', score_performance = ? WHERE id = ?",
START.plusSeconds(++sequence), index++ == 0 ? score1 : score2, row.get("id"));
evaluator.evaluateLab(WebPerfStore.asLong(row.get("id")));
}
}
/*******************************************************************************
** One editable budget.
*******************************************************************************/
private long budget(String scope, String scopeValue, String source, String strategy, String metric, String comparator, double threshold, String severity) throws Exception
{
Map<String, Object> row = new LinkedHashMap<>();
row.put("name", "Test budget");
row.put("scope", scope);
row.put("scope_value", scopeValue);
row.put("source", source);
row.put("strategy", strategy);
row.put("metric", metric);
row.put("comparator", comparator);
row.put("threshold", threshold);
row.put("severity", severity);
row.put("enabled", true);
return (store.insert(WebPerfSchema.BUDGET, row));
}
/*******************************************************************************
** A source event without automatic queueing for fixtures.
*******************************************************************************/
private long change(String kind, String component, String value, Instant time) throws Exception
{
return (new WebPerfChangeService(store).record(new ChangeEventSource.Change(kind, component, "before", value, time), "WEBHOOK", "test", false));
}
/*******************************************************************************
** A real or unavailable CrUX collection period.
*******************************************************************************/
private long field(String scope, String status, Integer lcp, int day) throws Exception
{
Map<String, Object> row = new LinkedHashMap<>();
row.put("source", "CRUX_API");
row.put("scope", scope);
row.put("scope_key", "https://www.example.com/");
row.put("form_factor", "PHONE");
row.put("status", status);
row.put("p75_lcp_ms", lcp);
row.put("period_end", LocalDate.of(2026, 10, day));
row.put("last_checked_at", START.plusSeconds(day));
return (store.insert(WebPerfSchema.FIELD_SNAPSHOT, row));
}
/*******************************************************************************
** The first incident, retained through status changes.
*******************************************************************************/
private Map<String, Object> regression() throws Exception
{
return (store.query("SELECT * FROM {regression} ORDER BY id").get(0));
}
/*******************************************************************************
** The count shown in Tested Pages.
*******************************************************************************/
private int openCount() throws Exception
{
return (WebPerfStore.asInt(store.query("SELECT open_regression_count FROM {target} WHERE id = ?", target).get(0).get("open_regression_count")));
}
/*******************************************************************************
** Run the actual ETL process, following the existing frontend-skip convention.
*******************************************************************************/
private RunProcessOutput runAction(String name, Map<String, Serializable> values) throws Exception
{
RunProcessInput input = new RunProcessInput();
input.setPermissionEnforcement(com.kingsrook.qqq.backend.core.actions.permissions.PermissionEnforcement.ENFORCE);
input.setProcessName(config.qualify(name));
input.setValues(new HashMap<>(values));
input.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.SKIP);
input.setCallback(new QProcessCallback()
{
/*******************************************************************************
** Select this test's records.
*******************************************************************************/
@Override
public QQueryFilter getQueryFilter()
{
return (new QQueryFilter());
}
});
return (new RunProcessAction().execute(input));
}
/*******************************************************************************
** Core alert sink, without any Alerts QBit dependency.
*******************************************************************************/
public static class RecordingSink implements AlertSinkInterface
{
static final List<AlertSignal> signals = new ArrayList<>();
static final List<AlertResolution> resolutions = new ArrayList<>();
/*******************************************************************************
** Capture a raise.
*******************************************************************************/
@Override
public void raise(AlertSignal signal)
{
signals.add(signal);
}
/*******************************************************************************
** Capture a resolution.
*******************************************************************************/
@Override
public void resolve(AlertResolution resolution)
{
resolutions.add(resolution);
}
/*******************************************************************************
** This producer does not emit heartbeats.
*******************************************************************************/
@Override
public void heartbeat(String key, Instant at)
{
}
}
/*******************************************************************************
** A probe controlled by the test.
*******************************************************************************/
private static class MutableProbe implements DeployProbe
{
String build = "build-1";
/*******************************************************************************
** Frontend component.
*******************************************************************************/
@Override
public String component()
{
return ("FRONTEND");
}
/*******************************************************************************
** Build id without transport.
*******************************************************************************/
@Override
public String currentBuildId()
{
return (build);
}
}
/*******************************************************************************
** A minimal Lighthouse runner without transport.
*******************************************************************************/
private static class FakeRunner implements LabRunner
{
/*******************************************************************************
** Stored runner name.
*******************************************************************************/
@Override
public String name()
{
return ("TEST");
}
/*******************************************************************************
** Return a successful response.
*******************************************************************************/
@Override
public LabResponse run(LabRequest request)
{
JSONObject body = new JSONObject().put("lighthouseVersion", "13.0").put("finalUrl", request.url())
.put("categories", new JSONObject().put("performance", new JSONObject().put("score", 0.96)))
.put("audits", new JSONObject());
return (new LabResponse(200, body.toString(), Instant.now()));
}
}
}