Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3d72a5d05d
|
||
|
|
4fb8d74458 | ||
|
|
aba80ff62f
|
||
|
|
55a2ec2a09 | ||
|
|
1262f567c8
|
||
|
|
bbf1b55eac | ||
|
|
1cd8810e32
|
||
|
|
d3995734dd
|
||
|
|
8bb2c4eb4f
|
No files matched your search
@@ -5,12 +5,17 @@ All notable changes to this project are documented here. The format follows Keep
|
||||
## [0.1.1] - Unreleased
|
||||
|
||||
### Added
|
||||
- Central `cdn.provider.<id>` dependency health checks for enabled providers, every 900 seconds with a 20000 ms timeout, reusing the existing connection test with bounded, non-secret results.
|
||||
- Ship the host-neutral in-app help bundle with QQQ registration, live references, and coverage checks for all CDN metadata and generated purge actions. Hosts can override entries with a HOST bundle.
|
||||
- Origin header check: Check Origin Headers on Hostnames (and a nightly run over hostnames that are not Off) asks the origin for sample pages and writes `cdn_origin_check` rows. The table had no writer before.
|
||||
- Guided empty states: help bundle `help/qbit-cdn.yaml` (app and all nine tables, with "How rows get created") applied by `CdnHelp`, and a Setup card on the CDN home.
|
||||
- "Create default provider" (No CDN), "Create default policies" (two turned-off examples) and "Create default rules" (a tags-only rule per purgeable table) table actions; all insert-only.
|
||||
- `hostnameRole` is documented as a label that changes nothing; cache policies are documented as applying only where the application calls `CdnResponseHeaderPolicy`.
|
||||
|
||||
### Removed
|
||||
- Connection testing from the daily drift step; drift checks and the manual Test Connection action remain.
|
||||
- The `cdnHealthCard` HTML widget, its renderer and dedicated status summary.
|
||||
|
||||
### Changed
|
||||
- Pin the released QQQ BOM to 4.1.0-RC.7 and replace the temporary help loader with the core bundle registry.
|
||||
- Require released QQQ BOM 4.1.0-RC.5; revalidation hooks default to core Standard Webhooks signing. Per-hook `signatureFormat=LEGACY` preserves the prior header and signature bytes.
|
||||
|
||||
@@ -25,12 +25,26 @@ Design: QRun-IO/qqq `docs/specs/2026-10-12-qbit-cdn-design.md`.
|
||||
Purge Everything, Test Connection, Import Edge Stats, Check Config Drift;
|
||||
scheduled: dispatcher, daily stats import, daily drift check.
|
||||
- **Admin -> CDN app:** sections Connection / Settings / Activity / Performance,
|
||||
a home widget with honest empty states, and a Health status card.
|
||||
a home widget with honest empty states.
|
||||
- **Providers:** CloudFront (invalidations, connection test, CloudWatch stats,
|
||||
drift against the OpenTofu `expected_config`), Souin (self-hosted, tags),
|
||||
NONE. See `docs/cloudfront-expected-config.md`.
|
||||
- **Permissions:** `cdn.view`, `cdn.purge`, `cdn.purgeAll`, `cdn.admin`.
|
||||
|
||||
## Health checks
|
||||
|
||||
Central System Health registers `cdn.provider.<id>` for each enabled
|
||||
`cdn_provider` row: **DEPENDENCY**, component `cdn`, category **CDN**, every
|
||||
**900 seconds**, with a **20000 ms** timeout. It reuses the read-only connection
|
||||
test: failure is DOWN; success without the origin header is DEGRADED; otherwise
|
||||
UP. Providers without connection-test support report UNKNOWN. Disable a check
|
||||
centrally with `health_check.enabled`.
|
||||
|
||||
Details contain only provider type, a known provider status (or UNKNOWN), and
|
||||
whether the origin header is configured. The manual `cdnTestConnection` action
|
||||
and its `lastHealth*` fields remain; the daily scheduled step checks drift only.
|
||||
The host must use QQQ **4.1.0-RC.8** or later for the central health SPI.
|
||||
|
||||
## Setup
|
||||
|
||||
This qbit ships no DDL and seeds nothing from a changelog (a host chooses its own column names). Open the CDN app: the Setup card lists each part with the next step. On a fresh install run **Create default provider** (No CDN: purges are recorded, nothing is sent), **Create default rules** (tags for every purgeable table), and optionally **Create default policies** (two turned-off examples). Add your CDN and hostnames when you have them. Cache policies change a response only where the application calls `CdnResponseHeaderPolicy.decide`. **Check Origin Headers** on a hostname (and the nightly run) asks the origin, the provider's `originHost` when set, for the home page, prefix policy paths and literal tag-rule paths, and records how many have the right Cache-Control.
|
||||
@@ -78,7 +92,7 @@ variable named by the provider setting `expectedConfigEnv`).
|
||||
|
||||
## Revalidation hooks
|
||||
|
||||
Requires QQQ **4.1.0-RC.7** (the default `qqq-snapshot` profile imports the released BOM;
|
||||
Requires QQQ **4.1.0-RC.8** (the default `qqq-snapshot` profile imports the released BOM;
|
||||
its historical property name does not select a core SNAPSHOT).
|
||||
|
||||
New `cdn_revalidation_hook` rows default `signatureFormat` to `STANDARD_WEBHOOKS`;
|
||||
@@ -144,7 +158,7 @@ with purging enabled. Purge From CDN, Purge Everything, and Retry Purge show a
|
||||
Review warning and a recorded-only Result when sending is unavailable. Requests
|
||||
for enabled providers remain visible with `SKIPPED_NONE_PROVIDER` or
|
||||
`SKIPPED_STAGE_OFF`; with no enabled provider, no new request row is created.
|
||||
The home Connection and Health cards explain the unconfigured state, and purge
|
||||
The home Connection card explains the unconfigured state, and purge
|
||||
activity labels recorded-only requests when disconnected.
|
||||
|
||||
Run **Check Origin Headers** on a hostname to populate origin-check results.
|
||||
|
||||
@@ -106,7 +106,34 @@
|
||||
</dependencies>
|
||||
|
||||
<build>
|
||||
<resources>
|
||||
<!-- help bundles carry the artifact version (QQQ help bundle provenance) -->
|
||||
<resource>
|
||||
<directory>src/main/resources</directory>
|
||||
<filtering>true</filtering>
|
||||
<includes>
|
||||
<include>help/*.yaml</include>
|
||||
</includes>
|
||||
</resource>
|
||||
<resource>
|
||||
<directory>src/main/resources</directory>
|
||||
<filtering>false</filtering>
|
||||
<excludes>
|
||||
<exclude>help/*.yaml</exclude>
|
||||
</excludes>
|
||||
</resource>
|
||||
</resources>
|
||||
<plugins>
|
||||
<!-- the artifact version, for the help bundle version test -->
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-surefire-plugin</artifactId>
|
||||
<configuration>
|
||||
<systemPropertyVariables combine.children="append">
|
||||
<qbit.artifactVersion>${project.version}</qbit.artifactVersion>
|
||||
</systemPropertyVariables>
|
||||
</configuration>
|
||||
</plugin>
|
||||
<plugin>
|
||||
<groupId>org.apache.maven.plugins</groupId>
|
||||
<artifactId>maven-checkstyle-plugin</artifactId>
|
||||
@@ -194,7 +221,7 @@
|
||||
<activeByDefault>true</activeByDefault>
|
||||
</activation>
|
||||
<properties>
|
||||
<qqq.snapshot.version>4.1.0-RC.7</qqq.snapshot.version>
|
||||
<qqq.snapshot.version>4.1.0-RC.8</qqq.snapshot.version>
|
||||
</properties>
|
||||
<dependencyManagement>
|
||||
<dependencies>
|
||||
|
||||
@@ -20,10 +20,13 @@ package com.kingsrook.qbits.cdn;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import com.kingsrook.qbits.cdn.app.CdnWidgets;
|
||||
import com.kingsrook.qbits.cdn.health.CdnProviderHealthChecks;
|
||||
import com.kingsrook.qbits.cdn.model.CdnTables;
|
||||
import com.kingsrook.qbits.cdn.processes.CdnProcesses;
|
||||
import com.kingsrook.qbits.cdn.purge.CdnRecordChangeListener;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.health.HealthCheckSupplierMetaData;
|
||||
import com.kingsrook.qqq.backend.core.health.HealthChecksMetaData;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
|
||||
@@ -162,6 +165,8 @@ public class CdnQBitProducer implements QBitMetaDataProducer<CdnQBitConfig>
|
||||
}
|
||||
output.add(CdnWidgets.app(qBitConfig, tables));
|
||||
output.addSelfToInstance(qInstance);
|
||||
HealthChecksMetaData.ofOrWithNew(qInstance).withSupplier(new HealthCheckSupplierMetaData()
|
||||
.withNamePrefix("cdn.provider").withComponent("cdn").withSupplier(new QCodeReference(CdnProviderHealthChecks.class)));
|
||||
com.kingsrook.qbits.cdn.model.CdnHelp.register(qInstance, qBitConfig);
|
||||
|
||||
if(!qBitConfig.getPurgeableTables().isEmpty())
|
||||
|
||||
@@ -252,24 +252,6 @@ public record CdnHomeData(List<ProviderView> providers, List<HostnameView> hostn
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Overall state for the Health card: NOT_CONNECTED, ATTENTION or OK.
|
||||
*******************************************************************************/
|
||||
public String overallState()
|
||||
{
|
||||
if(!isConnected())
|
||||
{
|
||||
return ("NOT_CONNECTED");
|
||||
}
|
||||
boolean attention = dead24h > 0 || !openDriftByArea.isEmpty()
|
||||
|| providers.stream().anyMatch(p -> p.isConnected() && ("FAILED".equals(p.lastHealthStatus())
|
||||
|| (p.credentialAgeDays() != null && p.credentialAgeDays() >= ROTATION_WARNING_DAYS)))
|
||||
|| budgets.stream().anyMatch(b -> b.billed() > b.soft());
|
||||
return (attention ? "ATTENTION" : "OK");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Paths billed this month for a provider.
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -72,26 +72,6 @@ public final class CdnHomeView
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The Health status card.
|
||||
*******************************************************************************/
|
||||
public static String renderHealthCard(CdnHomeData data, String appPath)
|
||||
{
|
||||
String state = data.overallState();
|
||||
String text = "All CDN checks are OK.";
|
||||
if("NOT_CONNECTED".equals(state))
|
||||
{
|
||||
text = "No CDN provider configured: purges are recorded only.";
|
||||
}
|
||||
else if("ATTENTION".equals(state))
|
||||
{
|
||||
text = "Needs attention: see the CDN home.";
|
||||
}
|
||||
return ("<div style=\"" + CARD + "\"><strong>CDN</strong><p>" + esc(text) + "</p><a href=\"" + esc(appPath) + "\">Open the CDN home</a></div>");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Connection card.
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -37,13 +37,12 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
|
||||
|
||||
/*******************************************************************************
|
||||
** The Admin -> CDN app: a home widget (the single owner of CDN data), four
|
||||
** sections of standard tables, and a status card for the app's Health page.
|
||||
** sections of standard tables.
|
||||
** Processes are never added to the nav.
|
||||
*******************************************************************************/
|
||||
public final class CdnWidgets
|
||||
{
|
||||
public static final String HOME_WIDGET = "cdnHome";
|
||||
public static final String HEALTH_CARD_WIDGET = "cdnHealthCard";
|
||||
public static final String HOME_WIDGET = "cdnHome";
|
||||
|
||||
|
||||
|
||||
@@ -63,9 +62,7 @@ public final class CdnWidgets
|
||||
{
|
||||
return (List.of(
|
||||
new QWidgetMetaData().withName(HOME_WIDGET).withLabel("CDN").withType(WidgetType.HTML.getType())
|
||||
.withCodeReference(new QCodeReference(HomeRenderer.class)).withGridColumns(12).withIsCard(false).withPermissionRules(CdnTables.permissionRules()),
|
||||
new QWidgetMetaData().withName(HEALTH_CARD_WIDGET).withLabel("CDN").withType(WidgetType.HTML.getType())
|
||||
.withCodeReference(new QCodeReference(HealthCardRenderer.class)).withGridColumns(12).withIsCard(true).withPermissionRules(CdnTables.permissionRules())));
|
||||
.withCodeReference(new QCodeReference(HomeRenderer.class)).withGridColumns(12).withIsCard(false).withPermissionRules(CdnTables.permissionRules())));
|
||||
}
|
||||
|
||||
|
||||
@@ -132,21 +129,4 @@ public final class CdnWidgets
|
||||
return (new RenderWidgetOutput(new RawHTML(null, CdnHomeView.render(CdnHomeData.load(), tablePathPrefix()))));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Health status card renderer.
|
||||
*******************************************************************************/
|
||||
public static class HealthCardRenderer extends AbstractWidgetRenderer
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Render.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
|
||||
{
|
||||
return (new RenderWidgetOutput(new RawHTML("CDN", CdnHomeView.renderHealthCard(CdnHomeData.load(), appPath()))));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,149 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.cdn.health;
|
||||
|
||||
|
||||
import java.io.Serializable;
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.Locale;
|
||||
import java.util.concurrent.ExecutionException;
|
||||
import java.util.concurrent.FutureTask;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import java.util.concurrent.TimeoutException;
|
||||
import com.kingsrook.qbits.cdn.CdnRuntime;
|
||||
import com.kingsrook.qbits.cdn.model.CdnTables;
|
||||
import com.kingsrook.qbits.cdn.purge.CdnPurgeDispatcher;
|
||||
import com.kingsrook.qbits.cdn.spi.CdnOperations;
|
||||
import com.kingsrook.qbits.cdn.spi.CdnProviderConfig;
|
||||
import com.kingsrook.qbits.cdn.spi.CdnProviderInterface;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.core.health.HealthCategory;
|
||||
import com.kingsrook.qqq.backend.core.health.HealthCheckInput;
|
||||
import com.kingsrook.qqq.backend.core.health.HealthCheckKind;
|
||||
import com.kingsrook.qqq.backend.core.health.HealthCheckMetaData;
|
||||
import com.kingsrook.qqq.backend.core.health.HealthCheckOutcome;
|
||||
import com.kingsrook.qqq.backend.core.health.HealthCheckSupplier;
|
||||
import com.kingsrook.qqq.backend.core.health.HealthDetails;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Central health checks reuse the read-only provider connection test.
|
||||
*******************************************************************************/
|
||||
public class CdnProviderHealthChecks implements HealthCheckSupplier
|
||||
{
|
||||
/*******************************************************************************
|
||||
** Resolve the enabled provider rows each time the registry refreshes.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
public List<HealthCheckMetaData> supply(QInstance qInstance) throws QException
|
||||
{
|
||||
List<HealthCheckMetaData> checks = new ArrayList<>();
|
||||
for(QRecord provider : new QueryAction().execute(new QueryInput(CdnTables.PROVIDER)
|
||||
.withFilter(new QQueryFilter(new QFilterCriteria("enabled", QCriteriaOperator.EQUALS, true)))).getRecords())
|
||||
{
|
||||
Serializable id = provider.getValue("id");
|
||||
String name = "cdn.provider." + id.toString().toLowerCase(Locale.ROOT);
|
||||
checks.add(new HealthCheckMetaData().withName(name).withLabel(name).withComponent("cdn")
|
||||
.withKind(HealthCheckKind.DEPENDENCY).withCategory(HealthCategory.CDN)
|
||||
.withIntervalSeconds(900).withTimeoutMs(20000).withCheckInstance(input -> run(id, input)));
|
||||
}
|
||||
return (checks);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Reload configuration so disabled or deleted providers are not probed.
|
||||
** Never copy provider messages, addresses or exception text into outcomes.
|
||||
*******************************************************************************/
|
||||
private static HealthCheckOutcome run(Serializable id, HealthCheckInput input) throws QException
|
||||
{
|
||||
if(!Instant.now().isBefore(input.getDeadline()))
|
||||
{
|
||||
return (HealthCheckOutcome.down("CDN connection test timed out"));
|
||||
}
|
||||
QRecord provider = new GetAction().executeForRecord(new GetInput(CdnTables.PROVIDER).withPrimaryKey(id));
|
||||
if(provider == null || !Boolean.TRUE.equals(provider.getValueBoolean("enabled")))
|
||||
{
|
||||
return (HealthCheckOutcome.unknown("CDN provider is disabled or removed"));
|
||||
}
|
||||
CdnProviderConfig config = CdnPurgeDispatcher.configOf(provider);
|
||||
CdnProviderInterface adapter = CdnRuntime.provider(config.type());
|
||||
if(!(adapter instanceof CdnOperations operations))
|
||||
{
|
||||
return (HealthCheckOutcome.unknown("CDN provider does not support connection tests"));
|
||||
}
|
||||
return (probe(operations, config, input));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The existing SPI has no timeout parameter. Bound its wait by the remaining
|
||||
** deadline and interrupt the probe on timeout or runner cancellation.
|
||||
*******************************************************************************/
|
||||
private static HealthCheckOutcome probe(CdnOperations operations, CdnProviderConfig config, HealthCheckInput input)
|
||||
{
|
||||
long remaining = Duration.between(Instant.now(), input.getDeadline()).toNanos();
|
||||
if(remaining <= 0)
|
||||
{
|
||||
return (HealthCheckOutcome.down("CDN connection test timed out"));
|
||||
}
|
||||
FutureTask<CdnOperations.ConnectionTest> task = new FutureTask<>(() -> operations.connectionTest(config));
|
||||
Thread.startVirtualThread(task);
|
||||
try
|
||||
{
|
||||
CdnOperations.ConnectionTest test = task.get(remaining, TimeUnit.NANOSECONDS);
|
||||
HealthCheckOutcome outcome = !test.ok() ? HealthCheckOutcome.down("CDN connection test failed")
|
||||
: !test.originHeaderConfigured() ? HealthCheckOutcome.degraded("CDN origin header is not configured")
|
||||
: HealthCheckOutcome.up("CDN connection test passed");
|
||||
// Only known provider states are safe; arbitrary status text may contain secrets.
|
||||
String status = "Deployed".equals(test.status()) ? "Deployed" : "InProgress".equals(test.status()) ? "InProgress" : "UNKNOWN";
|
||||
return (outcome.withDetails(new HealthDetails().withValue("providerType", config.type().name())
|
||||
.withValue("status", status).withValue("originHeaderConfigured", test.originHeaderConfigured())));
|
||||
}
|
||||
catch(TimeoutException e)
|
||||
{
|
||||
return (HealthCheckOutcome.down("CDN connection test timed out"));
|
||||
}
|
||||
catch(InterruptedException e)
|
||||
{
|
||||
Thread.currentThread().interrupt();
|
||||
return (HealthCheckOutcome.down("CDN connection test interrupted"));
|
||||
}
|
||||
catch(ExecutionException e)
|
||||
{
|
||||
return (HealthCheckOutcome.down("CDN connection test failed"));
|
||||
}
|
||||
finally
|
||||
{
|
||||
task.cancel(true);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -525,7 +525,7 @@ public final class CdnProviderProcesses
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Scheduled: daily drift check (and connection test) for every operable
|
||||
** Scheduled: daily drift check for every operable
|
||||
** provider.
|
||||
*******************************************************************************/
|
||||
public static class ScheduledDriftStep implements BackendStep
|
||||
@@ -538,7 +538,6 @@ public final class CdnProviderProcesses
|
||||
{
|
||||
for(QRecord provider : operableProviders())
|
||||
{
|
||||
testAndRecord(provider);
|
||||
driftAndRecord(provider);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
bundle: qbit-cdn
|
||||
version: 0.1.0-SNAPSHOT
|
||||
version: ${project.version}
|
||||
layer: QBIT
|
||||
entries:
|
||||
- key: app:cdnAdmin;slot:help
|
||||
@@ -324,7 +324,7 @@ entries:
|
||||
content: |
|
||||
**{{label:cdnCheckDrift}}** (and a daily job when a scheduler is configured) reads the live CDN settings and compares them with the expected setup defined by the infrastructure code. Each difference becomes a finding with what was expected and what was found.
|
||||
|
||||
A finding stays open until a later check no longer sees it, then it is marked resolved. Open findings turn the CDN status on the Health screen to needing attention. You cannot edit findings; fix the CDN or the expected setup instead.
|
||||
A finding stays open until a later check no longer sees it, then it is marked resolved. You cannot edit findings; fix the CDN or the expected setup instead.
|
||||
|
||||
**How rows get created:** Check Config Drift on a provider writes it, and, when a scheduler is configured, a nightly run does the same. It needs the expected configuration in the environment variable the provider names (CDN_EXPECTED_CONFIG by default); a finding is resolved when it no longer appears.
|
||||
- key: table:cdn_drift_finding;field:cdnProviderId
|
||||
@@ -386,18 +386,14 @@ entries:
|
||||
content: |
|
||||
When your application configures a scheduler, this job imports the last three complete UTC days for enabled providers that support statistics. Use **{{label:cdnImportEdgeStats}}** to refresh on demand.
|
||||
- key: process:cdnScheduledCheckDrift;slot:help
|
||||
summary: Test enabled CDN providers and check their configuration on the configured schedule.
|
||||
summary: Check enabled CDN providers for configuration drift on the configured schedule.
|
||||
content: |
|
||||
When your application configures a scheduler, this job updates connection health and drift findings for supported providers. Use **{{label:cdnCheckDrift}}** to check drift on demand.
|
||||
When your application configures a scheduler, this job updates drift findings for supported providers. Connection health is checked separately by central System Health. Use **{{label:cdnCheckDrift}}** to check drift on demand.
|
||||
- key: widget:cdnHome;slot:label
|
||||
summary: 'The CDN at a glance: provider, hostnames, traffic, purge activity, invalidation budget, origin checks and
|
||||
drift.'
|
||||
content: |
|
||||
Each card says plainly when there is nothing yet, for example when no CDN is connected or no statistics have been imported. Traffic figures cover the last 7 and 30 days; purge failures cover 24 hours and 7 days.
|
||||
- key: widget:cdnHealthCard;slot:label
|
||||
summary: 'A one-line CDN status for the Health screen: not connected, needs attention, or all checks OK.'
|
||||
content: |
|
||||
It needs attention when a purge went dead in the last 24 hours, a drift finding is open, the last connection test failed, the credentials are 75 or more days old, or this month's billed paths passed the soft limit.
|
||||
- key: possibleValueSource:cdn_provider_type;slot:help
|
||||
summary: The CDN services a provider can be.
|
||||
content: |
|
||||
|
||||
@@ -65,7 +65,7 @@ class CdnHelpBundleTest
|
||||
assertThat(instance.getTable("helpContent")).isNull();
|
||||
var bundle = HelpBundles.of(instance).getBundles().get(0);
|
||||
assertThat(bundle.getId()).isEqualTo("qbit-cdn");
|
||||
assertThat(bundle.getVersion()).isEqualTo("0.1.0-SNAPSHOT");
|
||||
assertThat(bundle.getVersion()).as("the help bundle carries the artifact version").isEqualTo(System.getProperty("qbit.artifactVersion"));
|
||||
assertThat(bundle.getLayer()).isEqualTo(HelpBundleLayer.QBIT);
|
||||
assertThat(bundle.getEntries()).extracting(entry -> entry.getKey()).doesNotHaveDuplicates();
|
||||
// Core's gate does not require widgets or possible-value sources yet.
|
||||
|
||||
@@ -52,6 +52,7 @@ class CdnQBitProducerTest
|
||||
assertThat(instance.getProcess(CdnProcesses.PURGE_HOSTNAME)).isNotNull();
|
||||
assertThat(instance.getProcess(CdnProcesses.DISPATCH_PURGES).getIsHidden()).isTrue();
|
||||
assertThat(instance.getWidget(CdnWidgets.HOME_WIDGET)).isNotNull();
|
||||
assertThat(instance.getWidget("cdnHealthCard")).isNull();
|
||||
assertThat(instance.getApp("cdnAdmin").getSections()).extracting(section -> section.getLabel())
|
||||
.containsExactly("Connection", "Settings", "Activity", "Performance");
|
||||
assertThat(instance.getRecordChangeListeners()).isNotEmpty();
|
||||
|
||||
@@ -34,7 +34,7 @@ import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The CDN home and Health card: honest empty states, then real data.
|
||||
** The CDN home: honest empty states, then real data.
|
||||
*******************************************************************************/
|
||||
class CdnHomeTest
|
||||
{
|
||||
@@ -61,7 +61,6 @@ class CdnHomeTest
|
||||
assertThat(html).contains("No CDN provider configured: purges are recorded only.").contains("No edge statistics yet.").contains("No purges recorded yet.")
|
||||
.contains("No path-billed provider connected yet.").contains("No origin check yet.").contains("Nothing to compare yet.")
|
||||
.contains("minmax(min(100%,22rem),1fr)");
|
||||
assertThat(render(CdnWidgets.HEALTH_CARD_WIDGET)).contains("No CDN provider configured: purges are recorded only.").contains("href=\"/cdnAdmin\"");
|
||||
}
|
||||
|
||||
|
||||
@@ -91,7 +90,6 @@ class CdnHomeTest
|
||||
CdnTestSupport.insert(CdnTables.DRIFT_FINDING, new QRecord().withValue("cdnProviderId", provider).withValue("area", "BEHAVIOR").withValue("checkedDate", CdnTestSupport.NOW));
|
||||
|
||||
CdnHomeData data = CdnHomeData.load();
|
||||
assertEquals("ATTENTION", data.overallState());
|
||||
assertEquals(1, data.dead24h());
|
||||
assertEquals(4, data.budgets().get(0).billed());
|
||||
assertEquals(age, data.providers().get(0).credentialAgeDays());
|
||||
@@ -100,7 +98,6 @@ class CdnHomeTest
|
||||
assertThat(html).contains("d1.cloudfront.net").contains(age + " days (rotate every 90) - rotation due").contains("1,500").contains("91.5%")
|
||||
.contains("<polyline").contains("396 of 396").contains("BEHAVIOR").contains("1 open").contains("/cdnAdmin/cdn_purge_request/")
|
||||
.contains("www.example.com").doesNotContain("recorded only (not sent)", "nothing has been checked");
|
||||
assertThat(CdnHomeView.renderHealthCard(data, "/cdnAdmin")).contains("Needs attention");
|
||||
}
|
||||
|
||||
|
||||
@@ -142,7 +139,6 @@ class CdnHomeTest
|
||||
assertThat(render(CdnWidgets.HOME_WIDGET)).contains("No CDN provider configured: purges are recorded only.")
|
||||
.contains("The CloudFront distribution is created in the cutover window.").contains("recorded only (not sent)")
|
||||
.contains("No origin check yet.");
|
||||
assertThat(render(CdnWidgets.HEALTH_CARD_WIDGET)).contains("No CDN provider configured: purges are recorded only.");
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,333 @@
|
||||
/*
|
||||
* Copyright 2026 Kingsrook, LLC
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.cdn.health;
|
||||
|
||||
|
||||
import java.time.Duration;
|
||||
import java.time.Instant;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
import java.util.concurrent.TimeUnit;
|
||||
import com.kingsrook.qbits.cdn.CdnRuntime;
|
||||
import com.kingsrook.qbits.cdn.CdnTestSupport;
|
||||
import com.kingsrook.qbits.cdn.model.CdnTables;
|
||||
import com.kingsrook.qbits.cdn.spi.CdnOperations;
|
||||
import com.kingsrook.qbits.cdn.spi.CdnProviderConfig;
|
||||
import com.kingsrook.qbits.cdn.spi.CdnProviderType;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.GetAction;
|
||||
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
|
||||
import com.kingsrook.qqq.backend.core.context.QContext;
|
||||
import com.kingsrook.qqq.backend.core.health.HealthCategory;
|
||||
import com.kingsrook.qqq.backend.core.health.HealthCheckInput;
|
||||
import com.kingsrook.qqq.backend.core.health.HealthCheckKind;
|
||||
import com.kingsrook.qqq.backend.core.health.HealthCheckMetaData;
|
||||
import com.kingsrook.qqq.backend.core.health.HealthCheckOutcome;
|
||||
import com.kingsrook.qqq.backend.core.health.HealthChecksMetaData;
|
||||
import com.kingsrook.qqq.backend.core.health.HealthStatus;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput;
|
||||
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
|
||||
import com.kingsrook.qqq.backend.core.model.data.QRecord;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.core.model.session.QSession;
|
||||
import com.kingsrook.qqq.backend.core.model.session.QUser;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeEach;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.junit.jupiter.api.Assertions.assertTimeoutPreemptively;
|
||||
import static org.mockito.ArgumentMatchers.any;
|
||||
import static org.mockito.Mockito.doAnswer;
|
||||
import static org.mockito.Mockito.doReturn;
|
||||
import static org.mockito.Mockito.doThrow;
|
||||
import static org.mockito.Mockito.never;
|
||||
import static org.mockito.Mockito.spy;
|
||||
import static org.mockito.Mockito.verify;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Provider checks reuse the probe, honor deadlines and expose only safe facts.
|
||||
*******************************************************************************/
|
||||
class CdnProviderHealthChecksTest
|
||||
{
|
||||
private static final String SECRET = "S3NT1NEL-secret";
|
||||
private QInstance instance;
|
||||
private Integer providerId;
|
||||
private CdnTestSupport.FakePathProvider adapter;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A memory-backed instance and the existing fake adapter, with no network.
|
||||
*******************************************************************************/
|
||||
@BeforeEach
|
||||
void setUp() throws Exception
|
||||
{
|
||||
instance = CdnTestSupport.setUp();
|
||||
providerId = CdnTestSupport.liveProvider(CdnProviderType.CLOUDFRONT, null);
|
||||
adapter = spy(new CdnTestSupport.FakePathProvider());
|
||||
CdnRuntime.registerProvider(CdnProviderType.CLOUDFRONT, () -> adapter);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Restore process-wide test state.
|
||||
*******************************************************************************/
|
||||
@AfterEach
|
||||
void tearDown()
|
||||
{
|
||||
CdnRuntime.setEnvironment(null);
|
||||
CdnRuntime.setClock(null);
|
||||
Thread.interrupted();
|
||||
QContext.clear();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The producer registers a dynamic dependency supplier with the spec defaults.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void registrationAndRefresh() throws Exception
|
||||
{
|
||||
var registry = HealthChecksMetaData.ofOrWithNew(instance);
|
||||
assertThat(registry.getSuppliers()).singleElement().satisfies(supplier ->
|
||||
{
|
||||
assertThat(supplier.getNamePrefix()).isEqualTo("cdn.provider");
|
||||
assertThat(supplier.getComponent()).isEqualTo("cdn");
|
||||
});
|
||||
CdnTestSupport.insert(CdnTables.PROVIDER, new QRecord().withValue("name", "disabled").withValue("providerType", "CLOUDFRONT").withValue("enabled", false));
|
||||
// Match the runner's automated context rather than relying on CDN UI permissions.
|
||||
QContext.init(instance, new QSession().withUser(new QUser().withIdReference("health_runner").withFullName("health_runner"))
|
||||
.withSecurityKeyValues(Map.of()).withPermissions("health.read", "health.write"));
|
||||
assertThat(registry.resolveAll(instance)).singleElement().satisfies(check ->
|
||||
{
|
||||
assertThat(check.getName()).isEqualTo("cdn.provider." + providerId).matches("[a-z0-9]+([._-][a-z0-9]+)*");
|
||||
assertThat(check.getComponent()).isEqualTo("cdn");
|
||||
assertThat(check.getKind()).isEqualTo(HealthCheckKind.DEPENDENCY);
|
||||
assertThat(check.getCategory()).isEqualTo(HealthCategory.CDN);
|
||||
assertThat(check.getIntervalSeconds()).isEqualTo(900);
|
||||
assertThat(check.getTimeoutMs()).isEqualTo(20000);
|
||||
assertThat(check.getIncludeInReadiness()).isFalse();
|
||||
});
|
||||
assertThat(check().getCheckInstance().run(input()).getStatus()).isEqualTo(HealthStatus.UP);
|
||||
update(new QRecord().withValue("enabled", false));
|
||||
assertThat(registry.resolveAll(instance)).isEmpty();
|
||||
update(new QRecord().withValue("enabled", true));
|
||||
assertThat(registry.resolveAll(instance)).hasSize(1);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Map success, missing origin protection and failure without writing lastHealth.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void connectionStates() throws Exception
|
||||
{
|
||||
var check = check().getCheckInstance();
|
||||
HealthCheckOutcome up = check.run(input());
|
||||
assertThat(up.getStatus()).isEqualTo(HealthStatus.UP);
|
||||
assertThat(up.getDetails().getValues()).containsExactlyInAnyOrderEntriesOf(Map.of(
|
||||
"providerType", "CLOUDFRONT", "status", "Deployed", "originHeaderConfigured", true));
|
||||
doReturn(new CdnOperations.ConnectionTest(true, null, null, null, false, "InProgress", "unused")).when(adapter).connectionTest(any());
|
||||
HealthCheckOutcome degraded = check.run(input());
|
||||
assertThat(degraded.getStatus()).isEqualTo(HealthStatus.DEGRADED);
|
||||
assertThat(degraded.getDetails().getValues()).containsEntry("originHeaderConfigured", false).containsEntry("status", "InProgress");
|
||||
doReturn(new CdnOperations.ConnectionTest(false, null, null, null, false, null, "unused")).when(adapter).connectionTest(any());
|
||||
assertThat(check.run(input()).getStatus()).isEqualTo(HealthStatus.DOWN);
|
||||
QRecord row = new GetAction().executeForRecord(new GetInput(CdnTables.PROVIDER).withPrimaryKey(providerId));
|
||||
assertThat(row.getValue("lastHealthCheckDate")).isNull();
|
||||
assertThat(row.getValue("lastHealthStatus")).isNull();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A previously resolved check reloads config and skips disabled/deleted rows.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void reloadsProviderAndHandlesUnsupported() throws Exception
|
||||
{
|
||||
var check = check().getCheckInstance();
|
||||
update(new QRecord().withValue("providerType", "NONE"));
|
||||
assertThat(check.run(input()).getStatus()).isEqualTo(HealthStatus.UNKNOWN);
|
||||
update(new QRecord().withValue("enabled", false));
|
||||
assertThat(check.run(input()).getStatus()).isEqualTo(HealthStatus.UNKNOWN);
|
||||
new DeleteAction().execute(new DeleteInput(CdnTables.PROVIDER).withPrimaryKeys(List.of(providerId)));
|
||||
assertThat(check.run(input()).getStatus()).isEqualTo(HealthStatus.UNKNOWN);
|
||||
verify(adapter, never()).connectionTest(any());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Poison credentials, every free-form probe field and thrown error messages.
|
||||
** HealthRunner.sanitize is package-private, so assert the raw boundary here.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void noSecretsInOutcomes() throws Exception
|
||||
{
|
||||
update(new QRecord().withValue("secretRef", "TEST_CDN").withValue("name", SECRET)
|
||||
.withValue("zoneOrDistributionId", SECRET).withValue("settingsJson", "{\"originVerifyHeader\":\"" + SECRET + "\"}"));
|
||||
CdnRuntime.setEnvironment(name -> SECRET);
|
||||
for(HealthStatus expected : List.of(HealthStatus.UP, HealthStatus.DEGRADED, HealthStatus.DOWN))
|
||||
{
|
||||
doAnswer(invocation ->
|
||||
{
|
||||
CdnProviderConfig config = invocation.getArgument(0);
|
||||
assertThat(CdnRuntime.environment(config.secretRef() + "_ACCESS_KEY_ID")).isEqualTo(SECRET);
|
||||
assertThat(CdnRuntime.environment(config.secretRef() + "_SECRET_ACCESS_KEY")).isEqualTo(SECRET);
|
||||
return (new CdnOperations.ConnectionTest(expected != HealthStatus.DOWN, SECRET, SECRET, 443,
|
||||
expected == HealthStatus.UP, SECRET, SECRET));
|
||||
}).when(adapter).connectionTest(any());
|
||||
HealthCheckOutcome outcome = check().getCheckInstance().run(input());
|
||||
assertThat(outcome.getStatus()).isEqualTo(expected);
|
||||
assertClean(outcome);
|
||||
}
|
||||
doThrow(new IllegalStateException(SECRET)).when(adapter).connectionTest(any());
|
||||
HealthCheckOutcome failed = check().getCheckInstance().run(input());
|
||||
assertThat(failed.getStatus()).isEqualTo(HealthStatus.DOWN);
|
||||
assertClean(failed);
|
||||
update(new QRecord().withValue("providerType", "NONE"));
|
||||
assertClean(check().getCheckInstance().run(input()));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** An expired deadline never starts the existing probe.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void expiredDeadline() throws Exception
|
||||
{
|
||||
HealthCheckOutcome outcome = check().getCheckInstance().run(input().withDeadline(Instant.now().minusSeconds(1)));
|
||||
assertThat(outcome.getStatus()).isEqualTo(HealthStatus.DOWN);
|
||||
assertThat(outcome.getMessage()).contains("timed out");
|
||||
verify(adapter, never()).connectionTest(any());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A slow adapter is cancelled at the deadline, without a blocking executor close.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void deadlineCancelsProbe() throws Exception
|
||||
{
|
||||
CountDownLatch cancelled = new CountDownLatch(1);
|
||||
doAnswer(invocation ->
|
||||
{
|
||||
try
|
||||
{
|
||||
new CountDownLatch(1).await();
|
||||
return (null);
|
||||
}
|
||||
finally
|
||||
{
|
||||
cancelled.countDown();
|
||||
}
|
||||
}).when(adapter).connectionTest(any());
|
||||
HealthCheckOutcome outcome = assertTimeoutPreemptively(Duration.ofSeconds(3), () ->
|
||||
{
|
||||
QContext.init(instance, new QSession());
|
||||
try
|
||||
{
|
||||
return (check().getCheckInstance().run(input().withDeadline(Instant.now().plusMillis(200))));
|
||||
}
|
||||
finally
|
||||
{
|
||||
QContext.clear();
|
||||
}
|
||||
});
|
||||
assertThat(outcome.getStatus()).isEqualTo(HealthStatus.DOWN);
|
||||
assertThat(outcome.getMessage()).contains("timed out");
|
||||
assertThat(cancelled.await(2, TimeUnit.SECONDS)).isTrue();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Runner cancellation preserves the caller's interrupt status.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void interruptedProbe() throws Exception
|
||||
{
|
||||
doAnswer(invocation ->
|
||||
{
|
||||
new CountDownLatch(1).await();
|
||||
return (null);
|
||||
}).when(adapter).connectionTest(any());
|
||||
var check = check().getCheckInstance();
|
||||
Thread.currentThread().interrupt();
|
||||
try
|
||||
{
|
||||
HealthCheckOutcome outcome = check.run(input());
|
||||
assertThat(outcome.getStatus()).isEqualTo(HealthStatus.DOWN);
|
||||
assertThat(outcome.getMessage()).contains("interrupted");
|
||||
assertThat(Thread.currentThread().isInterrupted()).isTrue();
|
||||
}
|
||||
finally
|
||||
{
|
||||
Thread.interrupted();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Resolve through the actual registration, including QCodeReference loading.
|
||||
*******************************************************************************/
|
||||
private HealthCheckMetaData check() throws Exception
|
||||
{
|
||||
return (HealthChecksMetaData.ofOrWithNew(instance).resolveAll(instance).getFirst());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fresh deadline for each call.
|
||||
*******************************************************************************/
|
||||
private HealthCheckInput input()
|
||||
{
|
||||
return (new HealthCheckInput().withQInstance(instance).withDeadline(Instant.now().plusSeconds(5)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Modify the real memory row to exercise fresh reads.
|
||||
*******************************************************************************/
|
||||
private void update(QRecord values) throws Exception
|
||||
{
|
||||
new UpdateAction().execute(new UpdateInput(CdnTables.PROVIDER).withRecord(values.withValue("id", providerId)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** No provider-controlled text can reach the central health boundary.
|
||||
*******************************************************************************/
|
||||
private static void assertClean(HealthCheckOutcome outcome)
|
||||
{
|
||||
assertThat(outcome.getMessage()).doesNotContain(SECRET, "\n", "\r");
|
||||
assertThat(outcome.getDetails().toJson()).doesNotContain(SECRET, "TEST_CDN", "origin.example.com", "d1.cloudfront.net");
|
||||
}
|
||||
}
|
||||
@@ -199,9 +199,14 @@ class CdnProviderProcessesTest
|
||||
@Test
|
||||
void scheduledSteps() throws Exception
|
||||
{
|
||||
FakePathProvider adapter = org.mockito.Mockito.spy(new FakePathProvider());
|
||||
CdnRuntime.registerProvider(CdnProviderType.CLOUDFRONT, () -> adapter);
|
||||
FakePathProvider.drift = List.of(new CdnOperations.DriftItem("origin.header", "present", "missing"));
|
||||
new CdnProviderProcesses.ScheduledStatsStep().run(null, null);
|
||||
new CdnProviderProcesses.ScheduledDriftStep().run(null, null);
|
||||
assertThat(new QueryAction().execute(new QueryInput(CdnTables.EDGE_STAT_DAILY)).getRecords()).hasSize(2);
|
||||
assertEquals("OK", row().getValueString("lastHealthStatus"));
|
||||
assertThat(new QueryAction().execute(new QueryInput(CdnTables.DRIFT_FINDING)).getRecords()).hasSize(1);
|
||||
assertNull(row().getValueString("lastHealthStatus"));
|
||||
org.mockito.Mockito.verify(adapter, org.mockito.Mockito.never()).connectionTest(org.mockito.ArgumentMatchers.any());
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user