Compare commits

...
8 Commits
Author SHA1 Message Date
james.maesandClaude Opus 5.5 8b9a14df59 Merge develop into release/0.1.0 for 0.1.6 (#21 !field ignore entries) [skip ci]
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 11:33:20 -05:00
james.maes b4d08dc065 Merge pull request #21 from QRun-IO/feature/tag-rule-ignored-fields
feat(purge): "!field" entries in onlyWhenFieldsChange ignore a field
2026-10-14 11:32:41 -05:00
james.maesandClaude Opus 5.5 49c0db82f3 feat(purge): "!field" entries in onlyWhenFieldsChange ignore a field
onlyWhenFieldsChange was only an allow-list, so a rule could not say
"purge on any change except this derived counter" without listing every
rendered field. An entry starting with "!" now names an ignored field:
with only ignored entries, any other changed field still purges; with a
mixed list the allow-list applies and ignored fields never count.
Inserts and deletes are unaffected. No schema change.

For kof22: the cms_blog_post rule gets "!viewCount" so the nightly view
count refresh purges nothing (Website-Backend#206).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 11:30:17 -05:00
james.maes 675b9fb32c Merge pull request #19 from QRun-IO/feature/orb-0.6.9
ci: qqq-orb 0.6.9 (refresh SNAPSHOT dependencies)
2026-10-14 10:56:43 -05:00
james.maesandClaude Opus 5.5 440f69fc5c Merge develop into release/0.1.0 for 0.1.5 (#20 purge messaging) [skip ci]
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 10:50:48 -05:00
james.maes 37fb879f51 Merge pull request #20 from QRun-IO/fix/no-cdn-provider-honest
fix(purge): say when purges are recorded only, not sent (no live CDN)
2026-10-14 10:50:21 -05:00
james.maesandClaude Opus 5.5 ecceaf31ee fix(purge): say when purges are recorded only, not sent
With no live CDN (provider NONE, or every hostname at stage OFF) the
Purge From CDN, Purge Everything and Retry Purge actions said the
requests were "sent within seconds" while the dispatcher skipped them.
Review and Result now carry a warning line ("No CDN provider configured:
purges are recorded only, not sent." or "CDN hostnames are not live"),
from the same status logic CdnPurgeService records (sendingStatus).
The home's Connection, Health and Purges cards say the same, and the
Origin headers card no longer implies a check is scheduled.

Refs KofTwentyTwo/Website-Backend#199

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 10:39:55 -05:00
james.maesandClaude Opus 5.5 4546a17d6b ci: qqq-orb 0.6.9 (refresh SNAPSHOT dependencies)
Moves qqq-orb@0.6.8 to qqq-orb@0.6.9, which passes --update-snapshots so cached SNAPSHOT dependencies are re-checked on every build (QRun-IO/qqq-orb#22).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 10:06:19 -05:00
13 changed files with 466 additions and 26 deletions

No files matched your search

+1 -1
View File
@@ -1,7 +1,7 @@
version: 2.1
orbs:
qqq-orb: kingsrook/qqq-orb@0.6.8
qqq-orb: kingsrook/qqq-orb@0.6.9
commands:
configure-package-install:
+2
View File
@@ -10,6 +10,8 @@ All notable changes to this project are documented here. The format follows Keep
### Fixed
- Tag rules accept ignored fields in `onlyWhenFieldsChange`, such as `!viewCount`, so counter-only updates do not purge rendered pages. Plain allow-lists still work, and ignored fields take precedence in mixed lists. No schema change is needed.
- Purge actions warn when providers are NONE or hostnames are OFF, and report requests as recorded but not sent. The home explains recorded-only purges and that this version does not run origin header checks.
- Tag rule `{prefix.field}` placeholders now resolve the joined table from the `<prefix>Id` field's possible-value source, so prefixed tables (`cms_blog`) work; a missing joined table is logged instead of silently skipped.
## [0.1.0] - Unreleased
+30 -1
View File
@@ -31,7 +31,21 @@ Design: QRun-IO/qqq `docs/specs/2026-10-12-qbit-cdn-design.md`.
NONE. See `docs/cloudfront-expected-config.md`.
- **Permissions:** `cdn.view`, `cdn.purge`, `cdn.purgeAll`, `cdn.admin`.
## Purge-path placeholders
## Tag rules
`cdn_table_tag_rule.onlyWhenFieldsChange` is a comma-separated list of QQQ field
names. Plain entries (for example `title,slug`) purge only when at least one
listed field changes. Prefix a field with `!` to ignore it: `!viewCount` prevents
a view-count-only update from purging, while a title change still purges.
With `!viewCount, !lastIndexedAt`, any change outside those fields counts.
Mixed lists keep the allow-list and exclude ignored fields, even if also listed
plainly: `title,viewCount,!viewCount` counts only title changes.
Names are trimmed and case-sensitive. Partial updates retain omitted old values;
only unequal old and new values count. This filter does not restrict inserts or
deletes. The separate `publishedField` filter still applies to all changes.
### Purge-path placeholders
`cdn_table_tag_rule` templates (`tagTemplates`, `invalidationPaths`) can use `{field}` for the changed record's own fields and `{prefix.field}` for a joined record, e.g. `/blog/{blog.slug}/{slug}*`.
@@ -97,3 +111,18 @@ KoF22 receiver and exact changes required.
## License
Apache License 2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).
### Before a CDN is live
With no enabled CDN provider (including providers of type `NONE`), purges are
recorded only, not sent. A real provider also needs a `LIVE` or `LOCKED` hostname
with purging enabled. Purge From CDN, Purge Everything, and Retry Purge show a
Review warning and a recorded-only Result when sending is unavailable. Requests
for enabled providers remain visible with `SKIPPED_NONE_PROVIDER` or
`SKIPPED_STAGE_OFF`; with no enabled provider, no new request row is created.
The home Connection and Health cards explain the unconfigured state, and purge
activity labels recorded-only requests when disconnected.
This version does not run origin header checks. The Origin headers card says
nothing has been checked until the host application writes results to the
standard `cdn_origin_check` table; those results continue to appear normally.
@@ -61,14 +61,14 @@ public record CdnHomeData(List<ProviderView> providers, List<HostnameView> hostn
** A provider row as shown.
*******************************************************************************/
public record ProviderView(Integer id, String name, String type, String distributionId, String dnsTarget, String originHost,
String lastHealthStatus, String lastHealthMessage, Instant lastSuccessfulTest, Long credentialAgeDays)
String lastHealthStatus, String lastHealthMessage, Instant lastSuccessfulTest, Long credentialAgeDays, Boolean enabled)
{
/*******************************************************************************
** Whether this is a real CDN (not the NONE placeholder).
*******************************************************************************/
public boolean isConnected()
{
return (!CdnProviderType.NONE.name().equals(type));
return (Boolean.TRUE.equals(enabled) && !CdnProviderType.NONE.name().equals(type));
}
}
@@ -119,7 +119,7 @@ public record CdnHomeData(List<ProviderView> providers, List<HostnameView> hostn
LocalDate rotated = ValueUtils.getValueAsLocalDate(row.getValue("credentialRotatedDate"));
providers.add(new ProviderView(id, row.getValueString("name"), row.getValueString("providerType"), row.getValueString("zoneOrDistributionId"),
row.getValueString("dnsTarget"), row.getValueString("originHost"), row.getValueString("lastHealthStatus"), row.getValueString("lastHealthMessage"),
ValueUtils.getValueAsInstant(row.getValue("lastSuccessfulTestDate")), rotated == null ? null : ChronoUnit.DAYS.between(rotated, today)));
ValueUtils.getValueAsInstant(row.getValue("lastSuccessfulTestDate")), rotated == null ? null : ChronoUnit.DAYS.between(rotated, today), row.getValueBoolean("enabled")));
if(CdnProviderType.CLOUDFRONT.name().equals(row.getValueString("providerType")))
{
PathPlannerSettings settings = PathPlannerSettings.from(CdnPurgeDispatcher.configOf(row));
@@ -80,7 +80,7 @@ public final class CdnHomeView
String text = "All CDN checks are OK.";
if("NOT_CONNECTED".equals(state))
{
text = "No CDN connected yet.";
text = "No CDN provider configured: purges are recorded only.";
}
else if("ATTENTION".equals(state))
{
@@ -99,7 +99,7 @@ public final class CdnHomeView
open(html, "Connection");
if(!data.isConnected())
{
empty(html, "No CDN connected yet. The CloudFront distribution is created in the cutover window.");
empty(html, "No CDN provider configured: purges are recorded only. The CloudFront distribution is created in the cutover window.");
}
for(CdnHomeData.ProviderView provider : data.providers())
{
@@ -182,6 +182,7 @@ public final class CdnHomeView
Instant created = ValueUtils.getValueAsInstant(row.getValue("createDate"));
html.append("<li><a href=\"").append(esc(appPath + "/" + CdnTables.PURGE_REQUEST + "/" + row.getValue("id"))).append("\">#")
.append(esc(String.valueOf(row.getValue("id")))).append("</a> ").append(esc(row.getValueString("status")))
.append(data.isConnected() ? "" : " - recorded only (not sent)")
.append(created == null ? "" : ", " + WHEN.format(created)).append("</li>");
}
html.append("</ul>");
@@ -224,7 +225,7 @@ public final class CdnHomeView
open(html, "Origin headers");
if(data.originChecks().isEmpty())
{
empty(html, "No origin header check has run yet.");
empty(html, "Origin header checks are not run by this version; nothing has been checked.");
}
for(QRecord check : data.originChecks())
{
@@ -24,6 +24,7 @@ import com.kingsrook.qbits.cdn.spi.CdnProviderType;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
@@ -273,7 +274,8 @@ public final class CdnTables
.withField(new QFieldMetaData("tableName", QFieldType.STRING).withIsRequired(true).withMaxLength(100))
.withField(new QFieldMetaData("tagTemplates", QFieldType.TEXT))
.withField(new QFieldMetaData("invalidationPaths", QFieldType.TEXT))
.withField(new QFieldMetaData("onlyWhenFieldsChange", QFieldType.STRING).withMaxLength(1000))
.withField(new QFieldMetaData("onlyWhenFieldsChange", QFieldType.STRING).withMaxLength(1000)
.withHelpContent(new QHelpContent("Comma-separated, case-sensitive field names. Plain entries allow only those changes; !viewCount ignores viewCount changes. With only ignored entries, any other changed field counts. In mixed lists, ignored fields never count. Inserts and deletes are unaffected by this filter.")))
.withField(new QFieldMetaData("publishedField", QFieldType.STRING).withMaxLength(100))
.withField(new QFieldMetaData("enabled", QFieldType.BOOLEAN).withDefaultValue(true)));
}
@@ -167,6 +167,56 @@ public final class CdnProcesses
/*******************************************************************************
** Summary counts for requests that cannot be sent, shared by all actions.
*******************************************************************************/
private static class UnsentSummary
{
private final ProcessSummaryLine absent = line(Status.WARNING,
"record has no enabled CDN provider", "records have no enabled CDN provider",
"record had no enabled CDN provider (no purge request recorded)", "records had no enabled CDN provider (no purge requests recorded)");
private final ProcessSummaryLine none = line(Status.WARNING,
"No CDN provider configured: purges are recorded only, not sent.", "No CDN provider configured: purges are recorded only, not sent.",
"purge request was recorded (not sent: no CDN provider)", "purge requests were recorded (not sent: no CDN provider)");
private final ProcessSummaryLine off = line(Status.WARNING,
"CDN hostnames are not live: purges are recorded only, not sent.", "CDN hostnames are not live: purges are recorded only, not sent.",
"purge request was recorded (not sent: CDN hostnames are not live)", "purge requests were recorded (not sent: CDN hostnames are not live)");
/*******************************************************************************
** Count a skipped request; returns whether it cannot be sent.
*******************************************************************************/
boolean add(CdnPurgeStatus status)
{
if(status == CdnPurgeStatus.SKIPPED_NONE_PROVIDER)
{
none.incrementCount(1);
return (true);
}
if(status == CdnPurgeStatus.SKIPPED_STAGE_OFF)
{
off.incrementCount(1);
return (true);
}
return (false);
}
/*******************************************************************************
** Include only reasons encountered by this action.
*******************************************************************************/
void append(ArrayList<ProcessSummaryLineInterface> lines)
{
absent.addSelfToListIfAnyCount(lines);
none.addSelfToListIfAnyCount(lines);
off.addSelfToListIfAnyCount(lines);
}
}
/*******************************************************************************
** Scheduled: send due purges and poll in-progress ones.
*******************************************************************************/
@@ -189,6 +239,7 @@ public final class CdnProcesses
*******************************************************************************/
public static class PurgeRecordsTransform extends AbstractTransformStep implements ProcessSummaryProviderInterface
{
private final UnsentSummary unsent = new UnsentSummary();
private final ProcessSummaryLine records = line(Status.OK, "record will be purged", "records will be purged",
"record was purged", "records were purged");
private final ProcessSummaryLine noRule = line(Status.INFO, "record has no CDN tag rule (no change)", "records have no CDN tag rule (no change)",
@@ -207,6 +258,7 @@ public final class CdnProcesses
String table = input.getValueString(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE);
List<QRecord> rules = rules(table);
TagRuleEngine engine = new TagRuleEngine();
CdnPurgeStatus sending = CdnPurgeService.sendingStatus(null);
for(QRecord record : input.getRecords())
{
List<String> tags = new ArrayList<>();
@@ -226,7 +278,10 @@ public final class CdnProcesses
recordPaths.forEach(path -> paths.add(path.path()));
record.setValue("cdnTags", String.join("\n", tags));
record.setValue("cdnPaths", com.kingsrook.qbits.cdn.purge.PurgeText.paths(recordPaths));
records.incrementCountAndAddPrimaryKey(record.getValue("id"));
if(!unsent.add(sending))
{
records.incrementCountAndAddPrimaryKey(record.getValue("id"));
}
output.getRecords().add(record);
}
}
@@ -249,6 +304,7 @@ public final class CdnProcesses
lines.add(pathLine);
}
noRule.addSelfToListIfAnyCount(lines);
unsent.append(lines);
return (lines);
}
}
@@ -256,10 +312,11 @@ public final class CdnProcesses
/*******************************************************************************
** Purge From CDN, execute: one purge request (sent at once, hubs included).
** Purge From CDN, execute: record requests with hubs included.
*******************************************************************************/
public static class PurgeRecordsLoad extends AbstractLoadStep implements ProcessSummaryProviderInterface
{
private final UnsentSummary unsent = new UnsentSummary();
private final ProcessSummaryLine requested = line(Status.OK, "purge request will be recorded", "purge requests will be recorded",
"purge request was recorded (sent within seconds)", "purge requests were recorded (sent within seconds)");
@@ -277,7 +334,17 @@ public final class CdnProcesses
List<QRecord> rows = CdnPurgeService.enqueue(new CdnPurgeService.Request(CdnPurgeKind.TAG.name(),
com.kingsrook.qbits.cdn.purge.PurgeText.tags(record.getValueString("cdnTags")),
com.kingsrook.qbits.cdn.purge.PurgeText.paths(record.getValueString("cdnPaths")), true, CdnPurgeSource.RECORD_ACTION, table, record.getValue("id")));
requested.incrementCount(rows.size());
if(rows.isEmpty())
{
unsent.absent.incrementCount(1);
}
for(QRecord row : rows)
{
if(!unsent.add(CdnPurgeStatus.valueOf(row.getValueString("status"))))
{
requested.incrementCount(1);
}
}
output.getRecords().add(record);
}
}
@@ -296,6 +363,7 @@ public final class CdnProcesses
{
transform.noRule.addSelfToListIfAnyCount(lines);
}
unsent.append(lines);
return (lines);
}
}
@@ -307,6 +375,7 @@ public final class CdnProcesses
*******************************************************************************/
public static class RetryTransform extends AbstractTransformStep implements ProcessSummaryProviderInterface
{
private final UnsentSummary unsent = new UnsentSummary();
private final ProcessSummaryLine retry = line(Status.OK, "purge request will be sent again", "purge requests will be sent again",
"purge request was queued again", "purge requests were queued again");
private final ProcessSummaryLine skip = line(Status.INFO, "purge request has not failed (no change)", "purge requests have not failed (no change)",
@@ -325,7 +394,10 @@ public final class CdnProcesses
String status = record.getValueString("status");
if(CdnPurgeStatus.FAILED.name().equals(status) || CdnPurgeStatus.DEAD.name().equals(status) || CdnPurgeStatus.DEFERRED_BUDGET.name().equals(status))
{
retry.incrementCountAndAddPrimaryKey(record.getValue("id"));
if(!unsent.add(CdnPurgeService.sendingStatus(record.getValue("cdnProviderId"))))
{
retry.incrementCountAndAddPrimaryKey(record.getValue("id"));
}
output.getRecords().add(record);
}
else
@@ -346,6 +418,7 @@ public final class CdnProcesses
ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
retry.addSelfToListIfAnyCount(lines);
skip.addSelfToListIfAnyCount(lines);
unsent.append(lines);
return (lines);
}
}
@@ -353,10 +426,16 @@ public final class CdnProcesses
/*******************************************************************************
** Retry Purge, execute: back to PENDING (sent on the next dispatcher pass).
** Retry Purge, execute: queue live providers or record why sending is skipped.
*******************************************************************************/
public static class RetryLoad extends AbstractLoadStep
public static class RetryLoad extends AbstractLoadStep implements ProcessSummaryProviderInterface
{
private final UnsentSummary unsent = new UnsentSummary();
private final ProcessSummaryLine requested = line(Status.OK, "purge request will be recorded", "purge requests will be recorded",
"purge request was queued again", "purge requests were queued again");
/*******************************************************************************
** Reset the rows.
*******************************************************************************/
@@ -366,7 +445,12 @@ public final class CdnProcesses
List<QRecord> updates = new ArrayList<>();
for(QRecord record : input.getRecords())
{
updates.add(new QRecord().withValue("id", record.getValue("id")).withValue("status", CdnPurgeStatus.PENDING.name())
CdnPurgeStatus status = CdnPurgeService.sendingStatus(record.getValue("cdnProviderId"));
if(!unsent.add(status))
{
requested.incrementCount(1);
}
updates.add(new QRecord().withValue("id", record.getValue("id")).withValue("status", status.name())
.withValue("source", CdnPurgeSource.RETRY.name()).withValue("attempts", 0).withValue("nextAttemptDate", null)
.withValue("errorMessage", null).withValue("completedDate", null).withValue("modifyDate", CdnRuntime.clock().instant()));
}
@@ -375,6 +459,24 @@ public final class CdnProcesses
output.getRecords().addAll(new UpdateAction().execute(new UpdateInput(CdnTables.PURGE_REQUEST).withRecords(updates)).getRecords());
}
}
/*******************************************************************************
** Result lines from the requests recorded during execution.
*******************************************************************************/
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
{
ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
requested.addSelfToListIfAnyCount(lines);
if(getTransformStep() instanceof RetryTransform transform)
{
transform.skip.addSelfToListIfAnyCount(lines);
}
unsent.append(lines);
return (lines);
}
}
@@ -384,6 +486,7 @@ public final class CdnProcesses
*******************************************************************************/
public static class PurgeHostnameTransform extends AbstractTransformStep implements ProcessSummaryProviderInterface
{
private final UnsentSummary unsent = new UnsentSummary();
private final ProcessSummaryLine providers = line(Status.WARNING,
"CDN distribution will be purged completely (every hostname on it)", "CDN distributions will be purged completely (every hostname on them)",
"CDN distribution purge was recorded", "CDN distribution purges were recorded");
@@ -402,7 +505,10 @@ public final class CdnProcesses
Serializable providerId = record.getValue("cdnProviderId");
if(providerId != null && seen.add(providerId))
{
providers.incrementCountAndAddPrimaryKey(record.getValue("id"));
if(!unsent.add(CdnPurgeService.sendingStatus(providerId)))
{
providers.incrementCountAndAddPrimaryKey(record.getValue("id"));
}
output.getRecords().add(record);
}
}
@@ -418,6 +524,7 @@ public final class CdnProcesses
{
ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
providers.addSelfToListIfAnyCount(lines);
unsent.append(lines);
return (lines);
}
}
@@ -427,8 +534,14 @@ public final class CdnProcesses
/*******************************************************************************
** Purge Everything, execute.
*******************************************************************************/
public static class PurgeHostnameLoad extends AbstractLoadStep
public static class PurgeHostnameLoad extends AbstractLoadStep implements ProcessSummaryProviderInterface
{
private final UnsentSummary unsent = new UnsentSummary();
private final ProcessSummaryLine requested = line(Status.OK, "purge request will be recorded", "purge requests will be recorded",
"CDN distribution purge was recorded", "CDN distribution purges were recorded");
/*******************************************************************************
** Record an ALL purge for each provider.
*******************************************************************************/
@@ -437,11 +550,36 @@ public final class CdnProcesses
{
for(QRecord record : input.getRecords())
{
CdnPurgeService.enqueueFor(record.getValue("cdnProviderId"), new CdnPurgeService.Request(CdnPurgeKind.ALL.name(), List.of(), List.of(), true,
List<QRecord> rows = CdnPurgeService.enqueueFor(record.getValue("cdnProviderId"), new CdnPurgeService.Request(CdnPurgeKind.ALL.name(), List.of(), List.of(), true,
CdnPurgeSource.MANUAL, CdnTables.HOSTNAME, record.getValue("id")));
if(rows.isEmpty())
{
unsent.absent.incrementCount(1);
}
for(QRecord row : rows)
{
if(!unsent.add(CdnPurgeStatus.valueOf(row.getValueString("status"))))
{
requested.incrementCount(1);
}
}
output.getRecords().add(record);
}
}
/*******************************************************************************
** Result lines from the requests recorded during execution.
*******************************************************************************/
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen)
{
ArrayList<ProcessSummaryLineInterface> lines = new ArrayList<>();
requested.addSelfToListIfAnyCount(lines);
unsent.append(lines);
return (lines);
}
}
@@ -120,6 +120,36 @@ public final class CdnPurgeService
/*******************************************************************************
** Sending eligibility for one provider, or all enabled providers when null.
** A pending status means at least one provider can send; otherwise the
** status gives the same reason used when recording a new request.
*******************************************************************************/
public static CdnPurgeStatus sendingStatus(Serializable providerId) throws QException
{
QQueryFilter filter = new QQueryFilter(new QFilterCriteria("enabled", QCriteriaOperator.EQUALS, true));
if(providerId != null)
{
filter.addCriteria(new QFilterCriteria("id", QCriteriaOperator.EQUALS, providerId));
}
CdnPurgeStatus status = CdnPurgeStatus.SKIPPED_NONE_PROVIDER;
for(QRecord provider : new QueryAction().execute(new QueryInput(CdnTables.PROVIDER).withFilter(filter)).getRecords())
{
CdnPurgeStatus candidate = initialStatus(provider);
if(candidate == CdnPurgeStatus.PENDING)
{
return (candidate);
}
if(candidate == CdnPurgeStatus.SKIPPED_STAGE_OFF)
{
status = candidate;
}
}
return (status);
}
/*******************************************************************************
** Status of a new request for a provider.
*******************************************************************************/
@@ -55,8 +55,8 @@ import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair;
** expires by TTL instead of being invalidated;
** - publishedField: when set, only records that are (or were) published
** count, so draft edits purge nothing;
** - onlyWhenFieldsChange: comma list; an update that changes none of them
** purges nothing.
** - onlyWhenFieldsChange: comma list of watched fields; !field ignores a
** field. With only ignored fields, any other changed field counts.
**
** A dotted placeholder "{blog.slug}" reads field "slug" of the record that
** field "blogId" points to (the xId convention). The joined table is the one
@@ -159,7 +159,27 @@ public class TagRuleEngine
String watched = rule.getValueString("onlyWhenFieldsChange");
if(current != null && old != null && StringUtils.hasContent(watched))
{
return (Arrays.stream(watched.split(",")).map(String::strip).filter(StringUtils::hasContent)
Set<String> included = new LinkedHashSet<>();
Set<String> ignored = new LinkedHashSet<>();
for(String entry : watched.split(","))
{
String field = entry.strip();
if(field.startsWith("!"))
{
ignored.add(field.substring(1).strip());
}
else if(StringUtils.hasContent(field))
{
included.add(field);
}
}
if(included.isEmpty() && !ignored.isEmpty())
{
included.addAll(current.getValues().keySet());
included.addAll(old.getValues().keySet());
}
included.removeAll(ignored);
return (included.stream()
.anyMatch(field -> !Objects.equals(current.getValue(field), old.getValue(field))));
}
return (true);
@@ -58,10 +58,10 @@ class CdnHomeTest
{
CdnTestSupport.setUp();
String html = render(CdnWidgets.HOME_WIDGET);
assertThat(html).contains("No CDN connected yet.").contains("No edge statistics yet.").contains("No purges recorded yet.")
.contains("No path-billed provider connected yet.").contains("No origin header check has run yet.").contains("Nothing to compare yet.")
assertThat(html).contains("No CDN provider configured: purges are recorded only.").contains("No edge statistics yet.").contains("No purges recorded yet.")
.contains("No path-billed provider connected yet.").contains("Origin header checks are not run by this version; nothing has been checked.").contains("Nothing to compare yet.")
.contains("minmax(min(100%,22rem),1fr)");
assertThat(render(CdnWidgets.HEALTH_CARD_WIDGET)).contains("No CDN connected yet.").contains("href=\"/cdnAdmin\"");
assertThat(render(CdnWidgets.HEALTH_CARD_WIDGET)).contains("No CDN provider configured: purges are recorded only.").contains("href=\"/cdnAdmin\"");
}
@@ -99,7 +99,7 @@ class CdnHomeTest
String html = CdnHomeView.render(data, "/cdnAdmin");
assertThat(html).contains("d1.cloudfront.net").contains(age + " days (rotate every 90) - rotation due").contains("1,500").contains("91.5%")
.contains("<polyline").contains("396 of 396").contains("BEHAVIOR").contains("1 open").contains("/cdnAdmin/cdn_purge_request/")
.contains("www.example.com");
.contains("www.example.com").doesNotContain("recorded only (not sent)", "nothing has been checked");
assertThat(CdnHomeView.renderHealthCard(data, "/cdnAdmin")).contains("Needs attention");
}
@@ -119,4 +119,31 @@ class CdnHomeTest
assertEquals("50.0%", CdnHomeView.percent(new BigDecimal("0.5")));
assertEquals("91.5%", CdnHomeView.percent(new BigDecimal("91.5")));
}
/*******************************************************************************
** NONE and disabled providers do not imply a connection or a sent purge.
*******************************************************************************/
@Test
void recordedOnlyHome() throws Exception
{
for(boolean none : List.of(true, false))
{
CdnTestSupport.setUp();
Integer provider = CdnTestSupport.liveProvider(none ? CdnProviderType.NONE : CdnProviderType.CLOUDFRONT, null);
if(!none)
{
new com.kingsrook.qqq.backend.core.actions.tables.UpdateAction().execute(
new com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput(CdnTables.PROVIDER)
.withRecord(new QRecord().withValue("id", provider).withValue("enabled", false)));
}
CdnTestSupport.insert(CdnTables.PURGE_REQUEST, new QRecord().withValue("cdnProviderId", provider).withValue("status", "SKIPPED_NONE_PROVIDER"));
assertThat(render(CdnWidgets.HOME_WIDGET)).contains("No CDN provider configured: purges are recorded only.")
.contains("The CloudFront distribution is created in the cutover window.").contains("recorded only (not sent)")
.contains("Origin header checks are not run by this version; nothing has been checked.");
assertThat(render(CdnWidgets.HEALTH_CARD_WIDGET)).contains("No CDN provider configured: purges are recorded only.");
}
}
}
@@ -70,6 +70,16 @@ class CdnProcessesTest
RunProcessOutput review = new RunProcessAction().execute(input);
List<ProcessSummaryLineInterface> reviewLines = (List<ProcessSummaryLineInterface>) review.getValues().get(StreamedETLWithFrontendProcess.FIELD_VALIDATION_SUMMARY);
for(ProcessSummaryLineInterface reviewLine : reviewLines)
{
ProcessSummaryLine summary = (ProcessSummaryLine) reviewLine;
summary.pickMessage(false);
if(summary.getMessage().contains("purges are recorded only"))
{
assertEquals(com.kingsrook.qqq.backend.core.model.actions.processes.Status.WARNING, summary.getStatus());
}
}
RunProcessInput go = new RunProcessInput();
go.setProcessName(process);
go.setProcessUUID(review.getProcessUUID());
@@ -137,6 +147,7 @@ class CdnProcessesTest
new QRecord().withValue("cdnProviderId", 1).withValue("status", "SUCCEEDED"));
List<List<String>> lines = run(CdnProcesses.RETRY_PURGE, rows.get(0).getValue("id") + "," + rows.get(1).getValue("id"));
assertThat(lines.get(0)).containsExactly("1 purge request will be sent again", "1 purge request has not failed (no change)");
assertThat(lines.get(1)).contains("1 purge request was queued again");
QRecord retried = purges().get(0);
assertEquals("PENDING", retried.getValueString("status"));
assertEquals(0, retried.getValueInteger("attempts"));
@@ -155,9 +166,99 @@ class CdnProcessesTest
CdnTestSupport.insert(CdnTables.PROVIDER, new QRecord().withValue("name", "other").withValue("providerType", "CLOUDFRONT").withValue("enabled", true));
List<List<String>> lines = run(CdnProcesses.PURGE_HOSTNAME, "1");
assertThat(lines.get(0)).containsExactly("1 CDN distribution will be purged completely (every hostname on it)");
assertThat(lines.get(1)).containsExactly("1 CDN distribution purge was recorded");
List<QRecord> rows = purges();
assertThat(rows).hasSize(1);
assertEquals("ALL", rows.get(0).getValueString("kind"));
assertEquals(1, rows.get(0).getValueInteger("cdnProviderId"));
}
/*******************************************************************************
** Every action reports NONE and OFF honestly, including plural results.
*******************************************************************************/
@Test
void recordedOnlySummaries() throws Exception
{
for(boolean none : List.of(true, false))
{
for(String process : List.of(CdnProcesses.PURGE_PREFIX + "post", CdnProcesses.PURGE_HOSTNAME, CdnProcesses.RETRY_PURGE))
{
setUp();
String table = none ? CdnTables.PROVIDER : CdnTables.HOSTNAME;
new com.kingsrook.qqq.backend.core.actions.tables.UpdateAction().execute(
new com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput(table)
.withRecord(new QRecord().withValue("id", 1).withValue(none ? "providerType" : "stage", none ? "NONE" : "OFF")));
String ids = "1";
if(process.equals(CdnProcesses.PURGE_PREFIX + "post"))
{
CdnTestSupport.insert("post", new QRecord().withValue("slug", "one"), new QRecord().withValue("slug", "two"));
ids = "1,2";
}
else if(process.equals(CdnProcesses.RETRY_PURGE))
{
CdnTestSupport.insert(CdnTables.PURGE_REQUEST,
new QRecord().withValue("cdnProviderId", 1).withValue("status", "FAILED"),
new QRecord().withValue("cdnProviderId", 1).withValue("status", "DEAD"));
ids = "1,2";
}
List<List<String>> lines = run(process, ids);
String warning = none ? "No CDN provider configured" : "CDN hostnames are not live";
assertThat(lines.get(0)).anyMatch(line -> line.contains(warning + ": purges are recorded only, not sent."));
assertThat(lines.get(0)).noneMatch(line -> line.contains("will be purged") || line.contains("will be sent again"));
String reason = none ? "no CDN provider" : "CDN hostnames are not live";
assertThat(lines.get(1)).containsExactly(process.equals(CdnProcesses.PURGE_HOSTNAME)
? "1 purge request was recorded (not sent: " + reason + ")"
: "2 purge requests were recorded (not sent: " + reason + ")");
assertThat(purges()).allMatch(row -> (none ? "SKIPPED_NONE_PROVIDER" : "SKIPPED_STAGE_OFF").equals(row.getValueString("status")));
}
}
}
/*******************************************************************************
** Eligibility considers enabled providers and their purge-enabled hosts.
*******************************************************************************/
@Test
void sendingEligibility() throws Exception
{
CdnTestSupport.setUp();
assertEquals(com.kingsrook.qbits.cdn.model.CdnPurgeStatus.SKIPPED_NONE_PROVIDER,
com.kingsrook.qbits.cdn.purge.CdnPurgeService.sendingStatus(null));
CdnTestSupport.liveProvider(CdnProviderType.NONE, null);
Integer live = CdnTestSupport.liveProvider(CdnProviderType.CLOUDFRONT, null);
assertEquals(com.kingsrook.qbits.cdn.model.CdnPurgeStatus.PENDING,
com.kingsrook.qbits.cdn.purge.CdnPurgeService.sendingStatus(null));
new com.kingsrook.qqq.backend.core.actions.tables.UpdateAction().execute(
new com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput(CdnTables.HOSTNAME)
.withRecord(new QRecord().withValue("id", 2).withValue("purgeEnabled", false)));
assertEquals(com.kingsrook.qbits.cdn.model.CdnPurgeStatus.SKIPPED_STAGE_OFF,
com.kingsrook.qbits.cdn.purge.CdnPurgeService.sendingStatus(live));
new com.kingsrook.qqq.backend.core.actions.tables.UpdateAction().execute(
new com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput(CdnTables.PROVIDER)
.withRecord(new QRecord().withValue("id", live).withValue("enabled", false)));
assertEquals(com.kingsrook.qbits.cdn.model.CdnPurgeStatus.SKIPPED_NONE_PROVIDER,
com.kingsrook.qbits.cdn.purge.CdnPurgeService.sendingStatus(null));
}
/*******************************************************************************
** Without an enabled provider, do not claim a request row was recorded.
*******************************************************************************/
@Test
void absentProvider() throws Exception
{
CdnTestSupport.setUp();
CdnTestSupport.postRule();
CdnTestSupport.insert("post", new QRecord().withValue("slug", "one"));
List<List<String>> lines = run(CdnProcesses.PURGE_PREFIX + "post", "1");
assertThat(lines.get(0)).contains("1 No CDN provider configured: purges are recorded only, not sent.");
assertThat(lines.get(1)).containsExactly("1 record had no enabled CDN provider (no purge request recorded)");
assertThat(purges()).isEmpty();
}
}
@@ -28,11 +28,14 @@ import com.kingsrook.qbits.cdn.spi.CdnProviderType;
import com.kingsrook.qbits.cdn.spi.CdnPurgeKind;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
@@ -138,6 +141,32 @@ class PurgePipelineTest
/*******************************************************************************
** The record-change listener skips counter updates but records title changes.
*******************************************************************************/
@Test
void ignoredCounterChangeRecordsNoPurge() throws Exception
{
QContext.getQInstance().getTable("post").withField(new QFieldMetaData("viewCount", QFieldType.INTEGER));
CdnTestSupport.liveProvider(CdnProviderType.CLOUDFRONT, null);
QRecord rule = new QueryAction().execute(new QueryInput(CdnTables.TABLE_TAG_RULE)).getRecords().get(0);
new UpdateAction().execute(new UpdateInput(CdnTables.TABLE_TAG_RULE).withRecord(new QRecord().withValue("id", rule.getValue("id"))
.withValue("onlyWhenFieldsChange", "!viewCount")));
QRecord post = CdnTestSupport.insert("post", new QRecord().withValue("slug", "counter-post").withValue("title", "Old")
.withValue("blogId", 1).withValue("isPublished", true).withValue("viewCount", 10)).get(0);
assertThat(purges()).hasSize(1);
new UpdateAction().execute(new UpdateInput("post").withRecord(new QRecord().withValue("id", post.getValue("id")).withValue("viewCount", 11)));
assertThat(purges()).as("counter-only update adds no purge request").hasSize(1);
new UpdateAction().execute(new UpdateInput("post").withRecord(new QRecord().withValue("id", post.getValue("id")).withValue("title", "New")));
assertThat(purges()).hasSize(2);
assertThat(purges().get(1).getValueString("paths")).contains("/blog/quick-bits/counter-post*");
assertThat(purges().get(1).getValueString("tags")).contains("post:" + post.getValue("id"));
}
/*******************************************************************************
** Retryable failures back off and end DEAD; permanent ones go DEAD at once.
*******************************************************************************/
@@ -135,6 +135,67 @@ class TagRuleEngineTest
/*******************************************************************************
** Ignore-only rules compare actual values after merging partial updates.
*******************************************************************************/
@Test
void ignoresCounterOnlyUpdates() throws Exception
{
QRecord rule = new QRecord().withValue("onlyWhenFieldsChange", " !viewCount, !lastIndexedAt ").withValue("tagTemplates", "post:{id}");
QRecord old = new QRecord().withValue("id", 1).withValue("title", "Old").withValue("viewCount", 10).withValue("lastIndexedAt", "yesterday");
TagRuleEngine engine = new TagRuleEngine();
assertTrue(engine.evaluate(rule, List.of(new QRecord(old)), List.of(old)).isEmpty());
assertTrue(engine.evaluate(rule, List.of(new QRecord().withValue("id", 1).withValue("viewCount", 11).withValue("lastIndexedAt", "today")), List.of(old)).isEmpty());
assertTrue(engine.evaluate(rule, List.of(new QRecord().withValue("id", 1).withValue("title", "Old")), List.of(old)).isEmpty());
assertThat(engine.evaluate(rule, List.of(new QRecord().withValue("id", 1).withValue("title", "New").withValue("viewCount", 11)), List.of(old)).tags()).containsExactly("post:1");
assertFalse(engine.evaluate(rule, List.of(new QRecord().withValue("id", 1).withValue("title", null)), List.of(old)).isEmpty());
assertFalse(engine.evaluate(rule, List.of(new QRecord().withValue("id", 1).withValue("newField", "value")), List.of(old)).isEmpty());
assertFalse(engine.evaluate(rule, List.of(new QRecord().withValue("id", 1).withValue("ViewCount", 11)), List.of(old)).isEmpty());
}
/*******************************************************************************
** Plain lists retain allow-list behavior; ignored entries always win.
*******************************************************************************/
@Test
void appliesAllowListAndMixedRules()
{
QRecord old = new QRecord().withValue("title", "Old").withValue("viewCount", 10);
QRecord counterChange = new QRecord(old).withValue("viewCount", 11);
QRecord titleChange = new QRecord(old).withValue("title", "New");
QRecord rule = new QRecord().withValue("onlyWhenFieldsChange", " title, ,slug ");
assertFalse(TagRuleEngine.counts(rule, counterChange, old));
assertTrue(TagRuleEngine.counts(rule, titleChange, old));
assertFalse(TagRuleEngine.counts(rule, new QRecord(old), old));
rule.setValue("onlyWhenFieldsChange", "title, viewCount, !viewCount");
assertFalse(TagRuleEngine.counts(rule, counterChange, old));
assertTrue(TagRuleEngine.counts(rule, titleChange, old));
assertFalse(TagRuleEngine.counts(rule, new QRecord(old).withValue("slug", "new"), old));
rule.setValue("onlyWhenFieldsChange", "viewCount,!viewCount");
assertFalse(TagRuleEngine.counts(rule, titleChange, old));
assertFalse(TagRuleEngine.counts(rule, counterChange, old));
}
/*******************************************************************************
** Changed-field filters never suppress inserts or deletes.
*******************************************************************************/
@Test
void insertsAndDeletesIgnoreFieldFilters() throws Exception
{
QRecord post = new QRecord().withValue("id", 1).withValue("viewCount", 10);
for(String fields : List.of("!viewCount", "title", "title,!viewCount"))
{
QRecord rule = new QRecord().withValue("onlyWhenFieldsChange", fields).withValue("tagTemplates", "post:{id}");
assertThat(new TagRuleEngine().evaluate(rule, List.of(post), List.of()).tags()).containsExactly("post:1");
assertThat(new TagRuleEngine().evaluate(rule, List.of(), List.of(post)).tags()).containsExactly("post:1");
}
}
/*******************************************************************************
** Stored text round-trips; invalid lines are dropped.
*******************************************************************************/