Compare commits

...
12 Commits
Author SHA1 Message Date
james.maesandClaude Opus 5.5 0092574b6e Merge develop into release/0.1.0 for 0.1.4 (#18 QQQ RC.6, parent 2.1.0, guardrails) [skip ci]
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 07:26:42 -05:00
james.maes fb4a536494 Merge pull request #18 from QRun-IO/feature/guardrails-2.1.0
build: QQQ 4.1.0-RC.6 + guardrails (parent 2.1.0, enforced)
2026-10-14 07:21:58 -05:00
james.maesandClaude Opus 5.5 967c52f8bc build: QQQ 4.1.0-RC.6 and qbit-build-parent 2.1.0 with guardrails enforced
Move the core baseline to the released 4.1.0-RC.6 (architecture review
finding F19, guardrail G7). Adopt qbit-build-parent 2.1.0 with the
architecture guardrails enforced (G1, G3, G4, G7) and tests in strict
raw-route mode (G2). Everything passes with no source changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 07:18:29 -05:00
james.maesandClaude Opus 5.5 253f862ecd Merge develop into release/0.1.0 for 0.1.3 (core signer, signed revalidation hooks, #16) [skip ci]
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:53:00 -05:00
james.maes 5801ff2ebd Merge pull request #16 from QRun-IO/feature/arch-core-signer
feat(revalidation): core WebhookSigner + SafeHttpClient for hooks (#15)
2026-10-13 19:36:55 -05:00
james.maesandClaude Opus 5.5 7510392166 feat(revalidation): sign hooks with core WebhookSigner and the safe HTTP client
Replace the hand-rolled HMAC in RevalidationHookRunner with core
WebhookSigner (Standard Webhooks) from QQQ 4.1.0-RC.5, and send through
core SafeHttpClient.

New cdn_revalidation_hook fields:
- signatureFormat: STANDARD_WEBHOOKS (the default for new rows) or LEGACY,
  which keeps X-Revalidate-Signature byte-for-byte. A null format, as on
  existing rows, stays LEGACY.
- allowPlainHttp and internalAllowlist: explicit exceptions to the SSRF
  policy.

Closes #15 (architecture review QRun-IO/qqq#1150, finding F11).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 19:33:36 -05:00
james.maes 59fc98ccb2 Merge pull request #14 from QRun-IO/feature/release-rc-automation
ci: automate RC releases; hotfix/* pushes no longer publish
2026-10-13 14:43:05 -05:00
james.maesandClaude Opus 5.5 1e2af2acf1 ci: automate RC releases; hotfix/* pushes no longer publish
Removes publish_hotfix_release: a hotfix/* branch push published a GA
release to Maven Central with no tag or review (QRun-IO/qbit-cdn#12,
QRun-IO/qqq#1121). hotfix/* branches now run test_only; a hotfix
release publishes from a reviewed vX.Y.Z tag through publish_release.
Adds .github/workflows/release-rc.yml (reusable QRun-IO/.github
qbit-release-rc): a manual dispatch, or merging a develop PR labelled
release:rc, merges develop into the release line and the release/* push
runs the existing CircleCI RC publish.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 14:29:07 -05:00
james.maesandClaude Opus 5.5 2c41409fb2 Merge develop into release/0.1.0 for 0.1.2 [skip ci]
AWS SDK for Java v2 2.41.10 -> 2.41.34 (GHSA-443w-3rq3-5m5h, #13) on
top of 0.1.1 (TagRuleEngine PVS fix, #10). Published from the GA tag
v0.1.2 on this commit; [skip ci] so the release-branch push does not
also publish a 0.1.0 RC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 10:41:57 -05:00
james.maes b1d7d9543f Merge pull request #13 from QRun-IO/feature/awssdk-2.41.34
build(deps): AWS SDK for Java v2 2.41.34 (GHSA-443w-3rq3-5m5h)
2026-10-13 10:40:35 -05:00
james.maesandClaude Opus 5.5 15439d2fba build(deps): AWS SDK for Java v2 2.41.10 -> 2.41.34 (GHSA-443w-3rq3-5m5h)
2.41.10 carries the HIGH advisory GHSA-443w-3rq3-5m5h (flagged by a Trivy scan); 2.41.30+ fixes it. Same minor line, so no API change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 10:23:30 -05:00
CircleCI 77d2fdd55f Bump version to 0.1.0-RC.1 [skip ci] 2026-10-12 21:24:51 +00:00
12 changed files with 381 additions and 70 deletions

No files matched your search

+1 -12
View File
@@ -22,7 +22,7 @@ workflows:
context: [qqq-maven-registry-credentials]
filters:
branches:
ignore: /(develop|release\/.*|hotfix\/.*)/
ignore: /(develop|release\/.*)/
publish_snapshot:
jobs:
@@ -58,14 +58,3 @@ workflows:
ignore: /.*/
tags:
only: [/^v\d+\.\d+\.\d+$/]
publish_hotfix_release:
jobs:
- qqq-orb/mvn_publish:
pre-steps:
- configure-package-install
context: [qqq-maven-registry-credentials]
branch_type: hotfix
filters:
branches:
only: [/hotfix\/.*/]
+25
View File
@@ -0,0 +1,25 @@
# Cut the next release candidate (QRun-IO/qqq docs/QRUN-QBIT-REPO-STANDARD.md, C6):
# run this workflow by hand, or merge a develop PR labelled `release:rc`. It merges develop into
# the release line and the release/* push runs CircleCI publish_release_candidate.
name: Release candidate
on:
workflow_dispatch:
inputs:
dry_run:
description: Print the plan without merging or creating branches
type: boolean
default: false
pull_request:
types: [closed]
branches: [develop]
permissions:
contents: write
jobs:
release-rc:
if: github.event_name == 'workflow_dispatch' || (github.event.pull_request.merged == true && contains(github.event.pull_request.labels.*.name, 'release:rc'))
uses: QRun-IO/.github/.github/workflows/qbit-release-rc.yml@main
with:
dry_run: ${{ github.event_name == 'workflow_dispatch' && inputs.dry_run }}
+1 -1
View File
@@ -35,7 +35,7 @@ This file provides guidance to Codex when working with code in this repository.
## Dependencies
- Java 21.
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-RC.4` ahead of it, as in qbit-quick-search (ADR-0007). Remove that activation when the parent moves to 4.1.
- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-RC.5` ahead of it, as in qbit-quick-search (ADR-0007). Remove that activation when the parent moves to 4.1.
- Tests: see pom.xml.
## Release
+5
View File
@@ -4,6 +4,11 @@ All notable changes to this project are documented here. The format follows Keep
## [0.1.1] - Unreleased
### Changed
- Require released QQQ BOM 4.1.0-RC.5; revalidation hooks default to core Standard Webhooks signing. Per-hook `signatureFormat=LEGACY` preserves the prior header and signature bytes.
- Use core `SafeHttpClient` for revalidation requests: HTTPS/public targets by default, explicit `allowPlainHttp`/`internalAllowlist` exceptions, bounded responses and checked DNS pinning. Fan-out preserves Host/TLS identity. Host schema migration and receiver migration are described in README and `docs/revalidation-hook-migration.md`.
### Fixed
- Tag rule `{prefix.field}` placeholders now resolve the joined table from the `<prefix>Id` field's possible-value source, so prefixed tables (`cms_blog`) work; a missing joined table is logged instead of silently skipped.
+30
View File
@@ -58,6 +58,36 @@ CloudFront credentials: `<secretRef>_ACCESS_KEY_ID` / `<secretRef>_SECRET_ACCESS
Drift: the OpenTofu `expected_config` JSON in `CDN_EXPECTED_CONFIG` (or the
variable named by the provider setting `expectedConfigEnv`).
## Revalidation hooks
Requires QQQ **4.1.0-RC.5** (the default `qqq-snapshot` profile imports the released BOM;
its historical property name does not select a core SNAPSHOT).
New `cdn_revalidation_hook` rows default `signatureFormat` to `STANDARD_WEBHOOKS`;
existing rows with a null format keep the `LEGACY` signature. Signed requests use core `WebhookSigner` and
`WebhookSecret`: `webhook-id`, `webhook-timestamp`, and `webhook-signature`.
Rows on `LEGACY` (or null) keep sending
`X-Revalidate-Signature: sha256=<lowercase hex HMAC-SHA256 of the body>`.
Legacy keys remain the exact UTF-8 environment value, with no trimming or decoding.
Standard keys must contain at least 16 bytes after core parsing (trimmed raw text,
or base64-decoded `whsec_` text). `secretRef` names the environment variable;
as before, no resolved secret means no signature headers.
Core `SafeHttpClient` now enforces HTTPS and rejects internal addresses by default.
For an internal HTTP pod endpoint explicitly set `allowPlainHttp=true` and
`internalAllowlist` to its exact hostname, a `.suffix`, IP, or CIDR (comma/whitespace
separated). Metadata and unspecified addresses remain blocked. Every DNS address
is checked before fan-out; the original hostname is retained for Host/TLS, and
connections use the checked addresses without resolving again. Redirects fail,
responses are bounded to 64 KiB, and `timeoutMs` is a positive per-request total
limit (default 5000). These transport rules apply to both signature formats.
Host apps must migrate `cdn_revalidation_hook` to add `signatureFormat` (string),
`allowPlainHttp` (boolean), and `internalAllowlist` (text). Backfill `LEGACY` for
old receivers; configure internal network exceptions explicitly before dispatch.
See [receiver migration](docs/revalidation-hook-migration.md) for the inspected
KoF22 receiver and exact changes required.
## Build
```sh
+68
View File
@@ -0,0 +1,68 @@
# Revalidation hook migration (F11 / qbit-cdn #15)
Reference: core source read directly at tag `v4.1.0-RC.5`, including
`WebhookSigner`, `WebhookSecret`, `WebhookVerifier`, `SafeHttpClient`,
`SafeHttpPolicy`, and `AddressRules`. No temporary core bridge is needed.
## Receiver found locally
The plain `Website-Frontend` and `Website-Publisher` checkouts have no match for
`X-Revalidate-Signature`. The implementation is in
`R:/Git.Local/KofTwentyTwo/Website-Frontend-wt-isr`, branch `feature/cdn-isr`,
HEAD `476318b` at inspection (working files inspected):
- `src/lib/revalidate.ts:5`: names `x-revalidate-signature`.
- `src/lib/revalidate.ts:9`: `signBody` returns `sha256=` plus hex HMAC-SHA256
over the body using the raw string key (the calculation is line 10).
- `src/lib/revalidate.ts:14`: `verifySignature` compares this value in constant
time; line 21 lowercases the incoming legacy header.
- `src/app/api/revalidate/route.ts:30`: reads `CDN_REVALIDATE_SECRET`.
- `src/app/api/revalidate/route.ts:36`: reads the body before JSON parsing.
- `src/app/api/revalidate/route.ts:40`: reads only the legacy signature header
and calls the verifier; standard requests currently return 401.
- `src/__tests__/revalidate-route.test.ts:12`: test requests send only the legacy header.
These receiver files were inspected, not modified.
## Deployment and receiver changes
1. Add the three hook columns documented in README. Existing rows (null
`signatureFormat`) keep the LEGACY signature; switch a hook to `STANDARD_WEBHOOKS`
after its receiver verifies Standard Webhooks. New rows default to `STANDARD_WEBHOOKS`.
For internal HTTP endpoints also explicitly allow HTTP and the intended host.
2. Update `src/lib/revalidate.ts:14` to verify Standard Webhooks: require all
three headers, validate an integer Unix timestamp within 300 seconds of the
receiver clock (reject malformed/non-finite values), then HMAC-SHA256 the
bytes `<webhook-id>.<webhook-timestamp>.<raw body>` and compare decoded
signatures in constant time. Accept any matching `v1,<base64>` entry in the
space-separated signature list. Do not lowercase base64 signatures.
Match core secret parsing: trim text; decode `whsec_` base64 keys, otherwise
UTF-8; use at least 16 key bytes. Rotate short existing keys on both sides.
3. At `src/app/api/revalidate/route.ts:36`, retain raw body bytes for verification;
at line 40 pass all three headers to the new verifier before parsing JSON.
Update its header import/documentation (lines 5 and 22). During migration the
receiver may explicitly support both formats; do not fall back to legacy
when standard headers are present but invalid.
4. Update `src/__tests__/revalidate-route.test.ts:12` and helper tests to cover
valid standard requests, altered body/id/timestamp, missing headers, stale
or future timestamps, malformed timestamps, encoded keys, and legacy mode.
5. Deploy the receiver, then switch its hook to `STANDARD_WEBHOOKS`.
The JSON payload remains `{"tags":[...]}`. One UUID identifies a hook invocation,
shared across its fan-out requests; each request gets its own current timestamp.
A later dispatcher invocation generates a new UUID. Revalidation remains
idempotent, so receivers must not rely on this ID to deduplicate purge retries;
any optional deduplication must remain local to each pod, not suppress other pods.
## Compatibility and verification
The legacy HMAC helper remains solely for the explicitly supported legacy format,
not as a temporary replacement for core. Golden tests use fixed digests calculated
independently of that helper, including a short key and a whitespace/prefix key.
Standard requests are checked using RC.5 `WebhookVerifier`; transport tests cover
HTTP/private-address refusal, explicit local exceptions and mixed-address fan-out.
The test/collaborator constructor now takes `(secrets, resolver)`; callers no longer
inject `java.net.http.HttpClient`. The production no-argument constructor remains.
The only existing test adaptation outside the runner tests is `PurgePipelineTest`
using this constructor. The RC.5 bump required no unrelated code fixes.
+6 -4
View File
@@ -18,17 +18,19 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<properties>
<revision>0.1.0-SNAPSHOT</revision>
<qbit.guardrails.enforce>true</qbit.guardrails.enforce>
<qbit.guardrails.strictUiApiRoutes>true</qbit.guardrails.strictUiApiRoutes>
<revision>0.1.0-RC.1</revision>
<maven.compiler.source>21</maven.compiler.source>
<maven.compiler.target>21</maven.compiler.target>
<jacoco.version>0.8.15</jacoco.version>
<aws.sdk.version>2.41.10</aws.sdk.version>
<aws.sdk.version>2.41.34</aws.sdk.version>
</properties>
<parent>
<groupId>com.kingsrook</groupId>
<artifactId>qbit-build-parent</artifactId>
<version>2.0.0</version>
<version>2.1.0</version>
</parent>
<groupId>com.kingsrook.qbits</groupId>
@@ -192,7 +194,7 @@
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<qqq.snapshot.version>4.1.0-RC.4</qqq.snapshot.version>
<qqq.snapshot.version>4.1.0-RC.6</qqq.snapshot.version>
</properties>
<dependencyManagement>
<dependencies>
@@ -0,0 +1,66 @@
/*
* Copyright 2026 Kingsrook, LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.cdn.model;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PossibleValueEnum;
/*******************************************************************************
** Signature format selected on each revalidation hook.
*******************************************************************************/
public enum CdnSignatureFormat implements PossibleValueEnum<String>
{
STANDARD_WEBHOOKS("STANDARD_WEBHOOKS", "Standard Webhooks"),
LEGACY("LEGACY", "Legacy X-Revalidate-Signature");
private final String id;
private final String label;
/*******************************************************************************
** Constructor.
*******************************************************************************/
CdnSignatureFormat(String id, String label)
{
this.id = id;
this.label = label;
}
/*******************************************************************************
** Getter for possibleValueId.
*******************************************************************************/
@Override
public String getPossibleValueId()
{
return (id);
}
/*******************************************************************************
** Getter for possibleValueLabel.
*******************************************************************************/
@Override
public String getPossibleValueLabel()
{
return (label);
}
}
@@ -54,11 +54,12 @@ public final class CdnTables
public static final List<String> ALL = List.of(PROVIDER, HOSTNAME, CACHE_POLICY, TABLE_TAG_RULE, PURGE_REQUEST, REVALIDATION_HOOK,
EDGE_STAT_DAILY, ORIGIN_CHECK, DRIFT_FINDING);
public static final String PVS_PROVIDER_TYPE = "cdn_provider_type";
public static final String PVS_STAGE = "cdn_hostname_stage";
public static final String PVS_PURGE_STATUS = "cdn_purge_status";
public static final String PVS_PURGE_SOURCE = "cdn_purge_source";
public static final String PVS_POLICY_CLASS = "cdn_policy_class";
public static final String PVS_PROVIDER_TYPE = "cdn_provider_type";
public static final String PVS_STAGE = "cdn_hostname_stage";
public static final String PVS_PURGE_STATUS = "cdn_purge_status";
public static final String PVS_PURGE_SOURCE = "cdn_purge_source";
public static final String PVS_SIGNATURE_FORMAT = "cdn_signature_format";
public static final String PVS_POLICY_CLASS = "cdn_policy_class";
@@ -95,6 +96,7 @@ public final class CdnTables
QPossibleValueSource.newForEnum(PVS_PURGE_STATUS, CdnPurgeStatus.values()),
QPossibleValueSource.newForEnum(PVS_PURGE_SOURCE, CdnPurgeSource.values()),
QPossibleValueSource.newForEnum(PVS_POLICY_CLASS, CdnPolicyClass.values()),
QPossibleValueSource.newForEnum(PVS_SIGNATURE_FORMAT, CdnSignatureFormat.values()),
QPossibleValueSource.newForTable(PROVIDER),
QPossibleValueSource.newForTable(HOSTNAME)));
}
@@ -318,6 +320,9 @@ public final class CdnTables
.withField(new QFieldMetaData("url", QFieldType.STRING).withIsRequired(true).withMaxLength(500))
.withField(new QFieldMetaData("fanOutDns", QFieldType.BOOLEAN).withDefaultValue(false))
.withField(new QFieldMetaData("secretRef", QFieldType.STRING).withMaxLength(100))
.withField(new QFieldMetaData("signatureFormat", QFieldType.STRING).withPossibleValueSourceName(PVS_SIGNATURE_FORMAT).withDefaultValue(CdnSignatureFormat.STANDARD_WEBHOOKS.name()))
.withField(new QFieldMetaData("allowPlainHttp", QFieldType.BOOLEAN).withDefaultValue(false))
.withField(new QFieldMetaData("internalAllowlist", QFieldType.TEXT))
.withField(new QFieldMetaData("timeoutMs", QFieldType.INTEGER).withDefaultValue(5000))
.withField(new QFieldMetaData("enabled", QFieldType.BOOLEAN).withDefaultValue(true)));
}
@@ -21,16 +21,18 @@ import javax.crypto.Mac;
import javax.crypto.spec.SecretKeySpec;
import java.net.InetAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.ArrayList;
import java.time.Instant;
import java.util.Arrays;
import java.util.HexFormat;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.UUID;
import java.util.function.Function;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.kingsrook.qbits.cdn.model.CdnSignatureFormat;
import com.kingsrook.qbits.cdn.model.CdnTables;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.exceptions.QException;
@@ -41,17 +43,20 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.utils.StringUtils;
import com.kingsrook.qqq.backend.core.utils.ValueUtils;
import com.kingsrook.qqq.backend.core.utils.http.SafeHttpClient;
import com.kingsrook.qqq.backend.core.utils.http.SafeHttpPolicy;
import com.kingsrook.qqq.backend.core.utils.webhooks.WebhookSecret;
import com.kingsrook.qqq.backend.core.utils.webhooks.WebhookSigner;
/*******************************************************************************
** Calls every enabled cdn_revalidation_hook before the CDN is purged, so the
** edge never refetches a stale page (design spec section 4.5 step 3).
**
** Request: POST {"tags":[...]} with header
** "X-Revalidate-Signature: sha256=<hex HMAC-SHA256 of the body>", keyed by
** the secret in the environment variable the hook's secretRef names. With
** fanOutDns, the hook's host is resolved and every address is called (one
** per pod behind a headless Service). Every call must answer 2xx.
** Request: POST {"tags":[...]} signed with core Standard Webhooks headers.
** LEGACY keeps X-Revalidate-Signature with a hex HMAC of the body. The
** secretRef names an environment variable. SafeHttpClient checks and pins
** every address; fan-out preserves the original Host and TLS server name.
*******************************************************************************/
public class RevalidationHookRunner
{
@@ -59,9 +64,8 @@ public class RevalidationHookRunner
private static final ObjectMapper JSON = new ObjectMapper();
private final HttpClient http;
private final Function<String, String> secrets;
private final Resolver resolver;
private final Function<String, String> secrets;
private final Resolver resolver;
@@ -83,7 +87,7 @@ public class RevalidationHookRunner
*******************************************************************************/
public RevalidationHookRunner()
{
this(HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(5)).build(), System::getenv,
this(System::getenv,
host -> java.util.Arrays.stream(InetAddress.getAllByName(host)).map(InetAddress::getHostAddress).toList());
}
@@ -92,9 +96,8 @@ public class RevalidationHookRunner
/*******************************************************************************
** Constructor with collaborators.
*******************************************************************************/
public RevalidationHookRunner(HttpClient http, Function<String, String> secrets, Resolver resolver)
public RevalidationHookRunner(Function<String, String> secrets, Resolver resolver)
{
this.http = http;
this.secrets = secrets;
this.resolver = resolver;
}
@@ -133,42 +136,63 @@ public class RevalidationHookRunner
byte[] body = JSON.writeValueAsBytes(java.util.Map.of("tags", tags));
String secretRef = hook.getValueString("secretRef");
String secret = StringUtils.hasContent(secretRef) ? secrets.apply(secretRef) : null;
String formatValue = hook.getValueString("signatureFormat");
CdnSignatureFormat format = formatValue == null ? CdnSignatureFormat.LEGACY : CdnSignatureFormat.valueOf(formatValue);
WebhookSecret signingSecret = secret != null && format == CdnSignatureFormat.STANDARD_WEBHOOKS ? WebhookSecret.parse(secret) : null;
String messageId = UUID.randomUUID().toString();
URI uri = URI.create(hook.getValueString("url"));
List<URI> targets = new ArrayList<>();
if(Boolean.TRUE.equals(ValueUtils.getValueAsBoolean(hook.getValue("fanOutDns"))))
Integer timeout = ValueUtils.getValueAsInteger(hook.getValue("timeoutMs"));
if(timeout != null && timeout <= 0)
{
for(String address : resolver.resolve(uri.getHost()))
{
String host = address.contains(":") ? "[" + address + "]" : address;
targets.add(new URI(uri.getScheme(), null, host, uri.getPort(), uri.getPath(), uri.getQuery(), null));
}
throw (new QException("Revalidation hook timeoutMs must be positive"));
}
else
String allowlist = hook.getValueString("internalAllowlist");
SafeHttpPolicy policy = SafeHttpPolicy.defaults()
.withAllowPlainHttp(Boolean.TRUE.equals(ValueUtils.getValueAsBoolean(hook.getValue("allowPlainHttp"))))
.withInternalAllowlist(allowlist == null ? List.of() : Arrays.asList(allowlist.split("[,\\s]+")))
.withTotalTimeout(Duration.ofMillis(timeout == null ? 5000 : timeout));
List<String> resolved = resolver.resolve(uri.getHost());
InetAddress[] addresses = new InetAddress[resolved.size()];
for(int i = 0; i < resolved.size(); i++)
{
targets.add(uri);
addresses[i] = InetAddress.getByName(resolved.get(i));
}
if(targets.isEmpty())
if(addresses.length == 0)
{
throw (new QException("Revalidation hook " + name + " resolved to no addresses"));
}
Integer timeout = ValueUtils.getValueAsInteger(hook.getValue("timeoutMs"));
for(URI target : targets)
try(SafeHttpClient validator = new SafeHttpClient(policy, host -> addresses))
{
HttpRequest.Builder request = HttpRequest.newBuilder(target)
.timeout(Duration.ofMillis(timeout == null ? 5000 : timeout))
.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofByteArray(body));
validator.checkUrl(uri.toString());
}
boolean fanOut = Boolean.TRUE.equals(ValueUtils.getValueAsBoolean(hook.getValue("fanOutDns")));
int calls = fanOut ? addresses.length : 1;
for(int i = 0; i < calls; i++)
{
InetAddress[] targetAddresses = fanOut ? new InetAddress[] {addresses[i]} : addresses;
Map<String, String> headers = new LinkedHashMap<>();
headers.put("Content-Type", "application/json");
if(secret != null)
{
request.header(SIGNATURE_HEADER, "sha256=" + sign(secret, body));
if(format == CdnSignatureFormat.LEGACY)
{
headers.put(SIGNATURE_HEADER, "sha256=" + sign(secret, body));
}
else
{
headers.putAll(WebhookSigner.headers(messageId, Instant.now(), body, List.of(signingSecret)));
}
}
HttpResponse<String> response = http.send(request.build(), HttpResponse.BodyHandlers.ofString());
if(response.statusCode() / 100 != 2)
try(SafeHttpClient client = new SafeHttpClient(policy, host -> targetAddresses))
{
throw (new QException("Revalidation hook " + name + " answered " + response.statusCode() + " from " + target.getHost()));
int status = client.post(uri.toString(), headers, body).statusCode();
if(status / 100 != 2)
{
throw (new QException("Revalidation hook " + name + " answered " + status + " from " + uri.getHost()));
}
}
}
return (targets.size());
return (calls);
}
catch(QException e)
{
@@ -188,7 +212,7 @@ public class RevalidationHookRunner
/*******************************************************************************
** Hex HMAC-SHA256.
** Legacy compatibility: hex HMAC-SHA256 over the body with raw UTF-8 key bytes.
*******************************************************************************/
public static String sign(String secret, byte[] body) throws Exception
{
@@ -17,12 +17,11 @@
package com.kingsrook.qbits.cdn.purge;
import java.net.http.HttpClient;
import java.util.List;
import com.kingsrook.qbits.cdn.CdnRuntime;
import com.kingsrook.qbits.cdn.CdnTestSupport;
import com.kingsrook.qbits.cdn.CdnTestSupport.FakePathProvider;
import com.kingsrook.qbits.cdn.CdnTestSupport.FakeTagProvider;
import com.kingsrook.qbits.cdn.CdnTestSupport;
import com.kingsrook.qbits.cdn.model.CdnTables;
import com.kingsrook.qbits.cdn.spi.CdnException;
import com.kingsrook.qbits.cdn.spi.CdnProviderType;
@@ -46,7 +45,7 @@ import static org.junit.jupiter.api.Assertions.assertEquals;
*******************************************************************************/
class PurgePipelineTest
{
private final CdnPurgeDispatcher dispatcher = new CdnPurgeDispatcher(new RevalidationHookRunner(HttpClient.newHttpClient(), name -> null, host -> List.of()));
private final CdnPurgeDispatcher dispatcher = new CdnPurgeDispatcher(new RevalidationHookRunner(name -> null, host -> List.of()));
@@ -18,15 +18,19 @@ package com.kingsrook.qbits.cdn.purge;
import java.net.InetSocketAddress;
import java.net.http.HttpClient;
import java.nio.charset.StandardCharsets;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CopyOnWriteArrayList;
import java.util.concurrent.atomic.AtomicInteger;
import com.kingsrook.qbits.cdn.CdnTestSupport;
import com.kingsrook.qbits.cdn.model.CdnTables;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.utils.webhooks.WebhookSecret;
import com.kingsrook.qqq.backend.core.utils.webhooks.WebhookVerificationException;
import com.kingsrook.qqq.backend.core.utils.webhooks.WebhookVerifier;
import com.sun.net.httpserver.HttpServer;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
@@ -46,6 +50,7 @@ class RevalidationHookRunnerTest
private final List<String> signatures = new CopyOnWriteArrayList<>();
private final AtomicInteger status = new AtomicInteger(200);
private HttpServer server;
private final List<Map<String, String>> headers = new CopyOnWriteArrayList<>();
@@ -59,6 +64,9 @@ class RevalidationHookRunnerTest
server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
server.createContext("/api/revalidate", exchange ->
{
Map<String, String> received = new LinkedHashMap<>();
exchange.getRequestHeaders().forEach((key, values) -> received.put(key.toLowerCase(java.util.Locale.ROOT), values.get(0)));
headers.add(received);
bodies.add(new String(exchange.getRequestBody().readAllBytes(), StandardCharsets.UTF_8));
String signature = exchange.getRequestHeaders().getFirst(RevalidationHookRunner.SIGNATURE_HEADER);
signatures.add(signature == null ? "none" : signature);
@@ -88,13 +96,21 @@ class RevalidationHookRunnerTest
void signsAndFansOut() throws Exception
{
CdnTestSupport.insert(CdnTables.REVALIDATION_HOOK, new QRecord().withValue("name", "next").withValue("enabled", true).withValue("fanOutDns", true)
.withValue("secretRef", "SECRET").withValue("url", "http://frontend.invalid:" + server.getAddress().getPort() + "/api/revalidate"));
RevalidationHookRunner runner = new RevalidationHookRunner(HttpClient.newHttpClient(), name -> "SECRET".equals(name) ? "s3cret" : null,
.withValue("secretRef", "SECRET").withValue("allowPlainHttp", true).withValue("internalAllowlist", "frontend.invalid").withValue("url", "http://frontend.invalid:" + server.getAddress().getPort() + "/api/revalidate"));
RevalidationHookRunner runner = new RevalidationHookRunner(name -> "SECRET".equals(name) ? "test-secret-at-least-16-bytes" : null,
host -> List.of("127.0.0.1", "127.0.0.1"));
assertEquals(2, runner.run(List.of("post:1")));
assertThat(bodies).hasSize(2).allMatch("{\"tags\":[\"post:1\"]}"::equals);
String expected = "sha256=" + RevalidationHookRunner.sign("s3cret", bodies.get(0).getBytes(StandardCharsets.UTF_8));
assertThat(signatures).allMatch(expected::equals);
WebhookVerifier verifier = new WebhookVerifier(List.of(WebhookSecret.parse("test-secret-at-least-16-bytes")));
for(int i = 0; i < bodies.size(); i++)
{
verifier.verify(headers.get(i), bodies.get(i).getBytes(StandardCharsets.UTF_8));
}
assertThat(signatures).containsExactly("none", "none");
assertThat(headers.get(0).get("webhook-id")).isEqualTo(headers.get(1).get("webhook-id"));
assertThat(headers.get(0).get("host")).startsWith("frontend.invalid:");
assertThatThrownBy(() -> verifier.verify(headers.get(0), "tampered".getBytes(StandardCharsets.UTF_8)))
.isInstanceOf(WebhookVerificationException.class);
assertEquals(0, runner.run(List.of()));
}
@@ -108,12 +124,94 @@ class RevalidationHookRunnerTest
{
status.set(500);
CdnTestSupport.insert(CdnTables.REVALIDATION_HOOK, new QRecord().withValue("name", "next").withValue("enabled", true)
.withValue("url", "http://127.0.0.1:" + server.getAddress().getPort() + "/api/revalidate"));
RevalidationHookRunner runner = new RevalidationHookRunner(HttpClient.newHttpClient(), name -> null, host -> List.of());
.withValue("allowPlainHttp", true).withValue("internalAllowlist", "127.0.0.1").withValue("url", "http://127.0.0.1:" + server.getAddress().getPort() + "/api/revalidate"));
RevalidationHookRunner runner = new RevalidationHookRunner(name -> null, host -> List.of("127.0.0.1"));
assertThatThrownBy(() -> runner.run(List.of("post:1"))).isInstanceOf(QException.class).hasMessageContaining("500");
assertThat(signatures).containsExactly("none");
QRecord fanOut = new QRecord().withValue("name", "empty").withValue("fanOutDns", true).withValue("url", "http://x.invalid/api/revalidate");
assertThatThrownBy(() -> runner.call(fanOut, List.of("post:1"))).isInstanceOf(QException.class).hasMessageContaining("no addresses");
assertThatThrownBy(() -> new RevalidationHookRunner(name -> null, host -> List.of()).call(fanOut, List.of("post:1"))).isInstanceOf(QException.class).hasMessageContaining("no addresses");
}
/*******************************************************************************
** Legacy mode preserves fixed signatures, including whitespace and whsec_ keys.
*******************************************************************************/
@Test
void legacyGoldenVectors() throws Exception
{
new RevalidationHookRunner(name -> "s3cret", host -> List.of("127.0.0.1"))
.call(hook().withValue("signatureFormat", "LEGACY"), List.of("post:1"));
assertThat(signatures.get(0)).isEqualTo("sha256=540457336a45ef962b165f8732f71ebf327ffa67c0cb1047c446df0231fa0db2");
new RevalidationHookRunner(name -> " whsec_test-key-with-spaces ", host -> List.of("127.0.0.1"))
.call(hook().withValue("signatureFormat", "LEGACY"), List.of("post:1"));
assertThat(signatures.get(1)).isEqualTo("sha256=a4dd022bffe6f799633102c415fdc18e22f078bfc98357f909b5df58d5318b05");
new RevalidationHookRunner(name -> "s3cret", host -> List.of("127.0.0.1"))
.call(hook(), List.of("post:1"));
assertThat(signatures.get(2)).as("a null format (existing rows) keeps the legacy signature").isEqualTo(signatures.get(0));
assertThat(bodies).containsExactly("{\"tags\":[\"post:1\"]}", "{\"tags\":[\"post:1\"]}", "{\"tags\":[\"post:1\"]}");
assertThat(headers).allSatisfy(value -> assertThat(value).doesNotContainKeys("webhook-id", "webhook-timestamp", "webhook-signature"));
}
/*******************************************************************************
** An explicit standard format accepts encoded secrets.
*******************************************************************************/
@Test
void explicitStandardFormat() throws Exception
{
WebhookSecret secret = WebhookSecret.generate();
new RevalidationHookRunner(name -> secret.reveal(), host -> List.of("127.0.0.1"))
.call(hook().withValue("signatureFormat", "STANDARD_WEBHOOKS"), List.of("post:1"));
new WebhookVerifier(List.of(secret)).verify(headers.get(0), bodies.get(0).getBytes(StandardCharsets.UTF_8));
}
/*******************************************************************************
** Invalid format and short standard secrets fail before sending.
*******************************************************************************/
@Test
void invalidSigningConfiguration()
{
RevalidationHookRunner runner = new RevalidationHookRunner(name -> "short", host -> List.of("127.0.0.1"));
assertThatThrownBy(() -> runner.call(hook().withValue("signatureFormat", "STANDARD_WEBHOOKS"), List.of("post:1"))).isInstanceOf(QException.class).hasMessageContaining("at least 16 bytes");
assertThatThrownBy(() -> runner.call(hook().withValue("signatureFormat", "UNKNOWN"), List.of("post:1"))).isInstanceOf(QException.class);
assertThat(bodies).isEmpty();
}
/*******************************************************************************
** Safe defaults reject HTTP, private addresses and mixed fan-out before sending.
*******************************************************************************/
@Test
void safeHttpPolicy()
{
RevalidationHookRunner runner = new RevalidationHookRunner(name -> null, host -> List.of("127.0.0.1"));
assertThatThrownBy(() -> runner.call(hook().withValue("allowPlainHttp", null), List.of("post:1")))
.isInstanceOf(QException.class).hasMessageContaining("Only https");
assertThatThrownBy(() -> runner.call(hook().withValue("internalAllowlist", null), List.of("post:1")))
.isInstanceOf(QException.class).hasMessageContaining("Blocked");
RevalidationHookRunner mixed = new RevalidationHookRunner(name -> null, host -> List.of("127.0.0.1", "169.254.169.254"));
assertThatThrownBy(() -> mixed.call(hook().withValue("fanOutDns", true), List.of("post:1")))
.isInstanceOf(QException.class).hasMessageContaining("Blocked");
assertThatThrownBy(() -> runner.call(hook().withValue("timeoutMs", 0), List.of("post:1")))
.isInstanceOf(QException.class).hasMessageContaining("positive");
assertThat(bodies).isEmpty();
}
/*******************************************************************************
** A local test hook explicitly opts into HTTP and loopback.
*******************************************************************************/
private QRecord hook()
{
return (new QRecord().withValue("name", "test").withValue("secretRef", "SECRET")
.withValue("allowPlainHttp", true).withValue("internalAllowlist", "127.0.0.1")
.withValue("url", "http://127.0.0.1:" + server.getAddress().getPort() + "/api/revalidate"));
}
}