test(help): compare the help bundle version with the artifact version, not a hardcoded SNAPSHOT

The bundle version is now Maven-filtered, so a literal SNAPSHOT in the test fails every release build. Surefire passes the artifact version in qbit.artifactVersion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge pull request #20 from QRun-IO/fix/help-bundle-version
2026-10-14 23:07:56 -05:00 · 2026-10-14 22:37:20 -05:00 · 2026-10-14 22:04:41 -05:00 · 2026-10-14 21:27:00 -05:00 · 2026-10-14 21:15:07 -05:00 · 2026-10-14 20:29:38 -05:00
17 changed files with 1498 additions and 208 deletions
@@ -6,15 +6,23 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and
 ## [Unreleased]

 ### Added
+- Central `webhooks.endpoint.<id>` and `webhooks.dispatcher` dependency checks, every 300 seconds, using persisted endpoint health and outbox age.
+- A native home status line linking to the webhooks checks in System Health.
+- Complete host-neutral in-app help bundle for Webhooks, with live metadata references and a `HelpBundleAssertions` coverage test including the optional ESB process.
 - Foreign keys between all five tables in `docs/ddl/webhooks-postgres.sql`, named `fk_<table>_<column>`: an endpoint's secrets and subscriptions are deleted with it (CASCADE), deliveries and attempts keep their endpoint (RESTRICT: cancel and Delete Deliveries before deleting an endpoint with history), a delivery outlives its subscription (SET NULL) and attempts go with their delivery (CASCADE). New index `webhooks_delivery_subscription_ix`.
 - `docs/ddl/webhooks-foreign-keys-upgrade.sql` for existing databases: records orphan rows in `qqq_fk_quarantine` (nulls nullable references, deletes rows with a missing required parent and keeps them as JSON), replaces the NO ACTION endpoint keys, and is idempotent.
 - `ForeignKeyGuardrailTest` applies the DDL to PostgreSQL 17 (Testcontainers) and fails when a join or table possible-value field between webhooks tables has no foreign key.
- Every table explains itself: a help bundle (`help/qbit-webhooks.yaml`, QQQ help-bundle format) says what each `webhooks_*` table is for and how its rows get created, including where event types come from. It shows on the table screen and when the table is empty. Applied by `WebhooksHelp` (STAND-IN for QRun-IO/qqq#1198); a host's own help is kept.
+- Every table explains itself: a help bundle (`help/qbit-webhooks.yaml`, QQQ help-bundle format) says what each `webhooks_*` table is for and how its rows get created, including where event types come from. It shows on the table screen and when the table is empty. Registered through QQQ `HelpBundles`, with inline empty-state help retained for hosts without `helpContent`; a host's own help is kept.
 - A Setup card on the Webhooks home lists event types, encryption key, endpoints, signing secrets and subscriptions with their state and the next step.
 - `WebhooksTablesGuardrailTest` (each table is used at runtime, seeded or explained, and has help) and a test that a previous secret stops signing at its Signs Until date even before the maintenance job retires it.
 - Nothing is seeded, deliberately: the tables hold the owner's receivers and secrets, or logs.

+### Removed
+- The home's separate endpoint-health attention tile, list and diagnostic queries; endpoint state, alerts and Send Test Event remain unchanged.
+
 ### Changed
+- Pin QQQ to 4.1.0-RC.8 for the released central health API.
+- Pin QQQ to 4.1.0-RC.7 and use its help-bundle parser, registry and guarded startup synchronization. Hosts can override bundle text with a `HOST` bundle.
 - Require QQQ 4.1.0-RC.5 and use its webhook signer, verifier, secret and verification exception (QRun-IO/qqq#1160, F11). Removed the internal signing package; Java consumers must import `com.kingsrook.qqq.backend.core.utils.webhooks` instead.
 - Signing wire format, timestamp tolerance, rotation, endpoint management, encrypted secret storage and delivery behavior are unchanged.

@@ -7,7 +7,7 @@ Design: [QRun-IO/qqq#1105](https://github.com/QRun-IO/qqq/pull/1105) (`docs/spec
 ## Requirements

 - Java 21
- QQQ 4.1.0-RC.5 or later (`qqq-backend-core`). Two modules are optional:
+- QQQ 4.1.0-RC.8 or later (`qqq-backend-core`). Two modules are optional:
  - `qqq-middleware-api`, for payloads rendered through a QQQ API version.
  - `qqq-esb`, for ESB topics as an event source.
 - A SQL backend for the five tables (DDL in [docs/ddl/webhooks-postgres.sql](docs/ddl/webhooks-postgres.sql)), and a QQQ scheduler for delivery.
@@ -56,6 +56,14 @@ Nothing in this qbit is seeded: endpoints, secrets and subscriptions are your ch

 The event types you can subscribe to come from code, not from a table: the host's `tableEventSources`, `processEventSources` and ESB settings, custom events it fires with `Webhooks.fire`, and a built-in Test event, all held in the `WebhooksRegistry`.

+## In-app help
+
+The producer registers [help/qbit-webhooks.yaml](src/main/resources/help/qbit-webhooks.yaml), a `QBIT` bundle covering the app, tables, fields, actions, widgets and possible-value sources. QQQ synchronizes it on startup when the host has the `helpContent` table; registration is safe without that table, and inline app/table empty-state help remains available.
+
+Hosts can register their own YAML bundle with `layer: HOST` through `HelpBundles.register(qInstance, "help/my-host.yaml")`. Use the same entry keys and roles to override selected text with deployment-specific guidance. Keep live `{{label:...}}`, `{{values:...}}` and `{{default:...}}` references where appropriate. If you customize the app name, supply its help under the customized app key in your host bundle.
+
+`WebhooksHelpBundleTest` uses QQQ's `HelpBundleAssertions` against the produced instance, including the optional ESB process, to check coverage, summary length and live references. Run it with `./mvnw -q -Dtest=WebhooksHelpBundleTest test`; the full `./mvnw -q verify` also runs it. Keep the bundle's literal version aligned with the POM revision when releasing.
+
 ## Permissions

 | Permission | Grants |
@@ -123,6 +131,22 @@ Upgrading from 0.x: see [docs/migration-0.x-to-1.0.md](docs/migration-0.x-to-1.0
 ./mvnw verify      # tests, Checkstyle and the JaCoCo gate (always use the wrapper)
 ```

+## Health checks
+
+The producer registers passive checks with QQQ central health monitoring (requires QQQ 4.1.0-RC.8 or later):
+
+| Name | Kind | Category | Interval |
+|---|---|---|---|
+| `webhooks.endpoint.<id>` | DEPENDENCY | INTEGRATION | 300 seconds |
+| `webhooks.dispatcher` | DEPENDENCY | INTEGRATION | 300 seconds |
+
+Both belong to component `webhooks`. Endpoint checks read the existing state machine: HEALTHY means UP; UNHEALTHY or PROBATION means DEGRADED; automatically disabled means DOWN. Active and automatically disabled endpoints are discovered dynamically; paused and manually disabled endpoints are excluded. Details contain only healthStatus, consecutiveFailures and unhealthySince when present.
+
+The dispatcher reports `oldestPendingSeconds`, measured from the oldest queued delivery's createDate across PENDING, RETRY_WAIT and SENDING. More than 15 minutes is DEGRADED; more than 60 minutes is DOWN. An empty queue reports UP with zero age. Checks do not send requests or change endpoint state.
+
+The home shows one native central status line linking to the webhooks checks in System Health. Disable checks centrally with `health_check.enabled`. The endpoint state machine, alerts and Send Test Event action remain unchanged.
+
+
 ## License

 Apache-2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).
@@ -142,7 +142,34 @@
   </dependencies>

   <build>
+      <resources>
+         <!-- help bundles carry the artifact version (QQQ help bundle provenance) -->
+         <resource>
+            <directory>src/main/resources</directory>
+            <filtering>true</filtering>
+            <includes>
+               <include>help/*.yaml</include>
+            </includes>
+         </resource>
+         <resource>
+            <directory>src/main/resources</directory>
+            <filtering>false</filtering>
+            <excludes>
+               <exclude>help/*.yaml</exclude>
+            </excludes>
+         </resource>
+      </resources>
      <plugins>
+         <!-- the artifact version, for the help bundle version test -->
+         <plugin>
+            <groupId>org.apache.maven.plugins</groupId>
+            <artifactId>maven-surefire-plugin</artifactId>
+            <configuration>
+               <systemPropertyVariables combine.children="append">
+                  <qbit.artifactVersion>${project.version}</qbit.artifactVersion>
+               </systemPropertyVariables>
+            </configuration>
+         </plugin>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-checkstyle-plugin</artifactId>
@@ -230,7 +257,7 @@
            <activeByDefault>true</activeByDefault>
         </activation>
         <properties>
-            <qqq.snapshot.version>4.1.0-RC.6</qqq.snapshot.version>
+            <qqq.snapshot.version>4.1.0-RC.8</qqq.snapshot.version>
         </properties>
         <dependencyManagement>
            <dependencies>
@@ -28,6 +28,7 @@ import java.util.Map;
 import com.kingsrook.qbits.webhooks.app.WebhooksApp;
 import com.kingsrook.qbits.webhooks.events.WebhooksProcessListener;
 import com.kingsrook.qbits.webhooks.events.WebhooksRecordChangeListener;
+import com.kingsrook.qbits.webhooks.health.WebhooksHealthChecks;
 import com.kingsrook.qbits.webhooks.metadata.WebhooksHelp;
 import com.kingsrook.qbits.webhooks.metadata.WebhooksTableCustomizers;
 import com.kingsrook.qbits.webhooks.metadata.WebhooksTables;
@@ -43,6 +44,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
 import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundles;
 import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
 import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
@@ -200,6 +202,8 @@ public class WebhooksQBitProducer implements QBitMetaDataProducer<WebhooksQBitCo
      output.add(WebhooksApp.app(qBitConfig, tables));
      output.addSelfToInstance(qInstance);

+      WebhooksHealthChecks.register(qInstance);
+
      WebhooksRegistry registry = WebhooksRegistry.of(qInstance);
      registry.registerFromConfig(qInstance, qBitConfig);
      registry.register(new WebhookEventType().withName(WebhooksProcesses.TEST_EVENT_TYPE).withKind(WebhookEventKind.CUSTOM)
@@ -210,6 +214,8 @@ public class WebhooksQBitProducer implements QBitMetaDataProducer<WebhooksQBitCo
      {
         qInstance.withProcessLifecycleListener(new QCodeReference(WebhooksProcessListener.class));
      }
+      // Registration is metadata-only; core guards startup sync when helpContent is absent.
+      HelpBundles.register(qInstance, "help/qbit-webhooks.yaml");
      WebhooksHelp.apply(qInstance, qBitConfig);
      return (new MetaDataProducerMultiOutput());
   }
@@ -26,6 +26,7 @@ import java.util.List;
 import java.util.Map;
 import com.kingsrook.qbits.webhooks.WebhooksQBitConfig;
 import com.kingsrook.qbits.webhooks.WebhooksRuntime;
+import com.kingsrook.qbits.webhooks.health.WebhooksHealthWidget;
 import com.kingsrook.qbits.webhooks.metadata.WebhooksEnums.SecretSource;
 import com.kingsrook.qbits.webhooks.metadata.WebhooksEnums.SecretStatus;
 import com.kingsrook.qbits.webhooks.metadata.WebhooksTables;
@@ -34,7 +35,10 @@ import com.kingsrook.qbits.webhooks.permissions.WebhooksPermissions;
 import com.kingsrook.qbits.webhooks.secrets.SecretCipher;
 import com.kingsrook.qbits.webhooks.support.SystemWrites;
 import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer;
+import com.kingsrook.qqq.backend.core.actions.permissions.TablePermissionSubType;
+import com.kingsrook.qqq.backend.core.actions.permissions.UseTablePermissionCustomPermissionChecker;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.health.HealthMetaDataProvider;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
@@ -47,6 +51,8 @@ import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
 import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
+import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
+import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;


@@ -57,6 +63,7 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 *******************************************************************************/
 public final class WebhooksApp
 {
+   public static final String HEALTH_WIDGET       = "webhooksHealth";
   public static final String HOME_WIDGET         = "webhooksHome";
   public static final String SECRET_VALUE_WIDGET = "webhooksSecretValue";

@@ -77,6 +84,10 @@ public final class WebhooksApp
   public static List<QWidgetMetaData> widgets()
   {
      return (List.of(
+         new QWidgetMetaData().withName(HEALTH_WIDGET).withLabel("System Health").withType(WidgetType.STATISTICS.getType())
+            .withCodeReference(new QCodeReference(WebhooksHealthWidget.class)).withGridColumns(12)
+            .withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.HAS_ACCESS_PERMISSION)
+               .withCustomPermissionChecker(UseTablePermissionCustomPermissionChecker.build(HealthMetaDataProvider.CHECK_TABLE, TablePermissionSubType.READ))),
         new QWidgetMetaData().withName(HOME_WIDGET).withLabel("Overview").withType(WidgetType.HTML.getType())
            .withCodeReference(new QCodeReference(HomeRenderer.class)).withGridColumns(12).withIsCard(false)
            .withPermissionRules(WebhooksTables.permissionRules()),
@@ -97,7 +108,7 @@ public final class WebhooksApp
         .withLabel(config.getAppLabel())
         .withIcon(new QIcon("webhook"))
         .withPermissionRules(WebhooksTables.permissionRules())
-         .withWidgets(List.of(HOME_WIDGET));
+         .withWidgets(List.of(HEALTH_WIDGET, HOME_WIDGET));
      for(String name : WebhooksTables.MAIN)
      {
         if(tables.containsKey(name))
@@ -26,15 +26,10 @@ import java.time.Instant;
 import java.time.temporal.ChronoUnit;
 import java.util.ArrayList;
 import java.util.Collections;
-import java.util.HashSet;
 import java.util.List;
-import java.util.Set;
 import com.kingsrook.qbits.webhooks.WebhooksRuntime;
-import com.kingsrook.qbits.webhooks.metadata.WebhooksEnums.ActiveStatus;
 import com.kingsrook.qbits.webhooks.metadata.WebhooksEnums.AttemptOutcome;
 import com.kingsrook.qbits.webhooks.metadata.WebhooksEnums.DeliveryStatus;
-import com.kingsrook.qbits.webhooks.metadata.WebhooksEnums.HealthStatus;
-import com.kingsrook.qbits.webhooks.metadata.WebhooksEnums.SecretStatus;
 import com.kingsrook.qbits.webhooks.metadata.WebhooksTables;
 import com.kingsrook.qqq.backend.core.actions.tables.CountAction;
 import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
@@ -59,7 +54,6 @@ public final class WebhooksHomeData
   private static final int ATTEMPT_SAMPLE = 20000;

   private Instant       now;
-   private int           endpointCount;
   private int           delivered24h;
   private int           failedAttempts24h;
   private int           rateLimited24h;
@@ -69,8 +63,6 @@ public final class WebhooksHomeData
   private Long          p95Millis;
   private final int[]   hourlyOk     = new int[24];
   private final int[]   hourlyFailed = new int[24];
-   private final List<QRecord> attention   = new ArrayList<>();
-   private final List<String>  attentionWhy = new ArrayList<>();
   private final List<QRecord> recentDeadLetters = new ArrayList<>();
   private List<WebhooksSetupStatus.Step> setup = new ArrayList<>();

@@ -85,24 +77,6 @@ public final class WebhooksHomeData
      data.now = WebhooksRuntime.clock().instant();
      Instant since = data.now.minus(Duration.ofHours(24));

-      List<QRecord> endpoints = query(WebhooksTables.ENDPOINT, new QQueryFilter().withOrderBy(new QFilterOrderBy("name")).withLimit(500));
-      data.endpointCount = endpoints.size();
-
-      Set<Integer> withSecret = new HashSet<>();
-      for(QRecord secret : query(WebhooksTables.SECRET, new QQueryFilter(new QFilterCriteria("status", QCriteriaOperator.IN, SecretStatus.ACTIVE.name(), SecretStatus.PREVIOUS.name()))))
-      {
-         withSecret.add(secret.getValueInteger("endpointId"));
-      }
-      for(QRecord endpoint : endpoints)
-      {
-         String why = attentionReason(endpoint, withSecret.contains(endpoint.getValueInteger("id")));
-         if(why != null)
-         {
-            data.attention.add(endpoint);
-            data.attentionWhy.add(why);
-         }
-      }
-
      data.delivered24h = count(WebhooksTables.DELIVERY, new QQueryFilter(new QFilterCriteria("deliveredDate", QCriteriaOperator.GREATER_THAN_OR_EQUALS, since)));
      data.deadLetters = count(WebhooksTables.DELIVERY, new QQueryFilter(new QFilterCriteria("status", QCriteriaOperator.EQUALS, DeliveryStatus.DEAD_LETTER.name())));
      data.backlog = count(WebhooksTables.DELIVERY, new QQueryFilter(new QFilterCriteria("status", QCriteriaOperator.IN,
@@ -157,39 +131,6 @@ public final class WebhooksHomeData



-   /*******************************************************************************
-    ** Why an endpoint needs attention, or null.
-    *******************************************************************************/
-   static String attentionReason(QRecord endpoint, boolean hasSecret)
-   {
-      String health = endpoint.getValueString("healthStatus");
-      String active = endpoint.getValueString("activeStatus");
-      if(ActiveStatus.DISABLED.name().equals(active) && endpoint.getValue("autoDisabledDate") != null)
-      {
-         return ("Disabled after failing on probation");
-      }
-      if(HealthStatus.UNHEALTHY.name().equals(health))
-      {
-         return ("Unhealthy: deliveries paused");
-      }
-      if(HealthStatus.PROBATION.name().equals(health))
-      {
-         return ("On probation: one try per run");
-      }
-      if(ActiveStatus.ACTIVE.name().equals(active) && !hasSecret && !Boolean.TRUE.equals(endpoint.getValueBoolean("allowUnsigned")))
-      {
-         return ("No signing secret: deliveries are waiting");
-      }
-      Integer failures = endpoint.getValueInteger("consecutiveFailures");
-      if(failures != null && failures > 0)
-      {
-         return (failures + (failures == 1 ? " failure" : " failures") + " in a row");
-      }
-      return (null);
-   }
-
-
-
   /*******************************************************************************
    ** Query as the viewer.
    *******************************************************************************/
@@ -221,16 +162,6 @@ public final class WebhooksHomeData



-   /*******************************************************************************
-    ** Getter for endpointCount.
-    *******************************************************************************/
-   public int getEndpointCount()
-   {
-      return (endpointCount);
-   }
-
-
-
   /*******************************************************************************
    ** Getter for delivered24h.
    *******************************************************************************/
@@ -321,26 +252,6 @@ public final class WebhooksHomeData



-   /*******************************************************************************
-    ** Endpoints that need attention.
-    *******************************************************************************/
-   public List<QRecord> getAttention()
-   {
-      return (attention);
-   }
-
-
-
-   /*******************************************************************************
-    ** The reason for each endpoint in {@link #getAttention()}.
-    *******************************************************************************/
-   public List<String> getAttentionWhy()
-   {
-      return (attentionWhy);
-   }
-
-
-
   /*******************************************************************************
    ** The five newest dead letters.
    *******************************************************************************/
@@ -25,7 +25,6 @@ import java.io.Serializable;
 import java.time.Instant;
 import java.time.ZoneOffset;
 import java.time.format.DateTimeFormatter;
-import java.util.List;
 import java.util.Locale;
 import com.kingsrook.qbits.webhooks.metadata.WebhooksTables;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
@@ -88,7 +87,6 @@ public final class WebhooksHomeView
      setup(html, data);
      kpis(html, data, tablePrefix);
      volume(html, data);
-      attention(html, data, tablePrefix);
      deadLetters(html, data, tablePrefix);
      return (html.append("</div>").toString());
   }
@@ -123,7 +121,6 @@ public final class WebhooksHomeView
      kpi(html, String.valueOf(data.getFailedAttempts24h()), "Failed attempts (24 h)", data.getFailedAttempts24h() > 0, prefix + "/" + WebhooksTables.DELIVERY);
      kpi(html, String.valueOf(data.getDeadLetters()), "Dead letters", data.getDeadLetters() > 0, prefix + "/" + WebhooksTables.DELIVERY);
      kpi(html, String.valueOf(data.getBacklog()), "Waiting to send", false, prefix + "/" + WebhooksTables.DELIVERY);
-      kpi(html, String.valueOf(data.getAttention().size()), "Endpoints needing attention", !data.getAttention().isEmpty(), prefix + "/" + WebhooksTables.ENDPOINT);
      kpi(html, data.getP50Millis() == null ? "None yet" : data.getP50Millis() + " / " + data.getP95Millis() + " ms", "Latency, median / 95th (24 h)", false, null);
      html.append("</div>");
   }
@@ -196,41 +193,6 @@ public final class WebhooksHomeView



-   /*******************************************************************************
-    ** Endpoints needing attention, one card row each.
-    *******************************************************************************/
-   private static void attention(StringBuilder html, WebhooksHomeData data, String prefix)
-   {
-      html.append("<div style=\"").append(CARD).append("\"><h3 style=\"").append(TITLE).append("\">Endpoints needing attention</h3>");
-      if(data.getEndpointCount() == 0)
-      {
-         html.append("<p style=\"").append(MUTED).append("\">No endpoints yet. <a href=\"").append(esc(prefix + "/" + WebhooksTables.ENDPOINT))
-            .append("\">Add an endpoint</a>, then subscribe it to events.</p></div>");
-         return;
-      }
-      if(data.getAttention().isEmpty())
-      {
-         html.append("<p style=\"").append(MUTED).append("\">All ").append(data.getEndpointCount()).append(" endpoints are healthy.</p></div>");
-         return;
-      }
-      List<QRecord> rows = data.getAttention();
-      for(int i = 0; i < rows.size(); i++)
-      {
-         QRecord endpoint = rows.get(i);
-         String  message  = endpoint.getValueString("lastFailureMessage");
-         html.append("<div style=\"").append(ROW).append("\">")
-            .append(cell("Endpoint", "<a href=\"" + esc(prefix + "/" + WebhooksTables.ENDPOINT + "/" + endpoint.getValue("id")) + "\">"
-               + esc(endpoint.getValueString("name")) + "</a>"))
-            .append(cell("Why", esc(data.getAttentionWhy().get(i)) + (message == null ? "" : "<br><span style=\"opacity:.75\">" + esc(clamp(message, 140)) + "</span>")))
-            .append(cell("Last failure", when(endpoint.getValue("lastFailureDate"))))
-            .append(cell("Last success", when(endpoint.getValue("lastSuccessDate"))))
-            .append("</div>");
-      }
-      html.append("</div>");
-   }
-
-
-
   /*******************************************************************************
    ** Newest dead letters.
    *******************************************************************************/
@@ -0,0 +1,230 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.webhooks.health;
+
+
+import java.time.Duration;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+import java.util.Set;
+import com.kingsrook.qbits.webhooks.WebhooksRuntime;
+import com.kingsrook.qbits.webhooks.metadata.WebhooksEnums.ActiveStatus;
+import com.kingsrook.qbits.webhooks.metadata.WebhooksEnums.DeliveryStatus;
+import com.kingsrook.qbits.webhooks.metadata.WebhooksEnums.HealthStatus;
+import com.kingsrook.qbits.webhooks.metadata.WebhooksTables;
+import com.kingsrook.qbits.webhooks.support.SystemWrites;
+import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.health.HealthCategory;
+import com.kingsrook.qqq.backend.core.health.HealthCheck;
+import com.kingsrook.qqq.backend.core.health.HealthCheckInput;
+import com.kingsrook.qqq.backend.core.health.HealthCheckKind;
+import com.kingsrook.qqq.backend.core.health.HealthCheckMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthCheckOutcome;
+import com.kingsrook.qqq.backend.core.health.HealthCheckSupplier;
+import com.kingsrook.qqq.backend.core.health.HealthCheckSupplierMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthChecksMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthDetails;
+import com.kingsrook.qqq.backend.core.health.HealthMetric;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+
+
+/*******************************************************************************
+ ** Passive central checks over the existing endpoint state machine and outbox.
+ *******************************************************************************/
+public class WebhooksHealthChecks implements HealthCheckSupplier
+{
+   /*******************************************************************************
+    ** Register the dispatcher and dynamic endpoint supplier from the producer.
+    *******************************************************************************/
+   public static void register(QInstance instance)
+   {
+      HealthChecksMetaData.ofOrWithNew(instance)
+         .withCheck(definition("webhooks.dispatcher", WebhooksHealthChecks::dispatcher))
+         .withSupplier(new HealthCheckSupplierMetaData().withNamePrefix("webhooks.endpoint").withComponent("webhooks")
+            .withSupplier(new QCodeReference(WebhooksHealthChecks.class)));
+   }
+
+
+
+   /*******************************************************************************
+    ** Keep auto-disabled endpoints visible until an operator recovers them.
+    *******************************************************************************/
+   @Override
+   public List<HealthCheckMetaData> supply(QInstance qInstance) throws QException
+   {
+      List<HealthCheckMetaData> checks = new ArrayList<>();
+      for(QRecord endpoint : SystemWrites.call(() -> new QueryAction().execute(new QueryInput(WebhooksTables.ENDPOINT)
+         .withFilter(new QQueryFilter(new QFilterCriteria("activeStatus", QCriteriaOperator.IN, ActiveStatus.ACTIVE.name(), ActiveStatus.DISABLED.name())))
+         .withFieldNamesToInclude(Set.of("id", "activeStatus", "autoDisabledDate")).withTimeoutSeconds(5)).getRecords()))
+      {
+         if(monitored(endpoint))
+         {
+            Integer id = endpoint.getValueInteger("id");
+            checks.add(definition("webhooks.endpoint." + id.toString().toLowerCase(Locale.ROOT), input -> endpoint(input, id)));
+         }
+      }
+      return (checks);
+   }
+
+
+
+   /*******************************************************************************
+    ** Common metadata and a boundary that never exposes backend error text.
+    *******************************************************************************/
+   private static HealthCheckMetaData definition(String name, HealthCheck check)
+   {
+      return (new HealthCheckMetaData().withName(name).withLabel(name).withComponent("webhooks")
+         .withCategory(HealthCategory.INTEGRATION).withKind(HealthCheckKind.DEPENDENCY).withIntervalSeconds(300).withTimeoutMs(5000)
+         .withCheckInstance(input ->
+         {
+            if(expired(input))
+            {
+               return (HealthCheckOutcome.down("Health check timed out"));
+            }
+            try
+            {
+               HealthCheckOutcome outcome = check.run(input);
+               return (expired(input) ? HealthCheckOutcome.down("Health check timed out") : outcome);
+            }
+            catch(Exception e)
+            {
+               return (HealthCheckOutcome.down("Unable to read webhook health"));
+            }
+         }));
+   }
+
+
+
+   /*******************************************************************************
+    ** Read the latest EndpointHealth state, without probing or mutating it.
+    *******************************************************************************/
+   private static HealthCheckOutcome endpoint(HealthCheckInput input, Integer id) throws QException
+   {
+      List<QRecord> records = query(input, WebhooksTables.ENDPOINT,
+         new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.EQUALS, id)).withLimit(1),
+         Set.of("id", "activeStatus", "autoDisabledDate", "healthStatus", "consecutiveFailures", "unhealthySinceDate"));
+      if(records.isEmpty() || !monitored(records.getFirst()))
+      {
+         return (HealthCheckOutcome.unknown("Endpoint is absent or disabled"));
+      }
+      QRecord endpoint = records.getFirst();
+      String health = endpoint.getValueString("healthStatus");
+      HealthCheckOutcome outcome;
+      if(ActiveStatus.DISABLED.name().equals(endpoint.getValueString("activeStatus")))
+      {
+         outcome = HealthCheckOutcome.down("Endpoint automatically disabled");
+      }
+      else if(HealthStatus.HEALTHY.name().equals(health))
+      {
+         outcome = HealthCheckOutcome.up("Endpoint healthy");
+      }
+      else if(HealthStatus.UNHEALTHY.name().equals(health) || HealthStatus.PROBATION.name().equals(health))
+      {
+         outcome = HealthCheckOutcome.degraded("Endpoint unhealthy or on probation");
+      }
+      else
+      {
+         return (HealthCheckOutcome.unknown("Endpoint health is unavailable"));
+      }
+      // Only known state names may enter details, even if persisted data is corrupt.
+      String safeHealth = List.of("HEALTHY", "UNHEALTHY", "PROBATION").contains(health == null ? "" : health) ? health : "UNKNOWN";
+      Integer failures = endpoint.getValueInteger("consecutiveFailures");
+      HealthDetails details = new HealthDetails().withValue("healthStatus", safeHealth)
+         .withValue("consecutiveFailures", failures == null ? 0 : failures);
+      Instant unhealthySince = endpoint.getValueInstant("unhealthySinceDate");
+      if(unhealthySince != null)
+      {
+         details.withValue("unhealthySince", unhealthySince);
+      }
+      return (outcome.withDetails(details));
+   }
+
+
+
+   /*******************************************************************************
+    ** Age of the oldest queued delivery, including retries and in-flight work.
+    *******************************************************************************/
+   private static HealthCheckOutcome dispatcher(HealthCheckInput input) throws QException
+   {
+      List<QRecord> records = query(input, WebhooksTables.DELIVERY,
+         new QQueryFilter(new QFilterCriteria("status", QCriteriaOperator.IN,
+            DeliveryStatus.PENDING.name(), DeliveryStatus.RETRY_WAIT.name(), DeliveryStatus.SENDING.name()))
+            .withOrderBy(new QFilterOrderBy("createDate", true)).withLimit(1), Set.of("createDate"));
+      long seconds = 0;
+      if(!records.isEmpty())
+      {
+         Instant queued = records.getFirst().getValueInstant("createDate");
+         if(queued == null)
+         {
+            return (HealthCheckOutcome.unknown("Pending delivery age is unavailable"));
+         }
+         seconds = Math.max(0, Duration.between(queued, WebhooksRuntime.clock().instant()).getSeconds());
+      }
+      HealthCheckOutcome outcome = seconds > 3600 ? HealthCheckOutcome.down("Delivery backlog exceeds one hour")
+         : seconds > 900 ? HealthCheckOutcome.degraded("Delivery backlog exceeds fifteen minutes") : HealthCheckOutcome.up("Delivery backlog within threshold");
+      return (outcome.withMetrics(List.of(new HealthMetric().withName("oldestPendingSeconds").withValue(seconds).withUnit("seconds"))));
+   }
+
+
+
+   /*******************************************************************************
+    ** Only active or automatically disabled endpoints participate.
+    *******************************************************************************/
+   private static boolean monitored(QRecord endpoint)
+   {
+      String active = endpoint.getValueString("activeStatus");
+      return (ActiveStatus.ACTIVE.name().equals(active)
+         || (ActiveStatus.DISABLED.name().equals(active) && endpoint.getValue("autoDisabledDate") != null));
+   }
+
+
+
+   /*******************************************************************************
+    ** Query only allow-listed fields with the remaining database time budget.
+    ** The runner enforces the precise deadline when the backend blocks.
+    *******************************************************************************/
+   private static List<QRecord> query(HealthCheckInput input, String table, QQueryFilter filter, Set<String> fields) throws QException
+   {
+      int seconds = (int) Math.max(1, Math.min(5, Duration.between(Instant.now(), input.getDeadline()).toSeconds()));
+      return (SystemWrites.call(() -> new QueryAction().execute(new QueryInput(table).withFilter(filter)
+         .withFieldNamesToInclude(fields).withTimeoutSeconds(seconds)).getRecords()));
+   }
+
+
+
+   /*******************************************************************************
+    ** Use wall time for the runner deadline, independently of the domain clock.
+    *******************************************************************************/
+   private static boolean expired(HealthCheckInput input)
+   {
+      return (!Instant.now().isBefore(input.getDeadline()));
+   }
+}
@@ -0,0 +1,70 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.webhooks.health;
+
+
+import java.util.List;
+import java.util.Set;
+import com.kingsrook.qqq.backend.core.actions.ai.output.AiLinkBuilder;
+import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer;
+import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.health.HealthMetaDataProvider;
+import com.kingsrook.qqq.backend.core.health.HealthStatus;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
+import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
+import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput;
+import com.kingsrook.qqq.backend.core.model.dashboard.widgets.StatisticsData;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+
+
+/*******************************************************************************
+ ** One native status line linking to central health, using the viewer's access.
+ *******************************************************************************/
+public class WebhooksHealthWidget extends AbstractWidgetRenderer
+{
+   /*******************************************************************************
+    ** Read the central component rollup; never run a probe from the home.
+    *******************************************************************************/
+   @Override
+   public RenderWidgetOutput render(RenderWidgetInput input) throws QException
+   {
+      String status = "No checks have run yet";
+      if(QContext.getQInstance().getTable(HealthMetaDataProvider.COMPONENT_TABLE) != null)
+      {
+         List<QRecord> records = new QueryAction().execute(new QueryInput(HealthMetaDataProvider.COMPONENT_TABLE)
+            .withFilter(new QQueryFilter(new QFilterCriteria("name", QCriteriaOperator.EQUALS, "webhooks")).withLimit(1))
+            .withFieldNamesToInclude(Set.of("status"))).getRecords();
+         if(!records.isEmpty())
+         {
+            String stored = records.getFirst().getValueString("status");
+            status = List.of("UP", "DEGRADED", "DOWN", "UNKNOWN").contains(stored == null ? "" : stored) ? stored : HealthStatus.UNKNOWN.name();
+         }
+      }
+      return (new RenderWidgetOutput(new StatisticsData().withCount(status).withCountContext("Webhooks - System Health")
+         .withCountURL(AiLinkBuilder.filteredTableHref(HealthMetaDataProvider.CHECK_TABLE,
+            new QQueryFilter(new QFilterCriteria("component", QCriteriaOperator.EQUALS, "webhooks"))))));
+   }
+}
@@ -21,17 +21,14 @@
 package com.kingsrook.qbits.webhooks.metadata;


-import java.io.IOException;
-import java.io.InputStream;
 import java.util.LinkedHashMap;
 import java.util.Map;
-import com.fasterxml.jackson.databind.JsonNode;
-import com.fasterxml.jackson.databind.ObjectMapper;
-import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
 import com.kingsrook.qbits.webhooks.WebhooksQBitConfig;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
 import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
+import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundleEntry;
+import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundles;
 import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;

@@ -40,12 +37,12 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 ** Applies help/qbit-webhooks.yaml (the QQQ help-bundle format) to the
 ** Webhooks app and its webhooks_* tables: what each table is for and how its
 ** rows get created, shown on the table screen and when it is empty.
- ** STAND-IN(core=QRun-IO/qqq#1198, removeBy=4.1.0-RC.8): replace with HelpBundles.register once the pinned
- ** QQQ release includes it; the bundle file stays the same.
+ ** Retains inline empty-state help for hosts without the helpContent table.
+ ** Core owns bundle parsing, registration and persistent help synchronization.
 *******************************************************************************/
 public final class WebhooksHelp
 {
-   public static final String RESOURCE = "/help/qbit-webhooks.yaml";
+   public static final String RESOURCE = "help/qbit-webhooks.yaml";
   public static final String SLOT     = "help";


@@ -62,35 +59,14 @@ public final class WebhooksHelp
   /*******************************************************************************
    ** The bundle's entries by key, validated (fails startup if malformed).
    *******************************************************************************/
-   public static Map<String, JsonNode> entries() throws QException
+   public static Map<String, HelpBundleEntry> entries() throws QException
   {
-      try(InputStream stream = WebhooksHelp.class.getResourceAsStream(RESOURCE))
+      Map<String, HelpBundleEntry> entries = new LinkedHashMap<>();
+      for(HelpBundleEntry entry : HelpBundles.load(RESOURCE, WebhooksHelp.class.getClassLoader()).getEntries())
      {
-         if(stream == null)
-         {
-            throw (new IOException("missing resource"));
-         }
-         JsonNode bundle = new ObjectMapper(new YAMLFactory()).readTree(stream);
-         if(!"qbit-webhooks".equals(bundle.path("bundle").asText()) || !"QBIT".equals(bundle.path("layer").asText()))
-         {
-            throw (new IOException("invalid bundle header"));
-         }
-         Map<String, JsonNode> entries = new LinkedHashMap<>();
-         for(JsonNode entry : bundle.path("entries"))
-         {
-            String key     = entry.path("key").asText();
-            String summary = entry.path("summary").asText();
-            if(key.isBlank() || summary.isBlank() || summary.length() > 120 || entry.path("content").asText().isBlank() || entries.put(key, entry) != null)
-            {
-               throw (new IOException("invalid or duplicate entry: " + key));
-            }
-         }
-         return (entries);
-      }
-      catch(IOException | RuntimeException e)
-      {
-         throw (new QException("Could not load " + RESOURCE, e));
+         entries.put(entry.getKey(), entry);
      }
+      return (entries);
   }


@@ -102,17 +78,17 @@ public final class WebhooksHelp
    *******************************************************************************/
   public static void apply(QInstance qInstance, WebhooksQBitConfig config) throws QException
   {
-      Map<String, JsonNode> entries = entries();
+      Map<String, HelpBundleEntry> entries = entries();
      for(QTableMetaData table : qInstance.getTables().values())
      {
-         JsonNode entry = entries.get("table:" + table.getName() + ";slot:" + SLOT);
+         HelpBundleEntry entry = entries.get("table:" + table.getName() + ";slot:" + SLOT);
         if(entry != null && (table.getHelpContent() == null || !table.getHelpContent().containsKey(SLOT)))
         {
            table.withHelpContent(SLOT, content(entry));
         }
      }
-      QAppMetaData app      = qInstance.getApp(config.getAppName());
-      JsonNode     appEntry = entries.get("app:webhooks;slot:" + SLOT);
+      QAppMetaData    app      = qInstance.getApp(config.getAppName());
+      HelpBundleEntry appEntry = entries.get("app:webhooks;slot:" + SLOT);
      if(app != null && appEntry != null && (app.getHelpContent() == null || !app.getHelpContent().containsKey(SLOT)))
      {
         app.withHelpContent(SLOT, content(appEntry));
@@ -124,9 +100,9 @@ public final class WebhooksHelp
   /*******************************************************************************
    ** Summary as the first paragraph, then the content, as markdown.
    *******************************************************************************/
-   static QHelpContent content(JsonNode entry)
+   static QHelpContent content(HelpBundleEntry entry)
   {
-      return (new QHelpContent().withContentAsMarkdown(entry.path("summary").asText().strip() + "\n\n" + entry.path("content").asText().strip()));
+      return (new QHelpContent().withContentAsMarkdown(entry.getSummary().strip() + "\n\n" + entry.getContent().strip()));
   }

 }
@@ -1,38 +1,576 @@
 bundle: qbit-webhooks
-version: 1
+version: ${project.version}
 layer: QBIT
-# Help for the Webhooks app and every webhooks_* table, in the QQQ help-bundle
-# format (qqq#1198). Each table entry says what the table is for and how its rows
-# get created, so an empty table explains itself. WebhooksHelp applies it until
-# the core HelpBundles registry ships in a QQQ release (a stand-in until that ships).
-# Summaries stay under 120 characters (WebhooksTablesGuardrailTest).
 entries:
-  - key: app:webhooks;slot:help
-    summary: "Send signed HTTP notifications to other systems when things happen in this application."
-    content: |
-      The Setup card on the home lists each step with its state and the next step. Nothing is seeded: you choose where events go. Add an endpoint (a URL), give it a signing secret, then subscribe it to event types. The event types you can pick come from the code: the host lists the tables and processes that raise events in the qbit's configuration (`tableEventSources`, `processEventSources`, ESB topics) and can fire custom ones, and the qbit adds a Test event.
+- key: app:webhooks;slot:help
+  summary: Send signed event messages to other systems from your application.
+  content: |
+    Webhooks tell outside services (a build, a chat room, another app) that something happened in your application. You add an **Endpoint** (the address to call), then add a **Subscription** that says which events it wants. Each matching event becomes one **Delivery**, which is sent with a signature, retried when it fails, and kept as a record you can inspect.

-  - key: table:webhooks_endpoint;slot:help
-    summary: "A receiver URL that gets webhooks: its status, health, rate limit, timeout and unsigned or internal-network switches."
-    content: |
-      **How rows get created:** an administrator adds one with New (a name and an `https` URL). Then add a signing secret and a subscription. Send Test Event checks it end to end. Health moves on its own (Healthy, Probation, Unhealthy) as deliveries succeed or fail, and an endpoint that keeps failing on probation is disabled.
+    Your administrator configures which table changes, process results, message-bus events and custom events are available. Use the **Setup** card to check event types, endpoints, signing secrets and subscriptions. Nothing is seeded; you choose your receivers.

-  - key: table:webhooks_endpoint_secret;slot:help
-    summary: "The signing secrets for an endpoint: generated and encrypted, or the name of an environment variable."
-    content: |
-      **How rows get created:** an administrator adds one on the endpoint, either a generated secret (needs the encryption key variable, `WEBHOOKS_SECRET_KEY` by default) or the name of an environment variable that holds it. The Rotate Signing Secret action adds a new one and keeps the old one signing until Signs Until, so receivers can switch over. Secrets past Signs Until stop signing and are retired by the maintenance job. Secret values are never shown here, only fingerprints.
+    The home screen shows deliveries and failures for the last 24 hours, dead letters, how many are waiting, delivery speed, and central endpoint and dispatcher status with a link to System Health. Under it you can open **Endpoints**, **{{label:webhooks_subscription}}** and **Deliveries**. Signing secrets and attempts are shown inside the endpoint and the delivery.

-  - key: table:webhooks_subscription;slot:help
-    summary: "Which events an endpoint receives: event type, optional record filter, detail level and field limits."
-    content: |
-      **How rows get created:** an administrator adds one on the endpoint or with New: pick the endpoint and an event type. The event types listed come from the code-side registry: `<table>.inserted|updated|deleted` for each table in `tableEventSources`, `process.<name>.completed|failed` for `processEventSources`, ESB topics, custom events the host fires, and the Test event. A record filter limits which records fire it.
+    Viewing needs the webhooks view permission, changing endpoints and re-sending needs manage, and secrets, internal-network targets and deleting need admin. Delivery is at least once, so receivers must tolerate a repeat.
+- key: table:webhooks_endpoint;slot:help
+  summary: A web address that receives signed event messages, with its sending limits and health.
+  content: |
+    **How rows get created:** You add a receiver with **New**, then give it a signing secret and subscription.

-  - key: table:webhooks_delivery;slot:help
-    summary: "One row per event per endpoint: the stored payload, status, attempts, next attempt and last error."
-    content: |
-      **How rows get created:** written only by the qbit, when an event matches a subscription. Empty means no subscribed event has happened yet. Use Send Now, Cancel Delivery or Replay on dead letters to act on them; the Deliver Webhooks job sends them and retries on a schedule.
+    An endpoint is one receiver. Give it a name, the https address, and a status, then add **{{label:webhooks_subscription}}** to choose what it hears about. Open the record to see its subscriptions, signing secrets and recent deliveries.

-  - key: table:webhooks_delivery_attempt;slot:help
-    summary: "One row per HTTP attempt to an endpoint: outcome, status code, duration, response excerpt and error."
-    content: |
-      **How rows get created:** written only by the qbit each time it tries to send a delivery (the claim that stops two replicas sending the same attempt). Empty means nothing has been sent yet. Old rows are purged by the retention job.
+    - By default, the address must be https, with no user name or password in it. It is checked when you save and again on every send.
+    - Addresses on private or internal networks are refused unless an administrator allows them under **Internal Network (Admin Only)**.
+    - Without a signing secret, deliveries wait and are not sent. Add one with **{{label:webhooks_rotate_secret}}** or an environment reference.
+    - Use **{{label:webhooks_send_test}}** to check that the receiver answers and verifies the signature.
+
+    Endpoints that keep failing are paused automatically and eventually disabled; see **{{label:webhooks_endpoint.healthStatus}}**. Deleting an endpoint needs the webhooks admin permission.
+- key: table:webhooks_endpoint;section:identity
+  summary: The name, address and on/off state of the endpoint.
+- key: table:webhooks_endpoint;section:delivery
+  summary: How fast, how patiently and how strictly messages are sent to this endpoint.
+- key: table:webhooks_endpoint;section:health
+  summary: Failure counts and dates the system keeps up to date as it sends.
+- key: table:webhooks_endpoint;section:network
+  summary: Lets this endpoint be an address on an internal network. Only webhook administrators can change it.
+- key: table:webhooks_endpoint;section:subscriptions
+  summary: The events this endpoint is subscribed to.
+- key: table:webhooks_endpoint;section:secrets
+  summary: The secrets used to sign messages to this endpoint.
+- key: table:webhooks_endpoint;section:deliveries
+  summary: The newest deliveries sent to this endpoint.
+- key: table:webhooks_endpoint;section:dates
+  summary: When the endpoint was created and last changed.
+- key: table:webhooks_endpoint;field:name
+  summary: A short unique name for this receiver, used in lists and on the home screen.
+- key: table:webhooks_endpoint;field:description
+  summary: Optional note on what this receiver is for.
+- key: table:webhooks_endpoint;field:url
+  summary: 'Where messages are sent: a public https address, with no user name or password in it.'
+  content: |
+    Messages are sent as a POST with a JSON body. Redirects are not followed, so use the final address.
+
+    The address is checked when you save. A name that cannot be looked up yet is accepted, because DNS may not exist yet, but every send checks it again. Addresses that resolve to private, loopback or other internal networks are refused unless **{{label:webhooks_endpoint.allowInternalTargets}}** is on for this endpoint.
+- key: table:webhooks_endpoint;field:activeStatus
+  summary: 'Whether this endpoint receives messages: {{values:webhooks_active_status}}.'
+  content: |
+    - **Active**: events are queued and sent.
+    - **Paused**: events are still queued, but nothing is sent until you set it back to Active.
+    - **Disabled**: nothing is queued or sent. The system sets this itself after repeated failed probation attempts; see **{{label:webhooks_endpoint.autoDisabledDate}}**.
+- key: table:webhooks_endpoint;field:healthStatus
+  summary: How well sending to this endpoint is going, set automatically from the results.
+  content: |
+    - **Healthy**: sending normally.
+    - **Unhealthy**: after the configured number of failures in a row (10 by default), sending stops.
+    - **Probation**: after the configured unhealthy period (60 minutes by default), one attempt is made per run. A success makes it Healthy again; a failure makes it Unhealthy.
+
+    After the configured number of failed probations (3 by default) the endpoint is set to Disabled. Replies of "too many requests" (429) and policy blocks do not count as failures. **{{label:webhooks_send_now}}** and **{{label:webhooks_send_test}}** ignore health.
+- key: table:webhooks_endpoint;field:serialDelivery
+  summary: Send one message at a time, oldest first, when the receiver needs events in order.
+  content: |
+    When on, the oldest unfinished delivery for this endpoint blocks the rest until it is delivered or becomes a dead letter. When off (default: {{default:webhooks_endpoint.serialDelivery}}), newer messages go ahead while a failing one waits to retry, so order is not guaranteed.
+- key: table:webhooks_endpoint;field:maxPerMinute
+  summary: Caps how many send attempts this endpoint gets per minute. Leave empty for no limit.
+  content: |
+    Must be at least 1. The count is based on the attempts made in the last minute, so it is approximate: a few extra can slip through.
+- key: table:webhooks_endpoint;field:timeoutSeconds
+  summary: How long to wait for the receiver to answer, in seconds, subject to the limits configured for your application.
+  content: |
+    Leave empty to use your application's timeout (10 seconds by default). A send that takes longer counts as a failed attempt.
+- key: table:webhooks_endpoint;field:allowUnsigned
+  summary: Sends messages without a signature when no signing secret exists. Normally leave this off.
+  content: |
+    Unsigned messages still carry the id and timestamp headers, but the receiver cannot prove they came from your application. With this off, an endpoint with no usable secret does not get anything sent.
+- key: table:webhooks_endpoint;field:allowInternalTargets
+  summary: Permits this endpoint to be an address on an internal or private network. Administrators only.
+  content: |
+    Only webhook administrators can turn this on or change it. Even when on, the cloud metadata addresses are always blocked, and when an allowlist is given only those entries are permitted.
+- key: table:webhooks_endpoint;field:internalAllowlist
+  summary: The internal addresses this endpoint may call, separated by commas or spaces. Administrators only.
+  content: |
+    Each entry is a network range (for example 10.0.0.0/24), a single address, an exact host name, or a host name suffix starting with a dot. An entry that cannot be read is rejected when you save. Only used when **{{label:webhooks_endpoint.allowInternalTargets}}** is on.
+- key: table:webhooks_endpoint;field:consecutiveFailures
+  summary: Failed attempts in a row since the last success, counted automatically.
+- key: table:webhooks_endpoint;field:probationFailures
+  summary: How many times a probation attempt has failed, counted automatically. The configured limit controls when it is disabled.
+- key: table:webhooks_endpoint;field:lastSuccessDate
+  summary: When the last delivery to this endpoint succeeded, filled in automatically.
+- key: table:webhooks_endpoint;field:lastFailureDate
+  summary: When the last attempt to this endpoint failed, filled in automatically.
+- key: table:webhooks_endpoint;field:lastFailureMessage
+  summary: The reason the last attempt failed, such as an HTTP status or a network error. Filled in automatically.
+- key: table:webhooks_endpoint;field:unhealthySinceDate
+  summary: When the endpoint last became unhealthy, filled in automatically. Empty while healthy.
+- key: table:webhooks_endpoint;field:autoDisabledDate
+  summary: When the system disabled this endpoint after repeated failed probation attempts. Set automatically.
+  content: |
+    To use the endpoint again, fix the receiver and set **{{label:webhooks_endpoint.activeStatus}}** back to Active.
+- key: table:webhooks_endpoint_secret;slot:help
+  summary: A secret used to sign the messages sent to one endpoint, so the receiver can verify them.
+  content: |
+    **How rows get created:** You use **{{label:webhooks_rotate_secret}}** to generate a secret, or add an environment reference.
+
+    Every message carries a signature made with the endpoint's active secret. The receiver holds the same secret and checks it.
+
+    There are two kinds. A generated secret is created by **{{label:webhooks_rotate_secret}}** and stored encrypted. An environment reference only names a server setting that holds the secret, so the value never lives in the database.
+
+    The value of a generated secret is never shown in lists or results. Webhook administrators can open the secret record and reveal it behind "Show the secret". During a rotation the old secret stays valid for the configured overlap (24 hours by default) and both sign every message, so the receiver can switch over without missing anything. Adding or changing secrets needs the webhooks admin permission.
+- key: table:webhooks_endpoint_secret;section:identity
+  summary: Which endpoint this secret belongs to, and whether it is in use.
+- key: table:webhooks_endpoint_secret;section:data
+  summary: Where the secret comes from, when it signs until, and its fingerprint.
+- key: table:webhooks_endpoint_secret;section:value
+  summary: Lets webhook administrators reveal a generated secret so they can copy it to the receiver.
+- key: table:webhooks_endpoint_secret;section:dates
+  summary: When the secret was created and last changed.
+- key: table:webhooks_endpoint_secret;field:endpointId
+  summary: The endpoint whose messages this secret signs.
+- key: table:webhooks_endpoint_secret;field:status
+  summary: 'Where the secret is in its life: {{values:webhooks_secret_status}}.'
+  content: |
+    - **Active**: the secret in use.
+    - **Previous (still signing until it expires)**: replaced by a newer one but still signing for the overlap, 24 hours by default, so receivers can switch.
+    - **Retired**: expired and no longer used. The maintenance job retires expired secrets when scheduling is enabled.
+- key: table:webhooks_endpoint_secret;field:source
+  summary: Whether the secret is stored encrypted here, or read from a server setting.
+  content: |
+    - **Generated (stored encrypted)**: created by **{{label:webhooks_rotate_secret}}**. You cannot add these by hand.
+    - **Environment reference**: you name a server setting under **{{label:webhooks_endpoint_secret.secretRef}}**; its value is read each time a message is sent.
+- key: table:webhooks_endpoint_secret;field:secretRef
+  summary: The name of the server setting that holds the secret, for environment references only.
+  content: |
+    Enter only the variable name (letters, digits and underscores), never the secret itself. The value in that variable must be a whsec_ secret or at least 16 bytes long; a shorter one is rejected when you save.
+- key: table:webhooks_endpoint_secret;field:encryptedSecret
+  summary: The encrypted form of a generated secret. Filled in automatically and not readable by people.
+- key: table:webhooks_endpoint_secret;field:fingerprint
+  summary: A short code that identifies the secret without revealing it. Filled in automatically.
+  content: |
+    The same fingerprint appears under **Signed With (Fingerprints)** on each attempt, so you can tell which secret signed a message.
+- key: table:webhooks_endpoint_secret;field:expiresDate
+  summary: When a Previous secret stops signing. Set automatically when a secret is replaced.
+  content: |
+    By default, it is 24 hours after the rotation; your application can configure another overlap. An active secret has no end date.
+- key: table:webhooks_endpoint_secret;field:retiredDate
+  summary: When the secret was retired, filled in automatically.
+- key: table:webhooks_endpoint_secret;field:note
+  summary: Optional note, such as who holds the matching copy of the secret.
+- key: table:webhooks_subscription;slot:help
+  summary: Chooses which events go to an endpoint, optionally filtered, and how much of the record is sent.
+  content: |
+    **How rows get created:** You add a subscription and choose an endpoint and event type. Your administrator configures the available events.
+
+    A subscription links one **{{label:webhooks_subscription.endpointId}}** to one **{{label:webhooks_subscription.eventTypeName}}**. When a matching event happens, one delivery is queued for that endpoint.
+
+    An event matches when all of these hold:
+    - the subscription is Active and its endpoint is not Disabled;
+    - the event type is the same;
+    - the **{{label:webhooks_subscription.filterJson}}**, if any, matches the record;
+    - the record is one the subscription is allowed to read.
+
+    Event types look like a table name plus inserted, updated or deleted, or the completed or failed result of a configured process. You can only subscribe to a table's events if you can read that table, and an unknown event type is rejected.
+
+    Use **{{label:webhooks_subscription.detailLevel}}**, **{{label:webhooks_subscription.includeFields}}** and **{{label:webhooks_subscription.excludeFields}}** to control what the message contains. The message is built when the event happens, so retries send exactly the same content.
+- key: table:webhooks_subscription;section:identity
+  summary: The endpoint, the event type and whether the subscription is on.
+- key: table:webhooks_subscription;section:filter
+  summary: An optional note and a filter that limits which records trigger a message.
+- key: table:webhooks_subscription;section:payload
+  summary: How much of the record is sent and how field names are shown.
+- key: table:webhooks_subscription;section:deliveries
+  summary: The newest deliveries made for this subscription.
+- key: table:webhooks_subscription;section:dates
+  summary: When the subscription was created and last changed.
+- key: table:webhooks_subscription;field:endpointId
+  summary: The endpoint that receives the messages for this subscription.
+- key: table:webhooks_subscription;field:eventTypeName
+  summary: The event this subscription listens for, for example a table being inserted, updated or deleted.
+  content: |
+    Pick from the list. Table events are named after the table plus inserted, updated or deleted. Configured process events report completed or failed. Custom and message-bus events may also be available. You need permission to read the table.
+- key: table:webhooks_subscription;field:activeStatus
+  summary: 'Whether this subscription is used: {{values:webhooks_active_status}}.'
+  content: |
+    Only Active subscriptions queue new deliveries.
+- key: table:webhooks_subscription;field:description
+  summary: Optional note on why this subscription exists.
+- key: table:webhooks_subscription;field:filterJson
+  summary: Only send events for records that match this filter. Leave empty to send them all.
+  content: |
+    Enter a query filter as JSON, with criteria on the record's field names. It is checked against the record as it is after the change, with the old values filled in for anything not changed. If the JSON cannot be read, saving fails with a message.
+- key: table:webhooks_subscription;field:detailLevel
+  summary: 'How much of the record each message carries: the full record, only what changed, or just the keys.'
+  content: |
+    - **Full record** (default: {{default:webhooks_subscription.detailLevel}}): the whole record, plus the old record for updates.
+    - **Changed fields only**: the fields that changed, and the record key.
+    - **Keys only**: just the record key, so the receiver fetches the rest itself.
+- key: table:webhooks_subscription;field:includeFields
+  summary: If set, only these fields are sent. Separate names with commas or spaces. Leave empty for all.
+  content: |
+    Use the field names exactly as the record uses them. Combine with **{{label:webhooks_subscription.excludeFields}}** to remove a few.
+- key: table:webhooks_subscription;field:excludeFields
+  summary: Fields that are always left out of the message. Separate names with commas or spaces.
+- key: table:webhooks_subscription;field:apiName
+  summary: Send the record in the form of a published API, with that API's field names. Leave empty for the plain record.
+  content: |
+    Set together with **{{label:webhooks_subscription.apiVersion}}**. Both must be filled in for this to apply.
+- key: table:webhooks_subscription;field:apiVersion
+  summary: The version of the named API used to shape the record.
+- key: table:webhooks_delivery;slot:help
+  summary: One message queued for one endpoint, with its status, attempts and stored content.
+  content: |
+    **How rows get created:** A matching event queues a delivery automatically. Test sends and direct sends also create deliveries.
+
+    The system writes a delivery when an event matches a subscription, and updates it as it is sent. You cannot edit one. Use actions on selected deliveries instead.
+
+    A delivery moves from **Pending** to **Sending**, then **{{label:webhooks_delivery.deliveredDate}}**. If a send fails it follows your application's retry schedule. The default delays are 1, 5, 15, 60 and 240 minutes before the sixth attempt. After the configured attempt limit (6 by default) it becomes a **Dead letter**. A reply of "too many requests" (429) waits as the receiver asked and uses up no attempt.
+
+    - **{{label:webhooks_send_now}}** tries unfinished deliveries immediately.
+    - **{{label:webhooks_replay}}** queues finished or dead deliveries again with the same event id.
+    - **{{label:webhooks_cancel}}** stops pending ones.
+    - **{{label:webhooks_delete_deliveries}}** removes finished ones (administrators only).
+
+    Your application's retention settings determine when finished deliveries are purged. Open a delivery to see each attempt and the exact content that was sent.
+- key: table:webhooks_delivery;section:identity
+  summary: The event type, the endpoint it went to, and the current status.
+- key: table:webhooks_delivery;section:progress
+  summary: How many attempts were made, when the next one is due and how the last one ended.
+- key: table:webhooks_delivery;section:source
+  summary: 'What caused this delivery: the event id and the record or run it came from.'
+- key: table:webhooks_delivery;section:attempts
+  summary: Every send attempt for this delivery, newest included.
+- key: table:webhooks_delivery;section:payloadSection
+  summary: The exact JSON content that is sent to the endpoint.
+- key: table:webhooks_delivery;section:dates
+  summary: When the delivery was queued and last changed.
+- key: table:webhooks_delivery;field:eventId
+  summary: The unique id of the event, sent to the receiver as the webhook-id header so it can ignore repeats.
+  content: |
+    It starts with evt_ and stays the same on every retry and replay. Receivers should remember the ids they have already processed.
+- key: table:webhooks_delivery;field:eventTypeName
+  summary: The kind of event that was sent, for example a table insert or a publish completion.
+- key: table:webhooks_delivery;field:endpointId
+  summary: The endpoint this message is sent to.
+- key: table:webhooks_delivery;field:subscriptionId
+  summary: The subscription that caused this delivery. Empty for a test event.
+- key: table:webhooks_delivery;field:source
+  summary: What kind of thing produced the event.
+  content: |
+    - **Table change**: a record was inserted, updated or deleted in your application.
+    - **ESB event**: the same kind of change, reported by another service.
+    - **Process**: a configured process completed or failed.
+    - **Custom event**: an event raised by your application.
+    - **Channel (another qbit)**: sent straight to one endpoint by another part of the system, such as alerts.
+    - **Test send**: made by **{{label:webhooks_send_test}}**.
+- key: table:webhooks_delivery;field:sourceTable
+  summary: The table the changed record belongs to, when the event came from a record change.
+- key: table:webhooks_delivery;field:sourceRecordId
+  summary: The id of the record that changed, when the event came from a record change.
+- key: table:webhooks_delivery;field:sourceEventId
+  summary: The id of the original event or change this delivery came from, used to avoid queueing it twice.
+- key: table:webhooks_delivery;field:dedupKey
+  summary: An internal key that makes sure the same event is queued only once per subscription.
+- key: table:webhooks_delivery;field:status
+  summary: 'Where the delivery is: {{values:webhooks_delivery_status}}.'
+  content: |
+    - **Pending**: queued, not yet tried.
+    - **Sending**: an attempt is in progress.
+    - **Waiting to retry**: the last attempt failed; the next one is due at **{{label:webhooks_delivery.nextAttemptDate}}**.
+    - **{{label:webhooks_delivery.deliveredDate}}**: the receiver answered with success (2xx).
+    - **Dead letter**: reached the configured attempt limit since the last replay. You can replay or delete it; configured retention may also remove it.
+    - **Cancelled**: stopped with **{{label:webhooks_cancel}}**.
+- key: table:webhooks_delivery;field:attemptCount
+  summary: Total send attempts made for this delivery, including before any replay.
+- key: table:webhooks_delivery;field:cycleAttemptCount
+  summary: Attempts made since the delivery was queued or last replayed. The configured limit makes it a dead letter.
+  content: |
+    Replies of "too many requests" (429) do not count.
+- key: table:webhooks_delivery;field:replayCount
+  summary: How many times this delivery has been queued again with Replay.
+- key: table:webhooks_delivery;field:occurredDate
+  summary: When the event happened, which can be earlier than when the message was sent.
+- key: table:webhooks_delivery;field:nextAttemptDate
+  summary: When the next attempt is due. Empty once the delivery is finished.
+  content: |
+    Retries follow your application's configured delays and attempt limit. A "too many requests" reply honors the receiver's delay up to the configured maximum (one hour by default).
+- key: table:webhooks_delivery;field:leaseExpiresDate
+  summary: While Sending, the time after which another attempt may take over if the first one has stalled.
+  content: |
+    Normally 2 minutes after the attempt started.
+- key: table:webhooks_delivery;field:lastHttpStatus
+  summary: The HTTP status code the receiver answered with on the last attempt. Empty if it did not answer.
+- key: table:webhooks_delivery;field:lastError
+  summary: Why the last attempt failed, such as a network error, a blocked address or an HTTP status.
+- key: table:webhooks_delivery;field:deliveredDate
+  summary: When the receiver accepted the message. Empty until delivered.
+- key: table:webhooks_delivery;field:deadLetterDate
+  summary: When the delivery ran out of attempts and became a dead letter.
+- key: table:webhooks_delivery;field:payloadBytes
+  summary: Size of the stored JSON content in bytes.
+- key: table:webhooks_delivery;field:payload
+  summary: The exact JSON that is sent, built when the event happened, so every retry sends the same content.
+  content: |
+    It contains the event id, type, time, source and a data section with the record. Which fields appear is decided by the subscription's detail level and field lists.
+- key: table:webhooks_delivery_attempt;slot:help
+  summary: One try at sending a delivery, with the result, timing and what the receiver said.
+  content: |
+    **How rows get created:** The system records each attempt to send a delivery. An empty list means no attempt has been recorded.
+
+    The system adds a row each time it sends a delivery, so you can see exactly what happened: the outcome, HTTP status, how long it took, a short excerpt of the receiver's reply, the error if any, and which secrets signed it. Attempts cannot be edited, and they are shown inside the delivery they belong to. Your application's retention settings determine when old attempts are removed.
+- key: table:webhooks_delivery_attempt;section:identity
+  summary: The delivery and endpoint this attempt belongs to, and how it ended.
+- key: table:webhooks_delivery_attempt;section:data
+  summary: The request, the receiver's response and the timing of this attempt.
+- key: table:webhooks_delivery_attempt;section:dates
+  summary: When this attempt record was created and last changed.
+- key: table:webhooks_delivery_attempt;field:deliveryId
+  summary: The delivery that was being sent.
+- key: table:webhooks_delivery_attempt;field:endpointId
+  summary: The endpoint the request went to.
+- key: table:webhooks_delivery_attempt;field:attemptNumber
+  summary: 'Which try this was for the delivery: 1, 2, 3 and so on.'
+- key: table:webhooks_delivery_attempt;field:outcome
+  summary: How the attempt ended.
+  content: |
+    - **In flight**: still being sent.
+    - **Succeeded**: the receiver answered with success (2xx).
+    - **Failed**: an error status, a network problem or a timeout.
+    - **Rate limited (429)**: the receiver asked us to slow down; this uses no attempt.
+    - **Blocked by policy**: the address is not allowed, for example an internal network. It does not count against endpoint health.
+    - **Lease expired**: the sender stopped before recording a result, so a new attempt took over.
+- key: table:webhooks_delivery_attempt;field:claimedBy
+  summary: Which server instance made this attempt, filled in automatically.
+- key: table:webhooks_delivery_attempt;field:leaseExpiresDate
+  summary: The time this attempt was allowed to run until before another could take over.
+- key: table:webhooks_delivery_attempt;field:startDate
+  summary: When the request started.
+- key: table:webhooks_delivery_attempt;field:endDate
+  summary: When the request finished.
+- key: table:webhooks_delivery_attempt;field:durationMillis
+  summary: How long the request took, in milliseconds.
+- key: table:webhooks_delivery_attempt;field:httpStatus
+  summary: The HTTP status code in the receiver's reply. Empty if there was no reply.
+- key: table:webhooks_delivery_attempt;field:responseExcerpt
+  summary: The start of the receiver's reply, up to 2,000 characters; longer replies are marked as cut off.
+- key: table:webhooks_delivery_attempt;field:errorMessage
+  summary: What went wrong, if the attempt did not succeed.
+- key: table:webhooks_delivery_attempt;field:remoteAddress
+  summary: The network address the request actually connected to.
+- key: table:webhooks_delivery_attempt;field:requestUrl
+  summary: The address the request was sent to, as it was when this attempt ran.
+- key: table:webhooks_delivery_attempt;field:signedWith
+  summary: Fingerprints of the secrets that signed this request, separated by spaces.
+  content: |
+    Two fingerprints mean a secret rotation is in progress. Match them against the fingerprints under **Signing Secrets**.
+- key: process:webhooks_send_now;slot:help
+  summary: Tries to send the selected deliveries right now instead of waiting for their turn.
+  content: |
+    Select one or more deliveries and run it. Each unfinished delivery gets an attempt immediately, even if its endpoint is unhealthy or paused.
+
+    Deliveries that are already delivered, are dead letters (use **{{label:webhooks_replay}}** for those) or are cancelled are skipped. The result tells you how many were delivered and how many failed; failed ones go on retrying by the normal schedule. A delivery that another sender is working on at that moment is skipped too.
+
+    Needs the webhooks manage permission.
+- key: process:webhooks_replay;slot:help
+  summary: Queues finished deliveries again, with the same event id and a fresh set of attempts.
+  content: |
+    Use it to resend something the receiver missed: a dead letter, or a delivery you want sent a second time. The delivery goes back to Pending, its attempt count since replay resets, and its **Replays** counter goes up. The event id stays the same, so a receiver that deduplicates will notice the repeat.
+
+    Deliveries that are still being delivered are skipped. It does not send straight away; the next scheduled run picks it up when scheduling is enabled (every 15 seconds by default), or use **{{label:webhooks_send_now}}**.
+
+    Needs the webhooks manage permission.
+- key: process:webhooks_cancel;slot:help
+  summary: Stops the selected deliveries that have not been sent yet, so they will not be tried again.
+  content: |
+    Works on Pending and Waiting to retry deliveries. Deliveries being sent at that moment, and those already delivered, dead or cancelled, are skipped. A cancelled delivery can be queued again later with **{{label:webhooks_replay}}**.
+
+    Needs the webhooks manage permission.
+- key: process:webhooks_delete_deliveries;slot:help
+  summary: Permanently deletes finished deliveries and all of their attempts.
+  content: |
+    Only Delivered, Dead letter and Cancelled deliveries are deleted; anything still pending or waiting is skipped, so cancel it first. This cannot be undone, and the history is lost with it. Replay dead letters before deleting if the receiver still needs them.
+
+    Needs the webhooks admin permission.
+- key: process:webhooks_send_test;slot:help
+  summary: Sends a signed test message to the selected endpoint now and shows whether it was accepted.
+  content: |
+    Use it after adding an endpoint or changing its address or secret. The message has the type webhooks.test and a short note in its data. It ignores the endpoint's health and subscriptions, and it is recorded as a normal delivery with its attempt, so you can inspect it afterwards.
+
+    Disabled endpoints are skipped. The result shows delivered with the HTTP status, or the error. Needs the webhooks manage permission.
+- key: process:webhooks_rotate_secret;slot:help
+  summary: Creates a new signing secret for the endpoint. The old one keeps signing during the configured overlap.
+  content: |
+    The current Active secret becomes Previous and signs alongside the new one during the configured overlap (24 hours by default), so you can update the receiver without losing any message. After that the old secret is retired automatically.
+
+    The new value is not shown in the result. Open the endpoint, go to **Signing Secrets**, open the new secret and reveal it, then copy it to the receiver. It is stored encrypted.
+
+    If the server has no encryption key set, the action is skipped; add an environment reference instead. Needs the webhooks admin permission.
+- key: process:webhooks_deliver;slot:help
+  summary: Send due deliveries and retries when your application enables scheduling.
+  content: |
+    It sends pending deliveries and retries, honoring each endpoint's rate limit, order setting and health, and updates endpoint health from the results. With scheduling enabled, it runs every 15 seconds by default; use **{{label:webhooks_send_now}}** for a single delivery.
+- key: process:webhooks_maintain;slot:help
+  summary: Maintain endpoint health and retire expired secrets, every five minutes when scheduling is enabled.
+  content: |
+    After an endpoint has been Unhealthy for the configured period (60 minutes by default) it goes to Probation so one attempt can be made. Secrets whose overlap has ended are set to Retired.
+- key: process:webhooks_retention;slot:help
+  summary: Remove old finished deliveries and attempts according to your application's retention settings.
+  content: |
+    When scheduling is enabled, this job runs daily. By default, delivered and cancelled deliveries and completed attempts are kept for 400 days; dead letters are kept indefinitely. Your administrator can configure each retention period; zero disables that purge. Deleting a delivery also deletes its attempts. Export any history you need before it expires.
+- key: widget:webhooksEndpointSubscriptions;slot:label
+  summary: The subscriptions that decide which events this endpoint receives.
+- key: widget:webhooksEndpointSecrets;slot:label
+  summary: The signing secrets for this endpoint, with their status and fingerprints. Values are not shown here.
+- key: widget:webhooksEndpointDeliveries;slot:label
+  summary: The most recent deliveries sent to this endpoint.
+- key: widget:webhooksSubscriptionDeliveries;slot:label
+  summary: The most recent deliveries made for this subscription.
+- key: widget:webhooksDeliveryAttempts;slot:label
+  summary: Every send attempt for this delivery, with result, status and timing.
+- key: widget:webhooksHome;slot:label
+  summary: Setup steps, deliveries, failures, dead letters, backlog and speed over the last 24 hours.
+  content: |
+    The Setup card shows configuration steps and their next actions. The tiles show delivered and failed attempts for the last 24 hours, dead letters, deliveries waiting to send and median and 95th percentile response time. Below are an hourly chart and the newest dead letters. Endpoint and dispatcher health appear in the System Health status widget.
+- key: widget:webhooksHealth;slot:label
+  summary: Central endpoint and dispatcher health, with a link to the webhooks checks in System Health.
+- key: widget:webhooksSecretValue;slot:label
+  summary: Reveals the secret value behind a Show button. Webhook administrators only.
+- key: possibleValueSource:webhooks_active_status;slot:help
+  summary: Whether an endpoint or subscription is in use.
+  content: |
+    - **Active**: used normally.
+    - **Paused**: events are still queued for a paused endpoint but not sent; a paused subscription queues nothing new.
+    - **Disabled**: not used. An endpoint can be disabled automatically after repeated failures.
+- key: possibleValueSource:webhooks_health_status;slot:help
+  summary: How well sending to an endpoint is going.
+  content: |
+    - **Healthy**: sending normally.
+    - **Probation**: after an unhealthy spell, one attempt at a time is made to see if it works again.
+    - **Unhealthy**: reached the configured consecutive-failure limit (10 by default), so sending is stopped for a while.
+- key: possibleValueSource:webhooks_delivery_status;slot:help
+  summary: The stage of a delivery.
+  content: |
+    - **Pending**: queued and not yet tried.
+    - **Sending**: an attempt is in progress.
+    - **Waiting to retry**: the last attempt failed and another is scheduled.
+    - **Delivered**: accepted by the receiver.
+    - **Dead letter**: failed too many times and was given up on.
+    - **Cancelled**: stopped by a person.
+- key: possibleValueSource:webhooks_attempt_outcome;slot:help
+  summary: How one send attempt ended.
+  content: |
+    - **In flight**: still running.
+    - **Succeeded**: the receiver accepted it.
+    - **Failed**: error status, network problem or timeout.
+    - **Rate limited (429)**: the receiver asked to slow down.
+    - **Blocked by policy**: the address is not allowed.
+    - **Lease expired**: the sender stopped without recording a result.
+- key: possibleValueSource:webhooks_event_source;slot:help
+  summary: What produced an event.
+  content: |
+    - **Table change**: a record insert, update or delete in your application.
+    - **ESB event**: a change reported by another service.
+    - **Process**: a configured process completing or failing.
+    - **Custom event**: an event raised by your application.
+    - **Channel (another qbit)**: sent directly to one endpoint, for example by alerts.
+    - **Test send**: from **{{label:webhooks_send_test}}**.
+- key: possibleValueSource:webhooks_secret_status;slot:help
+  summary: Where a signing secret is in its life.
+  content: |
+    - **Active**: the secret in use.
+    - **Previous (still signing until it expires)**: replaced, but still signs for the configured overlap (24 hours by default).
+    - **Retired**: no longer used.
+- key: possibleValueSource:webhooks_secret_source;slot:help
+  summary: How a signing secret is kept.
+  content: |
+    - **Generated (stored encrypted)**: created by the system and stored encrypted.
+    - **Environment reference**: only the name of a server setting is stored; the value is read when sending.
+- key: possibleValueSource:webhooks_detail_level;slot:help
+  summary: How much of a record a message contains.
+  content: |
+    - **Full record**: the whole record, plus the old record for updates.
+    - **Changed fields only**: just the fields that changed and the key.
+    - **Keys only**: only the record key.
+- key: possibleValueSource:webhooks_event_type;slot:help
+  summary: 'The events you can subscribe to: table inserts, updates and deletes, and process results and custom events.'
+  content: |
+    Table events are the table name followed by inserted, updated or deleted. You can only pick events for tables you are allowed to read.
+- key: process:webhooks_esb_ingest;slot:help
+  summary: Turns events arriving on the message bus into webhook deliveries; it runs by itself and is not started by hand.
+  content: |
+    It listens on the ESB destinations configured for webhooks. A table event such as a record being inserted, updated or deleted becomes the matching webhook event, and a finished or failed process becomes a process event. Events of unregistered types are ignored. By default, events this server published itself are skipped so nothing is sent twice, and a message delivered again queues nothing new.
+- key: process:webhooks_endpoint.bulkInsert;slot:help
+  summary: Add multiple endpoints from a file.
+  content: |
+    Upload your file, map its columns, and review the preview before running the import.
+
+    You need webhook manage permission to add or edit endpoints and admin permission to delete them. Configure signing secrets and subscriptions separately.
+- key: process:webhooks_endpoint.bulkEdit;slot:help
+  summary: Change selected endpoints together.
+  content: |
+    Select the records you want to change, enter the new values, and review the preview before applying them.
+
+    You need webhook manage permission to add or edit endpoints and admin permission to delete them. Configure signing secrets and subscriptions separately.
+- key: process:webhooks_endpoint.bulkEditWithFile;slot:help
+  summary: Update endpoints using values from a file.
+  content: |
+    Upload your file and map its columns to identify and update existing records. Review the preview before applying changes.
+
+    You need webhook manage permission to add or edit endpoints and admin permission to delete them. Configure signing secrets and subscriptions separately.
+- key: process:webhooks_endpoint.bulkDelete;slot:help
+  summary: Delete selected endpoints.
+  content: |
+    Review the selected records before confirming deletion. This permanently removes records and cannot be undone.
+
+    You need webhook manage permission to add or edit endpoints and admin permission to delete them. Configure signing secrets and subscriptions separately.
+- key: process:webhooks_endpoint_secret.bulkInsert;slot:help
+  summary: Add multiple signing secrets from a file.
+  content: |
+    Upload your file, map its columns, and review the preview before running the import.
+
+    You need webhook admin permission. Add only environment references here; use **{{label:webhooks_rotate_secret}}** to generate an encrypted secret. Never put a secret value in an import file.
+- key: process:webhooks_endpoint_secret.bulkEdit;slot:help
+  summary: Change selected signing secrets together.
+  content: |
+    Select the records you want to change, enter the new values, and review the preview before applying them.
+
+    You need webhook admin permission. Add only environment references here; use **{{label:webhooks_rotate_secret}}** to generate an encrypted secret. Never put a secret value in an import file.
+- key: process:webhooks_endpoint_secret.bulkEditWithFile;slot:help
+  summary: Update signing secrets using values from a file.
+  content: |
+    Upload your file and map its columns to identify and update existing records. Review the preview before applying changes.
+
+    You need webhook admin permission. Add only environment references here; use **{{label:webhooks_rotate_secret}}** to generate an encrypted secret. Never put a secret value in an import file.
+- key: process:webhooks_endpoint_secret.bulkDelete;slot:help
+  summary: Delete selected signing secrets.
+  content: |
+    Review the selected records before confirming deletion. This permanently removes records and cannot be undone.
+
+    You need webhook admin permission. Add only environment references here; use **{{label:webhooks_rotate_secret}}** to generate an encrypted secret. Never put a secret value in an import file.
+- key: process:webhooks_subscription.bulkInsert;slot:help
+  summary: Add multiple subscriptions from a file.
+  content: |
+    Upload your file, map its columns, and review the preview before running the import.
+
+    You need webhook manage permission. Choose valid endpoints and available event types; your access to source records still applies.
+- key: process:webhooks_subscription.bulkEdit;slot:help
+  summary: Change selected subscriptions together.
+  content: |
+    Select the records you want to change, enter the new values, and review the preview before applying them.
+
+    You need webhook manage permission. Choose valid endpoints and available event types; your access to source records still applies.
+- key: process:webhooks_subscription.bulkEditWithFile;slot:help
+  summary: Update subscriptions using values from a file.
+  content: |
+    Upload your file and map its columns to identify and update existing records. Review the preview before applying changes.
+
+    You need webhook manage permission. Choose valid endpoints and available event types; your access to source records still applies.
+- key: process:webhooks_subscription.bulkDelete;slot:help
+  summary: Delete selected subscriptions.
+  content: |
+    Review the selected records before confirming deletion. This permanently removes records and cannot be undone.
+
+    You need webhook manage permission. Choose valid endpoints and available event types; your access to source records still applies.
+- key: possibleValueSource:webhooks_endpoint;slot:help
+  summary: Choose the endpoint that receives your messages.
+- key: possibleValueSource:webhooks_subscription;slot:help
+  summary: Choose the subscription that selects events for an endpoint.
+- key: possibleValueSource:webhooks_delivery;slot:help
+  summary: Choose the delivery whose send attempts you want to inspect.
@@ -0,0 +1,81 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.webhooks;
+
+
+import java.util.List;
+import com.kingsrook.qbits.webhooks.esb.WebhooksEsbSource;
+import com.kingsrook.qbits.webhooks.metadata.WebhooksHelp;
+import com.kingsrook.qqq.backend.core.actions.help.HelpTargetCatalog;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundles;
+import com.kingsrook.qqq.backend.core.testing.HelpBundleAssertions;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+
+
+/*******************************************************************************
+ ** Shipped help covers the qbit, including its optional ESB integration.
+ *******************************************************************************/
+class WebhooksHelpBundleTest
+{
+   /*******************************************************************************
+    ** Validate all entries and live references against produced metadata.
+    *******************************************************************************/
+   @Test
+   void bundleIsComplete() throws Exception
+   {
+      QInstance instance = WebhooksTestSupport.setUp(config -> config.withEsbDestinations(List.of("events")));
+      WebhooksEsbSource.configure(instance);
+      HelpBundleAssertions.forInstance(instance)
+         .bundles("help/qbit-webhooks.yaml")
+         .owning(target -> target.name() != null && target.name().startsWith("webhooks"))
+         .assertComplete();
+      // RC.7 requires app/table/process/field coverage; also enforce our widgets and value sources.
+      var entries = WebhooksHelp.entries();
+      for(var target : HelpTargetCatalog.collect(instance))
+      {
+         if(target.name() != null && target.name().startsWith("webhooks")
+            && List.of("widget", "possibleValueSource").contains(target.kind()))
+         {
+            assertThat(entries).as(target.key()).containsKey(target.key());
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Hosts need no help table or ESB integration to register the bundle.
+    *******************************************************************************/
+   @Test
+   void registersWithoutHelpTable() throws Exception
+   {
+      QInstance instance = WebhooksTestSupport.setUp();
+      assertThat(instance.getTable("helpContent")).isNull();
+      HelpBundles.syncOnStartup(instance);
+      assertThat(HelpBundles.of(instance).getBundles()).singleElement().satisfies(bundle ->
+      {
+         assertThat(bundle.getId()).isEqualTo("qbit-webhooks");
+         assertThat(bundle.getVersion()).as("the help bundle carries the artifact version").isEqualTo(System.getProperty("qbit.artifactVersion"));
+      });
+   }
+}
@@ -90,7 +90,7 @@ class WebhooksQBitProducerTest
      {
         assertThat(child).isNotInstanceOf(QProcessMetaData.class);
      }
-      assertThat(app.getWidgets()).containsExactly("webhooksHome");
+      assertThat(app.getWidgets()).containsExactly("webhooksHealth", "webhooksHome");
      Set<String> endpointWidgets = instance.getTable(WebhooksTables.ENDPOINT).getSections().stream().map(s -> s.getWidgetName()).filter(w -> w != null).collect(Collectors.toSet());
      assertThat(endpointWidgets).contains(WebhooksTables.WIDGET_ENDPOINT_SUBSCRIPTIONS, WebhooksTables.WIDGET_ENDPOINT_SECRETS, WebhooksTables.WIDGET_ENDPOINT_DELIVERIES);
   }
@@ -104,7 +104,7 @@ class WebhooksTablesGuardrailTest
   void everyTableHasEmptyStateHelp() throws Exception
   {
      QInstance instance = WebhooksTestSupport.setUp();
-      assertThat(WebhooksHelp.entries()).allSatisfy((key, entry) -> assertThat(entry.path("summary").asText().length()).as(key).isLessThanOrEqualTo(120));
+      assertThat(WebhooksHelp.entries()).allSatisfy((key, entry) -> assertThat(entry.getSummary().length()).as(key).isLessThanOrEqualTo(120));
      for(String table : WebhooksTables.ALL)
      {
         List<QHelpContent> help = instance.getTable(table).getHelpContent() == null ? null : instance.getTable(table).getHelpContent().get(WebhooksHelp.SLOT);
@@ -112,7 +112,7 @@ class WebhooksTablesGuardrailTest
         assertThat(help.get(0).getContent()).as(table + " help says how rows get created").contains("How rows get created");
      }
      assertThat(instance.getApp("webhooks").getHelpContent()).containsKey(WebhooksHelp.SLOT);
-      assertThat(WebhooksHelp.entries().get("table:webhooks_subscription;slot:help").path("content").asText()).contains("registry");
+      assertThat(WebhooksHelp.entries().get("table:webhooks_subscription;slot:help").getContent()).contains("available events");
      assertThat(WebhooksHelp.entries().keySet()).allMatch(key -> !key.startsWith("table:") || instance.getTable(key.substring(6, key.indexOf(';'))) != null,
         "help names only real tables");
   }
@@ -90,7 +90,7 @@ class WebhooksHomeTest
   void emptyHome() throws Exception
   {
      String html = home();
-      assertThat(html).contains("Setup").contains("Needs setup").contains("No endpoints yet").contains("/admin/webhooks/webhooks_endpoint").contains("No delivery attempts in the last 24 hours")
+      assertThat(html).doesNotContain("Endpoints needing attention").contains("Setup").contains("Needs setup").contains("Add an endpoint: the URL that receives events").contains("/admin/webhooks/webhooks_endpoint").contains("No delivery attempts in the last 24 hours")
         .contains("No dead letters").contains("None yet");
   }

@@ -120,12 +120,10 @@ class WebhooksHomeTest
      assertThat(data.getDelivered24h()).isEqualTo(3);
      assertThat(data.getFailedAttempts24h()).isEqualTo(3);
      assertThat(data.getDeadLetters()).isEqualTo(1);
-      assertThat(data.getAttention()).hasSize(1);
-      assertThat(data.getAttentionWhy().get(0)).contains("Unhealthy");
      assertThat(data.getP50Millis()).isNotNull();

      String html = home();
-      assertThat(html).contains("Delivered (24 h)").contains("attempts in the last 24 hours, by hour").contains("bad &lt;b&gt;").doesNotContain("bad <b>")
+      assertThat(html).contains("Delivered (24 h)").contains("attempts in the last 24 hours, by hour").doesNotContain("Endpoints needing attention").doesNotContain("bad <b>")
         .contains("/admin/webhooks/webhooks_delivery/" + deadOne.getValue("id")).contains("Replay");
      ////////////////////////////////////////////////////////////////
      // readability rules: no fixed widths that force a scroll bar   //
@@ -0,0 +1,366 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.webhooks.health;
+
+
+import java.time.Instant;
+import java.util.List;
+import java.util.Map;
+import java.util.UUID;
+import com.kingsrook.qbits.webhooks.WebhooksRuntime;
+import com.kingsrook.qbits.webhooks.WebhooksTestSupport;
+import com.kingsrook.qbits.webhooks.delivery.EndpointHealth;
+import com.kingsrook.qbits.webhooks.metadata.WebhooksTables;
+import com.kingsrook.qbits.webhooks.support.SystemWrites;
+import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
+import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.health.HealthCategory;
+import com.kingsrook.qqq.backend.core.health.HealthCheckInput;
+import com.kingsrook.qqq.backend.core.health.HealthCheckKind;
+import com.kingsrook.qqq.backend.core.health.HealthCheckMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthCheckOutcome;
+import com.kingsrook.qqq.backend.core.health.HealthChecksMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthStatus;
+import com.kingsrook.qqq.backend.core.model.actions.tables.QueryOrGetInputInterface;
+import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
+import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import com.kingsrook.qqq.backend.core.model.session.QSession;
+import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+
+
+/*******************************************************************************
+ ** Passive health checks use memory records and never contact a receiver.
+ *******************************************************************************/
+class WebhooksHealthChecksTest
+{
+   private static final String SENTINEL = "S3NT1NEL-secret";
+   private QInstance instance;
+
+
+
+   /*******************************************************************************
+    ** Set up the producer and a sentinel credential reference.
+    *******************************************************************************/
+   @BeforeEach
+   void setUp() throws Exception
+   {
+      instance = WebhooksTestSupport.setUp();
+      System.setProperty("WEBHOOKS_HEALTH_SECRET", SENTINEL + "-long-enough");
+   }
+
+
+
+   /*******************************************************************************
+    ** Remove the test credential.
+    *******************************************************************************/
+   @AfterEach
+   void tearDown()
+   {
+      System.clearProperty("WEBHOOKS_HEALTH_SECRET");
+   }
+
+
+
+   /*******************************************************************************
+    ** Supplier refresh finds new endpoints and excludes manual disable/pause.
+    *******************************************************************************/
+   @Test
+   void registrationAndDynamicDiscovery() throws Exception
+   {
+      HealthChecksMetaData registry = HealthChecksMetaData.ofOrWithNew(instance);
+      assertThat(registry.getChecks()).extracting(HealthCheckMetaData::getName).containsExactly("webhooks.dispatcher");
+      assertThat(registry.getSuppliers()).hasSize(1);
+      assertThat(registry.getSuppliers().getFirst().getNamePrefix()).isEqualTo("webhooks.endpoint");
+      assertThat(registry.getSuppliers().getFirst().getComponent()).isEqualTo("webhooks");
+      assertThat(registry.resolveAll(instance)).hasSize(1);
+      Integer active = endpoint();
+      Integer disabled = endpoint();
+      update(disabled, "activeStatus", "DISABLED");
+      Integer paused = endpoint();
+      update(paused, "activeStatus", "PAUSED");
+      Integer automatic = endpoint();
+      update(automatic, "activeStatus", "DISABLED");
+      update(automatic, "autoDisabledDate", WebhooksTestSupport.CLOCK.instant());
+      QContext.init(instance, new QSession());
+      assertThat(registry.resolveAll(instance)).extracting(HealthCheckMetaData::getName)
+         .containsExactlyInAnyOrder("webhooks.dispatcher", "webhooks.endpoint." + active, "webhooks.endpoint." + automatic);
+      assertThat(registry.resolveAll(instance)).allSatisfy(check ->
+      {
+         assertThat(check.getKind()).isEqualTo(HealthCheckKind.DEPENDENCY);
+         assertThat(check.getCategory()).isEqualTo(HealthCategory.INTEGRATION);
+         assertThat(check.getComponent()).isEqualTo("webhooks");
+         assertThat(check.getIntervalSeconds()).isEqualTo(300);
+         assertThat(check.getTimeoutMs()).isLessThan(300000);
+         assertThat(check.getIncludeInReadiness()).isFalse();
+      });
+   }
+
+
+
+   /*******************************************************************************
+    ** Read fresh persisted states and keep the existing state machine intact.
+    *******************************************************************************/
+   @Test
+   void endpointStatesAndRecovery() throws Exception
+   {
+      Integer id = endpoint();
+      HealthCheckMetaData check = check("webhooks.endpoint." + id);
+      assertOutcome(check, HealthStatus.UP);
+      for(String state : List.of("UNHEALTHY", "PROBATION"))
+      {
+         update(id, "healthStatus", state);
+         update(id, "consecutiveFailures", 3);
+         update(id, "unhealthySinceDate", WebhooksTestSupport.CLOCK.instant());
+         HealthCheckOutcome outcome = assertOutcome(check, HealthStatus.DEGRADED);
+         assertThat(outcome.getDetails().getValues()).containsExactlyInAnyOrderEntriesOf(Map.of(
+            "healthStatus", state, "consecutiveFailures", 3, "unhealthySince", WebhooksTestSupport.CLOCK.instant()));
+      }
+      update(id, "probationFailures", 1);
+      QRecord row = SystemWrites.query(WebhooksTables.ENDPOINT, new QQueryFilter()).getFirst();
+      EndpointHealth.recordFailure(WebhooksRuntime.config(), row, WebhooksTestSupport.CLOCK.instant(), SENTINEL);
+      assertOutcome(check, HealthStatus.DOWN);
+      assertThat(check("webhooks.endpoint." + id)).isNotNull();
+      update(id, "activeStatus", "ACTIVE");
+      EndpointHealth.recordSuccess(row, WebhooksTestSupport.CLOCK.instant());
+      assertOutcome(check, HealthStatus.UP);
+      update(id, "healthStatus", SENTINEL);
+      assertOutcome(check, HealthStatus.UNKNOWN);
+      update(id, "healthStatus", null);
+      assertOutcome(check, HealthStatus.UNKNOWN);
+      update(id, "activeStatus", "DISABLED");
+      assertOutcome(check, HealthStatus.DOWN);
+      update(id, "autoDisabledDate", null);
+      assertOutcome(check, HealthStatus.UNKNOWN);
+      SystemWrites.delete(WebhooksTables.ENDPOINT, new QQueryFilter());
+      assertOutcome(check, HealthStatus.UNKNOWN);
+   }
+
+
+
+   /*******************************************************************************
+    ** Strict thresholds and terminal-status exclusion across every queue state.
+    *******************************************************************************/
+   @Test
+   void dispatcherThresholds() throws Exception
+   {
+      HealthCheckMetaData check = check("webhooks.dispatcher");
+      assertThat(assertOutcome(check, HealthStatus.UP).getMetrics().getFirst().getValue().longValue()).isZero();
+      for(String terminal : List.of("DELIVERED", "CANCELLED", "DEAD_LETTER"))
+      {
+         delivery(terminal, 10000);
+      }
+      assertOutcome(check, HealthStatus.UP);
+      for(String status : List.of("PENDING", "RETRY_WAIT", "SENDING"))
+      {
+         for(long age : new long[] {-10, 0, 900, 901, 3600, 3601})
+         {
+            QRecord record = delivery(status, age);
+            HealthCheckOutcome outcome = assertOutcome(check, age > 3600 ? HealthStatus.DOWN : age > 900 ? HealthStatus.DEGRADED : HealthStatus.UP);
+            assertThat(outcome.getMetrics().getFirst().getName()).isEqualTo("oldestPendingSeconds");
+            assertThat(outcome.getMetrics().getFirst().getValue().longValue()).isEqualTo(Math.max(0, age));
+            SystemWrites.updateOne(WebhooksTables.DELIVERY, new QRecord().withValue("id", record.getValue("id")).withValue("status", "DELIVERED"));
+         }
+      }
+      delivery("PENDING", 4000);
+      delivery("PENDING", 1);
+      assertThat(assertOutcome(check, HealthStatus.DOWN).getMetrics().getFirst().getValue().longValue()).isEqualTo(4000);
+   }
+
+
+
+   /*******************************************************************************
+    ** A missing queue timestamp cannot be reported as a healthy backlog.
+    *******************************************************************************/
+   @Test
+   void missingQueueTimestamp() throws Exception
+   {
+      QRecord record = delivery("PENDING", 0);
+      MemoryRecordStore.getInstance().update(new UpdateInput(WebhooksTables.DELIVERY).withRecords(List.of(
+         new QRecord().withValue("id", record.getValue("id")).withValue("createDate", null))), false);
+      assertOutcome(check("webhooks.dispatcher"), HealthStatus.UNKNOWN);
+   }
+
+
+
+   /*******************************************************************************
+    ** No reads start after the deadline; late responses are not accepted as UP.
+    *******************************************************************************/
+   @Test
+   void deadlines() throws Exception
+   {
+      endpoint();
+      delivery("PENDING", 0);
+      for(HealthCheckMetaData check : HealthChecksMetaData.ofOrWithNew(instance).resolveAll(instance))
+      {
+         assertThat(check.getCheckInstance().run(input().withDeadline(Instant.now().minusSeconds(1))).getStatus()).isEqualTo(HealthStatus.DOWN);
+      }
+      instance.getTable(WebhooksTables.DELIVERY).withCustomizer(TableCustomizers.POST_QUERY_RECORD, new QCodeReference(SlowQuery.class));
+      HealthCheckOutcome outcome = check("webhooks.dispatcher").getCheckInstance().run(input().withDeadline(Instant.now().plusMillis(50)));
+      assertThat(outcome.getStatus()).isEqualTo(HealthStatus.DOWN);
+      assertThat(outcome.getMessage()).contains("timed out");
+   }
+
+
+
+   /*******************************************************************************
+    ** Exception messages, URLs, failure text and credentials never leave checks.
+    *******************************************************************************/
+   @Test
+   void secretBearingFailures() throws Exception
+   {
+      Integer id = endpoint();
+      HealthCheckMetaData endpoint = check("webhooks.endpoint." + id);
+      delivery("PENDING", 0);
+      assertOutcome(endpoint, HealthStatus.UP);
+      assertOutcome(check("webhooks.dispatcher"), HealthStatus.UP);
+      for(String table : List.of(WebhooksTables.ENDPOINT, WebhooksTables.DELIVERY))
+      {
+         instance.getTable(table).withCustomizer(TableCustomizers.POST_QUERY_RECORD, new QCodeReference(FailingQuery.class));
+      }
+      assertOutcome(endpoint, HealthStatus.DOWN);
+      assertOutcome(HealthChecksMetaData.ofOrWithNew(instance).getChecks().getFirst(), HealthStatus.DOWN);
+   }
+
+
+
+   /*******************************************************************************
+    ** Persist a sentinel credential without ever resolving it for a health check.
+    *******************************************************************************/
+   private Integer endpoint() throws Exception
+   {
+      Integer id = WebhooksTestSupport.insert(WebhooksTables.ENDPOINT, new QRecord().withValue("name", SENTINEL + UUID.randomUUID())
+         .withValue("url", "https://example.invalid/" + SENTINEL)).getValueInteger("id");
+      WebhooksTestSupport.insert(WebhooksTables.SECRET, new QRecord().withValue("endpointId", id)
+         .withValue("source", "REFERENCE").withValue("secretRef", "WEBHOOKS_HEALTH_SECRET"));
+      update(id, "lastFailureMessage", SENTINEL);
+      return (id);
+   }
+
+
+
+   /*******************************************************************************
+    ** Change an existing domain field as the qbit.
+    *******************************************************************************/
+   private void update(Integer id, String field, java.io.Serializable value) throws Exception
+   {
+      SystemWrites.updateOne(WebhooksTables.ENDPOINT, new QRecord().withValue("id", id).withValue(field, value));
+   }
+
+
+
+   /*******************************************************************************
+    ** Insert a pending or terminal delivery with deterministic queue time.
+    *******************************************************************************/
+   private QRecord delivery(String status, long age) throws Exception
+   {
+      QRecord record = SystemWrites.insertOne(WebhooksTables.DELIVERY, new QRecord().withValue("status", status)
+         .withValue("payload", SENTINEL).withValue("lastError", SENTINEL));
+      MemoryRecordStore.getInstance().update(new UpdateInput(WebhooksTables.DELIVERY).withRecords(List.of(
+         new QRecord().withValue("id", record.getValue("id")).withValue("createDate", WebhooksTestSupport.CLOCK.instant().minusSeconds(age)))), false);
+      return (record);
+   }
+
+
+
+   /*******************************************************************************
+    ** Find a producer-registered check, resolving the supplier afresh.
+    *******************************************************************************/
+   private HealthCheckMetaData check(String name) throws Exception
+   {
+      return (HealthChecksMetaData.ofOrWithNew(instance).resolveAll(instance).stream().filter(c -> name.equals(c.getName())).findFirst().orElseThrow());
+   }
+
+
+
+   /*******************************************************************************
+    ** Bounded runner input.
+    *******************************************************************************/
+   private HealthCheckInput input()
+   {
+      return (new HealthCheckInput().withQInstance(instance).withDeadline(Instant.now().plusSeconds(5)));
+   }
+
+
+
+   /*******************************************************************************
+    ** Assert raw outcomes are clean; HealthRunner.sanitize is package-private.
+    *******************************************************************************/
+   private HealthCheckOutcome assertOutcome(HealthCheckMetaData check, HealthStatus status) throws Exception
+   {
+      HealthCheckOutcome outcome = check.getCheckInstance().run(input());
+      assertThat(outcome.getStatus()).isEqualTo(status);
+      assertThat(outcome.getMessage()).doesNotContain(SENTINEL).doesNotContain("example.invalid").doesNotContain("\n");
+      assertThat(outcome.getDetails().toJson()).doesNotContain(SENTINEL).doesNotContain("example.invalid");
+      return (outcome);
+   }
+
+
+
+   /*******************************************************************************
+    ** A backend failure containing sensitive text.
+    *******************************************************************************/
+   public static class FailingQuery implements TableCustomizerInterface
+   {
+      /***************************************************************************
+       ** Throw after the read, before the check sees any records.
+       ***************************************************************************/
+      @Override
+      public List<QRecord> postQuery(QueryOrGetInputInterface input, List<QRecord> records) throws QException
+      {
+         throw (new QException(SENTINEL));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** A database that responds after the check's deadline.
+    *******************************************************************************/
+   public static class SlowQuery implements TableCustomizerInterface
+   {
+      /***************************************************************************
+       ** Simulate a slow database, with no network.
+       ***************************************************************************/
+      @Override
+      public List<QRecord> postQuery(QueryOrGetInputInterface input, List<QRecord> records) throws QException
+      {
+         try
+         {
+            Thread.sleep(100);
+         }
+         catch(InterruptedException e)
+         {
+            Thread.currentThread().interrupt();
+            throw (new QException("Interrupted", e));
+         }
+         return (records);
+      }
+   }
+}
@@ -0,0 +1,82 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.webhooks.health;
+
+
+import java.net.URLDecoder;
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import com.kingsrook.qbits.webhooks.WebhooksTestSupport;
+import com.kingsrook.qbits.webhooks.app.WebhooksApp;
+import com.kingsrook.qbits.webhooks.support.SystemWrites;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.health.HealthMetaDataProvider;
+import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput;
+import com.kingsrook.qqq.backend.core.model.dashboard.widgets.StatisticsData;
+import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.session.QSession;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+
+
+/*******************************************************************************
+ ** The native home line displays only the persisted central component state.
+ *******************************************************************************/
+class WebhooksHealthWidgetTest
+{
+   /*******************************************************************************
+    ** No data means unknown, with a component-filtered link to System Health.
+    *******************************************************************************/
+   @Test
+   void emptyAndStoredComponent() throws Exception
+   {
+      var instance = WebhooksTestSupport.setUp();
+      assertThat(instance.getWidget(WebhooksApp.HEALTH_WIDGET).getType()).isEqualTo(WidgetType.STATISTICS.getType());
+      StatisticsData data = render();
+      assertThat(data.getCount()).isEqualTo("No checks have run yet");
+      assertThat(data.getCountURL()).startsWith("/app/health_check?filter=");
+      String filter = new String(Base64.getDecoder().decode(URLDecoder.decode(data.getCountURL().split("filter=")[1], StandardCharsets.UTF_8)), StandardCharsets.UTF_8);
+      assertThat(filter).contains("component", "webhooks");
+      new HealthMetaDataProvider().defineAll(instance, WebhooksTestSupport.MEMORY, null);
+      QContext.init(instance, new QSession().withPermissions(List.of("health.read")));
+      assertThat(render().getCount()).isEqualTo("No checks have run yet");
+      QRecord component = SystemWrites.insertOne(HealthMetaDataProvider.COMPONENT_TABLE, new QRecord().withValue("name", "webhooks").withValue("status", "DOWN"));
+      for(String status : List.of("UP", "DEGRADED", "DOWN", "UNKNOWN", "S3NT1NEL-secret"))
+      {
+         SystemWrites.updateOne(HealthMetaDataProvider.COMPONENT_TABLE, new QRecord().withValue("id", component.getValue("id")).withValue("status", status));
+         assertThat(render().getCount()).isEqualTo(status.equals("S3NT1NEL-secret") ? "UNKNOWN" : status);
+      }
+      SystemWrites.updateOne(HealthMetaDataProvider.COMPONENT_TABLE, new QRecord().withValue("id", component.getValue("id")).withValue("status", null));
+      assertThat(render().getCount()).isEqualTo("UNKNOWN");
+   }
+
+
+
+   /*******************************************************************************
+    ** Render the one-line native widget.
+    *******************************************************************************/
+   private StatisticsData render() throws Exception
+   {
+      return ((StatisticsData) new WebhooksHealthWidget().render(new RenderWidgetInput()).getWidgetData());
+   }
+}
Author	SHA1	Message	Date
james.maesandClaude Opus 5.5	d03c902f58	test(help): compare the help bundle version with the artifact version, not a hardcoded SNAPSHOT The bundle version is now Maven-filtered, so a literal SNAPSHOT in the test fails every release build. Surefire passes the artifact version in qbit.artifactVersion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 23:07:56 -05:00
james.maes	673e665aba	Merge pull request #20 from QRun-IO/fix/help-bundle-version fix(help): help bundle version comes from the build, not a hardcoded SNAPSHOT	2026-10-14 22:37:20 -05:00
james.maesandClaude Opus 5.5	9dfe565df0	fix(help): the help bundle carries the artifact version (Maven-filtered) instead of a hardcoded SNAPSHOT On release branches the bundle's version and the producer's differed, so the bundle test that asserts they match failed, and released bundles recorded SNAPSHOT as their provenance. help/*.yaml is now filtered: version: ${project.version}. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 22:04:41 -05:00
james.maes	b67abf4ff4	Merge pull request #16 from QRun-IO/feature/health-checks feat(health): register webhook endpoint and dispatcher checks with core central health	2026-10-14 21:27:00 -05:00
james.maesandClaude Opus 5.5	0a8ccb3442	Merge origin/develop into feature/health-checks; pin QQQ 4.1.0-RC.8 RC.8 ships the central health kernel (QRun-IO/qqq#1207). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 21:15:07 -05:00
james.maes	b31e65f40d	Merge pull request #19 from QRun-IO/feature/help-bundle feat(help): qbit-webhooks help bundle via QQQ help bundles, with coverage gate	2026-10-14 20:29:38 -05:00
james.maesandClaude Opus 5.5	e84ec8b9e5	feat(help): ship the qbit-webhooks help bundle through QQQ help bundles, with coverage gate Host-neutral help (deployment-specific facts removed), registered with HelpBundles, gated by HelpBundleAssertions. QQQ 4.1.0-RC.7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 20:17:15 -05:00
james.maes	35acb94756	Merge pull request #17 from QRun-IO/feature/foreign-keys Foreign keys between the webhooks tables, with an orphan-safe upgrade	2026-10-14 18:00:28 -05:00
james.maesandClaude Opus 5.5	d650deb46f	docs: webhooks health checks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 14:55:12 -05:00
james.maesandClaude Opus 5.5	d988e00be6	refactor(webhooks): home links to central health instead of its own endpoint health tile Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 14:55:11 -05:00
james.maesandClaude Opus 5.5	3cf073e2d5	feat(health): register webhook endpoint and dispatcher checks with core central health Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 14:55:11 -05:00