build: QQQ 4.1.0-RC.14 (ships ClusterCaches)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
feat(cluster): register backups caches with ClusterCaches
2026-10-15 14:05:06 -05:00 · 2026-10-15 12:11:11 -05:00 · 2026-10-15 01:42:56 -05:00 · 2026-10-15 01:27:59 -05:00 · 2026-10-14 22:37:20 -05:00 · 2026-10-14 22:05:12 -05:00
22 changed files with 1247 additions and 304 deletions
@@ -5,6 +5,7 @@ All notable changes to this project are documented here. The format follows Keep
 ## [0.1.0] - Unreleased

 ### Added
+- Central `backups.sla` and per-active-plan `backups.destination.<planId>` health checks, with bounded read-only destination diagnostics and secret-free status details.
 - Always-durable backup, resume, scheduled backup, restore and verify processes with conservative `NONE` idempotency and stable backup identities on explicit job retry.
 - Weighted backup phases, table/file counts, stored bytes, safe per-table/artifact logs, warnings, failures and resume notices through QQQ `JobProgress`.
 - Cooperative job cancellation during backup pages, media copying, restore pages and verification scans; local media-reference spooling keeps phase separation from growing heap usage across tables.
@@ -15,6 +16,9 @@ All notable changes to this project are documented here. The format follows Keep
 - Repository scaffold: `BackupsQBitConfig` and `BackupsQBitProducer` (table prefix `backups_`, permissions `backups.*`), build, CI and repository standard files.

 ### Changed
+- Multi-node deployments: the cached Prometheus text (`BackupsMetrics`) is registered with QQQ `ClusterCaches` as `qbit-backups.metrics` and cleared on every node when a run finishes or a plan is inserted, updated or deleted; the 60 s TTL remains the safety net. `SlaChecker`'s manifest-head and bucket-audit throttles and `BackupsRuntime`'s process-permission map are registered as node-local, with reasons. `BackupsClusterCachesTest` audits the package for unregistered static caches and covers a remote invalidation.
+- Builds against QQQ `4.1.0-SNAPSHOT` (`qqq.snapshot.version`) for `ClusterCaches` until the next QQQ RC ships; a release must pin that RC.
+- Read the QBit metadata version from the Maven artifact version so release builds report their released version.
 - Target QQQ `4.1.0-RC.8` (job progress).
 - Persist controlled backup failure messages instead of untrusted provider exception text.
 - Pin QQQ to released 4.1.0-RC.7 and replace the temporary help loader with core HelpBundles registration.
@@ -26,3 +30,6 @@ All notable changes to this project are documented here. The format follows Keep
 - New manifests identify `qqq-table-schema-v1` and carry core schema descriptors; restore and content verification still accept pre-RC.6 manifests with the original fingerprint algorithm.
 - Preserve primary-key ordering, exclusive resume cursors, bounded pages and caller-owned transactions; reject non-orderable primary keys and scope filters with skip/limit.
 - Require the core `backups.restore` permission during Review when using a core restore writer, including anonymous-mode hosts; preserve supplied timestamps through `RestoreWriteFlag`.
+
+### Removed
+- The legacy `BackupsHealthIndicator` and its optional `qqq-middleware-health` dependency; central health registration replaces manual indicator configuration.
@@ -43,6 +43,11 @@ Follow the QQQ [adoption checklist](https://github.com/QRun-IO/qqq/blob/develop/
 See [the RC.6 migration notes](docs/RC6-MIGRATION.md) for fingerprint compatibility,
 paging guarantees and restore permissions.

+Health checks use the core health API released in QQQ `4.1.0-RC.8`.
+The producer registers `backups.sla` and a read-only destination check for each
+active plan (`backups.destination.<planId>`). With dependencies cached locally,
+run `./mvnw -o verify`; no enforcer skip flags are required.
+
 ## In-app help

 The producer registers [help/qbit-backups.yaml](src/main/resources/help/qbit-backups.yaml)
@@ -95,3 +100,23 @@ The backup SLA checker still owns expected intervals and grace periods.
 RC.5 requires hyphenated alert names: `backups.run.missing-in-storage` and
 `backups.plan.owner-invalid` replace the former underscore spellings, including
 their deduplication keys. Update host routing rules that used those spellings.
+
+## Health checks
+
+The producer registers these central checks under component `backups`, category `STORAGE`:
+
+| Name | Kind | Interval | Timeout |
+| --- | --- | --- | --- |
+| `backups.sla` | DEPENDENCY | 900s (15 minutes) | 5000ms |
+| `backups.destination.<planId>` | DEPENDENCY | 86400s (24 hours) | 30000ms |
+
+The SLA check reads persisted `slaStatus`: critical active plans that are LATE,
+FAILING or MISSING_IN_STORAGE yield DOWN; non-critical failures yield DEGRADED.
+Only UNVERIFIED/UNKNOWN plans yield UNKNOWN; no active plans yield UP. Details
+contain counts per SLA status. The scheduled `backupCheckSla` still computes SLA state.
+
+Destination checks are supplied for active plans and reuse `DestinationProbe` in
+read-only list/head mode, including on unlocked buckets. They never write, delete,
+or resolve prune credentials. The Test destination action retains its write tests.
+Storage calls are bounded by the central runner, with deadline checks between calls.
+Override intervals in `health_check` and disable checks through `health_check.enabled`.
@@ -74,13 +74,6 @@
         <artifactId>qqq-backend-core</artifactId>
      </dependency>

-      <!-- optional: the BackupsHealthIndicator (apps without qqq-middleware-health never load it) -->
-      <dependency>
-         <groupId>com.kingsrook.qqq</groupId>
-         <artifactId>qqq-middleware-health</artifactId>
-         <optional>true</optional>
-      </dependency>
-
      <!-- optional: ESB events (apps without qqq-esb never load it) -->
      <dependency>
         <groupId>com.kingsrook.qqq</groupId>
@@ -131,11 +124,33 @@
   </dependencies>

   <build>
+      <resources>
+         <!-- help bundles carry the artifact version (QQQ help bundle provenance) -->
+         <resource>
+            <directory>src/main/resources</directory>
+            <filtering>true</filtering>
+            <includes>
+               <include>help/*.yaml</include>
+               <include>qbit-backups.properties</include>
+            </includes>
+         </resource>
+         <resource>
+            <directory>src/main/resources</directory>
+            <filtering>false</filtering>
+            <excludes>
+               <exclude>help/*.yaml</exclude>
+               <exclude>qbit-backups.properties</exclude>
+            </excludes>
+         </resource>
+      </resources>
      <plugins>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <configuration>
+               <systemPropertyVariables combine.children="append">
+                  <qbit.artifactVersion>${project.version}</qbit.artifactVersion>
+               </systemPropertyVariables>
               <!-- the MinIO test container default credentials, resolved through ${env.X} refs like production -->
               <environmentVariables>
                  <STORAGE_TEST_ACCESS_KEY>minioadmin</STORAGE_TEST_ACCESS_KEY>
@@ -201,7 +216,7 @@
            <activeByDefault>true</activeByDefault>
         </activation>
         <properties>
-            <qqq.snapshot.version>4.1.0-RC.8</qqq.snapshot.version>
+            <qqq.snapshot.version>4.1.0-RC.14</qqq.snapshot.version>
         </properties>
         <dependencyManagement>
            <dependencies>
@@ -17,6 +17,17 @@
 package com.kingsrook.qbits.backups;


+import java.io.IOException;
+import java.io.InputStream;
+import java.util.Properties;
+import com.kingsrook.qbits.backups.health.BackupsDestinationHealthChecks;
+import com.kingsrook.qbits.backups.health.BackupsSlaHealthCheck;
+import com.kingsrook.qqq.backend.core.health.HealthCategory;
+import com.kingsrook.qqq.backend.core.health.HealthCheckKind;
+import com.kingsrook.qqq.backend.core.health.HealthCheckMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthCheckSupplierMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthChecksMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
 import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundles;
 import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaDataProducer;
@@ -30,13 +41,41 @@ public class BackupsQBitProducer implements QBitMetaDataProducer<BackupsQBitConf
 {
   public static final String GROUP_ID    = "com.kingsrook.qbits";
   public static final String ARTIFACT_ID = "qbit-backups";
-   public static final String VERSION     = "0.1.0";
+   public static final String VERSION     = loadVersion();

   private BackupsQBitConfig qBitConfig;
   private String namespace;



+   /*******************************************************************************
+    ** Read the artifact version from the Maven-filtered resource.
+    *******************************************************************************/
+   private static String loadVersion()
+   {
+      try(InputStream inputStream = BackupsQBitProducer.class.getClassLoader().getResourceAsStream("qbit-backups.properties"))
+      {
+         if(inputStream == null)
+         {
+            throw (new IllegalStateException("Missing qbit-backups.properties; build with Maven to generate the artifact version"));
+         }
+         Properties properties = new Properties();
+         properties.load(inputStream);
+         String version = properties.getProperty("version");
+         if(version == null || version.isBlank() || version.contains("${"))
+         {
+            throw (new IllegalStateException("Missing or unfiltered version in qbit-backups.properties; build with Maven"));
+         }
+         return (version);
+      }
+      catch(IOException e)
+      {
+         throw (new IllegalStateException("Could not read artifact version from qbit-backups.properties", e));
+      }
+   }
+
+
+
   /*******************************************************************************
    ** Fluent setter for the config.
    *******************************************************************************/
@@ -164,6 +203,13 @@ public class BackupsQBitProducer implements QBitMetaDataProducer<BackupsQBitConf
      // Registration is metadata-only; core guards startup sync when helpContent is absent.
      HelpBundles.register(qInstance, "help/qbit-backups.yaml");

+      HealthChecksMetaData.ofOrWithNew(qInstance)
+         .withCheck(new HealthCheckMetaData().withName(BackupsSlaHealthCheck.NAME).withLabel("Backup SLA")
+            .withComponent("backups").withKind(HealthCheckKind.DEPENDENCY).withCategory(HealthCategory.STORAGE)
+            .withIntervalSeconds(900).withCheck(new QCodeReference(BackupsSlaHealthCheck.class)))
+         .withSupplier(new HealthCheckSupplierMetaData().withNamePrefix(BackupsDestinationHealthChecks.PREFIX)
+            .withComponent("backups").withSupplier(new QCodeReference(BackupsDestinationHealthChecks.class)));
+
      // A parent app only lists the apps that are its children, so attach ourselves (for example Admin, then Backups).
      com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData parent = qBitConfig.getParentAppName() == null ? null : qInstance.getApp(qBitConfig.getParentAppName());
      if(parent != null && (parent.getChildren() == null || parent.getChildren().stream().noneMatch(child -> app.getName().equals(child.getName()))))
@@ -25,6 +25,7 @@ import com.kingsrook.qbits.backups.alerts.BackupAlertSink;
 import com.kingsrook.qbits.backups.alerts.NoOpBackupAlertSink;
 import com.kingsrook.qbits.backups.storage.BackupStorage;
 import com.kingsrook.qbits.backups.storage.BackupStorageException;
+import com.kingsrook.qqq.backend.core.cluster.ClusterCaches;


 /*******************************************************************************
@@ -34,7 +35,8 @@ import com.kingsrook.qbits.backups.storage.BackupStorageException;
 *******************************************************************************/
 public final class BackupsRuntime
 {
-   private static final Map<String, String> PROCESS_PERMISSIONS = new ConcurrentHashMap<>();
+   private static final Map<String, String> PROCESS_PERMISSIONS = ClusterCaches.nodeLocal(new ConcurrentHashMap<>(),
+      "built from the qbit config at startup, the same on every node");

   private static volatile BackupsQBitConfig config;
   private static volatile Clock             clock           = Clock.systemUTC();
@@ -0,0 +1,95 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.backups.health;
+
+
+import com.kingsrook.qbits.backups.model.BackupsTables;
+import com.kingsrook.qbits.backups.plan.DestinationProbe;
+import com.kingsrook.qbits.backups.run.BackupCatalog;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.health.HealthCheck;
+import com.kingsrook.qqq.backend.core.health.HealthCheckInput;
+import com.kingsrook.qqq.backend.core.health.HealthCheckOutcome;
+import com.kingsrook.qqq.backend.core.health.HealthDetails;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
+
+
+/*******************************************************************************
+ ** Reuse the destination diagnostic in read-only mode; never publish its text.
+ *******************************************************************************/
+public class BackupsDestinationHealthCheck implements HealthCheck
+{
+   private final Integer planId;
+
+
+
+   /*******************************************************************************
+    ** Retain only the id; destination and active state are read on each run.
+    *******************************************************************************/
+   public BackupsDestinationHealthCheck(Integer planId)
+   {
+      this.planId = planId;
+   }
+
+
+
+   /*******************************************************************************
+    ** A removed or disabled plan can remain registered until supplier refresh.
+    *******************************************************************************/
+   @Override
+   public HealthCheckOutcome run(HealthCheckInput input)
+   {
+      boolean ownsContext = QContext.getQInstance() == null;
+      if(ownsContext)
+      {
+         QContext.init(input.getQInstance(), new QSystemUserSession(input.getQInstance()));
+      }
+      try
+      {
+         if(BackupsSlaHealthCheck.expired(input))
+         {
+            return (HealthCheckOutcome.down("Backup destination check timed out"));
+         }
+         QRecord plan = new BackupCatalog().get(BackupsTables.PLAN, planId).orElse(null);
+         if(plan == null || !Boolean.TRUE.equals(plan.getValueBoolean("isActive")))
+         {
+            return (HealthCheckOutcome.unknown("Backup plan is no longer active"));
+         }
+         DestinationProbe.Report report = DestinationProbe.probeReadOnly(plan, input.getDeadline());
+         if(BackupsSlaHealthCheck.expired(input))
+         {
+            return (HealthCheckOutcome.down("Backup destination check timed out"));
+         }
+         return ((report.ok() ? HealthCheckOutcome.up("Backup destination is readable") : HealthCheckOutcome.down("Backup destination probe failed"))
+            .withDetails(new HealthDetails().withValue("planId", planId).withValue("readOnly", true)
+               .withValue("checks", report.checks().size()).withValue("failedChecks", report.checks().stream().filter(check -> !check.ok()).count())));
+      }
+      catch(QException | RuntimeException e)
+      {
+         return (HealthCheckOutcome.down("Backup destination could not be checked"));
+      }
+      finally
+      {
+         if(ownsContext)
+         {
+            QContext.clear();
+         }
+      }
+   }
+}
@@ -0,0 +1,77 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.backups.health;
+
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+import com.kingsrook.qbits.backups.model.BackupsTables;
+import com.kingsrook.qbits.backups.run.BackupCatalog;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.health.HealthCategory;
+import com.kingsrook.qqq.backend.core.health.HealthCheckKind;
+import com.kingsrook.qqq.backend.core.health.HealthCheckMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthCheckSupplier;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
+
+
+/*******************************************************************************
+ ** Discover one daily, read-only destination check per active backup plan.
+ *******************************************************************************/
+public class BackupsDestinationHealthChecks implements HealthCheckSupplier
+{
+   public static final String PREFIX = "backups.destination";
+
+
+
+   /*******************************************************************************
+    ** Resolve from current catalog data, without opening any storage connections.
+    *******************************************************************************/
+   @Override
+   public List<HealthCheckMetaData> supply(QInstance qInstance) throws QException
+   {
+      boolean ownsContext = QContext.getQInstance() == null;
+      if(ownsContext)
+      {
+         QContext.init(qInstance, new QSystemUserSession(qInstance));
+      }
+      try
+      {
+         List<HealthCheckMetaData> checks = new ArrayList<>();
+         for(QRecord plan : new BackupCatalog().where(BackupsTables.PLAN, "isActive", true))
+         {
+            Integer planId = plan.getValueInteger("id");
+            checks.add(new HealthCheckMetaData().withName(PREFIX + "." + planId.toString().toLowerCase(Locale.ROOT))
+               .withLabel("Backup destination " + planId).withComponent("backups")
+               .withKind(HealthCheckKind.DEPENDENCY).withCategory(HealthCategory.STORAGE)
+               .withIntervalSeconds(86400).withTimeoutMs(30000).withCheckInstance(new BackupsDestinationHealthCheck(planId)));
+         }
+         return (checks);
+      }
+      finally
+      {
+         if(ownsContext)
+         {
+            QContext.clear();
+         }
+      }
+   }
+}
@@ -1,133 +0,0 @@
-/*
- * Copyright 2026 Kingsrook, LLC
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package com.kingsrook.qbits.backups.health;
-
-
-import java.util.ArrayList;
-import java.util.LinkedHashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import com.kingsrook.qbits.backups.model.BackupsTables;
-import com.kingsrook.qbits.backups.run.BackupCatalog;
-import com.kingsrook.qqq.backend.core.context.QContext;
-import com.kingsrook.qqq.backend.core.exceptions.QException;
-import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
-import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
-import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
-import com.kingsrook.qqq.backend.core.model.data.QRecord;
-import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
-import com.kingsrook.qqq.backend.core.model.session.QSession;
-import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
-import com.kingsrook.qqq.middleware.health.HealthIndicator;
-import com.kingsrook.qqq.middleware.health.model.metadata.HealthCheckResult;
-import com.kingsrook.qqq.middleware.health.model.metadata.HealthStatus;
-
-
-/*******************************************************************************
- ** The Backups health indicator (design spec section 10.4), for
- ** qqq-middleware-health (an optional dependency: an app that does not have that
- ** module never loads this class).  It reports DOWN if any active plan marked
- ** critical is LATE, FAILING or MISSING_IN_STORAGE, and UP with details otherwise.
- ** It reads the slaStatus the SLA checker maintains; it never touches storage, so
- ** a health call is cheap.
- **
- ** It is OFF the liveness path by design.  Add it to the indicators of the
- ** instance's health check (the readiness and monitoring view), and do not point a
- ** Kubernetes liveness probe at a check that includes it: a backup problem must
- ** never restart pods.
- *******************************************************************************/
-public class BackupsHealthIndicator implements HealthIndicator
-{
-   public static final String NAME = "backups";
-
-   private static final Set<String> CRITICAL_STATUSES = Set.of("LATE", "FAILING", "MISSING_IN_STORAGE");
-
-
-
-   /*******************************************************************************
-    ** {@inheritDoc}
-    *******************************************************************************/
-   @Override
-   public String getName()
-   {
-      return (NAME);
-   }
-
-
-
-   /*******************************************************************************
-    ** {@inheritDoc}
-    *******************************************************************************/
-   @Override
-   public HealthCheckResult check(QInstance qInstance) throws QException
-   {
-      long                started = System.currentTimeMillis();
-      boolean             ownsContext = QContext.getQInstance() == null;
-      if(ownsContext)
-      {
-         QContext.init(qInstance, systemSession(qInstance));
-      }
-      try
-      {
-         List<QRecord> plans = new BackupCatalog().query(BackupsTables.PLAN, new QQueryFilter(new QFilterCriteria("isActive", QCriteriaOperator.EQUALS, true)));
-
-         List<String> criticalProblems = new ArrayList<>();
-         Map<String, Integer> byStatus = new LinkedHashMap<>();
-         for(QRecord plan : plans)
-         {
-            String status = plan.getValueString("slaStatus") == null ? "UNKNOWN" : plan.getValueString("slaStatus");
-            byStatus.merge(status, 1, Integer::sum);
-            if(Boolean.TRUE.equals(plan.getValueBoolean("critical")) && CRITICAL_STATUSES.contains(status))
-            {
-               criticalProblems.add(plan.getValueString("planKey") + ": " + status);
-            }
-         }
-
-         Map<String, Object> details = new LinkedHashMap<>();
-         details.put("activePlans", plans.size());
-         details.put("plansByStatus", byStatus);
-         if(!criticalProblems.isEmpty())
-         {
-            details.put("criticalPlansWithProblems", criticalProblems);
-         }
-         return (new HealthCheckResult()
-            .withStatus(criticalProblems.isEmpty() ? HealthStatus.UP : HealthStatus.DOWN)
-            .withDurationMs(System.currentTimeMillis() - started)
-            .withDetails(details));
-      }
-      finally
-      {
-         if(ownsContext)
-         {
-            QContext.clear();
-         }
-      }
-   }
-
-
-
-   /*******************************************************************************
-    ** The session for a health request that has no QQQ context. It must be the
-    ** core system user: token-based authentication modules (OAuth2, Auth0)
-    ** reject any other tokenless session, which failed the check on such hosts.
-    *******************************************************************************/
-   static QSession systemSession(QInstance qInstance)
-   {
-      return (new QSystemUserSession(qInstance));
-   }
-}
@@ -0,0 +1,132 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.backups.health;
+
+
+import java.time.Instant;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import com.kingsrook.qbits.backups.model.BackupsTables;
+import com.kingsrook.qbits.backups.run.BackupCatalog;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.health.HealthCheck;
+import com.kingsrook.qqq.backend.core.health.HealthCheckInput;
+import com.kingsrook.qqq.backend.core.health.HealthCheckOutcome;
+import com.kingsrook.qqq.backend.core.health.HealthDetails;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
+
+
+/*******************************************************************************
+ ** Read persisted SLA state using the former indicator's critical-plan rule.
+ ** SLA calculation and storage verification remain the scheduled checker's job.
+ *******************************************************************************/
+public class BackupsSlaHealthCheck implements HealthCheck
+{
+   public static final String NAME = "backups.sla";
+
+   private static final Set<String> BAD_STATUSES = Set.of("LATE", "FAILING", "MISSING_IN_STORAGE");
+   private static final List<String> STATUSES = List.of("OK", "LATE", "FAILING", "MISSING_IN_STORAGE", "UNVERIFIED", "ABANDONED", "OWNER_INVALID", "UNKNOWN");
+
+
+
+   /*******************************************************************************
+    ** Counts use a fixed vocabulary so stored strings cannot become detail keys.
+    *******************************************************************************/
+   @Override
+   public HealthCheckOutcome run(HealthCheckInput input)
+   {
+      boolean ownsContext = QContext.getQInstance() == null;
+      if(ownsContext)
+      {
+         QContext.init(input.getQInstance(), new QSystemUserSession(input.getQInstance()));
+      }
+      try
+      {
+         if(expired(input))
+         {
+            return (HealthCheckOutcome.down("Backup SLA check timed out"));
+         }
+         List<QRecord> plans = new BackupCatalog().where(BackupsTables.PLAN, "isActive", true);
+         Map<String, Integer> counts = new LinkedHashMap<>();
+         STATUSES.forEach(status -> counts.put(status, 0));
+         boolean criticalBad = false;
+         boolean ordinaryBad = false;
+         for(QRecord plan : plans)
+         {
+            String status = plan.getValueString("slaStatus");
+            status = status != null && STATUSES.contains(status) ? status : "UNKNOWN";
+            counts.merge(status, 1, Integer::sum);
+            if(BAD_STATUSES.contains(status))
+            {
+               if(Boolean.TRUE.equals(plan.getValueBoolean("critical")))
+               {
+                  criticalBad = true;
+               }
+               else
+               {
+                  ordinaryBad = true;
+               }
+            }
+         }
+         if(expired(input))
+         {
+            return (HealthCheckOutcome.down("Backup SLA check timed out"));
+         }
+         HealthDetails details = new HealthDetails().withValue("activePlans", plans.size());
+         counts.forEach((status, count) -> details.withValue("slaStatus." + status, count));
+         HealthCheckOutcome outcome = HealthCheckOutcome.up("Backup SLA has no failures");
+         if(criticalBad)
+         {
+            outcome = HealthCheckOutcome.down("Critical backup plans have SLA failures");
+         }
+         else if(ordinaryBad)
+         {
+            outcome = HealthCheckOutcome.degraded("Backup plans have SLA failures");
+         }
+         else if(!plans.isEmpty() && counts.get("UNVERIFIED") + counts.get("UNKNOWN") == plans.size())
+         {
+            outcome = HealthCheckOutcome.unknown("Backup SLA is not verified yet");
+         }
+         return (outcome.withDetails(details));
+      }
+      catch(QException | RuntimeException e)
+      {
+         return (HealthCheckOutcome.down("Backup SLA state could not be read"));
+      }
+      finally
+      {
+         if(ownsContext)
+         {
+            QContext.clear();
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Stop between catalog operations when the runner's budget is exhausted.
+    *******************************************************************************/
+   static boolean expired(HealthCheckInput input)
+   {
+      return (Thread.currentThread().isInterrupted() || !Instant.now().isBefore(input.getDeadline()));
+   }
+}
@@ -20,10 +20,12 @@ package com.kingsrook.qbits.backups.metrics;
 import java.time.Duration;
 import java.time.Instant;
 import java.util.List;
+import java.util.concurrent.atomic.AtomicReference;
 import com.kingsrook.qbits.backups.BackupsQBitConfig;
 import com.kingsrook.qbits.backups.BackupsRuntime;
 import com.kingsrook.qbits.backups.model.BackupsTables;
 import com.kingsrook.qbits.backups.run.BackupCatalog;
+import com.kingsrook.qqq.backend.core.cluster.ClusterCaches;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
@@ -56,10 +58,21 @@ public final class BackupsMetrics
 {
   public static final String ENDPOINT_PATH = "/metrics/backups";

+   /** Cluster cache name of the rendered text: cleared on every node when a run finishes or a plan changes. */
+   public static final String CACHE_NAME = "qbit-backups.metrics";
+
   private static final Duration CACHE = Duration.ofSeconds(60);

-   private static String  cachedText;
-   private static Instant cachedAt;
+   private static final AtomicReference<Rendered> CACHED = ClusterCaches.shared(CACHE_NAME, new AtomicReference<>(), BackupsMetrics::resetCache, null);
+
+
+
+   /*******************************************************************************
+    ** The rendered text and when it was rendered.
+    *******************************************************************************/
+   private record Rendered(String text, Instant at)
+   {
+   }



@@ -84,12 +97,21 @@ public final class BackupsMetrics


   /*******************************************************************************
-    ** Forget the cached text (tests).
+    ** Forget the cached text on this node.
    *******************************************************************************/
-   public static synchronized void resetCache()
+   public static void resetCache()
   {
-      cachedText = null;
-      cachedAt = null;
+      CACHED.set(null);
+   }
+
+
+
+   /*******************************************************************************
+    ** Forget the cached text on every node (the backup data changed).
+    *******************************************************************************/
+   public static void invalidate()
+   {
+      ClusterCaches.invalidate(CACHE_NAME);
   }


@@ -99,13 +121,14 @@ public final class BackupsMetrics
    *******************************************************************************/
   public static synchronized String renderCached() throws QException
   {
-      Instant now = BackupsRuntime.getClock().instant();
-      if(cachedText == null || cachedAt == null || cachedAt.plus(CACHE).isBefore(now))
+      Instant  now    = BackupsRuntime.getClock().instant();
+      Rendered cached = CACHED.get();
+      if(cached == null || cached.at().plus(CACHE).isBefore(now))
      {
-         cachedText = render();
-         cachedAt = now;
+         cached = new Rendered(render(), now);
+         CACHED.set(cached);
      }
-      return (cachedText);
+      return (cached.text());
   }


@@ -413,15 +413,22 @@ public final class BackupsTables
      QCodeReference validation = new QCodeReference(BackupPlanTableCustomizer.class);
      table.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, validation);
      table.withCustomizer(TableCustomizers.PRE_UPDATE_RECORD, validation);
+
+      /////////////////////////////////////////////////////////////////////////////
+      // after a save or delete, the plan customizer clears the cached metrics on //
+      // every node                                                               //
+      /////////////////////////////////////////////////////////////////////////////
+      QCodeReference post = validation;
      if(StringUtils.hasContent(config.getSchedulerName()))
      {
-         QCodeReferenceWithProperties sync = MultiCustomizer.of(
+         post = MultiCustomizer.of(
            BaseSyncToScheduledJobTableCustomizer.makeCodeReference(table, new BackupPlanSyncToScheduledJobProcess()),
-            BaseSyncToScheduledJobTableCustomizer.makeCodeReference(table, new BackupPlanVerifySyncToScheduledJobProcess()));
-         table.withCustomizer(TableCustomizers.POST_INSERT_RECORD, sync);
-         table.withCustomizer(TableCustomizers.POST_UPDATE_RECORD, sync);
-         table.withCustomizer(TableCustomizers.POST_DELETE_RECORD, sync);
+            BaseSyncToScheduledJobTableCustomizer.makeCodeReference(table, new BackupPlanVerifySyncToScheduledJobProcess()),
+            validation);
      }
+      table.withCustomizer(TableCustomizers.POST_INSERT_RECORD, post);
+      table.withCustomizer(TableCustomizers.POST_UPDATE_RECORD, post);
+      table.withCustomizer(TableCustomizers.POST_DELETE_RECORD, post);
      return (table);
   }

@@ -22,6 +22,7 @@ import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
 import java.util.Optional;
+import com.kingsrook.qbits.backups.metrics.BackupsMetrics;
 import com.kingsrook.qbits.backups.model.BackupsTables;
 import com.kingsrook.qbits.backups.run.BackupCatalog;
 import com.kingsrook.qbits.backups.run.BackupRunFactory;
@@ -31,6 +32,9 @@ import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterfa
 import com.kingsrook.qqq.backend.core.context.QContext;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
+import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.session.QSession;
 import com.kingsrook.qqq.backend.core.model.statusmessages.BadInputStatusMessage;
@@ -43,7 +47,8 @@ import com.kingsrook.qqq.backend.core.model.statusmessages.QWarningMessage;
 ** configuration (the processes write the last run and SLA fields) is not
 ** validated again.
 **
- ** The saving user becomes the plan owner when none is given.
+ ** The saving user becomes the plan owner when none is given.  After an insert,
+ ** update or delete, the cached Prometheus text is cleared on every node.
 *******************************************************************************/
 public class BackupPlanTableCustomizer implements TableCustomizerInterface
 {
@@ -122,6 +127,42 @@ public class BackupPlanTableCustomizer implements TableCustomizerInterface



+   /*******************************************************************************
+    ** A saved plan changes the metrics on every node.
+    *******************************************************************************/
+   @Override
+   public List<QRecord> postInsert(InsertInput insertInput, List<QRecord> records) throws QException
+   {
+      BackupsMetrics.invalidate();
+      return (records);
+   }
+
+
+
+   /*******************************************************************************
+    ** A saved plan changes the metrics on every node.
+    *******************************************************************************/
+   @Override
+   public List<QRecord> postUpdate(UpdateInput updateInput, List<QRecord> records, Optional<List<QRecord>> oldRecordList) throws QException
+   {
+      BackupsMetrics.invalidate();
+      return (records);
+   }
+
+
+
+   /*******************************************************************************
+    ** A deleted plan changes the metrics on every node.
+    *******************************************************************************/
+   @Override
+   public List<QRecord> postDelete(DeleteInput deleteInput, List<QRecord> records) throws QException
+   {
+      BackupsMetrics.invalidate();
+      return (records);
+   }
+
+
+
   /*******************************************************************************
    ** The scope rules to resolve: the ones sent with the plan (associated rows), or
    ** the saved ones.
@@ -20,6 +20,7 @@ package com.kingsrook.qbits.backups.plan;
 import java.io.InputStream;
 import java.io.OutputStream;
 import java.nio.charset.StandardCharsets;
+import java.time.Instant;
 import java.util.ArrayList;
 import java.util.List;
 import java.util.UUID;
@@ -81,6 +82,28 @@ public final class DestinationProbe
    ** Probe the destination of a plan.
    *******************************************************************************/
   public static Report probe(QRecord plan)
+   {
+      return (probe(plan, false, null));
+   }
+
+
+
+   /*******************************************************************************
+    ** Health monitoring only lists and heads; it never resolves prune credentials
+    ** or probes capabilities. The runner bounds in-flight storage calls; check the
+    ** deadline between calls because BackupStorage has no per-call timeout API.
+    *******************************************************************************/
+   public static Report probeReadOnly(QRecord plan, Instant deadline)
+   {
+      return (probe(plan, true, deadline));
+   }
+
+
+
+   /*******************************************************************************
+    ** Share storage resolution and the existing locked-bucket read path.
+    *******************************************************************************/
+   private static Report probe(QRecord plan, boolean readOnly, Instant deadline)
   {
      List<Check>  checks   = new ArrayList<>();
      List<String> warnings = new ArrayList<>();
@@ -89,15 +112,23 @@ public final class DestinationProbe
      BackupStorage storage;
      try
      {
+         requireTime(deadline);
         storage = BackupsRuntime.storageFor(alias);
+         requireTime(deadline);
      }
      catch(RuntimeException e)
      {
         checks.add(new Check("Open the backup bucket " + alias, false, e.getMessage()));
-         return (new Report(false, checks, List.of(), warnings));
+         return (new Report(readOnly, checks, List.of(), warnings));
      }
      checks.add(new Check("Open the backup bucket " + alias, true, "found"));

+      if(readOnly)
+      {
+         headOnly(plan, storage, BackupPruner.planPrefix(plan), checks, deadline);
+         return (new Report(true, checks, List.of(), warnings));
+      }
+
      BackupStorage deleteSide = storage;
      String        pruneAlias = plan.getValueString("pruneBucketAlias");
      if(pruneAlias == null || pruneAlias.isBlank())
@@ -133,7 +164,7 @@ public final class DestinationProbe
      String prefix = BackupPruner.planPrefix(plan);
      if(locked)
      {
-         headOnly(plan, storage, prefix, checks);
+         headOnly(plan, storage, prefix, checks, null);
      }
      else
      {
@@ -147,10 +178,11 @@ public final class DestinationProbe
   /*******************************************************************************
    ** The probe for a locked bucket: list the prefix, and head the newest manifest.
    *******************************************************************************/
-   private static void headOnly(QRecord plan, BackupStorage storage, String prefix, List<Check> checks)
+   private static void headOnly(QRecord plan, BackupStorage storage, String prefix, List<Check> checks, Instant deadline)
   {
-      checks.add(attempt("Write and delete a probe object", () -> "skipped: object lock is on, a probe object could not be deleted again"));
-      checks.add(attempt("List the plan prefix", () ->
+      checks.add(new Check("Write and delete a probe object", true,
+         deadline == null ? "skipped: object lock is on, a probe object could not be deleted again" : "skipped: read-only probe"));
+      checks.add(attempt("List the plan prefix", deadline, () ->
      {
         List<StoredObjectInfo> objects = storage.list(prefix);
         return (objects.size() + " objects under " + prefix);
@@ -162,10 +194,11 @@ public final class DestinationProbe
         checks.add(new Check("Head the newest manifest", true, "there is no successful run yet"));
         return;
      }
-      checks.add(attempt("Head the newest manifest", () ->
+      checks.add(attempt("Head the newest manifest", deadline, () ->
      {
         QRecord run = new com.kingsrook.qbits.backups.run.BackupCatalog().get(com.kingsrook.qbits.backups.model.BackupsTables.RUN, lastSuccess).orElseThrow();
         String  key = run.getValueString("manifestKey");
+         requireTime(deadline);
         if(key == null || storage.head(key).isEmpty())
         {
            throw (new IllegalStateException("the manifest of the newest successful run is not in the bucket"));
@@ -238,6 +271,9 @@ public final class DestinationProbe
   @FunctionalInterface
   private interface Attempt
   {
+      /*******************************************************************************
+       ** Execute one diagnostic operation.
+       *******************************************************************************/
      String run() throws Exception;
   }

@@ -247,14 +283,40 @@ public final class DestinationProbe
    **
    *******************************************************************************/
   private static Check attempt(String name, Attempt attempt)
+   {
+      return (attempt(name, null, attempt));
+   }
+
+
+
+   /*******************************************************************************
+    ** Never begin another storage operation after the health budget expires.
+    *******************************************************************************/
+   private static Check attempt(String name, Instant deadline, Attempt attempt)
   {
      try
      {
-         return (new Check(name, true, attempt.run()));
+         requireTime(deadline);
+         String detail = attempt.run();
+         requireTime(deadline);
+         return (new Check(name, true, detail));
      }
      catch(Exception e)
      {
         return (new Check(name, false, e.getMessage() == null ? e.getClass().getSimpleName() : e.getMessage()));
      }
   }
+
+
+
+   /*******************************************************************************
+    ** Existing on-demand diagnostics have no deadline; health probes always do.
+    *******************************************************************************/
+   private static void requireTime(Instant deadline)
+   {
+      if(deadline != null && (Thread.currentThread().isInterrupted() || !Instant.now().isBefore(deadline)))
+      {
+         throw (new IllegalStateException("Destination probe timed out"));
+      }
+   }
 }
@@ -47,6 +47,7 @@ import com.kingsrook.qbits.backups.files.FileReference;
 import com.kingsrook.qbits.backups.files.FileReferenceProvider;
 import com.kingsrook.qbits.backups.manifest.BackupManifest;
 import com.kingsrook.qbits.backups.manifest.TableFingerprints;
+import com.kingsrook.qbits.backups.metrics.BackupsMetrics;
 import com.kingsrook.qbits.backups.model.BackupsTables;
 import com.kingsrook.qbits.backups.scope.ScopeResolver;
 import com.kingsrook.qbits.backups.storage.BackupStorage;
@@ -774,6 +775,7 @@ public class BackupEngine
      result.setStatus(status);
      result.setErrorMessage(message);
      signal(request, result, status, message);
+      BackupsMetrics.invalidate();
   }


@@ -35,6 +35,7 @@ import com.kingsrook.qbits.backups.model.BackupsTables;
 import com.kingsrook.qbits.backups.run.BackupCatalog;
 import com.kingsrook.qbits.backups.storage.BackupStorage;
 import com.kingsrook.qbits.backups.storage.StoredObjectInfo;
+import com.kingsrook.qqq.backend.core.cluster.ClusterCaches;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
@@ -95,8 +96,11 @@ public class SlaChecker
   private static final Set<String> FAILED_STATUSES   = Set.of("FAILED", "TIMED_OUT", "ABANDONED");
   private static final Set<String> FINISHED_STATUSES = Set.of("SUCCEEDED", "SUCCEEDED_WITH_WARNINGS", "FAILED", "TIMED_OUT", "ABANDONED");

-   private static final Map<String, Instant> MANIFEST_OK_AT = new ConcurrentHashMap<>();
-   private static final Map<String, Instant> LAST_AUDIT_AT  = new ConcurrentHashMap<>();
+   private static final Map<String, Instant> MANIFEST_OK_AT = ClusterCaches.nodeLocal(new ConcurrentHashMap<>(),
+      "throttles this node's storage HEAD of a manifest to once per MANIFEST_CACHE; it caches storage state, not editable data, "
+         + "and a pruned run is skipped by its prunedDate, so each node is as fresh as a single node");
+   private static final Map<String, Instant> LAST_AUDIT_AT  = ClusterCaches.nodeLocal(new ConcurrentHashMap<>(),
+      "throttles this node's bucket audit to once per BUCKET_AUDIT_INTERVAL; another node auditing again is only repeated work");

   private final BackupCatalog catalog = new BackupCatalog();

@@ -1,5 +1,5 @@
 bundle: qbit-backups
-version: 0.1.0-SNAPSHOT
+version: ${project.version}
 layer: QBIT
 entries:
 - key: table:backups_plan;field:name
@@ -0,0 +1 @@
+version=${project.version}
@@ -0,0 +1,164 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.backups;
+
+
+import java.util.EnumSet;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.function.Consumer;
+import com.kingsrook.qbits.backups.metrics.BackupsMetrics;
+import com.kingsrook.qbits.backups.model.BackupsTables;
+import com.kingsrook.qbits.backups.run.BackupCatalog;
+import com.kingsrook.qqq.backend.core.cluster.ClusterBus;
+import com.kingsrook.qqq.backend.core.cluster.ClusterBusInterface;
+import com.kingsrook.qqq.backend.core.cluster.ClusterBusMetaData;
+import com.kingsrook.qqq.backend.core.cluster.ClusterCacheAudit;
+import com.kingsrook.qqq.backend.core.cluster.ClusterEvent;
+import com.kingsrook.qqq.backend.core.cluster.QNodeRoles;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+
+
+/*******************************************************************************
+ ** The Backups caches are correct on several nodes: every static cache is
+ ** registered with ClusterCaches, and the cached metrics are cleared on every
+ ** node when a plan changes.
+ *******************************************************************************/
+class BackupsClusterCachesTest
+{
+   /** the "wire": events every node's transport sees */
+   static final List<Consumer<ClusterEvent>> WIRE = new CopyOnWriteArrayList<>();
+
+
+
+   /*******************************************************************************
+    ** A transport over the static WIRE, so a "remote" publish reaches us.
+    *******************************************************************************/
+   public static class WireBus implements ClusterBusInterface
+   {
+      /*******************************************************************************
+       **
+       *******************************************************************************/
+      @Override
+      public void publish(ClusterEvent event)
+      {
+         WIRE.forEach(listener -> listener.accept(event));
+      }
+
+
+
+      /*******************************************************************************
+       **
+       *******************************************************************************/
+      @Override
+      public AutoCloseable subscribe(Consumer<ClusterEvent> listener)
+      {
+         WIRE.add(listener);
+         return (() -> WIRE.remove(listener));
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** A backed-up test instance, connected to the wire as node-a.
+    *******************************************************************************/
+   @BeforeEach
+   void setUp() throws Exception
+   {
+      VerifyTestSupport.setUp(1);
+      QInstance instance = QContext.getQInstance();
+      instance.withSupplementalMetaData(new ClusterBusMetaData().withBus(new QCodeReference(WireBus.class)));
+      QNodeRoles.set(new QNodeRoles.Settings(EnumSet.allOf(QNodeRoles.Role.class), true, true, true, null, "node-a"));
+      ClusterBus.connect(instance);
+   }
+
+
+
+   /*******************************************************************************
+    **
+    *******************************************************************************/
+   @AfterEach
+   void afterEach()
+   {
+      ClusterBus.close(QContext.getQInstance());
+      WIRE.clear();
+      QNodeRoles.set(null);
+   }
+
+
+
+   /*******************************************************************************
+    ** Every static Memoization in the QBit is registered (shared or node-local).
+    *******************************************************************************/
+   @Test
+   void testNoUnregisteredStaticCaches() throws Exception
+   {
+      assertThat(ClusterCacheAudit.unregisteredStaticCaches("com.kingsrook.qbits.backups")).isEmpty();
+   }
+
+
+
+   /*******************************************************************************
+    ** Another node's invalidation clears this node's cached metrics.
+    *******************************************************************************/
+   @Test
+   void testRemoteInvalidationClearsMetricsCache() throws Exception
+   {
+      String cached = BackupsMetrics.renderCached();
+      assertThat(BackupsMetrics.renderCached()).isSameAs(cached);
+
+      ClusterEvent remote = new ClusterEvent(ClusterEvent.TYPE_MEMO_INVALIDATE, BackupsMetrics.CACHE_NAME, null);
+      remote.setOriginNodeId("node-b");
+      WIRE.forEach(listener -> listener.accept(remote));
+
+      assertThat(BackupsMetrics.renderCached()).isNotSameAs(cached);
+   }
+
+
+
+   /*******************************************************************************
+    ** A plan edit here clears the metrics here and tells the other nodes.
+    *******************************************************************************/
+   @Test
+   void testPlanEditPublishesMetricsInvalidation() throws Exception
+   {
+      QRecord plan = VerifyTestSupport.plan(Map.of());
+      String  cached = BackupsMetrics.renderCached();
+
+      List<ClusterEvent> sent = new CopyOnWriteArrayList<>();
+      WIRE.add(sent::add);
+
+      new BackupCatalog().update(BackupsTables.PLAN, plan.getValueInteger("id"), Map.of("slaHours", 48));
+
+      assertThat(BackupsMetrics.renderCached()).isNotSameAs(cached);
+      assertThat(sent).anySatisfy(event ->
+      {
+         assertThat(event.getType()).isEqualTo(ClusterEvent.TYPE_MEMO_INVALIDATE);
+         assertThat(event.getName()).isEqualTo(BackupsMetrics.CACHE_NAME);
+         assertThat(event.getOriginNodeId()).isEqualTo("node-a");
+      });
+   }
+}
@@ -18,6 +18,7 @@ package com.kingsrook.qbits.backups;


 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundles;
 import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitConfigValidationException;
 import org.junit.jupiter.api.Test;
 import static org.assertj.core.api.Assertions.assertThat;
@@ -40,12 +41,16 @@ class BackupsQBitProducerTest
      BackupsQBitConfig config    = new BackupsQBitConfig().withBackendName("rdbms");
      BackupsQBitProducer producer  = new BackupsQBitProducer().withQBitConfig(config).withNamespace("backups");

+      String artifactVersion = System.getProperty("qbit.artifactVersion");
+      assertThat(artifactVersion).isNotBlank();
+      assertThat(producer.getQBitMetaData().getVersion()).isEqualTo(artifactVersion).isEqualTo(BackupsQBitProducer.VERSION);
      assertThat(producer.isEnabled()).isTrue();
      assertThat(producer.getQBitConfig()).isSameAs(config);
      assertThat(producer.getNamespace()).isEqualTo("backups");
      assertThat(config.getDefaultBackendNameForTables()).isEqualTo("rdbms");

      producer.produce(qInstance);
+      assertThat(HelpBundles.of(qInstance).getBundles().get(0).getVersion()).isEqualTo(artifactVersion).isEqualTo(BackupsQBitProducer.VERSION);
      assertThat(qInstance.getQBits()).hasSize(1);
      assertThat(qInstance.getQBits().values().iterator().next().getArtifactId()).isEqualTo("qbit-backups");
   }
@@ -0,0 +1,495 @@
+/*
+ * Copyright 2026 Kingsrook, LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.backups.health;
+
+
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.time.Instant;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import com.kingsrook.qbits.backups.BackupsRuntime;
+import com.kingsrook.qbits.backups.BackupsTestUtils;
+import com.kingsrook.qbits.backups.VerifyTestSupport;
+import com.kingsrook.qbits.backups.model.BackupsTables;
+import com.kingsrook.qbits.backups.plan.DestinationProbe;
+import com.kingsrook.qbits.backups.run.BackupCatalog;
+import com.kingsrook.qbits.backups.storage.InMemoryBackupStorage;
+import com.kingsrook.qbits.backups.storage.StoredObjectInfo;
+import com.kingsrook.qbits.backups.storage.WriteOptions;
+import com.kingsrook.qqq.backend.core.actions.customizers.QCodeLoader;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.health.HealthCategory;
+import com.kingsrook.qqq.backend.core.health.HealthCheck;
+import com.kingsrook.qqq.backend.core.health.HealthCheckInput;
+import com.kingsrook.qqq.backend.core.health.HealthCheckKind;
+import com.kingsrook.qqq.backend.core.health.HealthCheckMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthCheckOutcome;
+import com.kingsrook.qqq.backend.core.health.HealthChecksMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthStatus;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.session.QSession;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+
+
+/*******************************************************************************
+ ** Central registration, migrated SLA rules, read-only probes and secret safety.
+ *******************************************************************************/
+class BackupsHealthChecksTest
+{
+   private static final String SECRET = "S3NT1NEL-secret";
+
+   private final BackupCatalog catalog = new BackupCatalog();
+   private ReadOnlyStorage storage;
+
+
+
+   /*******************************************************************************
+    ** All probes use memory storage; the resolver represents credential injection.
+    *******************************************************************************/
+   @BeforeEach
+   void setUp() throws Exception
+   {
+      VerifyTestSupport.setUp(3);
+      storage = new ReadOnlyStorage(SECRET);
+      BackupsRuntime.getConfig().withStorageResolver(alias ->
+      {
+         assertThat(alias).as("never resolve prune credentials").isNotEqualTo("prune");
+         return (storage);
+      });
+   }
+
+
+
+   /*******************************************************************************
+    ** Use the actual producer registry, including its reflective supplier loading.
+    *******************************************************************************/
+   @Test
+   void testRegistrationAndDynamicPlans() throws Exception
+   {
+      HealthChecksMetaData registry = HealthChecksMetaData.ofOrWithNew(QContext.getQInstance());
+      assertThat(registry.getChecks()).extracting(HealthCheckMetaData::getName).containsExactly("backups.sla");
+      assertThat(registry.getSuppliers()).singleElement().satisfies(supplier ->
+      {
+         assertThat(supplier.getNamePrefix()).isEqualTo("backups.destination");
+         assertThat(supplier.getComponent()).isEqualTo("backups");
+      });
+      QRecord active = VerifyTestSupport.plan(Map.of());
+      VerifyTestSupport.plan(Map.of("name", "Off", "planKey", "off", "isActive", false));
+      List<HealthCheckMetaData> checks = registry.resolveAll(QContext.getQInstance());
+      assertThat(checks).hasSize(2).allSatisfy(check ->
+      {
+         assertThat(check.getComponent()).isEqualTo("backups");
+         assertThat(check.getKind()).isEqualTo(HealthCheckKind.DEPENDENCY);
+         assertThat(check.getCategory()).isEqualTo(HealthCategory.STORAGE);
+         assertThat(check.getIncludeInReadiness()).isFalse();
+         assertThat(check.getName()).matches("[a-z0-9]+([._-][a-z0-9]+)*");
+      });
+      assertThat(checks.get(0).getIntervalSeconds()).isEqualTo(900);
+      assertThat(checks.get(0).getTimeoutMs()).isEqualTo(5000);
+      assertThat(QCodeLoader.getAdHoc(HealthCheck.class, checks.get(0).getCheck()).run(input()).getStatus()).isEqualTo(HealthStatus.UNKNOWN);
+      HealthCheckMetaData destination = checks.get(1);
+      assertThat(destination.getName()).isEqualTo("backups.destination." + active.getValueInteger("id"));
+      assertThat(destination.getIntervalSeconds()).isEqualTo(86400);
+      assertThat(destination.getTimeoutMs()).isEqualTo(30000);
+      assertThat(destination.getCheckInstance().run(input()).getStatus()).isEqualTo(HealthStatus.UP);
+      catalog.update(BackupsTables.PLAN, active.getValueInteger("id"), Map.of("isActive", false));
+      assertThat(destination.getCheckInstance().run(input()).getStatus()).isEqualTo(HealthStatus.UNKNOWN);
+      assertThat(registry.resolveAll(QContext.getQInstance())).hasSize(1);
+      VerifyTestSupport.plan(Map.of("name", "New", "planKey", "new"));
+      assertThat(registry.resolveAll(QContext.getQInstance())).hasSize(2);
+   }
+
+
+
+   /*******************************************************************************
+    ** Carry over the indicator's empty, critical, ordinary and inactive cases.
+    *******************************************************************************/
+   @Test
+   void testSlaUpDownAndDegraded() throws Exception
+   {
+      BackupsSlaHealthCheck check = new BackupsSlaHealthCheck();
+      HealthCheckOutcome empty = check.run(input());
+      assertThat(empty.getStatus()).isEqualTo(HealthStatus.UP);
+      assertThat(empty.getDetails().getValues()).containsEntry("activePlans", 0);
+      QRecord critical = VerifyTestSupport.plan(Map.of("critical", true));
+      QRecord ordinary = VerifyTestSupport.plan(Map.of("name", "Other", "planKey", "other"));
+      QRecord inactive = VerifyTestSupport.plan(Map.of("name", "Off", "planKey", "off", "isActive", false, "critical", true));
+      setStatus(inactive, "LATE");
+      setStatus(critical, "OK");
+      setStatus(ordinary, "OK");
+      assertThat(check.run(input()).getStatus()).isEqualTo(HealthStatus.UP);
+      for(String status : List.of("LATE", "FAILING", "MISSING_IN_STORAGE"))
+      {
+         setStatus(ordinary, status);
+         assertThat(check.run(input()).getStatus()).as(status).isEqualTo(HealthStatus.DEGRADED);
+         setStatus(critical, status);
+         HealthCheckOutcome down = check.run(input());
+         assertThat(down.getStatus()).as(status).isEqualTo(HealthStatus.DOWN);
+         assertThat(down.getDetails().getValues()).containsEntry("activePlans", 2).containsEntry("slaStatus." + status, 2);
+         assertClean(down);
+         setStatus(critical, "OK");
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Unknown or unverified plans do not become a false UP or a critical DOWN.
+    *******************************************************************************/
+   @Test
+   void testSlaUnknownAndMixedStates() throws Exception
+   {
+      QRecord plan = VerifyTestSupport.plan(Map.of("critical", true));
+      BackupsSlaHealthCheck check = new BackupsSlaHealthCheck();
+      assertThat(check.run(input()).getStatus()).isEqualTo(HealthStatus.UNKNOWN);
+      for(String status : List.of("UNVERIFIED", "UNKNOWN"))
+      {
+         setStatus(plan, status);
+         assertThat(check.run(input()).getStatus()).isEqualTo(HealthStatus.UNKNOWN);
+      }
+      QRecord other = VerifyTestSupport.plan(Map.of("name", "Other", "planKey", "other"));
+      setStatus(other, "UNVERIFIED");
+      assertThat(check.run(input()).getStatus()).isEqualTo(HealthStatus.UNKNOWN);
+      setStatus(other, "OK");
+      assertThat(check.run(input()).getStatus()).isEqualTo(HealthStatus.UP);
+   }
+
+
+
+   /*******************************************************************************
+    ** Preserve the old indicator's no-context behavior and existing sessions.
+    *******************************************************************************/
+   @Test
+   void testWithoutContext() throws Exception
+   {
+      QRecord plan = VerifyTestSupport.plan(Map.of());
+      QInstance instance = QContext.getQInstance();
+      QSession session = QContext.getQSession();
+      new BackupsSlaHealthCheck().run(input());
+      assertThat(QContext.getQSession()).isSameAs(session);
+      QContext.clear();
+      try
+      {
+         HealthCheckInput input = new HealthCheckInput().withQInstance(instance).withDeadline(Instant.now().plusSeconds(5));
+         assertThat(new BackupsSlaHealthCheck().run(input).getStatus()).isEqualTo(HealthStatus.UNKNOWN);
+         assertThat(QContext.getQInstance()).isNull();
+         assertThat(new BackupsDestinationHealthChecks().supply(instance)).hasSize(1);
+         assertThat(QContext.getQInstance()).isNull();
+         BackupsRuntime.getConfig().withStorageResolver(alias ->
+         {
+            assertThat(QContext.getQSession()).isInstanceOf(com.kingsrook.qqq.backend.core.model.session.QSystemUserSession.class);
+            return (storage);
+         });
+         assertThat(destination(plan).run(input).getStatus()).isEqualTo(HealthStatus.UP);
+         assertThat(QContext.getQInstance()).isNull();
+      }
+      finally
+      {
+         QContext.init(instance, session);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Locked and unlocked plans only list/head; neither reads object contents.
+    *******************************************************************************/
+   @Test
+   void testDestinationReadOnlyAndMissingManifest() throws Exception
+   {
+      for(String lockMode : List.of("NONE", "GOVERNANCE"))
+      {
+         BackupsRuntime.getConfig().withStorageResolver(alias -> BackupsTestUtils.BACKUPS);
+         QRecord plan = VerifyTestSupport.plan(Map.of("name", lockMode, "planKey", lockMode.toLowerCase(), "objectLockMode", lockMode, "objectLockDays", 3, "pruneBucketAlias", "prune"));
+         BackupsRuntime.getConfig().withStorageResolver(alias ->
+         {
+            assertThat(alias).as("never resolve prune credentials").isNotEqualTo("prune");
+            return (storage);
+         });
+         Integer runId = catalog.insert(BackupsTables.RUN, Map.of("manifestKey", "manifest.json"));
+         catalog.update(BackupsTables.PLAN, plan.getValueInteger("id"), Map.of("lastSuccessRunId", runId));
+         storage.putBytes("manifest.json", new byte[] { 1 });
+         assertThat(destination(plan).run(input()).getStatus()).isEqualTo(HealthStatus.UP);
+         storage.clear();
+         assertThat(destination(plan).run(input()).getStatus()).isEqualTo(HealthStatus.DOWN);
+      }
+      assertThat(storage.lists).isEqualTo(4);
+      assertThat(storage.heads).isEqualTo(4);
+   }
+
+
+
+   /*******************************************************************************
+    ** A deleted plan is UNKNOWN and does not touch storage.
+    *******************************************************************************/
+   @Test
+   void testDeletedPlan() throws Exception
+   {
+      QRecord plan = VerifyTestSupport.plan(Map.of());
+      catalog.deleteByIds(BackupsTables.PLAN, List.of(plan.getValueInteger("id")));
+      assertThat(destination(plan).run(input()).getStatus()).isEqualTo(HealthStatus.UNKNOWN);
+      assertThat(storage.lists).isZero();
+   }
+
+
+
+   /*******************************************************************************
+    ** No work starts with an expired deadline, including direct probe calls.
+    *******************************************************************************/
+   @Test
+   void testExpiredDeadline() throws Exception
+   {
+      QRecord plan = VerifyTestSupport.plan(Map.of());
+      HealthCheckInput expired = input().withDeadline(Instant.now().minusSeconds(1));
+      assertThat(new BackupsSlaHealthCheck().run(expired).getStatus()).isEqualTo(HealthStatus.DOWN);
+      assertThat(destination(plan).run(expired).getStatus()).isEqualTo(HealthStatus.DOWN);
+      assertThat(DestinationProbe.probeReadOnly(plan, expired.getDeadline()).ok()).isFalse();
+      assertThat(storage.lists).isZero();
+      assertThat(storage.heads).isZero();
+   }
+
+
+
+   /*******************************************************************************
+    ** Runner cancellation after list prevents the following manifest head.
+    *******************************************************************************/
+   @Test
+   void testCancellationBetweenStorageCalls() throws Exception
+   {
+      QRecord plan = VerifyTestSupport.plan(Map.of());
+      Integer runId = catalog.insert(BackupsTables.RUN, Map.of("manifestKey", "manifest.json"));
+      catalog.update(BackupsTables.PLAN, plan.getValueInteger("id"), Map.of("lastSuccessRunId", runId));
+      storage.interruptAfterList = true;
+      try
+      {
+         assertThat(destination(plan).run(input()).getStatus()).isEqualTo(HealthStatus.DOWN);
+         assertThat(storage.lists).isEqualTo(1);
+         assertThat(storage.heads).isZero();
+      }
+      finally
+      {
+         Thread.interrupted();
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Poison configuration, persisted state and provider errors. sanitize is
+    ** package-private in core; assert raw outcomes already contain no secrets.
+    *******************************************************************************/
+   @Test
+   void testNoSecretsInSuccessAndFailureOutcomes() throws Exception
+   {
+      QRecord plan = VerifyTestSupport.plan(Map.of("name", SECRET, "storageBucketAlias", SECRET, "critical", true));
+      Integer runId = catalog.insert(BackupsTables.RUN, Map.of("manifestKey", SECRET));
+      catalog.update(BackupsTables.PLAN, plan.getValueInteger("id"), Map.of("lastSuccessRunId", runId));
+      storage.putBytes(SECRET, new byte[] { 1 });
+      for(String status : List.of("OK", "LATE", "UNVERIFIED", SECRET))
+      {
+         setStatus(plan, status);
+         assertClean(new BackupsSlaHealthCheck().run(input()));
+         assertClean(destination(plan).run(input()));
+      }
+      storage.failList = true;
+      HealthCheckOutcome failure = destination(plan).run(input());
+      assertThat(failure.getStatus()).isEqualTo(HealthStatus.DOWN);
+      assertClean(failure);
+      storage.failList = false;
+      storage.failHead = true;
+      failure = destination(plan).run(input());
+      assertThat(failure.getStatus()).isEqualTo(HealthStatus.DOWN);
+      assertClean(failure);
+      BackupsRuntime.getConfig().withStorageResolver(alias ->
+      {
+         throw (new IllegalStateException(SECRET));
+      });
+      failure = destination(plan).run(input());
+      assertThat(failure.getStatus()).isEqualTo(HealthStatus.DOWN);
+      assertClean(failure);
+      assertClean(new BackupsSlaHealthCheck().run(input()));
+   }
+
+
+
+   /*******************************************************************************
+    ** Catalog failures are DOWN with fixed messages, even if metadata is poisoned.
+    *******************************************************************************/
+   @Test
+   void testCatalogFailureDoesNotExposeSecrets()
+   {
+      QContext.getQInstance().getTable(BackupsTables.PLAN).setBackendName(SECRET);
+      HealthCheckOutcome sla = new BackupsSlaHealthCheck().run(input());
+      HealthCheckOutcome destination = new BackupsDestinationHealthCheck(1).run(input());
+      assertThat(sla.getStatus()).isEqualTo(HealthStatus.DOWN);
+      assertThat(destination.getStatus()).isEqualTo(HealthStatus.DOWN);
+      assertClean(sla);
+      assertClean(destination);
+   }
+
+
+
+   /*******************************************************************************
+    ** Update persisted SLA state without running the scheduled calculator.
+    *******************************************************************************/
+   private void setStatus(QRecord plan, String status) throws Exception
+   {
+      catalog.update(BackupsTables.PLAN, plan.getValueInteger("id"), Map.of("slaStatus", status));
+   }
+
+
+
+   /*******************************************************************************
+    ** Fresh bounded input for each call.
+    *******************************************************************************/
+   private HealthCheckInput input()
+   {
+      return (new HealthCheckInput().withQInstance(QContext.getQInstance()).withDeadline(Instant.now().plusSeconds(5)));
+   }
+
+
+
+   /*******************************************************************************
+    ** Construct the same per-plan check as the supplier.
+    *******************************************************************************/
+   private BackupsDestinationHealthCheck destination(QRecord plan)
+   {
+      return (new BackupsDestinationHealthCheck(plan.getValueInteger("id")));
+   }
+
+
+
+   /*******************************************************************************
+    ** Validate the raw public message and flattened detail representation.
+    *******************************************************************************/
+   private void assertClean(HealthCheckOutcome outcome)
+   {
+      assertThat(outcome.getMessage()).doesNotContain(SECRET, "\n", "\r");
+      assertThat(outcome.getDetails().toJson()).doesNotContain(SECRET);
+   }
+
+
+
+   /*******************************************************************************
+    ** Existing memory fake with read failures and assertions against side effects.
+    *******************************************************************************/
+   private static class ReadOnlyStorage extends InMemoryBackupStorage
+   {
+      private final String credential;
+      private boolean failList;
+      private boolean failHead;
+      private boolean interruptAfterList;
+      private int lists;
+      private int heads;
+
+
+
+      /*******************************************************************************
+       ** Mimic the credential held by a host-provided storage adapter.
+       *******************************************************************************/
+      ReadOnlyStorage(String credential)
+      {
+         this.credential = credential;
+      }
+
+
+
+      /*******************************************************************************
+       ** Record list calls and inject credential-bearing failures or cancellation.
+       *******************************************************************************/
+      @Override
+      public List<StoredObjectInfo> list(String prefix)
+      {
+         lists++;
+         if(failList)
+         {
+            throw (new IllegalStateException(credential));
+         }
+         if(interruptAfterList)
+         {
+            Thread.currentThread().interrupt();
+         }
+         return (super.list(prefix));
+      }
+
+
+
+      /*******************************************************************************
+       ** Record head calls and inject credential-bearing failures.
+       *******************************************************************************/
+      @Override
+      public Optional<StoredObjectInfo> head(String key)
+      {
+         heads++;
+         if(failHead)
+         {
+            throw (new IllegalStateException(credential));
+         }
+         return (super.head(key));
+      }
+
+
+
+      /*******************************************************************************
+       ** Health probes must not write, even to an unlocked bucket.
+       *******************************************************************************/
+      @Override
+      public OutputStream openWrite(String key, WriteOptions options)
+      {
+         throw (new AssertionError("Health probe wrote an object"));
+      }
+
+
+
+      /*******************************************************************************
+       ** Health probes must not fetch backup contents.
+       *******************************************************************************/
+      @Override
+      public InputStream openRead(String key)
+      {
+         throw (new AssertionError("Health probe read object contents"));
+      }
+
+
+
+      /*******************************************************************************
+       ** Health probes must not delete.
+       *******************************************************************************/
+      @Override
+      public DeleteResult deletePermanent(String key)
+      {
+         throw (new AssertionError("Health probe deleted an object"));
+      }
+
+
+
+      /*******************************************************************************
+       ** Capability diagnostics can be active probes; health must not invoke them.
+       *******************************************************************************/
+      @Override
+      public Set<StorageCapability> capabilities()
+      {
+         throw (new AssertionError("Health probe requested capabilities"));
+      }
+   }
+}
@@ -1,132 +0,0 @@
-/*
- * Copyright 2026 Kingsrook, LLC
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package com.kingsrook.qbits.backups.health;
-
-
-import java.util.Map;
-import com.kingsrook.qbits.backups.VerifyTestSupport;
-import com.kingsrook.qbits.backups.model.BackupsTables;
-import com.kingsrook.qbits.backups.run.BackupCatalog;
-import com.kingsrook.qqq.backend.core.context.QContext;
-import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
-import com.kingsrook.qqq.backend.core.model.data.QRecord;
-import com.kingsrook.qqq.middleware.health.model.metadata.HealthCheckResult;
-import com.kingsrook.qqq.middleware.health.model.metadata.HealthStatus;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-import static org.assertj.core.api.Assertions.assertThat;
-
-
-/*******************************************************************************
- ** The health indicator: DOWN only when a critical, active plan is LATE, FAILING or
- ** MISSING_IN_STORAGE.
- *******************************************************************************/
-class BackupsHealthIndicatorTest
-{
-   /*******************************************************************************
-    **
-    *******************************************************************************/
-   @BeforeEach
-   void setUp() throws Exception
-   {
-      VerifyTestSupport.setUp(3);
-   }
-
-
-
-   /*******************************************************************************
-    **
-    *******************************************************************************/
-   private void setStatus(QRecord plan, String status) throws Exception
-   {
-      new BackupCatalog().update(BackupsTables.PLAN, plan.getValueInteger("id"), Map.of("slaStatus", status));
-   }
-
-
-
-   /*******************************************************************************
-    ** UP with details when nothing critical is wrong, including a non-critical plan
-    ** that is LATE; DOWN when a critical plan is LATE, FAILING or MISSING_IN_STORAGE.
-    *******************************************************************************/
-   @Test
-   void testUpAndDown() throws Exception
-   {
-      BackupsHealthIndicator indicator = new BackupsHealthIndicator();
-      assertThat(indicator.getName()).isEqualTo("backups");
-
-      HealthCheckResult empty = indicator.check(QContext.getQInstance());
-      assertThat(empty.getStatus()).isEqualTo(HealthStatus.UP);
-      assertThat(empty.getDetails()).containsEntry("activePlans", 0);
-
-      QRecord critical = VerifyTestSupport.plan(Map.of("critical", true));
-      QRecord ordinary = VerifyTestSupport.plan(Map.of("name", "Other", "planKey", "other"));
-      VerifyTestSupport.plan(Map.of("name", "Off", "planKey", "off", "isActive", false, "critical", true));
-
-      setStatus(critical, "OK");
-      setStatus(ordinary, "LATE");
-      HealthCheckResult up = indicator.check(QContext.getQInstance());
-      assertThat(up.getStatus()).isEqualTo(HealthStatus.UP);
-      assertThat(up.getDetails()).containsEntry("activePlans", 2).doesNotContainKey("criticalPlansWithProblems");
-      assertThat(up.getDurationMs()).isNotNull();
-
-      for(String status : new String[] { "LATE", "FAILING", "MISSING_IN_STORAGE" })
-      {
-         setStatus(critical, status);
-         HealthCheckResult down = indicator.check(QContext.getQInstance());
-         assertThat(down.getStatus()).as(status).isEqualTo(HealthStatus.DOWN);
-         assertThat(down.getDetails().get("criticalPlansWithProblems").toString()).contains("nightly: " + status);
-      }
-
-      setStatus(critical, "UNVERIFIED");
-      assertThat(indicator.check(QContext.getQInstance()).getStatus()).as("not verified is not down").isEqualTo(HealthStatus.UP);
-   }
-
-
-
-   /*******************************************************************************
-    ** On a thread with no QQQ context (a health request) the indicator makes its own
-    ** and cleans up.
-    *******************************************************************************/
-   @Test
-   void testWorksWithoutAContext() throws Exception
-   {
-      QInstance qInstance = QContext.getQInstance();
-      QContext.clear();
-      try
-      {
-         assertThat(new BackupsHealthIndicator().check(qInstance).getStatus()).isEqualTo(HealthStatus.UP);
-         assertThat(QContext.getQInstance()).isNull();
-      }
-      finally
-      {
-         QContext.init(qInstance, new com.kingsrook.qqq.backend.core.model.session.QSession());
-      }
-   }
-
-
-
-   /*******************************************************************************
-    ** The indicator's own context runs as the core system user, which token-based
-    ** authentication modules accept without a token.
-    *******************************************************************************/
-   @Test
-   void testOwnContextUsesSystemUserSession()
-   {
-      assertThat(BackupsHealthIndicator.systemSession(QContext.getQInstance()))
-         .isInstanceOf(com.kingsrook.qqq.backend.core.model.session.QSystemUserSession.class);
-   }
-}
@@ -87,10 +87,15 @@ class BackupsMetricsTest
      String empty = BackupsMetrics.renderCached();
      assertThat(empty).doesNotContain("{instance_name");

-      VerifyTestSupport.backup(VerifyTestSupport.plan(Map.of()));
      VerifyTestSupport.clock.advance(Duration.ofSeconds(30));
-      assertThat(BackupsMetrics.renderCached()).as("cached").isEqualTo(empty);
+      assertThat(BackupsMetrics.renderCached()).as("cached").isSameAs(empty);
      VerifyTestSupport.clock.advance(Duration.ofSeconds(31));
+      assertThat(BackupsMetrics.renderCached()).as("expired").isNotSameAs(empty);
+
+      /////////////////////////////////////////////////////////////
+      // a saved plan and a finished run clear the cache at once //
+      /////////////////////////////////////////////////////////////
+      VerifyTestSupport.backup(VerifyTestSupport.plan(Map.of()));
      assertThat(BackupsMetrics.renderCached()).contains("plan=\"nightly\"");

      assertThat(BackupsMetrics.escape("a\"b\\c\nd")).isEqualTo("a\\\"b\\\\c\\nd");
Author	SHA1	Message	Date
james.maesandClaude Opus 5.5	2f1964ddd1	build: QQQ 4.1.0-RC.14 (ships ClusterCaches) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 14:05:06 -05:00
james.maesandClaude Opus 5.5	e57ab62193	feat(cluster): register backups caches with ClusterCaches The cached Prometheus text is shared as qbit-backups.metrics and cleared on every node when a run finishes or a plan is saved or deleted (the plan customizer now also runs post insert/update/delete); the 60 s TTL stays as the safety net. SlaChecker's manifest-head and bucket-audit throttles and the process-permission map are node-local. Builds against QQQ 4.1.0-SNAPSHOT (qqq.snapshot.version) until the QQQ RC with ClusterCaches is published; a release must pin that RC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 12:11:11 -05:00
james.maes	e263f3bc8b	Merge pull request #29 from QRun-IO/fix/producer-version-from-build fix: qbit metadata version comes from the build	2026-10-15 01:42:56 -05:00
james.maesandClaude Opus 5.5	ab969bf99b	fix: the qbit metadata version comes from the build, not a hardcoded constant The producer read a literal version, so released artifacts reported a stale or -SNAPSHOT version. qbit-backups.properties (version=${project.version}) is Maven-filtered and the producer loads it; tests assert producer == bundle == artifact version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-15 01:27:59 -05:00
james.maes	b17664db8c	Merge pull request #28 from QRun-IO/fix/help-bundle-version fix(help): help bundle version comes from the build, not a hardcoded SNAPSHOT	2026-10-14 22:37:20 -05:00
james.maes	eb413b6e15	Merge pull request #23 from QRun-IO/feature/health-checks feat(health): register backups SLA and destination checks with core central health	2026-10-14 22:05:12 -05:00
james.maes	1e17dada29	Merge remote-tracking branch 'origin/develop' into feature/health-checks # Conflicts: # CHANGELOG.md	2026-10-14 22:02:35 -05:00
james.maesandClaude Opus 5.5	991f310ccc	fix(help): the help bundle carries the artifact version (Maven-filtered) instead of a hardcoded SNAPSHOT On release branches the bundle's version and the producer's differed, so the bundle test that asserts they match failed, and released bundles recorded SNAPSHOT as their provenance. help/*.yaml is now filtered: version: ${project.version}. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 22:01:33 -05:00
james.maes	0e798e3079	Merge pull request #26 from QRun-IO/feature/job-progress feat(progress): backup phases, counts, bytes and log lines (durable job progress)	2026-10-14 21:15:10 -05:00
james.maesandClaude Opus 5.5	9a2a1dd88e	Merge origin/develop into feature/health-checks; pin QQQ 4.1.0-RC.8 RC.8 ships the central health kernel (QRun-IO/qqq#1207). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 21:14:59 -05:00
james.maesandClaude Opus 5.5	f9767ede83	chore: drop optional qqq-middleware-health dependency; docs for backups health checks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 14:55:03 -05:00
james.maesandClaude Opus 5.5	8d56613f7d	feat(health): register backups SLA and destination checks; drop BackupsHealthIndicator Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 14:55:03 -05:00
james.maesandClaude Opus 5.5	0f9178dc14	feat(health): read-only destination probe mode Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 14:55:02 -05:00