fix(help): the help bundle carries the artifact version (Maven-filtered) instead of a hardcoded SNAPSHOT

On release branches the bundle's version and the producer's differed, so the bundle test that asserts they match failed, and released bundles recorded SNAPSHOT as their provenance. help/*.yaml is now filtered: version: ${project.version}. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merge pull request #17 from QRun-IO/feature/health-checks
2026-10-14 22:07:58 -05:00 · 2026-10-14 21:27:01 -05:00 · 2026-10-14 21:18:29 -05:00 · 2026-10-14 20:45:30 -05:00 · 2026-10-14 20:37:04 -05:00 · 2026-10-14 17:51:25 -05:00
20 changed files with 2324 additions and 119 deletions
@@ -5,7 +5,10 @@ All notable changes to this project are documented here. The format follows Keep
 ## [0.1.0] - Unreleased

 ### Added
- Every table explains itself: a help bundle (`help/qbit-object-storage.yaml`, QQQ help-bundle format) says what each `object_storage_*` table is for and how its rows get created; it shows on the table screen and when the table is empty. Applied by `ObjectStorageHelp` (STAND-IN for QRun-IO/qqq#1198); a host's own help is kept.
+- Central health supplier `object_storage.connection.<id>` (DEPENDENCY/STORAGE, 300s interval, 20000ms timeout), reusing bounded read-only capability probes with safe problem kinds and clock-skew details. Turn a connection's check off in the central `health_check` table.
+- In-memory health tests for registration, refresh, UP/DOWN/DEGRADED/UNKNOWN outcomes, deadlines, read-only execution and sentinel-secret exclusion.
+- Expanded the host-neutral in-app help bundle to cover storage fields, actions, widgets and possible values, with live references and a `HelpBundleAssertions` coverage gate. Registered it for QQQ startup sync and HOST overrides; retained inline table/app help for hosts without help-content tables.
+- Every table explains itself: a help bundle (`help/qbit-object-storage.yaml`, QQQ help-bundle format) says what each `object_storage_*` table is for and how its rows get created; it shows on the table screen and when the table is empty. Applied inline by `ObjectStorageHelp` for hosts without help-content tables; a host's own help is kept.
 - A Setup card on the Storage home (connections, buckets, header rules, protection rules) with the state and next step of each, resolved the way the runtime resolves them.
 - A starter changelog `db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql`: a disabled `default-mutable` header rule. Insert-only, run through real Liquibase in `DefaultsChangelogTest`.
 - `ObjectStorageTablesGuardrailTest` (each table is used at runtime, seeded or explained, and has help).
@@ -20,12 +23,19 @@ All notable changes to this project are documented here. The format follows Keep
 - **The index (P2).** `object_storage_object` and `object_storage_folder` kept in step with each indexed bucket: resumable full scan with a mass-removal safety stop, HEAD backlog, write-through on every service write, soft-deleted rows, folder rollups. Orphaned and missing objects from `FieldReferenceProvider`, daily usage snapshots, quota and freshness alerts, the nightly job.
 - **The Storage app (P3, P4).** Admin -> Storage with a responsive home; record screens with sections, child lists and a thumbnail preview; actions Rescan bucket, Check configuration, Refresh from bucket, Apply header policy (replaces app-specific backfills), Change headers and metadata, Edit tags, Copy, Move or rename, Delete, Restore, Create download link, Create upload link, folder rename and delete, Upload files, Test connection; every one a standard Review and Result with a bulk cap.
 - `ObjectStorageSchema`: PostgreSQL DDL generated from the table metadata.
+- Foreign keys on every relation between the qbit tables (`fk_<table>_<column>`), generated by `ObjectStorageSchema`: owned observed state (capabilities, folders, objects, versions, usage) cascades from its bucket or object, `object.folder_id` and `object_reference.object_id` are set null, and a bucket's connection, header and protection rules and operation log restrict. Each key is added by an idempotent block that first records rows whose parent is gone in `qqq_fk_quarantine` and clears (nullable column) or deletes (required column) them, so it also upgrades an existing schema. New indexes `ix_object_storage_bucket_1` (connection_id), `ix_object_storage_header_rule_1` and `ix_object_storage_protection_rule_1` (bucket_id) back the keys. Upgrade: run the generated statements again (or the host's copy of them).
 - Conformance suite against MinIO and Garage (Testcontainers), service, index and action tests, and the index and actions on PostgreSQL.

+### Removed
+- None for the health migration; both Test connection record actions, including on-demand write tests, are retained.
+
 ### Changed
+- Rechecked the released RC.8 JARs: core still has no S3 session-token/provider hook or file-reference interpreter. Retained both compatibility shims and moved their review deadline to RC.9 under issue #10.
+- Raised the QQQ BOM baseline to released `4.1.0-RC.8` for the central health API.
+- Raised the QQQ BOM baseline to released `4.1.0-RC.7` for the core help-bundle registry and coverage assertions.
 - Raised the QQQ BOM baseline to released `4.1.0-RC.5` and delegated S3 client construction to core `S3ClientFactory.buildClient`. Removed local HTTP client and trust-manager implementations; existing endpoint, region, addressing, checksum, TLS/CA, credentials and timeout settings remain supported.
 - Clients remain owned by each connection runtime and are rebuilt on reset/reinstall. Retry behavior now follows core/SDK defaults instead of forcing `ADAPTIVE_V2`; hosts can select the SDK retry mode through `AWS_RETRY_MODE` or `aws.retryMode`.
- Retained local presigning (including a separate endpoint/key). A request-credentials bridge preserves static session tokens because RC.5 core metadata/factory has no session-token field or credentials-provider hook. This bridge and the existing `${file.PATH}` resolver are marked for removal by RC.7 under issue #10.
+- Retained local presigning (including a separate endpoint/key). A request-credentials bridge preserves static session tokens because RC.5 core metadata/factory has no session-token field or credentials-provider hook. This bridge and the existing `${file.PATH}` resolver remain necessary in released RC.7 (no session-token/provider hook or file-reference interpreter); their review deadline is now RC.8 under issue #10.
 - Added the filesystem module with its unused SFTP/SSH stack excluded. Resolved dependencies otherwise only change QQQ RC.4 to RC.5 and jsoup 1.23.1 to 1.23.2; see [dependency details](docs/S3-FACTORY-MIGRATION.md).

 ### Notes
@@ -15,7 +15,7 @@ S3-compatible storage as a capability for any QQQ app, and an admin app to run i
 ## Requirements

 - Java 21.
- QQQ `4.1.0-RC.5` (released BOM), including `qqq-backend-module-filesystem`. `ConnectionRuntime` maps connection settings to core's `S3BackendMetaData` and uses `S3ClientFactory.buildClient`; core owns HTTP, TLS and checksum setup. The unused SFTP/SSH dependency is excluded. See the [dependency diff](docs/S3-FACTORY-MIGRATION.md).
+- QQQ `4.1.0-RC.8` (released BOM), including `qqq-backend-module-filesystem`. `ConnectionRuntime` maps connection settings to core's `S3BackendMetaData` and uses `S3ClientFactory.buildClient`; core owns HTTP, TLS and checksum setup. The unused SFTP/SSH dependency is excluded. See the [dependency diff](docs/S3-FACTORY-MIGRATION.md).
 - A QQQ backend for the qbit's tables (PostgreSQL is the intended store). `ObjectStorageSchema.postgresScript(config)` generates the DDL from the same metadata the app runs on, for your Liquibase or Flyway migration.

 ## Install
@@ -47,6 +47,20 @@ Connections and buckets come from this code only. Secrets are **references** (`$

 Client settings take effect when the qbit configuration is reinstalled or the runtime is reset. Presigning keeps its separate public endpoint and credentials. Retry behavior now follows core/SDK defaults; hosts requiring adaptive retries can set `AWS_RETRY_MODE=adaptive` or `aws.retryMode=adaptive`. Static session tokens remain supported through a small local bridge pending core support.

+## In-app help
+
+The producer registers [help/qbit-object-storage.yaml](src/main/resources/help/qbit-object-storage.yaml)
+as a `QBIT` bundle. QQQ syncs it when the host has the help-content tables installed;
+registration is safe without them, and inline table/app help still explains empty lists.
+The bundle version is the literal POM version; update both when releasing.
+
+Hosts can register their own `layer: HOST` bundle with `HelpBundles.register(qInstance, "help/my-host.yaml")`.
+Use the same entry key and role to override qbit text, keeping live `{{...}}` references where useful.
+Hosts that rename the optional Storage app should provide an app entry for their configured name.
+`ObjectStorageHelpBundleTest` checks all owned tables, fields, processes, apps, widgets and
+possible-value sources using `HelpBundleAssertions`, including valid keys, live references and
+plain summaries of at most 120 characters. It runs with `./mvnw verify`.
+
 ## Use

 ```java
@@ -78,6 +92,21 @@ Every write is a standard streamed ETL: **Review** (the dry run, with summary li

 Permissions: `object_storage.view`, `.write`, `.delete`, `.admin` (admin implies the rest). Per-bucket scoping uses record security locks on the `object_storage_bucket` and `object_storage_connection` security key types (switch off with `withBucketScoping(false)` for a single-administrator app).

+## Health checks
+
+The producer registers `object_storage.connection.<id>` for each configured
+`object_storage_connection` row: **DEPENDENCY**, **STORAGE**, every **300s**,
+with a **20000ms** timeout and component `object_storage`. The central health runner
+refreshes the supplier as rows change. The existing `CapabilityProbe` runs only
+read-only calls within the deadline; it never runs write or presign tests or stores
+free-form capability diagnostics. Connection failures are DOWN, clock skew above
+900 seconds is DEGRADED, and connections without buckets are UNKNOWN. Details
+contain only problem kind, clock skew seconds and bucket count.
+
+Turn a connection's check off or change its interval in the central `health_check`
+table. Both **Test connection** record actions remain available, including
+on-demand write tests.
+
 ## Setup and starter defaults

 The Storage home has a **Setup** card: connections and buckets (declared in code, synced into their tables), header rules, and protection rules, each with its state and next step. Every table has help (`help/qbit-object-storage.yaml`) saying what it is for and how its rows get created, which also shows when the table is empty.
@@ -172,6 +172,23 @@
   </dependencies>

   <build>
+      <resources>
+         <!-- help bundles carry the artifact version (QQQ help bundle provenance) -->
+         <resource>
+            <directory>src/main/resources</directory>
+            <filtering>true</filtering>
+            <includes>
+               <include>help/*.yaml</include>
+            </includes>
+         </resource>
+         <resource>
+            <directory>src/main/resources</directory>
+            <filtering>false</filtering>
+            <excludes>
+               <exclude>help/*.yaml</exclude>
+            </excludes>
+         </resource>
+      </resources>
      <plugins>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
@@ -255,14 +272,14 @@
   <profiles>
      <!-- Built against qqq 4.1 (QBitMetaDataProducer). Until qbit-build-parent  -->
      <!-- imports the released 4.1 BOM, this profile is active by default and     -->
-      <!-- imports qqq-bom-pom 4.1.0-RC.5 ahead of the parent's 4.0.0 BOM.         -->
+      <!-- imports qqq-bom-pom 4.1.0-RC.7 ahead of the parent's 4.0.0 BOM.         -->
      <profile>
         <id>qqq-snapshot</id>
         <activation>
            <activeByDefault>true</activeByDefault>
         </activation>
         <properties>
-            <qqq.snapshot.version>4.1.0-RC.6</qqq.snapshot.version>
+            <qqq.snapshot.version>4.1.0-RC.8</qqq.snapshot.version>
         </properties>
         <dependencyManagement>
            <dependencies>
@@ -24,6 +24,7 @@ package com.kingsrook.qbits.objectstorage;
 import java.util.ArrayList;
 import java.util.List;
 import com.kingsrook.qbits.objectstorage.app.StorageWidgets;
+import com.kingsrook.qbits.objectstorage.health.ObjectStorageConnectionChecks;
 import com.kingsrook.qbits.objectstorage.internal.ConfigSync;
 import com.kingsrook.qbits.objectstorage.internal.DefaultObjectHeaderPolicy;
 import com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime;
@@ -31,9 +32,13 @@ import com.kingsrook.qbits.objectstorage.model.ObjectStorageHelp;
 import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables;
 import com.kingsrook.qbits.objectstorage.processes.ObjectStorageProcesses;
 import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.health.HealthCheckSupplierMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthChecksMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
 import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundles;
 import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
@@ -187,6 +192,9 @@ public class ObjectStorageQBitProducer implements QBitMetaDataProducer<ObjectSto
         output.add(app);
      }
      output.addSelfToInstance(qInstance);
+      HealthChecksMetaData.ofOrWithNew(qInstance).withSupplier(new HealthCheckSupplierMetaData()
+         .withNamePrefix("object_storage.connection").withComponent("object_storage")
+         .withSupplier(new QCodeReference(ObjectStorageConnectionChecks.class)));
      if(Boolean.TRUE.equals(qBitConfig.getAdminApp()) && qBitConfig.getParentAppName() != null && qInstance.getApp(qBitConfig.getParentAppName()) != null)
      {
         ///////////////////////////////////////////////////////////////////////////////
@@ -194,6 +202,8 @@ public class ObjectStorageQBitProducer implements QBitMetaDataProducer<ObjectSto
         ///////////////////////////////////////////////////////////////////////////////
         qInstance.getApp(qBitConfig.getParentAppName()).withChild(qInstance.getApp(qBitConfig.getAppName()));
      }
+      // Registration is metadata-only; core skips startup sync without a helpContent table.
+      HelpBundles.register(qInstance, "help/qbit-object-storage.yaml");
      ObjectStorageHelp.apply(qInstance, qBitConfig);
      return (new MetaDataProducerMultiOutput());
   }
@@ -0,0 +1,121 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.objectstorage.health;
+
+
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Locale;
+import com.kingsrook.qbits.objectstorage.internal.CapabilityProbe;
+import com.kingsrook.qbits.objectstorage.internal.ConfigSync;
+import com.kingsrook.qbits.objectstorage.internal.Db;
+import com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime;
+import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables;
+import com.kingsrook.qqq.backend.core.alerts.AlertSeverity;
+import com.kingsrook.qqq.backend.core.exceptions.QException;
+import com.kingsrook.qqq.backend.core.health.HealthCategory;
+import com.kingsrook.qqq.backend.core.health.HealthCheckInput;
+import com.kingsrook.qqq.backend.core.health.HealthCheckKind;
+import com.kingsrook.qqq.backend.core.health.HealthCheckMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthCheckOutcome;
+import com.kingsrook.qqq.backend.core.health.HealthCheckSupplier;
+import com.kingsrook.qqq.backend.core.health.HealthDetails;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+
+
+/*******************************************************************************
+ ** Dynamic, read-only checks for configured connection rows. Diagnostic text and
+ ** resolved settings never cross the health boundary.
+ *******************************************************************************/
+public class ObjectStorageConnectionChecks implements HealthCheckSupplier
+{
+   /*******************************************************************************
+    ** Resolve current rows on every refresh; removed configuration is not probed.
+    *******************************************************************************/
+   @Override
+   public List<HealthCheckMetaData> supply(QInstance qInstance) throws QException
+   {
+      ConfigSync.ensureSynced();
+      List<HealthCheckMetaData> checks = new ArrayList<>();
+      for(QRecord row : Db.asSystem(() -> Db.query(ObjectStorageTables.CONNECTION)))
+      {
+         if(ObjectStorageRuntime.getConfig().findConnection(row.getValueString("name")) == null)
+         {
+            continue;
+         }
+         Integer id = row.getValueInteger("id");
+         String name = "object_storage.connection." + id.toString().toLowerCase(Locale.ROOT);
+         checks.add(new HealthCheckMetaData().withName(name).withLabel("Storage connection " + id)
+            .withComponent("object_storage").withCategory(HealthCategory.STORAGE).withKind(HealthCheckKind.DEPENDENCY)
+            .withIntervalSeconds(300).withTimeoutMs(20000).withSeverity(AlertSeverity.ERROR)
+            .withCheckInstance(input -> run(id, input)));
+      }
+      return (checks);
+   }
+
+
+
+   /*******************************************************************************
+    ** Check the current row, reuse the bounded capability probe, and expose only
+    ** numeric facts and a fixed problem vocabulary (never exception messages).
+    *******************************************************************************/
+   private static HealthCheckOutcome run(Integer id, HealthCheckInput input) throws QException
+   {
+      if(!Instant.now().isBefore(input.getDeadline()) || Thread.currentThread().isInterrupted())
+      {
+         return (HealthCheckOutcome.down("Storage probe timed out").withDetails(new HealthDetails().withValue("problemKind", "timeout")));
+      }
+      QRecord row = Db.asSystem(() -> Db.first(ObjectStorageTables.CONNECTION, "id", id));
+      if(row == null || ObjectStorageRuntime.getConfig().findConnection(row.getValueString("name")) == null)
+      {
+         return (HealthCheckOutcome.unknown("Storage connection is disabled or no longer configured"));
+      }
+      try
+      {
+         CapabilityProbe.Report report = CapabilityProbe.probeConnection(row.getValueString("name"), input.getDeadline());
+         HealthDetails details = new HealthDetails().withValue("bucketsProbed", report.getByBucket().size());
+         if(report.getClockSkewSeconds() != null)
+         {
+            details.withValue("clockSkewSeconds", report.getClockSkewSeconds());
+         }
+         if(report.getProblemKind() != null)
+         {
+            return (HealthCheckOutcome.down("Storage connection probe failed").withDetails(details.withValue("problemKind", report.getProblemKind())));
+         }
+         if(report.getClockSkewSeconds() != null && report.getClockSkewSeconds() > CapabilityProbe.CLOCK_SKEW_THRESHOLD_SECONDS)
+         {
+            return (HealthCheckOutcome.degraded("Storage clock skew exceeds the signing threshold").withDetails(details));
+         }
+         if(report.getByBucket().isEmpty())
+         {
+            return (HealthCheckOutcome.unknown("No storage buckets configured to probe").withDetails(details));
+         }
+         return (HealthCheckOutcome.up("Storage connection is reachable").withDetails(details));
+      }
+      catch(RuntimeException e)
+      {
+         return (HealthCheckOutcome.down("Storage connection probe failed")
+            .withDetails(new HealthDetails().withValue("problemKind", CapabilityProbe.problemKind(e))));
+      }
+   }
+}
@@ -21,7 +21,9 @@
 package com.kingsrook.qbits.objectstorage.internal;


+import javax.net.ssl.SSLException;
 import java.net.URI;
+import java.net.UnknownHostException;
 import java.net.http.HttpClient;
 import java.net.http.HttpRequest;
 import java.net.http.HttpResponse;
@@ -35,6 +37,7 @@ import java.util.EnumMap;
 import java.util.List;
 import java.util.Map;
 import java.util.UUID;
+import java.util.function.Function;
 import com.kingsrook.qbits.objectstorage.api.Capability;
 import com.kingsrook.qbits.objectstorage.api.CapabilityStatus;
 import com.kingsrook.qbits.objectstorage.api.PresignedPost;
@@ -45,7 +48,9 @@ import com.kingsrook.qbits.objectstorage.config.StorageConnectionConfig;
 import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables;
 import com.kingsrook.qqq.backend.core.logging.QLogger;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import software.amazon.awssdk.awscore.AwsRequestOverrideConfiguration;
 import software.amazon.awssdk.core.sync.RequestBody;
+import software.amazon.awssdk.services.s3.DelegatingS3Client;
 import software.amazon.awssdk.services.s3.S3Client;
 import software.amazon.awssdk.services.s3.model.CompleteMultipartUploadRequest;
 import software.amazon.awssdk.services.s3.model.CompletedMultipartUpload;
@@ -61,6 +66,7 @@ import software.amazon.awssdk.services.s3.model.ListObjectsV2Response;
 import software.amazon.awssdk.services.s3.model.ObjectIdentifier;
 import software.amazon.awssdk.services.s3.model.PutObjectRequest;
 import software.amazon.awssdk.services.s3.model.S3Exception;
+import software.amazon.awssdk.services.s3.model.S3Request;
 import software.amazon.awssdk.services.s3.model.UploadPartRequest;
 import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;
 import software.amazon.awssdk.services.s3.presigner.model.PutObjectPresignRequest;
@@ -79,6 +85,7 @@ public final class CapabilityProbe
   private static final QLogger LOG = QLogger.getLogger(CapabilityProbe.class);

   public static final String PROBE_PREFIX = ".qqq-probe/";
+   public static final int CLOCK_SKEW_THRESHOLD_SECONDS = 900;

   private static final int MIN_PART_BYTES = 5 * 1024 * 1024;

@@ -94,6 +101,17 @@ public final class CapabilityProbe
      private final List<String>                                   problems = new ArrayList<>();
      private Long                                                 clockSkewSeconds;
      private Boolean                                              presignHostReachable;
+      private String                                               problemKind;
+
+
+
+      /*******************************************************************************
+       ** Safe connection failure category, without provider messages or identifiers.
+       *******************************************************************************/
+      public String getProblemKind()
+      {
+         return (problemKind);
+      }



@@ -209,7 +227,7 @@ public final class CapabilityProbe
    *******************************************************************************/
   public static Report probeConnection(String connectionName, boolean includeWrite)
   {
-      details.clear();
+      details.get().clear();
      Report                  report     = new Report();
      ConnectionRuntime       connection = ObjectStorageRuntime.connection(connectionName);
      StorageConnectionConfig config     = connection.getConfig();
@@ -230,7 +248,7 @@ public final class CapabilityProbe
         List<String> notes = new ArrayList<>();
         report.notes.put(bucket.getAlias(), notes);

-         readProbes(connection, bucket, results, report);
+         readProbes(connection, bucket, results, report, null);
         recordVersioningState(bucket);

         if(includeWrite)
@@ -251,7 +269,7 @@ public final class CapabilityProbe
         }
         for(Map.Entry<Capability, CapabilityStatus> e : results.entrySet())
         {
-            CapabilityStore.record(bucket, e.getKey(), e.getValue(), details.getOrDefault(bucket.getAlias() + ":" + e.getKey(), null));
+            CapabilityStore.record(bucket, e.getKey(), e.getValue(), details.get().getOrDefault(bucket.getAlias() + ":" + e.getKey(), null));
         }
      }

@@ -262,7 +280,7 @@ public final class CapabilityProbe
         report.byBucket.get(writable.getAlias()).putAll(presign);
         for(Map.Entry<Capability, CapabilityStatus> e : presign.entrySet())
         {
-            CapabilityStore.record(writable, e.getKey(), e.getValue(), details.get(writable.getAlias() + ":" + e.getKey()));
+            CapabilityStore.record(writable, e.getKey(), e.getValue(), details.get().get(writable.getAlias() + ":" + e.getKey()));
         }
      }

@@ -281,7 +299,116 @@ public final class CapabilityProbe
      return (report);
   }

-   private static final Map<String, String> details = new java.util.concurrent.ConcurrentHashMap<>();
+   private static final ThreadLocal<Map<String, String>> details = ThreadLocal.withInitial(java.util.HashMap::new);
+
+
+
+   /*******************************************************************************
+    ** Reuse the read probes within a health deadline, without storing diagnostics
+    ** or running write/presign probes. Only the returned report is consumed.
+    *******************************************************************************/
+   public static Report probeConnection(String connectionName, Instant deadline)
+   {
+      try
+      {
+         return (probeReadOnly(connectionName, deadline));
+      }
+      finally
+      {
+         details.remove();
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Run the shared read-only calls without updating capability or bucket rows.
+    *******************************************************************************/
+   private static Report probeReadOnly(String connectionName, Instant deadline)
+   {
+      Report report = new Report();
+      ConnectionRuntime connection = ObjectStorageRuntime.connection(connectionName);
+      for(StorageBucketConfig bucket : ObjectStorageRuntime.getConfig().getBuckets())
+      {
+         if(!connectionName.equals(bucket.getConnection()))
+         {
+            continue;
+         }
+         if(!Instant.now().isBefore(deadline) || Thread.currentThread().isInterrupted())
+         {
+            report.problemKind = "timeout";
+            break;
+         }
+         Map<Capability, CapabilityStatus> results = new EnumMap<>(Capability.class);
+         report.byBucket.put(bucket.getAlias(), results);
+         readProbes(connection, bucket, results, report, deadline);
+      }
+      if(!Instant.now().isBefore(deadline))
+      {
+         report.problemKind = "timeout";
+      }
+      return (report);
+   }
+
+
+
+   /*******************************************************************************
+    ** Classify failures using types and known protocol codes, never message text.
+    *******************************************************************************/
+   public static String problemKind(Throwable error)
+   {
+      for(Throwable cause = error; cause != null; cause = cause.getCause())
+      {
+         if(cause instanceof UnknownHostException)
+         {
+            return ("dns");
+         }
+         if(cause instanceof SSLException)
+         {
+            return ("tls");
+         }
+         String code = ErrorMapper.errorCode(cause);
+         if("RequestTimeTooSkewed".equals(code) || "RequestExpired".equals(code))
+         {
+            return ("clock");
+         }
+         if(cause instanceof S3Exception s3 && (s3.statusCode() == 401 || s3.statusCode() == 403
+            || "InvalidAccessKeyId".equals(code) || "SignatureDoesNotMatch".equals(code)))
+         {
+            return ("auth");
+         }
+      }
+      return ("connection");
+   }
+
+
+
+   /*******************************************************************************
+    ** Bound each SDK call (including retries) by the remaining probe budget.
+    *******************************************************************************/
+   private static S3Client boundedClient(S3Client client, Instant deadline)
+   {
+      return (new DelegatingS3Client(client)
+      {
+         /*******************************************************************************
+          ** Preserve credentials and other overrides while limiting the request.
+          *******************************************************************************/
+         @Override
+         @SuppressWarnings("unchecked") // toBuilder retains the concrete request type.
+         protected <T extends S3Request, ReturnT> ReturnT invokeOperation(T request, Function<T, ReturnT> operation)
+         {
+            Duration remaining = Duration.between(Instant.now(), deadline);
+            if(remaining.isNegative() || remaining.isZero() || Thread.currentThread().isInterrupted())
+            {
+               throw (new IllegalStateException("Storage probe deadline exceeded"));
+            }
+            AwsRequestOverrideConfiguration.Builder overrides = request.overrideConfiguration()
+               .map(AwsRequestOverrideConfiguration::toBuilder).orElseGet(AwsRequestOverrideConfiguration::builder);
+            T bounded = (T) request.toBuilder().overrideConfiguration(overrides.apiCallTimeout(remaining).apiCallAttemptTimeout(remaining).build()).build();
+            return (operation.apply(bounded));
+         }
+      });
+   }



@@ -291,7 +418,7 @@ public final class CapabilityProbe
    *******************************************************************************/
   private static void recordVersioningState(StorageBucketConfig bucket)
   {
-      String seen = details.get(bucket.getAlias() + ":" + Capability.VERSIONING);
+      String seen = details.get().get(bucket.getAlias() + ":" + Capability.VERSIONING);
      if(seen == null || !Db.available(ObjectStorageTables.BUCKET))
      {
         return;
@@ -317,14 +444,14 @@ public final class CapabilityProbe
   /*******************************************************************************
    ** The read-only probes of one bucket.
    *******************************************************************************/
-   private static void readProbes(ConnectionRuntime connection, StorageBucketConfig bucket, Map<Capability, CapabilityStatus> results, Report report)
+   private static void readProbes(ConnectionRuntime connection, StorageBucketConfig bucket, Map<Capability, CapabilityStatus> results, Report report, Instant deadline)
   {
-      S3Client client = connection.client();
+      S3Client client = deadline == null ? connection.client() : boundedClient(connection.client(), deadline);
      String   name   = name(bucket);
      String   alias  = bucket.getAlias();
      String   sampleKey = null;

-      try(RateLimiter.Permit ignored = connection.getRateLimiter().acquire())
+      try(RateLimiter.Permit ignored = deadline == null ? connection.getRateLimiter().acquire() : connection.getRateLimiter().acquire(deadline))
      {
         ListObjectsV2Response r = client.listObjectsV2(ListObjectsV2Request.builder().bucket(name).maxKeys(1).build());
         results.put(Capability.LIST_V2, CapabilityStatus.SUPPORTED);
@@ -352,6 +479,7 @@ public final class CapabilityProbe
      {
         results.put(Capability.LIST_V2, classify(alias, Capability.LIST_V2, e));
         report.problems.add(alias + ": listing failed (" + shortMessage(e) + ")");
+         report.problemKind = problemKind(e);
      }

      probe(results, alias, Capability.VERSIONING, () ->
@@ -359,10 +487,10 @@ public final class CapabilityProbe
         String status = client.getBucketVersioning(b -> b.bucket(name)).statusAsString();
         if(status == null && ProviderDefaults.defaultStatus(connection.getConfig().getProviderType(), Capability.VERSIONING) == CapabilityStatus.UNSUPPORTED)
         {
-            details.put(alias + ":" + Capability.VERSIONING, "the provider reports versioning off and documents it as a stub");
+            details.get().put(alias + ":" + Capability.VERSIONING, "the provider reports versioning off and documents it as a stub");
            return (CapabilityStatus.UNSUPPORTED);
         }
-         details.put(alias + ":" + Capability.VERSIONING, status == null ? "not enabled" : status);
+         details.get().put(alias + ":" + Capability.VERSIONING, status == null ? "not enabled" : status);
         return (CapabilityStatus.SUPPORTED);
      });
      probe(results, alias, Capability.OBJECT_LOCK, () ->
@@ -376,7 +504,7 @@ public final class CapabilityProbe
         {
            if("ObjectLockConfigurationNotFoundError".equals(ErrorMapper.errorCode(e)))
            {
-               details.put(alias + ":" + Capability.OBJECT_LOCK, "object lock is not enabled on this bucket");
+               details.get().put(alias + ":" + Capability.OBJECT_LOCK, "object lock is not enabled on this bucket");
               return (CapabilityStatus.UNSUPPORTED);
            }
            throw (e);
@@ -391,7 +519,7 @@ public final class CapabilityProbe
      if(key == null)
      {
         results.put(Capability.OBJECT_TAGGING, CapabilityStatus.UNKNOWN);
-         details.put(alias + ":" + Capability.OBJECT_TAGGING, "no object in the bucket to test");
+         details.get().put(alias + ":" + Capability.OBJECT_TAGGING, "no object in the bucket to test");
      }
      else
      {
@@ -505,7 +633,7 @@ public final class CapabilityProbe
            try
            {
               client.putObject(PutObjectRequest.builder().bucket(name).key(key).ifNoneMatch("*").build(), RequestBody.fromString("2"));
-               details.put(alias + ":" + Capability.CONDITIONAL_WRITE, "If-None-Match was ignored");
+               details.get().put(alias + ":" + Capability.CONDITIONAL_WRITE, "If-None-Match was ignored");
               return (CapabilityStatus.UNSUPPORTED);
            }
            catch(S3Exception e)
@@ -564,7 +692,7 @@ public final class CapabilityProbe
            report.presignHostReachable = true;
            if(r.statusCode() == 200 && !"probe".equals(r.body()))
            {
-               details.put(alias + ":" + Capability.PRESIGNED_GET, "the presigned GET returned unexpected content (does it reach the same store?)");
+               details.get().put(alias + ":" + Capability.PRESIGNED_GET, "the presigned GET returned unexpected content (does it reach the same store?)");
               return (CapabilityStatus.UNKNOWN);
            }
            return (httpStatus(alias, Capability.PRESIGNED_GET, r.statusCode(), r.body()));
@@ -618,7 +746,7 @@ public final class CapabilityProbe
      {
         return (CapabilityStatus.SUPPORTED);
      }
-      details.put(alias + ":" + capability, "HTTP " + status + (body == null ? "" : " " + body.replaceAll("\\s+", " ").substring(0, Math.min(200, body.length()))));
+      details.get().put(alias + ":" + capability, "HTTP " + status + (body == null ? "" : " " + body.replaceAll("\\s+", " ").substring(0, Math.min(200, body.length()))));
      if(status == 403)
      {
         return (CapabilityStatus.NOT_PERMITTED);
@@ -669,7 +797,7 @@ public final class CapabilityProbe
   static CapabilityStatus classify(String alias, Capability capability, Exception e)
   {
      String detail = shortMessage(e);
-      details.put(alias + ":" + capability, detail);
+      details.get().put(alias + ":" + capability, detail);
      if(e instanceof S3Exception s3)
      {
         String code = ErrorMapper.errorCode(s3);
@@ -291,8 +291,9 @@ public class ConnectionRuntime


   /*******************************************************************************
-    ** STAND-IN(core=QRun-IO/qqq#1165, removeBy=4.1.0-RC.7)
-    ** RC.5 metadata/factory lacks a session token or credentials-provider hook.
+    ** STAND-IN(core=QRun-IO/qqq#1165, removeBy=4.1.0-RC.9)
+    ** RC.8 S3BackendMetaData and S3ClientFactory still lack a session token or
+    ** credentials-provider hook (verified against the released JAR).
    ** Override request credentials only; core still owns the client and transport.
    *******************************************************************************/
   private static class SessionCredentialsClient extends DelegatingS3Client
@@ -40,8 +40,8 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
 import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
-import com.kingsrook.qqq.backend.core.model.metadata.security.QSecurityKeyType;
 import com.kingsrook.qqq.backend.core.model.session.QSession;
+import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;


 /*******************************************************************************
@@ -131,30 +131,15 @@ public final class Db


   /*******************************************************************************
-    ** A session that holds every permission and every all-access security key of
-    ** the given instance (computed per instance, never cached across instances).
+    ** The core system-user session (every permission and every all-access
+    ** security key of the given instance, read per instance). It must be a
+    ** QSystemUserSession: token-based authentication modules (OAuth2, Auth0)
+    ** accept that class without a token and reject any other tokenless session,
+    ** which made every system read and write fail on such hosts.
    *******************************************************************************/
   private static QSession systemSession(QInstance instance)
   {
-      QSession session = new QSession()
-      {
-         /*******************************************************************************
-          ** The system user holds every permission.
-          *******************************************************************************/
-         @Override
-         public boolean hasPermission(String permissionName)
-         {
-            return (true);
-         }
-      };
-      for(QSecurityKeyType keyType : instance.getSecurityKeyTypes().values())
-      {
-         if(keyType.getAllAccessKeyName() != null)
-         {
-            session.withSecurityKeyValue(keyType.getAllAccessKeyName(), true);
-         }
-      }
-      return (session);
+      return (new QSystemUserSession(instance));
   }


@@ -21,7 +21,10 @@
 package com.kingsrook.qbits.objectstorage.internal;


+import java.time.Duration;
+import java.time.Instant;
 import java.util.concurrent.Semaphore;
+import java.util.concurrent.TimeUnit;


 /*******************************************************************************
@@ -57,17 +60,52 @@ public class RateLimiter
    *******************************************************************************/
   public Permit acquire()
   {
+      return (acquire(null));
+   }
+
+
+
+   /*******************************************************************************
+    ** Wait no later than a caller's deadline, releasing the slot on interruption.
+    *******************************************************************************/
+   public Permit acquire(Instant deadline)
+   {
+      boolean acquired = false;
      try
      {
-         concurrency.acquire();
-         waitForToken();
+         if(deadline == null)
+         {
+            concurrency.acquire();
+            acquired = true;
+         }
+         else
+         {
+            acquired = concurrency.tryAcquire(Math.max(0, Duration.between(Instant.now(), deadline).toNanos()), TimeUnit.NANOSECONDS);
+            if(!acquired)
+            {
+               throw (new IllegalStateException("Storage probe deadline exceeded"));
+            }
+         }
+         waitForToken(deadline);
         return (new Permit(concurrency));
      }
      catch(InterruptedException e)
      {
+         if(acquired)
+         {
+            concurrency.release();
+         }
         Thread.currentThread().interrupt();
         throw (new IllegalStateException("Interrupted while waiting for the storage rate limiter", e));
      }
+      catch(RuntimeException e)
+      {
+         if(acquired)
+         {
+            concurrency.release();
+         }
+         throw (e);
+      }
   }


@@ -75,10 +113,14 @@ public class RateLimiter
   /*******************************************************************************
    ** Block until a token is available.
    *******************************************************************************/
-   private void waitForToken() throws InterruptedException
+   private void waitForToken(Instant deadline) throws InterruptedException
   {
      while(true)
      {
+         if(deadline != null && !Instant.now().isBefore(deadline))
+         {
+            throw (new IllegalStateException("Storage probe deadline exceeded"));
+         }
         long sleepMillis;
         synchronized(this)
         {
@@ -93,7 +135,7 @@ public class RateLimiter
            throttledWaits++;
            sleepMillis = Math.max(1, (long) ((1.0 - tokens) / opsPerSecond * 1000));
         }
-         Thread.sleep(sleepMillis);
+         Thread.sleep(deadline == null ? sleepMillis : Math.min(sleepMillis, Math.max(1, Duration.between(Instant.now(), deadline).toMillis())));
      }
   }

@@ -36,7 +36,8 @@ import com.kingsrook.qqq.backend.core.utils.StringUtils;
 ** the allow-list. Resolved values live only in memory, never in a record, a log
 ** line or a process value.
 **
- ** STAND-IN(core=QRun-IO/qqq#1165, removeBy=4.1.0-RC.7)
+ ** STAND-IN(core=QRun-IO/qqq#1165, removeBy=4.1.0-RC.9)
+ ** RC.8 QMetaDataVariableInterpreter still supports env/prop, not file references.
 ** ${file.PATH} moves to QMetaDataVariableInterpreter in QQQ core
 ** (spec section 8.2); then this class delegates entirely.
 *******************************************************************************/
@@ -40,9 +40,8 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 ** Applies help/qbit-object-storage.yaml (the QQQ help-bundle format) to the
 ** Storage app and its object_storage_* tables: what each table is for and how
 ** its rows get created, shown on the table screen and when it is empty.
- ** STAND-IN(core=QRun-IO/qqq#1198, removeBy=4.1.0-RC.8): replace with
- ** HelpBundles.register once the pinned QQQ release includes it; the bundle file
- ** stays the same.
+ ** Inline defaults preserve empty-state help for hosts without helpContent.
+ ** The producer also registers the bundle for core sync and HOST overrides.
 *******************************************************************************/
 public final class ObjectStorageHelp
 {
@@ -81,7 +80,7 @@ public final class ObjectStorageHelp
         {
            String key     = entry.path("key").asText();
            String summary = entry.path("summary").asText();
-            if(key.isBlank() || summary.isBlank() || summary.length() > 120 || entry.path("content").asText().isBlank() || entries.put(key, entry) != null)
+            if(key.isBlank() || summary.isBlank() || summary.length() > 120 || entries.put(key, entry) != null)
            {
               throw (new IOException("invalid or duplicate entry: " + key));
            }
@@ -38,7 +38,9 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 **
 ** The statements are idempotent (CREATE ... IF NOT EXISTS) and include the
 ** unique constraints and the browse indexes the design calls for
- ** ((bucket, key) unique; bucket with folder, last modified, size, content type).
+ ** ((bucket, key) unique; bucket with folder, last modified, size, content type),
+ ** then the foreign keys, each in a block that also upgrades an existing schema
+ ** (orphans recorded in qqq_fk_quarantine first; see addForeignKey).
 *******************************************************************************/
 public final class ObjectStorageSchema
 {
@@ -52,14 +54,79 @@ public final class ObjectStorageSchema
      ObjectStorageTables.OBJECT, List.of(List.of("bucketId", "objectKey")),
      ObjectStorageTables.USAGE_DAILY, List.of(List.of("bucketId", "snapshotDate")));

-   /** Browse and lookup indexes: table to column lists. */
+   /** Browse and lookup indexes: table to column lists. New entries go at the end of
+    ** a table's list, so the generated names (ix_<table>_<n>) of existing ones never change.
+    ** The child lists on record pages (folder -> objects, object -> versions and references)
+    ** filter on the parent id alone, so those columns lead an index of their own. */
   private static final Map<String, List<List<String>>> INDEXES = Map.of(
      ObjectStorageTables.OBJECT, List.of(List.of("bucketId", "folderId"), List.of("bucketId", "lastModified"), List.of("bucketId", "size"), List.of("bucketId", "contentType"),
-         List.of("bucketId", "deleted", "referenceCount"), List.of("bucketId", "headersKnown", "deleted")),
-      ObjectStorageTables.OBJECT_REFERENCE, List.of(List.of("bucketId", "objectId"), List.of("bucketId", "missing")),
+         List.of("bucketId", "deleted", "referenceCount"), List.of("bucketId", "headersKnown", "deleted"), List.of("folderId")),
+      ObjectStorageTables.OBJECT_REFERENCE, List.of(List.of("bucketId", "objectId"), List.of("bucketId", "missing"), List.of("objectId")),
      ObjectStorageTables.OPERATION, List.of(List.of("bucketId", "occurredAt"), List.of("objectKey", "occurredAt")),
      ObjectStorageTables.FOLDER, List.of(List.of("parentFolderId")),
-      ObjectStorageTables.OBJECT_VERSION, List.of(List.of("bucketId", "objectKey")));
+      ObjectStorageTables.OBJECT_VERSION, List.of(List.of("bucketId", "objectKey"), List.of("objectId")),
+      ObjectStorageTables.BUCKET, List.of(List.of("connectionId")),
+      ObjectStorageTables.HEADER_RULE, List.of(List.of("bucketId")),
+      ObjectStorageTables.PROTECTION_RULE, List.of(List.of("bucketId")));
+
+   /** The table that records the rows an upgrade changed so a foreign key could be added
+    ** (identical DDL in every qbit, so whichever runs first creates it). */
+   public static final String QUARANTINE_DDL = "CREATE TABLE IF NOT EXISTS qqq_fk_quarantine (id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, "
+      + "table_name VARCHAR(100) NOT NULL, column_name VARCHAR(100) NOT NULL, row_id VARCHAR(100), missing_parent_id VARCHAR(100), action VARCHAR(20) NOT NULL, "
+      + "row_data TEXT, changeset VARCHAR(200), quarantined_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP)";
+
+   /** Every parent/child relation, enforced in the database. The qbit never deletes a
+    ** connection, bucket or folder row (ConfigSync only upserts; folders are only
+    ** created), and its only deletes are object rows (scan removal, trash retention,
+    ** hard-delete write-through) and reference rows (rebuilt by each reference scan),
+    ** through QQQ deletes with no associations. So:
+    ** - observed state that belongs to a parent (capabilities, folders, objects, versions,
+    **   usage) CASCADEs: an object delete takes its versions, and an admin deleting a
+    **   bucket row takes its index with it;
+    ** - optional pointers into the index (object.folderId, reference.objectId) SET NULL:
+    **   the next scan sets the folder again, and a reference to a removed object is
+    **   missing until the next reference scan rebuilds the rows;
+    ** - admin-authored config (header and protection rules), the operation audit log and
+    **   a bucket's connection RESTRICT (NO ACTION): deleting a bucket row that has rules
+    **   or history, or a connection that buckets use, is refused rather than silently
+    **   dropping protection or audit rows (ConfigSync would re-create a configured bucket
+    **   under a new id, with its rules gone).
+    ** Inserts are parents first everywhere (connections before buckets, a folder's parent
+    ** before the folder, folders before objects), so nothing needs to be deferred.
+    ** New entries go at the end. */
+   private static final List<ForeignKey> FOREIGN_KEYS = List.of(
+      new ForeignKey(ObjectStorageTables.BUCKET, "connectionId", ObjectStorageTables.CONNECTION, "NO ACTION"),
+      new ForeignKey(ObjectStorageTables.CONNECTION_CAPABILITY, "connectionId", ObjectStorageTables.CONNECTION, "CASCADE"),
+      new ForeignKey(ObjectStorageTables.BUCKET_CAPABILITY, "bucketId", ObjectStorageTables.BUCKET, "CASCADE"),
+      new ForeignKey(ObjectStorageTables.HEADER_RULE, "bucketId", ObjectStorageTables.BUCKET, "NO ACTION"),
+      new ForeignKey(ObjectStorageTables.PROTECTION_RULE, "bucketId", ObjectStorageTables.BUCKET, "NO ACTION"),
+      new ForeignKey(ObjectStorageTables.OPERATION, "bucketId", ObjectStorageTables.BUCKET, "NO ACTION"),
+      new ForeignKey(ObjectStorageTables.USAGE_DAILY, "bucketId", ObjectStorageTables.BUCKET, "CASCADE"),
+      new ForeignKey(ObjectStorageTables.FOLDER, "bucketId", ObjectStorageTables.BUCKET, "CASCADE"),
+      new ForeignKey(ObjectStorageTables.FOLDER, "parentFolderId", ObjectStorageTables.FOLDER, "CASCADE"),
+      new ForeignKey(ObjectStorageTables.OBJECT, "bucketId", ObjectStorageTables.BUCKET, "CASCADE"),
+      new ForeignKey(ObjectStorageTables.OBJECT, "folderId", ObjectStorageTables.FOLDER, "SET NULL"),
+      new ForeignKey(ObjectStorageTables.OBJECT_VERSION, "bucketId", ObjectStorageTables.BUCKET, "CASCADE"),
+      new ForeignKey(ObjectStorageTables.OBJECT_VERSION, "objectId", ObjectStorageTables.OBJECT, "CASCADE"),
+      new ForeignKey(ObjectStorageTables.OBJECT_REFERENCE, "bucketId", ObjectStorageTables.BUCKET, "CASCADE"),
+      new ForeignKey(ObjectStorageTables.OBJECT_REFERENCE, "objectId", ObjectStorageTables.OBJECT, "SET NULL"));
+
+
+
+   /*******************************************************************************
+    ** One foreign key: the child table and field, the parent table (always its id)
+    ** and the ON DELETE rule.
+    *******************************************************************************/
+   public record ForeignKey(String table, String field, String parent, String onDelete)
+   {
+      /*******************************************************************************
+       ** The constraint name: fk_<child table>_<child column>.
+       *******************************************************************************/
+      public String name()
+      {
+         return ("fk_" + table + "_" + column(field));
+      }
+   }



@@ -99,11 +166,56 @@ public final class ObjectStorageSchema
            }
         }
      }
+      statements.add(QUARANTINE_DDL);
+      for(ForeignKey fk : FOREIGN_KEYS)
+      {
+         statements.add(addForeignKey(fk, tables));
+      }
      return (statements);
   }



+   /*******************************************************************************
+    ** The foreign keys the schema declares.
+    *******************************************************************************/
+   public static List<ForeignKey> foreignKeys()
+   {
+      return (FOREIGN_KEYS);
+   }
+
+
+
+   /*******************************************************************************
+    ** One idempotent block that adds a foreign key and upgrades an existing schema
+    ** safely: rows whose parent is gone are first recorded in qqq_fk_quarantine, then
+    ** their reference is cleared (nullable column) or the row is deleted (required
+    ** column; the whole row is recorded), so adding the constraint cannot fail. A
+    ** no-op once the constraint exists.
+    *******************************************************************************/
+   private static String addForeignKey(ForeignKey fk, List<QTableMetaData> tables)
+   {
+      String  col      = column(fk.field());
+      boolean required = tables.stream().filter(t -> t.getName().equals(fk.table())).findFirst()
+         .map(t -> Boolean.TRUE.equals(t.getField(fk.field()).getIsRequired())).orElse(false);
+      String  orphans  = "c." + col + " IS NOT NULL AND NOT EXISTS (SELECT 1 FROM " + fk.parent() + " p WHERE p.id = c." + col + ")";
+      return ("DO $$\n"
+         + "BEGIN\n"
+         + "   IF NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conname = '" + fk.name() + "' AND conrelid = '" + fk.table() + "'::regclass) THEN\n"
+         + "      INSERT INTO qqq_fk_quarantine (table_name, column_name, row_id, missing_parent_id, action, row_data, changeset)\n"
+         + "         SELECT '" + fk.table() + "', '" + col + "', c.id::text, c." + col + "::text, '" + (required ? "DELETED" : "NULLED") + "', row_to_json(c)::text, "
+         + "'qbit-object-storage ObjectStorageSchema'\n"
+         + "         FROM " + fk.table() + " c WHERE " + orphans + ";\n"
+         + (required
+            ? "      DELETE FROM " + fk.table() + " c WHERE " + orphans + ";\n"
+            : "      UPDATE " + fk.table() + " c SET " + col + " = NULL WHERE " + orphans + ";\n")
+         + "      ALTER TABLE " + fk.table() + " ADD CONSTRAINT " + fk.name() + " FOREIGN KEY (" + col + ") REFERENCES " + fk.parent() + " (id) ON DELETE " + fk.onDelete() + ";\n"
+         + "   END IF;\n"
+         + "END $$");
+   }
+
+
+
   /*******************************************************************************
    ** The statements as one script.
    *******************************************************************************/
@@ -227,7 +227,7 @@ public final class TestConnectionProcess
            }
            if(report.getClockSkewSeconds() != null)
            {
-               boolean skewed = report.getClockSkewSeconds() > 900;
+               boolean skewed = report.getClockSkewSeconds() > CapabilityProbe.CLOCK_SKEW_THRESHOLD_SECONDS;
               ok = ok && !skewed;
               lines.add(new ProcessSummaryLine(skewed ? Status.ERROR : Status.INFO, "Clock skew against the store: " + report.getClockSkewSeconds() + " seconds"
                  + (skewed ? " (SigV4 fails beyond 15 minutes)" : "")));
@@ -0,0 +1,76 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.objectstorage;
+
+
+import java.util.Set;
+import java.util.stream.Collectors;
+import com.kingsrook.qbits.objectstorage.testsupport.StorageTestSupport;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundles;
+import com.kingsrook.qqq.backend.core.testing.HelpBundleAssertions;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+
+
+/*******************************************************************************
+ ** Coverage of every object owned by the storage qbit, including its optional app.
+ *******************************************************************************/
+class ObjectStorageHelpBundleTest
+{
+   /*******************************************************************************
+    ** Release the installed runtime after each test.
+    *******************************************************************************/
+   @AfterEach
+   void tearDown()
+   {
+      StorageTestSupport.tearDown();
+   }
+
+
+
+   /*******************************************************************************
+    ** Register safely without help tables and cover all owned metadata.
+    *******************************************************************************/
+   @Test
+   void bundleIsCompleteWithoutHelpTables() throws Exception
+   {
+      QInstance instance = StorageTestSupport.setUp(new ObjectStorageQBitConfig()
+         .withBackendName(StorageTestSupport.MEMORY).withBucketScoping(false).withOpenWhenAnonymous(true).withAdminApp(true));
+
+      assertThat(instance.getTable("helpContent")).isNull();
+      assertThat(HelpBundles.of(instance).getBundles()).singleElement()
+         .satisfies(bundle -> assertThat(bundle.getId()).isEqualTo("qbit-object-storage"));
+      HelpBundles.syncOnStartup(instance);
+      Set<String> keys = HelpBundles.of(instance).getBundles().get(0).getEntries().stream()
+         .map(entry -> entry.getKey()).collect(Collectors.toSet());
+      // RC.7 validates these entries but only requires app/table/process/field coverage.
+      instance.getWidgets().keySet().stream().filter(name -> name.startsWith("object_storage_") || name.startsWith("objectStorage"))
+         .forEach(name -> assertThat(keys).contains("widget:" + name + ";slot:label"));
+      instance.getPossibleValueSources().keySet().stream().filter(name -> name.startsWith("object_storage_"))
+         .forEach(name -> assertThat(keys).contains("possibleValueSource:" + name + ";slot:help"));
+      HelpBundleAssertions.forInstance(instance)
+         .bundles("help/qbit-object-storage.yaml")
+         .owning(target -> target.name().startsWith("object_storage_") || target.name().startsWith("objectStorage"))
+         .assertComplete();
+   }
+}
@@ -0,0 +1,364 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.objectstorage.health;
+
+
+import javax.net.ssl.SSLHandshakeException;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationHandler;
+import java.lang.reflect.Proxy;
+import java.net.UnknownHostException;
+import java.time.Duration;
+import java.time.Instant;
+import java.time.ZoneOffset;
+import java.time.format.DateTimeFormatter;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig;
+import com.kingsrook.qbits.objectstorage.config.StorageBucketConfig;
+import com.kingsrook.qbits.objectstorage.config.StorageConnectionConfig;
+import com.kingsrook.qbits.objectstorage.internal.CapabilityProbe;
+import com.kingsrook.qbits.objectstorage.internal.ConfigSync;
+import com.kingsrook.qbits.objectstorage.internal.ConnectionRuntime;
+import com.kingsrook.qbits.objectstorage.internal.Db;
+import com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime;
+import com.kingsrook.qbits.objectstorage.internal.RateLimiter;
+import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables;
+import com.kingsrook.qbits.objectstorage.processes.TestConnectionProcess;
+import com.kingsrook.qbits.objectstorage.testsupport.StorageTestSupport;
+import com.kingsrook.qqq.backend.core.health.HealthCategory;
+import com.kingsrook.qqq.backend.core.health.HealthCheckInput;
+import com.kingsrook.qqq.backend.core.health.HealthCheckKind;
+import com.kingsrook.qqq.backend.core.health.HealthCheckMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthCheckOutcome;
+import com.kingsrook.qqq.backend.core.health.HealthChecksMetaData;
+import com.kingsrook.qqq.backend.core.health.HealthStatus;
+import com.kingsrook.qqq.backend.core.model.data.QRecord;
+import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+import software.amazon.awssdk.awscore.exception.AwsErrorDetails;
+import software.amazon.awssdk.http.SdkHttpResponse;
+import software.amazon.awssdk.services.s3.S3Client;
+import software.amazon.awssdk.services.s3.model.ListObjectsV2Response;
+import software.amazon.awssdk.services.s3.model.S3Exception;
+import software.amazon.awssdk.services.s3.model.S3Request;
+import software.amazon.awssdk.utils.builder.SdkBuilder;
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+
+/*******************************************************************************
+ ** Exercise the real probe using an in-memory S3 client and QQQ record store.
+ *******************************************************************************/
+class ObjectStorageConnectionChecksTest
+{
+   private static final String SECRET = "S3NT1NEL-secret";
+   private QInstance instance;
+   private ObjectStorageQBitConfig config;
+   private final List<String> calls = new ArrayList<>();
+   private RuntimeException failure;
+   private long skew;
+   private boolean expireAfterList;
+   private Instant deadline;
+
+
+
+   /*******************************************************************************
+    ** Poison every credential reference; the fake prevents all network access.
+    *******************************************************************************/
+   @BeforeEach
+   void setUp() throws Exception
+   {
+      System.setProperty("os.health.secret", SECRET);
+      String ref = "${prop.os.health.secret}";
+      config = new ObjectStorageQBitConfig().withBackendName(StorageTestSupport.MEMORY)
+         .withAllowedSecretRefPatterns(List.of("prop.os.health.*"))
+         .withConnection(new StorageConnectionConfig("PRIMARY").withEndpoint("https://user:" + SECRET + "@private.example")
+            .withAccessKeyRef(ref).withSecretKeyRef(ref).withSessionTokenRef(ref)
+            .withPresignAccessKeyRef(ref).withPresignSecretKeyRef(ref).withCaBundleRef(ref).withMaxConcurrency(1))
+         .withBucket(new StorageBucketConfig("files").withConnection("PRIMARY").withBucketName("private-bucket").withIndexed(false));
+      instance = StorageTestSupport.setUp(config);
+      S3Client client = (S3Client) Proxy.newProxyInstance(S3Client.class.getClassLoader(), new Class<?>[] { S3Client.class }, (proxy, method, args) ->
+      {
+         if(method.getName().equals("close"))
+         {
+            return (null);
+         }
+         if(args != null && args.length == 1 && args[0] instanceof java.util.function.Consumer<?>)
+         {
+            return (InvocationHandler.invokeDefault(proxy, method, args));
+         }
+         calls.add(method.getName());
+         assertThat(method.getName()).matches("listObjectsV2|getBucket.*|getObjectLockConfiguration|getObjectTagging");
+         S3Request request = (S3Request) args[0];
+         Duration timeout = request.overrideConfiguration().orElseThrow().apiCallTimeout().orElseThrow();
+         assertThat(timeout).isPositive().isLessThanOrEqualTo(Duration.ofSeconds(20));
+         assertThat(request.overrideConfiguration().orElseThrow().apiCallAttemptTimeout()).contains(timeout);
+         if(failure != null)
+         {
+            throw (failure);
+         }
+         if(method.getName().equals("listObjectsV2"))
+         {
+            if(expireAfterList)
+            {
+               Thread.sleep(Math.max(1, Duration.between(Instant.now(), deadline).toMillis() + 10));
+            }
+            return (ListObjectsV2Response.builder().sdkHttpResponse(SdkHttpResponse.builder().statusCode(200)
+               .putHeader("Date", DateTimeFormatter.RFC_1123_DATE_TIME.format(Instant.now().minusSeconds(skew).atZone(ZoneOffset.UTC))).build()).build());
+         }
+         return (((SdkBuilder<?, ?>) method.getReturnType().getMethod("builder").invoke(null)).build());
+      });
+      Field field = ConnectionRuntime.class.getDeclaredField("client");
+      field.setAccessible(true);
+      field.set(ObjectStorageRuntime.connection("PRIMARY"), client);
+   }
+
+
+
+   /*******************************************************************************
+    ** Restore process-wide fixtures and the poisoned property.
+    *******************************************************************************/
+   @AfterEach
+   void tearDown()
+   {
+      StorageTestSupport.tearDown();
+      System.clearProperty("os.health.secret");
+   }
+
+
+
+   /*******************************************************************************
+    ** Producer registration includes the required metadata and keeps both actions.
+    *******************************************************************************/
+   @Test
+   void registrationAndRefresh() throws Exception
+   {
+      var registry = HealthChecksMetaData.ofOrWithNew(instance);
+      assertThat(registry.getSuppliers()).singleElement().satisfies(supplier ->
+      {
+         assertThat(supplier.getNamePrefix()).isEqualTo("object_storage.connection");
+         assertThat(supplier.getComponent()).isEqualTo("object_storage");
+      });
+      HealthCheckMetaData check = check();
+      assertThat(check.getName()).isEqualTo("object_storage.connection." + ConfigSync.connectionId("PRIMARY")).matches("[a-z0-9]+([._-][a-z0-9]+)*");
+      assertThat(check.getKind()).isEqualTo(HealthCheckKind.DEPENDENCY);
+      assertThat(check.getCategory()).isEqualTo(HealthCategory.STORAGE);
+      assertThat(check.getComponent()).isEqualTo("object_storage");
+      assertThat(check.getIntervalSeconds()).isEqualTo(300);
+      assertThat(check.getTimeoutMs()).isEqualTo(20000);
+      assertThat(instance.getProcess(TestConnectionProcess.READ_NAME)).isNotNull();
+      assertThat(instance.getProcess(TestConnectionProcess.WRITE_NAME)).isNotNull();
+      ConfigSync.sync();
+      assertThat(registry.resolveAll(instance)).hasSize(1);
+      config.withConnection(new StorageConnectionConfig("SECOND"));
+      ConfigSync.sync();
+      assertThat(registry.resolveAll(instance)).hasSize(2);
+      config.getConnections().clear();
+      assertThat(registry.resolveAll(instance)).isEmpty();
+      assertClean(check.getCheckInstance().run(input()), HealthStatus.UNKNOWN);
+   }
+
+
+
+   /*******************************************************************************
+    ** Successful checks read buckets without persisting any capability diagnostics.
+    *******************************************************************************/
+   @Test
+   void upReadOnlyAndNoSecrets() throws Exception
+   {
+      HealthCheckOutcome outcome = run();
+      assertClean(outcome, HealthStatus.UP);
+      assertThat(outcome.getDetails().getValues()).containsEntry("bucketsProbed", 1);
+      assertThat(calls).contains("listObjectsV2", "getBucketVersioning", "getBucketAcl");
+      assertThat(Db.asSystem(() -> Db.query(ObjectStorageTables.CONNECTION_CAPABILITY))).isEmpty();
+      assertThat(Db.asSystem(() -> Db.query(ObjectStorageTables.BUCKET_CAPABILITY))).isEmpty();
+      assertThat(Db.asSystem(() -> Db.first(ObjectStorageTables.CONNECTION, "name", "PRIMARY")).getValue("lastTestedAt")).isNull();
+   }
+
+
+
+   /*******************************************************************************
+    ** Clock skew uses the same threshold as the record action.
+    *******************************************************************************/
+   @Test
+   void clockSkew() throws Exception
+   {
+      skew = CapabilityProbe.CLOCK_SKEW_THRESHOLD_SECONDS - 2;
+      assertClean(run(), HealthStatus.UP);
+      skew = CapabilityProbe.CLOCK_SKEW_THRESHOLD_SECONDS + 10;
+      HealthCheckOutcome outcome = run();
+      assertClean(outcome, HealthStatus.DEGRADED);
+      assertThat((Long) outcome.getDetails().getValues().get("clockSkewSeconds")).isGreaterThan(900);
+   }
+
+
+
+   /*******************************************************************************
+    ** Expected failures carry only a fixed kind even when errors contain secrets.
+    *******************************************************************************/
+   @ParameterizedTest
+   @ValueSource(strings = { "auth", "tls", "dns", "clock", "connection" })
+   void downAndNoSecrets(String kind) throws Exception
+   {
+      failure = switch(kind)
+      {
+         case "auth" -> (S3Exception) S3Exception.builder().statusCode(403).message(SECRET).build();
+         case "tls" -> new IllegalStateException(SECRET, new SSLHandshakeException(SECRET));
+         case "dns" -> new IllegalStateException(SECRET, new UnknownHostException(SECRET));
+         case "clock" -> (S3Exception) S3Exception.builder().statusCode(403).message(SECRET)
+            .awsErrorDetails(AwsErrorDetails.builder().errorCode("RequestTimeTooSkewed").build()).build();
+         default -> new IllegalStateException(SECRET);
+      };
+      skew = 1000;
+      HealthCheckOutcome outcome = run();
+      assertClean(outcome, HealthStatus.DOWN);
+      assertThat(outcome.getDetails().getValues()).containsEntry("problemKind", kind);
+      assertThat(Db.asSystem(() -> Db.query(ObjectStorageTables.CONNECTION_CAPABILITY))).isEmpty();
+   }
+
+
+
+   /*******************************************************************************
+    ** An empty configuration cannot establish connectivity.
+    *******************************************************************************/
+   @Test
+   void noBuckets() throws Exception
+   {
+      config.getBuckets().clear();
+      assertClean(run(), HealthStatus.UNKNOWN);
+      assertThat(calls).isEmpty();
+   }
+
+
+
+   /*******************************************************************************
+    ** Client initialization errors also cannot expose poisoned configuration.
+    *******************************************************************************/
+   @Test
+   @SuppressWarnings("unchecked")
+   void clientInitializationFailure() throws Exception
+   {
+      Field connections = ObjectStorageRuntime.class.getDeclaredField("CONNECTIONS");
+      connections.setAccessible(true);
+      ((Map<String, ConnectionRuntime>) connections.get(null)).put("PRIMARY", new ConnectionRuntime(config.getConnections().getFirst())
+      {
+         /*******************************************************************************
+          ** Simulate a credential or endpoint failure before any request is sent.
+          *******************************************************************************/
+         @Override
+         public synchronized S3Client client()
+         {
+            throw (new IllegalStateException(SECRET));
+         }
+      });
+      assertClean(run(), HealthStatus.DOWN);
+      assertThat(calls).isEmpty();
+   }
+
+
+
+   /*******************************************************************************
+    ** Expired checks perform no I/O; a deadline reached mid-probe stops SDK calls.
+    *******************************************************************************/
+   @Test
+   void deadlines() throws Exception
+   {
+      assertClean(check().getCheckInstance().run(input().withDeadline(Instant.now().minusSeconds(1))), HealthStatus.DOWN);
+      assertThat(calls).isEmpty();
+      expireAfterList = true;
+      deadline = Instant.now().plusMillis(500);
+      HealthCheckOutcome outcome = check().getCheckInstance().run(input().withDeadline(deadline));
+      assertClean(outcome, HealthStatus.DOWN);
+      assertThat(outcome.getDetails().getValues()).containsEntry("problemKind", "timeout");
+      assertThat(calls).containsExactly("listObjectsV2");
+   }
+
+
+
+   /*******************************************************************************
+    ** Rate limiting also respects the budget, both for slots and depleted tokens.
+    *******************************************************************************/
+   @Test
+   void rateLimiterDeadline() throws Exception
+   {
+      RateLimiter limiter = ObjectStorageRuntime.connection("PRIMARY").getRateLimiter();
+      try(var held = limiter.acquire())
+      {
+         assertClean(check().getCheckInstance().run(input().withDeadline(Instant.now().plusMillis(50))), HealthStatus.DOWN);
+         assertThat(calls).isEmpty();
+      }
+      RateLimiter onePerSecond = new RateLimiter(1, 1);
+      onePerSecond.acquire().close();
+      assertThatThrownBy(() -> onePerSecond.acquire(Instant.now().plusMillis(20))).isInstanceOf(IllegalStateException.class);
+      try(var permit = onePerSecond.acquire(Instant.now().plusSeconds(2)))
+      {
+         assertThat(permit).isNotNull();
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Fetch checks through the registry, exercising the supplier code reference.
+    *******************************************************************************/
+   private HealthCheckMetaData check() throws Exception
+   {
+      return (HealthChecksMetaData.ofOrWithNew(instance).resolveAll(instance).getFirst());
+   }
+
+
+
+   /*******************************************************************************
+    ** Default bounded input.
+    *******************************************************************************/
+   private HealthCheckInput input()
+   {
+      return (new HealthCheckInput().withQInstance(instance).withDeadline(Instant.now().plusSeconds(20)));
+   }
+
+
+
+   /*******************************************************************************
+    ** Run the registered check.
+    *******************************************************************************/
+   private HealthCheckOutcome run() throws Exception
+   {
+      return (check().getCheckInstance().run(input()));
+   }
+
+
+
+   /*******************************************************************************
+    ** HealthRunner.sanitize is package-private; enforce cleanliness on raw output.
+    *******************************************************************************/
+   private void assertClean(HealthCheckOutcome outcome, HealthStatus expected)
+   {
+      assertThat(outcome.getStatus()).isEqualTo(expected);
+      assertThat(outcome.getMessage() + outcome.getDetails().toJson()).doesNotContain(SECRET, "private.example", "private-bucket");
+      assertThat(outcome.getMessage()).doesNotContain("\n", "\r");
+   }
+}
@@ -23,10 +23,14 @@ package com.kingsrook.qbits.objectstorage.index;

 import java.sql.Connection;
 import java.sql.DriverManager;
+import java.sql.ResultSet;
 import java.sql.Statement;
+import java.util.ArrayList;
 import java.util.List;
 import java.util.Map;
 import java.util.Set;
+import java.util.TreeMap;
+import java.util.TreeSet;
 import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig;
 import com.kingsrook.qbits.objectstorage.api.FieldReferenceProvider;
 import com.kingsrook.qbits.objectstorage.api.ObjectStorage;
@@ -46,6 +50,11 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
 import com.kingsrook.qqq.backend.core.model.data.QRecord;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
 import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
+import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
+import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
 import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
 import com.kingsrook.qqq.backend.core.model.session.QSession;
 import com.kingsrook.qqq.backend.core.model.session.QUser;
@@ -270,4 +279,203 @@ class PostgresIndexTest
         .withSecurityKeyValue(ObjectStorageTables.KEY_TYPE_BUCKET, ConfigSync.bucketId("indexed")));
      assertThat(new QueryAction().execute(new QueryInput(ObjectStorageTables.OBJECT)).getRecords()).extracting(r -> r.getValueString("objectKey")).containsExactly("d/ref.txt");
   }
+
+
+
+   /** Relations (child.column -> parent) that are deliberately not database foreign keys,
+    ** each with its reason. Empty: every join and table possible value is enforced. */
+   private static final Map<String, String> FK_EXEMPTIONS = Map.of();
+
+
+
+   /*******************************************************************************
+    ** The guardrail: every join and table possible-value field between the qbit's
+    ** tables is a foreign key in the generated schema (or exempt, with a reason),
+    ** with the declared ON DELETE rule, and every foreign key column leads an index.
+    ** A new relation without a foreign key fails here.
+    *******************************************************************************/
+   @Test
+   void everyRelationIsAForeignKeyWithAnIndex() throws Exception
+   {
+      Set<String> expected = new TreeSet<>();
+      for(QJoinMetaData join : QContext.getQInstance().getJoins().values())
+      {
+         if(ObjectStorageTables.ALL.contains(join.getLeftTable()) && ObjectStorageTables.ALL.contains(join.getRightTable()) && join.getType() == JoinType.ONE_TO_MANY)
+         {
+            for(JoinOn on : join.getJoinOns())
+            {
+               expected.add(join.getRightTable() + "." + ObjectStorageSchemaColumns.of(on.getRightField()) + " -> " + join.getLeftTable());
+            }
+         }
+      }
+      for(String tableName : ObjectStorageTables.ALL)
+      {
+         for(QFieldMetaData field : QContext.getQInstance().getTable(tableName).getFields().values())
+         {
+            QPossibleValueSource pvs = field.getPossibleValueSourceName() == null ? null : QContext.getQInstance().getPossibleValueSource(field.getPossibleValueSourceName());
+            if(pvs != null && pvs.getType() == QPossibleValueSourceType.TABLE && ObjectStorageTables.ALL.contains(pvs.getTableName()))
+            {
+               expected.add(tableName + "." + ObjectStorageSchemaColumns.of(field.getName()) + " -> " + pvs.getTableName());
+            }
+         }
+      }
+      assertThat(expected).as("the qbit declares relations").hasSizeGreaterThan(10);
+
+      Map<String, String> actual  = new TreeMap<>();
+      List<String>        noIndex = new ArrayList<>();
+      try(Connection c = DriverManager.getConnection(postgres.getJdbcUrl(), "qqq", "qqq"); Statement st = c.createStatement();
+         ResultSet rs = st.executeQuery("""
+            SELECT cl.relname AS child, a.attname AS col, pcl.relname AS parent, con.confdeltype AS rule,
+               EXISTS (SELECT 1 FROM pg_index i WHERE i.indrelid = con.conrelid AND i.indkey[0] = con.conkey[1]) AS indexed
+            FROM pg_constraint con
+            JOIN pg_class cl ON cl.oid = con.conrelid
+            JOIN pg_class pcl ON pcl.oid = con.confrelid
+            JOIN pg_attribute a ON a.attrelid = con.conrelid AND a.attnum = con.conkey[1]
+            WHERE con.contype = 'f'"""))
+      {
+         while(rs.next())
+         {
+            String key = rs.getString("child") + "." + rs.getString("col") + " -> " + rs.getString("parent");
+            actual.put(key, rs.getString("rule"));
+            if(!rs.getBoolean("indexed"))
+            {
+               noIndex.add(key);
+            }
+         }
+      }
+
+      List<String> missing = new ArrayList<>();
+      for(String relation : expected)
+      {
+         if(!actual.containsKey(relation) && !FK_EXEMPTIONS.containsKey(relation))
+         {
+            missing.add(relation);
+         }
+      }
+      assertThat(missing).as("relations without a database foreign key (add one to ObjectStorageSchema, or an exemption with a reason)").isEmpty();
+      assertThat(noIndex).as("foreign key columns that do not lead an index").isEmpty();
+
+      Map<String, String> rules = Map.of("NO ACTION", "a", "RESTRICT", "r", "CASCADE", "c", "SET NULL", "n");
+      for(ObjectStorageSchema.ForeignKey fk : ObjectStorageSchema.foreignKeys())
+      {
+         String key = fk.table() + "." + ObjectStorageSchemaColumns.of(fk.field()) + " -> " + fk.parent();
+         assertThat(actual.get(key)).as(fk.name() + " ON DELETE").isEqualTo(rules.get(fk.onDelete()));
+         assertThat(fk.name().length()).as(fk.name()).isLessThanOrEqualTo(63);
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Upgrading a schema created before the foreign keys: orphans are recorded in
+    ** qqq_fk_quarantine and cleared (nullable) or deleted (required), the keys are
+    ** added, and running the statements again changes nothing.
+    *******************************************************************************/
+   @Test
+   void foreignKeyUpgradeQuarantinesOrphansAndIsIdempotent() throws Exception
+   {
+      Integer bucketId = ConfigSync.bucketId("indexed");
+      try(Connection c = DriverManager.getConnection(postgres.getJdbcUrl(), "qqq", "qqq"); Statement st = c.createStatement())
+      {
+         for(ObjectStorageSchema.ForeignKey fk : ObjectStorageSchema.foreignKeys())
+         {
+            st.execute("ALTER TABLE " + fk.table() + " DROP CONSTRAINT " + fk.name());
+         }
+         st.execute("INSERT INTO object_storage_object (bucket_id, object_key, folder_id) VALUES (" + bucketId + ", 'kept.txt', 424242)");
+         st.execute("INSERT INTO object_storage_protection_rule (bucket_id, prefix_glob, mode) VALUES (434343, 'x/*', 'NO_DELETE')");
+         st.execute("INSERT INTO object_storage_protection_rule (bucket_id, prefix_glob, mode) VALUES (" + bucketId + ", 'y/*', 'NO_DELETE')");
+
+         ObjectStorageQBitConfig forSchema = new ObjectStorageQBitConfig().withBackendName("rdbms");
+         for(int pass = 0; pass < 2; pass++)
+         {
+            for(String sql : ObjectStorageSchema.postgresStatements(forSchema))
+            {
+               st.execute(sql);
+            }
+         }
+
+         try(ResultSet rs = st.executeQuery("SELECT table_name, column_name, missing_parent_id, action, row_data FROM qqq_fk_quarantine ORDER BY id"))
+         {
+            List<String> rows = new ArrayList<>();
+            while(rs.next())
+            {
+               rows.add(rs.getString(1) + "." + rs.getString(2) + "=" + rs.getString(3) + " " + rs.getString(4) + (rs.getString(5).contains("x/*") ? " with row" : ""));
+            }
+            assertThat(rows).as("recorded once, although the statements ran twice")
+               .containsExactly("object_storage_protection_rule.bucket_id=434343 DELETED with row", "object_storage_object.folder_id=424242 NULLED");
+         }
+         try(ResultSet rs = st.executeQuery("SELECT folder_id FROM object_storage_object WHERE object_key = 'kept.txt'"))
+         {
+            assertThat(rs.next()).isTrue();
+            assertThat(rs.getObject(1)).isNull();
+         }
+         try(ResultSet rs = st.executeQuery("SELECT prefix_glob FROM object_storage_protection_rule ORDER BY id"))
+         {
+            assertThat(rs.next()).isTrue();
+            assertThat(rs.getString(1)).isEqualTo("y/*");
+            assertThat(rs.next()).isFalse();
+         }
+         try(ResultSet rs = st.executeQuery("SELECT COUNT(*) FROM pg_constraint WHERE contype = 'f' AND conname LIKE 'fk_object_storage_%'"))
+         {
+            rs.next();
+            assertThat(rs.getInt(1)).isEqualTo(ObjectStorageSchema.foreignKeys().size());
+         }
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** The delete paths the qbit uses still work with the keys: removing an object
+    ** row (scan removal, hard-delete write-through) takes its versions and leaves
+    ** its references pointing nowhere, and a bucket row with an audit history or
+    ** rules cannot be deleted.
+    *******************************************************************************/
+   @Test
+   void deletesFollowTheForeignKeyRules() throws Exception
+   {
+      rawPut("f/one.txt", 3);
+      ObjectIndex.scan("indexed");
+      Integer bucketId = ConfigSync.bucketId("indexed");
+      Integer objectId = Db.first(ObjectStorageTables.OBJECT, "objectKey", "f/one.txt").getValueInteger("id");
+      try(Connection c = DriverManager.getConnection(postgres.getJdbcUrl(), "qqq", "qqq"); Statement st = c.createStatement())
+      {
+         st.execute("INSERT INTO object_storage_object_version (bucket_id, object_id, object_key) VALUES (" + bucketId + ", " + objectId + ", 'f/one.txt')");
+      }
+      Db.insert("media", List.of(new QRecord().withValue("storageKey", "f/one.txt")));
+      ReferenceScan.run("indexed");
+      assertThat(Db.first(ObjectStorageTables.OBJECT_REFERENCE, "keyOrPrefix", "f/one.txt").getValueInteger("objectId")).isEqualTo(objectId);
+
+      try(S3Client raw = target.rawClient())
+      {
+         raw.deleteObject(DeleteObjectRequest.builder().bucket(target.indexBucket).key("f/one.txt").build());
+      }
+      assertThat(ObjectIndex.scan("indexed").status()).isEqualTo("OK");
+      assertThat(Db.first(ObjectStorageTables.OBJECT, "objectKey", "f/one.txt")).isNull();
+      assertThat(Db.query(ObjectStorageTables.OBJECT_VERSION, "objectId", objectId)).as("versions cascade").isEmpty();
+      assertThat(Db.first(ObjectStorageTables.OBJECT_REFERENCE, "keyOrPrefix", "f/one.txt").getValueInteger("objectId")).as("reference set null").isNull();
+
+      try(Connection c = DriverManager.getConnection(postgres.getJdbcUrl(), "qqq", "qqq"); Statement st = c.createStatement())
+      {
+         st.execute("INSERT INTO object_storage_operation (bucket_id, bucket_alias, action) VALUES (" + bucketId + ", 'indexed', 'PUT')");
+         org.assertj.core.api.Assertions.assertThatThrownBy(() -> st.execute("DELETE FROM object_storage_bucket WHERE id = " + bucketId))
+            .hasMessageContaining("fk_object_storage_operation_bucket_id");
+      }
+   }
+
+
+
+   /*******************************************************************************
+    ** Column names as the schema derives them.
+    *******************************************************************************/
+   private static final class ObjectStorageSchemaColumns
+   {
+      /*******************************************************************************
+       ** The snake_case column of a field.
+       *******************************************************************************/
+      static String of(String field)
+      {
+         return (com.kingsrook.qqq.backend.core.instances.QInstanceEnricher.inferBackendName(field));
+      }
+   }
 }
@@ -0,0 +1,72 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.kingsrook.qbits.objectstorage.internal;
+
+
+import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig;
+import com.kingsrook.qbits.objectstorage.testsupport.StorageTestSupport;
+import com.kingsrook.qqq.backend.core.context.QContext;
+import com.kingsrook.qqq.backend.core.model.metadata.security.QSecurityKeyType;
+import com.kingsrook.qqq.backend.core.model.session.QSession;
+import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.Test;
+import static org.assertj.core.api.Assertions.assertThat;
+
+
+/*******************************************************************************
+ ** Db.asSystem runs work as the core system user. Token-based authentication
+ ** modules (OAuth2, Auth0) accept a QSystemUserSession without a token and
+ ** reject any other tokenless session, so the class matters, not just the
+ ** permissions.
+ *******************************************************************************/
+class DbSystemSessionTest
+{
+   /*******************************************************************************
+    ** Reset the runtime and context after each test.
+    *******************************************************************************/
+   @AfterEach
+   void tearDown()
+   {
+      StorageTestSupport.tearDown();
+   }
+
+
+
+   /*******************************************************************************
+    ** The work sees a QSystemUserSession with every all-access key, and the
+    ** caller's session is restored afterwards.
+    *******************************************************************************/
+   @Test
+   void testAsSystemUsesCoreSystemUserSession() throws Exception
+   {
+      ObjectStorageQBitConfig config = new ObjectStorageQBitConfig().withBackendName(StorageTestSupport.MEMORY).withBucketScoping(false);
+      StorageTestSupport.setUp(config, instance -> instance.addSecurityKeyType(new QSecurityKeyType().withName("tenant").withAllAccessKeyName("tenantAllAccess")));
+      QSession caller = QContext.getQSession();
+
+      QSession inside = Db.asSystem(QContext::getQSession);
+
+      assertThat(inside).isInstanceOf(QSystemUserSession.class);
+      assertThat(inside.hasPermission("anything")).isTrue();
+      assertThat(inside.hasSecurityKeyValue("tenantAllAccess", true)).isTrue();
+      assertThat(QContext.getQSession()).isSameAs(caller);
+   }
+}
@@ -218,6 +218,11 @@ public class DefaultsChangelogTest
      {
         for(String sql : ObjectStorageSchema.postgresStatements(config()))
         {
+            if(sql.startsWith("DO $$"))
+            {
+               // the foreign-key blocks are PL/pgSQL (Postgres only); PostgresIndexTest runs them on real Postgres
+               continue;
+            }
            statement.execute(sql);
         }
      }
@@ -129,6 +129,25 @@ class TablesAndConfigTest



+   /*******************************************************************************
+    ** The record-page child lists filter on the parent id alone (folder ->
+    ** objects, object -> versions and references), so the generated schema
+    ** gives each of those columns its own index, appended so existing generated
+    ** index names keep their numbers.
+    *******************************************************************************/
+   @Test
+   void testChildListColumnsAreIndexed()
+   {
+      String schema = String.join("\n", com.kingsrook.qbits.objectstorage.model.ObjectStorageSchema.postgresStatements(new ObjectStorageQBitConfig().withBackendName("rdbms")));
+      assertThat(schema)
+         .contains("ix_object_storage_object_1 ON object_storage_object (bucket_id, folder_id)")
+         .contains("ix_object_storage_object_7 ON object_storage_object (folder_id)")
+         .contains("ix_object_storage_object_version_2 ON object_storage_object_version (object_id)")
+         .contains("ix_object_storage_object_reference_3 ON object_storage_object_reference (object_id)");
+   }
+
+
+
   /*******************************************************************************
    ** Owner naming rules: every table is lowercase snake_case with the full qbit
    ** name as prefix; no table has a field that could hold a secret value; the nav
Author	SHA1	Message	Date
james.maesandClaude Opus 5.5	eb04621d4e	fix(help): the help bundle carries the artifact version (Maven-filtered) instead of a hardcoded SNAPSHOT On release branches the bundle's version and the producer's differed, so the bundle test that asserts they match failed, and released bundles recorded SNAPSHOT as their provenance. help/*.yaml is now filtered: version: ${project.version}. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 22:07:58 -05:00
james.maes	38e74cc07f	Merge pull request #17 from QRun-IO/feature/health-checks feat(health): register object storage connection checks with core central health	2026-10-14 21:27:01 -05:00
james.maesandClaude Opus 5.5	8aa4511839	Merge origin/develop into feature/health-checks; pin QQQ 4.1.0-RC.8 RC.8 ships the central health kernel (QRun-IO/qqq#1207). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 21:18:29 -05:00
james.maes	9b2158bf97	Merge pull request #24 from QRun-IO/feature/help-bundle feat(help): qbit-object-storage help bundle via QQQ help bundles, with coverage gate	2026-10-14 20:45:30 -05:00
james.maesandClaude Opus 5.5	d00c230140	feat(help): ship the qbit-object-storage help bundle through QQQ help bundles, with coverage gate Host-neutral help (deployment-specific facts removed), registered with HelpBundles, gated by HelpBundleAssertions. QQQ 4.1.0-RC.7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 20:37:04 -05:00
james.maes	11f1fbff3d	Merge pull request #23 from QRun-IO/feature/fix-fk-dbms-postgresql Fix develop CI: skip Postgres-only FK DO blocks in H2 DefaultsChangelogTest	2026-10-14 17:51:25 -05:00
james.maesandClaude Opus 5.5	2964b91437	test: skip Postgres-only FK blocks when building the H2 schema in DefaultsChangelogTest The FK changesets from #20 are PL/pgSQL DO blocks; H2 cannot run them, which broke #21's DefaultsChangelogTest. FKs stay real on Postgres (PostgresIndexTest covers them). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 17:43:59 -05:00
james.maes	0b6ea1914a	Merge pull request #20 from QRun-IO/feature/foreign-keys feat(schema): foreign keys on every relation between the qbit tables	2026-10-14 17:36:55 -05:00
james.maes	deccc44fd2	Merge pull request #19 from QRun-IO/feature/object-id-indexes fix(schema): index the record-page child columns folder_id and object_id	2026-10-14 17:36:48 -05:00
james.maes	4498eac60b	Merge pull request #18 from QRun-IO/feature/system-session-qsystemusersession fix(internal): run system work as a QSystemUserSession	2026-10-14 17:36:42 -05:00
james.maesandClaude Opus 5.5	1ef599f695	feat(schema): foreign keys on every relation between the qbit tables ObjectStorageSchema now declares a foreign key for each join and table possible value (fk_<table>_<column>): capabilities, folders, objects, versions, references and usage cascade from their bucket or object; object.folder_id and object_reference.object_id are set null; a bucket's connection, its header and protection rules and the operation log restrict. Each key is added by an idempotent DO block that first records orphans in qqq_fk_quarantine and clears (nullable) or deletes (required) them, so the same statements upgrade an existing schema. Three new indexes back keys that had none. PostgresIndexTest gains the guardrail (every relation has a database foreign key with its rule, every key column leads an index), the upgrade path with orphans run twice, and the delete paths the scan uses. Builds on #19 (feature/object-id-indexes). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 16:21:04 -05:00
james.maesandClaude Opus 5.5	ed1bbcfdf9	fix(schema): index the record-page child columns folder_id and object_id The folder -> objects, object -> versions and object -> references child lists filter on the parent id alone, but every existing index on those tables leads with bucket_id. The generated schema now adds ix_object_storage_object_7 (folder_id), ix_object_storage_object_version_2 (object_id) and ix_object_storage_object_reference_3 (object_id). They are appended, so existing generated index names keep their numbers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 15:25:21 -05:00
james.maesandClaude Opus 5.5	7f8e926f0c	fix(internal): run system work as a QSystemUserSession Db.asSystem built an anonymous QSession subclass with every permission but no token. Token-based authentication modules (OAuth2, Auth0) accept a QSystemUserSession without a token and reject any other tokenless session, so on such hosts every system read and write failed in session validation (JWTDecodeException: The token is null). ConfigSync never mirrored the code-defined connections and buckets, and the operation log, object index, capabilities and protection/header rule scoping silently did nothing. Use core QSystemUserSession(instance), which carries the same permissions and all-access keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 15:12:32 -05:00
james.maesandClaude Opus 5.5	fd81c88f32	docs: object storage health checks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 13:39:40 -05:00
james.maesandClaude Opus 5.5	63c441e6cd	feat(health): register object storage connection checks with core central health Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 13:39:39 -05:00
james.maesandClaude Opus 5.5	b13122f1b0	feat(health): deadline-aware read-only capability probe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-14 13:39:38 -05:00