Compare commits

...
Author SHA1 Message Date
james.maesandClaude Opus 5.5 d00c230140 feat(help): ship the qbit-object-storage help bundle through QQQ help bundles, with coverage gate
Host-neutral help (deployment-specific facts removed), registered with
HelpBundles, gated by HelpBundleAssertions. QQQ 4.1.0-RC.7.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 20:37:04 -05:00
james.maes 11f1fbff3d Merge pull request #23 from QRun-IO/feature/fix-fk-dbms-postgresql
Fix develop CI: skip Postgres-only FK DO blocks in H2 DefaultsChangelogTest
2026-10-14 17:51:25 -05:00
james.maesandClaude Opus 5.5 2964b91437 test: skip Postgres-only FK blocks when building the H2 schema in DefaultsChangelogTest
The FK changesets from #20 are PL/pgSQL DO blocks; H2 cannot run them, which broke
#21's DefaultsChangelogTest. FKs stay real on Postgres (PostgresIndexTest covers them).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:43:59 -05:00
james.maes 0b6ea1914a Merge pull request #20 from QRun-IO/feature/foreign-keys
feat(schema): foreign keys on every relation between the qbit tables
2026-10-14 17:36:55 -05:00
james.maes deccc44fd2 Merge pull request #19 from QRun-IO/feature/object-id-indexes
fix(schema): index the record-page child columns folder_id and object_id
2026-10-14 17:36:48 -05:00
james.maes 4498eac60b Merge pull request #18 from QRun-IO/feature/system-session-qsystemusersession
fix(internal): run system work as a QSystemUserSession
2026-10-14 17:36:42 -05:00
james.maes 632c2cfa2d Merge pull request #21 from QRun-IO/feature/tables-populated-defaults
feat: guided empty states, Setup card, starter header rule and bucket presign overrides
2026-10-14 17:19:50 -05:00
james.maesandClaude Opus 5.5 e9fd0cef4c fix: presign row limits are a cached lower cap, no config mutation; setup card reads runtime config; empty rollback
- effective limit is min(code, row); row 0/null/higher is ignored; config never mutated
- applied to presignGet/Put/Post, previewUrl and the download-link action; row read cached 15 s
- Setup card takes connections and buckets from the runtime config and counts rules as system
- seed changeset rollback is empty; test that a host rule survives rollback
- host help kept test, stand-in marker prose in the bundle

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 17:10:53 -05:00
james.maesandClaude Opus 5.5 25e57e0315 feat: guided empty states, Setup card, starter header rule and bucket presign overrides
Help bundle for all 12 tables and the app (ObjectStorageHelp), Setup card on
the home, disabled default-mutable header rule via a jar changelog tested
through real Liquibase on H2, tables guardrail test. The bucket row's presign
limits are now read at runtime (ConfigSync.withBucketOverrides). The version
table is documented as reserved and not used yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:57:43 -05:00
james.maesandClaude Opus 5.5 1ef599f695 feat(schema): foreign keys on every relation between the qbit tables
ObjectStorageSchema now declares a foreign key for each join and table
possible value (fk_<table>_<column>): capabilities, folders, objects,
versions, references and usage cascade from their bucket or object;
object.folder_id and object_reference.object_id are set null; a bucket's
connection, its header and protection rules and the operation log
restrict. Each key is added by an idempotent DO block that first records
orphans in qqq_fk_quarantine and clears (nullable) or deletes (required)
them, so the same statements upgrade an existing schema. Three new
indexes back keys that had none.

PostgresIndexTest gains the guardrail (every relation has a database
foreign key with its rule, every key column leads an index), the
upgrade path with orphans run twice, and the delete paths the scan uses.

Builds on #19 (feature/object-id-indexes).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 16:21:04 -05:00
james.maesandClaude Opus 5.5 ed1bbcfdf9 fix(schema): index the record-page child columns folder_id and object_id
The folder -> objects, object -> versions and object -> references child
lists filter on the parent id alone, but every existing index on those
tables leads with bucket_id. The generated schema now adds
ix_object_storage_object_7 (folder_id), ix_object_storage_object_version_2
(object_id) and ix_object_storage_object_reference_3 (object_id). They are
appended, so existing generated index names keep their numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 15:25:21 -05:00
24 changed files with 2742 additions and 18 deletions

No files matched your search

+10 -1
View File
@@ -5,6 +5,13 @@ All notable changes to this project are documented here. The format follows Keep
## [0.1.0] - Unreleased
### Added
- Expanded the host-neutral in-app help bundle to cover storage fields, actions, widgets and possible values, with live references and a `HelpBundleAssertions` coverage gate. Registered it for QQQ startup sync and HOST overrides; retained inline table/app help for hosts without help-content tables.
- Every table explains itself: a help bundle (`help/qbit-object-storage.yaml`, QQQ help-bundle format) says what each `object_storage_*` table is for and how its rows get created; it shows on the table screen and when the table is empty. Applied inline by `ObjectStorageHelp` for hosts without help-content tables; a host's own help is kept.
- A Setup card on the Storage home (connections, buckets, header rules, protection rules) with the state and next step of each, resolved the way the runtime resolves them.
- A starter changelog `db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql`: a disabled `default-mutable` header rule. Insert-only, run through real Liquibase in `DefaultsChangelogTest`.
- `ObjectStorageTablesGuardrailTest` (each table is used at runtime, seeded or explained, and has help).
- The presign limits on the bucket row (`maxPresignGetSeconds`, `maxPresignPutSeconds`) are now read at runtime (`ConfigSync.effectiveMaxPresignGetSeconds` / `Put`) as a lower cap: the effective limit is the smaller of the code config and the row. It applies to presigned GET, PUT, POST, previews and the Create download link action. Previously the row value was never read.
- `object_storage_object_version` is documented as reserved and not used yet (the indexer lists current versions only).
- `StorageBucket.maxUploadBytes()` in the `api` package: the bucket's upload limit in bytes (null when none), so other qbits need not call `config()`, which returns an internal type.
- **The service (P1).** `ObjectStorage.bucket(alias)` and `ObjectStorage.asSystem(caller, runId)`: put (multipart above the part size), get with ranges, head, copy (in parts for very large objects), header and metadata changes, delete (delete marker, private trash, or permanent), restore, list, presigned GET, PUT and POST, public URLs, tags. Key building and URL encoding, the header policy (`object_storage_header_rule`), `PublicUrlResolver`, `StorageChangeListener`, `StorageReferenceProvider`, `StorageAlertSink`.
- Configuration in code, upserted into visible `object_storage_*` tables; secret references with an allow-list; per-connection rate limits; per-bucket and per-connection record security locks.
@@ -14,12 +21,14 @@ All notable changes to this project are documented here. The format follows Keep
- **The index (P2).** `object_storage_object` and `object_storage_folder` kept in step with each indexed bucket: resumable full scan with a mass-removal safety stop, HEAD backlog, write-through on every service write, soft-deleted rows, folder rollups. Orphaned and missing objects from `FieldReferenceProvider`, daily usage snapshots, quota and freshness alerts, the nightly job.
- **The Storage app (P3, P4).** Admin -> Storage with a responsive home; record screens with sections, child lists and a thumbnail preview; actions Rescan bucket, Check configuration, Refresh from bucket, Apply header policy (replaces app-specific backfills), Change headers and metadata, Edit tags, Copy, Move or rename, Delete, Restore, Create download link, Create upload link, folder rename and delete, Upload files, Test connection; every one a standard Review and Result with a bulk cap.
- `ObjectStorageSchema`: PostgreSQL DDL generated from the table metadata.
- Foreign keys on every relation between the qbit tables (`fk_<table>_<column>`), generated by `ObjectStorageSchema`: owned observed state (capabilities, folders, objects, versions, usage) cascades from its bucket or object, `object.folder_id` and `object_reference.object_id` are set null, and a bucket's connection, header and protection rules and operation log restrict. Each key is added by an idempotent block that first records rows whose parent is gone in `qqq_fk_quarantine` and clears (nullable column) or deletes (required column) them, so it also upgrades an existing schema. New indexes `ix_object_storage_bucket_1` (connection_id), `ix_object_storage_header_rule_1` and `ix_object_storage_protection_rule_1` (bucket_id) back the keys. Upgrade: run the generated statements again (or the host's copy of them).
- Conformance suite against MinIO and Garage (Testcontainers), service, index and action tests, and the index and actions on PostgreSQL.
### Changed
- Raised the QQQ BOM baseline to released `4.1.0-RC.7` for the core help-bundle registry and coverage assertions.
- Raised the QQQ BOM baseline to released `4.1.0-RC.5` and delegated S3 client construction to core `S3ClientFactory.buildClient`. Removed local HTTP client and trust-manager implementations; existing endpoint, region, addressing, checksum, TLS/CA, credentials and timeout settings remain supported.
- Clients remain owned by each connection runtime and are rebuilt on reset/reinstall. Retry behavior now follows core/SDK defaults instead of forcing `ADAPTIVE_V2`; hosts can select the SDK retry mode through `AWS_RETRY_MODE` or `aws.retryMode`.
- Retained local presigning (including a separate endpoint/key). A request-credentials bridge preserves static session tokens because RC.5 core metadata/factory has no session-token field or credentials-provider hook. This bridge and the existing `${file.PATH}` resolver are marked for removal by RC.7 under issue #10.
- Retained local presigning (including a separate endpoint/key). A request-credentials bridge preserves static session tokens because RC.5 core metadata/factory has no session-token field or credentials-provider hook. This bridge and the existing `${file.PATH}` resolver remain necessary in released RC.7 (no session-token/provider hook or file-reference interpreter); their review deadline is now RC.8 under issue #10.
- Added the filesystem module with its unused SFTP/SSH stack excluded. Resolved dependencies otherwise only change QQQ RC.4 to RC.5 and jsoup 1.23.1 to 1.23.2; see [dependency details](docs/S3-FACTORY-MIGRATION.md).
### Notes
+27 -1
View File
@@ -15,7 +15,7 @@ S3-compatible storage as a capability for any QQQ app, and an admin app to run i
## Requirements
- Java 21.
- QQQ `4.1.0-RC.5` (released BOM), including `qqq-backend-module-filesystem`. `ConnectionRuntime` maps connection settings to core's `S3BackendMetaData` and uses `S3ClientFactory.buildClient`; core owns HTTP, TLS and checksum setup. The unused SFTP/SSH dependency is excluded. See the [dependency diff](docs/S3-FACTORY-MIGRATION.md).
- QQQ `4.1.0-RC.7` (released BOM), including `qqq-backend-module-filesystem`. `ConnectionRuntime` maps connection settings to core's `S3BackendMetaData` and uses `S3ClientFactory.buildClient`; core owns HTTP, TLS and checksum setup. The unused SFTP/SSH dependency is excluded. See the [dependency diff](docs/S3-FACTORY-MIGRATION.md).
- A QQQ backend for the qbit's tables (PostgreSQL is the intended store). `ObjectStorageSchema.postgresScript(config)` generates the DDL from the same metadata the app runs on, for your Liquibase or Flyway migration.
## Install
@@ -47,6 +47,20 @@ Connections and buckets come from this code only. Secrets are **references** (`$
Client settings take effect when the qbit configuration is reinstalled or the runtime is reset. Presigning keeps its separate public endpoint and credentials. Retry behavior now follows core/SDK defaults; hosts requiring adaptive retries can set `AWS_RETRY_MODE=adaptive` or `aws.retryMode=adaptive`. Static session tokens remain supported through a small local bridge pending core support.
## In-app help
The producer registers [help/qbit-object-storage.yaml](src/main/resources/help/qbit-object-storage.yaml)
as a `QBIT` bundle. QQQ syncs it when the host has the help-content tables installed;
registration is safe without them, and inline table/app help still explains empty lists.
The bundle version is the literal POM version; update both when releasing.
Hosts can register their own `layer: HOST` bundle with `HelpBundles.register(qInstance, "help/my-host.yaml")`.
Use the same entry key and role to override qbit text, keeping live `{{...}}` references where useful.
Hosts that rename the optional Storage app should provide an app entry for their configured name.
`ObjectStorageHelpBundleTest` checks all owned tables, fields, processes, apps, widgets and
possible-value sources using `HelpBundleAssertions`, including valid keys, live references and
plain summaries of at most 120 characters. It runs with `./mvnw verify`.
## Use
```java
@@ -78,6 +92,18 @@ Every write is a standard streamed ETL: **Review** (the dry run, with summary li
Permissions: `object_storage.view`, `.write`, `.delete`, `.admin` (admin implies the rest). Per-bucket scoping uses record security locks on the `object_storage_bucket` and `object_storage_connection` security key types (switch off with `withBucketScoping(false)` for a single-administrator app).
## Setup and starter defaults
The Storage home has a **Setup** card: connections and buckets (declared in code, synced into their tables), header rules, and protection rules, each with its state and next step. Every table has help (`help/qbit-object-storage.yaml`) saying what it is for and how its rows get created, which also shows when the table is empty.
The jar ships one optional seed changelog. After the changesets that create the tables (the script from `ObjectStorageSchema.postgresScript`), include it from your master changelog:
```yaml
- include: { file: db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql }
```
It is insert-only and inserts a **disabled** generic header rule, `default-mutable` (every key, `public, max-age=3600`, priority 1000), so nothing changes on deploy: enable and edit it when you want it. Edits survive re-runs. Protection rules are not seeded (a rule needs a bucket). Presign limits on the bucket row are read at runtime: `maxPresignGetSeconds` and `maxPresignPutSeconds` on the bucket row can lower the code config limits (the effective limit is the smaller of the two, cached up to 15 seconds); the row cannot raise them. `object_storage_object_version` is reserved and not used yet.
## Deployment notes
- **Uploads** go through the app server (100 MB per file by default, per bucket `maxUploadBytes`), using QQQ's standard file-upload field, so the app needs `uploadedFileArchiveTableName` set in its Javalin metadata, as for any file upload.
+8
View File
@@ -0,0 +1,8 @@
# Upgrading
## Unreleased
- Nothing is required. Help and the Setup card appear when the qbit is produced; the schema is unchanged.
- Optional: include `db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql` after the changesets that create the tables, to get a disabled generic header rule (`default-mutable`). It is insert-only and changes no behavior until you enable the rule.
- Behavior change: the presign limits on an `object_storage_bucket` row (`maxPresignGetSeconds`, `maxPresignPutSeconds`) are now read, as a lower cap only. Any bucket row whose presign values are lower than your code config shortens links after this upgrade (the config sync copied the code values into the row on first start, so a row someone lowered earlier counts). Check the values on your bucket rows; 0 or empty means no row limit, and a value above the code limit has no effect.
- `object_storage_object_version` stays empty: it is reserved.
+16 -2
View File
@@ -109,6 +109,20 @@
<scope>test</scope>
</dependency>
<!-- DefaultsChangelogTest runs the shipped seed changelog through real Liquibase on H2 (PostgreSQL mode). -->
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<version>2.3.232</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.liquibase</groupId>
<artifactId>liquibase-core</artifactId>
<version>4.33.0</version>
<scope>test</scope>
</dependency>
<!-- the Postgres index test: the real SQL of the object index (src/test only) -->
<dependency>
<groupId>com.kingsrook.qqq</groupId>
@@ -241,14 +255,14 @@
<profiles>
<!-- Built against qqq 4.1 (QBitMetaDataProducer). Until qbit-build-parent -->
<!-- imports the released 4.1 BOM, this profile is active by default and -->
<!-- imports qqq-bom-pom 4.1.0-RC.5 ahead of the parent's 4.0.0 BOM. -->
<!-- imports qqq-bom-pom 4.1.0-RC.7 ahead of the parent's 4.0.0 BOM. -->
<profile>
<id>qqq-snapshot</id>
<activation>
<activeByDefault>true</activeByDefault>
</activation>
<properties>
<qqq.snapshot.version>4.1.0-RC.6</qqq.snapshot.version>
<qqq.snapshot.version>4.1.0-RC.7</qqq.snapshot.version>
</properties>
<dependencyManagement>
<dependencies>
@@ -27,12 +27,14 @@ import com.kingsrook.qbits.objectstorage.app.StorageWidgets;
import com.kingsrook.qbits.objectstorage.internal.ConfigSync;
import com.kingsrook.qbits.objectstorage.internal.DefaultObjectHeaderPolicy;
import com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime;
import com.kingsrook.qbits.objectstorage.model.ObjectStorageHelp;
import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables;
import com.kingsrook.qbits.objectstorage.processes.ObjectStorageProcesses;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundles;
import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
@@ -193,6 +195,9 @@ public class ObjectStorageQBitProducer implements QBitMetaDataProducer<ObjectSto
///////////////////////////////////////////////////////////////////////////////
qInstance.getApp(qBitConfig.getParentAppName()).withChild(qInstance.getApp(qBitConfig.getAppName()));
}
// Registration is metadata-only; core skips startup sync without a helpContent table.
HelpBundles.register(qInstance, "help/qbit-object-storage.yaml");
ObjectStorageHelp.apply(qInstance, qBitConfig);
return (new MetaDataProducerMultiOutput());
}
}
@@ -36,6 +36,7 @@ import com.kingsrook.qbits.objectstorage.api.StorageException.Reason;
import com.kingsrook.qbits.objectstorage.config.BucketRole;
import com.kingsrook.qbits.objectstorage.config.StorageBucketConfig;
import com.kingsrook.qbits.objectstorage.internal.CapabilityStore;
import com.kingsrook.qbits.objectstorage.internal.ConfigSync;
import com.kingsrook.qbits.objectstorage.internal.ConnectionRuntime;
import com.kingsrook.qbits.objectstorage.internal.ErrorMapper;
import com.kingsrook.qbits.objectstorage.internal.IndexHook;
@@ -1028,7 +1029,7 @@ public final class StorageBucket
{
checkPermission("object_storage.view");
String key = StorageKeys.normalize(request.getKey());
Duration ttl = cap(request.getTtl(), Duration.ofMinutes(5), bucket.getMaxPresignGetSeconds());
Duration ttl = cap(request.getTtl(), Duration.ofMinutes(5), ConfigSync.effectiveMaxPresignGetSeconds(bucket));
countPresign();
GetObjectRequest.Builder b = GetObjectRequest.builder().bucket(name()).key(key);
if(StringUtils.hasContent(request.getDownloadAs()))
@@ -1061,7 +1062,7 @@ public final class StorageBucket
String key = StorageKeys.normalize(rawKey);
try
{
return (connection.presigner().presignGetObject(GetObjectPresignRequest.builder().signatureDuration(Duration.ofMinutes(5))
return (connection.presigner().presignGetObject(GetObjectPresignRequest.builder().signatureDuration(cap(null, Duration.ofMinutes(5), ConfigSync.effectiveMaxPresignGetSeconds(bucket)))
.getObjectRequest(GetObjectRequest.builder().bucket(name()).key(key).build()).build()).url());
}
catch(RuntimeException e)
@@ -1150,7 +1151,7 @@ public final class StorageBucket
{
checkPermission("object_storage.write");
String key = StorageKeys.normalize(request.getKey());
Duration ttl = cap(request.getTtl(), Duration.ofMinutes(15), bucket.getMaxPresignPutSeconds());
Duration ttl = cap(request.getTtl(), Duration.ofMinutes(15), ConfigSync.effectiveMaxPresignPutSeconds(bucket));
countPresign();
PutObjectRequest.Builder b = PutObjectRequest.builder().bucket(name()).key(key);
if(StringUtils.hasContent(request.getContentType()))
@@ -1192,7 +1193,7 @@ public final class StorageBucket
throw (new StorageException(Reason.UNSUPPORTED, "Presigned POST is not supported on this provider; verify the size after upload"));
}
String key = StorageKeys.normalize(request.getKey());
Duration ttl = cap(request.getTtl(), Duration.ofMinutes(15), bucket.getMaxPresignPutSeconds());
Duration ttl = cap(request.getTtl(), Duration.ofMinutes(15), ConfigSync.effectiveMaxPresignPutSeconds(bucket));
countPresign();
String[] creds = connection.presignCredentials();
PresignedPost post = PostPolicySigner.sign(creds[0], creds[1], creds[2], creds[3], connection.getConfig().effectiveAddressingStyle(), name(), key,
@@ -95,6 +95,7 @@ public final class StorageHome
LocalDate.ofInstant(now, ZoneOffset.UTC).minusDays(30))).withOrderBy(new com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy("snapshotDate")));
StringBuilder html = new StringBuilder("<div class=\"object-storage-home\" style=\"" + WRAP + "\">");
setupSection(html, tablePathPrefix);
bucketsSection(html, buckets, connections, caps, tablePathPrefix);
healthSection(html, buckets, tablePathPrefix);
recentSection(html, operations, tablePathPrefix);
@@ -116,6 +117,22 @@ public final class StorageHome
/*******************************************************************************
** 0. Setup: each configuration table, its state and the next step.
*******************************************************************************/
private static void setupSection(StringBuilder html, String prefix) throws QException
{
open(html, "Setup");
for(StorageSetupStatus.Step step : StorageSetupStatus.load())
{
html.append("<div style=\"").append(ROW).append("\"><strong><a href=\"").append(esc(prefix + "/" + step.table())).append("\">").append(esc(step.label())).append("</a></strong>: ")
.append(esc(step.state())).append(". <span style=\"").append(MUTED).append("\">").append(esc(step.next())).append("</span></div>");
}
close(html);
}
/*******************************************************************************
** 1. Buckets: one card per bucket, linking to its record. Cards flow into as
** many columns as fit, so the section reads at any width without scrolling.
@@ -0,0 +1,113 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.objectstorage.app;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig;
import com.kingsrook.qbits.objectstorage.internal.Db;
import com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime;
import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables;
import com.kingsrook.qqq.backend.core.exceptions.QException;
/*******************************************************************************
** The Setup card on the Storage home: each configuration table, its state and
** the next step. States use the same resolution the runtime uses: connections
** and buckets come from code (so "set up" means declared and synced), and the
** header policy is the host's own when the config names one, otherwise the
** enabled rows of object_storage_header_rule.
*******************************************************************************/
public final class StorageSetupStatus
{
public static final String SET_UP = "Set up";
public static final String NEEDS_SETUP = "Needs setup";
public static final String USING_DEFAULTS = "Using defaults";
public static final String OPTIONAL = "Optional";
/*******************************************************************************
** One line of the card.
*******************************************************************************/
public record Step(String label, String state, String next, String table)
{
}
/*******************************************************************************
** Utility class.
*******************************************************************************/
private StorageSetupStatus()
{
}
/*******************************************************************************
** Read the current state. Connections and buckets come from the runtime
** config (what the runtime uses, whoever is looking), and the rule counts
** are read as the system, so a bucket-scoped user is never shown a false
** "needs setup".
*******************************************************************************/
public static List<Step> load() throws QException
{
ObjectStorageQBitConfig config = ObjectStorageRuntime.getConfig();
int headerRules = Db.asSystem(() -> Db.query(ObjectStorageTables.HEADER_RULE, "enabled", true).size());
int protection = Db.asSystem(() -> Db.query(ObjectStorageTables.PROTECTION_RULE, "enabled", true).size());
return (steps(config.getConnections().size(), config.getBuckets().size(), headerRules, config.getHeaderPolicy() != null, protection));
}
/*******************************************************************************
** The steps, from the numbers (pure, for tests).
*******************************************************************************/
static List<Step> steps(int connections, int buckets, int enabledHeaderRules, boolean hostHeaderPolicy, int enabledProtectionRules)
{
List<Step> steps = new ArrayList<>();
steps.add(connections > 0
? new Step("Connections", SET_UP, connections + " declared in code and synced here; set limits and timeouts on the row", ObjectStorageTables.CONNECTION)
: new Step("Connections", NEEDS_SETUP, "Declare a connection in the qbit config (withConnection) and restart: endpoint, region and the names of the key variables", ObjectStorageTables.CONNECTION));
steps.add(buckets > 0
? new Step("Buckets", SET_UP, buckets + " declared in code and synced here; set the presign limits on the row", ObjectStorageTables.BUCKET)
: new Step("Buckets", NEEDS_SETUP, "Declare a bucket in the qbit config (withBucket) and restart", ObjectStorageTables.BUCKET));
if(hostHeaderPolicy)
{
steps.add(new Step("Header rules", SET_UP, "The host's own header policy (code) decides; these rows are not read", ObjectStorageTables.HEADER_RULE));
}
else if(enabledHeaderRules > 0)
{
steps.add(new Step("Header rules", SET_UP, enabledHeaderRules + " enabled; the first match by priority sets Cache-Control", ObjectStorageTables.HEADER_RULE));
}
else
{
steps.add(new Step("Header rules", USING_DEFAULTS, "No rule is enabled, so objects keep the headers they were written with. Enable the starter rule or add one", ObjectStorageTables.HEADER_RULE));
}
steps.add(enabledProtectionRules > 0
? new Step("Protection rules", SET_UP, enabledProtectionRules + " enabled", ObjectStorageTables.PROTECTION_RULE)
: new Step("Protection rules", OPTIONAL, "Add a rule to make a bucket prefix read-only or delete-proof", ObjectStorageTables.PROTECTION_RULE));
return (steps);
}
}
@@ -51,6 +51,9 @@ public final class ConfigSync
private static final Map<String, Integer> BUCKET_IDS = new ConcurrentHashMap<>();
private static final Map<String, Integer> CONNECTION_IDS = new ConcurrentHashMap<>();
private static final Map<String, RowLimits> ROW_LIMITS = new ConcurrentHashMap<>();
private static final long ROW_LIMIT_CACHE_MILLIS = 15_000L;
private static volatile boolean synced = false;
@@ -72,6 +75,7 @@ public final class ConfigSync
synced = false;
BUCKET_IDS.clear();
CONNECTION_IDS.clear();
ROW_LIMITS.clear();
}
@@ -273,4 +277,90 @@ public final class ConfigSync
}
return (base);
}
/*******************************************************************************
** What a bucket row says about presign limits (row values, cached briefly).
*******************************************************************************/
private record RowLimits(long readAt, Integer get, Integer put)
{
}
/*******************************************************************************
** The longest a presigned download link may last, in seconds: the smaller of
** the code config and the bucket row (Settings on object_storage_bucket). The
** row can only lower the code limit; a missing row, null or a value below one
** second means "no row limit". The config object is never modified.
*******************************************************************************/
public static int effectiveMaxPresignGetSeconds(StorageBucketConfig bucket)
{
return (lowest(bucket.getMaxPresignGetSeconds(), rowLimits(bucket.getAlias()).get()));
}
/*******************************************************************************
** The longest a presigned upload link may last, in seconds (see the GET twin).
*******************************************************************************/
public static int effectiveMaxPresignPutSeconds(StorageBucketConfig bucket)
{
return (lowest(bucket.getMaxPresignPutSeconds(), rowLimits(bucket.getAlias()).put()));
}
/*******************************************************************************
** Forget the cached row limits (after a row edit in a test; they also expire).
*******************************************************************************/
public static void clearRowLimits()
{
ROW_LIMITS.clear();
}
/*******************************************************************************
** The smaller of the code value (3600 when unset) and a usable row value.
*******************************************************************************/
private static int lowest(Integer code, Integer row)
{
int base = code == null ? 3600 : code;
return (row != null && row > 0 && row < base ? row : base);
}
/*******************************************************************************
** The row's presign limits, read at most every 15 seconds per alias (the way
** the header rules are cached); never throws.
*******************************************************************************/
private static RowLimits rowLimits(String alias)
{
RowLimits cached = ROW_LIMITS.get(alias);
if(cached != null && System.currentTimeMillis() - cached.readAt() < ROW_LIMIT_CACHE_MILLIS)
{
return (cached);
}
RowLimits limits = new RowLimits(System.currentTimeMillis(), null, null);
if(Db.available(ObjectStorageTables.BUCKET))
{
try
{
QRecord row = Db.asSystem(() -> Db.first(ObjectStorageTables.BUCKET, "alias", alias));
if(row != null)
{
limits = new RowLimits(System.currentTimeMillis(), row.getValueInteger("maxPresignGetSeconds"), row.getValueInteger("maxPresignPutSeconds"));
}
}
catch(Exception e)
{
LOG.debug("Could not read bucket presign limits", e);
}
}
ROW_LIMITS.put(alias, limits);
return (limits);
}
}
@@ -291,8 +291,9 @@ public class ConnectionRuntime
/*******************************************************************************
** STAND-IN(core=QRun-IO/qqq#1165, removeBy=4.1.0-RC.7)
** RC.5 metadata/factory lacks a session token or credentials-provider hook.
** STAND-IN(core=QRun-IO/qqq#1165, removeBy=4.1.0-RC.8)
** RC.7 S3BackendMetaData and S3ClientFactory still lack a session token or
** credentials-provider hook (verified against the released JAR).
** Override request credentials only; core still owns the client and transport.
*******************************************************************************/
private static class SessionCredentialsClient extends DelegatingS3Client
@@ -36,7 +36,8 @@ import com.kingsrook.qqq.backend.core.utils.StringUtils;
** the allow-list. Resolved values live only in memory, never in a record, a log
** line or a process value.
**
** STAND-IN(core=QRun-IO/qqq#1165, removeBy=4.1.0-RC.7)
** STAND-IN(core=QRun-IO/qqq#1165, removeBy=4.1.0-RC.8)
** RC.7 QMetaDataVariableInterpreter still supports env/prop, not file references.
** ${file.PATH} moves to QMetaDataVariableInterpreter in QQQ core
** (spec section 8.2); then this class delegates entirely.
*******************************************************************************/
@@ -0,0 +1,132 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.objectstorage.model;
import java.io.IOException;
import java.io.InputStream;
import java.util.LinkedHashMap;
import java.util.Map;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.dataformat.yaml.YAMLFactory;
import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Applies help/qbit-object-storage.yaml (the QQQ help-bundle format) to the
** Storage app and its object_storage_* tables: what each table is for and how
** its rows get created, shown on the table screen and when it is empty.
** Inline defaults preserve empty-state help for hosts without helpContent.
** The producer also registers the bundle for core sync and HOST overrides.
*******************************************************************************/
public final class ObjectStorageHelp
{
public static final String RESOURCE = "/help/qbit-object-storage.yaml";
public static final String SLOT = "help";
/*******************************************************************************
** Utility class.
*******************************************************************************/
private ObjectStorageHelp()
{
}
/*******************************************************************************
** The bundle's entries by key, validated (fails startup if malformed).
*******************************************************************************/
public static Map<String, JsonNode> entries() throws QException
{
try(InputStream stream = ObjectStorageHelp.class.getResourceAsStream(RESOURCE))
{
if(stream == null)
{
throw (new IOException("missing resource"));
}
JsonNode bundle = new ObjectMapper(new YAMLFactory()).readTree(stream);
if(!"qbit-object-storage".equals(bundle.path("bundle").asText()) || !"QBIT".equals(bundle.path("layer").asText()))
{
throw (new IOException("invalid bundle header"));
}
Map<String, JsonNode> entries = new LinkedHashMap<>();
for(JsonNode entry : bundle.path("entries"))
{
String key = entry.path("key").asText();
String summary = entry.path("summary").asText();
if(key.isBlank() || summary.isBlank() || summary.length() > 120 || entries.put(key, entry) != null)
{
throw (new IOException("invalid or duplicate entry: " + key));
}
}
return (entries);
}
catch(IOException | RuntimeException e)
{
throw (new QException("Could not load " + RESOURCE, e));
}
}
/*******************************************************************************
** Apply the help to the app and every object_storage_* table in the
** instance. A host's own help on a table or the app is kept (the bundle only
** fills the slot).
*******************************************************************************/
public static void apply(QInstance qInstance, ObjectStorageQBitConfig config) throws QException
{
Map<String, JsonNode> entries = entries();
for(QTableMetaData table : qInstance.getTables().values())
{
JsonNode entry = entries.get("table:" + table.getName() + ";slot:" + SLOT);
if(entry != null && (table.getHelpContent() == null || !table.getHelpContent().containsKey(SLOT)))
{
table.withHelpContent(SLOT, content(entry));
}
}
QAppMetaData app = qInstance.getApp(config.getAppName());
JsonNode appEntry = entries.get("app:objectStorageAdmin;slot:" + SLOT);
if(app != null && appEntry != null && (app.getHelpContent() == null || !app.getHelpContent().containsKey(SLOT)))
{
app.withHelpContent(SLOT, content(appEntry));
}
}
/*******************************************************************************
** Summary as the first paragraph, then the content, as markdown.
*******************************************************************************/
static QHelpContent content(JsonNode entry)
{
return (new QHelpContent().withContentAsMarkdown(entry.path("summary").asText().strip() + "\n\n" + entry.path("content").asText().strip()));
}
}
@@ -38,7 +38,9 @@ import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
**
** The statements are idempotent (CREATE ... IF NOT EXISTS) and include the
** unique constraints and the browse indexes the design calls for
** ((bucket, key) unique; bucket with folder, last modified, size, content type).
** ((bucket, key) unique; bucket with folder, last modified, size, content type),
** then the foreign keys, each in a block that also upgrades an existing schema
** (orphans recorded in qqq_fk_quarantine first; see addForeignKey).
*******************************************************************************/
public final class ObjectStorageSchema
{
@@ -52,14 +54,79 @@ public final class ObjectStorageSchema
ObjectStorageTables.OBJECT, List.of(List.of("bucketId", "objectKey")),
ObjectStorageTables.USAGE_DAILY, List.of(List.of("bucketId", "snapshotDate")));
/** Browse and lookup indexes: table to column lists. */
/** Browse and lookup indexes: table to column lists. New entries go at the end of
** a table's list, so the generated names (ix_<table>_<n>) of existing ones never change.
** The child lists on record pages (folder -> objects, object -> versions and references)
** filter on the parent id alone, so those columns lead an index of their own. */
private static final Map<String, List<List<String>>> INDEXES = Map.of(
ObjectStorageTables.OBJECT, List.of(List.of("bucketId", "folderId"), List.of("bucketId", "lastModified"), List.of("bucketId", "size"), List.of("bucketId", "contentType"),
List.of("bucketId", "deleted", "referenceCount"), List.of("bucketId", "headersKnown", "deleted")),
ObjectStorageTables.OBJECT_REFERENCE, List.of(List.of("bucketId", "objectId"), List.of("bucketId", "missing")),
List.of("bucketId", "deleted", "referenceCount"), List.of("bucketId", "headersKnown", "deleted"), List.of("folderId")),
ObjectStorageTables.OBJECT_REFERENCE, List.of(List.of("bucketId", "objectId"), List.of("bucketId", "missing"), List.of("objectId")),
ObjectStorageTables.OPERATION, List.of(List.of("bucketId", "occurredAt"), List.of("objectKey", "occurredAt")),
ObjectStorageTables.FOLDER, List.of(List.of("parentFolderId")),
ObjectStorageTables.OBJECT_VERSION, List.of(List.of("bucketId", "objectKey")));
ObjectStorageTables.OBJECT_VERSION, List.of(List.of("bucketId", "objectKey"), List.of("objectId")),
ObjectStorageTables.BUCKET, List.of(List.of("connectionId")),
ObjectStorageTables.HEADER_RULE, List.of(List.of("bucketId")),
ObjectStorageTables.PROTECTION_RULE, List.of(List.of("bucketId")));
/** The table that records the rows an upgrade changed so a foreign key could be added
** (identical DDL in every qbit, so whichever runs first creates it). */
public static final String QUARANTINE_DDL = "CREATE TABLE IF NOT EXISTS qqq_fk_quarantine (id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, "
+ "table_name VARCHAR(100) NOT NULL, column_name VARCHAR(100) NOT NULL, row_id VARCHAR(100), missing_parent_id VARCHAR(100), action VARCHAR(20) NOT NULL, "
+ "row_data TEXT, changeset VARCHAR(200), quarantined_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP)";
/** Every parent/child relation, enforced in the database. The qbit never deletes a
** connection, bucket or folder row (ConfigSync only upserts; folders are only
** created), and its only deletes are object rows (scan removal, trash retention,
** hard-delete write-through) and reference rows (rebuilt by each reference scan),
** through QQQ deletes with no associations. So:
** - observed state that belongs to a parent (capabilities, folders, objects, versions,
** usage) CASCADEs: an object delete takes its versions, and an admin deleting a
** bucket row takes its index with it;
** - optional pointers into the index (object.folderId, reference.objectId) SET NULL:
** the next scan sets the folder again, and a reference to a removed object is
** missing until the next reference scan rebuilds the rows;
** - admin-authored config (header and protection rules), the operation audit log and
** a bucket's connection RESTRICT (NO ACTION): deleting a bucket row that has rules
** or history, or a connection that buckets use, is refused rather than silently
** dropping protection or audit rows (ConfigSync would re-create a configured bucket
** under a new id, with its rules gone).
** Inserts are parents first everywhere (connections before buckets, a folder's parent
** before the folder, folders before objects), so nothing needs to be deferred.
** New entries go at the end. */
private static final List<ForeignKey> FOREIGN_KEYS = List.of(
new ForeignKey(ObjectStorageTables.BUCKET, "connectionId", ObjectStorageTables.CONNECTION, "NO ACTION"),
new ForeignKey(ObjectStorageTables.CONNECTION_CAPABILITY, "connectionId", ObjectStorageTables.CONNECTION, "CASCADE"),
new ForeignKey(ObjectStorageTables.BUCKET_CAPABILITY, "bucketId", ObjectStorageTables.BUCKET, "CASCADE"),
new ForeignKey(ObjectStorageTables.HEADER_RULE, "bucketId", ObjectStorageTables.BUCKET, "NO ACTION"),
new ForeignKey(ObjectStorageTables.PROTECTION_RULE, "bucketId", ObjectStorageTables.BUCKET, "NO ACTION"),
new ForeignKey(ObjectStorageTables.OPERATION, "bucketId", ObjectStorageTables.BUCKET, "NO ACTION"),
new ForeignKey(ObjectStorageTables.USAGE_DAILY, "bucketId", ObjectStorageTables.BUCKET, "CASCADE"),
new ForeignKey(ObjectStorageTables.FOLDER, "bucketId", ObjectStorageTables.BUCKET, "CASCADE"),
new ForeignKey(ObjectStorageTables.FOLDER, "parentFolderId", ObjectStorageTables.FOLDER, "CASCADE"),
new ForeignKey(ObjectStorageTables.OBJECT, "bucketId", ObjectStorageTables.BUCKET, "CASCADE"),
new ForeignKey(ObjectStorageTables.OBJECT, "folderId", ObjectStorageTables.FOLDER, "SET NULL"),
new ForeignKey(ObjectStorageTables.OBJECT_VERSION, "bucketId", ObjectStorageTables.BUCKET, "CASCADE"),
new ForeignKey(ObjectStorageTables.OBJECT_VERSION, "objectId", ObjectStorageTables.OBJECT, "CASCADE"),
new ForeignKey(ObjectStorageTables.OBJECT_REFERENCE, "bucketId", ObjectStorageTables.BUCKET, "CASCADE"),
new ForeignKey(ObjectStorageTables.OBJECT_REFERENCE, "objectId", ObjectStorageTables.OBJECT, "SET NULL"));
/*******************************************************************************
** One foreign key: the child table and field, the parent table (always its id)
** and the ON DELETE rule.
*******************************************************************************/
public record ForeignKey(String table, String field, String parent, String onDelete)
{
/*******************************************************************************
** The constraint name: fk_<child table>_<child column>.
*******************************************************************************/
public String name()
{
return ("fk_" + table + "_" + column(field));
}
}
@@ -99,11 +166,56 @@ public final class ObjectStorageSchema
}
}
}
statements.add(QUARANTINE_DDL);
for(ForeignKey fk : FOREIGN_KEYS)
{
statements.add(addForeignKey(fk, tables));
}
return (statements);
}
/*******************************************************************************
** The foreign keys the schema declares.
*******************************************************************************/
public static List<ForeignKey> foreignKeys()
{
return (FOREIGN_KEYS);
}
/*******************************************************************************
** One idempotent block that adds a foreign key and upgrades an existing schema
** safely: rows whose parent is gone are first recorded in qqq_fk_quarantine, then
** their reference is cleared (nullable column) or the row is deleted (required
** column; the whole row is recorded), so adding the constraint cannot fail. A
** no-op once the constraint exists.
*******************************************************************************/
private static String addForeignKey(ForeignKey fk, List<QTableMetaData> tables)
{
String col = column(fk.field());
boolean required = tables.stream().filter(t -> t.getName().equals(fk.table())).findFirst()
.map(t -> Boolean.TRUE.equals(t.getField(fk.field()).getIsRequired())).orElse(false);
String orphans = "c." + col + " IS NOT NULL AND NOT EXISTS (SELECT 1 FROM " + fk.parent() + " p WHERE p.id = c." + col + ")";
return ("DO $$\n"
+ "BEGIN\n"
+ " IF NOT EXISTS (SELECT 1 FROM pg_constraint WHERE conname = '" + fk.name() + "' AND conrelid = '" + fk.table() + "'::regclass) THEN\n"
+ " INSERT INTO qqq_fk_quarantine (table_name, column_name, row_id, missing_parent_id, action, row_data, changeset)\n"
+ " SELECT '" + fk.table() + "', '" + col + "', c.id::text, c." + col + "::text, '" + (required ? "DELETED" : "NULLED") + "', row_to_json(c)::text, "
+ "'qbit-object-storage ObjectStorageSchema'\n"
+ " FROM " + fk.table() + " c WHERE " + orphans + ";\n"
+ (required
? " DELETE FROM " + fk.table() + " c WHERE " + orphans + ";\n"
: " UPDATE " + fk.table() + " c SET " + col + " = NULL WHERE " + orphans + ";\n")
+ " ALTER TABLE " + fk.table() + " ADD CONSTRAINT " + fk.name() + " FOREIGN KEY (" + col + ") REFERENCES " + fk.parent() + " (id) ON DELETE " + fk.onDelete() + ";\n"
+ " END IF;\n"
+ "END $$");
}
/*******************************************************************************
** The statements as one script.
*******************************************************************************/
@@ -37,6 +37,7 @@ import com.kingsrook.qbits.objectstorage.config.StorageBucketConfig;
import com.kingsrook.qbits.objectstorage.index.ObjectIndex;
import com.kingsrook.qbits.objectstorage.index.ReferenceScan;
import com.kingsrook.qbits.objectstorage.internal.CapabilityStore;
import com.kingsrook.qbits.objectstorage.internal.ConfigSync;
import com.kingsrook.qbits.objectstorage.internal.ConnectionRuntime;
import com.kingsrook.qbits.objectstorage.internal.Db;
import com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime;
@@ -689,7 +690,7 @@ public final class BucketProcesses
request.setContentLength(Long.valueOf(String.valueOf(input.getValue(FIELD_LINK_LENGTH))));
}
URL url = ActionSupport.storage(alias).presignPut(request);
int minutes = (int) Math.min(ActionSupport.config(alias).getMaxPresignPutSeconds() / 60, 15);
int minutes = (int) Math.min(ConfigSync.effectiveMaxPresignPutSeconds(ActionSupport.config(alias)) / 60, 15);
ProcessSummaryLine line = EtlSupport.text(Status.OK, "Upload link for " + alias + "/" + request.getKey() + " (expires in about " + minutes + " minutes; shown once)");
line.setBulletsOfText(new ArrayList<>(List.of(url.toString())));
results.add(line);
@@ -27,6 +27,7 @@ import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qbits.objectstorage.api.PresignGet;
import com.kingsrook.qbits.objectstorage.config.StorageBucketConfig;
import com.kingsrook.qbits.objectstorage.internal.ConfigSync;
import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables;
import com.kingsrook.qbits.objectstorage.permissions.ObjectStoragePermissions;
import com.kingsrook.qqq.backend.core.exceptions.QException;
@@ -150,7 +151,7 @@ public final class LinkProcesses
{
String alias = ActionSupport.aliasOf(record);
StorageBucketConfig config = ActionSupport.config(alias);
Duration ttl = Duration.ofSeconds(Math.min(300, config.getMaxPresignGetSeconds()));
Duration ttl = Duration.ofSeconds(Math.min(300, ConfigSync.effectiveMaxPresignGetSeconds(config)));
URL url = ActionSupport.storage(alias).presignGet(PresignGet.of(record.getValueString("objectKey")).ttl(ttl).downloadAs(record.getValueString("name")));
ProcessSummaryLine line = EtlSupport.text(Status.OK, "Download link for " + alias + "/" + record.getValueString("objectKey") + " (expires in " + ttl.toMinutes() + " minutes; shown once)");
line.setBulletsOfText(new ArrayList<>(List.of(url.toString())));
@@ -0,0 +1,34 @@
-- liquibase formatted sql
-- qbit-object-storage starter defaults.
-- Include it from the host's master changelog AFTER the changesets that create
-- the object_storage_* tables (the script from ObjectStorageSchema.postgresScript):
--
-- - include: { file: db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql }
--
-- Rules for this file:
-- * insert-only: a row is inserted only when its natural key is absent, so an
-- administrator's edits always survive and nothing is ever updated;
-- * a seed never changes what an existing host does on deploy: the rows below
-- are inserted DISABLED, so they are visible examples until someone turns
-- them on;
-- * rollback is empty: a seed row may have been edited or adopted by the host, so
-- rolling back never deletes data;
-- * no secrets, no host-specific values;
-- * portable SQL (PostgreSQL and H2): INSERT ... SELECT ... WHERE NOT EXISTS;
-- * every changeset has a precondition, so a host that has not created the
-- table yet simply skips it (MARK_RAN).
-- DefaultsChangelogTest runs this file through Liquibase, twice, on H2.
--
-- Not seeded: object_storage_protection_rule (a rule needs a bucket, which is
-- host-specific), and the connection and bucket rows (written from code by the
-- qbit's config sync).
-- changeset qbit-object-storage:defaults-1
-- preconditions onFail:MARK_RAN onError:MARK_RAN
-- precondition-sql-check expectedResult:0 SELECT COUNT(*) FROM object_storage_header_rule WHERE name = 'default-mutable'
-- comment: A generic header rule for mutable objects, disabled. Enable it to give every object (any bucket) Cache-Control public, max-age=3600; lower the priority number to rank it ahead of other rules.
INSERT INTO object_storage_header_rule (name, key_glob, cache_control, priority, enabled, create_date, modify_date)
SELECT 'default-mutable', '**', 'public, max-age=3600', 1000, FALSE, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP
WHERE NOT EXISTS (SELECT 1 FROM object_storage_header_rule WHERE name = 'default-mutable');
-- rollback empty
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,76 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.objectstorage;
import java.util.Set;
import java.util.stream.Collectors;
import com.kingsrook.qbits.objectstorage.testsupport.StorageTestSupport;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.help.bundles.HelpBundles;
import com.kingsrook.qqq.backend.core.testing.HelpBundleAssertions;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** Coverage of every object owned by the storage qbit, including its optional app.
*******************************************************************************/
class ObjectStorageHelpBundleTest
{
/*******************************************************************************
** Release the installed runtime after each test.
*******************************************************************************/
@AfterEach
void tearDown()
{
StorageTestSupport.tearDown();
}
/*******************************************************************************
** Register safely without help tables and cover all owned metadata.
*******************************************************************************/
@Test
void bundleIsCompleteWithoutHelpTables() throws Exception
{
QInstance instance = StorageTestSupport.setUp(new ObjectStorageQBitConfig()
.withBackendName(StorageTestSupport.MEMORY).withBucketScoping(false).withOpenWhenAnonymous(true).withAdminApp(true));
assertThat(instance.getTable("helpContent")).isNull();
assertThat(HelpBundles.of(instance).getBundles()).singleElement()
.satisfies(bundle -> assertThat(bundle.getId()).isEqualTo("qbit-object-storage"));
HelpBundles.syncOnStartup(instance);
Set<String> keys = HelpBundles.of(instance).getBundles().get(0).getEntries().stream()
.map(entry -> entry.getKey()).collect(Collectors.toSet());
// RC.7 validates these entries but only requires app/table/process/field coverage.
instance.getWidgets().keySet().stream().filter(name -> name.startsWith("object_storage_") || name.startsWith("objectStorage"))
.forEach(name -> assertThat(keys).contains("widget:" + name + ";slot:label"));
instance.getPossibleValueSources().keySet().stream().filter(name -> name.startsWith("object_storage_"))
.forEach(name -> assertThat(keys).contains("possibleValueSource:" + name + ";slot:help"));
HelpBundleAssertions.forInstance(instance)
.bundles("help/qbit-object-storage.yaml")
.owning(target -> target.name().startsWith("object_storage_") || target.name().startsWith("objectStorage"))
.assertComplete();
}
}
@@ -0,0 +1,218 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.objectstorage;
import java.lang.reflect.Field;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.List;
import java.util.Map;
import java.util.stream.Stream;
import com.kingsrook.qbits.objectstorage.internal.DefaultsChangelogTest;
import com.kingsrook.qbits.objectstorage.model.ObjectStorageHelp;
import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables;
import com.kingsrook.qbits.objectstorage.testsupport.StorageTestSupport;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** Guardrail (owner requirement 2026-10-14: "all tables should be populated and
** the qbit should be fully using them"). For every object_storage_* table:
** - runtime code (not just the model, metadata or home screen) reads or writes
** it, or it is listed in NOT_USED_YET with what is missing;
** - it is either seeded by the shipped defaults changelog, or listed in
** STARTS_EMPTY with why;
** - it has bundle help that says how its rows get created (shown on the table
** screen and when it is empty).
** A new table fails this test until it is wired, seeded or explained.
*******************************************************************************/
class ObjectStorageTablesGuardrailTest
{
private static final Path MAIN = Path.of("src", "main", "java", "com", "kingsrook", "qbits", "objectstorage");
/*******************************************************************************
** Tables that no runtime code writes yet, and what is missing. Their help says
** "Not used yet".
*******************************************************************************/
static final Map<String, String> NOT_USED_YET = Map.of(
ObjectStorageTables.OBJECT_VERSION, "the indexer lists current versions only (ListObjectsV2); indexing version history is a later release");
/*******************************************************************************
** Tables that legitimately start empty, and where their rows come from.
*******************************************************************************/
static final Map<String, String> STARTS_EMPTY = Map.ofEntries(
Map.entry(ObjectStorageTables.CONNECTION, "written from code by the config sync: the host declares its connections"),
Map.entry(ObjectStorageTables.BUCKET, "written from code by the config sync: the host declares its buckets"),
Map.entry(ObjectStorageTables.CONNECTION_CAPABILITY, "runtime: Test connection probes the provider"),
Map.entry(ObjectStorageTables.BUCKET_CAPABILITY, "runtime: Test connection and Check configuration probe the bucket"),
Map.entry(ObjectStorageTables.FOLDER, "runtime: the index scan and uploads"),
Map.entry(ObjectStorageTables.OBJECT, "runtime: the index scan and write-through"),
Map.entry(ObjectStorageTables.OBJECT_VERSION, "not used yet: see NOT_USED_YET"),
Map.entry(ObjectStorageTables.OBJECT_REFERENCE, "runtime: the reference scan, when the host registers a reference provider"),
Map.entry(ObjectStorageTables.PROTECTION_RULE, "a rule needs a bucket, which is host-specific: an administrator adds each rule"),
Map.entry(ObjectStorageTables.OPERATION, "log: every operation the qbit performs"),
Map.entry(ObjectStorageTables.USAGE_DAILY, "runtime: the nightly job"));
/*******************************************************************************
** Tear down.
*******************************************************************************/
@AfterEach
void tearDown()
{
StorageTestSupport.tearDown();
}
/*******************************************************************************
** Every table is read or written by runtime code, or explicitly not used yet.
*******************************************************************************/
@Test
void everyTableIsUsedAtRuntime() throws Exception
{
List<String> runtimeSource;
try(Stream<Path> files = Files.walk(MAIN))
{
runtimeSource = files.filter(p -> p.toString().endsWith(".java"))
.filter(p -> !p.startsWith(MAIN.resolve("model")) && !p.startsWith(MAIN.resolve("app")))
.map(ObjectStorageTablesGuardrailTest::read).toList();
}
for(String table : ObjectStorageTables.ALL)
{
String constant = "ObjectStorageTables." + constantFor(table);
boolean used = runtimeSource.stream().anyMatch(source -> source.contains(constant));
if(NOT_USED_YET.containsKey(table))
{
assertThat(used).as(table + " is listed as not used yet, but runtime code uses it: remove it from NOT_USED_YET").isFalse();
}
else
{
assertThat(used).as(table + " must be read or written by runtime code, not only declared").isTrue();
}
}
}
/*******************************************************************************
** Every table is seeded or explained.
*******************************************************************************/
@Test
void everyTableIsSeededOrExplained() throws Exception
{
String changelog = DefaultsChangelogTest.changelogText();
for(String table : ObjectStorageTables.ALL)
{
boolean seeded = DefaultsChangelogTest.SEEDED_TABLES.contains(table);
assertThat(seeded || STARTS_EMPTY.containsKey(table)).as(table + " must be seeded by the defaults changelog or listed in STARTS_EMPTY").isTrue();
assertThat(seeded && STARTS_EMPTY.containsKey(table)).as(table + " cannot be both seeded and listed as starting empty").isFalse();
assertThat(changelog.contains("INSERT INTO " + table + " ")).as(table + " seeded iff the changelog inserts it").isEqualTo(seeded);
}
assertThat(STARTS_EMPTY.keySet()).allMatch(ObjectStorageTables.ALL::contains, "STARTS_EMPTY names only real tables");
assertThat(NOT_USED_YET.keySet()).allMatch(ObjectStorageTables.ALL::contains, "NOT_USED_YET names only real tables");
}
/*******************************************************************************
** Every table and the app have bundle help that says how rows get created,
** and the not-yet-used tables say so.
*******************************************************************************/
@Test
void everyTableHasEmptyStateHelp() throws Exception
{
QInstance instance = StorageTestSupport.setUp(new ObjectStorageQBitConfig().withBackendName(StorageTestSupport.MEMORY).withBucketScoping(false).withOpenWhenAnonymous(true).withAdminApp(true));
assertThat(ObjectStorageHelp.entries()).allSatisfy((key, entry) -> assertThat(entry.path("summary").asText().length()).as(key).isLessThanOrEqualTo(120));
for(String table : ObjectStorageTables.ALL)
{
List<QHelpContent> help = instance.getTable(table).getHelpContent() == null ? null : instance.getTable(table).getHelpContent().get(ObjectStorageHelp.SLOT);
assertThat(help).as(table + " table help").isNotEmpty();
assertThat(help.get(0).getContent()).as(table + " help says how rows get created").contains("How rows get created");
if(NOT_USED_YET.containsKey(table))
{
assertThat(help.get(0).getContent()).as(table + " help says it is not used yet").contains("Not used yet");
}
}
assertThat(instance.getApp("objectStorageAdmin").getHelpContent()).containsKey(ObjectStorageHelp.SLOT);
assertThat(ObjectStorageHelp.entries().keySet()).allMatch(key -> !key.startsWith("table:") || instance.getTable(key.substring(6, key.indexOf(';'))) != null,
"help names only real tables");
}
/*******************************************************************************
** The ObjectStorageTables constant name of a table.
*******************************************************************************/
private static String constantFor(String table) throws Exception
{
for(Field field : ObjectStorageTables.class.getFields())
{
if(field.getType() == String.class && table.equals(field.get(null)) && !field.getName().startsWith("KEY_") && !field.getName().startsWith("PVS_") && !field.getName().startsWith("ALL_"))
{
return (field.getName());
}
}
throw (new IllegalStateException("no constant for " + table));
}
/*******************************************************************************
** Read a file.
*******************************************************************************/
private static String read(Path path)
{
try
{
return (Files.readString(path, StandardCharsets.UTF_8));
}
catch(Exception e)
{
throw (new IllegalStateException(e));
}
}
/*******************************************************************************
** A host's own help in the slot is kept; the bundle only fills empty slots.
*******************************************************************************/
@Test
void hostHelpIsKept() throws Exception
{
ObjectStorageQBitConfig config = new ObjectStorageQBitConfig().withBackendName(StorageTestSupport.MEMORY).withBucketScoping(false).withOpenWhenAnonymous(true).withAdminApp(true);
QInstance instance = StorageTestSupport.setUp(config);
instance.getTable(ObjectStorageTables.BUCKET).withHelpContent(ObjectStorageHelp.SLOT, new QHelpContent().withContentAsMarkdown("host text"));
instance.getApp("objectStorageAdmin").withHelpContent(ObjectStorageHelp.SLOT, new QHelpContent().withContentAsMarkdown("host app text"));
ObjectStorageHelp.apply(instance, config);
assertThat(instance.getTable(ObjectStorageTables.BUCKET).getHelpContent().get(ObjectStorageHelp.SLOT)).singleElement().extracting(QHelpContent::getContent).isEqualTo("host text");
assertThat(instance.getApp("objectStorageAdmin").getHelpContent().get(ObjectStorageHelp.SLOT)).singleElement().extracting(QHelpContent::getContent).isEqualTo("host app text");
}
}
@@ -0,0 +1,126 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.objectstorage.app;
import java.util.List;
import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig;
import com.kingsrook.qbits.objectstorage.api.ObjectHeaderPolicy;
import com.kingsrook.qbits.objectstorage.app.StorageSetupStatus.Step;
import com.kingsrook.qbits.objectstorage.config.BucketRole;
import com.kingsrook.qbits.objectstorage.config.ProviderType;
import com.kingsrook.qbits.objectstorage.config.StorageBucketConfig;
import com.kingsrook.qbits.objectstorage.config.StorageConnectionConfig;
import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables;
import com.kingsrook.qbits.objectstorage.testsupport.StorageTestSupport;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** The Setup card: states follow what is configured, resolved the way the
** runtime resolves it (a host's own header policy beats the rows).
*******************************************************************************/
class StorageSetupStatusTest
{
/*******************************************************************************
** Tear down.
*******************************************************************************/
@AfterEach
void tearDown()
{
StorageTestSupport.tearDown();
}
/*******************************************************************************
** A bare install needs a connection and a bucket; headers use defaults; the
** protection rules are optional.
*******************************************************************************/
@Test
void bareInstall()
{
List<Step> steps = StorageSetupStatus.steps(0, 0, 0, false, 0);
assertThat(steps).extracting(Step::label).containsExactly("Connections", "Buckets", "Header rules", "Protection rules");
assertThat(steps).extracting(Step::state).containsExactly("Needs setup", "Needs setup", "Using defaults", "Optional");
assertThat(steps).allSatisfy(step -> assertThat(step.next()).isNotBlank());
}
/*******************************************************************************
** A host policy in code means the rows are not what decides, so it is never
** reported as needing setup.
*******************************************************************************/
@Test
void hostPolicyIsRespected()
{
assertThat(StorageSetupStatus.steps(1, 1, 0, true, 0).get(2)).satisfies(step ->
{
assertThat(step.state()).isEqualTo("Set up");
assertThat(step.next()).contains("host's own header policy");
});
assertThat(StorageSetupStatus.steps(1, 1, 2, false, 1)).extracting(Step::state).containsOnly("Set up");
}
/*******************************************************************************
** The live card reads the tables and the config.
*******************************************************************************/
@Test
void liveStatus() throws Exception
{
ObjectStorageQBitConfig config = new ObjectStorageQBitConfig().withBackendName(StorageTestSupport.MEMORY).withBucketScoping(false).withOpenWhenAnonymous(true).withAdminApp(true)
.withConnection(new StorageConnectionConfig("main").withProviderType(ProviderType.MINIO).withEndpoint("http://localhost:1").withRegion("us-east-1"))
.withBucket(new StorageBucketConfig("media").withConnection("main").withBucketName("media-bucket").withRole(BucketRole.PRIVATE).withIndexed(false));
StorageTestSupport.setUp(config);
assertThat(StorageSetupStatus.load()).extracting(Step::state).containsExactly("Set up", "Set up", "Using defaults", "Optional");
StorageTestSupport.insert(ObjectStorageTables.HEADER_RULE, new QRecord().withValue("name", "r").withValue("keyGlob", "*").withValue("cacheControl", "no-cache").withValue("priority", 1).withValue("enabled", true));
assertThat(StorageSetupStatus.load().get(2).state()).isEqualTo("Set up");
ObjectHeaderPolicy hostPolicy = (bucket, key, contentType) -> new com.kingsrook.qbits.objectstorage.api.ObjectHeaders();
config.withHeaderPolicy(hostPolicy);
assertThat(StorageSetupStatus.load().get(2).next()).contains("host's own");
}
/*******************************************************************************
** A bucket-scoped user with no bucket keys sees no rows, but the card reads
** the runtime config and counts rules as the system, so it never shows a
** false "needs setup".
*******************************************************************************/
@Test
void scopedUserSeesTrueStatus() throws Exception
{
ObjectStorageQBitConfig config = new ObjectStorageQBitConfig().withBackendName(StorageTestSupport.MEMORY).withBucketScoping(true).withOpenWhenAnonymous(true).withAdminApp(true)
.withConnection(new StorageConnectionConfig("main").withProviderType(ProviderType.MINIO).withEndpoint("http://localhost:1").withRegion("us-east-1"))
.withBucket(new StorageBucketConfig("media").withConnection("main").withBucketName("media-bucket").withRole(BucketRole.PRIVATE).withIndexed(false));
StorageTestSupport.setUp(config);
StorageTestSupport.insert(ObjectStorageTables.HEADER_RULE, new QRecord().withValue("name", "r").withValue("keyGlob", "*").withValue("cacheControl", "no-cache").withValue("priority", 1).withValue("enabled", true));
assertThat(StorageSetupStatus.load()).extracting(Step::state).containsExactly("Set up", "Set up", "Set up", "Optional");
}
}
@@ -23,10 +23,14 @@ package com.kingsrook.qbits.objectstorage.index;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;
import java.util.TreeSet;
import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig;
import com.kingsrook.qbits.objectstorage.api.FieldReferenceProvider;
import com.kingsrook.qbits.objectstorage.api.ObjectStorage;
@@ -46,6 +50,11 @@ import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSourceType;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.model.session.QUser;
@@ -270,4 +279,203 @@ class PostgresIndexTest
.withSecurityKeyValue(ObjectStorageTables.KEY_TYPE_BUCKET, ConfigSync.bucketId("indexed")));
assertThat(new QueryAction().execute(new QueryInput(ObjectStorageTables.OBJECT)).getRecords()).extracting(r -> r.getValueString("objectKey")).containsExactly("d/ref.txt");
}
/** Relations (child.column -> parent) that are deliberately not database foreign keys,
** each with its reason. Empty: every join and table possible value is enforced. */
private static final Map<String, String> FK_EXEMPTIONS = Map.of();
/*******************************************************************************
** The guardrail: every join and table possible-value field between the qbit's
** tables is a foreign key in the generated schema (or exempt, with a reason),
** with the declared ON DELETE rule, and every foreign key column leads an index.
** A new relation without a foreign key fails here.
*******************************************************************************/
@Test
void everyRelationIsAForeignKeyWithAnIndex() throws Exception
{
Set<String> expected = new TreeSet<>();
for(QJoinMetaData join : QContext.getQInstance().getJoins().values())
{
if(ObjectStorageTables.ALL.contains(join.getLeftTable()) && ObjectStorageTables.ALL.contains(join.getRightTable()) && join.getType() == JoinType.ONE_TO_MANY)
{
for(JoinOn on : join.getJoinOns())
{
expected.add(join.getRightTable() + "." + ObjectStorageSchemaColumns.of(on.getRightField()) + " -> " + join.getLeftTable());
}
}
}
for(String tableName : ObjectStorageTables.ALL)
{
for(QFieldMetaData field : QContext.getQInstance().getTable(tableName).getFields().values())
{
QPossibleValueSource pvs = field.getPossibleValueSourceName() == null ? null : QContext.getQInstance().getPossibleValueSource(field.getPossibleValueSourceName());
if(pvs != null && pvs.getType() == QPossibleValueSourceType.TABLE && ObjectStorageTables.ALL.contains(pvs.getTableName()))
{
expected.add(tableName + "." + ObjectStorageSchemaColumns.of(field.getName()) + " -> " + pvs.getTableName());
}
}
}
assertThat(expected).as("the qbit declares relations").hasSizeGreaterThan(10);
Map<String, String> actual = new TreeMap<>();
List<String> noIndex = new ArrayList<>();
try(Connection c = DriverManager.getConnection(postgres.getJdbcUrl(), "qqq", "qqq"); Statement st = c.createStatement();
ResultSet rs = st.executeQuery("""
SELECT cl.relname AS child, a.attname AS col, pcl.relname AS parent, con.confdeltype AS rule,
EXISTS (SELECT 1 FROM pg_index i WHERE i.indrelid = con.conrelid AND i.indkey[0] = con.conkey[1]) AS indexed
FROM pg_constraint con
JOIN pg_class cl ON cl.oid = con.conrelid
JOIN pg_class pcl ON pcl.oid = con.confrelid
JOIN pg_attribute a ON a.attrelid = con.conrelid AND a.attnum = con.conkey[1]
WHERE con.contype = 'f'"""))
{
while(rs.next())
{
String key = rs.getString("child") + "." + rs.getString("col") + " -> " + rs.getString("parent");
actual.put(key, rs.getString("rule"));
if(!rs.getBoolean("indexed"))
{
noIndex.add(key);
}
}
}
List<String> missing = new ArrayList<>();
for(String relation : expected)
{
if(!actual.containsKey(relation) && !FK_EXEMPTIONS.containsKey(relation))
{
missing.add(relation);
}
}
assertThat(missing).as("relations without a database foreign key (add one to ObjectStorageSchema, or an exemption with a reason)").isEmpty();
assertThat(noIndex).as("foreign key columns that do not lead an index").isEmpty();
Map<String, String> rules = Map.of("NO ACTION", "a", "RESTRICT", "r", "CASCADE", "c", "SET NULL", "n");
for(ObjectStorageSchema.ForeignKey fk : ObjectStorageSchema.foreignKeys())
{
String key = fk.table() + "." + ObjectStorageSchemaColumns.of(fk.field()) + " -> " + fk.parent();
assertThat(actual.get(key)).as(fk.name() + " ON DELETE").isEqualTo(rules.get(fk.onDelete()));
assertThat(fk.name().length()).as(fk.name()).isLessThanOrEqualTo(63);
}
}
/*******************************************************************************
** Upgrading a schema created before the foreign keys: orphans are recorded in
** qqq_fk_quarantine and cleared (nullable) or deleted (required), the keys are
** added, and running the statements again changes nothing.
*******************************************************************************/
@Test
void foreignKeyUpgradeQuarantinesOrphansAndIsIdempotent() throws Exception
{
Integer bucketId = ConfigSync.bucketId("indexed");
try(Connection c = DriverManager.getConnection(postgres.getJdbcUrl(), "qqq", "qqq"); Statement st = c.createStatement())
{
for(ObjectStorageSchema.ForeignKey fk : ObjectStorageSchema.foreignKeys())
{
st.execute("ALTER TABLE " + fk.table() + " DROP CONSTRAINT " + fk.name());
}
st.execute("INSERT INTO object_storage_object (bucket_id, object_key, folder_id) VALUES (" + bucketId + ", 'kept.txt', 424242)");
st.execute("INSERT INTO object_storage_protection_rule (bucket_id, prefix_glob, mode) VALUES (434343, 'x/*', 'NO_DELETE')");
st.execute("INSERT INTO object_storage_protection_rule (bucket_id, prefix_glob, mode) VALUES (" + bucketId + ", 'y/*', 'NO_DELETE')");
ObjectStorageQBitConfig forSchema = new ObjectStorageQBitConfig().withBackendName("rdbms");
for(int pass = 0; pass < 2; pass++)
{
for(String sql : ObjectStorageSchema.postgresStatements(forSchema))
{
st.execute(sql);
}
}
try(ResultSet rs = st.executeQuery("SELECT table_name, column_name, missing_parent_id, action, row_data FROM qqq_fk_quarantine ORDER BY id"))
{
List<String> rows = new ArrayList<>();
while(rs.next())
{
rows.add(rs.getString(1) + "." + rs.getString(2) + "=" + rs.getString(3) + " " + rs.getString(4) + (rs.getString(5).contains("x/*") ? " with row" : ""));
}
assertThat(rows).as("recorded once, although the statements ran twice")
.containsExactly("object_storage_protection_rule.bucket_id=434343 DELETED with row", "object_storage_object.folder_id=424242 NULLED");
}
try(ResultSet rs = st.executeQuery("SELECT folder_id FROM object_storage_object WHERE object_key = 'kept.txt'"))
{
assertThat(rs.next()).isTrue();
assertThat(rs.getObject(1)).isNull();
}
try(ResultSet rs = st.executeQuery("SELECT prefix_glob FROM object_storage_protection_rule ORDER BY id"))
{
assertThat(rs.next()).isTrue();
assertThat(rs.getString(1)).isEqualTo("y/*");
assertThat(rs.next()).isFalse();
}
try(ResultSet rs = st.executeQuery("SELECT COUNT(*) FROM pg_constraint WHERE contype = 'f' AND conname LIKE 'fk_object_storage_%'"))
{
rs.next();
assertThat(rs.getInt(1)).isEqualTo(ObjectStorageSchema.foreignKeys().size());
}
}
}
/*******************************************************************************
** The delete paths the qbit uses still work with the keys: removing an object
** row (scan removal, hard-delete write-through) takes its versions and leaves
** its references pointing nowhere, and a bucket row with an audit history or
** rules cannot be deleted.
*******************************************************************************/
@Test
void deletesFollowTheForeignKeyRules() throws Exception
{
rawPut("f/one.txt", 3);
ObjectIndex.scan("indexed");
Integer bucketId = ConfigSync.bucketId("indexed");
Integer objectId = Db.first(ObjectStorageTables.OBJECT, "objectKey", "f/one.txt").getValueInteger("id");
try(Connection c = DriverManager.getConnection(postgres.getJdbcUrl(), "qqq", "qqq"); Statement st = c.createStatement())
{
st.execute("INSERT INTO object_storage_object_version (bucket_id, object_id, object_key) VALUES (" + bucketId + ", " + objectId + ", 'f/one.txt')");
}
Db.insert("media", List.of(new QRecord().withValue("storageKey", "f/one.txt")));
ReferenceScan.run("indexed");
assertThat(Db.first(ObjectStorageTables.OBJECT_REFERENCE, "keyOrPrefix", "f/one.txt").getValueInteger("objectId")).isEqualTo(objectId);
try(S3Client raw = target.rawClient())
{
raw.deleteObject(DeleteObjectRequest.builder().bucket(target.indexBucket).key("f/one.txt").build());
}
assertThat(ObjectIndex.scan("indexed").status()).isEqualTo("OK");
assertThat(Db.first(ObjectStorageTables.OBJECT, "objectKey", "f/one.txt")).isNull();
assertThat(Db.query(ObjectStorageTables.OBJECT_VERSION, "objectId", objectId)).as("versions cascade").isEmpty();
assertThat(Db.first(ObjectStorageTables.OBJECT_REFERENCE, "keyOrPrefix", "f/one.txt").getValueInteger("objectId")).as("reference set null").isNull();
try(Connection c = DriverManager.getConnection(postgres.getJdbcUrl(), "qqq", "qqq"); Statement st = c.createStatement())
{
st.execute("INSERT INTO object_storage_operation (bucket_id, bucket_alias, action) VALUES (" + bucketId + ", 'indexed', 'PUT')");
org.assertj.core.api.Assertions.assertThatThrownBy(() -> st.execute("DELETE FROM object_storage_bucket WHERE id = " + bucketId))
.hasMessageContaining("fk_object_storage_operation_bucket_id");
}
}
/*******************************************************************************
** Column names as the schema derives them.
*******************************************************************************/
private static final class ObjectStorageSchemaColumns
{
/*******************************************************************************
** The snake_case column of a field.
*******************************************************************************/
static String of(String field)
{
return (com.kingsrook.qqq.backend.core.instances.QInstanceEnricher.inferBackendName(field));
}
}
}
@@ -0,0 +1,146 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.objectstorage.internal;
import java.net.URL;
import java.time.Duration;
import java.util.List;
import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig;
import com.kingsrook.qbits.objectstorage.api.ObjectStorage;
import com.kingsrook.qbits.objectstorage.api.PresignGet;
import com.kingsrook.qbits.objectstorage.api.PresignPut;
import com.kingsrook.qbits.objectstorage.config.BucketRole;
import com.kingsrook.qbits.objectstorage.config.ProviderType;
import com.kingsrook.qbits.objectstorage.config.StorageBucketConfig;
import com.kingsrook.qbits.objectstorage.config.StorageConnectionConfig;
import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables;
import com.kingsrook.qbits.objectstorage.processes.ActionSupport;
import com.kingsrook.qbits.objectstorage.testsupport.StorageTestSupport;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** The presign limits on the bucket row (Settings on object_storage_bucket) are
** read at runtime, the way ConfigSync.withOverrides reads a connection row:
** editing the row changes how long presigned links last, and the identity
** fields stay owned by code.
*******************************************************************************/
class BucketRowOverridesTest
{
/*******************************************************************************
** Tear down.
*******************************************************************************/
@AfterEach
void tearDown()
{
StorageTestSupport.tearDown();
}
/*******************************************************************************
** A config whose buckets never touch a real store (presigning is offline).
*******************************************************************************/
private static ObjectStorageQBitConfig config()
{
System.setProperty("os.unit.ak", "ak");
System.setProperty("os.unit.sk", "sk");
return (new ObjectStorageQBitConfig().withBackendName(StorageTestSupport.MEMORY).withBucketScoping(false).withOpenWhenAnonymous(true).withAdminApp(true)
.withAllowedSecretRefPatterns(List.of("prop.os.unit.*"))
.withConnection(new StorageConnectionConfig("main").withProviderType(ProviderType.MINIO).withEndpoint("http://localhost:1").withRegion("us-east-1")
.withAccessKeyRef("${prop.os.unit.ak}").withSecretKeyRef("${prop.os.unit.sk}"))
.withBucket(new StorageBucketConfig("media").withConnection("main").withBucketName("media-bucket").withRole(BucketRole.PRIVATE).withIndexed(false)
.withPublicBaseUrl("https://cdn.example.com/media/")));
}
/*******************************************************************************
** The presigned URL's lifetime in seconds.
*******************************************************************************/
private static int expires(URL url)
{
return (Integer.parseInt(url.getQuery().replaceAll(".*X-Amz-Expires=(\\d+).*", "$1")));
}
/*******************************************************************************
** The row can only LOWER the presign limits: the effective limit is the
** smaller of the code config and the row. A larger, zero or null row value
** leaves the code limit, the shared config object is never modified, and the
** download-link action and the preview URL read it too.
*******************************************************************************/
@Test
void rowCanOnlyLowerPresignLimits() throws Exception
{
StorageTestSupport.setUp(config());
Integer bucketId = ConfigSync.bucketId("media");
Duration half = Duration.ofMinutes(30);
assertThat(expires(ObjectStorage.bucket("media").presignGet(PresignGet.of("a.txt").ttl(half)))).as("code limit is an hour").isEqualTo(1800);
setRow(bucketId, 60, 120);
assertThat(expires(ObjectStorage.bucket("media").presignGet(PresignGet.of("a.txt").ttl(half)))).as("the row lowers the cap").isEqualTo(60);
assertThat(expires(ObjectStorage.bucket("media").presignPut(PresignPut.of("a.txt").ttl(Duration.ofMinutes(10))))).isEqualTo(120);
assertThat(ConfigSync.effectiveMaxPresignGetSeconds(ActionSupport.config("media"))).as("the download-link action reads it too").isEqualTo(60);
assertThat(expires(ObjectStorage.bucket("media").previewUrl("a.txt"))).as("previews are capped as well").isEqualTo(60);
assertThat(ObjectStorageRuntime.bucketConfig("media").getMaxPresignGetSeconds()).as("the shared config is not mutated").isEqualTo(3600);
setRow(bucketId, 999_999, 999_999);
assertThat(expires(ObjectStorage.bucket("media").presignGet(PresignGet.of("a.txt").ttl(Duration.ofHours(2))))).as("the row cannot raise the code limit").isEqualTo(3600);
setRow(bucketId, 0, null);
assertThat(expires(ObjectStorage.bucket("media").presignGet(PresignGet.of("a.txt").ttl(half)))).as("zero: the code value applies again").isEqualTo(1800);
assertThat(ConfigSync.effectiveMaxPresignPutSeconds(ActionSupport.config("media"))).as("null: the code value applies again").isEqualTo(900);
}
/*******************************************************************************
** Set the row limits and drop the short-lived cache.
*******************************************************************************/
private static void setRow(Integer bucketId, Integer get, Integer put) throws Exception
{
Db.update(ObjectStorageTables.BUCKET, List.of(new QRecord().withValue("id", bucketId).withValue("maxPresignGetSeconds", get).withValue("maxPresignPutSeconds", put)));
ConfigSync.clearRowLimits();
}
/*******************************************************************************
** Identity fields stay code-owned: a tampered public base URL on the row is
** rewritten from code and never read back as an override.
*******************************************************************************/
@Test
void identityStaysWithCode() throws Exception
{
StorageTestSupport.setUp(config());
Integer bucketId = ConfigSync.bucketId("media");
Db.update(ObjectStorageTables.BUCKET, List.of(new QRecord().withValue("id", bucketId).withValue("publicBaseUrl", "https://evil.example.com/").withValue("trashPrefix", "x/")));
assertThat(ObjectStorageRuntime.bucketConfig("media").getPublicBaseUrl()).isEqualTo("https://cdn.example.com/media/");
ConfigSync.sync();
assertThat(Db.first(ObjectStorageTables.BUCKET, "alias", "media").getValueString("publicBaseUrl")).isEqualTo("https://cdn.example.com/media/");
}
}
@@ -0,0 +1,287 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.objectstorage.internal;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.Statement;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig;
import com.kingsrook.qbits.objectstorage.api.ObjectHeaders;
import com.kingsrook.qbits.objectstorage.config.BucketRole;
import com.kingsrook.qbits.objectstorage.config.ProviderType;
import com.kingsrook.qbits.objectstorage.config.StorageBucketConfig;
import com.kingsrook.qbits.objectstorage.config.StorageConnectionConfig;
import com.kingsrook.qbits.objectstorage.model.ObjectStorageSchema;
import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables;
import com.kingsrook.qbits.objectstorage.testsupport.StorageTestSupport;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import liquibase.Contexts;
import liquibase.Liquibase;
import liquibase.database.Database;
import liquibase.database.DatabaseFactory;
import liquibase.database.jvm.JdbcConnection;
import liquibase.resource.ClassLoaderResourceAccessor;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** The shipped starter defaults (db/liquibase/qbit-object-storage/
** object-storage-defaults.changelog.sql), run through real Liquibase on H2 in
** PostgreSQL mode against the schema ObjectStorageSchema emits: they apply,
** a second run inserts nothing and keeps an administrator's edits, a host that
** already has the rule or lacks the table is skipped (MARK_RAN), and enabling
** the seeded rule changes the headers the policy gives.
*******************************************************************************/
public class DefaultsChangelogTest
{
static final String CHANGELOG = "/db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql";
/** The tables the changelog seeds. */
public static final Set<String> SEEDED_TABLES = Set.of(ObjectStorageTables.HEADER_RULE);
/*******************************************************************************
** Tear down.
*******************************************************************************/
@AfterEach
void tearDown()
{
StorageTestSupport.tearDown();
}
/*******************************************************************************
** Fresh schema, seed twice: the same single disabled rule, and an edit survives.
*******************************************************************************/
@Test
void seedsAFreshSchemaInsertOnly() throws Exception
{
try(Connection connection = freshSchema(); Statement statement = connection.createStatement())
{
runChangelog(connection);
assertThat(scalar(connection, "SELECT COUNT(*) FROM object_storage_header_rule")).isEqualTo("1");
assertThat(scalar(connection, "SELECT name || '|' || key_glob || '|' || cache_control || '|' || priority || '|' || enabled FROM object_storage_header_rule"))
.isEqualTo("default-mutable|**|public, max-age=3600|1000|FALSE");
assertThat(scalar(connection, "SELECT bucket_id FROM object_storage_header_rule")).as("generic: not tied to a bucket").isNull();
statement.execute("UPDATE object_storage_header_rule SET cache_control = 'no-cache', enabled = TRUE WHERE name = 'default-mutable'");
runChangelog(connection);
assertThat(scalar(connection, "SELECT COUNT(*) FROM object_storage_header_rule")).isEqualTo("1");
assertThat(scalar(connection, "SELECT cache_control || '|' || enabled FROM object_storage_header_rule")).as("the edit survives").isEqualTo("no-cache|TRUE");
assertThat(ranChangesets(connection)).containsEntry("defaults-1", "EXECUTED");
}
}
/*******************************************************************************
** Rolling the seed back never deletes data: a header rule the host owns
** survives (the rollback is empty).
*******************************************************************************/
@Test
void rollbackKeepsHostOwnedRules() throws Exception
{
try(Connection connection = freshSchema(); Statement statement = connection.createStatement())
{
statement.execute("INSERT INTO object_storage_header_rule (name, key_glob, cache_control, priority, enabled, create_date, modify_date) "
+ "VALUES ('host-rule', '*.css', 'immutable', 5, TRUE, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)");
runChangelog(connection);
Database database = DatabaseFactory.getInstance().findCorrectDatabaseImplementation(new JdbcConnection(connection));
new Liquibase(CHANGELOG.substring(1), new ClassLoaderResourceAccessor(), database).rollback(1, "");
assertThat(scalar(connection, "SELECT COUNT(*) FROM object_storage_header_rule WHERE name = 'host-rule' AND enabled = TRUE")).isEqualTo("1");
assertThat(changelogText()).contains("-- rollback empty").doesNotContain("DELETE FROM");
}
}
/*******************************************************************************
** A host that already has a rule of that name keeps it untouched, and the
** changeset is marked ran. A host without the table is skipped, not failed.
*******************************************************************************/
@Test
void existingRuleAndMissingTableAreSkipped() throws Exception
{
try(Connection connection = freshSchema(); Statement statement = connection.createStatement())
{
statement.execute("INSERT INTO object_storage_header_rule (name, key_glob, cache_control, priority, enabled, create_date, modify_date) "
+ "VALUES ('default-mutable', '*.css', 'immutable', 5, TRUE, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)");
runChangelog(connection);
assertThat(scalar(connection, "SELECT COUNT(*) FROM object_storage_header_rule")).isEqualTo("1");
assertThat(scalar(connection, "SELECT key_glob || '|' || enabled FROM object_storage_header_rule")).isEqualTo("*.css|TRUE");
assertThat(ranChangesets(connection)).containsEntry("defaults-1", "MARK_RAN");
}
try(Connection connection = freshSchema(); Statement statement = connection.createStatement())
{
statement.execute("DROP TABLE object_storage_header_rule");
runChangelog(connection);
assertThat(ranChangesets(connection)).containsEntry("defaults-1", "MARK_RAN");
}
}
/*******************************************************************************
** Every changeset guards itself, uses only the sql-check precondition form,
** and inserts disabled rows only.
*******************************************************************************/
@Test
void everyChangesetIsGuarded() throws Exception
{
String text = changelogText();
String[] changesets = text.split("(?m)^-- changeset ");
assertThat(changesets).hasSize(2);
assertThat(changesets[1]).contains("-- preconditions onFail:MARK_RAN onError:MARK_RAN").contains("-- precondition-sql-check expectedResult:0 ")
.contains("-- rollback empty").contains("NOT EXISTS").contains("FALSE, CURRENT_TIMESTAMP");
assertThat(text).doesNotContain("precondition-table-exists");
for(String table : SEEDED_TABLES)
{
assertThat(text).contains("INSERT INTO " + table + " ");
}
}
/*******************************************************************************
** The seeded rule is inert until enabled (an existing host's behavior does not
** change on deploy), and enabling it gives every object the header.
*******************************************************************************/
@Test
void enablingTheSeededRuleChangesHeaders() throws Exception
{
ObjectStorageQBitConfig config = config();
StorageTestSupport.setUp(config);
StorageTestSupport.insert(ObjectStorageTables.HEADER_RULE, new QRecord().withValue("name", "default-mutable").withValue("keyGlob", "**")
.withValue("cacheControl", "public, max-age=3600").withValue("priority", 1000).withValue("enabled", false));
DefaultObjectHeaderPolicy.clearCache();
StorageBucketConfig media = config.findBucket("media");
assertThat(new DefaultObjectHeaderPolicy().headersFor(media, "any/key.bin", "application/octet-stream").isEmpty()).as("disabled: no behavior change").isTrue();
Db.update(ObjectStorageTables.HEADER_RULE, List.of(new QRecord().withValue("id", Db.first(ObjectStorageTables.HEADER_RULE, "name", "default-mutable").getValue("id")).withValue("enabled", true)));
DefaultObjectHeaderPolicy.clearCache();
ObjectHeaders headers = new DefaultObjectHeaderPolicy().headersFor(media, "any/key.bin", "application/octet-stream");
assertThat(headers.getCacheControl()).isEqualTo("public, max-age=3600");
}
/*******************************************************************************
** A small config.
*******************************************************************************/
private static ObjectStorageQBitConfig config()
{
return (new ObjectStorageQBitConfig().withBackendName(StorageTestSupport.MEMORY).withBucketScoping(false).withOpenWhenAnonymous(true).withAdminApp(true)
.withConnection(new StorageConnectionConfig("main").withProviderType(ProviderType.MINIO).withEndpoint("http://localhost:1").withRegion("us-east-1"))
.withBucket(new StorageBucketConfig("media").withConnection("main").withBucketName("media-bucket").withRole(BucketRole.PRIVATE).withIndexed(false)));
}
/*******************************************************************************
** A fresh H2 (PostgreSQL mode) with the schema the qbit emits.
*******************************************************************************/
static Connection freshSchema() throws Exception
{
Connection connection = DriverManager.getConnection("jdbc:h2:mem:osdefaults" + System.nanoTime() + ";MODE=PostgreSQL;DATABASE_TO_UPPER=FALSE;CASE_INSENSITIVE_IDENTIFIERS=TRUE;DEFAULT_NULL_ORDERING=HIGH");
try(Statement statement = connection.createStatement())
{
for(String sql : ObjectStorageSchema.postgresStatements(config()))
{
if(sql.startsWith("DO $$"))
{
// the foreign-key blocks are PL/pgSQL (Postgres only); PostgresIndexTest runs them on real Postgres
continue;
}
statement.execute(sql);
}
}
return (connection);
}
/*******************************************************************************
** Run the changelog through Liquibase, as a host does.
*******************************************************************************/
static void runChangelog(Connection connection) throws Exception
{
Database database = DatabaseFactory.getInstance().findCorrectDatabaseImplementation(new JdbcConnection(connection));
new Liquibase(CHANGELOG.substring(1), new ClassLoaderResourceAccessor(), database).update(new Contexts());
}
/*******************************************************************************
** changeset id to EXECTYPE (EXECUTED, MARK_RAN, ...).
*******************************************************************************/
static Map<String, String> ranChangesets(Connection connection) throws Exception
{
Map<String, String> rs = new HashMap<>();
try(Statement statement = connection.createStatement(); ResultSet result = statement.executeQuery("SELECT id, exectype FROM databasechangelog"))
{
while(result.next())
{
rs.put(result.getString(1), result.getString(2));
}
}
return (rs);
}
/*******************************************************************************
** The first column of the first row, as text (null when none).
*******************************************************************************/
static String scalar(Connection connection, String sql) throws Exception
{
try(Statement statement = connection.createStatement(); ResultSet result = statement.executeQuery(sql))
{
return (result.next() ? result.getString(1) : null);
}
}
/*******************************************************************************
** The changelog's text.
*******************************************************************************/
public static String changelogText() throws Exception
{
try(InputStream stream = DefaultsChangelogTest.class.getResourceAsStream(CHANGELOG))
{
assertThat(stream).as(CHANGELOG + " is on the classpath").isNotNull();
return (new String(stream.readAllBytes(), StandardCharsets.UTF_8));
}
}
}
@@ -129,6 +129,25 @@ class TablesAndConfigTest
/*******************************************************************************
** The record-page child lists filter on the parent id alone (folder ->
** objects, object -> versions and references), so the generated schema
** gives each of those columns its own index, appended so existing generated
** index names keep their numbers.
*******************************************************************************/
@Test
void testChildListColumnsAreIndexed()
{
String schema = String.join("\n", com.kingsrook.qbits.objectstorage.model.ObjectStorageSchema.postgresStatements(new ObjectStorageQBitConfig().withBackendName("rdbms")));
assertThat(schema)
.contains("ix_object_storage_object_1 ON object_storage_object (bucket_id, folder_id)")
.contains("ix_object_storage_object_7 ON object_storage_object (folder_id)")
.contains("ix_object_storage_object_version_2 ON object_storage_object_version (object_id)")
.contains("ix_object_storage_object_reference_3 ON object_storage_object_reference (object_id)");
}
/*******************************************************************************
** Owner naming rules: every table is lowercase snake_case with the full qbit
** name as prefix; no table has a field that could hold a secret value; the nav