From 25e57e03156ca30cb21cf49f5361c348c14c4b21 Mon Sep 17 00:00:00 2001 From: James Maes Date: Wed, 14 Oct 2026 16:57:43 -0500 Subject: [PATCH 1/2] feat: guided empty states, Setup card, starter header rule and bucket presign overrides Help bundle for all 12 tables and the app (ObjectStorageHelp), Setup card on the home, disabled default-mutable header rule via a jar changelog tested through real Liquibase on H2, tables guardrail test. The bucket row's presign limits are now read at runtime (ConfigSync.withBucketOverrides). The version table is documented as reserved and not used yet. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 6 + README.md | 12 + UPGRADE.md | 8 + pom.xml | 14 + .../ObjectStorageQBitProducer.java | 2 + .../qbits/objectstorage/app/StorageHome.java | 17 ++ .../objectstorage/app/StorageSetupStatus.java | 124 +++++++++ .../objectstorage/internal/ConfigSync.java | 40 +++ .../internal/ObjectStorageRuntime.java | 2 +- .../model/ObjectStorageHelp.java | 133 +++++++++ .../processes/ActionSupport.java | 2 +- .../object-storage-defaults.changelog.sql | 32 +++ .../resources/help/qbit-object-storage.yaml | 73 +++++ .../ObjectStorageTablesGuardrailTest.java | 201 ++++++++++++++ .../app/StorageSetupStatusTest.java | 108 ++++++++ .../internal/BucketRowOverridesTest.java | 127 +++++++++ .../internal/DefaultsChangelogTest.java | 261 ++++++++++++++++++ 17 files changed, 1160 insertions(+), 2 deletions(-) create mode 100644 UPGRADE.md create mode 100644 src/main/java/com/kingsrook/qbits/objectstorage/app/StorageSetupStatus.java create mode 100644 src/main/java/com/kingsrook/qbits/objectstorage/model/ObjectStorageHelp.java create mode 100644 src/main/resources/db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql create mode 100644 src/main/resources/help/qbit-object-storage.yaml create mode 100644 src/test/java/com/kingsrook/qbits/objectstorage/ObjectStorageTablesGuardrailTest.java create mode 100644 src/test/java/com/kingsrook/qbits/objectstorage/app/StorageSetupStatusTest.java create mode 100644 src/test/java/com/kingsrook/qbits/objectstorage/internal/BucketRowOverridesTest.java create mode 100644 src/test/java/com/kingsrook/qbits/objectstorage/internal/DefaultsChangelogTest.java diff --git a/CHANGELOG.md b/CHANGELOG.md index 2e26966..a5eb84d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,12 @@ All notable changes to this project are documented here. The format follows Keep ## [0.1.0] - Unreleased ### Added +- Every table explains itself: a help bundle (`help/qbit-object-storage.yaml`, QQQ help-bundle format) says what each `object_storage_*` table is for and how its rows get created; it shows on the table screen and when the table is empty. Applied by `ObjectStorageHelp` (STAND-IN for QRun-IO/qqq#1198); a host's own help is kept. +- A Setup card on the Storage home (connections, buckets, header rules, protection rules) with the state and next step of each, resolved the way the runtime resolves them. +- A starter changelog `db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql`: a disabled `default-mutable` header rule. Insert-only, run through real Liquibase in `DefaultsChangelogTest`. +- `ObjectStorageTablesGuardrailTest` (each table is used at runtime, seeded or explained, and has help). +- The presign limits on the bucket row (`maxPresignGetSeconds`, `maxPresignPutSeconds`) are now read at runtime (`ConfigSync.withBucketOverrides`), so editing them caps new presigned links and the Create download link action. Previously the row value was never read. +- `object_storage_object_version` is documented as reserved and not used yet (the indexer lists current versions only). - `StorageBucket.maxUploadBytes()` in the `api` package: the bucket's upload limit in bytes (null when none), so other qbits need not call `config()`, which returns an internal type. - **The service (P1).** `ObjectStorage.bucket(alias)` and `ObjectStorage.asSystem(caller, runId)`: put (multipart above the part size), get with ranges, head, copy (in parts for very large objects), header and metadata changes, delete (delete marker, private trash, or permanent), restore, list, presigned GET, PUT and POST, public URLs, tags. Key building and URL encoding, the header policy (`object_storage_header_rule`), `PublicUrlResolver`, `StorageChangeListener`, `StorageReferenceProvider`, `StorageAlertSink`. - Configuration in code, upserted into visible `object_storage_*` tables; secret references with an allow-list; per-connection rate limits; per-bucket and per-connection record security locks. diff --git a/README.md b/README.md index 4ead1a2..529af81 100644 --- a/README.md +++ b/README.md @@ -78,6 +78,18 @@ Every write is a standard streamed ETL: **Review** (the dry run, with summary li Permissions: `object_storage.view`, `.write`, `.delete`, `.admin` (admin implies the rest). Per-bucket scoping uses record security locks on the `object_storage_bucket` and `object_storage_connection` security key types (switch off with `withBucketScoping(false)` for a single-administrator app). +## Setup and starter defaults + +The Storage home has a **Setup** card: connections and buckets (declared in code, synced into their tables), header rules, and protection rules, each with its state and next step. Every table has help (`help/qbit-object-storage.yaml`) saying what it is for and how its rows get created, which also shows when the table is empty. + +The jar ships one optional seed changelog. After the changesets that create the tables (the script from `ObjectStorageSchema.postgresScript`), include it from your master changelog: + +```yaml +- include: { file: db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql } +``` + +It is insert-only and inserts a **disabled** generic header rule, `default-mutable` (every key, `public, max-age=3600`, priority 1000), so nothing changes on deploy: enable and edit it when you want it. Edits survive re-runs. Protection rules are not seeded (a rule needs a bucket). Presign limits on the bucket row are read at runtime: editing `maxPresignGetSeconds` or `maxPresignPutSeconds` caps every new presigned link. `object_storage_object_version` is reserved and not used yet. + ## Deployment notes - **Uploads** go through the app server (100 MB per file by default, per bucket `maxUploadBytes`), using QQQ's standard file-upload field, so the app needs `uploadedFileArchiveTableName` set in its Javalin metadata, as for any file upload. diff --git a/UPGRADE.md b/UPGRADE.md new file mode 100644 index 0000000..9997c35 --- /dev/null +++ b/UPGRADE.md @@ -0,0 +1,8 @@ +# Upgrading + +## Unreleased + +- Nothing is required. Help and the Setup card appear when the qbit is produced; the schema is unchanged. +- Optional: include `db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql` after the changesets that create the tables, to get a disabled generic header rule (`default-mutable`). It is insert-only and changes no behavior until you enable the rule. +- Behavior change: the presign limits edited on an `object_storage_bucket` row (`maxPresignGetSeconds`, `maxPresignPutSeconds`) now take effect. Rows are created from your code config and an edit is kept, so a row you edited earlier will start capping links now. Check the values on your bucket rows. +- `object_storage_object_version` stays empty: it is reserved. diff --git a/pom.xml b/pom.xml index 17dc6c3..adecfad 100644 --- a/pom.xml +++ b/pom.xml @@ -109,6 +109,20 @@ test + + + com.h2database + h2 + 2.3.232 + test + + + org.liquibase + liquibase-core + 4.33.0 + test + + com.kingsrook.qqq diff --git a/src/main/java/com/kingsrook/qbits/objectstorage/ObjectStorageQBitProducer.java b/src/main/java/com/kingsrook/qbits/objectstorage/ObjectStorageQBitProducer.java index 1ff7001..762665e 100644 --- a/src/main/java/com/kingsrook/qbits/objectstorage/ObjectStorageQBitProducer.java +++ b/src/main/java/com/kingsrook/qbits/objectstorage/ObjectStorageQBitProducer.java @@ -27,6 +27,7 @@ import com.kingsrook.qbits.objectstorage.app.StorageWidgets; import com.kingsrook.qbits.objectstorage.internal.ConfigSync; import com.kingsrook.qbits.objectstorage.internal.DefaultObjectHeaderPolicy; import com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime; +import com.kingsrook.qbits.objectstorage.model.ObjectStorageHelp; import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables; import com.kingsrook.qbits.objectstorage.processes.ObjectStorageProcesses; import com.kingsrook.qqq.backend.core.exceptions.QException; @@ -193,6 +194,7 @@ public class ObjectStorageQBitProducer implements QBitMetaDataProducer"); + setupSection(html, tablePathPrefix); bucketsSection(html, buckets, connections, caps, tablePathPrefix); healthSection(html, buckets, tablePathPrefix); recentSection(html, operations, tablePathPrefix); @@ -116,6 +117,22 @@ public final class StorageHome + /******************************************************************************* + ** 0. Setup: each configuration table, its state and the next step. + *******************************************************************************/ + private static void setupSection(StringBuilder html, String prefix) throws QException + { + open(html, "Setup"); + for(StorageSetupStatus.Step step : StorageSetupStatus.load()) + { + html.append("
").append(esc(step.label())).append(": ") + .append(esc(step.state())).append(". ").append(esc(step.next())).append("
"); + } + close(html); + } + + + /******************************************************************************* ** 1. Buckets: one card per bucket, linking to its record. Cards flow into as ** many columns as fit, so the section reads at any width without scrolling. diff --git a/src/main/java/com/kingsrook/qbits/objectstorage/app/StorageSetupStatus.java b/src/main/java/com/kingsrook/qbits/objectstorage/app/StorageSetupStatus.java new file mode 100644 index 0000000..f7e2907 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/objectstorage/app/StorageSetupStatus.java @@ -0,0 +1,124 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.objectstorage.app; + + +import java.util.ArrayList; +import java.util.List; +import com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime; +import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables; +import com.kingsrook.qqq.backend.core.actions.tables.CountAction; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; + + +/******************************************************************************* + ** The Setup card on the Storage home: each configuration table, its state and + ** the next step. States use the same resolution the runtime uses: connections + ** and buckets come from code (so "set up" means declared and synced), and the + ** header policy is the host's own when the config names one, otherwise the + ** enabled rows of object_storage_header_rule. + *******************************************************************************/ +public final class StorageSetupStatus +{ + public static final String SET_UP = "Set up"; + public static final String NEEDS_SETUP = "Needs setup"; + public static final String USING_DEFAULTS = "Using defaults"; + public static final String OPTIONAL = "Optional"; + + + + /******************************************************************************* + ** One line of the card. + *******************************************************************************/ + public record Step(String label, String state, String next, String table) + { + } + + + + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private StorageSetupStatus() + { + } + + + + /******************************************************************************* + ** Read the current state, as the viewer. + *******************************************************************************/ + public static List load() throws QException + { + return (steps(count(ObjectStorageTables.CONNECTION, new QQueryFilter()), count(ObjectStorageTables.BUCKET, new QQueryFilter()), + count(ObjectStorageTables.HEADER_RULE, new QQueryFilter(new QFilterCriteria("enabled", QCriteriaOperator.EQUALS, true))), + ObjectStorageRuntime.getConfig().getHeaderPolicy() != null, + count(ObjectStorageTables.PROTECTION_RULE, new QQueryFilter(new QFilterCriteria("enabled", QCriteriaOperator.EQUALS, true))))); + } + + + + /******************************************************************************* + ** The steps, from the numbers (pure, for tests). + *******************************************************************************/ + static List steps(int connections, int buckets, int enabledHeaderRules, boolean hostHeaderPolicy, int enabledProtectionRules) + { + List steps = new ArrayList<>(); + steps.add(connections > 0 + ? new Step("Connections", SET_UP, connections + " declared in code and synced here; set limits and timeouts on the row", ObjectStorageTables.CONNECTION) + : new Step("Connections", NEEDS_SETUP, "Declare a connection in the qbit config (withConnection) and restart: endpoint, region and the names of the key variables", ObjectStorageTables.CONNECTION)); + steps.add(buckets > 0 + ? new Step("Buckets", SET_UP, buckets + " declared in code and synced here; set the presign limits on the row", ObjectStorageTables.BUCKET) + : new Step("Buckets", NEEDS_SETUP, "Declare a bucket in the qbit config (withBucket) and restart", ObjectStorageTables.BUCKET)); + if(hostHeaderPolicy) + { + steps.add(new Step("Header rules", SET_UP, "The host's own header policy (code) decides; these rows are not read", ObjectStorageTables.HEADER_RULE)); + } + else if(enabledHeaderRules > 0) + { + steps.add(new Step("Header rules", SET_UP, enabledHeaderRules + " enabled; the first match by priority sets Cache-Control", ObjectStorageTables.HEADER_RULE)); + } + else + { + steps.add(new Step("Header rules", USING_DEFAULTS, "No rule is enabled, so objects keep the headers they were written with. Enable the starter rule or add one", ObjectStorageTables.HEADER_RULE)); + } + steps.add(enabledProtectionRules > 0 + ? new Step("Protection rules", SET_UP, enabledProtectionRules + " enabled", ObjectStorageTables.PROTECTION_RULE) + : new Step("Protection rules", OPTIONAL, "Add a rule to make a bucket prefix read-only or delete-proof", ObjectStorageTables.PROTECTION_RULE)); + return (steps); + } + + + + /******************************************************************************* + ** Count as the viewer. + *******************************************************************************/ + private static int count(String table, QQueryFilter filter) throws QException + { + Integer count = new CountAction().execute(new CountInput(table).withFilter(filter)).getCount(); + return (count == null ? 0 : count); + } + +} diff --git a/src/main/java/com/kingsrook/qbits/objectstorage/internal/ConfigSync.java b/src/main/java/com/kingsrook/qbits/objectstorage/internal/ConfigSync.java index 3ff0db2..49091a2 100644 --- a/src/main/java/com/kingsrook/qbits/objectstorage/internal/ConfigSync.java +++ b/src/main/java/com/kingsrook/qbits/objectstorage/internal/ConfigSync.java @@ -273,4 +273,44 @@ public final class ConfigSync } return (base); } + + + /******************************************************************************* + ** The bucket's presign limits edited in the UI (Settings on the bucket row), + ** applied to the code config so every presign cap reads the row. Only the + ** presign TTLs are row-owned: the public base URL, trash bucket and trash + ** prefix are identity, rewritten from code on every sync and read-only in + ** the UI, so they are never read back from the row. A missing row, an unset + ** value or a value below one second keeps the config value. + *******************************************************************************/ + public static StorageBucketConfig withBucketOverrides(StorageBucketConfig base) + { + if(base == null || !Db.available(ObjectStorageTables.BUCKET)) + { + return (base); + } + try + { + QRecord row = Db.asSystem(() -> Db.first(ObjectStorageTables.BUCKET, "alias", base.getAlias())); + if(row == null) + { + return (base); + } + Integer get = row.getValueInteger("maxPresignGetSeconds"); + Integer put = row.getValueInteger("maxPresignPutSeconds"); + if(get != null && get > 0) + { + base.setMaxPresignGetSeconds(get); + } + if(put != null && put > 0) + { + base.setMaxPresignPutSeconds(put); + } + } + catch(Exception e) + { + LOG.debug("Could not read bucket overrides", e); + } + return (base); + } } diff --git a/src/main/java/com/kingsrook/qbits/objectstorage/internal/ObjectStorageRuntime.java b/src/main/java/com/kingsrook/qbits/objectstorage/internal/ObjectStorageRuntime.java index 56ec9bd..d6afd51 100644 --- a/src/main/java/com/kingsrook/qbits/objectstorage/internal/ObjectStorageRuntime.java +++ b/src/main/java/com/kingsrook/qbits/objectstorage/internal/ObjectStorageRuntime.java @@ -163,7 +163,7 @@ public final class ObjectStorageRuntime { throw (new IllegalArgumentException("Unknown storage bucket alias: " + alias)); } - return (bucket); + return (ConfigSync.withBucketOverrides(bucket)); } diff --git a/src/main/java/com/kingsrook/qbits/objectstorage/model/ObjectStorageHelp.java b/src/main/java/com/kingsrook/qbits/objectstorage/model/ObjectStorageHelp.java new file mode 100644 index 0000000..8e16fc6 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/objectstorage/model/ObjectStorageHelp.java @@ -0,0 +1,133 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.objectstorage.model; + + +import java.io.IOException; +import java.io.InputStream; +import java.util.LinkedHashMap; +import java.util.Map; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.dataformat.yaml.YAMLFactory; +import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent; +import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; + + +/******************************************************************************* + ** Applies help/qbit-object-storage.yaml (the QQQ help-bundle format) to the + ** Storage app and its object_storage_* tables: what each table is for and how + ** its rows get created, shown on the table screen and when it is empty. + ** STAND-IN(core=QRun-IO/qqq#1198, removeBy=4.1.0-RC.8): replace with + ** HelpBundles.register once the pinned QQQ release includes it; the bundle file + ** stays the same. + *******************************************************************************/ +public final class ObjectStorageHelp +{ + public static final String RESOURCE = "/help/qbit-object-storage.yaml"; + public static final String SLOT = "help"; + + + + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private ObjectStorageHelp() + { + } + + + + /******************************************************************************* + ** The bundle's entries by key, validated (fails startup if malformed). + *******************************************************************************/ + public static Map entries() throws QException + { + try(InputStream stream = ObjectStorageHelp.class.getResourceAsStream(RESOURCE)) + { + if(stream == null) + { + throw (new IOException("missing resource")); + } + JsonNode bundle = new ObjectMapper(new YAMLFactory()).readTree(stream); + if(!"qbit-object-storage".equals(bundle.path("bundle").asText()) || !"QBIT".equals(bundle.path("layer").asText())) + { + throw (new IOException("invalid bundle header")); + } + Map entries = new LinkedHashMap<>(); + for(JsonNode entry : bundle.path("entries")) + { + String key = entry.path("key").asText(); + String summary = entry.path("summary").asText(); + if(key.isBlank() || summary.isBlank() || summary.length() > 120 || entry.path("content").asText().isBlank() || entries.put(key, entry) != null) + { + throw (new IOException("invalid or duplicate entry: " + key)); + } + } + return (entries); + } + catch(IOException | RuntimeException e) + { + throw (new QException("Could not load " + RESOURCE, e)); + } + } + + + + /******************************************************************************* + ** Apply the help to the app and every object_storage_* table in the + ** instance. A host's own help on a table or the app is kept (the bundle only + ** fills the slot). + *******************************************************************************/ + public static void apply(QInstance qInstance, ObjectStorageQBitConfig config) throws QException + { + Map entries = entries(); + for(QTableMetaData table : qInstance.getTables().values()) + { + JsonNode entry = entries.get("table:" + table.getName() + ";slot:" + SLOT); + if(entry != null && (table.getHelpContent() == null || !table.getHelpContent().containsKey(SLOT))) + { + table.withHelpContent(SLOT, content(entry)); + } + } + QAppMetaData app = qInstance.getApp(config.getAppName()); + JsonNode appEntry = entries.get("app:objectStorageAdmin;slot:" + SLOT); + if(app != null && appEntry != null && (app.getHelpContent() == null || !app.getHelpContent().containsKey(SLOT))) + { + app.withHelpContent(SLOT, content(appEntry)); + } + } + + + + /******************************************************************************* + ** Summary as the first paragraph, then the content, as markdown. + *******************************************************************************/ + static QHelpContent content(JsonNode entry) + { + return (new QHelpContent().withContentAsMarkdown(entry.path("summary").asText().strip() + "\n\n" + entry.path("content").asText().strip())); + } + +} diff --git a/src/main/java/com/kingsrook/qbits/objectstorage/processes/ActionSupport.java b/src/main/java/com/kingsrook/qbits/objectstorage/processes/ActionSupport.java index cf41e54..2e74c82 100644 --- a/src/main/java/com/kingsrook/qbits/objectstorage/processes/ActionSupport.java +++ b/src/main/java/com/kingsrook/qbits/objectstorage/processes/ActionSupport.java @@ -126,7 +126,7 @@ public final class ActionSupport *******************************************************************************/ public static StorageBucketConfig config(String alias) { - return (alias == null ? null : ObjectStorageRuntime.getConfig().findBucket(alias)); + return (alias == null ? null : ConfigSync.withBucketOverrides(ObjectStorageRuntime.getConfig().findBucket(alias))); } diff --git a/src/main/resources/db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql b/src/main/resources/db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql new file mode 100644 index 0000000..5649c5c --- /dev/null +++ b/src/main/resources/db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql @@ -0,0 +1,32 @@ +-- liquibase formatted sql + +-- qbit-object-storage starter defaults. +-- Include it from the host's master changelog AFTER the changesets that create +-- the object_storage_* tables (the script from ObjectStorageSchema.postgresScript): +-- +-- - include: { file: db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql } +-- +-- Rules for this file: +-- * insert-only: a row is inserted only when its natural key is absent, so an +-- administrator's edits always survive and nothing is ever updated; +-- * a seed never changes what an existing host does on deploy: the rows below +-- are inserted DISABLED, so they are visible examples until someone turns +-- them on; +-- * no secrets, no host-specific values; +-- * portable SQL (PostgreSQL and H2): INSERT ... SELECT ... WHERE NOT EXISTS; +-- * every changeset has a precondition, so a host that has not created the +-- table yet simply skips it (MARK_RAN). +-- DefaultsChangelogTest runs this file through Liquibase, twice, on H2. +-- +-- Not seeded: object_storage_protection_rule (a rule needs a bucket, which is +-- host-specific), and the connection and bucket rows (written from code by the +-- qbit's config sync). + +-- changeset qbit-object-storage:defaults-1 +-- preconditions onFail:MARK_RAN onError:MARK_RAN +-- precondition-sql-check expectedResult:0 SELECT COUNT(*) FROM object_storage_header_rule WHERE name = 'default-mutable' +-- comment: A generic header rule for mutable objects, disabled. Enable it to give every object (any bucket) Cache-Control public, max-age=3600; lower the priority number to rank it ahead of other rules. +INSERT INTO object_storage_header_rule (name, key_glob, cache_control, priority, enabled, create_date, modify_date) +SELECT 'default-mutable', '**', 'public, max-age=3600', 1000, FALSE, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP +WHERE NOT EXISTS (SELECT 1 FROM object_storage_header_rule WHERE name = 'default-mutable'); +-- rollback DELETE FROM object_storage_header_rule WHERE name = 'default-mutable' AND enabled = FALSE; diff --git a/src/main/resources/help/qbit-object-storage.yaml b/src/main/resources/help/qbit-object-storage.yaml new file mode 100644 index 0000000..6e59a17 --- /dev/null +++ b/src/main/resources/help/qbit-object-storage.yaml @@ -0,0 +1,73 @@ +bundle: qbit-object-storage +version: 1 +layer: QBIT +# Help for the Storage app and every object_storage_* table, in the QQQ help-bundle +# format (qqq#1198). Each table entry says what the table is for and how its rows +# get created, so an empty table explains itself. ObjectStorageHelp applies it until +# the core HelpBundles registry ships in a QQQ release (STAND-IN(core=#1198)). +# Summaries stay under 120 characters (ObjectStorageTablesGuardrailTest). +entries: + - key: app:objectStorageAdmin;slot:help + summary: "Browse and manage the S3-compatible buckets this application uses, with an index of their objects." + content: | + The Setup card on the home lists each configuration table with its state and next step. Connections and buckets are declared in code (the qbit config) and appear here after the first start; header rules and protection rules are yours to add; everything else is written by the qbit as it scans and serves. + + - key: table:object_storage_connection;slot:help + summary: "A connection to an S3-compatible service: endpoint, region, key variable names, limits and timeouts." + content: | + **How rows get created:** from code. Declare each connection in the qbit config (`withConnection`) and restart; the row is created on first start and its identity fields (endpoint, region, the names of the key variables) are rewritten from code each time. Rate limits and timeouts are yours to edit on the row and are read the next time the connection starts. No key is ever stored, only the names of the variables that hold them. Test connection checks it. + + - key: table:object_storage_connection_capability;slot:help + summary: "What a connection's provider supports (versioning, presigned POST, multipart and more), probed or assumed." + content: | + **How rows get created:** written by the qbit when a connection is tested (Test connection), starting from the provider defaults. Empty means the connection has not been probed yet. You do not edit these. + + - key: table:object_storage_bucket;slot:help + summary: "A bucket the application uses by alias: role, indexing, trash, public URL and presign limits." + content: | + **How rows get created:** from code. Declare each bucket in the qbit config (`withBucket`: alias, connection, bucket name, role) and restart; the row is created on first start and its identity (role, public base URL, trash location) is rewritten from code each time. The presign limits (maximum seconds for download and upload links) are yours to edit on the row and take effect immediately: every link is capped by them. Rescan bucket, Check configuration and Apply header policy are record actions. + + - key: table:object_storage_bucket_capability;slot:help + summary: "What a bucket supports (versioning, object lock, lifecycle, CORS and more), probed from the provider." + content: | + **How rows get created:** written by the qbit when a connection is tested or a bucket's configuration is checked (Test connection, Check configuration). Empty means the bucket has not been checked yet. You do not edit these. + + - key: table:object_storage_folder;slot:help + summary: "The folder tree of an indexed bucket, with object counts and sizes." + content: | + **How rows get created:** written by the qbit while it scans an indexed bucket and as objects are written through it. Empty means no indexed bucket has been scanned yet: run Rescan bucket on the bucket, or wait for the nightly job. Uploading files to a new folder path also creates it. + + - key: table:object_storage_object;slot:help + summary: "The searchable index of every object in the indexed buckets: key, size, type, headers and references." + content: | + **How rows get created:** written by the qbit: a full scan lists each indexed bucket (Rescan bucket, and nightly), and uploads, copies, moves and deletes made through the qbit update it at once. Empty means no indexed bucket has been scanned yet. Objects changed outside QQQ appear at the next scan. Record actions create download links, copy, move, delete and restore. + + - key: table:object_storage_object_version;slot:help + summary: "Reserved for per-object version history. Not used yet: nothing writes it." + content: | + **How rows get created:** they do not yet. The index lists only the current version of each object, so this table is always empty and is reserved for a later release that indexes version history. A versioned bucket still keeps its versions at the provider; deleting through the qbit writes a delete marker there. + + - key: table:object_storage_object_reference;slot:help + summary: "Which application records point at which objects, to find orphaned and missing files." + content: | + **How rows get created:** written by the reference scan when a bucket is marked `referenceTracked` and the host registers a reference provider (for example `FieldReferenceProvider` for a table field that holds a key). Empty means no reference provider is registered or no scan has run, so orphaned and missing counts are not available. + + - key: table:object_storage_header_rule;slot:help + summary: "Cache-Control and Content-Disposition rules by key pattern and type, first match by priority wins." + content: | + **How rows get created:** an administrator adds them with New, or the qbit's starter changelog adds a disabled `default-mutable` rule (all keys, one hour) that you can enable and edit. Only enabled rules are used, lowest priority number first. A rule can be limited to one bucket. They set the headers on every upload and on Apply header policy. If the host configures its own header policy in code, these rows are not read. Backup buckets are never touched. + + - key: table:object_storage_protection_rule;slot:help + summary: "Make a bucket prefix read-only or delete-proof, or only warn when it is changed." + content: | + **How rows get created:** an administrator adds one with New: pick the bucket (required), a prefix pattern and a mode (Read only, No delete or Warn). Empty is fine: no prefix is protected until you add a rule. Enabled rules are checked on every write and delete made through the qbit. + + - key: table:object_storage_operation;slot:help + summary: "The log of every write, delete, link and scan the qbit performed, with who, what and the result." + content: | + **How rows get created:** written by the qbit on every operation. Empty means nothing has been done through the qbit yet. Presigned URLs are never logged, only the key, the expiry and who asked. + + - key: table:object_storage_usage_daily;slot:help + summary: "A daily snapshot of object count and bytes per indexed bucket, for the usage trend." + content: | + **How rows get created:** written by the nightly job, one row per indexed bucket per day. Empty means the nightly job has not run yet, or no bucket is indexed. diff --git a/src/test/java/com/kingsrook/qbits/objectstorage/ObjectStorageTablesGuardrailTest.java b/src/test/java/com/kingsrook/qbits/objectstorage/ObjectStorageTablesGuardrailTest.java new file mode 100644 index 0000000..9bdfb3c --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/objectstorage/ObjectStorageTablesGuardrailTest.java @@ -0,0 +1,201 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.objectstorage; + + +import java.lang.reflect.Field; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.List; +import java.util.Map; +import java.util.stream.Stream; +import com.kingsrook.qbits.objectstorage.internal.DefaultsChangelogTest; +import com.kingsrook.qbits.objectstorage.model.ObjectStorageHelp; +import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables; +import com.kingsrook.qbits.objectstorage.testsupport.StorageTestSupport; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.metadata.help.QHelpContent; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; + + +/******************************************************************************* + ** Guardrail (owner requirement 2026-10-14: "all tables should be populated and + ** the qbit should be fully using them"). For every object_storage_* table: + ** - runtime code (not just the model, metadata or home screen) reads or writes + ** it, or it is listed in NOT_USED_YET with what is missing; + ** - it is either seeded by the shipped defaults changelog, or listed in + ** STARTS_EMPTY with why; + ** - it has bundle help that says how its rows get created (shown on the table + ** screen and when it is empty). + ** A new table fails this test until it is wired, seeded or explained. + *******************************************************************************/ +class ObjectStorageTablesGuardrailTest +{ + private static final Path MAIN = Path.of("src", "main", "java", "com", "kingsrook", "qbits", "objectstorage"); + + /******************************************************************************* + ** Tables that no runtime code writes yet, and what is missing. Their help says + ** "Not used yet". + *******************************************************************************/ + static final Map NOT_USED_YET = Map.of( + ObjectStorageTables.OBJECT_VERSION, "the indexer lists current versions only (ListObjectsV2); indexing version history is a later release"); + + /******************************************************************************* + ** Tables that legitimately start empty, and where their rows come from. + *******************************************************************************/ + static final Map STARTS_EMPTY = Map.ofEntries( + Map.entry(ObjectStorageTables.CONNECTION, "written from code by the config sync: the host declares its connections"), + Map.entry(ObjectStorageTables.BUCKET, "written from code by the config sync: the host declares its buckets"), + Map.entry(ObjectStorageTables.CONNECTION_CAPABILITY, "runtime: Test connection probes the provider"), + Map.entry(ObjectStorageTables.BUCKET_CAPABILITY, "runtime: Test connection and Check configuration probe the bucket"), + Map.entry(ObjectStorageTables.FOLDER, "runtime: the index scan and uploads"), + Map.entry(ObjectStorageTables.OBJECT, "runtime: the index scan and write-through"), + Map.entry(ObjectStorageTables.OBJECT_VERSION, "not used yet: see NOT_USED_YET"), + Map.entry(ObjectStorageTables.OBJECT_REFERENCE, "runtime: the reference scan, when the host registers a reference provider"), + Map.entry(ObjectStorageTables.PROTECTION_RULE, "a rule needs a bucket, which is host-specific: an administrator adds each rule"), + Map.entry(ObjectStorageTables.OPERATION, "log: every operation the qbit performs"), + Map.entry(ObjectStorageTables.USAGE_DAILY, "runtime: the nightly job")); + + + + /******************************************************************************* + ** Tear down. + *******************************************************************************/ + @AfterEach + void tearDown() + { + StorageTestSupport.tearDown(); + } + + + + /******************************************************************************* + ** Every table is read or written by runtime code, or explicitly not used yet. + *******************************************************************************/ + @Test + void everyTableIsUsedAtRuntime() throws Exception + { + List runtimeSource; + try(Stream files = Files.walk(MAIN)) + { + runtimeSource = files.filter(p -> p.toString().endsWith(".java")) + .filter(p -> !p.startsWith(MAIN.resolve("model")) && !p.startsWith(MAIN.resolve("app"))) + .map(ObjectStorageTablesGuardrailTest::read).toList(); + } + for(String table : ObjectStorageTables.ALL) + { + String constant = "ObjectStorageTables." + constantFor(table); + boolean used = runtimeSource.stream().anyMatch(source -> source.contains(constant)); + if(NOT_USED_YET.containsKey(table)) + { + assertThat(used).as(table + " is listed as not used yet, but runtime code uses it: remove it from NOT_USED_YET").isFalse(); + } + else + { + assertThat(used).as(table + " must be read or written by runtime code, not only declared").isTrue(); + } + } + } + + + + /******************************************************************************* + ** Every table is seeded or explained. + *******************************************************************************/ + @Test + void everyTableIsSeededOrExplained() throws Exception + { + String changelog = DefaultsChangelogTest.changelogText(); + for(String table : ObjectStorageTables.ALL) + { + boolean seeded = DefaultsChangelogTest.SEEDED_TABLES.contains(table); + assertThat(seeded || STARTS_EMPTY.containsKey(table)).as(table + " must be seeded by the defaults changelog or listed in STARTS_EMPTY").isTrue(); + assertThat(seeded && STARTS_EMPTY.containsKey(table)).as(table + " cannot be both seeded and listed as starting empty").isFalse(); + assertThat(changelog.contains("INSERT INTO " + table + " ")).as(table + " seeded iff the changelog inserts it").isEqualTo(seeded); + } + assertThat(STARTS_EMPTY.keySet()).allMatch(ObjectStorageTables.ALL::contains, "STARTS_EMPTY names only real tables"); + assertThat(NOT_USED_YET.keySet()).allMatch(ObjectStorageTables.ALL::contains, "NOT_USED_YET names only real tables"); + } + + + + /******************************************************************************* + ** Every table and the app have bundle help that says how rows get created, + ** and the not-yet-used tables say so. + *******************************************************************************/ + @Test + void everyTableHasEmptyStateHelp() throws Exception + { + QInstance instance = StorageTestSupport.setUp(new ObjectStorageQBitConfig().withBackendName(StorageTestSupport.MEMORY).withBucketScoping(false).withOpenWhenAnonymous(true).withAdminApp(true)); + assertThat(ObjectStorageHelp.entries()).allSatisfy((key, entry) -> assertThat(entry.path("summary").asText().length()).as(key).isLessThanOrEqualTo(120)); + for(String table : ObjectStorageTables.ALL) + { + List help = instance.getTable(table).getHelpContent() == null ? null : instance.getTable(table).getHelpContent().get(ObjectStorageHelp.SLOT); + assertThat(help).as(table + " table help").isNotEmpty(); + assertThat(help.get(0).getContent()).as(table + " help says how rows get created").contains("How rows get created"); + if(NOT_USED_YET.containsKey(table)) + { + assertThat(help.get(0).getContent()).as(table + " help says it is not used yet").contains("Not used yet"); + } + } + assertThat(instance.getApp("objectStorageAdmin").getHelpContent()).containsKey(ObjectStorageHelp.SLOT); + assertThat(ObjectStorageHelp.entries().keySet()).allMatch(key -> !key.startsWith("table:") || instance.getTable(key.substring(6, key.indexOf(';'))) != null, + "help names only real tables"); + } + + + + /******************************************************************************* + ** The ObjectStorageTables constant name of a table. + *******************************************************************************/ + private static String constantFor(String table) throws Exception + { + for(Field field : ObjectStorageTables.class.getFields()) + { + if(field.getType() == String.class && table.equals(field.get(null)) && !field.getName().startsWith("KEY_") && !field.getName().startsWith("PVS_") && !field.getName().startsWith("ALL_")) + { + return (field.getName()); + } + } + throw (new IllegalStateException("no constant for " + table)); + } + + + + /******************************************************************************* + ** Read a file. + *******************************************************************************/ + private static String read(Path path) + { + try + { + return (Files.readString(path, StandardCharsets.UTF_8)); + } + catch(Exception e) + { + throw (new IllegalStateException(e)); + } + } + +} diff --git a/src/test/java/com/kingsrook/qbits/objectstorage/app/StorageSetupStatusTest.java b/src/test/java/com/kingsrook/qbits/objectstorage/app/StorageSetupStatusTest.java new file mode 100644 index 0000000..a10ea9c --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/objectstorage/app/StorageSetupStatusTest.java @@ -0,0 +1,108 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.objectstorage.app; + + +import java.util.List; +import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig; +import com.kingsrook.qbits.objectstorage.api.ObjectHeaderPolicy; +import com.kingsrook.qbits.objectstorage.app.StorageSetupStatus.Step; +import com.kingsrook.qbits.objectstorage.config.BucketRole; +import com.kingsrook.qbits.objectstorage.config.ProviderType; +import com.kingsrook.qbits.objectstorage.config.StorageBucketConfig; +import com.kingsrook.qbits.objectstorage.config.StorageConnectionConfig; +import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables; +import com.kingsrook.qbits.objectstorage.testsupport.StorageTestSupport; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; + + +/******************************************************************************* + ** The Setup card: states follow what is configured, resolved the way the + ** runtime resolves it (a host's own header policy beats the rows). + *******************************************************************************/ +class StorageSetupStatusTest +{ + /******************************************************************************* + ** Tear down. + *******************************************************************************/ + @AfterEach + void tearDown() + { + StorageTestSupport.tearDown(); + } + + + + /******************************************************************************* + ** A bare install needs a connection and a bucket; headers use defaults; the + ** protection rules are optional. + *******************************************************************************/ + @Test + void bareInstall() + { + List steps = StorageSetupStatus.steps(0, 0, 0, false, 0); + assertThat(steps).extracting(Step::label).containsExactly("Connections", "Buckets", "Header rules", "Protection rules"); + assertThat(steps).extracting(Step::state).containsExactly("Needs setup", "Needs setup", "Using defaults", "Optional"); + assertThat(steps).allSatisfy(step -> assertThat(step.next()).isNotBlank()); + } + + + + /******************************************************************************* + ** A host policy in code means the rows are not what decides, so it is never + ** reported as needing setup. + *******************************************************************************/ + @Test + void hostPolicyIsRespected() + { + assertThat(StorageSetupStatus.steps(1, 1, 0, true, 0).get(2)).satisfies(step -> + { + assertThat(step.state()).isEqualTo("Set up"); + assertThat(step.next()).contains("host's own header policy"); + }); + assertThat(StorageSetupStatus.steps(1, 1, 2, false, 1)).extracting(Step::state).containsOnly("Set up"); + } + + + + /******************************************************************************* + ** The live card reads the tables and the config. + *******************************************************************************/ + @Test + void liveStatus() throws Exception + { + ObjectStorageQBitConfig config = new ObjectStorageQBitConfig().withBackendName(StorageTestSupport.MEMORY).withBucketScoping(false).withOpenWhenAnonymous(true).withAdminApp(true) + .withConnection(new StorageConnectionConfig("main").withProviderType(ProviderType.MINIO).withEndpoint("http://localhost:1").withRegion("us-east-1")) + .withBucket(new StorageBucketConfig("media").withConnection("main").withBucketName("media-bucket").withRole(BucketRole.PRIVATE).withIndexed(false)); + StorageTestSupport.setUp(config); + assertThat(StorageSetupStatus.load()).extracting(Step::state).containsExactly("Set up", "Set up", "Using defaults", "Optional"); + + StorageTestSupport.insert(ObjectStorageTables.HEADER_RULE, new QRecord().withValue("name", "r").withValue("keyGlob", "*").withValue("cacheControl", "no-cache").withValue("priority", 1).withValue("enabled", true)); + assertThat(StorageSetupStatus.load().get(2).state()).isEqualTo("Set up"); + + ObjectHeaderPolicy hostPolicy = (bucket, key, contentType) -> new com.kingsrook.qbits.objectstorage.api.ObjectHeaders(); + config.withHeaderPolicy(hostPolicy); + assertThat(StorageSetupStatus.load().get(2).next()).contains("host's own"); + } +} diff --git a/src/test/java/com/kingsrook/qbits/objectstorage/internal/BucketRowOverridesTest.java b/src/test/java/com/kingsrook/qbits/objectstorage/internal/BucketRowOverridesTest.java new file mode 100644 index 0000000..f7ba5d3 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/objectstorage/internal/BucketRowOverridesTest.java @@ -0,0 +1,127 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.objectstorage.internal; + + +import java.net.URL; +import java.time.Duration; +import java.util.List; +import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig; +import com.kingsrook.qbits.objectstorage.api.ObjectStorage; +import com.kingsrook.qbits.objectstorage.api.PresignGet; +import com.kingsrook.qbits.objectstorage.api.PresignPut; +import com.kingsrook.qbits.objectstorage.config.BucketRole; +import com.kingsrook.qbits.objectstorage.config.ProviderType; +import com.kingsrook.qbits.objectstorage.config.StorageBucketConfig; +import com.kingsrook.qbits.objectstorage.config.StorageConnectionConfig; +import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables; +import com.kingsrook.qbits.objectstorage.processes.ActionSupport; +import com.kingsrook.qbits.objectstorage.testsupport.StorageTestSupport; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; + + +/******************************************************************************* + ** The presign limits on the bucket row (Settings on object_storage_bucket) are + ** read at runtime, the way ConfigSync.withOverrides reads a connection row: + ** editing the row changes how long presigned links last, and the identity + ** fields stay owned by code. + *******************************************************************************/ +class BucketRowOverridesTest +{ + /******************************************************************************* + ** Tear down. + *******************************************************************************/ + @AfterEach + void tearDown() + { + StorageTestSupport.tearDown(); + } + + + + /******************************************************************************* + ** A config whose buckets never touch a real store (presigning is offline). + *******************************************************************************/ + private static ObjectStorageQBitConfig config() + { + System.setProperty("os.unit.ak", "ak"); + System.setProperty("os.unit.sk", "sk"); + return (new ObjectStorageQBitConfig().withBackendName(StorageTestSupport.MEMORY).withBucketScoping(false).withOpenWhenAnonymous(true).withAdminApp(true) + .withAllowedSecretRefPatterns(List.of("prop.os.unit.*")) + .withConnection(new StorageConnectionConfig("main").withProviderType(ProviderType.MINIO).withEndpoint("http://localhost:1").withRegion("us-east-1") + .withAccessKeyRef("${prop.os.unit.ak}").withSecretKeyRef("${prop.os.unit.sk}")) + .withBucket(new StorageBucketConfig("media").withConnection("main").withBucketName("media-bucket").withRole(BucketRole.PRIVATE).withIndexed(false) + .withPublicBaseUrl("https://cdn.example.com/media/"))); + } + + + + /******************************************************************************* + ** The presigned URL's lifetime in seconds. + *******************************************************************************/ + private static int expires(URL url) + { + return (Integer.parseInt(url.getQuery().replaceAll(".*X-Amz-Expires=(\\d+).*", "$1"))); + } + + + + /******************************************************************************* + ** Editing the row's presign limits caps new links; the config value applies + ** until then; a nonsense value (zero) is ignored. + *******************************************************************************/ + @Test + void rowPresignLimitsCapNewLinks() throws Exception + { + StorageTestSupport.setUp(config()); + Integer bucketId = ConfigSync.bucketId("media"); + + assertThat(expires(ObjectStorage.bucket("media").presignGet(PresignGet.of("a.txt").ttl(Duration.ofMinutes(30))))).as("the config limit (an hour)").isEqualTo(1800); + + Db.update(ObjectStorageTables.BUCKET, List.of(new QRecord().withValue("id", bucketId).withValue("maxPresignGetSeconds", 60).withValue("maxPresignPutSeconds", 120))); + assertThat(expires(ObjectStorage.bucket("media").presignGet(PresignGet.of("a.txt").ttl(Duration.ofMinutes(30))))).as("the row's limit caps the link").isEqualTo(60); + assertThat(expires(ObjectStorage.bucket("media").presignPut(PresignPut.of("a.txt").ttl(Duration.ofMinutes(10))))).isEqualTo(120); + assertThat(ActionSupport.config("media").getMaxPresignGetSeconds()).as("the Create download link action reads it too").isEqualTo(60); + + Db.update(ObjectStorageTables.BUCKET, List.of(new QRecord().withValue("id", bucketId).withValue("maxPresignGetSeconds", 0))); + assertThat(ObjectStorageRuntime.bucketConfig("media").getMaxPresignGetSeconds()).as("zero is ignored").isEqualTo(60); + } + + + + /******************************************************************************* + ** Identity fields stay code-owned: a tampered public base URL on the row is + ** rewritten from code and never read back as an override. + *******************************************************************************/ + @Test + void identityStaysWithCode() throws Exception + { + StorageTestSupport.setUp(config()); + Integer bucketId = ConfigSync.bucketId("media"); + Db.update(ObjectStorageTables.BUCKET, List.of(new QRecord().withValue("id", bucketId).withValue("publicBaseUrl", "https://evil.example.com/").withValue("trashPrefix", "x/"))); + assertThat(ObjectStorageRuntime.bucketConfig("media").getPublicBaseUrl()).isEqualTo("https://cdn.example.com/media/"); + ConfigSync.sync(); + assertThat(Db.first(ObjectStorageTables.BUCKET, "alias", "media").getValueString("publicBaseUrl")).isEqualTo("https://cdn.example.com/media/"); + } +} diff --git a/src/test/java/com/kingsrook/qbits/objectstorage/internal/DefaultsChangelogTest.java b/src/test/java/com/kingsrook/qbits/objectstorage/internal/DefaultsChangelogTest.java new file mode 100644 index 0000000..82d5b02 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/objectstorage/internal/DefaultsChangelogTest.java @@ -0,0 +1,261 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.objectstorage.internal; + + +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.ResultSet; +import java.sql.Statement; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig; +import com.kingsrook.qbits.objectstorage.api.ObjectHeaders; +import com.kingsrook.qbits.objectstorage.config.BucketRole; +import com.kingsrook.qbits.objectstorage.config.ProviderType; +import com.kingsrook.qbits.objectstorage.config.StorageBucketConfig; +import com.kingsrook.qbits.objectstorage.config.StorageConnectionConfig; +import com.kingsrook.qbits.objectstorage.model.ObjectStorageSchema; +import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables; +import com.kingsrook.qbits.objectstorage.testsupport.StorageTestSupport; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import liquibase.Contexts; +import liquibase.Liquibase; +import liquibase.database.Database; +import liquibase.database.DatabaseFactory; +import liquibase.database.jvm.JdbcConnection; +import liquibase.resource.ClassLoaderResourceAccessor; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; + + +/******************************************************************************* + ** The shipped starter defaults (db/liquibase/qbit-object-storage/ + ** object-storage-defaults.changelog.sql), run through real Liquibase on H2 in + ** PostgreSQL mode against the schema ObjectStorageSchema emits: they apply, + ** a second run inserts nothing and keeps an administrator's edits, a host that + ** already has the rule or lacks the table is skipped (MARK_RAN), and enabling + ** the seeded rule changes the headers the policy gives. + *******************************************************************************/ +public class DefaultsChangelogTest +{ + static final String CHANGELOG = "/db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql"; + + /** The tables the changelog seeds. */ + public static final Set SEEDED_TABLES = Set.of(ObjectStorageTables.HEADER_RULE); + + + + /******************************************************************************* + ** Tear down. + *******************************************************************************/ + @AfterEach + void tearDown() + { + StorageTestSupport.tearDown(); + } + + + + /******************************************************************************* + ** Fresh schema, seed twice: the same single disabled rule, and an edit survives. + *******************************************************************************/ + @Test + void seedsAFreshSchemaInsertOnly() throws Exception + { + try(Connection connection = freshSchema(); Statement statement = connection.createStatement()) + { + runChangelog(connection); + assertThat(scalar(connection, "SELECT COUNT(*) FROM object_storage_header_rule")).isEqualTo("1"); + assertThat(scalar(connection, "SELECT name || '|' || key_glob || '|' || cache_control || '|' || priority || '|' || enabled FROM object_storage_header_rule")) + .isEqualTo("default-mutable|**|public, max-age=3600|1000|FALSE"); + assertThat(scalar(connection, "SELECT bucket_id FROM object_storage_header_rule")).as("generic: not tied to a bucket").isNull(); + + statement.execute("UPDATE object_storage_header_rule SET cache_control = 'no-cache', enabled = TRUE WHERE name = 'default-mutable'"); + runChangelog(connection); + assertThat(scalar(connection, "SELECT COUNT(*) FROM object_storage_header_rule")).isEqualTo("1"); + assertThat(scalar(connection, "SELECT cache_control || '|' || enabled FROM object_storage_header_rule")).as("the edit survives").isEqualTo("no-cache|TRUE"); + assertThat(ranChangesets(connection)).containsEntry("defaults-1", "EXECUTED"); + } + } + + + + /******************************************************************************* + ** A host that already has a rule of that name keeps it untouched, and the + ** changeset is marked ran. A host without the table is skipped, not failed. + *******************************************************************************/ + @Test + void existingRuleAndMissingTableAreSkipped() throws Exception + { + try(Connection connection = freshSchema(); Statement statement = connection.createStatement()) + { + statement.execute("INSERT INTO object_storage_header_rule (name, key_glob, cache_control, priority, enabled, create_date, modify_date) " + + "VALUES ('default-mutable', '*.css', 'immutable', 5, TRUE, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)"); + runChangelog(connection); + assertThat(scalar(connection, "SELECT COUNT(*) FROM object_storage_header_rule")).isEqualTo("1"); + assertThat(scalar(connection, "SELECT key_glob || '|' || enabled FROM object_storage_header_rule")).isEqualTo("*.css|TRUE"); + assertThat(ranChangesets(connection)).containsEntry("defaults-1", "MARK_RAN"); + } + try(Connection connection = freshSchema(); Statement statement = connection.createStatement()) + { + statement.execute("DROP TABLE object_storage_header_rule"); + runChangelog(connection); + assertThat(ranChangesets(connection)).containsEntry("defaults-1", "MARK_RAN"); + } + } + + + + /******************************************************************************* + ** Every changeset guards itself, uses only the sql-check precondition form, + ** and inserts disabled rows only. + *******************************************************************************/ + @Test + void everyChangesetIsGuarded() throws Exception + { + String text = changelogText(); + String[] changesets = text.split("(?m)^-- changeset "); + assertThat(changesets).hasSize(2); + assertThat(changesets[1]).contains("-- preconditions onFail:MARK_RAN onError:MARK_RAN").contains("-- precondition-sql-check expectedResult:0 ") + .contains("-- rollback ").contains("NOT EXISTS").contains("FALSE, CURRENT_TIMESTAMP"); + assertThat(text).doesNotContain("precondition-table-exists"); + for(String table : SEEDED_TABLES) + { + assertThat(text).contains("INSERT INTO " + table + " "); + } + } + + + + /******************************************************************************* + ** The seeded rule is inert until enabled (an existing host's behavior does not + ** change on deploy), and enabling it gives every object the header. + *******************************************************************************/ + @Test + void enablingTheSeededRuleChangesHeaders() throws Exception + { + ObjectStorageQBitConfig config = config(); + StorageTestSupport.setUp(config); + StorageTestSupport.insert(ObjectStorageTables.HEADER_RULE, new QRecord().withValue("name", "default-mutable").withValue("keyGlob", "**") + .withValue("cacheControl", "public, max-age=3600").withValue("priority", 1000).withValue("enabled", false)); + DefaultObjectHeaderPolicy.clearCache(); + StorageBucketConfig media = config.findBucket("media"); + + assertThat(new DefaultObjectHeaderPolicy().headersFor(media, "any/key.bin", "application/octet-stream").isEmpty()).as("disabled: no behavior change").isTrue(); + + Db.update(ObjectStorageTables.HEADER_RULE, List.of(new QRecord().withValue("id", Db.first(ObjectStorageTables.HEADER_RULE, "name", "default-mutable").getValue("id")).withValue("enabled", true))); + DefaultObjectHeaderPolicy.clearCache(); + ObjectHeaders headers = new DefaultObjectHeaderPolicy().headersFor(media, "any/key.bin", "application/octet-stream"); + assertThat(headers.getCacheControl()).isEqualTo("public, max-age=3600"); + } + + + + /******************************************************************************* + ** A small config. + *******************************************************************************/ + private static ObjectStorageQBitConfig config() + { + return (new ObjectStorageQBitConfig().withBackendName(StorageTestSupport.MEMORY).withBucketScoping(false).withOpenWhenAnonymous(true).withAdminApp(true) + .withConnection(new StorageConnectionConfig("main").withProviderType(ProviderType.MINIO).withEndpoint("http://localhost:1").withRegion("us-east-1")) + .withBucket(new StorageBucketConfig("media").withConnection("main").withBucketName("media-bucket").withRole(BucketRole.PRIVATE).withIndexed(false))); + } + + + + /******************************************************************************* + ** A fresh H2 (PostgreSQL mode) with the schema the qbit emits. + *******************************************************************************/ + static Connection freshSchema() throws Exception + { + Connection connection = DriverManager.getConnection("jdbc:h2:mem:osdefaults" + System.nanoTime() + ";MODE=PostgreSQL;DATABASE_TO_UPPER=FALSE;CASE_INSENSITIVE_IDENTIFIERS=TRUE;DEFAULT_NULL_ORDERING=HIGH"); + try(Statement statement = connection.createStatement()) + { + for(String sql : ObjectStorageSchema.postgresStatements(config())) + { + statement.execute(sql); + } + } + return (connection); + } + + + + /******************************************************************************* + ** Run the changelog through Liquibase, as a host does. + *******************************************************************************/ + static void runChangelog(Connection connection) throws Exception + { + Database database = DatabaseFactory.getInstance().findCorrectDatabaseImplementation(new JdbcConnection(connection)); + new Liquibase(CHANGELOG.substring(1), new ClassLoaderResourceAccessor(), database).update(new Contexts()); + } + + + + /******************************************************************************* + ** changeset id to EXECTYPE (EXECUTED, MARK_RAN, ...). + *******************************************************************************/ + static Map ranChangesets(Connection connection) throws Exception + { + Map rs = new HashMap<>(); + try(Statement statement = connection.createStatement(); ResultSet result = statement.executeQuery("SELECT id, exectype FROM databasechangelog")) + { + while(result.next()) + { + rs.put(result.getString(1), result.getString(2)); + } + } + return (rs); + } + + + + /******************************************************************************* + ** The first column of the first row, as text (null when none). + *******************************************************************************/ + static String scalar(Connection connection, String sql) throws Exception + { + try(Statement statement = connection.createStatement(); ResultSet result = statement.executeQuery(sql)) + { + return (result.next() ? result.getString(1) : null); + } + } + + + + /******************************************************************************* + ** The changelog's text. + *******************************************************************************/ + public static String changelogText() throws Exception + { + try(InputStream stream = DefaultsChangelogTest.class.getResourceAsStream(CHANGELOG)) + { + assertThat(stream).as(CHANGELOG + " is on the classpath").isNotNull(); + return (new String(stream.readAllBytes(), StandardCharsets.UTF_8)); + } + } +} -- 2.54.0 From e9fd0cef4cbf2d77d95505dc6413a284d9725def Mon Sep 17 00:00:00 2001 From: James Maes Date: Wed, 14 Oct 2026 17:10:53 -0500 Subject: [PATCH 2/2] fix: presign row limits are a cached lower cap, no config mutation; setup card reads runtime config; empty rollback - effective limit is min(code, row); row 0/null/higher is ignored; config never mutated - applied to presignGet/Put/Post, previewUrl and the download-link action; row read cached 15 s - Setup card takes connections and buckets from the runtime config and counts rules as system - seed changeset rollback is empty; test that a host rule survives rollback - host help kept test, stand-in marker prose in the bundle Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 +- README.md | 2 +- UPGRADE.md | 2 +- .../objectstorage/api/StorageBucket.java | 9 +- .../objectstorage/app/StorageSetupStatus.java | 31 ++---- .../objectstorage/internal/ConfigSync.java | 102 +++++++++++++----- .../internal/ObjectStorageRuntime.java | 2 +- .../processes/ActionSupport.java | 2 +- .../processes/BucketProcesses.java | 3 +- .../processes/LinkProcesses.java | 3 +- .../object-storage-defaults.changelog.sql | 4 +- .../resources/help/qbit-object-storage.yaml | 4 +- .../ObjectStorageTablesGuardrailTest.java | 17 +++ .../app/StorageSetupStatusTest.java | 18 ++++ .../internal/BucketRowOverridesTest.java | 41 +++++-- .../internal/DefaultsChangelogTest.java | 23 +++- 16 files changed, 192 insertions(+), 73 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a5eb84d..87abcbe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ All notable changes to this project are documented here. The format follows Keep - A Setup card on the Storage home (connections, buckets, header rules, protection rules) with the state and next step of each, resolved the way the runtime resolves them. - A starter changelog `db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql`: a disabled `default-mutable` header rule. Insert-only, run through real Liquibase in `DefaultsChangelogTest`. - `ObjectStorageTablesGuardrailTest` (each table is used at runtime, seeded or explained, and has help). -- The presign limits on the bucket row (`maxPresignGetSeconds`, `maxPresignPutSeconds`) are now read at runtime (`ConfigSync.withBucketOverrides`), so editing them caps new presigned links and the Create download link action. Previously the row value was never read. +- The presign limits on the bucket row (`maxPresignGetSeconds`, `maxPresignPutSeconds`) are now read at runtime (`ConfigSync.effectiveMaxPresignGetSeconds` / `Put`) as a lower cap: the effective limit is the smaller of the code config and the row. It applies to presigned GET, PUT, POST, previews and the Create download link action. Previously the row value was never read. - `object_storage_object_version` is documented as reserved and not used yet (the indexer lists current versions only). - `StorageBucket.maxUploadBytes()` in the `api` package: the bucket's upload limit in bytes (null when none), so other qbits need not call `config()`, which returns an internal type. - **The service (P1).** `ObjectStorage.bucket(alias)` and `ObjectStorage.asSystem(caller, runId)`: put (multipart above the part size), get with ranges, head, copy (in parts for very large objects), header and metadata changes, delete (delete marker, private trash, or permanent), restore, list, presigned GET, PUT and POST, public URLs, tags. Key building and URL encoding, the header policy (`object_storage_header_rule`), `PublicUrlResolver`, `StorageChangeListener`, `StorageReferenceProvider`, `StorageAlertSink`. diff --git a/README.md b/README.md index 529af81..70ed397 100644 --- a/README.md +++ b/README.md @@ -88,7 +88,7 @@ The jar ships one optional seed changelog. After the changesets that create the - include: { file: db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql } ``` -It is insert-only and inserts a **disabled** generic header rule, `default-mutable` (every key, `public, max-age=3600`, priority 1000), so nothing changes on deploy: enable and edit it when you want it. Edits survive re-runs. Protection rules are not seeded (a rule needs a bucket). Presign limits on the bucket row are read at runtime: editing `maxPresignGetSeconds` or `maxPresignPutSeconds` caps every new presigned link. `object_storage_object_version` is reserved and not used yet. +It is insert-only and inserts a **disabled** generic header rule, `default-mutable` (every key, `public, max-age=3600`, priority 1000), so nothing changes on deploy: enable and edit it when you want it. Edits survive re-runs. Protection rules are not seeded (a rule needs a bucket). Presign limits on the bucket row are read at runtime: `maxPresignGetSeconds` and `maxPresignPutSeconds` on the bucket row can lower the code config limits (the effective limit is the smaller of the two, cached up to 15 seconds); the row cannot raise them. `object_storage_object_version` is reserved and not used yet. ## Deployment notes diff --git a/UPGRADE.md b/UPGRADE.md index 9997c35..f1821dc 100644 --- a/UPGRADE.md +++ b/UPGRADE.md @@ -4,5 +4,5 @@ - Nothing is required. Help and the Setup card appear when the qbit is produced; the schema is unchanged. - Optional: include `db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql` after the changesets that create the tables, to get a disabled generic header rule (`default-mutable`). It is insert-only and changes no behavior until you enable the rule. -- Behavior change: the presign limits edited on an `object_storage_bucket` row (`maxPresignGetSeconds`, `maxPresignPutSeconds`) now take effect. Rows are created from your code config and an edit is kept, so a row you edited earlier will start capping links now. Check the values on your bucket rows. +- Behavior change: the presign limits on an `object_storage_bucket` row (`maxPresignGetSeconds`, `maxPresignPutSeconds`) are now read, as a lower cap only. Any bucket row whose presign values are lower than your code config shortens links after this upgrade (the config sync copied the code values into the row on first start, so a row someone lowered earlier counts). Check the values on your bucket rows; 0 or empty means no row limit, and a value above the code limit has no effect. - `object_storage_object_version` stays empty: it is reserved. diff --git a/src/main/java/com/kingsrook/qbits/objectstorage/api/StorageBucket.java b/src/main/java/com/kingsrook/qbits/objectstorage/api/StorageBucket.java index ea5d573..c3e31ee 100644 --- a/src/main/java/com/kingsrook/qbits/objectstorage/api/StorageBucket.java +++ b/src/main/java/com/kingsrook/qbits/objectstorage/api/StorageBucket.java @@ -36,6 +36,7 @@ import com.kingsrook.qbits.objectstorage.api.StorageException.Reason; import com.kingsrook.qbits.objectstorage.config.BucketRole; import com.kingsrook.qbits.objectstorage.config.StorageBucketConfig; import com.kingsrook.qbits.objectstorage.internal.CapabilityStore; +import com.kingsrook.qbits.objectstorage.internal.ConfigSync; import com.kingsrook.qbits.objectstorage.internal.ConnectionRuntime; import com.kingsrook.qbits.objectstorage.internal.ErrorMapper; import com.kingsrook.qbits.objectstorage.internal.IndexHook; @@ -1028,7 +1029,7 @@ public final class StorageBucket { checkPermission("object_storage.view"); String key = StorageKeys.normalize(request.getKey()); - Duration ttl = cap(request.getTtl(), Duration.ofMinutes(5), bucket.getMaxPresignGetSeconds()); + Duration ttl = cap(request.getTtl(), Duration.ofMinutes(5), ConfigSync.effectiveMaxPresignGetSeconds(bucket)); countPresign(); GetObjectRequest.Builder b = GetObjectRequest.builder().bucket(name()).key(key); if(StringUtils.hasContent(request.getDownloadAs())) @@ -1061,7 +1062,7 @@ public final class StorageBucket String key = StorageKeys.normalize(rawKey); try { - return (connection.presigner().presignGetObject(GetObjectPresignRequest.builder().signatureDuration(Duration.ofMinutes(5)) + return (connection.presigner().presignGetObject(GetObjectPresignRequest.builder().signatureDuration(cap(null, Duration.ofMinutes(5), ConfigSync.effectiveMaxPresignGetSeconds(bucket))) .getObjectRequest(GetObjectRequest.builder().bucket(name()).key(key).build()).build()).url()); } catch(RuntimeException e) @@ -1150,7 +1151,7 @@ public final class StorageBucket { checkPermission("object_storage.write"); String key = StorageKeys.normalize(request.getKey()); - Duration ttl = cap(request.getTtl(), Duration.ofMinutes(15), bucket.getMaxPresignPutSeconds()); + Duration ttl = cap(request.getTtl(), Duration.ofMinutes(15), ConfigSync.effectiveMaxPresignPutSeconds(bucket)); countPresign(); PutObjectRequest.Builder b = PutObjectRequest.builder().bucket(name()).key(key); if(StringUtils.hasContent(request.getContentType())) @@ -1192,7 +1193,7 @@ public final class StorageBucket throw (new StorageException(Reason.UNSUPPORTED, "Presigned POST is not supported on this provider; verify the size after upload")); } String key = StorageKeys.normalize(request.getKey()); - Duration ttl = cap(request.getTtl(), Duration.ofMinutes(15), bucket.getMaxPresignPutSeconds()); + Duration ttl = cap(request.getTtl(), Duration.ofMinutes(15), ConfigSync.effectiveMaxPresignPutSeconds(bucket)); countPresign(); String[] creds = connection.presignCredentials(); PresignedPost post = PostPolicySigner.sign(creds[0], creds[1], creds[2], creds[3], connection.getConfig().effectiveAddressingStyle(), name(), key, diff --git a/src/main/java/com/kingsrook/qbits/objectstorage/app/StorageSetupStatus.java b/src/main/java/com/kingsrook/qbits/objectstorage/app/StorageSetupStatus.java index f7e2907..5a25199 100644 --- a/src/main/java/com/kingsrook/qbits/objectstorage/app/StorageSetupStatus.java +++ b/src/main/java/com/kingsrook/qbits/objectstorage/app/StorageSetupStatus.java @@ -23,14 +23,11 @@ package com.kingsrook.qbits.objectstorage.app; import java.util.ArrayList; import java.util.List; +import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig; +import com.kingsrook.qbits.objectstorage.internal.Db; import com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime; import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables; -import com.kingsrook.qqq.backend.core.actions.tables.CountAction; import com.kingsrook.qqq.backend.core.exceptions.QException; -import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput; -import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; -import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria; -import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; /******************************************************************************* @@ -68,14 +65,17 @@ public final class StorageSetupStatus /******************************************************************************* - ** Read the current state, as the viewer. + ** Read the current state. Connections and buckets come from the runtime + ** config (what the runtime uses, whoever is looking), and the rule counts + ** are read as the system, so a bucket-scoped user is never shown a false + ** "needs setup". *******************************************************************************/ public static List load() throws QException { - return (steps(count(ObjectStorageTables.CONNECTION, new QQueryFilter()), count(ObjectStorageTables.BUCKET, new QQueryFilter()), - count(ObjectStorageTables.HEADER_RULE, new QQueryFilter(new QFilterCriteria("enabled", QCriteriaOperator.EQUALS, true))), - ObjectStorageRuntime.getConfig().getHeaderPolicy() != null, - count(ObjectStorageTables.PROTECTION_RULE, new QQueryFilter(new QFilterCriteria("enabled", QCriteriaOperator.EQUALS, true))))); + ObjectStorageQBitConfig config = ObjectStorageRuntime.getConfig(); + int headerRules = Db.asSystem(() -> Db.query(ObjectStorageTables.HEADER_RULE, "enabled", true).size()); + int protection = Db.asSystem(() -> Db.query(ObjectStorageTables.PROTECTION_RULE, "enabled", true).size()); + return (steps(config.getConnections().size(), config.getBuckets().size(), headerRules, config.getHeaderPolicy() != null, protection)); } @@ -110,15 +110,4 @@ public final class StorageSetupStatus return (steps); } - - - /******************************************************************************* - ** Count as the viewer. - *******************************************************************************/ - private static int count(String table, QQueryFilter filter) throws QException - { - Integer count = new CountAction().execute(new CountInput(table).withFilter(filter)).getCount(); - return (count == null ? 0 : count); - } - } diff --git a/src/main/java/com/kingsrook/qbits/objectstorage/internal/ConfigSync.java b/src/main/java/com/kingsrook/qbits/objectstorage/internal/ConfigSync.java index 49091a2..425c83c 100644 --- a/src/main/java/com/kingsrook/qbits/objectstorage/internal/ConfigSync.java +++ b/src/main/java/com/kingsrook/qbits/objectstorage/internal/ConfigSync.java @@ -51,6 +51,9 @@ public final class ConfigSync private static final Map BUCKET_IDS = new ConcurrentHashMap<>(); private static final Map CONNECTION_IDS = new ConcurrentHashMap<>(); + private static final Map ROW_LIMITS = new ConcurrentHashMap<>(); + private static final long ROW_LIMIT_CACHE_MILLIS = 15_000L; + private static volatile boolean synced = false; @@ -72,6 +75,7 @@ public final class ConfigSync synced = false; BUCKET_IDS.clear(); CONNECTION_IDS.clear(); + ROW_LIMITS.clear(); } @@ -275,42 +279,88 @@ public final class ConfigSync } + /******************************************************************************* - ** The bucket's presign limits edited in the UI (Settings on the bucket row), - ** applied to the code config so every presign cap reads the row. Only the - ** presign TTLs are row-owned: the public base URL, trash bucket and trash - ** prefix are identity, rewritten from code on every sync and read-only in - ** the UI, so they are never read back from the row. A missing row, an unset - ** value or a value below one second keeps the config value. + ** What a bucket row says about presign limits (row values, cached briefly). *******************************************************************************/ - public static StorageBucketConfig withBucketOverrides(StorageBucketConfig base) + private record RowLimits(long readAt, Integer get, Integer put) { - if(base == null || !Db.available(ObjectStorageTables.BUCKET)) + } + + + + /******************************************************************************* + ** The longest a presigned download link may last, in seconds: the smaller of + ** the code config and the bucket row (Settings on object_storage_bucket). The + ** row can only lower the code limit; a missing row, null or a value below one + ** second means "no row limit". The config object is never modified. + *******************************************************************************/ + public static int effectiveMaxPresignGetSeconds(StorageBucketConfig bucket) + { + return (lowest(bucket.getMaxPresignGetSeconds(), rowLimits(bucket.getAlias()).get())); + } + + + + /******************************************************************************* + ** The longest a presigned upload link may last, in seconds (see the GET twin). + *******************************************************************************/ + public static int effectiveMaxPresignPutSeconds(StorageBucketConfig bucket) + { + return (lowest(bucket.getMaxPresignPutSeconds(), rowLimits(bucket.getAlias()).put())); + } + + + + /******************************************************************************* + ** Forget the cached row limits (after a row edit in a test; they also expire). + *******************************************************************************/ + public static void clearRowLimits() + { + ROW_LIMITS.clear(); + } + + + + /******************************************************************************* + ** The smaller of the code value (3600 when unset) and a usable row value. + *******************************************************************************/ + private static int lowest(Integer code, Integer row) + { + int base = code == null ? 3600 : code; + return (row != null && row > 0 && row < base ? row : base); + } + + + + /******************************************************************************* + ** The row's presign limits, read at most every 15 seconds per alias (the way + ** the header rules are cached); never throws. + *******************************************************************************/ + private static RowLimits rowLimits(String alias) + { + RowLimits cached = ROW_LIMITS.get(alias); + if(cached != null && System.currentTimeMillis() - cached.readAt() < ROW_LIMIT_CACHE_MILLIS) { - return (base); + return (cached); } - try + RowLimits limits = new RowLimits(System.currentTimeMillis(), null, null); + if(Db.available(ObjectStorageTables.BUCKET)) { - QRecord row = Db.asSystem(() -> Db.first(ObjectStorageTables.BUCKET, "alias", base.getAlias())); - if(row == null) + try { - return (base); + QRecord row = Db.asSystem(() -> Db.first(ObjectStorageTables.BUCKET, "alias", alias)); + if(row != null) + { + limits = new RowLimits(System.currentTimeMillis(), row.getValueInteger("maxPresignGetSeconds"), row.getValueInteger("maxPresignPutSeconds")); + } } - Integer get = row.getValueInteger("maxPresignGetSeconds"); - Integer put = row.getValueInteger("maxPresignPutSeconds"); - if(get != null && get > 0) + catch(Exception e) { - base.setMaxPresignGetSeconds(get); - } - if(put != null && put > 0) - { - base.setMaxPresignPutSeconds(put); + LOG.debug("Could not read bucket presign limits", e); } } - catch(Exception e) - { - LOG.debug("Could not read bucket overrides", e); - } - return (base); + ROW_LIMITS.put(alias, limits); + return (limits); } } diff --git a/src/main/java/com/kingsrook/qbits/objectstorage/internal/ObjectStorageRuntime.java b/src/main/java/com/kingsrook/qbits/objectstorage/internal/ObjectStorageRuntime.java index d6afd51..56ec9bd 100644 --- a/src/main/java/com/kingsrook/qbits/objectstorage/internal/ObjectStorageRuntime.java +++ b/src/main/java/com/kingsrook/qbits/objectstorage/internal/ObjectStorageRuntime.java @@ -163,7 +163,7 @@ public final class ObjectStorageRuntime { throw (new IllegalArgumentException("Unknown storage bucket alias: " + alias)); } - return (ConfigSync.withBucketOverrides(bucket)); + return (bucket); } diff --git a/src/main/java/com/kingsrook/qbits/objectstorage/processes/ActionSupport.java b/src/main/java/com/kingsrook/qbits/objectstorage/processes/ActionSupport.java index 2e74c82..cf41e54 100644 --- a/src/main/java/com/kingsrook/qbits/objectstorage/processes/ActionSupport.java +++ b/src/main/java/com/kingsrook/qbits/objectstorage/processes/ActionSupport.java @@ -126,7 +126,7 @@ public final class ActionSupport *******************************************************************************/ public static StorageBucketConfig config(String alias) { - return (alias == null ? null : ConfigSync.withBucketOverrides(ObjectStorageRuntime.getConfig().findBucket(alias))); + return (alias == null ? null : ObjectStorageRuntime.getConfig().findBucket(alias)); } diff --git a/src/main/java/com/kingsrook/qbits/objectstorage/processes/BucketProcesses.java b/src/main/java/com/kingsrook/qbits/objectstorage/processes/BucketProcesses.java index 5d4c71e..1632f6f 100644 --- a/src/main/java/com/kingsrook/qbits/objectstorage/processes/BucketProcesses.java +++ b/src/main/java/com/kingsrook/qbits/objectstorage/processes/BucketProcesses.java @@ -37,6 +37,7 @@ import com.kingsrook.qbits.objectstorage.config.StorageBucketConfig; import com.kingsrook.qbits.objectstorage.index.ObjectIndex; import com.kingsrook.qbits.objectstorage.index.ReferenceScan; import com.kingsrook.qbits.objectstorage.internal.CapabilityStore; +import com.kingsrook.qbits.objectstorage.internal.ConfigSync; import com.kingsrook.qbits.objectstorage.internal.ConnectionRuntime; import com.kingsrook.qbits.objectstorage.internal.Db; import com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime; @@ -689,7 +690,7 @@ public final class BucketProcesses request.setContentLength(Long.valueOf(String.valueOf(input.getValue(FIELD_LINK_LENGTH)))); } URL url = ActionSupport.storage(alias).presignPut(request); - int minutes = (int) Math.min(ActionSupport.config(alias).getMaxPresignPutSeconds() / 60, 15); + int minutes = (int) Math.min(ConfigSync.effectiveMaxPresignPutSeconds(ActionSupport.config(alias)) / 60, 15); ProcessSummaryLine line = EtlSupport.text(Status.OK, "Upload link for " + alias + "/" + request.getKey() + " (expires in about " + minutes + " minutes; shown once)"); line.setBulletsOfText(new ArrayList<>(List.of(url.toString()))); results.add(line); diff --git a/src/main/java/com/kingsrook/qbits/objectstorage/processes/LinkProcesses.java b/src/main/java/com/kingsrook/qbits/objectstorage/processes/LinkProcesses.java index 5dd85ce..d8bf533 100644 --- a/src/main/java/com/kingsrook/qbits/objectstorage/processes/LinkProcesses.java +++ b/src/main/java/com/kingsrook/qbits/objectstorage/processes/LinkProcesses.java @@ -27,6 +27,7 @@ import java.util.ArrayList; import java.util.List; import com.kingsrook.qbits.objectstorage.api.PresignGet; import com.kingsrook.qbits.objectstorage.config.StorageBucketConfig; +import com.kingsrook.qbits.objectstorage.internal.ConfigSync; import com.kingsrook.qbits.objectstorage.model.ObjectStorageTables; import com.kingsrook.qbits.objectstorage.permissions.ObjectStoragePermissions; import com.kingsrook.qqq.backend.core.exceptions.QException; @@ -150,7 +151,7 @@ public final class LinkProcesses { String alias = ActionSupport.aliasOf(record); StorageBucketConfig config = ActionSupport.config(alias); - Duration ttl = Duration.ofSeconds(Math.min(300, config.getMaxPresignGetSeconds())); + Duration ttl = Duration.ofSeconds(Math.min(300, ConfigSync.effectiveMaxPresignGetSeconds(config))); URL url = ActionSupport.storage(alias).presignGet(PresignGet.of(record.getValueString("objectKey")).ttl(ttl).downloadAs(record.getValueString("name"))); ProcessSummaryLine line = EtlSupport.text(Status.OK, "Download link for " + alias + "/" + record.getValueString("objectKey") + " (expires in " + ttl.toMinutes() + " minutes; shown once)"); line.setBulletsOfText(new ArrayList<>(List.of(url.toString()))); diff --git a/src/main/resources/db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql b/src/main/resources/db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql index 5649c5c..a13d807 100644 --- a/src/main/resources/db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql +++ b/src/main/resources/db/liquibase/qbit-object-storage/object-storage-defaults.changelog.sql @@ -12,6 +12,8 @@ -- * a seed never changes what an existing host does on deploy: the rows below -- are inserted DISABLED, so they are visible examples until someone turns -- them on; +-- * rollback is empty: a seed row may have been edited or adopted by the host, so +-- rolling back never deletes data; -- * no secrets, no host-specific values; -- * portable SQL (PostgreSQL and H2): INSERT ... SELECT ... WHERE NOT EXISTS; -- * every changeset has a precondition, so a host that has not created the @@ -29,4 +31,4 @@ INSERT INTO object_storage_header_rule (name, key_glob, cache_control, priority, enabled, create_date, modify_date) SELECT 'default-mutable', '**', 'public, max-age=3600', 1000, FALSE, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP WHERE NOT EXISTS (SELECT 1 FROM object_storage_header_rule WHERE name = 'default-mutable'); --- rollback DELETE FROM object_storage_header_rule WHERE name = 'default-mutable' AND enabled = FALSE; +-- rollback empty diff --git a/src/main/resources/help/qbit-object-storage.yaml b/src/main/resources/help/qbit-object-storage.yaml index 6e59a17..1e9b0cb 100644 --- a/src/main/resources/help/qbit-object-storage.yaml +++ b/src/main/resources/help/qbit-object-storage.yaml @@ -4,7 +4,7 @@ layer: QBIT # Help for the Storage app and every object_storage_* table, in the QQQ help-bundle # format (qqq#1198). Each table entry says what the table is for and how its rows # get created, so an empty table explains itself. ObjectStorageHelp applies it until -# the core HelpBundles registry ships in a QQQ release (STAND-IN(core=#1198)). +# the core HelpBundles registry ships in a QQQ release (a stand-in until that ships). # Summaries stay under 120 characters (ObjectStorageTablesGuardrailTest). entries: - key: app:objectStorageAdmin;slot:help @@ -25,7 +25,7 @@ entries: - key: table:object_storage_bucket;slot:help summary: "A bucket the application uses by alias: role, indexing, trash, public URL and presign limits." content: | - **How rows get created:** from code. Declare each bucket in the qbit config (`withBucket`: alias, connection, bucket name, role) and restart; the row is created on first start and its identity (role, public base URL, trash location) is rewritten from code each time. The presign limits (maximum seconds for download and upload links) are yours to edit on the row and take effect immediately: every link is capped by them. Rescan bucket, Check configuration and Apply header policy are record actions. + **How rows get created:** from code. Declare each bucket in the qbit config (`withBucket`: alias, connection, bucket name, role) and restart; the row is created on first start and its identity (role, public base URL, trash location) is rewritten from code each time. The presign limits (maximum seconds for download and upload links) can be lowered on the row: the effective limit is the smaller of the code config and the row, and a change applies within about 15 seconds. The row cannot raise the code limit. Rescan bucket, Check configuration and Apply header policy are record actions. - key: table:object_storage_bucket_capability;slot:help summary: "What a bucket supports (versioning, object lock, lifecycle, CORS and more), probed from the provider." diff --git a/src/test/java/com/kingsrook/qbits/objectstorage/ObjectStorageTablesGuardrailTest.java b/src/test/java/com/kingsrook/qbits/objectstorage/ObjectStorageTablesGuardrailTest.java index 9bdfb3c..ac0872e 100644 --- a/src/test/java/com/kingsrook/qbits/objectstorage/ObjectStorageTablesGuardrailTest.java +++ b/src/test/java/com/kingsrook/qbits/objectstorage/ObjectStorageTablesGuardrailTest.java @@ -198,4 +198,21 @@ class ObjectStorageTablesGuardrailTest } } + + + /******************************************************************************* + ** A host's own help in the slot is kept; the bundle only fills empty slots. + *******************************************************************************/ + @Test + void hostHelpIsKept() throws Exception + { + ObjectStorageQBitConfig config = new ObjectStorageQBitConfig().withBackendName(StorageTestSupport.MEMORY).withBucketScoping(false).withOpenWhenAnonymous(true).withAdminApp(true); + QInstance instance = StorageTestSupport.setUp(config); + instance.getTable(ObjectStorageTables.BUCKET).withHelpContent(ObjectStorageHelp.SLOT, new QHelpContent().withContentAsMarkdown("host text")); + instance.getApp("objectStorageAdmin").withHelpContent(ObjectStorageHelp.SLOT, new QHelpContent().withContentAsMarkdown("host app text")); + ObjectStorageHelp.apply(instance, config); + assertThat(instance.getTable(ObjectStorageTables.BUCKET).getHelpContent().get(ObjectStorageHelp.SLOT)).singleElement().extracting(QHelpContent::getContent).isEqualTo("host text"); + assertThat(instance.getApp("objectStorageAdmin").getHelpContent().get(ObjectStorageHelp.SLOT)).singleElement().extracting(QHelpContent::getContent).isEqualTo("host app text"); + } + } diff --git a/src/test/java/com/kingsrook/qbits/objectstorage/app/StorageSetupStatusTest.java b/src/test/java/com/kingsrook/qbits/objectstorage/app/StorageSetupStatusTest.java index a10ea9c..ee7ff55 100644 --- a/src/test/java/com/kingsrook/qbits/objectstorage/app/StorageSetupStatusTest.java +++ b/src/test/java/com/kingsrook/qbits/objectstorage/app/StorageSetupStatusTest.java @@ -105,4 +105,22 @@ class StorageSetupStatusTest config.withHeaderPolicy(hostPolicy); assertThat(StorageSetupStatus.load().get(2).next()).contains("host's own"); } + + + + /******************************************************************************* + ** A bucket-scoped user with no bucket keys sees no rows, but the card reads + ** the runtime config and counts rules as the system, so it never shows a + ** false "needs setup". + *******************************************************************************/ + @Test + void scopedUserSeesTrueStatus() throws Exception + { + ObjectStorageQBitConfig config = new ObjectStorageQBitConfig().withBackendName(StorageTestSupport.MEMORY).withBucketScoping(true).withOpenWhenAnonymous(true).withAdminApp(true) + .withConnection(new StorageConnectionConfig("main").withProviderType(ProviderType.MINIO).withEndpoint("http://localhost:1").withRegion("us-east-1")) + .withBucket(new StorageBucketConfig("media").withConnection("main").withBucketName("media-bucket").withRole(BucketRole.PRIVATE).withIndexed(false)); + StorageTestSupport.setUp(config); + StorageTestSupport.insert(ObjectStorageTables.HEADER_RULE, new QRecord().withValue("name", "r").withValue("keyGlob", "*").withValue("cacheControl", "no-cache").withValue("priority", 1).withValue("enabled", true)); + assertThat(StorageSetupStatus.load()).extracting(Step::state).containsExactly("Set up", "Set up", "Set up", "Optional"); + } } diff --git a/src/test/java/com/kingsrook/qbits/objectstorage/internal/BucketRowOverridesTest.java b/src/test/java/com/kingsrook/qbits/objectstorage/internal/BucketRowOverridesTest.java index f7ba5d3..f00a9ec 100644 --- a/src/test/java/com/kingsrook/qbits/objectstorage/internal/BucketRowOverridesTest.java +++ b/src/test/java/com/kingsrook/qbits/objectstorage/internal/BucketRowOverridesTest.java @@ -88,24 +88,43 @@ class BucketRowOverridesTest /******************************************************************************* - ** Editing the row's presign limits caps new links; the config value applies - ** until then; a nonsense value (zero) is ignored. + ** The row can only LOWER the presign limits: the effective limit is the + ** smaller of the code config and the row. A larger, zero or null row value + ** leaves the code limit, the shared config object is never modified, and the + ** download-link action and the preview URL read it too. *******************************************************************************/ @Test - void rowPresignLimitsCapNewLinks() throws Exception + void rowCanOnlyLowerPresignLimits() throws Exception { StorageTestSupport.setUp(config()); - Integer bucketId = ConfigSync.bucketId("media"); + Integer bucketId = ConfigSync.bucketId("media"); + Duration half = Duration.ofMinutes(30); + assertThat(expires(ObjectStorage.bucket("media").presignGet(PresignGet.of("a.txt").ttl(half)))).as("code limit is an hour").isEqualTo(1800); - assertThat(expires(ObjectStorage.bucket("media").presignGet(PresignGet.of("a.txt").ttl(Duration.ofMinutes(30))))).as("the config limit (an hour)").isEqualTo(1800); - - Db.update(ObjectStorageTables.BUCKET, List.of(new QRecord().withValue("id", bucketId).withValue("maxPresignGetSeconds", 60).withValue("maxPresignPutSeconds", 120))); - assertThat(expires(ObjectStorage.bucket("media").presignGet(PresignGet.of("a.txt").ttl(Duration.ofMinutes(30))))).as("the row's limit caps the link").isEqualTo(60); + setRow(bucketId, 60, 120); + assertThat(expires(ObjectStorage.bucket("media").presignGet(PresignGet.of("a.txt").ttl(half)))).as("the row lowers the cap").isEqualTo(60); assertThat(expires(ObjectStorage.bucket("media").presignPut(PresignPut.of("a.txt").ttl(Duration.ofMinutes(10))))).isEqualTo(120); - assertThat(ActionSupport.config("media").getMaxPresignGetSeconds()).as("the Create download link action reads it too").isEqualTo(60); + assertThat(ConfigSync.effectiveMaxPresignGetSeconds(ActionSupport.config("media"))).as("the download-link action reads it too").isEqualTo(60); + assertThat(expires(ObjectStorage.bucket("media").previewUrl("a.txt"))).as("previews are capped as well").isEqualTo(60); + assertThat(ObjectStorageRuntime.bucketConfig("media").getMaxPresignGetSeconds()).as("the shared config is not mutated").isEqualTo(3600); - Db.update(ObjectStorageTables.BUCKET, List.of(new QRecord().withValue("id", bucketId).withValue("maxPresignGetSeconds", 0))); - assertThat(ObjectStorageRuntime.bucketConfig("media").getMaxPresignGetSeconds()).as("zero is ignored").isEqualTo(60); + setRow(bucketId, 999_999, 999_999); + assertThat(expires(ObjectStorage.bucket("media").presignGet(PresignGet.of("a.txt").ttl(Duration.ofHours(2))))).as("the row cannot raise the code limit").isEqualTo(3600); + + setRow(bucketId, 0, null); + assertThat(expires(ObjectStorage.bucket("media").presignGet(PresignGet.of("a.txt").ttl(half)))).as("zero: the code value applies again").isEqualTo(1800); + assertThat(ConfigSync.effectiveMaxPresignPutSeconds(ActionSupport.config("media"))).as("null: the code value applies again").isEqualTo(900); + } + + + + /******************************************************************************* + ** Set the row limits and drop the short-lived cache. + *******************************************************************************/ + private static void setRow(Integer bucketId, Integer get, Integer put) throws Exception + { + Db.update(ObjectStorageTables.BUCKET, List.of(new QRecord().withValue("id", bucketId).withValue("maxPresignGetSeconds", get).withValue("maxPresignPutSeconds", put))); + ConfigSync.clearRowLimits(); } diff --git a/src/test/java/com/kingsrook/qbits/objectstorage/internal/DefaultsChangelogTest.java b/src/test/java/com/kingsrook/qbits/objectstorage/internal/DefaultsChangelogTest.java index 82d5b02..1e76247 100644 --- a/src/test/java/com/kingsrook/qbits/objectstorage/internal/DefaultsChangelogTest.java +++ b/src/test/java/com/kingsrook/qbits/objectstorage/internal/DefaultsChangelogTest.java @@ -104,6 +104,27 @@ public class DefaultsChangelogTest + /******************************************************************************* + ** Rolling the seed back never deletes data: a header rule the host owns + ** survives (the rollback is empty). + *******************************************************************************/ + @Test + void rollbackKeepsHostOwnedRules() throws Exception + { + try(Connection connection = freshSchema(); Statement statement = connection.createStatement()) + { + statement.execute("INSERT INTO object_storage_header_rule (name, key_glob, cache_control, priority, enabled, create_date, modify_date) " + + "VALUES ('host-rule', '*.css', 'immutable', 5, TRUE, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)"); + runChangelog(connection); + Database database = DatabaseFactory.getInstance().findCorrectDatabaseImplementation(new JdbcConnection(connection)); + new Liquibase(CHANGELOG.substring(1), new ClassLoaderResourceAccessor(), database).rollback(1, ""); + assertThat(scalar(connection, "SELECT COUNT(*) FROM object_storage_header_rule WHERE name = 'host-rule' AND enabled = TRUE")).isEqualTo("1"); + assertThat(changelogText()).contains("-- rollback empty").doesNotContain("DELETE FROM"); + } + } + + + /******************************************************************************* ** A host that already has a rule of that name keeps it untouched, and the ** changeset is marked ran. A host without the table is skipped, not failed. @@ -141,7 +162,7 @@ public class DefaultsChangelogTest String[] changesets = text.split("(?m)^-- changeset "); assertThat(changesets).hasSize(2); assertThat(changesets[1]).contains("-- preconditions onFail:MARK_RAN onError:MARK_RAN").contains("-- precondition-sql-check expectedResult:0 ") - .contains("-- rollback ").contains("NOT EXISTS").contains("FALSE, CURRENT_TIMESTAMP"); + .contains("-- rollback empty").contains("NOT EXISTS").contains("FALSE, CURRENT_TIMESTAMP"); assertThat(text).doesNotContain("precondition-table-exists"); for(String table : SEEDED_TABLES) { -- 2.54.0