feat: standalone mode, key-folder references, docs #6

Merged
KofTwentyTwo merged 1 commits from feature/storage-polish into develop 2026-10-12 22:43:55 +00:00
7 changed files with 237 additions and 12 deletions

No files matched your search

+13 -1
View File
@@ -5,4 +5,16 @@ All notable changes to this project are documented here. The format follows Keep
## [0.1.0] - Unreleased
### Added
- Repository scaffold: `ObjectStorageQBitConfig` and `ObjectStorageQBitProducer` (table prefix `object_storage_`, permissions `object_storage.*`), build, CI and repository standard files.
- **The service (P1).** `ObjectStorage.bucket(alias)` and `ObjectStorage.asSystem(caller, runId)`: put (multipart above the part size), get with ranges, head, copy (in parts for very large objects), header and metadata changes, delete (delete marker, private trash, or permanent), restore, list, presigned GET, PUT and POST, public URLs, tags. Key building and URL encoding, the header policy (`object_storage_header_rule`), `PublicUrlResolver`, `StorageChangeListener`, `StorageReferenceProvider`, `StorageAlertSink`.
- Configuration in code, upserted into visible `object_storage_*` tables; secret references with an allow-list; per-connection rate limits; per-bucket and per-connection record security locks.
- The capability model and the Test connection probe (read-only by default, write tests on request, clock skew, presign host), with seeded provider defaults.
- The `object_storage_operation` audit log (presigned URLs are logged without their signature) and protection rules (`READ_ONLY`, `NO_DELETE`, `WARN`, with allowed system callers).
- Backups follow-ups SF1 to SF5: `BucketRole.BACKUP`, object lock on put, system-caller permanent delete, probes that skip locked backup buckets, and the system-caller context.
- **The index (P2).** `object_storage_object` and `object_storage_folder` kept in step with each indexed bucket: resumable full scan with a mass-removal safety stop, HEAD backlog, write-through on every service write, soft-deleted rows, folder rollups. Orphaned and missing objects from `FieldReferenceProvider`, daily usage snapshots, quota and freshness alerts, the nightly job.
- **The Storage app (P3, P4).** Admin -> Storage with a responsive home; record screens with sections, child lists and a thumbnail preview; actions Rescan bucket, Check configuration, Refresh from bucket, Apply header policy (replaces app-specific backfills), Change headers and metadata, Edit tags, Copy, Move or rename, Delete, Restore, Create download link, Create upload link, folder rename and delete, Upload files, Test connection; every one a standard Review and Result with a bulk cap.
- `ObjectStorageSchema`: PostgreSQL DDL generated from the table metadata.
- Conformance suite against MinIO and Garage (Testcontainers), service, index and action tests, and the index and actions on PostgreSQL.
### Notes
- Until QQQ core's S3 module on AWS SDK v2 (qqq#1095, RC.5) is released, the qbit uses a small internal SDK v2 client factory.
- Deliberate differences from the design text: Test connection is two actions (read-only, and with write tests) so the write variant can require `object_storage.admin`; a folder record has no subfolder child list (core does not allow a self join as an exposed join); there is no `withTableNamePrefix` (the `object_storage_` prefix is fixed by the naming rule).
+75 -10
View File
@@ -5,38 +5,103 @@
> QQQ QBit: S3-compatible object storage service and Admin -> Storage app (browse, upload, presigned links, permission-checked and logged writes).
**Status:** design: [QRun-IO/qqq#1089](https://github.com/QRun-IO/qqq/pull/1089); implementation in progress. Nothing is published yet.
S3-compatible storage as a capability for any QQQ app, and an admin app to run it. Design: [QRun-IO/qqq#1089](https://github.com/QRun-IO/qqq/pull/1089).
1. **The service.** One Java API (`ObjectStorage.bucket("files")`) for put, get, head, copy, delete, restore, list, header and metadata changes, presigned upload and download URLs, public URLs and key building, over configured connections and buckets. Works against AWS S3, MinIO, Cloudflare R2, Backblaze B2, Wasabi, DigitalOcean Spaces, Ceph RGW and Garage. A capability model, filled in by a **Test connection** probe, says what each provider can do.
2. **The Storage app** (`Admin -> Storage`, optional). Buckets, objects, folders, connections, operations, header rules and protection rules as standard QQQ tables, an index of every indexed bucket, a home with real numbers, and standard Review/Result actions for every write.
**Status:** P1 (service), P2 (index, references, orphaned and missing objects, audit log) and P3/P4 (the app, read and write) are implemented. Not yet released: see [CHANGELOG.md](CHANGELOG.md).
## Requirements
- Java 21.
- QQQ 4.1 (built against `4.1.0-RC.4`).
- QQQ 4.1 (built against `4.1.0-RC.4`). Until QQQ core's S3 module on AWS SDK v2 (qqq#1095, RC.5) is released, the qbit carries a small internal SDK v2 client factory (`ConnectionRuntime`); it will be replaced by core's `S3ClientFactory` without changing the public API.
- A QQQ backend for the qbit's tables (PostgreSQL is the intended store). `ObjectStorageSchema.postgresScript(config)` generates the DDL from the same metadata the app runs on, for your Liquibase or Flyway migration.
## Install
Not yet published. Once released, add `com.kingsrook.qbits:qbit-object-storage` and produce the qbit into your `QInstance`:
```java
new ObjectStorageQBitProducer()
.withQBitConfig(new ObjectStorageQBitConfig().withBackendName("rdbms"))
.withQBitConfig(new ObjectStorageQBitConfig()
.withBackendName("rdbms")
.withConnection(new StorageConnectionConfig("primary")
.withProviderType(ProviderType.MINIO)
.withEndpoint("${env.S3_ENDPOINT}")
.withPresignEndpoint("${env.S3_PRESIGN_ENDPOINT}") // optional: the host clients reach the store by
.withRegion("${env.S3_REGION}")
.withAddressingStyle(AddressingStyle.PATH)
.withAccessKeyRef("${env.S3_ACCESS_KEY}")
.withSecretKeyRef("${env.S3_SECRET_KEY}")
.withPresignAccessKeyRef("${env.S3_PRESIGN_ACCESS_KEY}") // a least-privilege key for presigned URLs
.withPresignSecretKeyRef("${env.S3_PRESIGN_SECRET_KEY}"))
.withBucket(new StorageBucketConfig("files").withConnection("primary").withBucketName("${env.S3_BUCKET_NAME}")
.withRole(BucketRole.PUBLIC).withPublicBaseUrl("${env.PUBLIC_BASE_URL}").withIndexed(true).withReferenceTracked(true))
.withBucket(new StorageBucketConfig("incoming").withConnection("primary").withBucketName("${env.INCOMING_BUCKET}")
.withRole(BucketRole.INCOMING).withIndexed(false))
.withReferenceProvider(FieldReferenceProvider.keyPrefix("files", "media", "storageKey"))
.withAdminApp(true).withParentAppName("admin"))
.produce(qInstance);
```
Connections and buckets come from this code only. Secrets are **references** (`${env.X}`, `${prop.X}`, `${file.PATH}`) that must match `allowedSecretRefPatterns` (default `env.S3_*`, `env.STORAGE_*`, `file./var/run/secrets/storage/*`); no table stores a secret value. At startup the qbit upserts one `object_storage_connection` and one `object_storage_bucket` row per entry, so the tables always show the real configuration.
## Use
```java
StorageBucket files = ObjectStorage.bucket("files");
files.put(PutRequest.of("media/42/hero.jpg", bytes).withContentType("image/jpeg")); // headers come from the header policy
URL link = files.presignGet(PresignGet.of("media/42/hero.jpg").ttl(Duration.ofMinutes(5)));
String publicUrl = files.publicUrl("media/42/hero.jpg");
files.delete(DeleteRequest.of("media/42/hero.jpg")); // soft where the bucket allows it
// a calling qbit that checked its own permission (for example backups):
ObjectStorage.asSystem("backups", runId).bucket("backups").put(PutRequest.of(key, stream).withObjectLock(LockMode.COMPLIANCE, until));
```
Every call is synchronous, rate limited per connection, throws `StorageException` with a stable reason (`NOT_FOUND`, `ACCESS_DENIED`, `UNSUPPORTED`, `PRECONDITION_FAILED`, `THROTTLED`, `PROVIDER_ERROR`) and the provider's request id, and every write lands in `object_storage_operation`. Other qbits plug in with `ObjectHeaderPolicy`, `PublicUrlResolver`, `StorageChangeListener`, `StorageReferenceProvider` and `StorageAlertSink` (a no-op unless the Alerts qbit is installed).
## What is in the app
| Where | What |
|---|---|
| Home | Buckets, data health (orphaned and missing objects), recent changes, cache-header coverage, usage trend, connections. Each section says what is missing when it is empty. |
| Tables | `object_storage_bucket`, `_object`, `_folder`, `_connection`, `_operation`, `_header_rule`, `_protection_rule` in the nav; capabilities, versions, references and daily usage appear as child lists on their parents. |
| Bucket actions | Rescan bucket, Check configuration, Apply header policy, Create upload link |
| Object actions | Refresh from bucket, Apply header policy, Change headers and metadata, Edit tags, Copy objects, Move or rename objects, Delete objects, Restore objects, Create download link |
| Folder actions | Move or rename folder, Delete folder contents, Upload files to this folder |
| Table actions | Upload files |
| Connection actions | Test connection, Test connection with write tests |
Every write is a standard streamed ETL: **Review** (the dry run, with summary lines) then **Result**. Deletes are soft wherever the bucket has versioning or a private trash (Restore undoes them); referenced, protected and object-locked objects are skipped and the Review says why; bulk runs are capped at 10,000 objects unless the user has `object_storage.admin`.
Permissions: `object_storage.view`, `.write`, `.delete`, `.admin` (admin implies the rest). Per-bucket scoping uses record security locks on the `object_storage_bucket` and `object_storage_connection` security key types (switch off with `withBucketScoping(false)` for a single-administrator app).
## Deployment notes
- **Uploads** go through the app server (100 MB per file by default, per bucket `maxUploadBytes`), using QQQ's standard file-upload field, so the app needs `uploadedFileArchiveTableName` set in its Javalin metadata, as for any file upload.
- **Previews** are short-lived (5 minute) presigned GETs, so the Next dashboard's `img-src` must allow the storage host (it allows `https:` by default). Video, audio and PDF previews need same-origin media (the dashboard's `media-src` and `frame-src` are `'self'`); a same-origin download route is the planned follow-up. Until then those files show the file card with download and open links.
- **Scheduler.** Set `withSchedulerName(...)` to run the nightly scan, HEAD backlog, reference scan, usage snapshot and trash purge. Without it, run Rescan bucket by hand.
- **Alerts.** With the Alerts qbit installed, bind a `StorageAlertSink` adapter (`ObjectStorageRuntime.setAlertSink`); without it every condition is still visible on the home and in the tables.
## Conformance
`ObjectStorageConformanceSuite` runs the service against a store through Testcontainers on every PR: **MinIO** (the digest-pinned Chainguard build the QQQ tests use, since Docker Hub's `minio/minio` is gone) and **Garage** (digest-pinned). The index and the actions also run on **PostgreSQL**. AWS S3, R2 and B2 run from an owner-run workflow with scratch buckets; Wasabi, Spaces and Ceph results are community reported. See the design for the provider matrix.
## Demo server
`src/test/java/.../devserver/StorageDevServer` starts MinIO and an in-memory QQQ instance with generated images, header rules, references, an index and some history, and serves the Next dashboard, for looking at the app in a browser.
## Naming
Tables are `object_storage_<name>` (lowercase snake_case, backend name equal to the table name). Permissions are `object_storage.<action>`.
## Adopting in another app
Follow the QQQ [adoption checklist](https://github.com/QRun-IO/qqq/blob/develop/docs/migration/adoption-checklist.adoc) once the first release is published.
## Build
```bash
./mvnw verify
```
Always use the Maven wrapper (Maven 3.9.11); Maven 3.10 cannot read the parent POM.
Always use the Maven wrapper (Maven 3.9.11); Maven 3.10 cannot read the parent POM. The tests need Docker (Testcontainers); in CI a missing Docker fails the build instead of skipping.
## License
@@ -53,6 +53,7 @@ public class FieldReferenceProvider implements StorageReferenceProvider
{
KEY,
KEY_PREFIX,
KEY_FOLDER,
URL
}
@@ -97,6 +98,18 @@ public class FieldReferenceProvider implements StorageReferenceProvider
/*******************************************************************************
** The field holds the key of one object of a set that lives together in a
** folder (an original and its renditions): every object in that key's folder
** is referenced.
*******************************************************************************/
public static FieldReferenceProvider keyFolder(String bucketAlias, String tableName, String fieldName)
{
return (new FieldReferenceProvider(bucketAlias, tableName, fieldName, Holds.KEY_FOLDER));
}
/*******************************************************************************
** The field holds a public URL.
*******************************************************************************/
@@ -150,6 +163,15 @@ public class FieldReferenceProvider implements StorageReferenceProvider
.withRecordId(String.valueOf(row.getValue("id"))).withFieldName(fieldName).withLabel(label));
}
}
else if(holds == Holds.KEY_FOLDER)
{
int slash = value.lastIndexOf('/');
if(slash > 0)
{
references.add(new StorageReference().withBucketAlias(alias).withKeyOrPrefix(StorageKeys.normalizePrefix(value.substring(0, slash + 1))).withMatchMode(MatchMode.PREFIX)
.withTableName(tableName).withRecordId(String.valueOf(row.getValue("id"))).withFieldName(fieldName).withLabel(label));
}
}
else
{
references.add(new StorageReference().withBucketAlias(alias).withKeyOrPrefix(StorageKeys.normalizePrefix(value)).withMatchMode(holds == Holds.KEY_PREFIX ? MatchMode.PREFIX : MatchMode.KEY)
@@ -49,6 +49,27 @@ public final class ObjectStorage
/*******************************************************************************
** Install a configuration without a QQQ instance, for command-line tools and
** tests that use the service but not the tables or the app. Connections and
** buckets work as usual; nothing is written to the object_storage_* tables
** (there are none in this mode), so writes are logged to the application log
** only. Inside a QQQ application, produce the qbit instead.
*******************************************************************************/
public static void configureStandalone(com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig config)
{
java.util.List<String> errors = new java.util.ArrayList<>();
config.validate(null, errors);
if(!errors.isEmpty())
{
throw (new IllegalArgumentException("Object storage configuration is invalid: " + String.join("; ", errors)));
}
ObjectStorageRuntime.install(config);
ConfigSync.reset();
}
/*******************************************************************************
** The bucket behind an alias from config, called as the current session user.
*******************************************************************************/
@@ -206,6 +206,20 @@ public final class ObjectStorageTables
new String[] { OBJECT, OBJECT_VERSION, "objectId", "Versions" },
new String[] { OBJECT, OBJECT_REFERENCE, "objectId", "Referenced by" });
/** The columns a child list shows for each child table (the rest is one click away on the record). */
private static final Map<String, List<String>> CHILD_LIST_COLUMNS = Map.ofEntries(
Map.entry(OBJECT, List.of("objectKey", "size", "contentType", "lastModified", "referenceCount")),
Map.entry(FOLDER, List.of("path", "objectCount", "totalBytes")),
Map.entry(OBJECT_REFERENCE, List.of("label", "keyOrPrefix", "missing", "sourceTable")),
Map.entry(OPERATION, List.of("occurredAt", "action", "objectKey", "result", "actor")),
Map.entry(USAGE_DAILY, List.of("snapshotDate", "objectCount", "totalBytes")),
Map.entry(CONNECTION_CAPABILITY, List.of("capability", "status", "detail")),
Map.entry(BUCKET_CAPABILITY, List.of("capability", "status", "detail")),
Map.entry(BUCKET, List.of("alias", "role", "indexed", "objectCount")),
Map.entry(HEADER_RULE, List.of("name", "keyGlob", "cacheControl", "priority", "enabled")),
Map.entry(PROTECTION_RULE, List.of("prefixGlob", "mode", "enabled", "reason")),
Map.entry(OBJECT_VERSION, List.of("versionId", "isDeleteMarker", "isLatest", "lastModified")));
/** Tables that are nav entries (the rest appear as relations and child lists on their parents). */
public static final List<String> NAV_TABLES = List.of(BUCKET, OBJECT, FOLDER, CONNECTION, OPERATION, HEADER_RULE, PROTECTION_RULE);
@@ -367,7 +381,12 @@ public final class ObjectStorageTables
{
QJoinMetaData join = new QJoinMetaData().withName(joinName(relation)).withLeftTable(relation[0]).withRightTable(relation[1]).withType(JoinType.ONE_TO_MANY)
.withJoinOn(new JoinOn("id", relation[2]));
widgets.add(ChildRecordListRenderer.widgetMetaDataBuilder(join).withName(joinName(relation)).withLabel(relation[3]).withMaxRows(10).getWidgetMetaData());
ChildRecordListRenderer.Builder builder = ChildRecordListRenderer.widgetMetaDataBuilder(join).withName(joinName(relation)).withLabel(relation[3]).withMaxRows(10);
if(CHILD_LIST_COLUMNS.containsKey(relation[1]))
{
builder.withOnlyIncludeFieldNames(CHILD_LIST_COLUMNS.get(relation[1]));
}
widgets.add(builder.getWidgetMetaData());
}
return (widgets);
}
@@ -0,0 +1,64 @@
package com.kingsrook.qbits.objectstorage.conformance;
import java.nio.charset.StandardCharsets;
import com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig;
import com.kingsrook.qbits.objectstorage.api.DeleteRequest;
import com.kingsrook.qbits.objectstorage.api.ObjectStorage;
import com.kingsrook.qbits.objectstorage.api.PutRequest;
import com.kingsrook.qbits.objectstorage.api.Range;
import com.kingsrook.qbits.objectstorage.config.ChecksumMode;
import com.kingsrook.qbits.objectstorage.testsupport.StorageTestSupport;
import com.kingsrook.qqq.backend.core.context.QContext;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Assumptions;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** The service without a QQQ instance (command-line tools, tests): connections
** and buckets work, and nothing needs the object_storage_* tables.
*******************************************************************************/
class StandaloneTest
{
/*******************************************************************************
** Reset the runtime.
*******************************************************************************/
@AfterEach
void tearDown()
{
StorageTestSupport.tearDown();
}
@Test
void theServiceWorksWithoutAnInstanceOrTables() throws Exception
{
Assumptions.assumeTrue(Targets.dockerAvailable(), "Docker is not available");
ConformanceTarget target = Targets.minio();
ObjectStorageQBitConfig config = StorageTestSupport.config(target, ChecksumMode.WHEN_REQUIRED);
QContext.clear();
ObjectStorage.configureStandalone(config);
String key = "standalone/" + System.nanoTime() + ".txt";
ObjectStorage.bucket("files").put(PutRequest.of(key, "no tables needed".getBytes(StandardCharsets.UTF_8)).withContentType("text/plain"));
try(var in = ObjectStorage.bucket("files").get(key, Range.all()))
{
assertThat(new String(in.readAllBytes(), StandardCharsets.UTF_8)).isEqualTo("no tables needed");
}
ObjectStorage.bucket("files").delete(DeleteRequest.of(key));
assertThat(ObjectStorage.bucket("files").head(key)).isEmpty();
}
@Test
void aBadConfigIsRejectedWithEveryProblem()
{
ObjectStorageQBitConfig bad = new ObjectStorageQBitConfig();
assertThatThrownBy(() -> ObjectStorage.configureStandalone(bad)).isInstanceOf(IllegalArgumentException.class).hasMessageContaining("backendName");
}
}
@@ -473,6 +473,28 @@ class MinioIndexTest
@Test
void keyFolderReferencesCoverEveryObjectOfTheSet() throws Exception
{
rawPut(target.indexBucket, "set/one/original.jpg", 10);
rawPut(target.indexBucket, "set/one/thumbnail.jpg", 2);
rawPut(target.indexBucket, "set/two/original.jpg", 10);
ObjectIndex.scan("indexed");
StorageTestSupport.setUp(StorageTestSupport.config(target, com.kingsrook.qbits.objectstorage.config.ChecksumMode.WHEN_REQUIRED)
.withReferenceProvider(FieldReferenceProvider.keyFolder("indexed", "media", "storageKey")), instance ->
instance.addTable(new QTableMetaData().withName("media").withBackendName(StorageTestSupport.MEMORY).withPrimaryKeyField("id")
.withField(new QFieldMetaData("id", QFieldType.INTEGER)).withField(new QFieldMetaData("storageKey", QFieldType.STRING))));
ObjectIndex.scan("indexed");
StorageTestSupport.insert("media", new QRecord().withValue("storageKey", "set/one/original.jpg"), new QRecord().withValue("storageKey", "bare-name.jpg"));
ReferenceScan.Result result = ReferenceScan.run("indexed");
assertThat(result.references()).as("a key with no folder is not a reference").isEqualTo(1);
assertThat(rows("indexed").get("set/one/original.jpg").getValueInteger("referenceCount")).isEqualTo(1);
assertThat(rows("indexed").get("set/one/thumbnail.jpg").getValueInteger("referenceCount")).as("the rendition is covered").isEqualTo(1);
assertThat(rows("indexed").get("set/two/original.jpg").getValueInteger("referenceCount")).isZero();
}
@Test
void backupBucketsAreNeverReferenceScanned() throws Exception
{