Compare commits
16
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
73341e59c5
|
||
|
|
0d989683eb
|
||
|
|
6d733cde82 | ||
|
|
72e8ffb12d
|
||
|
|
b3d2f26548
|
||
|
|
0aa2f5180e | ||
|
|
cc08447020
|
||
|
|
6c16d73f28 | ||
|
|
266d101c91
|
||
|
|
cef1476d85
|
||
|
|
5493b598c6
|
||
|
|
07f846975b
|
||
|
|
a368187b44 | ||
|
|
4e6a2ae365
|
||
|
|
1cdd06098d
|
||
|
|
3f08c584d0
|
No files matched your search
@@ -3,5 +3,5 @@ on:
|
||||
push:
|
||||
jobs:
|
||||
maven:
|
||||
uses: self:KofTwentyTwo/ci-workflows/.gitea/workflows/maven.yaml@main
|
||||
uses: self:KofTwentyTwo/ci-workflows/.gitea/workflows/maven.yaml@v1
|
||||
secrets: inherit
|
||||
@@ -27,5 +27,3 @@ SOURCE-MANIFEST.tsv
|
||||
SOURCE-MANIFEST.sha256
|
||||
*.private.*
|
||||
*.log
|
||||
# Prepared caller retained as an inert documentation reference while CI/CD is on hold.
|
||||
.gitea/
|
||||
@@ -0,0 +1,29 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes are documented here, following [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and semantic versioning. Record Files is pre-1.0: a `0.1.0-RC.N` release candidate is a published, immutable artifact, not a production 1.0 claim (see [docs/RELEASE-STATUS.md](docs/RELEASE-STATUS.md)).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Changed
|
||||
|
||||
- PostgreSQL qualification (`RecordFilesPostgres`) accepts `full_page_writes = off` when, and only when, the server is Amazon Aurora PostgreSQL. Aurora reports `full_page_writes = off` (measured on engine 17.9) and the parameter is not in its cluster parameter group, so it cannot be changed; its storage layer does not write torn pages, so the protection that setting gives stock PostgreSQL is not needed there. Before this change Record Files refused every Aurora PostgreSQL database with its "exact writable PostgreSQL 17 database" message. Aurora is detected inside the existing settings query with `to_regprocedure('aurora_version()') IS NOT NULL`, which returns false and never raises on stock PostgreSQL. Nothing else is relaxed: stock PostgreSQL with `full_page_writes = off` is still refused, and on Aurora `fsync = on`, `synchronous_commit` of `on` or `remote_apply`, a writable primary, PostgreSQL 17, a UTC session, READ COMMITTED, a `lock_timeout` from 1 to 5000 ms and every owner-table check are required exactly as before. Covered by `PostgresDurabilityRuleTest` (every build) and `PostgresAuroraQualificationTest` (opt-in, real PostgreSQL 17, Aurora simulated by a test-only `aurora_version()` function).
|
||||
|
||||
## [0.1.0-RC.1] - 2026-11-06
|
||||
|
||||
Planned as **0.1.0-RC.1**, the first artifact published from this repository. It is published by pushing `develop` to `release/0.1`; the shared `KofTwentyTwo/ci-workflows` Maven workflow computes the `0.1.0-RC.N` version, deploys to the Gitea Maven registry (`https://git.kof22.com/api/packages/QRun-IO/maven`) and tags `v0.1.0-RC.N`. Nothing is published from `develop` except the `0.1.0-SNAPSHOT`.
|
||||
|
||||
### Added
|
||||
|
||||
- First published release candidate of the standalone QBit extracted from Website-Backend `b6fb2382`: retained files, ordered collections, review vocabulary, append-only audit, resumable upload sessions, tracked downloads, archive-and-delete, verified print observations, and the `tests` classifier with the public integration fixtures.
|
||||
- `CHANGELOG.md`, and a release-status document that records the publication path, the dependency cohort and the Forge development pin this release replaces.
|
||||
- Test-only PostgreSQL fixture diagnostics from the GG Forge qualification (`record-files-postgres-diagnostics`, `record-files-linux-fixture`): `POSTGRES_FIXTURE_HEALTH` container observation after each PostgreSQL intake case, and `FixtureEndpoint`, which resolves the disposable container's reachable host and published port (honouring `TESTCONTAINERS_HOST_OVERRIDE`, with the sibling bridge address for a JVM inside a build container) instead of hard-coding `127.0.0.1`. This is what lets the suite run on the shared `container-build` runners.
|
||||
|
||||
### Changed
|
||||
|
||||
- Archive capture (`ArchiveSecurity.digest`) now tolerates host table customizers that run only on insert, update or delete (`PRE_/POST_INSERT_RECORD`, `PRE_/POST_UPDATE_RECORD`, `PRE_/POST_DELETE_RECORD`), since they cannot change what a reader may see. Query, get and personalization customizers still refuse the capture with "Archive capture does not support this table's custom visibility policy." This is the source form of the change that GG Forge carried as the `-archive1` suffix of its development pin `0.1.0-forge.162af9a7b8cd-archive1`; it was recovered from that artifact's bytecode and is covered by `ArchiveTest.mutationCustomizersAreToleratedWhileVisibilityCustomizersRefuseCapture`.
|
||||
- `.gitea/` is no longer ignored, so the shared CI caller (`.gitea/workflows/build.yaml`, PR 2) is tracked normally; the retired platform-ci caller in `docs/release/qrun-release.reference.yaml` is marked superseded.
|
||||
|
||||
### Known limits carried into the release candidate
|
||||
|
||||
- The build needs a Collaboration artifact that carries the attachment lifecycle, upload journal and download SPI (Collaboration PR 59). Public Collaboration `0.1.0-RC.6` does not have it, and the `qbit-collaboration.version` property still names `0.1.0-SNAPSHOT`; pin the published Collaboration release candidate before pushing `release/0.1`.
|
||||
- Everything in [docs/KNOWN-ISSUES.md](docs/KNOWN-ISSUES.md) still applies: no malware scanning, no WORM proof, PostgreSQL script unqualified for production, no complete public-cohort consumer installation yet.
|
||||
@@ -1,6 +1,6 @@
|
||||
# QBit Record Files
|
||||
|
||||
> **Paused, unreleased work in progress — 2026-11-04.** Implementation has stopped. This is a source checkpoint, not a production-ready release or a complete installable package set.
|
||||
> **First release candidate in preparation — 2026-11-06.** `0.1.0-RC.1` will be the first artifact published from this repository (Gitea Maven registry, via the shared ci-workflows Maven workflow). It is a release candidate, not a production 1.0, and it is not published yet.
|
||||
|
||||
Start with [current status and build prerequisites](docs/RELEASE-STATUS.md) and [known issues](docs/KNOWN-ISSUES.md). These documents supersede older release-readiness statements below.
|
||||
|
||||
@@ -68,9 +68,33 @@ applies to all trusted generated outputs, not only previews. Hosts must trust ev
|
||||
in-process producer allowed to supply that origin; origin is provenance, not a
|
||||
cryptographic attestation of safe content.
|
||||
|
||||
`quarantineCapturedFiles` is a second optional host setting with the same shape
|
||||
(`getQuarantineCapturedFiles()` / `withQuarantineCapturedFiles(Boolean)`; null, the
|
||||
default, inherits `quarantineNewFiles`). It applies only to intake with origin
|
||||
`CAPTURE` **and** a nonblank device identity, and `GENERATED` keeps its own policy.
|
||||
In the upload path that identity is the server-persisted capture identity that the
|
||||
host's `CaptureIdentityProvider` resolved and revalidated; the manifest cannot carry
|
||||
a device, actor, policy or `CAPTURE` label (strict JSON rejects them; `CAMERA` and
|
||||
`GALLERY` items map to `CAPTURE`, so a device that passes the provider can label an
|
||||
item either way). A desktop request with no provider identity, or any `UPLOAD`,
|
||||
`SCANNER_IMPORT`, `IMPORT` or attachment-panel file, keeps `quarantineNewFiles`.
|
||||
Direct Java `RetainedFileService.intake` callers are trusted host code, like
|
||||
`GENERATED`: they could pass `CAPTURE` plus a device id, so hosts must not expose
|
||||
that call to untrusted input.
|
||||
|
||||
Host prerequisites before setting it to `false`: a `CaptureIdentityProvider` that
|
||||
actually approves each device and actor (a device id alone proves nothing, and Record
|
||||
Files does not decide approval); an `UploadConfiguration` that bounds media types,
|
||||
sizes and quotas to what the host accepts (for example JPEG/PNG only); and
|
||||
parent-table visibility and permissions as before. Nothing scans content, so this
|
||||
is a host trust decision, not a malware or human-review result. Existing quarantined
|
||||
files are not released by configuration; release them through the normal release
|
||||
process. Compatibility: the setting is additive and defaults to null, so existing
|
||||
hosts and persisted data behave exactly as before.
|
||||
|
||||
Upload manifests cannot select `GENERATED` or change host policy. Uploaded,
|
||||
captured, imported and standard attachment-panel originals retain
|
||||
`quarantineNewFiles`. Initial admission records `FILE_REGISTERED` with the chosen
|
||||
imported and standard attachment-panel originals, and captures without a device
|
||||
identity, retain `quarantineNewFiles`. Initial admission records `FILE_REGISTERED` with the chosen
|
||||
policy and lifecycle, without a release, scan or human-review claim. Hash failures
|
||||
still produce `FAILED`; permissions, current parent authorization and mandatory
|
||||
audit remain enforced. Changing configuration never releases existing quarantined
|
||||
@@ -93,3 +117,10 @@ Requires JDK 21 and a `qbit-collaboration` release that contains the attachment
|
||||
Published POMs are self-contained: the estate's `flatten-maven-plugin` (`ossrh` mode) resolves `${revision}` and the version
|
||||
properties at build time, so a consumer needs no `-Drevision` override. The version comes from `<revision>` in `pom.xml`.
|
||||
The inherited `qbit-build-parent` POM still warns until its owner publishes a literal version. Release candidate, not GA.
|
||||
|
||||
Artifacts are published to the Gitea Maven registry, `https://git.kof22.com/api/packages/QRun-IO/maven`, by the shared
|
||||
`KofTwentyTwo/ci-workflows` Maven workflow (`.gitea/workflows/build.yaml`): `develop` deploys `0.1.0-SNAPSHOT`, a push to
|
||||
`release/0.1` publishes the next `0.1.0-RC.N` and tags it, and a `v0.1.0` tag publishes the stable release. The registry
|
||||
requires sign-in for reads; configure a `<server>` for it in your private Maven settings. See
|
||||
[docs/RELEASE-STATUS.md](docs/RELEASE-STATUS.md) for the release steps and [CHANGELOG.md](CHANGELOG.md) for what each
|
||||
version contains.
|
||||
@@ -49,6 +49,8 @@ Final receipt: `{uploadId,collectionId,state,completedAt,items:[{itemId,fileId,v
|
||||
|
||||
Hard ceilings:20MiB per original,200MiB total,100items,64KiB–4MiB fixed chunk size,60–86400second staging lifetime,256KiB manifest JSON. Deployment limits can be lower and are frozen per upload. The original20MiB bound reflects the conditional single-PUT owner path, not a throughput benchmark. Accepted media types are declarations subject to quarantine/scanning, not verified file contents.
|
||||
|
||||
Initial lifecycle: finalized items are QUARANTINED unless the host sets the Java-only `quarantineCapturedFiles=false`, which makes only `CAMERA`/`GALLERY` items (origin `CAPTURE`) with the provider-resolved, revalidated device identity AVAILABLE. The manifest carries no policy, device or actor field and strict parsing rejects them. This is a host trust decision (approved devices, bounded media types), not a scan or review result; it emits no release event and leaves review status unchanged. See README "Not a 1.0".
|
||||
|
||||
Durable quota defaults:10 active uploads,512MiB reserved staging bytes and100 new uploads/hour per actor. Cancel/expiry does not immediately release capacity: bounded exact-key cleanup must finish and commit its cursor/outcome/quota release. `UploadMaintenance.cleanupOwn` permits currently authorized terminal cleanup. A trusted host scheduler can invoke bounded `cleanupBatch` under `record-files.maintain`; this does not grant content access. Expired finalization fences commit before irreversible staging deletion. Failed deletion/outcome retains cursor and quota for safe retry. Canonical originals are outside cleanup.
|
||||
|
||||
## Remaining integration and release evidence
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
# Known issues and unfinished work
|
||||
|
||||
Updated: 2026-11-04. These are unresolved items, not a plan currently being executed.
|
||||
Updated: 2026-11-06. These are unresolved items carried into the first release candidate, not a plan currently being executed.
|
||||
|
||||
- Public Collaboration 0.1.0-RC.6 does not provide the attachment lifecycle, upload journal, recovery and download APIs this module requires.
|
||||
- The corresponding canonical Collaboration integration is preserved on a separate work-in-progress branch. It requires confirmed rollback callbacks in QQQ Core/RDBMS; public QQQ 4.1.0-RC.21 is insufficient for that dependency chain.
|
||||
- Current snapshot dependency versions are placeholders for source integration, not published installation coordinates. No clean consumer installation of the complete public cohort has been demonstrated.
|
||||
- The corresponding canonical Collaboration integration is Collaboration PR 59 (`kof22/collab-v2`), not yet published. It requires confirmed rollback callbacks in QQQ Core/RDBMS; public QQQ 4.1.0-RC.21 is insufficient for that dependency chain.
|
||||
- `qbit-collaboration.version` is still `0.1.0-SNAPSHOT`, a placeholder for source integration. The release candidate must pin a published Collaboration RC before `release/0.1` is pushed (see RELEASE-STATUS.md). No clean consumer installation of the complete public cohort has been demonstrated.
|
||||
- Full Record Files verification, database integration, retained-file recovery and final standard-UI acceptance remain incomplete for this reconciled source.
|
||||
- Permanent access depends on the host's durable object storage, backups, retention and permission configuration. A source-level interface or unit test alone does not prove operational permanence.
|
||||
- The Amazon Aurora PostgreSQL rule (`full_page_writes = off` accepted on Aurora only) is tested on stock PostgreSQL 17 with Aurora simulated by a test-only `aurora_version()` function. No test in this repository runs against a real Aurora cluster, and the opt-in PostgreSQL suites do not run in the shared CI build. The detection resolves `aurora_version()` through the session `search_path`, so a role that can create functions in a schema on that path of a stock PostgreSQL database could make it report Aurora; the application role must not hold that privilege.
|
||||
+41
-8
@@ -1,6 +1,6 @@
|
||||
# Record Files — current status
|
||||
|
||||
Updated: 2026-11-04. **Work stopped at the user's request. Unreleased work in progress; not a production 1.0 release.**
|
||||
Updated: 2026-11-06. **Preparing the first published release candidate, `0.1.0-RC.1`.** Not a production 1.0; see [KNOWN-ISSUES.md](KNOWN-ISSUES.md) and [TODO.md](TODO.md) for the gates that remain.
|
||||
|
||||
## Purpose
|
||||
|
||||
@@ -10,18 +10,51 @@ Retained uploads, scanned files and photo collections associated with QQQ record
|
||||
|
||||
The root Maven module contains the QBit and its integration-test fixture classifier. Byte storage and attachment intake use the existing Collaboration attachment contracts. Generation and printing integrations live in their owning bridge modules, not here.
|
||||
|
||||
## Building and installation
|
||||
## Building
|
||||
|
||||
Use Java 21 and the included Maven wrapper. After compatible Core and Collaboration artifacts are available, run `./mvnw verify`. Integration tests require their Docker/Testcontainers services. Do not expect the currently published older dependencies to compile this branch.
|
||||
Use Java 21 and the included Maven wrapper: `./mvnw verify`. The PostgreSQL and MySQL integration suites are opt-in: they use Testcontainers (a Docker daemon is required) and run only with `-DpostgresRecordFilesTests=true` and `-DmysqlRecordFilesTests=true`; otherwise they are reported as skipped, so a plain `verify` is not provider evidence. The build needs a Collaboration artifact that carries the attachment lifecycle, upload journal and download SPI (`AttachmentLifecyclePolicy`, `AttachmentDownload`, `AttachmentContentIntegrityException`, `uploadjournal.UploadRecoveryNavigation`, `security.ParentRecordAccess`). Public Collaboration `0.1.0-RC.6` does not have them; that work is Collaboration [PR 59](https://git.kof22.com/QRun-IO/qbit-collaboration/pulls/59). Until it is published, build against a locally installed snapshot of that branch and override the property, for example `./mvnw -Dqbit-collaboration.version=<local snapshot> verify`.
|
||||
|
||||
## Version and publication path
|
||||
|
||||
Versioning follows the shared [KofTwentyTwo/ci-workflows](https://git.kof22.com/KofTwentyTwo/ci-workflows) Maven workflow, called from `.gitea/workflows/build.yaml` (PR 2). The version lives in the `<revision>` property of `pom.xml` and stays `0.1.0-SNAPSHOT` on `develop`; the workflow computes published versions and commits the bump back.
|
||||
|
||||
| Ref | Version | Action |
|
||||
| --- | --- | --- |
|
||||
| feature branches, `main` | unchanged | `mvn verify` only |
|
||||
| `develop` | `0.1.0-SNAPSHOT` | test, deploy snapshot |
|
||||
| `release/0.1` | `0.1.0-RC.N` | test, deploy, commit bump, tag `v0.1.0-RC.N` |
|
||||
| tag `v0.1.0` | `0.1.0` | test, deploy, commit to `main` |
|
||||
|
||||
Packages go to the Gitea Maven registry, `https://git.kof22.com/api/packages/QRun-IO/maven` (sign-in required for reads as well as writes). Nothing is published to Maven Central and nothing is GPG-signed. GitHub is a shadow/archive; never publish from it.
|
||||
|
||||
### First release candidate, step by step
|
||||
|
||||
1. Merge Collaboration PR 59 and publish its release candidate (`release/0.1` there), so the registry holds a Collaboration RC with the SPI above.
|
||||
2. Merge PR 2 (shared CI caller) and this release-preparation PR into `develop`; confirm the `develop` push run passes and deploys `0.1.0-SNAPSHOT`.
|
||||
3. Merge PR 1 (`quarantineCapturedFiles`) into `develop`; it has no source conflict with this branch and only adds to `RecordFilesQBitConfig` and `RetainedFileService`.
|
||||
4. On `develop`, set `qbit-collaboration.version` to the published Collaboration RC (a release candidate must not depend on a `-SNAPSHOT`), then `git push origin develop:release/0.1`. The push run publishes `0.1.0-RC.1` and tags `v0.1.0-RC.1`.
|
||||
5. Resolve `com.kingsrook.qbits:qbit-record-files:0.1.0-RC.1` (and its `tests` classifier if the fixture is wanted) from the registry in a fresh consumer cache and record the JAR and POM SHA-256 here. A passing build alone is not proof of publication.
|
||||
|
||||
The stable `0.1.0` comes later from a `v0.1.0` tag on `main`; it is not part of this candidate.
|
||||
|
||||
## What this candidate contains beyond the paused checkpoint (`162af9a`)
|
||||
|
||||
- The archive-capture customizer guard change that GG Forge ran as development pin `0.1.0-forge.162af9a7b8cd-archive1` (mutation-only host customizers no longer refuse a capture; see CHANGELOG).
|
||||
- The two test-only PostgreSQL fixture patches from the Forge qualification (`FixtureEndpoint`, fixture health reporting), unchanged from `scripts/local/patches/record-files-*.patch` in the Forge handoff.
|
||||
- Release documentation. No behaviour other than the archive guard changed.
|
||||
|
||||
### Relationship to the GG Forge development pin
|
||||
|
||||
Forge's pin `0.1.0-forge.162af9a7b8cd-archive1` was built from the same source as this candidate, but against development coordinates: `qqq-backend-core 4.1.0-forge.33464d37e815-echo1`, `qqq-backend-module-rdbms 4.1.0-forge.33464d37e815` and `qbit-collaboration 0.1.0-forge.8f348daf324d-files1`. `0.1.0-RC.1` is a drop-in replacement for the Record Files code itself (same classes, plus the `quarantineCapturedFiles` setting once PR 1 is merged), but its POM pins the public QQQ `4.1.0-RC.21` tuple and the published Collaboration RC, so the host must move the whole cohort together, not only this coordinate.
|
||||
|
||||
## Evidence retained
|
||||
|
||||
The extracted source passed Checkstyle and flattened-POM checks. Compilation against public Collaboration 0.1.0-RC.6 exposed missing attachment lifecycle APIs. End-to-end verification of this standalone package has not completed. The earlier scanner integration proof used a private dependency cohort and does not constitute a standalone public release.
|
||||
- Local `./mvnw -Dqbit-collaboration.version=0.1.0-rf53bd8f00-SNAPSHOT verify` on this branch (Collaboration commit `53bd8f00`, the commit PR 1 was verified against; Java 21): 284 tests, 0 failures, 0 errors, 58 skipped (the opt-in database suites), Checkstyle clean, BUILD SUCCESS.
|
||||
- Opt-in database suites with Docker 29.9.2 (`-DpostgresRecordFilesTests=true -DmysqlRecordFilesTests=true`): 64 tests, 1 failure, 1 error. MySQL 15/15, PostgresTrackedDownload 3/3, PostgresRetainedUploadRecovery 5/5 and FixtureEndpoint 5/5 pass; `POSTGRES_FIXTURE_HEALTH` reports `running=true oom=false restarting=false endpointMatches=true` after all 22 intake cases. The two failures pre-date this branch: `PostgresIntakeAndRetentionTest.unmanagedTablesKeepOrdinaryAttachmentBehavior` dies in Collaboration's `AttachmentsService.storeUpload` with `NoSuchMethodError QBackendTransaction.supportsRollbackCallbacks()` (the documented QQQ core after-rollback-callback gap: public 4.1.0-RC.21 lacks it, Forge builds against `4.1.0-forge.33464d37e815-echo1`), and `PostgresRecordFilesTest.refusesLabelsAndPhysicalDrift` fails its `isDurableBackend` re-check after `ALTER COLUMN at TYPE TIMESTAMPTZ` (line 258), a physical-drift probe untouched by this change.
|
||||
- The earlier scanner integration proof used a private dependency cohort and does not constitute a standalone public release.
|
||||
|
||||
## Source and release state
|
||||
|
||||
The starting source was extracted from Website-Backend commit b6fb23825a87412397dec703b9617148ae7eb601, then adapted for standalone packaging. This repository preserves the resulting source and existing licenses. It does not declare a published artifact cohort, completed deployment or complete acceptance.
|
||||
The starting source was extracted from Website-Backend commit b6fb23825a87412397dec703b9617148ae7eb601, then adapted for standalone packaging. This repository preserves the resulting source and existing licenses. The outstanding dependency order is Core, Collaboration, Scripting/Record Files, then the integrated Generation bridges and matching UI.
|
||||
|
||||
See [known issues](KNOWN-ISSUES.md) and the README for the existing API and integration documentation. No further implementation, tests, releases or deployment should be started without explicit user instruction. The outstanding dependency order is Core, Collaboration, Scripting/Record Files, then the integrated Generation bridges and matching UI.
|
||||
|
||||
The prepared release caller is saved as an inactive reference in docs/release/qrun-release.reference.yaml. No new workflow is being activated by this checkpoint; shared CI/CD design and enrollment remain unfinished.
|
||||
The old platform-ci caller is kept, inactive, as `docs/release/qrun-release.reference.yaml`; the shared ci-workflows caller supersedes it.
|
||||
@@ -1,3 +1,5 @@
|
||||
# SUPERSEDED, inactive reference. The repository now publishes through the shared KofTwentyTwo/ci-workflows Maven workflow
|
||||
# (.gitea/workflows/build.yaml, PR 2); see docs/RELEASE-STATUS.md. Kept only as a record of the retired platform-ci caller.
|
||||
# Standard QRun release caller (v2, thin contract): every push and manual dispatch delegates to the protected qrun-release-v1 channel. Do not edit per repository.
|
||||
name: qrun-release
|
||||
on:
|
||||
|
||||
@@ -18,22 +18,20 @@
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
|
||||
<properties>
|
||||
<revision>0.1.0-SNAPSHOT</revision>
|
||||
<revision>0.5.0-SNAPSHOT</revision>
|
||||
<maven.compiler.source>21</maven.compiler.source>
|
||||
<maven.compiler.target>21</maven.compiler.target>
|
||||
<jacoco.version>0.8.15</jacoco.version>
|
||||
<!-- Released qqq (Maven Central); one core for the whole module, imported through qqq-bom-pom (see the qqq-snapshot profile). -->
|
||||
<qqq.version>4.1.0-RC.21</qqq.version>
|
||||
<!-- NOTE: collaboration's lifecycle code also needs core after-rollback callbacks (CORE-DEPENDENCY.md); raise this to the core release that has them. -->
|
||||
<!-- Candidate collaboration carrying the attachment lifecycle SPI (feat/record-files-standalone, not yet released).
|
||||
Not in public 0.1.0-RC.6. Qualify with a locally installed 0.1.0-SNAPSHOT; see COLLABORATION-HANDOFF.md. -->
|
||||
<qbit-collaboration.version>0.1.0-SNAPSHOT</qbit-collaboration.version>
|
||||
<!-- QQQ Core 4.1.0-RC.22 from the Gitea registry, imported through qqq-bom-pom. It includes the after-rollback callbacks Collaboration needs. -->
|
||||
<qqq.version>4.1.0-RC.26</qqq.version>
|
||||
<!-- Released Collaboration carrying the attachment lifecycle SPI. -->
|
||||
<qbit-collaboration.version>1.2.0-RC.1</qbit-collaboration.version>
|
||||
</properties>
|
||||
|
||||
<parent>
|
||||
<groupId>com.kingsrook</groupId>
|
||||
<artifactId>qbit-build-parent</artifactId>
|
||||
<version>2.1.1</version>
|
||||
<version>2.2.1</version>
|
||||
</parent>
|
||||
|
||||
<groupId>com.kingsrook.qbits</groupId>
|
||||
@@ -238,7 +236,7 @@
|
||||
<activeByDefault>true</activeByDefault>
|
||||
</activation>
|
||||
<properties>
|
||||
<qqq.snapshot.version>${qqq.version}</qqq.snapshot.version>
|
||||
<qqq.snapshot.version>4.1.0-RC.26</qqq.snapshot.version>
|
||||
</properties>
|
||||
<dependencyManagement>
|
||||
<dependencies>
|
||||
|
||||
@@ -46,14 +46,25 @@ import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSTableBackendDe
|
||||
** Qualification of this QBit's actual PostgreSQL 17 target (the native QQQ
|
||||
** postgres module: vendor and backend type "postgres") and its owned tables.
|
||||
** Observes only; no migrations or session/global setting changes happen here.
|
||||
**
|
||||
** Amazon Aurora PostgreSQL is the one accepted variant of the durability rule:
|
||||
** it reports full_page_writes = off, and that alone is accepted there (see
|
||||
** durable). Every other check is identical on stock PostgreSQL and Aurora.
|
||||
*******************************************************************************/
|
||||
final class RecordFilesPostgres
|
||||
{
|
||||
static final String VENDOR = "postgres";
|
||||
|
||||
private static final String SETTINGS = "SELECT current_setting('server_version_num')::int, current_setting('transaction_read_only'), pg_is_in_recovery(),"
|
||||
/////////////////////////////////////////////////////////////////////////////////////////
|
||||
// Column 12 is true only on Amazon Aurora PostgreSQL, which defines aurora_version(). //
|
||||
// to_regprocedure returns NULL for a function that does not exist and never raises, //
|
||||
// so the probe is safe on stock PostgreSQL; calling aurora_version() directly there //
|
||||
// would abort the transaction, and pg_proc does not list the function on Aurora. //
|
||||
/////////////////////////////////////////////////////////////////////////////////////////
|
||||
static final String SETTINGS = "SELECT current_setting('server_version_num')::int, current_setting('transaction_read_only'), pg_is_in_recovery(),"
|
||||
+ " current_setting('synchronous_commit'), current_setting('fsync'), current_setting('full_page_writes'), current_setting('TimeZone'),"
|
||||
+ " (SELECT setting::bigint FROM pg_settings WHERE name='lock_timeout'), current_setting('transaction_isolation'), current_schema(), current_database()";
|
||||
+ " (SELECT setting::bigint FROM pg_settings WHERE name='lock_timeout'), current_setting('transaction_isolation'), current_schema(), current_database(),"
|
||||
+ " to_regprocedure('aurora_version()') IS NOT NULL";
|
||||
|
||||
private static final String TABLES = "SELECT c.relname, c.relkind, c.relpersistence FROM pg_class c JOIN pg_namespace n ON n.oid=c.relnamespace"
|
||||
+ " WHERE n.nspname=current_schema() AND c.relname = ANY(?)";
|
||||
@@ -131,7 +142,7 @@ final class RecordFilesPostgres
|
||||
try(ResultSet settings = statement.executeQuery(SETTINGS))
|
||||
{
|
||||
if(!settings.next() || settings.getInt(1) / 10000 != 17 || !"off".equals(settings.getString(2)) || settings.getBoolean(3)
|
||||
|| !Set.of("on", "remote_apply").contains(settings.getString(4)) || !"on".equals(settings.getString(5)) || !"on".equals(settings.getString(6))
|
||||
|| !durable(settings.getString(4), settings.getString(5), settings.getString(6), settings.getBoolean(12))
|
||||
|| !Set.of("UTC", "Etc/UTC").contains(settings.getString(7)) || settings.getLong(8) < 1 || settings.getLong(8) > 5000
|
||||
|| !"read committed".equals(settings.getString(9)) || settings.getString(10) == null || !catalog.equals(settings.getString(11)))
|
||||
{
|
||||
@@ -155,6 +166,30 @@ final class RecordFilesPostgres
|
||||
requireTables(connection, tables);
|
||||
}
|
||||
|
||||
/***************************************************************************
|
||||
** Committed durability from the observed server settings: synchronous_commit
|
||||
** on or remote_apply, fsync on, and torn-page protection.
|
||||
***************************************************************************/
|
||||
static boolean durable(String synchronousCommit, String fsync, String fullPageWrites, boolean aurora)
|
||||
{
|
||||
if(synchronousCommit == null || !Set.of("on", "remote_apply").contains(synchronousCommit) || !"on".equals(fsync))
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
|
||||
////////////////////////////////////////////////////////////////////////////////////////
|
||||
// Stock PostgreSQL needs full_page_writes = on: after a crash, a partly written page //
|
||||
// is repaired from the full page image in the write-ahead log. //
|
||||
// Amazon Aurora PostgreSQL reports full_page_writes = off (measured on engine 17.9) //
|
||||
// and the parameter cannot be changed there: it is not in the cluster parameter //
|
||||
// group. Its storage layer does not write torn pages, so the protection is not //
|
||||
// needed. Off is therefore accepted on Aurora only; on stock PostgreSQL it is still //
|
||||
// refused. fsync and synchronous_commit are required on Aurora exactly as on stock //
|
||||
// PostgreSQL. //
|
||||
////////////////////////////////////////////////////////////////////////////////////////
|
||||
return ("on".equals(fullPageWrites) || (aurora && "off".equals(fullPageWrites)));
|
||||
}
|
||||
|
||||
/***************************************************************************
|
||||
** Permanent tables, deterministic comparison, owner column types and complete unique keys.
|
||||
***************************************************************************/
|
||||
|
||||
@@ -81,6 +81,7 @@ public class RecordFilesQBitConfig implements QBitConfig
|
||||
private List<String> parentTables = List.of();
|
||||
private Boolean quarantineNewFiles = true;
|
||||
private Boolean quarantineGeneratedFiles;
|
||||
private Boolean quarantineCapturedFiles;
|
||||
private Integer maxItemsPerCollection = 500;
|
||||
private QCodeReference auditPort = new QCodeReference(TableAuditPort.class);
|
||||
|
||||
@@ -227,6 +228,32 @@ public class RecordFilesQBitConfig implements QBitConfig
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Optional trusted captured-intake policy. Null inherits quarantineNewFiles.
|
||||
** Applies only to CAPTURE intake that carries a nonblank server-resolved device
|
||||
** identity; never to GENERATED, UPLOAD, IMPORT or attachment-panel files.
|
||||
*******************************************************************************/
|
||||
public Boolean getQuarantineCapturedFiles()
|
||||
{
|
||||
return (quarantineCapturedFiles);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Host decision for device capture intake, not a scan or human review result.
|
||||
** False makes valid captured files AVAILABLE on registration. The host must
|
||||
** wire a CaptureIdentityProvider that approves each device and bound the
|
||||
** accepted media types; a device id alone proves no approval.
|
||||
*******************************************************************************/
|
||||
public RecordFilesQBitConfig withQuarantineCapturedFiles(Boolean quarantineCapturedFiles)
|
||||
{
|
||||
this.quarantineCapturedFiles = quarantineCapturedFiles;
|
||||
return (this);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Upper bound on the items of one collection.
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -150,6 +150,18 @@ public final class ArchiveSecurity
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Host customizers that only run on insert, update or delete cannot change what a
|
||||
** reader may see, so an archive capture tolerates them. Every other customizer role
|
||||
** (query, get, personalization) can shape visibility and is still refused.
|
||||
*******************************************************************************/
|
||||
private static final Set<String> MUTATION_ROLES = Set.of(
|
||||
TableCustomizers.PRE_INSERT_RECORD.getRole(), TableCustomizers.POST_INSERT_RECORD.getRole(),
|
||||
TableCustomizers.PRE_UPDATE_RECORD.getRole(), TableCustomizers.POST_UPDATE_RECORD.getRole(),
|
||||
TableCustomizers.PRE_DELETE_RECORD.getRole(), TableCustomizers.POST_DELETE_RECORD.getRole());
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fingerprint exact native lock topology/types/key definitions; permission
|
||||
** grants are intentionally not captured and are always evaluated currently.
|
||||
@@ -157,7 +169,7 @@ public final class ArchiveSecurity
|
||||
private static String digest(QTableMetaData table) throws QException
|
||||
{
|
||||
if(table == null || table.getPrimaryKeyField() == null || table.getCustomizers() != null
|
||||
&& table.getCustomizers().keySet().stream().anyMatch(role -> !TableCustomizers.PRE_DELETE_RECORD.getRole().equals(role))
|
||||
&& table.getCustomizers().keySet().stream().anyMatch(role -> !MUTATION_ROLES.contains(role))
|
||||
|| QContext.getQInstance().getSupplementalCustomizer(com.kingsrook.qqq.backend.core.actions.metadata.personalization.TableMetaDataPersonalizerInterface.CUSTOMIZER_TYPE) != null)
|
||||
{
|
||||
throw (new QException("Archive capture does not support this table's custom visibility policy."));
|
||||
|
||||
@@ -279,7 +279,15 @@ public class RetainedFileService
|
||||
|
||||
boolean generatedPolicy = pending != null && RecordFilesValues.ORIGIN_GENERATED.equals(pending.request().origin())
|
||||
&& config.getQuarantineGeneratedFiles() != null;
|
||||
boolean quarantine = Boolean.TRUE.equals(generatedPolicy ? config.getQuarantineGeneratedFiles() : config.getQuarantineNewFiles());
|
||||
// Captured policy needs the service-owned intake scope (never a plain standard upload), the CAPTURE origin
|
||||
// and a nonblank device identity. UploadFinalizer supplies that identity only from the server-persisted,
|
||||
// revalidated capture identity; a client cannot choose this policy through the upload JSON.
|
||||
boolean capturedPolicy = !generatedPolicy && pending != null && RecordFilesValues.ORIGIN_CAPTURE.equals(pending.request().origin())
|
||||
&& pending.request().deviceId() != null && !pending.request().deviceId().isBlank()
|
||||
&& config.getQuarantineCapturedFiles() != null;
|
||||
boolean quarantine = Boolean.TRUE.equals(generatedPolicy ? config.getQuarantineGeneratedFiles()
|
||||
: capturedPolicy ? config.getQuarantineCapturedFiles() : config.getQuarantineNewFiles());
|
||||
String admissionPolicy = generatedPolicy ? "quarantineGeneratedFiles" : capturedPolicy ? "quarantineCapturedFiles" : "quarantineNewFiles";
|
||||
String lifecycle = mismatch ? RecordFilesValues.LIFECYCLE_FAILED
|
||||
: quarantine ? RecordFilesValues.LIFECYCLE_QUARANTINED : RecordFilesValues.LIFECYCLE_AVAILABLE;
|
||||
QRecord row = new QRecord()
|
||||
@@ -313,7 +321,7 @@ public class RetainedFileService
|
||||
audit.withChange(new RecordFilesEvent(RecordFilesEventType.FILE_REGISTERED, actor, correlationId).withRequestId(requestId)
|
||||
.withFile(fileId, attachmentId).withSource(tableName, recordId)
|
||||
.withDetail("lifecycle=" + lifecycle + ", origin=" + row.getValueString("origin") + ", version=" + row.getValueInteger("versionNumber")
|
||||
+ ", admissionPolicy=" + (generatedPolicy ? "quarantineGeneratedFiles" : "quarantineNewFiles") + ", quarantine=" + quarantine
|
||||
+ ", admissionPolicy=" + admissionPolicy + ", quarantine=" + quarantine
|
||||
+ (mismatch ? ", declared SHA-256 did not match the stored bytes" : "")));
|
||||
if(pending != null)
|
||||
{
|
||||
|
||||
@@ -237,6 +237,46 @@ public class ArchiveTest extends RecordFilesFixture
|
||||
}
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Host insert/update/delete customizers cannot change reader visibility, so a capture
|
||||
** tolerates them; a query-shaping customizer still refuses the capture before any change.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void mutationCustomizersAreToleratedWhileVisibilityCustomizersRefuseCapture() throws Exception
|
||||
{
|
||||
var file = availableFile("archive-customizers", 1, "bol.txt", "evidence");
|
||||
var table = instance.getTable("document");
|
||||
table.withCustomizer(com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers.POST_QUERY_RECORD,
|
||||
new com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference(NoOpCustomizer.class));
|
||||
assertThatThrownBy(() -> new ArchiveService(attachments, new AuditTrail()).archiveAndDelete("document", 1))
|
||||
.isInstanceOf(QException.class).hasMessageContaining("custom visibility policy");
|
||||
assertThat(ArchiveService.member(RecordFilesTables.FILE, file.id())).isNull();
|
||||
assertThat(attachments.list("document", 1)).hasSize(1);
|
||||
|
||||
table.getCustomizers().remove(com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers.POST_QUERY_RECORD.getRole());
|
||||
for(var role : List.of(com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers.PRE_INSERT_RECORD,
|
||||
com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers.POST_INSERT_RECORD,
|
||||
com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers.PRE_UPDATE_RECORD,
|
||||
com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers.POST_UPDATE_RECORD,
|
||||
com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers.POST_DELETE_RECORD))
|
||||
{
|
||||
table.withCustomizer(role, new com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference(NoOpCustomizer.class));
|
||||
}
|
||||
assertThat(new ArchiveService(attachments, new AuditTrail()).archiveAndDelete("document", 1)).isPositive();
|
||||
assertThat(files.get(file.id()).recordId()).isEqualTo("1");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Host customizer with the interface's default (pass-through) behavior for every role.
|
||||
*******************************************************************************/
|
||||
public static class NoOpCustomizer implements com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Archived originals remain accessible through standard attachment reads and the exact reader
|
||||
** after context restart.
|
||||
|
||||
@@ -0,0 +1,214 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.recordfiles;
|
||||
|
||||
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.regex.Matcher;
|
||||
import java.util.regex.Pattern;
|
||||
import com.github.dockerjava.api.command.InspectContainerResponse;
|
||||
import com.github.dockerjava.api.model.ExposedPort;
|
||||
import com.github.dockerjava.api.model.PortBinding;
|
||||
import com.github.dockerjava.api.model.Ports;
|
||||
import org.testcontainers.DockerClientFactory;
|
||||
import org.testcontainers.containers.GenericContainer;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Where a disposable test container is reachable FROM THIS JVM.
|
||||
**
|
||||
** The original behavior is Testcontainers' own: the host it reports (which honors
|
||||
** TESTCONTAINERS_HOST_OVERRIDE) and the published port. The bridge address of the
|
||||
** sibling is used in exactly one topology: a JVM inside a build container that the
|
||||
** SAME daemon reports on its default bridge (a local full verify through the host
|
||||
** socket). Being inside some container is not enough: on the native runner the job
|
||||
** container is not a container of the isolated test daemon at all, so a sibling's
|
||||
** bridge address (a private address of that other daemon) is unreachable and the
|
||||
** published endpoint at the configured host must be dialed.
|
||||
**
|
||||
** The choice is made from the daemon's own description of the build container's
|
||||
** networks (no connection probing). If the build container cannot be identified or
|
||||
** is not on the default bridge (a job network, host networking, a remote or
|
||||
** separate daemon), the published endpoint is used. Timeouts, scenarios and
|
||||
** assertions are unchanged. The address is re-read from the daemon on every call
|
||||
** (a restarted container may get another bridge IP or published port).
|
||||
*******************************************************************************/
|
||||
final class FixtureEndpoint
|
||||
{
|
||||
private static final String DEFAULT_BRIDGE = "bridge";
|
||||
|
||||
private static final Pattern MOUNTINFO_CONTAINER_ID = Pattern.compile("/docker/containers/([0-9a-f]{64})/");
|
||||
|
||||
private static volatile Boolean buildContainerOnDefaultBridge;
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** Host and port to dial.
|
||||
***************************************************************************/
|
||||
record Address(String host, int port)
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** Utility class.
|
||||
***************************************************************************/
|
||||
private FixtureEndpoint()
|
||||
{
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** Resolve the endpoint of one exposed port of a running container, from a fresh
|
||||
** inspection (not Testcontainers' cached mapping).
|
||||
***************************************************************************/
|
||||
static Address of(GenericContainer<?> container, int internalPort)
|
||||
{
|
||||
InspectContainerResponse inspected = container.getDockerClient().inspectContainerCmd(container.getContainerId()).exec();
|
||||
var networks = inspected.getNetworkSettings().getNetworks();
|
||||
String bridgeIp = networks == null || networks.get(DEFAULT_BRIDGE) == null ? null : networks.get(DEFAULT_BRIDGE).getIpAddress();
|
||||
Integer publishedPort = null;
|
||||
var bindings = inspected.getNetworkSettings().getPorts().getBindings().get(new ExposedPort(internalPort));
|
||||
if(bindings != null && bindings.length > 0 && bindings[0].getHostPortSpec() != null)
|
||||
{
|
||||
publishedPort = Integer.valueOf(bindings[0].getHostPortSpec());
|
||||
}
|
||||
return (choose(onBridgeWithBuildContainer(), bridgeIp, internalPort, container.getHost(), publishedPort));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** Pure decision, separately tested. With the build container on the default
|
||||
** bridge: the sibling's bridge address and its own port. Otherwise the published
|
||||
** endpoint exactly as Testcontainers reports it.
|
||||
***************************************************************************/
|
||||
static Address choose(boolean buildContainerOnDefaultBridge, String bridgeIp, int internalPort, String publishedHost, Integer publishedPort)
|
||||
{
|
||||
if(buildContainerOnDefaultBridge)
|
||||
{
|
||||
if(bridgeIp == null || bridgeIp.isBlank())
|
||||
{
|
||||
throw new IllegalStateException("FIXTURE_ENDPOINT_NO_BRIDGE_ADDRESS: the build container cannot reach a sibling without its bridge address");
|
||||
}
|
||||
return (new Address(bridgeIp, internalPort));
|
||||
}
|
||||
if(publishedHost == null || publishedHost.isBlank() || publishedPort == null)
|
||||
{
|
||||
throw new IllegalStateException("FIXTURE_ENDPOINT_NOT_PUBLISHED: no published endpoint to dial");
|
||||
}
|
||||
return (new Address(publishedHost, publishedPort));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** How the fixture publishes its port: the historical loopback-only binding
|
||||
** wherever a loopback dial can work, otherwise the daemon's own default binding.
|
||||
***************************************************************************/
|
||||
static PortBinding publishedBinding(int internalPort)
|
||||
{
|
||||
boolean loopback = loopbackPublication(onBridgeWithBuildContainer(), DockerClientFactory.instance().dockerHostIpAddress());
|
||||
return (new PortBinding(loopback ? Ports.Binding.bindIpAndPort("127.0.0.1", 0) : Ports.Binding.empty(), new ExposedPort(internalPort)));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** Pure decision, separately tested. A loopback-only publication is reachable
|
||||
** when the build container dials the bridge address (publication unused) or the
|
||||
** Docker host is this machine; if Testcontainers dials another host (a remote or
|
||||
** separate daemon, TESTCONTAINERS_HOST_OVERRIDE) a loopback binding could never
|
||||
** be reached, so the daemon's configured default binding is used instead.
|
||||
***************************************************************************/
|
||||
static boolean loopbackPublication(boolean buildContainerOnDefaultBridge, String dockerHost)
|
||||
{
|
||||
return (buildContainerOnDefaultBridge || dockerHost == null || dockerHost.isBlank()
|
||||
|| dockerHost.equals("localhost") || dockerHost.equals("127.0.0.1") || dockerHost.equals("::1"));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** Whether this JVM runs in a container that the daemon reports on the default
|
||||
** bridge. Decided once per JVM from the daemon's description of this container.
|
||||
***************************************************************************/
|
||||
private static boolean onBridgeWithBuildContainer()
|
||||
{
|
||||
Boolean known = buildContainerOnDefaultBridge;
|
||||
if(known == null)
|
||||
{
|
||||
known = decideBuildContainerOnDefaultBridge();
|
||||
buildContainerOnDefaultBridge = known;
|
||||
}
|
||||
return (known);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** Inspect the build container; anything that cannot be established means the
|
||||
** original published endpoint (false).
|
||||
***************************************************************************/
|
||||
private static boolean decideBuildContainerOnDefaultBridge()
|
||||
{
|
||||
if(!Files.exists(Path.of("/.dockerenv")))
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
try
|
||||
{
|
||||
String id = ownContainerId(Files.readString(Path.of("/proc/self/mountinfo")), System.getenv("HOSTNAME"));
|
||||
if(id == null)
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
var networks = DockerClientFactory.instance().client().inspectContainerCmd(id).exec().getNetworkSettings().getNetworks();
|
||||
return (networks != null && networks.containsKey(DEFAULT_BRIDGE));
|
||||
}
|
||||
catch(Exception e)
|
||||
{
|
||||
return (false);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** The id of the container this JVM runs in: from the mount table when it names
|
||||
** the container directory, otherwise the host name (the container id unless
|
||||
** overridden); null when neither exists.
|
||||
***************************************************************************/
|
||||
static String ownContainerId(String mountInfo, String hostName)
|
||||
{
|
||||
Matcher matcher = MOUNTINFO_CONTAINER_ID.matcher(mountInfo == null ? "" : mountInfo);
|
||||
if(matcher.find())
|
||||
{
|
||||
return (matcher.group(1));
|
||||
}
|
||||
return (hostName == null || hostName.isBlank() ? null : hostName.trim());
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
package com.kingsrook.qbits.recordfiles;
|
||||
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertFalse;
|
||||
import static org.junit.jupiter.api.Assertions.assertNull;
|
||||
import static org.junit.jupiter.api.Assertions.assertThrows;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The topology decision of the sibling-container fixtures.
|
||||
*******************************************************************************/
|
||||
class FixtureEndpointTest
|
||||
{
|
||||
private static final String CONTAINER_ID = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** Build container on the default bridge: dial the sibling bridge IP and its own port, never the published port.
|
||||
***************************************************************************/
|
||||
@Test
|
||||
void buildContainerOnTheBridgeUsesTheBridgeAddress()
|
||||
{
|
||||
assertEquals(new FixtureEndpoint.Address("172.17.0.9", 5432), FixtureEndpoint.choose(true, "172.17.0.9", 5432, "localhost", 41111));
|
||||
assertEquals(new FixtureEndpoint.Address("172.17.0.9", 5432), FixtureEndpoint.choose(true, "172.17.0.9", 5432, null, null));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** Every other topology (JVM on the Docker host, a separate or remote daemon such as the native runner's isolated
|
||||
** test daemon, job network, host networking, unidentified build container) dials the published endpoint
|
||||
** at the host Testcontainers reports (including TESTCONTAINERS_HOST_OVERRIDE), never a private bridge address.
|
||||
***************************************************************************/
|
||||
@Test
|
||||
void everyOtherTopologyDialsThePublishedEndpoint()
|
||||
{
|
||||
assertEquals(new FixtureEndpoint.Address("localhost", 41111), FixtureEndpoint.choose(false, "172.17.0.9", 5432, "localhost", 41111));
|
||||
assertEquals(new FixtureEndpoint.Address("172.17.0.1", 32768), FixtureEndpoint.choose(false, "172.17.0.3", 6379, "172.17.0.1", 32768));
|
||||
assertEquals(new FixtureEndpoint.Address("host.docker.internal", 40000), FixtureEndpoint.choose(false, null, 5432, "host.docker.internal", 40000));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** An unknown endpoint is an error, never a guessed address.
|
||||
***************************************************************************/
|
||||
@Test
|
||||
void missingEndpointFailsClosed()
|
||||
{
|
||||
assertThrows(IllegalStateException.class, () -> FixtureEndpoint.choose(true, null, 5432, "localhost", 41111));
|
||||
assertThrows(IllegalStateException.class, () -> FixtureEndpoint.choose(true, " ", 5432, "localhost", 41111));
|
||||
assertThrows(IllegalStateException.class, () -> FixtureEndpoint.choose(false, "172.17.0.9", 5432, "localhost", null));
|
||||
assertThrows(IllegalStateException.class, () -> FixtureEndpoint.choose(false, "172.17.0.9", 5432, " ", 41111));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** The historical loopback-only publication stays wherever a loopback dial can work; when Testcontainers dials
|
||||
** another host (the native runner's separate daemon) the daemon's default binding is used so the dial can succeed.
|
||||
***************************************************************************/
|
||||
@Test
|
||||
void loopbackPublicationOnlyWhereALoopbackDialCanWork()
|
||||
{
|
||||
assertTrue(FixtureEndpoint.loopbackPublication(false, "localhost"));
|
||||
assertTrue(FixtureEndpoint.loopbackPublication(false, "127.0.0.1"));
|
||||
assertTrue(FixtureEndpoint.loopbackPublication(false, "::1"));
|
||||
assertTrue(FixtureEndpoint.loopbackPublication(false, null));
|
||||
assertTrue(FixtureEndpoint.loopbackPublication(true, "172.17.0.1"));
|
||||
assertFalse(FixtureEndpoint.loopbackPublication(false, "172.17.0.1"));
|
||||
assertFalse(FixtureEndpoint.loopbackPublication(false, "host.docker.internal"));
|
||||
assertFalse(FixtureEndpoint.loopbackPublication(false, "docker.example.test"));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/***************************************************************************
|
||||
** The build container id comes from the mount table when present, else the host name.
|
||||
***************************************************************************/
|
||||
@Test
|
||||
void ownContainerIdPrefersTheMountTableThenTheHostName()
|
||||
{
|
||||
String mountInfo = "700 699 0:50 /docker/containers/" + CONTAINER_ID + "/hostname /etc/hostname rw,relatime - ext4 /dev/sda1 rw";
|
||||
assertEquals(CONTAINER_ID, FixtureEndpoint.ownContainerId(mountInfo, "unrelated"));
|
||||
assertEquals("abc123", FixtureEndpoint.ownContainerId("no container mounts here", " abc123 "));
|
||||
assertNull(FixtureEndpoint.ownContainerId("", null));
|
||||
assertNull(FixtureEndpoint.ownContainerId(null, " "));
|
||||
}
|
||||
|
||||
}
|
||||
@@ -94,6 +94,105 @@ public class IntakeAndRetentionTest extends RecordFilesFixture
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Without the host opt-in, device capture keeps the product quarantine default.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void capturedIntakeQuarantinesByDefault() throws Exception
|
||||
{
|
||||
assertThat(config.getQuarantineCapturedFiles()).isNull();
|
||||
RetainedFile captured = files.intake(new IntakeRequest("captured-default", "document", 1, "photo.jpg", sha("photo"))
|
||||
.withOrigin(RecordFilesValues.ORIGIN_CAPTURE, "CAMERA").withDevice("device-a", null), stream("photo"));
|
||||
assertThat(captured.lifecycle()).isEqualTo(RecordFilesValues.LIFECYCLE_QUARANTINED);
|
||||
assertThat(events()).anySatisfy(event -> assertThat(event.getValueString("detail"))
|
||||
.contains("lifecycle=QUARANTINED", "admissionPolicy=quarantineNewFiles", "quarantine=true"));
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** An opted-in host makes a valid device capture AVAILABLE and audits the policy,
|
||||
** with no release, review or scan event and an unchanged initial review status.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void capturedPolicyMakesValidDeviceCaptureAvailableWithoutReviewClaim() throws Exception
|
||||
{
|
||||
RetainedFile quarantined = files.intake(new IntakeRequest("baseline-upload", "document", 1, "baseline.jpg", sha("baseline"))
|
||||
.withOrigin(RecordFilesValues.ORIGIN_UPLOAD, null), stream("baseline"));
|
||||
config.withQuarantineCapturedFiles(false);
|
||||
assertThat(config.getQuarantineNewFiles()).isTrue();
|
||||
RetainedFile captured = files.intake(new IntakeRequest("captured-open", "document", 1, "photo.jpg", sha("photo"))
|
||||
.withOrigin(RecordFilesValues.ORIGIN_CAPTURE, "CAMERA").withDevice("device-a", null), stream("photo"));
|
||||
assertThat(captured.lifecycle()).isEqualTo(RecordFilesValues.LIFECYCLE_AVAILABLE);
|
||||
assertThat(RecordFilesStore.first(RecordFilesTables.FILE, "id", captured.id()).getValueInteger("reviewStatusId"))
|
||||
.isEqualTo(RecordFilesStore.first(RecordFilesTables.FILE, "id", quarantined.id()).getValueInteger("reviewStatusId"));
|
||||
assertThat(events()).anySatisfy(event ->
|
||||
{
|
||||
assertThat(event.getValueString("eventType")).isEqualTo("FILE_REGISTERED");
|
||||
assertThat(event.getValueString("detail")).contains("lifecycle=AVAILABLE", "admissionPolicy=quarantineCapturedFiles", "quarantine=false");
|
||||
});
|
||||
assertThat(eventTypes()).doesNotContain("FILE_RELEASED", "REVIEW_CHANGE_ATTEMPT");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The captured opt-in needs CAPTURE origin plus a nonblank device identity.
|
||||
** Other origins, standard attachments and missing identity keep quarantine,
|
||||
** and the generated policy is neither widened nor narrowed by it.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void capturedPolicyDoesNotApplyToOtherOriginsOrMissingDeviceIdentity() throws Exception
|
||||
{
|
||||
config.withQuarantineCapturedFiles(false);
|
||||
for(String origin : List.of(RecordFilesValues.ORIGIN_UPLOAD, RecordFilesValues.ORIGIN_IMPORT, RecordFilesValues.ORIGIN_GENERATED))
|
||||
{
|
||||
assertThat(files.intake(new IntakeRequest("other-" + origin, "document", 1, "input.jpg", sha("input"))
|
||||
.withOrigin(origin, "CAMERA").withDevice("device-a", null), stream("input")).lifecycle())
|
||||
.as(origin).isEqualTo(RecordFilesValues.LIFECYCLE_QUARANTINED);
|
||||
}
|
||||
int index = 0;
|
||||
for(String device : new String[]{null, "", " "})
|
||||
{
|
||||
assertThat(files.intake(new IntakeRequest("no-device-" + index++, "document", 1, "nodevice.jpg", sha("nodevice"))
|
||||
.withOrigin(RecordFilesValues.ORIGIN_CAPTURE, "CAMERA").withDevice(device, null), stream("nodevice")).lifecycle())
|
||||
.as("device [" + device + "]").isEqualTo(RecordFilesValues.LIFECYCLE_QUARANTINED);
|
||||
}
|
||||
AttachmentEntry panel = attachments.upload("document", 1, "panel.pdf", "application/pdf", stream("pdf"), "CAMERA");
|
||||
assertThat(RecordFilesStore.first(RecordFilesTables.FILE, "attachmentId", panel.id()).getValueString("lifecycle"))
|
||||
.isEqualTo(RecordFilesValues.LIFECYCLE_QUARANTINED);
|
||||
|
||||
config.withQuarantineGeneratedFiles(true);
|
||||
assertThat(files.intake(new IntakeRequest("generated-still-quarantined", "document", 1, "gen.pdf", sha("gen"))
|
||||
.withOrigin(RecordFilesValues.ORIGIN_GENERATED, null).withDevice("device-a", null), stream("gen")).lifecycle())
|
||||
.isEqualTo(RecordFilesValues.LIFECYCLE_QUARANTINED);
|
||||
config.withQuarantineGeneratedFiles(false).withQuarantineCapturedFiles(true);
|
||||
assertThat(files.intake(new IntakeRequest("generated-open", "document", 1, "gen2.pdf", sha("gen2"))
|
||||
.withOrigin(RecordFilesValues.ORIGIN_GENERATED, null).withDevice("device-a", null), stream("gen2")).lifecycle())
|
||||
.isEqualTo(RecordFilesValues.LIFECYCLE_AVAILABLE);
|
||||
assertThat(events()).anySatisfy(event -> assertThat(event.getValueString("detail"))
|
||||
.contains("origin=GENERATED", "admissionPolicy=quarantineGeneratedFiles"));
|
||||
assertThat(eventTypes()).doesNotContain("FILE_RELEASED");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A hash mismatch is FAILED and inaccessible even when captured files are open.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void capturedPolicyStillFailsHashMismatch() throws Exception
|
||||
{
|
||||
config.withQuarantineCapturedFiles(false);
|
||||
IntakeRequest request = new IntakeRequest("captured-mismatch", "document", 1, "photo.jpg", sha("expected"))
|
||||
.withOrigin(RecordFilesValues.ORIGIN_CAPTURE, "CAMERA").withDevice("device-a", null);
|
||||
assertThatThrownBy(() -> files.intake(request, stream("different"))).isInstanceOf(QBadRequestException.class);
|
||||
assertThat(RecordFilesStore.first(RecordFilesTables.FILE, "fileName", "photo.jpg").getValueString("lifecycle"))
|
||||
.isEqualTo(RecordFilesValues.LIFECYCLE_FAILED);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Generated policy changes no permissions, integrity or mandatory audit checks.
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -0,0 +1,312 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
|
||||
package com.kingsrook.qbits.recordfiles;
|
||||
|
||||
|
||||
import java.sql.Connection;
|
||||
import java.sql.SQLException;
|
||||
import java.time.Duration;
|
||||
import com.kingsrook.qqq.backend.core.exceptions.QException;
|
||||
import com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager;
|
||||
import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData;
|
||||
import org.junit.jupiter.api.AfterAll;
|
||||
import org.junit.jupiter.api.AfterEach;
|
||||
import org.junit.jupiter.api.BeforeAll;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.junit.jupiter.api.condition.EnabledIfSystemProperty;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
import static org.assertj.core.api.Assertions.assertThatThrownBy;
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Opt-in qualification of the Amazon Aurora PostgreSQL rule on a real, stock
|
||||
** PostgreSQL 17 server (the disposable container of PostgresRecordFilesSupport).
|
||||
**
|
||||
** What is simulated: Aurora itself. The fixture superuser creates a function
|
||||
** public.aurora_version() in the test database, so the qualification's probe
|
||||
** to_regprocedure('aurora_version()') resolves as it does on Aurora, and sets
|
||||
** full_page_writes = off with ALTER SYSTEM and a configuration reload, the value
|
||||
** Aurora reports. Aurora's storage layer and its other settings are not
|
||||
** reproduced. Everything else is real: the server, the DML-only application
|
||||
** role, the shipped owner script, the settings query and the store calls.
|
||||
*******************************************************************************/
|
||||
@EnabledIfSystemProperty(named = "postgresRecordFilesTests", matches = "true")
|
||||
class PostgresAuroraQualificationTest extends RecordFilesFixture
|
||||
{
|
||||
private static final String REFUSAL = "Record Files requires its exact writable PostgreSQL 17 database";
|
||||
|
||||
private RDBMSBackendMetaData database;
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Start the provider with fsync and full_page_writes left off the server command
|
||||
** line (both default to on): a setting fixed there cannot be changed by ALTER
|
||||
** SYSTEM, and these cases change both.
|
||||
*******************************************************************************/
|
||||
@BeforeAll
|
||||
static void startPostgres() throws Exception
|
||||
{
|
||||
PostgresRecordFilesSupport.start("synchronous_commit=on");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Remove the provider created by this class.
|
||||
*******************************************************************************/
|
||||
@AfterAll
|
||||
static void stopPostgres()
|
||||
{
|
||||
PostgresRecordFilesSupport.stop();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Every case ends on a stock server with durable settings.
|
||||
*******************************************************************************/
|
||||
@AfterEach
|
||||
void restoreStockDurableServer() throws Exception
|
||||
{
|
||||
execute("DROP FUNCTION IF EXISTS public.aurora_version()");
|
||||
execute("ALTER ROLE " + PostgresRecordFilesSupport.APP_ROLE + " RESET synchronous_commit");
|
||||
execute("ALTER SYSTEM RESET fsync");
|
||||
execute("ALTER SYSTEM RESET full_page_writes");
|
||||
execute("SELECT pg_reload_conf()");
|
||||
awaitSetting("fsync", "on");
|
||||
awaitSetting("full_page_writes", "on");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The native postgres backend with the DML-only application identity.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
protected RDBMSBackendMetaData createBackend() throws Exception
|
||||
{
|
||||
database = PostgresRecordFilesSupport.backend(backendName);
|
||||
return (database);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Generic tables, the Record Files owner script and the journal's PostgreSQL script.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
protected void createSchema(RDBMSBackendMetaData backend) throws Exception
|
||||
{
|
||||
PostgresRecordFilesSupport.createSchema(instance, config);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Owner-level connection for test-only schema fault injection.
|
||||
*******************************************************************************/
|
||||
@Override
|
||||
protected Connection adminConnection(RDBMSBackendMetaData backend) throws Exception
|
||||
{
|
||||
return (PostgresRecordFilesSupport.root());
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The settings query runs as the application role on a stock server without an
|
||||
** error and reports "not Aurora" in column 12; with the function present it
|
||||
** reports Aurora.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void settingsQueryReportsAurora() throws Exception
|
||||
{
|
||||
assertThat(auroraColumn()).isFalse();
|
||||
simulateAurora();
|
||||
assertThat(auroraColumn()).isTrue();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Stock PostgreSQL with full_page_writes off is still refused, by the probe and
|
||||
** by the transaction the store opens.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void stockPostgresWithFullPageWritesOffIsRefused() throws Exception
|
||||
{
|
||||
assertThat(RecordFilesStore.isDurableBackend(database)).isTrue();
|
||||
server("full_page_writes", "off");
|
||||
assertThat(auroraColumn()).isFalse();
|
||||
assertThat(RecordFilesStore.isDurableBackend(database)).isFalse();
|
||||
assertThatThrownBy(RecordFilesStore::openTransaction).isInstanceOf(QException.class).hasMessageStartingWith(REFUSAL);
|
||||
assertThatThrownBy(() -> collections.create("Refused", "document", 1, 1, null, null)).isInstanceOf(QException.class);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Aurora with full_page_writes off qualifies: the probe, the owned transaction
|
||||
** and a committed owner write.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void auroraWithFullPageWritesOffIsAccepted() throws Exception
|
||||
{
|
||||
server("full_page_writes", "off");
|
||||
simulateAurora();
|
||||
assertThat(RecordFilesStore.isDurableBackend(database)).isTrue();
|
||||
var transaction = RecordFilesStore.openTransaction();
|
||||
try
|
||||
{
|
||||
RecordFilesStore.requireCompatibleTransaction(transaction);
|
||||
}
|
||||
finally
|
||||
{
|
||||
transaction.rollback();
|
||||
transaction.close();
|
||||
}
|
||||
var collection = collections.create("Aurora", "document", 1, 1, null, null);
|
||||
assertThat(RecordFilesStore.first(RecordFilesTables.COLLECTION, "id", collection.id()).getValueString("title")).isEqualTo("Aurora");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Aurora relaxes only full_page_writes: fsync off is refused.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void auroraWithFsyncOffIsRefused() throws Exception
|
||||
{
|
||||
server("full_page_writes", "off");
|
||||
simulateAurora();
|
||||
assertThat(RecordFilesStore.isDurableBackend(database)).isTrue();
|
||||
server("fsync", "off");
|
||||
assertThat(RecordFilesStore.isDurableBackend(database)).isFalse();
|
||||
assertThatThrownBy(RecordFilesStore::openTransaction).isInstanceOf(QException.class).hasMessageStartingWith(REFUSAL);
|
||||
server("fsync", "on");
|
||||
assertThat(RecordFilesStore.isDurableBackend(database)).isTrue();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Aurora relaxes only full_page_writes: synchronous_commit off or local is refused.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void auroraWithSynchronousCommitOffIsRefused() throws Exception
|
||||
{
|
||||
server("full_page_writes", "off");
|
||||
simulateAurora();
|
||||
assertThat(RecordFilesStore.isDurableBackend(database)).isTrue();
|
||||
for(String weak : new String[] {"off", "local"})
|
||||
{
|
||||
execute("ALTER ROLE " + PostgresRecordFilesSupport.APP_ROLE + " SET synchronous_commit='" + weak + "'");
|
||||
assertThat(setting("synchronous_commit")).isEqualTo(weak);
|
||||
assertThat(RecordFilesStore.isDurableBackend(database)).isFalse();
|
||||
assertThatThrownBy(RecordFilesStore::openTransaction).isInstanceOf(QException.class).hasMessageStartingWith(REFUSAL);
|
||||
}
|
||||
execute("ALTER ROLE " + PostgresRecordFilesSupport.APP_ROLE + " RESET synchronous_commit");
|
||||
assertThat(RecordFilesStore.isDurableBackend(database)).isTrue();
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** The stand-in for Aurora: a function with Aurora's name in the test database.
|
||||
*******************************************************************************/
|
||||
private void simulateAurora() throws SQLException
|
||||
{
|
||||
execute("CREATE FUNCTION public.aurora_version() RETURNS text LANGUAGE sql AS 'SELECT ''17.9.simulated''::text'");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Change one reload-time server setting and wait until new application sessions see it.
|
||||
*******************************************************************************/
|
||||
private void server(String name, String value) throws Exception
|
||||
{
|
||||
execute("ALTER SYSTEM SET " + name + " = " + value);
|
||||
execute("SELECT pg_reload_conf()");
|
||||
awaitSetting(name, value);
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** A reload is asynchronous; fail rather than assert against a stale value.
|
||||
*******************************************************************************/
|
||||
private void awaitSetting(String name, String value) throws Exception
|
||||
{
|
||||
long deadline = System.nanoTime() + Duration.ofSeconds(30).toNanos();
|
||||
while(!value.equals(setting(name)))
|
||||
{
|
||||
assertThat(System.nanoTime()).as("server setting " + name + " did not become " + value).isLessThan(deadline);
|
||||
Thread.sleep(50);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** One setting as a new application-role session observes it.
|
||||
*******************************************************************************/
|
||||
private String setting(String name) throws Exception
|
||||
{
|
||||
try(Connection connection = ConnectionManager.getConnection(database); var statement = connection.createStatement();
|
||||
var rows = statement.executeQuery("SELECT current_setting('" + name + "')"))
|
||||
{
|
||||
rows.next();
|
||||
return (rows.getString(1));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Column 12 of the shipped settings query, run as the application role.
|
||||
*******************************************************************************/
|
||||
private boolean auroraColumn() throws Exception
|
||||
{
|
||||
try(Connection connection = ConnectionManager.getConnection(database); var statement = connection.createStatement();
|
||||
var rows = statement.executeQuery(RecordFilesPostgres.SETTINGS))
|
||||
{
|
||||
assertThat(rows.next()).isTrue();
|
||||
assertThat(rows.getMetaData().getColumnCount()).isEqualTo(12);
|
||||
return (rows.getBoolean(12));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Fixture superuser statement, separate from native QQQ actions.
|
||||
*******************************************************************************/
|
||||
private void execute(String sql) throws SQLException
|
||||
{
|
||||
try(Connection connection = PostgresRecordFilesSupport.root(); var statement = connection.createStatement())
|
||||
{
|
||||
statement.execute(sql);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,90 @@
|
||||
/*
|
||||
* QQQ - Low-code Application Framework for Engineers.
|
||||
* Copyright (C) 2021-2025. Kingsrook, LLC
|
||||
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
|
||||
* contact@kingsrook.com
|
||||
* https://github.com/Kingsrook/
|
||||
*
|
||||
* Licensed under the Apache License, Version 2.0 (the "License");
|
||||
* you may not use this file except in compliance with the License.
|
||||
* You may obtain a copy of the License at
|
||||
*
|
||||
* https://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing, software
|
||||
* distributed under the License is distributed on an "AS IS" BASIS,
|
||||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
* See the License for the specific language governing permissions and
|
||||
* limitations under the License.
|
||||
*/
|
||||
|
||||
|
||||
package com.kingsrook.qbits.recordfiles;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
/***************************************************************************
|
||||
** The PostgreSQL durability decision as a pure function of the observed
|
||||
** setting values. No server is used here, so this runs in every build; the
|
||||
** same rule on a real PostgreSQL 17 server is PostgresAuroraQualificationTest.
|
||||
***************************************************************************/
|
||||
class PostgresDurabilityRuleTest
|
||||
{
|
||||
/************************************************************************
|
||||
** Stock PostgreSQL keeps the strict rule: full_page_writes must be on.
|
||||
************************************************************************/
|
||||
@Test
|
||||
void stockPostgresRequiresFullPageWrites()
|
||||
{
|
||||
assertThat(RecordFilesPostgres.durable("on", "on", "on", false)).isTrue();
|
||||
assertThat(RecordFilesPostgres.durable("remote_apply", "on", "on", false)).isTrue();
|
||||
assertThat(RecordFilesPostgres.durable("on", "on", "off", false)).isFalse();
|
||||
assertThat(RecordFilesPostgres.durable("remote_apply", "on", "off", false)).isFalse();
|
||||
assertThat(RecordFilesPostgres.durable("on", "on", null, false)).isFalse();
|
||||
}
|
||||
|
||||
/************************************************************************
|
||||
** Amazon Aurora PostgreSQL is accepted with full_page_writes off (and on).
|
||||
************************************************************************/
|
||||
@Test
|
||||
void auroraAcceptsFullPageWritesOff()
|
||||
{
|
||||
assertThat(RecordFilesPostgres.durable("on", "on", "off", true)).isTrue();
|
||||
assertThat(RecordFilesPostgres.durable("remote_apply", "on", "off", true)).isTrue();
|
||||
assertThat(RecordFilesPostgres.durable("on", "on", "on", true)).isTrue();
|
||||
assertThat(RecordFilesPostgres.durable("on", "on", null, true)).isFalse();
|
||||
assertThat(RecordFilesPostgres.durable("on", "on", "unknown", true)).isFalse();
|
||||
}
|
||||
|
||||
/************************************************************************
|
||||
** Aurora relaxes nothing else: fsync and synchronous_commit stay required.
|
||||
************************************************************************/
|
||||
@Test
|
||||
void auroraStillRequiresFsyncAndSynchronousCommit()
|
||||
{
|
||||
for(String fullPageWrites : new String[] {"off", "on"})
|
||||
{
|
||||
assertThat(RecordFilesPostgres.durable("on", "off", fullPageWrites, true)).isFalse();
|
||||
assertThat(RecordFilesPostgres.durable("on", null, fullPageWrites, true)).isFalse();
|
||||
assertThat(RecordFilesPostgres.durable(null, "on", fullPageWrites, true)).isFalse();
|
||||
for(String synchronousCommit : new String[] {"off", "local", "remote_write"})
|
||||
{
|
||||
assertThat(RecordFilesPostgres.durable(synchronousCommit, "on", fullPageWrites, true)).isFalse();
|
||||
assertThat(RecordFilesPostgres.durable(synchronousCommit, "on", fullPageWrites, false)).isFalse();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/************************************************************************
|
||||
** The settings query carries the Aurora probe as its last column, in the
|
||||
** form that cannot raise on stock PostgreSQL. Its execution on a real
|
||||
** server is PostgresAuroraQualificationTest.settingsQueryReportsAurora.
|
||||
************************************************************************/
|
||||
@Test
|
||||
void settingsQueryProbesAuroraWithoutCallingTheFunction()
|
||||
{
|
||||
assertThat(RecordFilesPostgres.SETTINGS).endsWith(", to_regprocedure('aurora_version()') IS NOT NULL");
|
||||
assertThat(RecordFilesPostgres.SETTINGS).doesNotContain("SELECT aurora_version()").doesNotContain("pg_proc");
|
||||
}
|
||||
}
|
||||
@@ -48,6 +48,16 @@ class PostgresIntakeAndRetentionTest extends IntakeAndRetentionTest
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Observe the exact disposable provider after each intake case.
|
||||
*******************************************************************************/
|
||||
@org.junit.jupiter.api.AfterEach
|
||||
void reportFixtureHealth()
|
||||
{
|
||||
PostgresRecordFilesSupport.reportHealth();
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Remove the provider created by this class.
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -30,9 +30,6 @@ import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.TimeZone;
|
||||
import java.util.UUID;
|
||||
import com.github.dockerjava.api.model.ExposedPort;
|
||||
import com.github.dockerjava.api.model.PortBinding;
|
||||
import com.github.dockerjava.api.model.Ports;
|
||||
import com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadJournalTable;
|
||||
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
|
||||
import com.kingsrook.qqq.backend.module.postgres.model.metadata.PostgreSQLBackendMetaData;
|
||||
@@ -54,7 +51,7 @@ public final class PostgresRecordFilesSupport
|
||||
{
|
||||
private static final String DATABASE = "record_files_test";
|
||||
private static final String OPTIONS = "?connectTimeout=5&socketTimeout=30&options=-c%20TimeZone%3DUTC";
|
||||
private static final String APP_ROLE = "record_files_app";
|
||||
public static final String APP_ROLE = "record_files_app";
|
||||
private static final String ROOT_PASSWORD = UUID.randomUUID().toString();
|
||||
private static final String APP_PASSWORD = UUID.randomUUID().toString();
|
||||
private static GenericContainer<?> postgres;
|
||||
@@ -78,19 +75,36 @@ public final class PostgresRecordFilesSupport
|
||||
*******************************************************************************/
|
||||
public static void start() throws Exception
|
||||
{
|
||||
start("fsync=on", "synchronous_commit=on", "full_page_writes=on");
|
||||
}
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Start the disposable server with exactly these name=value settings fixed on
|
||||
** its command line. A setting fixed there cannot be changed later by ALTER
|
||||
** SYSTEM, so a suite that changes a reload-time setting leaves it off this list.
|
||||
*******************************************************************************/
|
||||
public static void start(String... commandLineSettings) throws Exception
|
||||
{
|
||||
List<String> serverCommand = new ArrayList<>(List.of("postgres"));
|
||||
for(String setting : commandLineSettings)
|
||||
{
|
||||
serverCommand.add("-c");
|
||||
serverCommand.add(setting);
|
||||
}
|
||||
previousZone = TimeZone.getDefault();
|
||||
TimeZone.setDefault(TimeZone.getTimeZone("UTC"));
|
||||
postgres = new GenericContainer<>("postgres:17")
|
||||
.withEnv("POSTGRES_PASSWORD", ROOT_PASSWORD).withEnv("POSTGRES_DB", DATABASE)
|
||||
.withCommand("postgres", "-c", "fsync=on", "-c", "synchronous_commit=on", "-c", "full_page_writes=on")
|
||||
.withCommand(serverCommand.toArray(String[]::new))
|
||||
.withExposedPorts(5432)
|
||||
.withCreateContainerCmdModifier(command -> command.getHostConfig().withMemory(768L * 1024 * 1024)
|
||||
.withPortBindings(new PortBinding(Ports.Binding.bindIpAndPort("127.0.0.1", 0), new ExposedPort(5432))))
|
||||
.withPortBindings(FixtureEndpoint.publishedBinding(5432)))
|
||||
.waitingFor(Wait.forLogMessage(".*database system is ready to accept connections.*\\s", 2).withStartupTimeout(Duration.ofMinutes(2)));
|
||||
postgres.start();
|
||||
var inspected = postgres.getDockerClient().inspectContainerCmd(postgres.getContainerId()).exec();
|
||||
var binding = inspected.getNetworkSettings().getPorts().getBindings().get(new ExposedPort(5432))[0];
|
||||
server = "jdbc:postgresql://127.0.0.1:" + binding.getHostPortSpec() + "/";
|
||||
var address = FixtureEndpoint.of(postgres, 5432);
|
||||
server = "jdbc:postgresql://" + address.host() + ":" + address.port() + "/";
|
||||
try(Connection connection = root(); var statement = connection.createStatement())
|
||||
{
|
||||
statement.execute("CREATE ROLE " + APP_ROLE + " LOGIN PASSWORD '" + APP_PASSWORD + "' NOSUPERUSER NOCREATEDB NOCREATEROLE");
|
||||
@@ -100,6 +114,29 @@ public final class PostgresRecordFilesSupport
|
||||
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Bounded fixture diagnosis without environment, SQL, credentials or payloads.
|
||||
*******************************************************************************/
|
||||
public static void reportHealth()
|
||||
{
|
||||
try
|
||||
{
|
||||
var inspection = postgres.getDockerClient().inspectContainerCmd(postgres.getContainerId()).exec();
|
||||
var state = inspection.getState();
|
||||
var address = FixtureEndpoint.of(postgres, 5432);
|
||||
boolean endpointMatches = server.equals("jdbc:postgresql://" + address.host() + ":" + address.port() + "/");
|
||||
org.apache.logging.log4j.LogManager.getLogger(PostgresRecordFilesSupport.class).warn(
|
||||
"POSTGRES_FIXTURE_HEALTH running={} oom={} restarting={} endpointMatches={}",
|
||||
state.getRunning(), state.getOOMKilled(), state.getRestarting(), endpointMatches);
|
||||
}
|
||||
catch(Exception failure)
|
||||
{
|
||||
org.apache.logging.log4j.LogManager.getLogger(PostgresRecordFilesSupport.class).warn(
|
||||
"POSTGRES_FIXTURE_HEALTH inspectionFailure={}", failure.getClass().getSimpleName());
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/*******************************************************************************
|
||||
** Remove only this exact disposable container and restore the JVM zone.
|
||||
*******************************************************************************/
|
||||
|
||||
@@ -29,9 +29,6 @@ import java.time.Duration;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.UUID;
|
||||
import com.github.dockerjava.api.model.ExposedPort;
|
||||
import com.github.dockerjava.api.model.PortBinding;
|
||||
import com.github.dockerjava.api.model.Ports;
|
||||
import com.kingsrook.qbits.recordfiles.audit.AuditTrail;
|
||||
import com.kingsrook.qbits.recordfiles.audit.RecordFilesEvent;
|
||||
import com.kingsrook.qbits.recordfiles.audit.RecordFilesEventType;
|
||||
@@ -87,7 +84,7 @@ class PostgresRecordFilesTest extends RecordFilesFixture
|
||||
.withCommand("postgres", "-c", "fsync=on", "-c", "synchronous_commit=on", "-c", "full_page_writes=on")
|
||||
.withExposedPorts(5432)
|
||||
.withCreateContainerCmdModifier(command -> command.getHostConfig().withMemory(768L * 1024 * 1024)
|
||||
.withPortBindings(new PortBinding(Ports.Binding.bindIpAndPort("127.0.0.1", 0), new ExposedPort(5432))))
|
||||
.withPortBindings(FixtureEndpoint.publishedBinding(5432)))
|
||||
.waitingFor(Wait.forLogMessage(".*database system is ready to accept connections.*\\s", 2).withStartupTimeout(Duration.ofMinutes(2)));
|
||||
postgres.start();
|
||||
refreshEndpoint();
|
||||
@@ -647,8 +644,7 @@ class PostgresRecordFilesTest extends RecordFilesFixture
|
||||
***************************************************************************/
|
||||
private static void refreshEndpoint()
|
||||
{
|
||||
var inspected = postgres.getDockerClient().inspectContainerCmd(postgres.getContainerId()).exec();
|
||||
var binding = inspected.getNetworkSettings().getPorts().getBindings().get(new ExposedPort(5432))[0];
|
||||
server = "jdbc:postgresql://127.0.0.1:" + binding.getHostPortSpec() + "/";
|
||||
var address = FixtureEndpoint.of(postgres, 5432);
|
||||
server = "jdbc:postgresql://" + address.host() + ":" + address.port() + "/";
|
||||
}
|
||||
}
|
||||
@@ -83,6 +83,10 @@ class UploadJsonTest
|
||||
VALID.replace("\"origin\":\"CAMERA\"", "\"origin\":0"),
|
||||
VALID.replace("\"origin\":\"CAMERA\"", "\"origin\":\"GENERATED\""),
|
||||
VALID.replace("\"title\":", "\"quarantineGeneratedFiles\":false,\"title\":"),
|
||||
VALID.replace("\"title\":", "\"quarantineCapturedFiles\":false,\"title\":"),
|
||||
VALID.replace("\"caption\":null", "\"quarantineCapturedFiles\":false,\"caption\":null"),
|
||||
VALID.replace("\"title\":", "\"admissionPolicy\":\"quarantineCapturedFiles\",\"title\":"),
|
||||
VALID.replace("\"origin\":\"CAMERA\"", "\"origin\":\"CAPTURE\""),
|
||||
VALID.replace("\"title\":", "\"title\":null,\"title\":"),
|
||||
VALID + "{}", "null", "[]", "{}");
|
||||
for(String json : invalid)
|
||||
|
||||
+69
-1
@@ -87,10 +87,78 @@ class UploadSessionServiceTest extends RecordFilesFixture
|
||||
** A synthetic manifest preserves its parent alias until owner authorization.
|
||||
*******************************************************************************/
|
||||
private UploadManifest manifest(String title)
|
||||
{
|
||||
return (manifest(title, UploadManifest.Origin.UPLOAD));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** The same synthetic manifest with the client-declared item origin chosen.
|
||||
*******************************************************************************/
|
||||
private UploadManifest manifest(String title, UploadManifest.Origin origin)
|
||||
{
|
||||
return (new UploadManifest(REQUEST, new UploadManifest.Parent("document", "01"), title,
|
||||
List.of(new UploadManifest.Item(ITEM, 0, "receipt.pdf", "application/pdf", (long) CONTENT.length, UploadManifest.hash(CONTENT),
|
||||
UploadManifest.Origin.UPLOAD, null, null, null))));
|
||||
origin, null, null, null))));
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Create, upload and finalize one item; returns the committed lifecycle.
|
||||
*******************************************************************************/
|
||||
private String finalizeLifecycle(UploadManifest.Origin origin) throws Exception
|
||||
{
|
||||
String id = service.create(manifest("Policy", origin)).uploadId();
|
||||
service.putChunk(id, ITEM, 0, CONTENT.length, UploadManifest.hash(CONTENT), new ByteArrayInputStream(CONTENT));
|
||||
var completed = new UploadFinalizer(service, files, collections, clock).finalizeUpload(id, REQUEST);
|
||||
assertThat(completed.state()).isEqualTo("COMPLETE");
|
||||
return (completed.receipt().items().getFirst().lifecycleAtCommit());
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Without the host opt-in, an approved-device camera item is still quarantined.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void deviceCameraUploadQuarantinesByDefault() throws Exception
|
||||
{
|
||||
identity.captured = new CaptureIdentity("test", "alice", "device-a", "public-a");
|
||||
assertThat(finalizeLifecycle(UploadManifest.Origin.CAMERA)).as("default").isEqualTo("QUARANTINED");
|
||||
assertThat(eventTypes()).doesNotContain("FILE_RELEASED");
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** Opted-in host: a camera item with the provider-resolved device identity
|
||||
** becomes AVAILABLE and audits its policy, with no release/review event.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void optedInHostMakesProviderVerifiedDeviceCameraItemAvailable() throws Exception
|
||||
{
|
||||
config.withQuarantineCapturedFiles(false);
|
||||
identity.captured = new CaptureIdentity("test", "alice", "device-a", "public-a");
|
||||
assertThat(finalizeLifecycle(UploadManifest.Origin.CAMERA)).isEqualTo("AVAILABLE");
|
||||
assertThat(events()).anySatisfy(event -> assertThat(event.getValueString("detail"))
|
||||
.contains("origin=CAPTURE", "admissionPolicy=quarantineCapturedFiles", "lifecycle=AVAILABLE"));
|
||||
assertThat(eventTypes()).doesNotContain("FILE_RELEASED", "REVIEW_CHANGE_ATTEMPT");
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** A manifest origin label alone never opts in: no provider identity (desktop)
|
||||
** or a non-capture origin keeps quarantine even when the host opted in.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void manifestCaptureLabelWithoutProviderIdentityStaysQuarantined() throws Exception
|
||||
{
|
||||
config.withQuarantineCapturedFiles(false);
|
||||
assertThat(finalizeLifecycle(UploadManifest.Origin.CAMERA)).isEqualTo("QUARANTINED");
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
** An UPLOAD-origin item stays quarantined even with a device identity.
|
||||
*******************************************************************************/
|
||||
@Test
|
||||
void nonCaptureItemsWithDeviceIdentityStayQuarantined() throws Exception
|
||||
{
|
||||
config.withQuarantineCapturedFiles(false);
|
||||
identity.captured = new CaptureIdentity("test", "alice", "device-a", "public-a");
|
||||
assertThat(finalizeLifecycle(UploadManifest.Origin.UPLOAD)).isEqualTo("QUARANTINED");
|
||||
}
|
||||
|
||||
/*******************************************************************************
|
||||
|
||||
Reference in new issue
Block a user