From 3f08c584d0ba3d1ff41f84f1f15cf5aa50019254 Mon Sep 17 00:00:00 2001 From: James Maes Date: Thu, 5 Nov 2026 08:23:14 -0600 Subject: [PATCH] feat: allow host-controlled admission for captured files Keep the default quarantine policy for ordinary uploads and captures without a provider-resolved device identity. Audit the selected policy without fabricating a release or review event. Co-Authored-By: Claude Sonnet 5.5 --- README.md | 28 +++++- docs/INGESTION-CONTRACT.md | 2 + .../recordfiles/RecordFilesQBitConfig.java | 27 +++++ .../files/RetainedFileService.java | 12 ++- .../recordfiles/IntakeAndRetentionTest.java | 99 +++++++++++++++++++ .../recordfiles/ingestion/UploadJsonTest.java | 4 + .../ingestion/UploadSessionServiceTest.java | 70 ++++++++++++- 7 files changed, 237 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index c409df1..8ee0392 100644 --- a/README.md +++ b/README.md @@ -68,9 +68,33 @@ applies to all trusted generated outputs, not only previews. Hosts must trust ev in-process producer allowed to supply that origin; origin is provenance, not a cryptographic attestation of safe content. +`quarantineCapturedFiles` is a second optional host setting with the same shape +(`getQuarantineCapturedFiles()` / `withQuarantineCapturedFiles(Boolean)`; null, the +default, inherits `quarantineNewFiles`). It applies only to intake with origin +`CAPTURE` **and** a nonblank device identity, and `GENERATED` keeps its own policy. +In the upload path that identity is the server-persisted capture identity that the +host's `CaptureIdentityProvider` resolved and revalidated; the manifest cannot carry +a device, actor, policy or `CAPTURE` label (strict JSON rejects them; `CAMERA` and +`GALLERY` items map to `CAPTURE`, so a device that passes the provider can label an +item either way). A desktop request with no provider identity, or any `UPLOAD`, +`SCANNER_IMPORT`, `IMPORT` or attachment-panel file, keeps `quarantineNewFiles`. +Direct Java `RetainedFileService.intake` callers are trusted host code, like +`GENERATED`: they could pass `CAPTURE` plus a device id, so hosts must not expose +that call to untrusted input. + +Host prerequisites before setting it to `false`: a `CaptureIdentityProvider` that +actually approves each device and actor (a device id alone proves nothing, and Record +Files does not decide approval); an `UploadConfiguration` that bounds media types, +sizes and quotas to what the host accepts (for example JPEG/PNG only); and +parent-table visibility and permissions as before. Nothing scans content, so this +is a host trust decision, not a malware or human-review result. Existing quarantined +files are not released by configuration; release them through the normal release +process. Compatibility: the setting is additive and defaults to null, so existing +hosts and persisted data behave exactly as before. + Upload manifests cannot select `GENERATED` or change host policy. Uploaded, -captured, imported and standard attachment-panel originals retain -`quarantineNewFiles`. Initial admission records `FILE_REGISTERED` with the chosen +imported and standard attachment-panel originals, and captures without a device +identity, retain `quarantineNewFiles`. Initial admission records `FILE_REGISTERED` with the chosen policy and lifecycle, without a release, scan or human-review claim. Hash failures still produce `FAILED`; permissions, current parent authorization and mandatory audit remain enforced. Changing configuration never releases existing quarantined diff --git a/docs/INGESTION-CONTRACT.md b/docs/INGESTION-CONTRACT.md index f54ac30..7c2b200 100644 --- a/docs/INGESTION-CONTRACT.md +++ b/docs/INGESTION-CONTRACT.md @@ -49,6 +49,8 @@ Final receipt: `{uploadId,collectionId,state,completedAt,items:[{itemId,fileId,v Hard ceilings:20MiB per original,200MiB total,100items,64KiB–4MiB fixed chunk size,60–86400second staging lifetime,256KiB manifest JSON. Deployment limits can be lower and are frozen per upload. The original20MiB bound reflects the conditional single-PUT owner path, not a throughput benchmark. Accepted media types are declarations subject to quarantine/scanning, not verified file contents. +Initial lifecycle: finalized items are QUARANTINED unless the host sets the Java-only `quarantineCapturedFiles=false`, which makes only `CAMERA`/`GALLERY` items (origin `CAPTURE`) with the provider-resolved, revalidated device identity AVAILABLE. The manifest carries no policy, device or actor field and strict parsing rejects them. This is a host trust decision (approved devices, bounded media types), not a scan or review result; it emits no release event and leaves review status unchanged. See README "Not a 1.0". + Durable quota defaults:10 active uploads,512MiB reserved staging bytes and100 new uploads/hour per actor. Cancel/expiry does not immediately release capacity: bounded exact-key cleanup must finish and commit its cursor/outcome/quota release. `UploadMaintenance.cleanupOwn` permits currently authorized terminal cleanup. A trusted host scheduler can invoke bounded `cleanupBatch` under `record-files.maintain`; this does not grant content access. Expired finalization fences commit before irreversible staging deletion. Failed deletion/outcome retains cursor and quota for safe retry. Canonical originals are outside cleanup. ## Remaining integration and release evidence diff --git a/src/main/java/com/kingsrook/qbits/recordfiles/RecordFilesQBitConfig.java b/src/main/java/com/kingsrook/qbits/recordfiles/RecordFilesQBitConfig.java index 4d5cae0..0236f07 100644 --- a/src/main/java/com/kingsrook/qbits/recordfiles/RecordFilesQBitConfig.java +++ b/src/main/java/com/kingsrook/qbits/recordfiles/RecordFilesQBitConfig.java @@ -81,6 +81,7 @@ public class RecordFilesQBitConfig implements QBitConfig private List parentTables = List.of(); private Boolean quarantineNewFiles = true; private Boolean quarantineGeneratedFiles; + private Boolean quarantineCapturedFiles; private Integer maxItemsPerCollection = 500; private QCodeReference auditPort = new QCodeReference(TableAuditPort.class); @@ -227,6 +228,32 @@ public class RecordFilesQBitConfig implements QBitConfig + /******************************************************************************* + ** Optional trusted captured-intake policy. Null inherits quarantineNewFiles. + ** Applies only to CAPTURE intake that carries a nonblank server-resolved device + ** identity; never to GENERATED, UPLOAD, IMPORT or attachment-panel files. + *******************************************************************************/ + public Boolean getQuarantineCapturedFiles() + { + return (quarantineCapturedFiles); + } + + + + /******************************************************************************* + ** Host decision for device capture intake, not a scan or human review result. + ** False makes valid captured files AVAILABLE on registration. The host must + ** wire a CaptureIdentityProvider that approves each device and bound the + ** accepted media types; a device id alone proves no approval. + *******************************************************************************/ + public RecordFilesQBitConfig withQuarantineCapturedFiles(Boolean quarantineCapturedFiles) + { + this.quarantineCapturedFiles = quarantineCapturedFiles; + return (this); + } + + + /******************************************************************************* ** Upper bound on the items of one collection. *******************************************************************************/ diff --git a/src/main/java/com/kingsrook/qbits/recordfiles/files/RetainedFileService.java b/src/main/java/com/kingsrook/qbits/recordfiles/files/RetainedFileService.java index 3e5e38d..3aaa76e 100644 --- a/src/main/java/com/kingsrook/qbits/recordfiles/files/RetainedFileService.java +++ b/src/main/java/com/kingsrook/qbits/recordfiles/files/RetainedFileService.java @@ -279,7 +279,15 @@ public class RetainedFileService boolean generatedPolicy = pending != null && RecordFilesValues.ORIGIN_GENERATED.equals(pending.request().origin()) && config.getQuarantineGeneratedFiles() != null; - boolean quarantine = Boolean.TRUE.equals(generatedPolicy ? config.getQuarantineGeneratedFiles() : config.getQuarantineNewFiles()); + // Captured policy needs the service-owned intake scope (never a plain standard upload), the CAPTURE origin + // and a nonblank device identity. UploadFinalizer supplies that identity only from the server-persisted, + // revalidated capture identity; a client cannot choose this policy through the upload JSON. + boolean capturedPolicy = !generatedPolicy && pending != null && RecordFilesValues.ORIGIN_CAPTURE.equals(pending.request().origin()) + && pending.request().deviceId() != null && !pending.request().deviceId().isBlank() + && config.getQuarantineCapturedFiles() != null; + boolean quarantine = Boolean.TRUE.equals(generatedPolicy ? config.getQuarantineGeneratedFiles() + : capturedPolicy ? config.getQuarantineCapturedFiles() : config.getQuarantineNewFiles()); + String admissionPolicy = generatedPolicy ? "quarantineGeneratedFiles" : capturedPolicy ? "quarantineCapturedFiles" : "quarantineNewFiles"; String lifecycle = mismatch ? RecordFilesValues.LIFECYCLE_FAILED : quarantine ? RecordFilesValues.LIFECYCLE_QUARANTINED : RecordFilesValues.LIFECYCLE_AVAILABLE; QRecord row = new QRecord() @@ -313,7 +321,7 @@ public class RetainedFileService audit.withChange(new RecordFilesEvent(RecordFilesEventType.FILE_REGISTERED, actor, correlationId).withRequestId(requestId) .withFile(fileId, attachmentId).withSource(tableName, recordId) .withDetail("lifecycle=" + lifecycle + ", origin=" + row.getValueString("origin") + ", version=" + row.getValueInteger("versionNumber") - + ", admissionPolicy=" + (generatedPolicy ? "quarantineGeneratedFiles" : "quarantineNewFiles") + ", quarantine=" + quarantine + + ", admissionPolicy=" + admissionPolicy + ", quarantine=" + quarantine + (mismatch ? ", declared SHA-256 did not match the stored bytes" : ""))); if(pending != null) { diff --git a/src/test/java/com/kingsrook/qbits/recordfiles/IntakeAndRetentionTest.java b/src/test/java/com/kingsrook/qbits/recordfiles/IntakeAndRetentionTest.java index 7772054..05c2803 100644 --- a/src/test/java/com/kingsrook/qbits/recordfiles/IntakeAndRetentionTest.java +++ b/src/test/java/com/kingsrook/qbits/recordfiles/IntakeAndRetentionTest.java @@ -94,6 +94,105 @@ public class IntakeAndRetentionTest extends RecordFilesFixture + /******************************************************************************* + ** Without the host opt-in, device capture keeps the product quarantine default. + *******************************************************************************/ + @Test + void capturedIntakeQuarantinesByDefault() throws Exception + { + assertThat(config.getQuarantineCapturedFiles()).isNull(); + RetainedFile captured = files.intake(new IntakeRequest("captured-default", "document", 1, "photo.jpg", sha("photo")) + .withOrigin(RecordFilesValues.ORIGIN_CAPTURE, "CAMERA").withDevice("device-a", null), stream("photo")); + assertThat(captured.lifecycle()).isEqualTo(RecordFilesValues.LIFECYCLE_QUARANTINED); + assertThat(events()).anySatisfy(event -> assertThat(event.getValueString("detail")) + .contains("lifecycle=QUARANTINED", "admissionPolicy=quarantineNewFiles", "quarantine=true")); + } + + + + /******************************************************************************* + ** An opted-in host makes a valid device capture AVAILABLE and audits the policy, + ** with no release, review or scan event and an unchanged initial review status. + *******************************************************************************/ + @Test + void capturedPolicyMakesValidDeviceCaptureAvailableWithoutReviewClaim() throws Exception + { + RetainedFile quarantined = files.intake(new IntakeRequest("baseline-upload", "document", 1, "baseline.jpg", sha("baseline")) + .withOrigin(RecordFilesValues.ORIGIN_UPLOAD, null), stream("baseline")); + config.withQuarantineCapturedFiles(false); + assertThat(config.getQuarantineNewFiles()).isTrue(); + RetainedFile captured = files.intake(new IntakeRequest("captured-open", "document", 1, "photo.jpg", sha("photo")) + .withOrigin(RecordFilesValues.ORIGIN_CAPTURE, "CAMERA").withDevice("device-a", null), stream("photo")); + assertThat(captured.lifecycle()).isEqualTo(RecordFilesValues.LIFECYCLE_AVAILABLE); + assertThat(RecordFilesStore.first(RecordFilesTables.FILE, "id", captured.id()).getValueInteger("reviewStatusId")) + .isEqualTo(RecordFilesStore.first(RecordFilesTables.FILE, "id", quarantined.id()).getValueInteger("reviewStatusId")); + assertThat(events()).anySatisfy(event -> + { + assertThat(event.getValueString("eventType")).isEqualTo("FILE_REGISTERED"); + assertThat(event.getValueString("detail")).contains("lifecycle=AVAILABLE", "admissionPolicy=quarantineCapturedFiles", "quarantine=false"); + }); + assertThat(eventTypes()).doesNotContain("FILE_RELEASED", "REVIEW_CHANGE_ATTEMPT"); + } + + + + /******************************************************************************* + ** The captured opt-in needs CAPTURE origin plus a nonblank device identity. + ** Other origins, standard attachments and missing identity keep quarantine, + ** and the generated policy is neither widened nor narrowed by it. + *******************************************************************************/ + @Test + void capturedPolicyDoesNotApplyToOtherOriginsOrMissingDeviceIdentity() throws Exception + { + config.withQuarantineCapturedFiles(false); + for(String origin : List.of(RecordFilesValues.ORIGIN_UPLOAD, RecordFilesValues.ORIGIN_IMPORT, RecordFilesValues.ORIGIN_GENERATED)) + { + assertThat(files.intake(new IntakeRequest("other-" + origin, "document", 1, "input.jpg", sha("input")) + .withOrigin(origin, "CAMERA").withDevice("device-a", null), stream("input")).lifecycle()) + .as(origin).isEqualTo(RecordFilesValues.LIFECYCLE_QUARANTINED); + } + int index = 0; + for(String device : new String[]{null, "", " "}) + { + assertThat(files.intake(new IntakeRequest("no-device-" + index++, "document", 1, "nodevice.jpg", sha("nodevice")) + .withOrigin(RecordFilesValues.ORIGIN_CAPTURE, "CAMERA").withDevice(device, null), stream("nodevice")).lifecycle()) + .as("device [" + device + "]").isEqualTo(RecordFilesValues.LIFECYCLE_QUARANTINED); + } + AttachmentEntry panel = attachments.upload("document", 1, "panel.pdf", "application/pdf", stream("pdf"), "CAMERA"); + assertThat(RecordFilesStore.first(RecordFilesTables.FILE, "attachmentId", panel.id()).getValueString("lifecycle")) + .isEqualTo(RecordFilesValues.LIFECYCLE_QUARANTINED); + + config.withQuarantineGeneratedFiles(true); + assertThat(files.intake(new IntakeRequest("generated-still-quarantined", "document", 1, "gen.pdf", sha("gen")) + .withOrigin(RecordFilesValues.ORIGIN_GENERATED, null).withDevice("device-a", null), stream("gen")).lifecycle()) + .isEqualTo(RecordFilesValues.LIFECYCLE_QUARANTINED); + config.withQuarantineGeneratedFiles(false).withQuarantineCapturedFiles(true); + assertThat(files.intake(new IntakeRequest("generated-open", "document", 1, "gen2.pdf", sha("gen2")) + .withOrigin(RecordFilesValues.ORIGIN_GENERATED, null).withDevice("device-a", null), stream("gen2")).lifecycle()) + .isEqualTo(RecordFilesValues.LIFECYCLE_AVAILABLE); + assertThat(events()).anySatisfy(event -> assertThat(event.getValueString("detail")) + .contains("origin=GENERATED", "admissionPolicy=quarantineGeneratedFiles")); + assertThat(eventTypes()).doesNotContain("FILE_RELEASED"); + } + + + + /******************************************************************************* + ** A hash mismatch is FAILED and inaccessible even when captured files are open. + *******************************************************************************/ + @Test + void capturedPolicyStillFailsHashMismatch() throws Exception + { + config.withQuarantineCapturedFiles(false); + IntakeRequest request = new IntakeRequest("captured-mismatch", "document", 1, "photo.jpg", sha("expected")) + .withOrigin(RecordFilesValues.ORIGIN_CAPTURE, "CAMERA").withDevice("device-a", null); + assertThatThrownBy(() -> files.intake(request, stream("different"))).isInstanceOf(QBadRequestException.class); + assertThat(RecordFilesStore.first(RecordFilesTables.FILE, "fileName", "photo.jpg").getValueString("lifecycle")) + .isEqualTo(RecordFilesValues.LIFECYCLE_FAILED); + } + + + /******************************************************************************* ** Generated policy changes no permissions, integrity or mandatory audit checks. *******************************************************************************/ diff --git a/src/test/java/com/kingsrook/qbits/recordfiles/ingestion/UploadJsonTest.java b/src/test/java/com/kingsrook/qbits/recordfiles/ingestion/UploadJsonTest.java index d252248..d089ca3 100644 --- a/src/test/java/com/kingsrook/qbits/recordfiles/ingestion/UploadJsonTest.java +++ b/src/test/java/com/kingsrook/qbits/recordfiles/ingestion/UploadJsonTest.java @@ -83,6 +83,10 @@ class UploadJsonTest VALID.replace("\"origin\":\"CAMERA\"", "\"origin\":0"), VALID.replace("\"origin\":\"CAMERA\"", "\"origin\":\"GENERATED\""), VALID.replace("\"title\":", "\"quarantineGeneratedFiles\":false,\"title\":"), + VALID.replace("\"title\":", "\"quarantineCapturedFiles\":false,\"title\":"), + VALID.replace("\"caption\":null", "\"quarantineCapturedFiles\":false,\"caption\":null"), + VALID.replace("\"title\":", "\"admissionPolicy\":\"quarantineCapturedFiles\",\"title\":"), + VALID.replace("\"origin\":\"CAMERA\"", "\"origin\":\"CAPTURE\""), VALID.replace("\"title\":", "\"title\":null,\"title\":"), VALID + "{}", "null", "[]", "{}"); for(String json : invalid) diff --git a/src/test/java/com/kingsrook/qbits/recordfiles/ingestion/UploadSessionServiceTest.java b/src/test/java/com/kingsrook/qbits/recordfiles/ingestion/UploadSessionServiceTest.java index 852579c..31b57a0 100644 --- a/src/test/java/com/kingsrook/qbits/recordfiles/ingestion/UploadSessionServiceTest.java +++ b/src/test/java/com/kingsrook/qbits/recordfiles/ingestion/UploadSessionServiceTest.java @@ -87,10 +87,78 @@ class UploadSessionServiceTest extends RecordFilesFixture ** A synthetic manifest preserves its parent alias until owner authorization. *******************************************************************************/ private UploadManifest manifest(String title) + { + return (manifest(title, UploadManifest.Origin.UPLOAD)); + } + + /******************************************************************************* + ** The same synthetic manifest with the client-declared item origin chosen. + *******************************************************************************/ + private UploadManifest manifest(String title, UploadManifest.Origin origin) { return (new UploadManifest(REQUEST, new UploadManifest.Parent("document", "01"), title, List.of(new UploadManifest.Item(ITEM, 0, "receipt.pdf", "application/pdf", (long) CONTENT.length, UploadManifest.hash(CONTENT), - UploadManifest.Origin.UPLOAD, null, null, null)))); + origin, null, null, null)))); + } + + /******************************************************************************* + ** Create, upload and finalize one item; returns the committed lifecycle. + *******************************************************************************/ + private String finalizeLifecycle(UploadManifest.Origin origin) throws Exception + { + String id = service.create(manifest("Policy", origin)).uploadId(); + service.putChunk(id, ITEM, 0, CONTENT.length, UploadManifest.hash(CONTENT), new ByteArrayInputStream(CONTENT)); + var completed = new UploadFinalizer(service, files, collections, clock).finalizeUpload(id, REQUEST); + assertThat(completed.state()).isEqualTo("COMPLETE"); + return (completed.receipt().items().getFirst().lifecycleAtCommit()); + } + + /******************************************************************************* + ** Without the host opt-in, an approved-device camera item is still quarantined. + *******************************************************************************/ + @Test + void deviceCameraUploadQuarantinesByDefault() throws Exception + { + identity.captured = new CaptureIdentity("test", "alice", "device-a", "public-a"); + assertThat(finalizeLifecycle(UploadManifest.Origin.CAMERA)).as("default").isEqualTo("QUARANTINED"); + assertThat(eventTypes()).doesNotContain("FILE_RELEASED"); + } + + /******************************************************************************* + ** Opted-in host: a camera item with the provider-resolved device identity + ** becomes AVAILABLE and audits its policy, with no release/review event. + *******************************************************************************/ + @Test + void optedInHostMakesProviderVerifiedDeviceCameraItemAvailable() throws Exception + { + config.withQuarantineCapturedFiles(false); + identity.captured = new CaptureIdentity("test", "alice", "device-a", "public-a"); + assertThat(finalizeLifecycle(UploadManifest.Origin.CAMERA)).isEqualTo("AVAILABLE"); + assertThat(events()).anySatisfy(event -> assertThat(event.getValueString("detail")) + .contains("origin=CAPTURE", "admissionPolicy=quarantineCapturedFiles", "lifecycle=AVAILABLE")); + assertThat(eventTypes()).doesNotContain("FILE_RELEASED", "REVIEW_CHANGE_ATTEMPT"); + } + + /******************************************************************************* + ** A manifest origin label alone never opts in: no provider identity (desktop) + ** or a non-capture origin keeps quarantine even when the host opted in. + *******************************************************************************/ + @Test + void manifestCaptureLabelWithoutProviderIdentityStaysQuarantined() throws Exception + { + config.withQuarantineCapturedFiles(false); + assertThat(finalizeLifecycle(UploadManifest.Origin.CAMERA)).isEqualTo("QUARANTINED"); + } + + /******************************************************************************* + ** An UPLOAD-origin item stays quarantined even with a device identity. + *******************************************************************************/ + @Test + void nonCaptureItemsWithDeviceIdentityStayQuarantined() throws Exception + { + config.withQuarantineCapturedFiles(false); + identity.captured = new CaptureIdentity("test", "alice", "device-a", "public-a"); + assertThat(finalizeLifecycle(UploadManifest.Origin.UPLOAD)).isEqualTo("QUARANTINED"); } /******************************************************************************* -- 2.54.0