qbit-record-files/docs/ARCHIVED-SOURCE-DESIGN.md

Archived source access and database qualification proposal

Status: implementation and focused verification in progress, 2026-10-30. The archive association, native delete action, owner transfer and current-actor security path are implemented. The latest focused run passed 19 of 20 tests: eight archive cases and eleven content-reader/dispatch cases. The remaining archive assertion expected an exception where native H2 Get correctly returned no record; that assertion now tests non-disclosure directly, with a separate query absence check. New aggregate and corrupt-snapshot regressions await the next run. This is not a production qualification claim.

Evidence from the actual framework

• Core RecordReferenceSecurityHelper.secureFilter (line 87), visibleParent (153) and canReadReference (176) require a currently readable parent. Attachment lifecycle hooks alone cannot restore ordinary attachment Get/Query visibility after that parent disappears.
• ValidateRecordSecurityLockHelper.validateSecurityFields (92) evaluates the native READ lock tree and propagates AND/OR results through RecordWithErrors (117). It uses the current QContext session. Its separate allowedToReadRecord (185) returns false when any leaf inserted an error without propagating the Boolean tree; do not use that shortcut for archived OR policies.
• Native lock evaluation uses stored typed field values, current security keys and current all-access key definitions. Joined locks resolve stored joined rows (the branch beginning around line 282). Missing values cannot simply be omitted from a snapshot: native null-ALLOW semantics differ from an incomplete or corrupt snapshot.
• DeleteAction calls required record-change listeners after DML (398, 843), and post-delete customizers after that (403). Post-delete customizer exceptions are caught. The action does not own a transaction. A required listener's exception tells its caller to roll back; it cannot undo an already autocommitted delete.
• A table has one customizer per role (QTableMetaData.withCustomizer, 652). Installing a deletion guard must compose the existing pre-delete customizer, never silently replace it.

Recommended ownership boundary

Record Files owns one immutable archive association per source-record incarnation, not another byte store. An opaque association ID prevents primary-key reuse from attaching old evidence to a newly created record. The association preserves the original table/key, typed minimal security projection, policy version/digest, capture actor/time and delete event correlation. Original file provenance and hashes remain unchanged.

Collaboration owns a small, explicit retained-reference transfer operation. It changes only its attachment parent reference from the live source to the archive association, in the caller's validated real transaction. Its lifecycle policy must authorize that exact one-time transfer; default policy denies it. Record Files never updates Collaboration tables. The owner validates that the supplied archive association matches the old canonical source and that the operation participates in the same database transaction as the source delete and success audit.

The archive association is an ordinary QQQ table. Native attachment reference security then resolves that table through ordinary Get/Query, including its Record Files authorization customizer. No core dependency on Record Files and no special global archive-view permission are needed. Ordinary attachment preview, download and verified content reads keep the owner lifecycle/audit path.

Record Files must centralize source authorization for its own file, collection, event and reader paths. A file remains bound to its original source incarnation, and an archived association is resolved by that immutable binding. It is never selected by treating an invisible live parent as deleted. Existing content references continue to identify an exact retained version, never a substituted latest version; an owner reference transfer must not change the original source identity recorded in that reference.

Authorization and capture contract

For live sources, continue normal current-parent authorization. For sealed archived sources, require all of the following on every read:

1. Current Record Files/Collaboration operation permission and current original-table READ permission, including current channel and scoped-credential restrictions.
2. A complete server-captured projection with explicit typed nulls and a valid immutable policy digest; no caller-supplied security values or session objects.
3. Native READ-lock tree evaluation against the current caller's security keys. Preserve AND/OR, READ scope, all-access and null behavior. Never persist the capturing actor's access grants as an authorization decision.
4. Current trusted host policy for any visibility constraints beyond native locks. A post-query customizer or external tenant policy cannot be inferred from a lock snapshot.
5. Exact retained lifecycle, source-incarnation and content identity checks, plus durable access intent before bytes are released.

Capture should occur under the source deletion transaction after locking the authoritative stored source, not from presentation-transformed data or an intake-time snapshot that may precede a tenant transfer. Capture only security values, not the entire business record. Direct-field locks can be evaluated through a cloned metadata projection and the native evaluator. Joined locks need a supported capture operation preserving the same native joined-value resolution and missing-join semantics. Extracting this small generic capture/evaluate primitive into core is preferable to copying the private joined-lock query engine into Record Files. If a host has custom row visibility, a trusted Java archive policy must capture and recheck its extra constraints. Unsupported policies must block deletion with retained files until configured; they must not silently become public archives.

Policy removal or drift must not discard historic restrictions. A changed lock/key-type definition requires an explicit audited policy migration that retains the old constraints or a trusted compatibility decision. Table removal likewise requires a retained archive authority definition; absence of current metadata is not permission. The first release must document and test the exact supported policy forms, rather than claim arbitrary customizers are automatically supported.

Source deletion transaction

Expose a normal QQQ archive-and-delete action process backed by a narrow Record Files service. It authorizes the source's current DELETE permission and write locks, locks the source, captures security, inserts/seals its archive association, invokes the owner's retained-reference transfer, executes native DeleteAction with the same transaction, writes the success event and commits. An independent attempt precedes this transaction. Any failed transfer, delete or success audit rolls everything back.

A composed pre-delete guard rejects uncoordinated deletion when the source has retained evidence. The trusted service scope must bind exact source identity and exact transaction; a caller Boolean is insufficient. Existing host pre-delete behavior still runs. Preview is side-effect free. Deletion of sources with no retained files keeps ordinary behavior.

Initial atomic support requires the source, owner attachment and Record Files tables to share a qualified transactional database. Cross-database deletion needs a separately designed durable protocol; pretending a shared Java transaction spans multiple databases is not acceptable. Joined security data must have a consistent capture boundary. Parent updates/deletes racing capture and new intake must serialize against the same source/incarnation fence.

Acceptance tests before release

• Standard QQQ Get/Query and owner download/preview/readContent work after authorized source deletion and process/JDBC restart; bytes, checksum and original provenance are unchanged.
• Tenant A cannot read tenant B's archived files; revoked table READ, channel, credential scope, security key and custom host policy deny before storage access. Same source ID recreated for another tenant does not expose old evidence or merge new files into its archive.
• Direct and joined locks; nested AND/OR; READ versus WRITE scope; null ALLOW/DENY; all-access keys; typed numeric/string values; missing/corrupt projection and changed policy. Joined rows deleted after capture do not erase captured restrictions.
• Failed mandatory attempt, failed owner transfer, failed source delete and failed success audit each leave no committed partial archive/transfer/delete. Independent attempt remains truthful. Test real concurrent intake/update/delete and conflicting archive requests.
• Direct native DeleteAction cannot bypass the guard, including suppressed notifications and supplied ambient transactions. Existing pre-delete customizer executes once; preview makes no changes. No-retained-file deletes remain ordinary.
• Generic attachment tables and Record Files tables cannot bypass archive authorization through broad filters, aggregates, Get, preview customizers or historical content references.

Production database scope

Current RecordFilesStore.isDurableBackend (213) intentionally qualifies only exact RDBMS/H2 file URLs with WRITE_DELAY=0. It is evidence-based but is not broad host support. requireCompatibleTransaction (241) currently compares a literal configured H2 URL; lock (276) emits SELECT FOR UPDATE.

The actual framework provides MySQL, Aurora-MySQL and H2 defaults in ConnectionManager (151, 171). It also provides separate PostgreSQLBackendModule and SQLiteBackendModule subclasses with their own metadata and strategies. PostgreSQL metadata supplies vendor postgres and a generated JDBC URL. Consequently, merely adding vendor strings to the current exact-module test is insufficient, and metadata.getJdbcUrl may be null for valid native connection settings.

Recommended first production qualification: MySQL 8 with InnoDB and PostgreSQL through their native modules/strategies, retaining file-H2 as the deterministic local fixture. Use a small internal database dialect/qualification component for DDL, transaction identity and row locking. Open transactions through the actual registered backend, and validate connection/database/catalog/schema identity without treating arbitrary connection URL options or a self-asserted durability flag as proof. Require writable non-autocommit connections, supported isolation and actual transactional tables. Validate engine durability settings and document the operator/storage guarantees that application code cannot prove.

Run the same rollback, independent-intent, metadata-callback participation, uniqueness/idempotency, concurrent reservation/reorder/delete, disconnect/restart and actual generated-schema suites against each real engine. Test generated native connection URLs and both supported PostgreSQL schemas/MySQL catalogs; reject incompatible external owner transactions before mutation. Aurora-compatible SQL is not a failover/durability qualification: add a matching engine integration suite and deployment acceptance for failover. SQLite has a different writer/locking model and cannot reuse the current independent writer intents plus SELECT FOR UPDATE unchanged; keep it explicitly unsupported until its separate protocol is designed and tested.

The current PostgreSQL DDL is prospective, not evidence that runtime transaction dispatch, indexes, triggers, lock semantics and JDBC conversions have passed together. No database should be advertised as supported before those complete suites pass.