diff --git a/AGENTS.md b/AGENTS.md index 29e8802..ee35932 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -42,7 +42,7 @@ This file provides guidance to Codex when working with code in this repository. ## Dependencies - Java 21. -- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-RC.5` ahead of it, as in qbit-quick-search (ADR-0007). Remove that activation when the parent moves to 4.1. +- Parent `com.kingsrook:qbit-build-parent:2.1.1`. The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-SNAPSHOT`; build the matching Core integration branch first (see `docs/CORE-DEPENDENCY.md`). Public RC.21 lacks required rollback callbacks. - Tests: JUnit 5, AssertJ. ## Release diff --git a/CHANGELOG.md b/CHANGELOG.md index dc85e34..b5b5d94 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,8 @@ All notable changes are documented here, following [Keep a Changelog](https://ke ### Added +- Attachment lifecycle SPI (`AttachmentLifecyclePolicy`, `CollaborationQBitConfig.withAttachmentLifecycle`) for retention, registration and mandatory access events; durable upload journal (`collaboration_upload_journal`, H2/MySQL/PostgreSQL), upload recovery and existing-attachment adoption processes, durable cleanup queue with operator settlement, verified/authenticated delivery (`AttachmentDownload`, `AttachmentContentIntegrityException`), media policy, aggregate guard and bounded attachment pagination. See `docs/ATTACHMENT-LIFECYCLE.md`, `docs/ATTACHMENT-RECOVERY.md`, `docs/RETAINED-UPLOAD-JOURNAL.md`, `docs/ATTACHMENT-PAGINATION.md`. +- Requires a core whose `QBackendTransaction` provides `supportsRollbackCallbacks()` and `addAfterRollbackCallback(Runnable)` (not in 4.1.0-RC.21); see `docs/CORE-DEPENDENCY.md`. - Feature Settings table action **Create rows from configuration**: writes the code configuration out as editable `collaboration_feature_setting` rows (insert-only, only when the table is empty, no change in behavior). - Setup card on the Collaboration home: each record feature, whether it is on, for which tables, and where that comes from. - Host-neutral QBIT help bundle covering tables, fields, sections, built-in actions, the app, widgets and possible-value sources; core registration and `HelpBundleAssertions` coverage replace the temporary help loader. diff --git a/README.md b/README.md index 4f47ed6..ec0266f 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,8 @@ # QBit Collaboration +> **Version 1.0.** Published to the KOF22 Gitea Maven registry (`com.kingsrook.qbits`). Requires QQQ Core `4.1.0-RC.22` or later. See [current status](docs/QBIT-RELEASE-STATUS.md). + + Record collaboration for QQQ: attachments, tags, custom fields, comments with mentions, presence, sharing, favorites, follow, timeline and tasks. **Implemented now:** feature settings and people tables, favorites and recents services (phase b), per-user security, daily recents retention, phase-c attachments (upload/list/download/rename/delete, previews and parent cleanup), and the Collaboration app home with real session-scoped counts. Phase e adds comments/mentions, follow actions and digests, the audit/activity timeline, and automatic people-directory updates. Phase d adds tags, vocabulary policies, management and bulk actions, and core-integrated tag filtering/search. Phases h/i add sharing/access requests, tasks/reminders, assignments and My work services. Phases f/g add the custom-field designer and ephemeral replicated presence. Runtime overrides and TAGS mode settings resolve through core. Design: [QRun-IO/qqq PR #1136](https://github.com/QRun-IO/qqq/pull/1136), `docs/COLLABORATION.md`. See [the phase table](docs/DESIGN.md). @@ -28,10 +31,10 @@ Comment parents list the selected thread's root and replies. Feature-setting sco ## Requirements - Java 21. -- QQQ `4.1.0-RC.7` from Maven Central; middleware Javalin is a compile dependency. -- A configured QQQ backend supporting query, get, insert, update, delete, count and aggregate actions. PostgreSQL 15+ is the intended persistent store (the generated personal-target indexes use `NULLS NOT DISTINCT`); tests use the memory backend. +- QQQ `4.1.0-SNAPSHOT` built from the compatible Core integration branch; see [Core dependency setup](docs/CORE-DEPENDENCY.md). Middleware Javalin is a compile dependency. Public RC.21 lacks the required rollback API. +- A configured QQQ backend supporting query, get, insert, update, delete, count and aggregate actions. PostgreSQL 15+ is the intended persistent store (the generated personal-target indexes use `NULLS NOT DISTINCT`); tests use memory, H2 and disposable PostgreSQL/MySQL containers. - Attachments require `qbit-object-storage:0.1.0-RC.2` (a pinned dependency), installed with a PRIVATE bucket alias matching `attachmentsBucketName`. Configure its connection and credentials in the host; this qbit does not provision buckets. -- Core feature resolution, permissions, reference security, summaries, query expansion and realtime use QQQ `4.1.0-RC.7`. +- Core feature resolution, permissions, reference security, summaries, query expansion and realtime use the same matching Core snapshot. ## Followers and secured data @@ -54,7 +57,7 @@ Comment bodies remain user content; realtime field-name filtering is handled by ## Demo server `CollaborationDevServer` is a disposable, memory-backed Sales CRM for trying the -Next dashboard and taking screenshots. It needs Java 21 and QQQ `4.1.0-RC.7` from Maven Central. No database, Docker or MinIO is needed. It binds +Next dashboard and taking screenshots. It needs Java 21 and the matching Core integration snapshot. No database, Docker or MinIO is needed. It binds to `127.0.0.1`; stopping the JVM discards edits and uploaded attachments. Build the Next static export in `qqq-frontend-next`: diff --git a/docs/ATTACHMENT-LIFECYCLE.md b/docs/ATTACHMENT-LIFECYCLE.md new file mode 100644 index 0000000..3bfb3df --- /dev/null +++ b/docs/ATTACHMENT-LIFECYCLE.md @@ -0,0 +1,81 @@ +# Attachment lifecycle extension + +Collaboration owns standard attachment storage and record access. A host can configure `CollaborationQBitConfig.withAttachmentLifecycle(new QCodeReference(HostPolicy.class))` to enforce retention, registration and mandatory access-event persistence. `HostPolicy` extends `AttachmentLifecyclePolicy`. Collaboration has no dependency on Record Files, generation, scripting or a particular event store. + +The default policy preserves legacy behavior for ordinary attachments. Invalid policy loading fails before storage mutation. This is a trusted Java extension, never an entry point for untrusted user scripts. Policy implementations must be safe for concurrent requests and may not depend on per-instance state surviving between operations. + +## Contract + +| Hook | Timing and meaning | +| --- | --- | +| `isRetained(attachment)` | Called before object storage during intake with complete metadata but no generated id. Its result is persisted in the server-controlled `retained` field. Also consulted for legacy rows during deletion and display hints. A persisted true value always wins. | +| `beforeStored(attachment)` | Called with canonical metadata and actual checksum immediately before the first object PUT, after matching retry replay returns. Retained files default to refusal; a mandatory provider commits storage intent independently. Retained uploads reject ambient transactions. | +| `afterStored(attachment)` | Called after object and attachment metadata insertion, before upload returns. Register the retained identity and ingestion status. Failure preserves the object and inserted row for reconciliation when the enclosing transaction permits it. | +| `beforeAccess(attachment, inline)` | Called after parent authorization, before signing either standard preview or download URLs. Throw to deny. Runtime policy failures are converted to checked failures so preview code cannot silently swallow them. | +| `beforeMetadataChange(attachment, changes)` | Called after parent/ownership checks, before applying filename/description changes. The second record contains id and changed fields only. This represents an intent, not a committed change. | + +Callbacks receive record copies; scalar attachment metadata is the contract. They must not mutate the passed records to influence storage or authorization. They may throw `QException` to fail the operation. A failed registration must remain inaccessible through the policy until reconciliation succeeds; absence of a retained registry entry must never mean unrestricted access. + +## Persistent retention + +`collaboration_attachment.retained` is a required boolean with false default for existing ordinary attachments. Apply the generated PostgreSQL schema migration through the host's normal migration tooling before installing this version. The migration is `ALTER TABLE collaboration_attachment ADD COLUMN IF NOT EXISTS retained BOOLEAN NOT NULL DEFAULT FALSE`. It does not automatically classify pre-existing files; any historical backfill requires a deliberate host migration. + +There is no public API to clear retention. Direct user table writes remain denied. Deletion first authorizes and checks retention for the complete selection; it checks retention again immediately before storage deletion. Parent cleanup tombstones retained attachment rows and excludes completed tombstones from subsequent pages. The original bytes remain. The ordinary parent-based UI remains inaccessible after parent deletion; a retained-history view must supply its own preserved authorization policy. This extension does not grant archive access. + +Removing the configured provider cannot restore deletion of rows already marked retained. The no-op provider also refuses signed URLs and metadata edits for those retained rows. Replacing a provider is a trusted host operation: the replacement must still enforce audit and quarantine for every row marked retained, even if its table is no longer configured for new retention. + +Attachment DTOs expose separate `canEdit` and `canDelete` hints. Both require current ownership/admin and attach capability; retention disables deletion. Existing constructor overloads preserve source compatibility. The server rechecks all permissions on mutation. The old `canDelete` flag must no longer control client rename/description actions. + +## Verified content for integrations + +`AttachmentsService.readContent(attachmentId, maximumBytes)` authorizes the actual parent/current actor, enforces both content-read policy phases, validates stored length and SHA-256, closes storage on all paths and returns immutable `AttachmentContent(attachmentId, fileName, contentType, sha256, bytes)`. No storage coordinates or signed URL escape. The caller chooses a limit up to 20 MiB; ordinary large-file download remains a separate streaming/presigned path. No bytes leave this method before integrity verification and the result audit hook succeed. + +`beforeContentRead(attachment)` must authorize and durably record the attempt before storage opens. `afterContentRead(attachment, bytesRead)` must record verified server-read completion before returning content. Both default to refusal for retained files until the lifecycle provider implements them. These facts are distinct from URL issuance, user download, printer spooling and physical output. + +## Retryable intake + +An upload overload adds `intakeKey` and `manifestSha256` after the ordinary upload arguments. Both are server-coordinator SHA-256 identifiers. The owner additionally hashes actual authenticated actor and canonical parent into the storage scope; never trust a client key as authority. The intake fingerprint includes the caller's manifest, filename, declared type, description, actual length and actual content hash. Every retry reads and checks its supplied bytes. A matching attachment is returned and its idempotent registration hook is retried; changed inputs conflict. + +This path rejects ambient transactions. It conditionally creates a deterministic storage key with fingerprint metadata and never overwrites or deletes an uncertain original. If storage succeeded but metadata insertion failed, the next attempt verifies the existing bytes/fingerprint and completes registration. Host migration must add nullable `intake_key`/`intake_fingerprint` columns and the unique nonnull intake-key index from `CollaborationSchema`. That index is the database race boundary; sequential memory fixtures do not prove its concurrency behavior. The calling coordinator still owns durable request reservation, lifecycle and reconciliation scheduling. + +Retryable files are bounded to 20 MiB and use a conditional single-object PUT. The inspected object-storage multipart implementation does not carry conditional-create semantics through completion; it is deliberately not used here. Larger retryable originals need a separately qualified conditional multipart path before this bound changes. This is an advertised intake limit, not a claim of unlimited file sizes or storage immutability. Real storage conditional behavior, object version/lock and backup retention still require provider acceptance. + +## Atomic display metadata auditing + +`beforeMetadataChange` records an independent attempt. Before the update, `validateMetadataTransaction(attachment, transaction)` validates the actual transaction and audit backend; retained files default to denial. After the update, `afterMetadataChange(attachment, changes, transaction)` inserts the success event into that same transaction and also defaults to denial for retained files. The provider must not commit or close the supplied transaction. Reject no-op transactions before mutation; memory tests only establish callback behavior. + +When no caller transaction exists, Collaboration opens, commits, rolls back and closes the owner transaction. If a caller supplied a transaction, that caller owns completion and must roll back on any propagated failure. Successful operation and its event become durable together on commit. Record Files must qualify this with a real supported JDBC backend; this generic owner SPI cannot prove provider durability. + +## Durability boundaries and remaining integration + +The callback is not an audit database. Before returning from `beforeAccess`, a mandatory audit provider must have durably committed its intent independently of a caller transaction that may later roll back. An event means URL issuance was requested, not completed download or physical printing. Storage signing can still fail after the intent is recorded. Signed URLs can be reused until expiration; each use is not observed by Collaboration. Storage access telemetry or a serving gateway is required for transfer events. + +Retained originals are never deleted when metadata insertion fails; ordinary non-retained failed inserts retain legacy cleanup. A committed STORAGE_WRITE_ATTEMPT can identify an orphan even when no attachment row was inserted. + +An object put and QQQ metadata insert are not an atomic transaction. If a surrounding transaction rolls back after object creation, reconciliation must recover an orphaned object. The retained-only journal integration below provides explicit registration recovery; it does not provide malware scanning, WORM storage protection, bucket lifecycle protection or backup verification. Those remain deployment requirements. + +QQQ record-reference write validation issues an internal GetAction using global metadata. Attachment previews use the existing `QueryOrGetInputInterface.getShouldGenerateDisplayValues()` contract: visibility filtering always runs, while URL generation runs only when display values were requested. Core validation GETs default to false; the standard middleware's display GET/query executors explicitly request true. This avoids incidental signing during validation without removing validators or modifying QQQ core. Trusted callers requesting presentation URLs directly through GetAction/QueryAction must explicitly enable display values; `AttachmentsService.downloadUrl` remains an explicit authorized access operation. + +## Aggregate visibility policy + +The optional configuration-only `requiresPinnedAggregateParent()` hook defaults to false. Owners whose parent visibility relies on dynamic archive/project authorization opt in. AttachmentAggregateGuard preserves the normal collaboration permission checker and additionally requires Count/Aggregate inputs to contain one literal top-level AND equality for both `tableName` and `recordId`. It checks current native parent visibility before SQL; joins, set-operation filters, OR roots, missing and conflicting pins are rejected. It does not grant view access. + +Opt-in table metadata hides generic TABLE_COUNT and QUERY_STATS capabilities in both instance and table metadata. The current frontend's use-filter-setup hook already skips totals when TABLE_COUNT is absent and represents them as unknown; record listings and the attachment panel remain available. Native CountAction still permits an authorized pinned count. Do not substitute zero for an unavailable total or issue an unpinned aggregate and rely on a post-query filter: aggregate SQL cannot use that filter. Ordinary policies preserve the existing count capability. Verification is pending the new focused guard suite. + + +## Retained upload journal integration (verification in progress) + +Public attachment services always use JdbcUploadJournal for retained originals. Ordinary non-retained streaming behavior is unchanged. Apply UploadJournalTable.h2Script() as an owner migration before enabling this path: the immutable physical manifest includes private storage coordinates and lease tokens which deliberately are not QQQ metadata fields. Generating SQL from metadata alone is insufficient. Only shared file-H2 with actual WRITE_DELAY=0 is currently qualified; memory journals are explicitly injected test fixtures only. + +Retained uploads are capped at20MiB and use conditional single-request creation. The journal commits the candidate, fences a lease, and records UNKNOWN before PUT. A conditional collision or uncertain response is accepted only after reading the bounded original and verifying actual length and SHA-256. Metadata insertion and journal REGISTERED checkpoint share one real owner transaction. The mandatory idempotent afterStored lifecycle callback runs after that commit; COMPLETE follows only after it succeeds. A failed callback leaves the same attachment id recoverable. Source archival refuses transfer while the owner checkpoint is incomplete. + +Renewal occurs at synchronous stage boundaries. A storage call that outlives its lease cannot register under its expired fence; a newer recovery may inspect an existing matching original but never overwrite or delete it. No background task synthesizes or carries a browser session. Provider request deadlines and operational supervision remain host acceptance requirements. + +The native Retained Upload Journal table exposes status to its original actor only, subject to current source visibility; after a completed archive transfer, visibility follows the exact attachment's current parent. Native totals are disabled and private-field predicate/order/unique-key/join probes are denied. The Recover retained upload action reauthorizes the original actor and current attach capability, verifies existing bytes and retries registration. It does not accept bytes, a locator, another actor or a coordinator scope. Coordinated intakeKey entries must retry through their original coordinator, preserving origin/version/provenance. The beforeRecovery(candidate) lifecycle hook is default-deny for retained originals and records a recovery read attempt separately from STORAGE_WRITE_ATTEMPT and user downloads. + +Missing bytes or corrupt bytes remain unresolved, never timeout into cancellation or permission to delete possible evidence. A disabled/deleted original actor has no administrative impersonation escape; an explicitly audited trusted administrative reconciliation path remains to be designed. Existing attachment panels also need discoverable pending-upload recovery navigation; the standard table action alone is not a complete panel UX. + + +### Recovery navigation metadata + +Retention owners can call public UploadRecoveryNavigation.configure(sourceTable) after producing source metadata. It adds ATTACHMENTS.recoveryView containing the native owner table, label and actor/status filter. The generic Files panel adds literal canonical tableName/recordId AND criteria and navigates to the standard QQQ table and its recovery action. CurrentUser expressions are accepted only for createdByUserId/EQUALS with one value. Filters aid discovery and never replace original-actor/current-parent authorization. Missing or malformed metadata hides this optional navigation on ordinary installations. An interrupted-upload message does not claim saved/completed bytes; recovery still verifies the owner journal and exact object. diff --git a/docs/ATTACHMENT-PAGINATION.md b/docs/ATTACHMENT-PAGINATION.md new file mode 100644 index 0000000..c865d0f --- /dev/null +++ b/docs/ATTACHMENT-PAGINATION.md @@ -0,0 +1,45 @@ +# Standard attachment list pagination + +The standard `GET /records/{table}/{pk}/attachments` endpoint now applies its +page offset and limit inside the native QQQ attachment query. It reads at most +`limit + 1` rows (maximum 101), projects the requested page, and uses the extra +authorized row only to determine whether to emit the next cursor. It performs no +count query and does not load the complete attachment list before slicing it. + +The route, response schema, default limit 20, allowed limits 1–100, and existing +version-1 scoped offset cursor bytes are unchanged. Sorting is still descending +`createDate`, then descending `id`; equal timestamps therefore retain a stable tie +order. These are offset pages, not snapshots: concurrent insertion or deletion can +shift offsets. High offsets can require backend scanning; bounded returned rows +and host allocation are not a constant-time database performance guarantee. + +Each request still touches the current person, resolves and authorizes the actual +source record, canonicalizes its primary key, excludes deleted-parent attachments, +and uses the existing per-entry edit/delete policy. The same entry projection is +evaluated for the lookahead row. There is no generic post-filter refill or inference +from a denied row. Sources transferred to retained archive records use the exact +current parent identity and its native authorization policy. This owner artifact +does not import Record Files or reconstruct its archive policy. + +Metadata visibility is separate from content release. Listing retained or +quarantined metadata does not issue a download, preview or print capability; those +paths retain their lifecycle and current authorization checks. The cursor is a +routing/offset hint, not a signed authorization token. Current source access is +required even when the caller presents a previously valid cursor. + +`AttachmentsService.listPage` is the bounded public service operation. The older +unpaged `list` method remains for compatibility; new page consumers should use +`listPage`. Cursor parsing/encoding is shared with the existing complete-list +pagination helper, so other Collaboration endpoints keep their previous behavior. + +Verification: local focused run `target/attachment-pagination-focused.log` passed +40 tests on 2026-10-31 at 02:01:44, with zero failures/errors/skips and Checkstyle. +Two new tests execute the actual QQQ RDBMS action against H2 and observe native +offset/limit and returned-row bounds, equal timestamp ordering, deleted/other +source exclusion, retained flags, terminal pages and current revocation before +attachment SQL. The other 38 tests cover existing attachment service behavior and +the unchanged OpenAPI fixture. Root HTTP run47446 also passed all12 tests, including +the new regression covering three pages, +cursor scope, malformed/oversized input and a denied actor. Cross-qbit archive integration, full release verification and installation +of this new owner snapshot remain separate gates. No installed or frozen consumer +artifact has been replaced by this change. diff --git a/docs/ATTACHMENT-RECOVERY.md b/docs/ATTACHMENT-RECOVERY.md new file mode 100644 index 0000000..be5459c --- /dev/null +++ b/docs/ATTACHMENT-RECOVERY.md @@ -0,0 +1,37 @@ +# Attachment cleanup recovery + +Deletion writes a standard `collaboration_attachment_cleanup` row and removes attachment metadata in the same backend transaction. Only after commit does it delete the stored object, then acknowledge the cleanup row. PostgreSQL rejection of metadata deletion leaves both the original row and bytes intact. A provider failure or lost callback leaves the cleanup locator available after restart. The service returns a failure when immediate cleanup fails; the logical attachment deletion may already have committed. + +The cleanup table is visible in the Collaboration app to administrators. Direct table mutations are denied, including to administrators, so callers cannot forge deletion targets. Recovery runs through `collaborationRetryAttachmentCleanup`, guarded for system sessions or `collaboration.admin`. With the host's configured scheduler, it retries a bounded batch of 100 every minute. Kof22 already supplies a scheduler name. Without a scheduler, operators must invoke the recovery process; it is not safe to assume background retries exist. + +Apply the generated PostgreSQL schema before enabling this version; it creates the additional cleanup table and due-time index. Attachment ID is optional because rejected uploads may never have a committed ID. The upgrade preserves existing cleanup rows and removes the earlier attachment-ID uniqueness constraint. Do not remove that table during application rollback while cleanup remains pending. Existing releases do not know how to drain it. + +Verification covers real PostgreSQL metadata rejection, outer upload/delete rollback, commit, provider failure followed by a new recovery service, loss of post-commit callbacks, failed queue acknowledgement and repeat replay. Deterministic storage faults also cover a provider that writes bytes before reporting upload failure and two concurrent cleanup workers. A real MinIO fixture, accessed through the published object-storage QBit, verifies deletion replay in plain and versioned buckets while preserving another object's exact bytes. This is not evidence for every object-storage provider. The separate abrupt-JVM test described below uses fixture file storage. + +Failed upload rollback compensation now records its locator independently of the rolled-back transaction before attempting storage cleanup. Failed worker attempts persist a next-attempt time at least 60 seconds later; this lets healthy requests beyond a failing batch proceed. Tests verify later recovery once the failed requests become due. + +Upload I/O and runtime failures attempt durable compensation and preserve the original exception, adding cleanup failure as a suppressed exception. Deferred provider completion is covered by the settlement protocol below. + +Full Maven verification including operator settlement passes 206 tests, zero failures, errors or skips, plus packaging, help coverage, architecture guardrails and JaCoCo checks. PostgreSQL tests cover the retained receipt, repeated requests, active owners, existing references, rejected receipt writes, provider failure and the real form/Review/Result flow with permission revocation. Coordinated host adoption and staging rollback still require separate verification. + +On backends supporting transactional FOR UPDATE SKIP LOCKED (including the PostgreSQL host), upload now commits a cleanup locator before calling storage and holds its row lock for the upload transaction. Metadata insertion and removal of that locator share the same transaction. Recovery workers skip active uploads and hold the same lock while deleting abandoned bytes and acknowledging cleanup. An existing caller transaction retains ownership until it ends; otherwise the service starts its own transaction after spooling the input. Backends without this locking capability retain callback-based compensation and are not covered by this ownership guarantee. + +Focused PostgreSQL tests verify the locator exists before provider I/O, cleanup skips an active upload, and recovery succeeds after an interrupted upload whose rollback callbacks were deliberately suppressed. That test simulates callback loss. A separate child JVM executes the actual upload service against the disposable PostgreSQL database, persists fixture bytes, then calls Runtime.halt before returning from the provider. The parent verifies exit code 73, absent metadata, retained intent and bytes, and successful idempotent recovery using the original locator. This proves abrupt JVM termination recovery, not power-loss durability or late remote-provider completion. Another test rejects intent insertion in PostgreSQL and verifies zero provider writes or deletes; a completed upload in an open outer transaction remains protected from cleanup until commit. Concurrent workers verify that only one worker enters provider deletion for a locator. + +Issue #1289 remains open for coordinated host adoption and application rollback acceptance. Row locking does not make external object storage and the database one atomic transaction. + +## Unreleased late-upload settlement change + +A PostgreSQL regression reproduced a PUT finishing after timeout rollback had deleted the still-empty location and acknowledged its cleanup row. Upload intents (no attachment ID) now use AttachmentStorage.cleanupUpload, which removes current bytes and reports whether an earlier PUT can still complete. The generic adapter cannot prove settlement and returns false. Recovery retains the locator for later sweeps, with the existing 60-second backoff and 100-row batch. Ordinary committed deletion and successful upload acknowledgement remain unchanged. + +Adapters may report settlement only from authoritative completion/cancellation information, never merely object absence. A deterministic deferred-write fixture verifies the late-completion ordering; this does not establish remote S3 settlement. Generic uncertain-upload records remain pending until the provider proves settlement or an administrator performs the explicit operation below. There is no age-based expiry. + +## Resolve a settled upload + +In the Attachment Cleanup table, select upload intents and run **Resolve settled uploads**. An identified user with `collaboration.admin` must supply provider completion or cancellation evidence covering every selected locator, then confirm the standard Review screen. A support case or request receipt establishing completion is suitable; object absence, an elapsed timeout, or an application process stopping does not establish that an external provider has stopped writing. If settlement cannot be established, leave the intent pending. + +Apply rechecks permission, locks each row, rejects active ownership and existing attachment references, and deletes current bytes before saving completion. Provider or database failures leave the intent unresolved for retry. Committed attachment deletions use the ordinary cleanup retry operation instead. + +The service-managed row retains `resolvedAt`, `resolvedBy` and `settlementEvidence`, together with the original locator. Repeated settlement does not overwrite that receipt; scheduled retries exclude resolved rows. Direct edits and deletes remain denied even to administrators. These fields provide the durable operational receipt without relying on the framework's best-effort audit writer. Store a reference to provider evidence, never credentials or private object contents. + +Apply the additive settlement columns and pending-row index before this code runs. Kof22's corresponding migration is `2026-10-26-collaboration-upload-settlement.sql`. Retain receipts and pending intents during rollback. Do not start an earlier cleanup worker that ignores `resolvedAt`: it could consume retained receipts. Roll back to the pre-cleanup application with cleanup processing disabled, or use a version that understands settlement receipts. diff --git a/docs/CANDIDATE-RELEASE.md b/docs/CANDIDATE-RELEASE.md new file mode 100644 index 0000000..dba8ed7 --- /dev/null +++ b/docs/CANDIDATE-RELEASE.md @@ -0,0 +1,76 @@ +# Coordinated collaboration snapshot + +## Combined integration follow-up under qualification + +PR #51 combines current attachment settlement, My Work, presence recovery, +contributed-field and capacity follow-ups. Its source inventory and gate history +are in [INTEGRATION-20261029.md](INTEGRATION-20261029.md). The candidate selects +verified published Redis `0.1.0-redis-integration-20261029-551f1d5-SNAPSHOT` and +the existing core `4.1.0-virtual-field-values-1196-cc3981bb3-SNAPSHOT`. +The combined collaboration artifact is not yet published. Historical snapshot +and acceptance evidence below is not proof of this candidate's deployment. + +## Selectable-field follow-up snapshot + +The following focused upgrade is published for QQQ #1196: + +| Component | Version | +| --- | --- | +| QQQ BOM and modules | `4.1.0-virtual-field-values-1196-cc3981bb3-SNAPSHOT` | +| qbit-collaboration | `0.1.0-field-values-1196-ecff347-SNAPSHOT` | +| qbit-redis (unchanged prerequisite) | `0.1.0-redis-7fe5492-SNAPSHOT` | + +Import this QQQ BOM in the consuming host. Collaboration publisher 168 passed +from `ecff347efe2bb84949b8452a3383e3fdeb4bcede`; its POM and JAR resolve to +timestamp `20261026.133203-1`. Full collaboration verification against the +published core passed 198 tests with no failures, errors or skips, plus packaging +and coverage gates. An independent empty-cache Java 21 consumer downloaded the +pair and compiled both collaboration value providers against the core API. +See [the upgrade evidence](VIRTUAL-FIELD-UPGRADE.md) for checksums and scope. + +This is not a combined release of the separate attachment, My Work and presence +recovery follow-ups, nor a Kof22 deployment. The earlier candidate's browser +acceptance below must not be treated as browser qualification of this new pair. +A compatible packaged frontend and host staging qualification remain required. + +## Earlier main-line candidate + +The earlier main-line candidate is published. Its coordinated versions are: + +| Component | Version | +| --- | --- | +| QQQ BOM and modules | `4.1.0-collab-33464d37e-SNAPSHOT` | +| qbit-redis | `0.1.0-redis-7fe5492-SNAPSHOT` | +| qbit-collaboration | `0.1.0-collab-8f348da-SNAPSHOT` | +| qbit-session-store | `0.1.0-session-b5276ab-SNAPSHOT` | +| Next frontend source | `8d372142f8af0c41a75a353edc6d150c33ccf613` | + +Maven snapshot repository: https://central.sonatype.com/repository/maven-snapshots/ . Import the candidate QQQ BOM in the host to align transitive QQQ modules. The standalone consumer was verified from an empty cache. This is a feature snapshot, not a stable/GA release or a production deployment. + +Frontend build and full browser acceptance: https://github.com/QRun-IO/qqq-frontend-next/actions/runs/39683690943 . Its acceptance-build artifact contains the tested export. No dashboard container image publication is implied. Install the export through the normal host frontend packaging path; upgrade frontend and backend together because the shared-presence issued-lease protocol changed. + +## Host requirements + +- Configure a named Redis connection through qbit-redis and enable CollaborationQBitConfig.redisPresence. Use explicit deployment/application namespaces and a PresenceScopeProvider derived from authoritative authenticated tenant context. +- Start Redis infrastructure before dependent services and stop dependent services before closing its pools. Session-store may select SHARED_REDIS to reuse the named connection; its session semantics remain owned by session-store. +- Configure TLS/authentication and bounded timeouts/pools for the actual environment. The live acceptance fixture's plaintext loopback Redis and mock authentication are test-only. +- Deliver SSE without proxy buffering. The stream and its subscription updates must reach the owning web node; configure affinity for that connection. Shared Redis does not make node-local SSE connection IDs portable. +- Configure record versions on supported backend paths for stale-save protection. Presence is advisory and never replaces database compare-and-write. Unversioned records do not gain conflict protection merely by enabling presence. +- Use explicit QInstance registration when startup code runs outside QContext. Follow core docs/misc/SessionStoreIsolation.md and session-store docs/SESSION-REGISTRATION.md. JDBC shutdown and health operations must target the owning application; see core docs/lifecycle/application-shutdown.md. +- Apply generated schema/migrations through the host's migration tooling. Do not infer that publishing this library migrated an existing database or an external deployed host. + +## Verification + +All four standard backend publishers and the three QBit source CI runs passed. Tests against published prerequisites: Redis23, collaboration177, session-store96; zero failures/errors/skips and configured coverage gates passed. Core verification4209 and frontend acceptance39683690943 passed. Frontend local3664 tests and TypeScript passed. + +Independent JAR/POM downloads, exact timestamped versions, SHA-256 hashes and an empty-cache standalone consumer passed. Final browser checks used published backend JARs, the exact CI frontend export, two independent JVMs and one Redis endpoint. Cross-node avatars, self exclusion, concurrent editor field focus, preserved drafts, table presence and departure passed. Redis admin pool/subscription metrics, persisted framework UP health history and health-record navigation passed. + +The two-node fixture uses independent memory records and mock users; it proves published-artifact presence/admin integration. Database conflict and authentication behavior have separate core/frontend acceptance evidence. Publication does not prove unknown deployed hosts have migrated. + +Full evidence and issue reconciliation: https://github.com/QRun-IO/qqq/issues/1293#issuecomment-6329618640 . + +## Supported milestone and optional work + +The owner requested the main-line flow on N web nodes sharing one configured Redis endpoint. Browser delivery remains SSE plus HTTP lease writes; Redis Pub/Sub supplies invalidation hints. A new WebSocket protocol is not required. + +Data-loss/restore qualification (#1300), additional failover topologies (#1301), sustained scale/failure injection (#1302), Windows tooling qualifications (#1304/#1305/#1309), CI mirror resilience (#1307), search layout polish (#1310), long-version preflight (#1312) and test-only AssertJ maintenance (#1314) remain tracked separately in the QQQ project. No measured production capacity or disaster-recovery guarantee is claimed by this snapshot. diff --git a/docs/CORE-DEPENDENCY.md b/docs/CORE-DEPENDENCY.md new file mode 100644 index 0000000..6d6d430 --- /dev/null +++ b/docs/CORE-DEPENDENCY.md @@ -0,0 +1,34 @@ +# Core dependency: after-rollback callbacks + +Collaboration's retained-upload and cleanup paths compensate external object storage only after a transaction's rollback is confirmed. They need two methods on `com.kingsrook.qqq.backend.core.actions.QBackendTransaction`. These are included in QQQ Core `4.1.0-RC.22`, which this module imports through the BOM; Core `4.1.0-RC.21` does not declare them. + +## Required API (reviewed implementation: `b6fb2382`, `third-party-src/qqq-core`) + +`qqq-backend-core`, class `QBackendTransaction`: + +```java +public boolean supportsRollbackCallbacks() // default: false +public void addAfterRollbackCallback(Runnable callback) // throws UnsupportedOperationException unless supportsRollbackCallbacks(); Objects.requireNonNull(callback) +protected void runAfterRollbackCallbacks() // run once, in order, then clear; exceptions (incl. LinkageError) logged and swallowed +``` + +Pending rollback callbacks are preserved when a required operation marks the transaction rollback-only, and run only after confirmed JDBC rollback. A failed physical commit clears them because its outcome is uncertain. Successful commit discards them. + +`qqq-backend-module-rdbms`, class `RDBMSTransaction`: + +```java +@Override public boolean supportsRollbackCallbacks() // true +@Override public void addAfterRollbackCallback(Runnable callback) // super.add...; then rolledBackSinceCommit = false +// rollback()/close(): after connection.rollback() succeeds -> rolledBackSinceCommit = true; runAfterRollbackCallbacks() +// close() rolls back (and so runs callbacks) unless a rollback already happened since the last commit / callback registration +``` + +## Callers in this module +- `attachments/AttachmentsService.storeUpload`: refuses an ambient transaction where `!supportsRollbackCallbacks()`; registers storage compensation. +- `attachments/AttachmentCleanupQueue.beginUpload`: registers the same cleanup on both `addAfterRollbackCallback` and `addAfterCommitCallback`. + +## Core tests to carry with the change +- Base class: `supportsRollbackCallbacks()` is false; `addAfterRollbackCallback` throws `UnsupportedOperationException`. +- RDBMS: callback runs after `rollback()`; does not run after `commit()`; runs when a transaction with a registered callback is closed without commit; + a callback registered after an earlier rollback still runs on close; a throwing callback does not stop later callbacks. +- Integration (already in this module, needs the core): `attachments/AttachmentTransactionsPostgresTest` (needs Docker) incl. `interruptedUploadSurvivesLostRollbackCallback`. diff --git a/docs/INTEGRATION-20261029.md b/docs/INTEGRATION-20261029.md new file mode 100644 index 0000000..183fb05 --- /dev/null +++ b/docs/INTEGRATION-20261029.md @@ -0,0 +1,51 @@ +# Coordinated collaboration integration candidate + +This branch combines the current heads of the attachment, My Work, presence +recovery, virtual-field and capacity PRs. It is a qualification candidate, not +a published release or a deployed host. + +| PR | Included source | +| --- | --- | +| #45 attachment settlement | `dc65a826ce362416d55d91136fa39225f4a7cf94` | +| #46 bounded My Work | `cdcf7ee36509c7a1fc6f22fa864209eb80a6ff34` | +| #47 presence recovery | `776ebe3fcb6492881bc9d7ec3f6ca6adfa320200` | +| #48 contributed field values | `1da538cf43ba1ed80e3f8fd599729b5691563cbf` | +| #49 capacity qualification | `56a8b760d9c1a1862cb11e80fad07cf5cdc20b3a` | + +All five heads are ancestors of integration commit +`44a89d77b1ad3a39d26c512ada78af0884872013`. The stacked My Work and capacity +branches did not contain the latest heads of their base PRs, so those base +heads were merged explicitly as well. No production source was rewritten during +integration. The core prerequisite remains +`4.1.0-virtual-field-values-1196-cc3981bb3-SNAPSHOT`. The candidate now selects +published Redis `0.1.0-redis-integration-20261029-551f1d5-SNAPSHOT` after native +publisher job42 succeeded and independent timestamped POM/JAR downloads matched +their declared version and repository checksums. The timestamp is +`20261029.165102-1`; JAR SHA256 is +`385f69c5d84362dbab40a3044261fb9820e25126785d7f22901121302e6ce076`. + +The caller and manifest both select released `kingsrook/qqq-orb@0.8`. The +canonical validator extracted from released 0.8.32 and CircleCI schema validation +pass; no unpublished orb changes are used. + +Local `./mvnw.cmd verify -Dqqq.mywork.load=true` ran 238 tests with no skips, but +reported three failures and four errors. Failures include Redis command read +timeouts/rejoin assertions and PostgreSQL connection I/O during the attachment +fixture. Docker Desktop's published-port path has independently exhibited +300–900 ms replies after idle, compared with 0–58 ms on a direct Docker network. +This observation is not proof of every failure's cause. No timeout, assertion, +security boundary, or quality gate was weakened. Native hosted Linux results +are required before publication; the local run is **not** a passing gate. + +Native CircleCI job192 subsequently passed the full verification at integration +head `02085baff928c374e3b0d77367fbcc8182f81f93`; CodeQL also passed. Those checks +used the earlier Redis pin. The new published Redis pin above requires its own +current-head native verification before this collaboration artifact is published. +Use an explicit short feature branch (`feature/collab-int-20261029`) and dry-run +the released version calculator: generated help-bundle versions must remain at +most 50 characters. The longer `collaboration-integration-20261029` name would +produce 57 characters and is not a valid publication route. + +The successful earlier combined-source checks remain historical evidence only. +Host TLS/ACL, actual identity/proxy/provider acceptance, maintenance/restore +enforcement, artifact publication, and deployment are separate gates. diff --git a/docs/MY-WORK-QUERY-BOUNDS.md b/docs/MY-WORK-QUERY-BOUNDS.md new file mode 100644 index 0000000..0c27f99 --- /dev/null +++ b/docs/MY-WORK-QUERY-BOUNDS.md @@ -0,0 +1,92 @@ +# My Work summary query bounds + +The summary previously materialized every matching task, assignment and mention. +A regression fixture measured 601 parent lookups for 601 tasks on two parents. + +The summary now scans each category in primary-key pages of at most 200 rows. +It retains at most ten display tasks and sorts them using the existing due-date +order (earliest first, null last, then ID). Counts still include every visible +matching row, rather than silently truncating at a display limit. Assignment and +mention filters remain scoped to the signed-in user. + +Each page caches parent visibility only for that page, using the existing caller +permission and record-lock checks. Missing or revoked parents contribute neither +rows nor counts; unexpected authorization/backend failures propagate. No decision +is shared across requests. The starting maximum ID bounds each category scan so +concurrent inserts cannot extend it indefinitely. Like the earlier summary, this +is not a transactionally consistent snapshot across the three categories. + +This bounds row retention, not total work independently of data size. For N +matching rows, each category needs one high-watermark query plus approximately +ceil(N/200) page queries, and one parent lookup per distinct parent per page. +Many unique parents still require linear authorization work. Per-record heavy +field size is also outside this row-count bound. Database and browser latency, +N-node presence, saturation and production capacity remain part of #1291/#1302. + +Regression evidence: the 601-task case passes a budget of at most eight parent +queries, with no task query above 200 rows; correct top-ten ordering and revoked +access on a subsequent request are checked. Another case crosses assignment and +mention page boundaries and excludes hidden parents, other users and undelivered +mentions. These tests use the framework's real memory backend instrumentation. + +`MyWorkPostgresTest` repeats the 601-row case using PostgreSQL 17 and the generated +production schema. Independent `pg_stat_statements` observations verify exactly +five task queries and between one and eight parent queries. It also verifies the +same top ten, visible count and next-request permission revocation against SQL. +The fixture passes; these query counts are not sustained throughput or +deployment-capacity measurements. + +## Reproducible local workload + +Run `mvn -Dtest=MyWorkPostgresTest -Dqqq.mywork.load=true test` with Java21 and +Docker. The fixture owns a disposable PostgreSQL17 container, uses the generated +schema and real authorization checks, and writes `target/mywork-load.csv`. +It never connects to an application database. The optional workload runs after +the normal query-budget regression so ordinary CI keeps the smaller test. + +Each scenario has 2,000 tasks: one uses ten shared parents, the other 2,000 distinct +parents. Every tenth parent is inaccessible. Two warmups precede ten measured +sequential summaries. Every sample verifies 1,800 visible tasks and the exact top +ten. SQL calls and server execution time are collected independently using +pg_stat_statements after warmup. Latency includes service execution and database +round trips, excluding fixture setup and assertions. + +CSV percentiles use nearest rank. With only ten samples, p95 and p99 both equal +the maximum observed sample; they are descriptive, not stable tail estimates. +SQL figures are totals for all ten samples. This is a single-JVM local workload, +not HTTP/browser latency, sustained concurrency, pool saturation, multi-node +presence or a deployment-capacity claim. Do not derive production alert thresholds +from these numbers. No wall-clock performance assertion is added to CI. + +The fixture explicitly uses the framework C3P0 connection provider, matching the +host's provider, with initial/minimum1, maximum3 and a two-second checkout timeout. +It verifies the pool stays within that cap, has no checkout timeouts and returns +all borrowed connections after each scenario. The original fixture used the +unpooled default; the expanded workload failed with a Windows socket-allocation +error after repeatedly opening connections. This was a fixture mismatch, not +proof of a Kof22 outage. The same workload completed with pooling. + +Observed local run (Java21, PostgreSQL17 in Docker on Windows, ten sequential +samples after two warmups): shared-parent p50=91.192ms, max=112.105ms; distinct-parent +p50=960.565ms, max=1100.260ms. SQL calls per summary were113 and2013 respectively. +This confirms linear distinct-parent authorization work remains despite bounded +row retention. The existing exact-count semantics still require examining every +matching task. No runtime optimization or production-capacity claim follows from +this fixture measurement alone. + +## Compatibility with the later attachment changes + +On October 26, an isolated source export combined attachment commit +`dc65a826ce362416d55d91136fa39225f4a7cf94` with the four My Work files from +`2ac067bea4af199486f90e2dc5212baf8910b8fb`. Both branches share base +`15cb8cbf72aa9a5e033280cf5a38384d1295fc7d`; their changed files do not overlap. +This was a source-overlay verification, not a branch merge or publication. + +Java 21 `./mvnw.cmd verify -Dqqq.mywork.load=true` passed all 209 tests with zero +failures, errors or skips, including the 29 attachment transaction tests. +Packaging and JaCoCo gates passed. The optional PostgreSQL workload retained the +same visible-count, top-ten, query-budget and pool assertions. Shared-parent +p50/max were 76.301/96.044 ms; distinct-parent p50/max were 939.896/1035.402 ms. +SQL counts remained 113 and 2013 per summary. These local measurements preserve +the limitations above; they do not qualify an integrated published artifact or +Kof22 deployment. The eventual integrated branch still needs its own CI gates. diff --git a/docs/PRESENCE-RECOVERY.md b/docs/PRESENCE-RECOVERY.md new file mode 100644 index 0000000..43035ae --- /dev/null +++ b/docs/PRESENCE-RECOVERY.md @@ -0,0 +1,179 @@ +# Presence recovery candidate — QQQ #1300 + +This is an unreleased implementation candidate in collaboration draft PR47. +The existing store/service suites and CI pass; host adoption and enforced +maintenance remain incomplete. It is not part of the published snapshots. + +## Contract under qualification + +Each deployment/application/tenant/table has a persistent recovery guard in the +same Redis hash slot as its bounded room projections. Each Lua operation reads +that guard atomically with its room work. The guard contains a generation, a +readiness deadline and the latest issued expiry. A missing guard, or complete +loss of table projections before that expiry, starts a fresh generation and a +recovery interval of one lease lifetime. Existing live browsers can reserve and +renew replacement leases during that interval. Snapshots report recovery rather +than a successful empty room until the interval finishes. + +Lease entries carry the guard generation. Recovery removes invalid-generation +projections from touched rooms so they cannot consume all capacity and prevent +re-registration. Existing authorization remains in PresenceService; the guard +does not grant record, tenant or field access. The shared adapter maps recovery +to a sanitized service-unavailable response. Record-version conflict checks are +independent of this advisory state. + +An untouched new namespace is conservatively unknown too. Its first snapshot +therefore starts a recovery interval. Normal idle expiry retains the guard so +later empty snapshots can be known-empty. The guard has no TTL; tenant-retirement +operations and long-term guard storage must be included in operational capacity +qualification. + +## Controlled restore + +The host configuration has a bounded `recoveryGeneration`, default `initial`, +included in both the storage namespace and realtime audience. It belongs in +deployment configuration outside the Redis backup. All nodes must use the same +value. A restored guard can otherwise restore old authority; Redis process IDs +cannot detect arbitrary same-process data restoration. + +The required maintenance procedure is: stop admission and drain all old web +nodes; restore Redis; choose a never-reused generation outside Redis; start only +nodes with that generation; verify recovery and authenticated re-registration; +then reopen traffic. Do not roll back the generation when rolling back an app. +The host tooling that enforces and verifies this sequence is **not implemented** +yet. The branch does not claim transparent arbitrary restore or split-brain +recovery. Native Sentinel/Cluster routing remains the separate Redis issue #1301. + +The storage prefix is v2 so old executables cannot treat generation-bound entries +as old-format authority. This requires a coordinated deployment, not a mixed +version rolling deployment. Existing published v1 snapshots are unchanged. + +## Evidence and remaining work + +- Before the patch, a two-client disposable Redis test failed because total loss + returned an empty snapshot. +- Seven focused tests now pass: full loss, loss of volatile projections while the + guard survives, restored guard/lease bytes followed by a host generation + change, bounded generation validation, and rejection of inconsistent lease + lifetimes across nodes, guard loss across previously unvisited records, and + shared cold readiness followed by normal idle expiry. The latter verifies + that peer reads do not extend the deadline, the guard remains persistent, + expired projections remain known-empty and expired leases cannot renew. + These use independent clients in + one JVM, not a deployed HA cluster. +- Full wrapper verification passed on October 26: 184 tests, no failures, + errors or skips; packaging and coverage gates passed. This includes the + original seven store tests and two-node six-viewer/two-editor scenario. + The admin dashboard reads every table, so the fixture initializes every + dashboard table for both tenants through the real backend and waits for + recovery without writing Redis guard state. +- The service fixture socket read timeout is one second; its previous 200 ms + limit caused a Redis EVALSHA read timeout. Production timeouts are unchanged. +- The full run includes a real HTTP recovery test: real Redis + FLUSHDB produces 503 during recovery, the former lease returns 409 with + PRESENCE_LEASE_EXPIRED, hidden records and unauthorized viewers remain + inaccessible, and a new lease rejoins through the production recovery window. + This uses a loopback HTTP server with fixture authentication, not a browser + or a deployed host. The focused test and the combined full suite both pass. +- The store suite checks loss of either state or expiry data in both record + and table projections. Server-issued lease renewal must fail unavailable + without changing the surviving projection. It also covers complete loss + of either projection pair and stale departure ordering. +- Still required: deployed-host qualification and enforced host maintenance/reset. Local + per-operation cost measurements at 1,000 and 10,000 leases are recorded in + `acceptance/RECOVERY-COST.md`; sustained production capacity is unqualified. + The implementation uses no INFO call or lifecycle polling cache. +- This is an unreleased draft candidate. Publication and production deployment + have not occurred; the remaining acceptance work above is still required. + +## Independent-process acceptance + +`docs/acceptance/presence_recovery_processes.py` passed on October 26 using +two independent `SharedPresenceDevServer` JVMs and one disposable Redis container. +Each user renews through the opposite node. After FLUSHDB, both nodes return 503 +for presence and 409 for the former leases; fresh leases restore both viewers on +both nodes. The harness terminates only its owned processes and container. +This is fixture authentication with independently seeded memory records, not a +deployed HA application or a browser acceptance result. + +Reproduce from the repository root after `./mvnw test-compile`: + +```text +./mvnw dependency:build-classpath -Dmdep.includeScope=test -Dmdep.outputFile=target/recovery-process-classpath.txt +python docs/acceptance/presence_recovery_processes.py --java /path/to/java --repo /path/to/repository +``` + +Requires Python 3, Java 21 and Docker. JVM output and argument files stay under +`target/recovery-processes-*` for diagnosis. The harness uses the production +45-second recovery window, with bounded startup and recovery deadlines. + +## Browser recovery acceptance + +The same harness accepts `--frontend-repo /path/to/qqq-frontend-next`. It copies +that checkout's existing `out` export into the owned test output directory and +serves it through the real middleware on both JVMs. Playwright is loaded from +the frontend checkout; no API mocks are installed. Both browser users first +see the other viewer, then Redis FLUSHDB triggers the unavailable message and +removes stale people. Both viewer badges return without reloading either page. + +The final fixed-candidate run passed all five profiles on October 26 against +commit `4b0d6a32d6bd8ecba023857d3661d9f3f5980975`: Chromium, Firefox, WebKit, +Pixel 7 emulation and iPad emulation. Source/compiled Lua hashes were checked +before the run and after each profile. Traces and screenshots remain under +each `target/recovery-processes-*` directory. The earlier Chromium screenshot +in `target/recovery-processes-229f305e83` was also visually inspected. + +Select profiles using `--browser-profile chromium|firefox|webkit|mobile|tablet`. +These tests use the local export and do not establish provenance of a newly +published frontend artifact, physical-device behavior or deployed Kof22 +acceptance. Fixture frontend serving is opt-in via `collab.dev.frontend`; +its default remains false. + +## Controlled restore acceptance + +`acceptance/presence_restore_processes.py` exercises a disposable Redis RDB +restore with two independent JVMs. It saves the populated dataset, verifies +both old JVMs have terminated and their listeners have closed, clears the +dataset, reloads the saved image and starts both nodes with a shared fresh +generation supplied outside Redis. It checks recovery responses, rejects old +leases on both nodes, and requires two fresh viewers to rejoin across nodes. +The final run passed on October 26; logs are retained in +`target/restore-processes-aec5288289`. Both leases are explicitly renewed before +saving, and rejection must complete within 40 seconds of that renewal, before +their normal 45-second expiry. Old leases are rejected again after fresh +viewers have rejoined. + +The fixture enables Redis DEBUG RELOAD only on its owned loopback container. +Run after test compilation and classpath preparation: + +```text +python docs/acceptance/presence_restore_processes.py --java /path/to/java --repo /path/to/repository +``` + +This is a protocol acceptance fixture, not host maintenance automation. Actual +host admission closure, replacement prevention and generation configuration +still require implementation and staging validation. + +To qualify a packaged backend, add `--backend-jar /path/to/qbit-collaboration.jar`. +This excludes production `target/classes` from the JVM classpath, rejects a +duplicate collaboration dependency JAR, and records the artifact SHA-256 in +`target/restore-processes-*/result.json`. The receipt includes the elapsed time +for rejection of old leases (required to remain below their normal expiry), +the restore/rejoin checks and fixture cleanup. Cleanup failure fails the run. +The test repository still supplies compiled fixture classes and its dependency +classpath; this is not a standalone production launcher. + +## Packaged browser qualification + +The process harness accepts `--backend-jar` and `--frontend-jar` for artifact +verification. With a backend JAR, production `target/classes` is omitted from the +runtime classpath; test classes provide only the disposable server fixture. The +frontend JAR is served directly instead of copying a local `out` export. +`--frontend-repo` remains required for Playwright tooling. Dependency classpaths +must not contain a second collaboration or frontend artifact. + +Each run writes `result.json` with the selected artifact SHA-256 hashes, browser +profile, result and cleanup outcome. A successful test with failed fixture cleanup +exits unsuccessfully. This records local artifact qualification, not publication +or deployment provenance. Supply `--browser-profile` to qualify each required +browser independently. diff --git a/docs/QBIT-KNOWN-ISSUES.md b/docs/QBIT-KNOWN-ISSUES.md new file mode 100644 index 0000000..c75bae1 --- /dev/null +++ b/docs/QBIT-KNOWN-ISSUES.md @@ -0,0 +1,6 @@ +# Known issues + +- Deployment-specific object storage, database failover, backup/restore and retention policies need host qualification. +- Docker must be running for the attachment transaction tests; enable both `mysqlJournalTests` and `postgresJournalTests` for the additional journal-provider cases. On Windows use `-Dexec.skip=true` for the inherited POSIX console-summary command. +- Redis presence needs `qbit-redis` on the classpath; it is an optional dependency. +- The matching Next UI and host configuration must be verified before rollout. diff --git a/docs/QBIT-RELEASE-STATUS.md b/docs/QBIT-RELEASE-STATUS.md new file mode 100644 index 0000000..2fb4baa --- /dev/null +++ b/docs/QBIT-RELEASE-STATUS.md @@ -0,0 +1,13 @@ +# Collaboration — status + +**Version 1.0**, built and published by the shared `ci-workflows` Maven pipeline to the KOF22 Gitea Maven registry (`com.kingsrook.qbits`); `mvn deploy` runs the tests once. + +## What it contains + +Attachment lifecycle with the retained upload journal, recovery and cleanup APIs; presence including Redis-backed presence; tags, custom fields, comments with mentions, sharing, favorites, follow, timeline and tasks; virtual-field and fixture support. + +## Dependencies + +QQQ Core `4.1.0-RC.22` (imported through its BOM; it provides the rollback callbacks the upload paths need). `qbit-redis` is optional and only needed for Redis presence. Build with Java 21 and `./mvnw verify`; Docker must be running for the attachment transaction tests (set `mysqlJournalTests` and `postgresJournalTests` for the extra journal-provider cases). + +Host qualification of object storage, database failover, backup/restore and retention policy remains the deploying team's responsibility. See [known issues](QBIT-KNOWN-ISSUES.md). diff --git a/docs/REDIS-PRESENCE.md b/docs/REDIS-PRESENCE.md new file mode 100644 index 0000000..d10b7ea --- /dev/null +++ b/docs/REDIS-PRESENCE.md @@ -0,0 +1,47 @@ +# Shared Redis presence candidate + +Status: coordinated candidate artifacts are published; remote CI, empty-cache consumer resolution and published-artifact two-node browser checks passed. See [CANDIDATE-RELEASE.md](CANDIDATE-RELEASE.md) for exact versions and deployment requirements. This guide describes the current main-line scope. See MAINLINE-ACCEPTANCE.md for evidence and SESSION-STATE.md for live release checkpoints. + +## Host configuration and dependency ownership + +Multiple QQQ web nodes share one configured Redis endpoint. The host enables CollaborationQBitConfig.redisPresence, supplies a started named Redis connection, explicit deployment/application names, and a PresenceScopeProvider deriving one tenant scope from authenticated host context. Browser input cannot choose the tenant. + +Collaboration owns presence semantics, Lua scripts, record/table projections and authorization. qbit-redis owns connections, namespaces, script execution, pool lifecycle, admin management and framework health integration. Collaboration production code consumes only the Redis public API and does not import Jedis. Without shared configuration, the local presence backend remains available for a single node; its issued authority is not distributed. Runtime Redis failure never silently switches an HA host to local presence. + +Browser delivery uses the existing SSE stream with HTTP lease writes. Redis Pub/Sub carries scoped invalidation hints between nodes; a new WebSocket transport is not required. Hints contain table/record identifiers, not record contents or viewer lists. The recipient rereads through the authorized presence API. Core checks the authenticated recipient audience during live delivery and replay. Coordinate core/frontend/QBit versions and do not share a topic with audience-unaware hubs. + +## Issued lease lifecycle + +The candidate collaboration extension version is 1.3.0: + +- POST /presence/leases authorizes the table/record and creates an invisible, expiring reservation, returning a server-issued connectionId. +- POST /presence/leases/renew requires that issued handle and a positive safe-integer sequence. It activates or updates existing live authority; it cannot recreate an expired or departed lease. +- DELETE /presence performs ordered departure. Delayed renewals cannot reopen the departed issued handle. +- HTTP 409 with code PRESENCE_LEASE_EXPIRED tells an active browser to open a fresh lease. Input, permission and infrastructure failures retain their standard mappings. Unavailable shared storage returns 503 rather than an empty-room result. +- Shared presence rejects legacy heartbeat upserts. Local legacy behavior remains for compatibility. + +The Next candidate awaits open before activation, uses the handle across focus changes, and releases it on hide/unmount. It opens again on pageshow or explicit expiry. Generation checks discard old asynchronous responses; a late open result is released without activation. One immediate rejoin is allowed after expiry, then retries follow the normal 20-second heartbeat. Infrastructure errors do not churn handles. The UI preserves drafts and shows unavailable status when presence cannot be read or renewed. + +## State and permissions + +Record and table rooms use bounded expiry indexes and state hashes in the same Redis cluster hash slot for their deployment/application/tenant/feature namespace. Lua updates both projections atomically using Redis TIME. User/connection/record identifiers are encoded independently. No record values are stored. + +Snapshots merge tabs by user. EDITING takes precedence over VIEWING; focus follows the latest renewal within the selected mode. Table projections omit focused fields. Expired entries are pruned on reads/mutations and idle keys expire. Invisible reservations count toward capacity. Scripts use EVALSHA with fallback only for NOSCRIPT, through the Redis QBit API. + +Reads and renewals recheck parent and field access. Canonical record identifiers and recipient field filtering are enforced. Admin counts are limited to known rooms in the administrator's authenticated tenant; non-admins receive no counts. Capacity and TTL limits bound individual operations; they are not measured sustained production throughput guarantees. + +## Verified behavior + +- Real Redis tests cover six users on a record with two editors, duplicate-tab aggregation, ordering, departure, expiry, isolation, capacity and script-cache recovery. +- Service and real HTTP tests cover issued open/activation/renew/departure, 409 expiry, authorization, canonical keys, field projection and unavailable responses. +- Clean release-source collaboration verify passed 177 tests with coverage/architecture gates; focused frontend suites passed 34 tests, TypeScript and changed-file lint. +- Live browser acceptance with mocks disabled used three identities on two independent QQQ JVMs sharing Redis. It verified avatars, field focus, drafts, table projection and departure. Demo records were independently seeded memory data; this is presence evidence, not cross-node database save-consistency evidence. +- Live Redis admin acceptance verified pool/subscription status, persisted framework health results and navigation to standard health records. + +Presence is advisory. Atomic optimistic record-version checks protect saves independently. Coordinated cold-consumer acceptance against published artifacts remains required; local Maven cache success does not prove remote artifact availability. + +## Deferred work + +The user narrowed this release to the normal flow. Optional qualification is tracked in QQQ issues #1300 (data loss/backup restore), #1301 (failover/disaster topologies), and #1302 (sustained scale/failure injection), all on the QQQ Roadmap backlog. + +Issued leases do not provide disaster-recovery fencing: complete state loss can temporarily look empty, and restoring an old live lease can restore its authority. No recovery generation guard is implemented in this candidate. The deferred investigation is tracked in [QQQ #1300](https://github.com/QRun-IO/qqq/issues/1300). diff --git a/docs/RETAINED-UPLOAD-JOURNAL.md b/docs/RETAINED-UPLOAD-JOURNAL.md new file mode 100644 index 0000000..541eb87 --- /dev/null +++ b/docs/RETAINED-UPLOAD-JOURNAL.md @@ -0,0 +1,48 @@ +# Retained upload journal + +The Collaboration-owned journal is wired into retained `AttachmentsService` upload and recovery. It persists candidate metadata before object creation, owns the attachment/checkpoint transaction, and exposes native read-only recovery navigation. It has no Record Files dependency and stores no second copy of file bytes. + +The integrated owner snapshot passed all **214 tests**, including 10 real MySQL provider cases and 11 H2 journal regressions, and was installed under `0.1.0-record-files-SNAPSHOT`. The subsequent doublewrite/session-read-only delta compiles and awaits its provider rerun. This document does not qualify Aurora, other SQL vendors, external object storage, disks, backups, or a host deployment. + +## Ownership and recovery + +1. `prepare` independently commits the immutable server-built candidate before the first PUT. Matching retries reuse it; changed manifests conflict. +2. `reserve` locks the row and grants a bounded lease with a monotonically increasing fence. `renew` extends only a still-live exact reservation at synchronous stage boundaries. +3. `recordStorage(UNKNOWN)` precedes conditional object creation. `VERIFIED` means the storage owner proved the exact length and SHA-256. +4. `register` passes its real transaction to the attachment registrar. The attachment insert and REGISTERED checkpoint commit together; callback failure rolls both back. +5. The retained lifecycle registration runs after commit and is retried idempotently. The journal becomes COMPLETE only after it succeeds. Completed coordinated retries still invoke required lifecycle registration without adding a second attachment or activity event. +6. Recovery reauthorizes the original actor and current source. It verifies the stored original without overwriting it and resumes registration. Missing or unreadable bytes remain uncertain; timeout never proves absence. + +The owner rejects ambient/no-op transactions. SQL values are bound; mutable operations lock rows, check the token/fence/expiry, and require exactly one guarded update. Reads reconstruct manifest hashes. Private locator and lease columns are absent from QQQ metadata and forbidden in native filters/projections; native direct writes and unscoped totals are denied. Current actor/source authorization remains in the owner service and native journal security, not in the low-level JDBC port. + +Disabled or deleted original actors still need an explicitly designed administrative reconciliation flow. No fabricated session or broad archive permission substitutes for that flow. Broad post-filtered listings can contain short pages; source-pinned native recovery navigation is the supported user entry point. + +## Database boundaries + +Both owner tables must use the same QQQ backend. Hosts apply DDL through their migrations; runtime code never creates tables or changes server-global settings. + +| Backend | Runtime requirements | Schema | +| --- | --- | --- | +| File H2 | Native `h2`, `jdbc:h2:file:`, actual configured connection, writable real transaction, live `WRITE_DELAY=0` | `UploadJournalTable.h2Script()` | +| MySQL 8.4 | Native `mysql`, actual MySQL 8.4 product/version, exact configured URL target and catalog, writable real transaction, both owner tables InnoDB, fixed owner table mappings, journal `utf8mb4_0900_bin` columns/table, single-column unique locator/intake indexes, synchronous commit settings | `UploadJournalTable.mysqlScript()` | +| Aurora MySQL, MariaDB, PostgreSQL, other versions/providers | Refused; no inferred compatibility or durability claim | None | + +The MySQL adapter resolves explicit or native-generated URLs through QQQ `ConnectionManager`. It accepts a single ordinary `jdbc:mysql://host:port/catalog` target with a simple catalog identifier. Proxy/load-balancing URLs, alternate catalogs or table remapping fail closed. Error messages do not echo URLs or credentials. Driver options remain host-owned; TLS and credentials are not changed by the journal. + +Each MySQL operation checks `innodb_flush_log_at_trx_commit=1`, and `sync_binlog=1` when binary logging is enabled; a read-only server or session is refused. It also requires `innodb_doublewrite` to be `ON` or `DETECT_AND_RECOVER`; [detect-only mode does not retain recoverable page copies](https://dev.mysql.com/doc/refman/8.4/en/innodb-doublewrite-buffer.html). These settings follow the [MySQL 8.4 InnoDB durability documentation](https://dev.mysql.com/doc/refman/8.4/en/innodb-parameters.html#sysvar_innodb_flush_log_at_trx_commit) and [binary-log durability documentation](https://dev.mysql.com/doc/refman/8.4/en/replication-options-binary-log.html#sysvar_sync_binlog). Hardware that misreports flush completion can still lose data; replication/failover, backups, restore and disk guarantees remain deployment qualifications. Runtime settings are checked at transaction entry; an administrator changing server settings concurrently is outside the application trust boundary. + +The MySQL journal uses millisecond timestamps with explicit UTC JDBC calendars, InnoDB, enforced CHECK constraints and binary NO PAD string comparisons. Its live session must report UTC (+00:00/UTC, or SYSTEM with a UTC system zone); the journal does not silently change a pooled session timezone. Hosts should configure the native JDBC driver timezone consistently for generic QQQ date fields as well. The host must configure each journal connection with `innodb_lock_wait_timeout` between 1 and 5 seconds, through its backend JDBC sessionVariables/pool setup (or a dedicated database default). Qualification reads and preserves that session setting; it never changes it or other pooled-session/global settings. Each statement also has a ten-second query timeout. Duplicate error 1062 is distinguished from unrelated integrity failures; deadlock/lock-timeout failures remain typed BUSY responses, never fabricated success. + +## Reproducible tests + +Use the repository Maven wrapper and the same revision/QQQ dependency overrides as the integrated build. The default suite remains Docker-independent; explicitly enable the provider fixture: + +```powershell +.\mvnw.cmd '-Drevision=0.1.0-record-files-SNAPSHOT' '-DmysqlJournalTests=true' '-Dtest=MySqlUploadJournalTest,UploadJournalTest' test +``` + +`MySqlUploadJournalTest` creates only its own Testcontainers MySQL 8.4.13 container, random synthetic credentials and private anonymous data volume, and destroys that container on completion. It does not use existing host services. The image is pinned to `mysql@sha256:a0225d1ea6631c8bc2385dc4c2faba17bbf1834ecf369e1659a9a382bd8d5dd7`. Credentials are not printed by the fixture. + +The eleven provider cases cover acknowledged prepare surviving a real database process restart, attachment/checkpoint rollback and completion replay, simultaneous prepare/lease fencing, wrong backend/catalog/mapping/ambient/Aurora refusal, actual non-durable server settings and MyISAM refusal, binary identifier comparisons and native write denial, actual lock-timeout/BUSY behavior, column/index drift and immutable conflict, and lease instants across differing JVM default zones, preservation/refusal of host session lock-timeout settings, and refusal of detect-only doublewrite/read-only sessions without changing their settings. The attachment schema in this suite is deliberately a minimal transaction fixture; this is not an end-to-end MySQL object-storage or tenant/archive qualification. Record Files owns its separate provider integration tests. + +First provider run compiled but failed its restart fixture because Docker reassigned the dynamic port and Testcontainers retained a cached mapping; two subsequent setup errors followed that stale endpoint. The fixture now refreshes the inspected binding after restart and uses bounded JDBC readiness checks. Docker wire logs are suppressed before container creation. Expanded snapshot verification: **20 tests passed (9 MySQL, 11 H2), zero failures/errors/skips**, `target/mysql-journal-focused-verification.log`, completed 2026-10-31 00:01:44. The tenth provider case and full 214-test owner quality gates subsequently passed; the corrected development-GAV artifact SHA-256 is `72D6E36E77B92BE3ECFE9BE7042D292C76B308AA05A9E43BD490B229BD1A75E2`. The eleventh provider regression and doublewrite/read-only delta await fresh verification. diff --git a/docs/VIRTUAL-FIELD-UPGRADE.md b/docs/VIRTUAL-FIELD-UPGRADE.md new file mode 100644 index 0000000..fda061e --- /dev/null +++ b/docs/VIRTUAL-FIELD-UPGRADE.md @@ -0,0 +1,76 @@ +# Selectable collaboration fields: published feature snapshot + +Tracks [QQQ #1196](https://github.com/QRun-IO/qqq/issues/1196). + +This source branch requires the new core virtual-field value-provider API. Its +default BOM pin is now the published +`4.1.0-virtual-field-values-1196-cc3981bb3-SNAPSHOT`. Core publisher job 9716 +passed from commit `cc3981bb3c95e8f2dd7ef2a02e2233db7d0e2a79`; an independent +consumer with an initially empty Maven repository downloaded its BOM and core +and compiled the provider interface on October 26, 2026. Both artifacts resolve +to timestamp `20261026.130631-1` from Sonatype's snapshot repository. + +Published artifact SHA-256 receipts: + +| Artifact | SHA-256 | +| --- | --- | +| BOM POM | `d68cf837d7e07513c4e70977829f99e6236cb3e89f8af85fc75239d5b52564a5` | +| Core POM | `d89d491aaa9312051ec462e1cf165b948ee3e88c9aec2721ff25a00e956c431b` | +| Core JAR | `ae6dbeba8d0e4bc55f2714303ee3482dbf6a0cd34c03663c6009d8fe2c1da54d` | + +The collaboration follow-up is published as +`0.1.0-field-values-1196-ecff347-SNAPSHOT`, timestamp +`20261026.133203-1`. Publisher job 168 passed from commit +`ecff347efe2bb84949b8452a3383e3fdeb4bcede`, using signed tag +`publish-values1196-20261026-ecff347`. + +| Collaboration artifact | SHA-256 | +| --- | --- | +| POM | `cc12f59cbdfd4bc53ba545a9f2b6d7c14a316034b107ef5541ddc48df45fe6af` | +| JAR | `284fa370504dd82cee8c87d2b8f5c0cecba8c4b15b32a4f9d3d62a0a062e549d` | + +On October 26, an independent Java 21 consumer downloaded this pair into an +initially empty Maven repository and compiled both `TagValueProvider` and +`CustomFieldValueProvider` as `VirtualFieldValueProviderInterface` instances. +Maven recorded Sonatype `central-portal-snapshots` as the artifact source. The +downloaded collaboration POM pins the core version above; the independently +downloaded core JAR matches the receipt above. This checks published dependency +resolution and API compatibility; runtime behavior is covered by the 198-test +verification below, not by this compilation probe. + +This snapshot does not include the separate attachment, My Work or presence +recovery follow-up PRs. It has not been deployed to Kof22. + +Earlier local verification used core artifacts installed as +`4.1.0-values1196-SNAPSHOT`, with +`./mvnw.cmd -Dqqq.snapshot.version=4.1.0-values1196-SNAPSHOT verify`. +That version exists only in the local Maven repository; it is not an adoption pin. + +## Behavior + +- `collaborationTags` returns sorted, distinct approved labels from global or + matching-table vocabulary. Merged tags and deleted assignments are excluded. + The picker and filters retain their tag IDs; exports contain one Tags column. +- EAV custom fields return typed existing values and ordinary possible-value + display labels. Missing values are null internally and omitted from HTTP value + maps. This change does not enable EAV writes. +- Providers process batches and use the originating transaction. Parent record + authorization and field visibility remain enforced by core. Existing host + post-query customizers remain installed. +- SQL DISTINCT on provider values is explicitly unsupported. Backend primary-key + identity is required; joined heavy primary keys must be fetched explicitly. + +## Release sequence + +1. Publish and verify the coordinated core artifact containing the provider API (done). +2. Replace this branch's core BOM pin with that published version (done). +3. Run full collaboration verification against the published prerequisite + (done October 26: 198 tests, no failures/errors/skips; packaging and coverage + passed, including HTTP, PostgreSQL transaction and Redis tests). +4. Publish the collaboration snapshot and verify an independent consumer using + published artifacts (done; exact pair recorded in CANDIDATE-RELEASE.md). +5. Update Kof22 Website-Backend adoption work and qualify it in the selected + isolated staging environment before requesting production deployment. + +The published pair contains this implementation. Kof22 adoption, a compatible +packaged frontend and staging qualification remain separate release steps. diff --git a/docs/acceptance/RECOVERY-COST.md b/docs/acceptance/RECOVERY-COST.md new file mode 100644 index 0000000..74121a0 --- /dev/null +++ b/docs/acceptance/RECOVERY-COST.md @@ -0,0 +1,56 @@ +# Recovery cost diagnostic — October 26, 2026 + +One Java 21 process, one Redis connection, disposable loopback Redis 7.4-alpine, +1,000 issued leases, ten viewers per record, no TLS or HTTP. The probe keeps +all leases alive through the real 45-second recovery window and verifies that +the final table snapshot contains all 1,000 viewers. Each phase samples 200 +renewals with increasing sequence numbers. This is not a capacity or SLA claim. + +| Renewal latency (ms) | During recovery | Ready | +|---|---:|---:| +| p50 | 3.6779 | 0.5134 | +| p95 | 4.1956 | 0.7226 | +| p99 | 4.5447 | 0.9122 | +| Maximum | 6.8271 | 1.4886 | + +Population took 4,640.799 ms. The run passed all correctness checks. This first +implementation scanned touched projections on every request during recovery. + +After changing table cleanup to occur atomically only when creating its recovery +generation (individual record cleanup remains lazy), the same diagnostic passed: + +| Renewal latency (ms) | Optimized recovery | Ready | +|---|---:|---:| +| p50 | 0.6375 | 0.5029 | +| p95 | 0.8150 | 0.5762 | +| p99 | 0.9406 | 0.7500 | +| Maximum | 4.2280 | 1.2251 | + +Optimized population took 1,634.486 ms. All 1,000 leases survived. The focused +recovery/store suite also passed 13 tests, including guard loss with previously +unvisited record rooms. Full optimized verification passed 184 tests with no failures/errors/skips and coverage gates passed. These are +individual local runs and need further qualification before claiming supported +production capacity. They do not isolate every source +of latency, compare against an earlier release, or measure sustained concurrent +load, saturation, TLS, HTTP or browser propagation. + +At 10,000 entries the optimized diagnostic also passed, retaining all leases. +Population took 10,687.2105 ms; the same 200-sample phases measured: + +| Renewal latency (ms) | Optimized recovery, 10,000 entries | Ready | +|---|---:|---:| +| p50 | 0.4978 | 0.5341 | +| p95 | 0.5914 | 0.7108 | +| p99 | 0.7255 | 1.2964 | +| Maximum | 0.7544 | 1.8799 | + +Do not infer that larger tables are faster from these separate local runs. +JIT warmup, scheduling and concurrent machine activity were not controlled. +The useful result is that the optimization avoids a repeated whole-table scan +during recovery and both populated-table runs preserved lease correctness. + +The explicit `RedisPresenceRecoveryCostProbe` test-source main class is not +automatically run by the unit suite and imposes no machine-dependent CI gate. +After test compilation, run it on the test classpath with argument `1000`. +It fails if the real recovery window ends before the cold measurement, rather +than silently reporting ready-state latency as recovery latency. diff --git a/docs/acceptance/presence_recovery_browser.mjs b/docs/acceptance/presence_recovery_browser.mjs new file mode 100644 index 0000000..5171454 --- /dev/null +++ b/docs/acceptance/presence_recovery_browser.mjs @@ -0,0 +1,52 @@ +// Real browser recovery against the disposable two-JVM harness. No API mocks. +import { createRequire } from 'node:module' +import { execFileSync } from 'node:child_process' +import path from 'node:path' + +const [frontend, first, second, container, output, profile = 'chromium'] = process.argv.slice(2) +const require = createRequire(path.join(frontend, 'package.json')) +const { chromium, firefox, webkit, devices, expect } = require('@playwright/test') +const profiles = { + chromium: [chromium, devices['Desktop Chrome']], + firefox: [firefox, devices['Desktop Firefox']], + webkit: [webkit, devices['Desktop Safari']], + mobile: [chromium, devices['Pixel 7']], + tablet: [webkit, devices['iPad (gen 7)']], +} +if (!profiles[profile]) throw new Error(`Unknown browser profile: ${profile}`) +const [engine, device] = profiles[profile] +const browser = await engine.launch({ headless: true }) +const contexts = [] +try { + const pages = [] + for (const [port, user] of [[first, 'alice'], [second, 'bob']]) { + const context = await browser.newContext(device) + contexts.push(context) + await context.tracing.start({ screenshots: true, snapshots: true }) + const page = await context.newPage() + await page.goto(`http://127.0.0.1:${port}/?createMockSession=true&user=${user}`) + // The root export redirects after hydration; finish it before record navigation. + await page.waitForURL(url => /^\/app\/?$/.test(url.pathname), { waitUntil: 'domcontentloaded' }) + await page.goto(`http://127.0.0.1:${port}/app/person/1`) + pages.push(page) + } + const unavailable = page => page.getByText('Presence temporarily unavailable', { exact: true }) + await expect(pages[0].locator('[data-qqq-id="presence-bob"]')).toBeVisible({ timeout: 90000 }) + await expect(pages[1].locator('[data-qqq-id="presence-alice"]')).toBeVisible({ timeout: 90000 }) + execFileSync('docker', ['exec', container, 'redis-cli', 'FLUSHDB']) + for (const page of pages) await expect(unavailable(page)).toBeVisible({ timeout: 35000 }) + await expect(pages[0].locator('[data-qqq-id="presence-bob"]')).toHaveCount(0) + await expect(pages[1].locator('[data-qqq-id="presence-alice"]')).toHaveCount(0) + await expect(pages[0].locator('[data-qqq-id="presence-bob"]')).toBeVisible({ timeout: 90000 }) + await expect(pages[1].locator('[data-qqq-id="presence-alice"]')).toBeVisible({ timeout: 90000 }) + for (const [index, page] of pages.entries()) { + await expect(unavailable(page)).toHaveCount(0) + await page.screenshot({ path: path.join(output, `recovered-${index}.png`) }) + } + console.log(`PASS: ${profile} shows recovery unavailable, clears stale people, and rejoins both JVMs without reload.`) +} finally { + for (const [index, context] of contexts.entries()) { + await context.tracing.stop({ path: path.join(output, `browser-${index}.zip`) }) + } + await browser.close() +} diff --git a/docs/acceptance/presence_recovery_processes.py b/docs/acceptance/presence_recovery_processes.py new file mode 100644 index 0000000..e406f7a --- /dev/null +++ b/docs/acceptance/presence_recovery_processes.py @@ -0,0 +1,172 @@ +"""Exercise Redis loss through two independent SharedPresenceDevServer JVMs. + +Run after test-compile and dependency:build-classpath with test scope and output +target/recovery-process-classpath.txt. Uses only disposable loopback fixtures. +""" + +import argparse +import hashlib +import json +import os +from pathlib import Path +import socket +import shutil +import subprocess +import time +import urllib.error +import urllib.request +import uuid + + +def free_port(): + """Reserve a candidate loopback port; startup detects any bind race.""" + with socket.socket() as listener: + listener.bind(("127.0.0.1", 0)) + return listener.getsockname()[1] + + +def request(port, user, method, path, body=None): + """Use the fixture's mock session through the actual HTTP boundary.""" + separator = "&" if "?" in path else "?" + url = f"http://127.0.0.1:{port}/qqq/v1/qbit/collaboration{path}{separator}user={user}" + data = None if body is None else json.dumps(body).encode() + req = urllib.request.Request(url, data=data, method=method, headers={ + "Cookie": f"sessionUUID={user}-recovery-processes", + "Content-Type": "application/json", + }) + try: + response = urllib.request.urlopen(req, timeout=10) + except urllib.error.HTTPError as error: + response = error + with response: + return response.status, json.load(response) + + +def require(status, result): + """Fail with fixture-only response evidence when transport differs.""" + assert result[0] == status, result + return result[1] + + +def settle(ports, leases): + """Renew on the opposite node until both nodes report both people.""" + deadline = time.monotonic() + 60 + sequence = 0 + while time.monotonic() < deadline: + sequence += 1 + for index, (user, lease) in enumerate(leases.items()): + result = request(ports[1 - index], user, "POST", "/presence/leases/renew", { + "connectionId": lease, "table": "person", "pk": "1", + "mode": "VIEWING", "sequence": sequence, + }) + assert result[0] in (200, 503), result + if result[0] == 503: + assert "recovering" in result[1]["message"], result + snapshots = [request(port, "alice", "GET", "/presence?table=person&pk=1") for port in ports] + if all(status == 200 for status, _ in snapshots): + assert all(len(body["viewers"]) == 2 for _, body in snapshots), snapshots + return + assert all(status == 503 for status, _ in snapshots), snapshots + time.sleep(1) + raise AssertionError("Two-process recovery exceeded 60 seconds") + + +def main(): + """Own and clean up the exact child JVMs and disposable Redis container.""" + parser = argparse.ArgumentParser() + parser.add_argument("--java", required=True) + parser.add_argument("--repo", type=Path, required=True) + parser.add_argument("--frontend-repo", type=Path) + parser.add_argument("--frontend-jar", type=Path, help="Serve this packaged frontend instead of a local export") + parser.add_argument("--backend-jar", type=Path, help="Use this backend artifact instead of target/classes") + parser.add_argument("--browser-profile", choices=("chromium", "firefox", "webkit", "mobile", "tablet"), default="chromium") + args = parser.parse_args() + repo = args.repo.resolve() + output = repo / "target" / ("recovery-processes-" + uuid.uuid4().hex[:10]) + output.mkdir() + if args.frontend_jar and not args.frontend_repo: + parser.error("frontend-jar requires frontend-repo for browser tooling") + dependencies = (repo / "target/recovery-process-classpath.txt").read_text().strip().split(os.pathsep) + assert not any(Path(item).name.startswith(("qbit-collaboration-", "qqq-frontend-next-")) for item in dependencies) + backend = args.backend_jar.resolve(strict=True) if args.backend_jar else repo / "target/classes" + classpath = os.pathsep.join([str(repo / "target/test-classes"), str(backend), *dependencies]) + evidence = {"status": "FAIL", "browser_profile": args.browser_profile, "cleanup": False} + if args.backend_jar: + evidence["backend_jar_sha256"] = hashlib.sha256(backend.read_bytes()).hexdigest() + if args.frontend_jar: + frontend = args.frontend_jar.resolve(strict=True) + evidence["frontend_jar_sha256"] = hashlib.sha256(frontend.read_bytes()).hexdigest() + classpath = str(frontend) + os.pathsep + classpath + elif args.frontend_repo: + shutil.copytree(args.frontend_repo.resolve() / "out", output / "next-dashboard") + classpath = str(output) + os.pathsep + classpath + container = "qqq-recovery-" + uuid.uuid4().hex[:12] + processes, logs = [], [] + try: + subprocess.run(["docker", "run", "-d", "--name", container, "-p", "127.0.0.1::6379", "redis:7.4-alpine"], check=True, capture_output=True) + binding = subprocess.check_output(["docker", "port", container, "6379"], text=True).strip() + redis_port = int(binding.rsplit(":", 1)[1]) + ports = [free_port(), free_port()] + for port in ports: + argfile = output / f"node-{port}.args" + argfile.write_text('-Dcollab.dev.frontend=' + str(bool(args.frontend_repo)).lower() + '\n-cp\n"' + classpath.replace("\\", "/") + '"\n' + 'com.kingsrook.qbits.collaboration.devserver.SharedPresenceDevServer\n' + f'{port}\n{redis_port}\n', encoding="utf-8") + log = (output / f"node-{port}.log").open("w") + logs.append(log) + processes.append(subprocess.Popen([args.java, "@" + str(argfile)], cwd=repo, + stdout=log, stderr=subprocess.STDOUT, + creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0))) + deadline = time.monotonic() + 60 + for port, process in zip(ports, processes): + while f"SHARED_PRESENCE_READY {port}" not in (output / f"node-{port}.log").read_text(errors="replace"): + assert process.poll() is None, f"Node exited; see {output}" + assert time.monotonic() < deadline, f"Startup timed out; see {output}" + time.sleep(0.2) + if args.frontend_repo: + subprocess.run(["node", str(repo / "docs/acceptance/presence_recovery_browser.mjs"), + str(args.frontend_repo.resolve()), str(ports[0]), str(ports[1]), container, str(output), args.browser_profile], check=True) + evidence["status"] = "PASS" + return + def join(): + """Acquire each person's new lease on a different node.""" + return {user: require(200, request(port, user, "POST", "/presence/leases", { + "table": "person", "pk": "1"}))["connectionId"] for port, user in zip(ports, ("alice", "bob"))} + original = join() + settle(ports, original) + subprocess.run(["docker", "exec", container, "redis-cli", "FLUSHDB"], check=True, capture_output=True) + for port in ports: + require(503, request(port, "alice", "GET", "/presence?table=person&pk=1")) + for user, lease in original.items(): + error = require(409, request(port, user, "POST", "/presence/leases/renew", { + "connectionId": lease, "table": "person", "pk": "1", "mode": "VIEWING", "sequence": 1000})) + assert error["code"] == "PRESENCE_LEASE_EXPIRED", error + replacement = join() + assert all(replacement[user] != original[user] for user in original) + settle(ports, replacement) + print(f"PASS: two independent JVMs recovered two viewers after Redis loss. Logs: {output}") + evidence["status"] = "PASS" + except Exception as error: + evidence["failure"] = str(error) + raise + finally: + for process in processes: + process.terminate() + for process in processes: + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + process.kill() + process.wait(timeout=10) + for log in logs: + log.close() + cleanup = subprocess.run(["docker", "rm", "-f", container], capture_output=True, check=False) + evidence["cleanup"] = cleanup.returncode == 0 and all(process.poll() is not None for process in processes) + (output / "result.json").write_text(json.dumps(evidence, indent=2), encoding="utf-8") + print(output, flush=True) + if evidence["status"] == "PASS" and not evidence["cleanup"]: + raise RuntimeError("Recovery passed but owned fixture cleanup failed") + + +if __name__ == "__main__": + main() diff --git a/docs/acceptance/presence_restore_processes.py b/docs/acceptance/presence_restore_processes.py new file mode 100644 index 0000000..8b22710 --- /dev/null +++ b/docs/acceptance/presence_restore_processes.py @@ -0,0 +1,166 @@ +"""Restore a disposable Redis image while both fixture JVMs are stopped. + +This verifies the controlled-restore protocol, not production orchestration. +Requires the same compiled test classpath as presence_recovery_processes.py. +""" + +import argparse +import hashlib +import json +import os +from pathlib import Path +import socket +import subprocess +import time +import uuid + +from presence_recovery_processes import free_port, request, require, settle + + +def main(): + """Own a Redis fixture and two generations of independently running nodes.""" + parser = argparse.ArgumentParser() + parser.add_argument("--java", required=True) + parser.add_argument("--repo", type=Path, required=True) + parser.add_argument("--backend-jar", type=Path, help="Verify this artifact without production target/classes") + args = parser.parse_args() + repo = args.repo.resolve() + output = repo / "target" / ("restore-processes-" + uuid.uuid4().hex[:10]) + output.mkdir() + dependencies = (repo / "target/recovery-process-classpath.txt").read_text().strip().split(os.pathsep) + assert all(Path(item).is_file() and item.endswith('.jar') for item in dependencies) + assert not any(Path(item).name.startswith("qbit-collaboration-") for item in dependencies) + backend = args.backend_jar.resolve(strict=True) if args.backend_jar else repo / "target/classes" + classpath = os.pathsep.join([str(repo / "target/test-classes"), str(backend), *dependencies]) + evidence = {"status": "FAIL", "cleanup": False, "checks": []} + if args.backend_jar: + evidence["backend_jar_sha256"] = hashlib.sha256(backend.read_bytes()).hexdigest() + container = "qqq-restore-" + uuid.uuid4().hex[:12] + container_created = False + processes, logs = [], [] + ports = [free_port(), free_port()] + + def redis(*command): + """Run commands only against this owned, disposable Redis container.""" + return subprocess.check_output(["docker", "exec", container, "redis-cli", *command], text=True).strip() + + def stop_nodes(): + """Require process termination and closed old listeners before restore.""" + for process in processes: + if process.poll() is None: + process.terminate() + for process in processes: + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + process.kill() + process.wait(timeout=10) + processes.clear() + for port in ports: + with socket.socket() as connection: + connection.settimeout(1) + assert connection.connect_ex(("127.0.0.1", port)) != 0, "Old listener remains open" + + def start_nodes(generation, redis_port): + """Supply the shared generation outside the Redis image, on both JVMs.""" + for port in ports: + argfile = output / f"{generation}-{port}.args" + argfile.write_text(f'-Dcollab.dev.recoveryGeneration={generation}\n-cp\n"' + + classpath.replace("\\", "/") + '"\n' + 'com.kingsrook.qbits.collaboration.devserver.SharedPresenceDevServer\n' + f'{port}\n{redis_port}\n', encoding="utf-8") + logfile = output / f"{generation}-{port}.log" + log = logfile.open("w") + logs.append(log) + processes.append(subprocess.Popen([args.java, "@" + str(argfile)], cwd=repo, + stdout=log, stderr=subprocess.STDOUT, + creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0))) + deadline = time.monotonic() + 60 + for port, process in zip(ports, processes): + logfile = output / f"{generation}-{port}.log" + while f"SHARED_PRESENCE_READY {port}" not in logfile.read_text(errors="replace"): + assert process.poll() is None, f"Node exited; see {output}" + assert time.monotonic() < deadline, f"Startup timed out; see {output}" + time.sleep(0.2) + + def join(): + """Issue each person's lease on a separate node.""" + return {user: require(200, request(port, user, "POST", "/presence/leases", { + "table": "person", "pk": "1"}))["connectionId"] for port, user in zip(ports, ("alice", "bob"))} + + def reject_old(leases, sequence): + """Restored lease authority must fail on both new-generation nodes.""" + for port in ports: + for user, lease in leases.items(): + error = require(409, request(port, user, "POST", "/presence/leases/renew", { + "connectionId": lease, "table": "person", "pk": "1", + "mode": "VIEWING", "sequence": sequence})) + assert error["code"] == "PRESENCE_LEASE_EXPIRED", error + + try: + # DEBUG RELOAD is enabled only on this disposable loopback fixture. + subprocess.run(["docker", "run", "-d", "--name", container, "-p", "127.0.0.1::6379", + "redis:7.4-alpine", "redis-server", "--enable-debug-command", "yes", "--save", ""], + check=True, capture_output=True) + container_created = True + binding = subprocess.check_output(["docker", "port", container, "6379"], text=True).strip() + redis_port = int(binding.rsplit(":", 1)[1]) + start_nodes("before-restore", redis_port) + original = join() + settle(ports, original) + last_renewal = time.monotonic() + for port, (user, lease) in zip(ports, original.items()): + require(200, request(port, user, "POST", "/presence/leases/renew", { + "connectionId": lease, "table": "person", "pk": "1", + "mode": "VIEWING", "sequence": 100})) + assert redis("SAVE") == "OK" + before = int(redis("DBSIZE")) + assert before > 0 + stop_nodes() + evidence["checks"].append("old JVMs stopped and listeners closed before restore") + assert redis("FLUSHDB") == "OK" + assert redis("DBSIZE") == "0" + assert redis("DEBUG", "RELOAD", "NOSAVE") == "OK" + assert int(redis("DBSIZE")) > 0, "Saved Redis image was not restored" + evidence["checks"].append("nonempty saved Redis image restored") + start_nodes("restored-" + uuid.uuid4().hex, redis_port) + reject_old(original, 1000) + # Avoid a false pass caused only by the old 45-second lease expiring. + evidence["old_lease_rejection_seconds"] = time.monotonic() - last_renewal + assert evidence["old_lease_rejection_seconds"] < 40, "Restore exceeded the live-lease proof window" + evidence["checks"].append("both nodes reject old leases before normal expiry") + for port in ports: + result = require(503, request(port, "alice", "GET", "/presence?table=person&pk=1")) + assert "recovering" in result["message"], result + replacement = join() + assert all(replacement[user] != original[user] for user in original) + settle(ports, replacement) + reject_old(original, 1001) + evidence["checks"].append("both nodes recover with fresh viewers and still reject old leases") + if args.backend_jar: + assert hashlib.sha256(backend.read_bytes()).hexdigest() == evidence["backend_jar_sha256"] + evidence["status"] = "PASS" + print(f"PASS: restored Redis image cannot authorize old leases on either new-generation JVM. Logs: {output}") + except Exception as error: + evidence["failure"] = str(error) + raise + finally: + try: + try: + stop_nodes() + finally: + for log in logs: + log.close() + if container_created: + subprocess.run(["docker", "rm", "-f", container], capture_output=True, check=True) + evidence["cleanup"] = True + except Exception as error: + evidence["status"] = "FAIL" + evidence["cleanup_failure"] = str(error) + raise + finally: + (output / "result.json").write_text(json.dumps(evidence, indent=2), encoding="utf-8") + + +if __name__ == "__main__": + main() diff --git a/docs/performance/MULTI-NODE.md b/docs/performance/MULTI-NODE.md new file mode 100644 index 0000000..db3d1f0 --- /dev/null +++ b/docs/performance/MULTI-NODE.md @@ -0,0 +1,264 @@ +# Independent-node presence workload + +This fixture targets QQQ #1302/#1291. It runs 2–8 independent JVM HTTP servers +against an owned loopback Redis container. Every node serves the same configured +mock identities across 2–8 tenant scopes. The scope comes from the authenticated +fixture identity, never a supplied tenant parameter. Records are independently +seeded in memory; this does not qualify shared database or production IAM behavior. + +Build with Java 21: + +```powershell +./mvnw.cmd test-compile dependency:build-classpath '-Dmdep.includeScope=test' '-Dmdep.outputFile=target/capacity-classpath.txt' +python docs/performance/presence_nodes.py --java /path/to/java --nodes 3 --tenants 2 --users 6 --tabs 4 --rounds 100 +``` + +Four tabs per user means two duplicate tabs on each of two records. Every round +renews each issued lease on a rotating node, with sequential per-tab sequences. +All nodes must then return exactly the expected tenant's distinct people and +exactly two editors on both records. Any mismatch or request failure fails the +run. Requests use bounded worker concurrency and ten-second timeouts. Each node +has eight pooled Redis connections, a 500 ms pool wait and a 384 MiB heap ceiling. + +Results and per-node logs are retained under `target/capacity-nodes-`. +The result records parameters, node PIDs, elapsed time, sample counts and +nearest-rank HTTP p50/p95/p99/max. Renewal throughput includes snapshot checks +and is not maximum server throughput. All owned JVMs and the Redis container are +removed in `finally`; the receipt reports cleanup separately from test success. + +Each fixture JVM emits resource samples every 100 ms using Java management APIs +and the Redis QBit's existing pool metrics. The receipt summarizes sampled heap +used/committed bytes, active/idle/waiting pool counts and the configured maximum. +CPU core equivalents are process CPU time divided by observed wall time; they +can exceed one for a multithreaded JVM. Observation begins after server startup +and includes lease setup and workload execution. Unsupported CPU counters are +reported as null. Linux fixtures also read process `VmRSS` from `/proc/self/status` +every 100 ms and record `residentBytes_sampled_max`; unsupported platforms report +null. Linux-container runs require resident-memory samples. Heap is not total +resident memory, and process RSS is not the container's complete cgroup footprint. +Sampled peaks may miss +short bursts and must not be represented as exact maxima. Raw samples remain in +node logs; missing node samples or a sampled pool-bound violation fail the run. + +The final Redis INFO capture records `used_memory`, lifetime `used_memory_peak`, +and current `used_memory_rss`, plus process user/system CPU seconds since Redis +startup. Redis RSS is an end-of-run observation, not a sampled peak. These fresh, +owned Redis instances include fixture setup and diagnostics in their lifetime; +do not interpret those CPU values as workload-only measurements or add independent +process peaks and call the sum a measured simultaneous maximum. Missing Redis +memory/CPU diagnostics fail acceptance while cleanup still runs. + +The [100-user resource run](results/linux-100-users-resources.json) used three +Linux JVMs, two tenants with 50 people each, four leases per person, 16 request +workers and 200 rounds. All 80,000 renewals and 2,400 exact-membership snapshots +passed in 102.568 workload seconds. Renewal p50/p95/p99/max were +11.326/21.765/25.881/1,007.404 ms; snapshot p99 was 19.424 ms. The one-second +renewal maximum is retained, not discarded. Cleanup passed. + +Sampled JVM RSS peaks were 662,786,048 / 662,188,032 / 703,479,808 bytes +(approximately 632–671 MiB), while used-heap peaks were about 171–177 MiB. +CPU observations were 1.73–1.83 core equivalents per JVM; sampled pool-active +peaks were 7/7/6 of eight with no sampled waiters. Redis peak allocated memory +was 2,579,896 bytes; final RSS was 8,777,728 bytes. Redis process CPU since +startup was 23.738 system plus 35.526 user seconds. These measurements include +fixture overhead, omit browser load, and do not establish deployment limits. + +Initial smoke run: three JVMs, two tenants, six users each, four tabs per user, +three rounds: 144 renewals and 36 snapshots passed with tenant isolation, +duplicate-tab deduplication and two editors retained. Cleanup passed. This short +run validates the harness, not sustained capacity or operational thresholds. + +A second run with 12 users per tenant and 100 rounds passed 9,600 renewals and +1,200 snapshots in 23.110 seconds. Renewal p50/p95/p99 were 7.246/16.454/21.853 ms; +snapshot p50/p95/p99 were 12.908/15.776/18.239 ms. Every round checked exact +tenant membership, deduplication and two editors on both records at every node. +Cleanup passed. These are measurements from one local Windows/Docker run; +they do not establish deployment limits or browser propagation latency. + +## Sustained-run limitation under investigation + +The 1,000-round workload has intermittently failed with HTTP 503 on lease +renewal. Four runs failed; one completed 96,000 renewals and 12,000 snapshots. +The passing run does not resolve the failures. Temporary local diagnostics on +the third failure identified `JedisConnectionException` caused by +`SocketTimeoutException` while reading the `EVALSHA` response in +`RedisPresenceStore.mutate`. The configured socket timeout remained two seconds. +The temporary production-file diagnostics were removed after identifying this +path. No timeout increase, hidden request retry or production fix is included. + +Resource samples did not show pool waiters, but cannot rule out short bursts. +Redis slow-command timings and latency events are now captured before fixture +cleanup, alongside Redis CPU, memory and command counters, to distinguish server +execution cost from transport or scheduling delays. Slow-command summaries omit +command arguments. These diagnostics belong only to the disposable fixture. +Until the cause is established and qualified, this workload establishes no +supported sustained throughput or production alert thresholds. + +The timing-instrumented Windows-JVM run failed at round 480, with no commands +at or above the configured one-millisecond Redis slow-log threshold and no +latency-monitor events. Redis reported about 93 microseconds per `EVALSHA` on +average. This weakens a slow-Lua explanation but does not identify the delayed +component or exclude time spent outside command execution. + +## Linux comparison + +`--node-image eclipse-temurin:21-jre` optionally runs each JVM in its own Linux +container on an owned Docker network shared with Redis. The HTTP driver remains +on the host. Only loopback HTTP ports are published. Compiled fixture classes +and copied dependency JARs are mounted read-only; Redis remains disposable. +The existing loopback defaults remain unchanged when this option is absent. +Container mode records Docker client PIDs separately from container identities. +It removes the Windows-JVM-to-published-Redis-port path, but also changes the JVM +platform and scheduling environment; it is a diagnostic comparison, not proof +of a specific operating-system or Docker defect. Cleanup includes every owned +node container and the owned network. + +`--churn-every 20` departs all tabs belonging to the last user in each tenant +every 20 rounds. Every departed lease must reject a later renewal with HTTP 409 +and `PRESENCE_LEASE_EXPIRED` on another node. All surviving nodes must show that +user absent on both records, then show the original exact membership after +fresh, distinct leases are acquired and activated. Only this explicit rejoin +path acquires new leases; the node-crash path preserves existing ones. + +`--pause-redis` adds a controlled failure phase after the ordinary rounds. It +freezes only the owned Redis container, sends up to 64 concurrent renewal +requests, and requires HTTP 503 within an eight-second fixture bound. Snapshots +must also fail explicitly rather than return an empty room. Redis is unpaused +in `finally`; once reachable, higher-sequence renewals must recover the same +leases and exact membership. The test requires observed pool waiters and rejects +sampled active counts above the configured maximum. It does not retry failed +requests invisibly or raise production timeouts. + +The [Linux Redis-stall receipt](results/linux-redis-stall.json) passed with three +nodes, two tenants, 24 users and 96 tab leases. All 96 stalled renewals and all +three node snapshots returned the expected unavailable response. Existing +leases recovered after Redis resumed, and cleanup passed. This proves bounded +failure and recovery for an injected stall; it does not explain the intermittent +Windows timeout or establish a saturation throughput threshold. + +```powershell +python docs/performance/presence_nodes.py --java java --node-image eclipse-temurin:21-jre --nodes 3 --tenants 2 --users 12 --tabs 4 --rounds 1000 +``` + +The first Linux comparison completed 1,000 rounds in 250.811 seconds: 96,000 +renewals and 12,000 snapshots, with exact membership/editor assertions and +cleanup passing. Renewal p95/p99 were 15.331/17.760 ms. The sampled active pool +peak was five of eight, with no sampled waiters. This one run is separate from +the unresolved Windows failures and does not establish a production SLA. + +`--crash-round 50` abruptly kills the first node before that round, then sends +all existing leases through surviving nodes. It does not acquire replacement +leases or hide failed requests with retries. Exact tenant membership, duplicate +tabs and editor counts must remain correct on every surviving node. This tests +HTTP lease continuity after process loss, not browser/SSE reconnect, ingress +failover, database durability or a Redis outage. + +The Linux crash run passed 100 rounds (9,600 renewals and 996 snapshots) in +25.148 seconds. The first JVM exited with code 137 at round 50; the remaining +two retained all issued leases and expected membership. Cleanup passed. + +A [combined churn and node-loss run](results/linux-node-crash-churn.json) passed +100 rounds with the first node killed at round 50 and churn every 20 rounds. +Five cycles rejected all 40 departed handles and checked absence/restored +membership across both tenants and records on every surviving node. Cleanup +passed. This is bounded HTTP lifecycle qualification; browser reconnect and +missed-event behavior remain separate requirements. + +Recorded receipts retain both successful and failing evidence: + +- [Windows timeout with Redis diagnostics](results/windows-timeout.json) +- [Linux 1,000-round run](results/linux-1000-rounds.json) +- [Linux node crash](results/linux-node-crash.json) + +Only the Java executable path is normalized in these receipts. The Linux image +used for these runs was +`eclipse-temurin@sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99`. + +## Browser propagation baseline + +`browser_subscription_affinity.mjs FRONTEND_REPO BASE_URL RESULT_PATH` verifies +the current owning-node subscription contract against the two-node fixture. It +uses the connection ID from the real browser's successful subscription request, +then preserves the same session cookies while changing only the test routing +cookie. The [published-artifact receipt](results/browser-subscription-affinity.json) +records owner 200, other node 404, owner 200, with the requested channel active +on both successful responses and the unsaved draft retained. This proves why +subscription requests need owning-node routing even when presence state is shared. +It does not test the deployed Traefik cookie, HTTPS, buffering or failover. +The packaged runner accepts this script through `--affinity-script`. + +`browser_presence_latency.mjs` measures Chromium field-focus changes until a +different editor sees the corresponding remote marker. Supply a caller-owned +two-node loopback fixture serving the published frontend with `alice`/`admin` +mock identities and the acceptance router's `presence_node=first/second` +affinity cookies: + +```powershell +node docs/performance/browser_presence_latency.mjs /path/to/frontend http://localhost:PORT result.json 25 +``` + +Use `presence_router.mjs FIRST_NODE_PORT SECOND_NODE_PORT ROUTER_PORT` for this +test. It binds only to loopback and adds a test-only POST endpoint that severs +first-node SSE responses. Never deploy this fault-injection router. The packaged +acceptance fixture accepts it through `--router-script` alongside `--latency-script`. + +The test verifies the target marker is absent before every change, the old +marker disappears afterward, both real SSE streams opened successfully, +issued-lease writes reached distinct nodes, and unsaved drafts survive. API +responses are not mocked. The fixture owns backend/router/Redis cleanup; this +script owns and closes its browser. + +After timing the normal transitions, the script takes the observer offline and +explicitly severs its existing SSE response through the fixture router. It requires +an observed stream failure, a failed offline fetch, three acknowledged sender focus +changes, and absence of the new marker while disconnected. After network restoration +it requires a new successful SSE response and the current marker with the previous +marker removed, without reload or loss of either unsaved draft. Recovery may use +the shipped reconnect and polling behavior; this does not prove event replay. + +The [disconnect/recovery receipt](results/browser-reconnect.json) passed against +the public RC37 JAR: one stream severed, three missed changes, recovery in 1,858 ms, +both drafts preserved. The accompanying six-user and Redis admin checks passed; +fixture cleanup passed. This is one Chromium disconnect case without concurrent +load, not a reconnect storm, lease-expiry test, or production recovery objective. + +Append `load` to the browser command (or use `--latency-load` on the packaged +fixture) to add 16 concurrent HTTP lease workers: four mock identities with four +leases each on the same record, split between both nodes. Each worker renews +sequentially with a 100 ms pause after each response. The browser observations +are spread over at least 60 seconds; every observation interval must include +completed background renewals. The workload continues during disconnect/reconnect. +Any request or lease-cleanup failure fails the test. Responses are disposed after +reading, and workers stop before the contexts close. The receipt reports completed +renewals, observed node counts, elapsed time and any errors. These are HTTP load +workers, not 16 additional browser tabs or distinct people. + +The [bounded load/reconnect receipt](results/browser-load-reconnect.json) passed +9,046 renewals over 69.728 seconds, with no worker or lease-release errors. +Across 25 observed focus transitions, p50/p95/p99 were 32.384/85.391/785.495 ms; +p99 is the maximum of this small sample, not a reliable tail-latency estimate. +The observer recovered in 1,855.887 ms after missing three updates while load +continued. Both drafts survived. Six-user and Redis admin acceptance also passed +against the public RC37 JAR; fixture cleanup passed. An initial run failed only +because the new load helper used the wrong lease-release endpoint; that receipt +is retained in the local packaged fixture history, not counted as a passing run. + +The [25-transition receipt](results/browser-focus-25.json) passed against the +downloaded public RC37 JAR and published backend snapshots. Focus-to-observed-DOM +p50/p95/p99 were 31.648/33.078/36.092 ms; p99 is the maximum in this small sample. +There were no page errors or failed lease responses. Six-user presence and Redis +admin browser checks also passed in the same fixture run; fixture cleanup passed. + +This baseline includes browser automation, UI debounce, network, invalidation +and rendering. It is not a server-only transport measurement, an SSE-only causal +proof, an under-load result, or a production latency objective. The separate load +receipt above covers one bounded workload and disconnect scenario; larger loads, +reconnect storms and deployment-specific behavior remain unqualified. + +Remaining qualification includes longer soak and larger workloads, container +resource limits, reconnect storms and the actual deployment's capacity and alert +thresholds. The receipts above cover specific bounded HTTP, browser, saturation, +churn, missed-update and process-crash cases, not every combination. The native +Windows intermittent Redis read timeout remains unresolved. Loopback HTTP latency must not be reported +as browser propagation or a production SLA. No deployed capacity threshold is +established by this fixture. diff --git a/docs/performance/browser_presence_latency.mjs b/docs/performance/browser_presence_latency.mjs new file mode 100644 index 0000000..be5944b --- /dev/null +++ b/docs/performance/browser_presence_latency.mjs @@ -0,0 +1,145 @@ +// Real UI observations against a caller-owned two-node acceptance fixture. +import { createRequire } from 'node:module'; +import { resolve } from 'node:path'; +import { writeFile } from 'node:fs/promises'; +import { performance } from 'node:perf_hooks'; +import { setTimeout as delay } from 'node:timers/promises'; +import { startPresenceLoad } from './browser_presence_load.mjs'; + +const [frontendRepo, base, output, countValue = '25', loadMode] = process.argv.slice(2); +if (loadMode && loadMode !== 'load') throw new Error('Optional final argument must be load'); +const samples = Number(countValue); +if (!frontendRepo || !base || !output || !Number.isInteger(samples) || samples < 5 || samples > 100) { + throw new Error('Expected frontend repo, loopback base URL, result path and 5..100 samples'); +} +if (!['localhost', '127.0.0.1'].includes(new URL(base).hostname)) { + throw new Error('Only the disposable loopback fixture is supported'); +} +const require = createRequire(resolve(frontendRepo, 'package.json')); +const { chromium, expect } = require('@playwright/test'); +const result = { status: 'FAIL', samples: [], streams: [], requests: [], errors: [], + scope: 'Chromium focus-to-remote-DOM observation, including automation and UI debounce; no concurrent load' }; +const browser = await chromium.launch({ headless: true }); +let load; +try { + async function editor(user, node) { + const context = await browser.newContext(); + await context.addCookies([ + { name: 'collab_dev_user', value: user, url: base }, + { name: 'presence_node', value: node, url: base }, + ]); + const page = await context.newPage(); + page.setDefaultTimeout(15000); + page.on('pageerror', error => result.errors.push(error.message)); + page.on('response', response => { + const path = new URL(response.url()).pathname; + if (path.endsWith('/realtime/stream')) { + result.streams.push({ user, status: response.status(), type: response.headers()['content-type'] }); + } + if (path.includes('/presence/leases')) { + result.requests.push({ user, status: response.status(), node: response.headers()['x-acceptance-node'] }); + } + }); + await page.goto(`${base}/app/person/1/edit?createMockSession=true`); + await expect(page.locator('[data-qqq-id="firstName"]')).toBeVisible(); + return page; + } + const observer = await editor('alice', 'first'); + const sender = await editor('admin', 'second'); + await observer.locator('[data-qqq-id="firstName"]').fill('Retained observer draft'); + await sender.locator('[data-qqq-id="firstName"]').fill('Retained sender draft'); + await expect(observer.locator('[data-presence-field="firstName"]')).toContainText('Demo Administrator'); + if (loadMode) { + load = await startPresenceLoad(browser, base, expect); + result.load = load.state; + result.scope = 'Chromium focus-to-remote-DOM and reconnect under 16 concurrent HTTP lease workers; loopback, four background identities'; + } + for (let index = 0; index < samples; index++) { + const previousCompleted = load?.state.completed; + if (load) await delay(60000 / samples); + const field = index % 2 === 0 ? 'lastName' : 'firstName'; + const previous = field === 'firstName' ? 'lastName' : 'firstName'; + await expect(observer.locator(`[data-presence-field="${field}"]`)).not.toContainText('Demo Administrator'); + const before = performance.now(); + await sender.locator(`[data-qqq-id="${field}"]`).focus(); + await expect(observer.locator(`[data-presence-field="${field}"]`)).toContainText('Demo Administrator'); + result.samples.push({ field, propagation_ms: performance.now() - before }); + await expect(observer.locator(`[data-presence-field="${previous}"]`)).not.toContainText('Demo Administrator'); + if (load) { + expect(load.state.errors).toEqual([]); + expect(load.state.completed).toBeGreaterThan(previousCompleted); + } + } + // Drop a real browser connection, miss multiple focus updates, then converge + // through the shipped reconnect/polling behavior without reloading the form. + const oldField = samples % 2 === 0 ? 'firstName' : 'lastName'; + const newField = oldField === 'firstName' ? 'lastName' : 'firstName'; + const streamsBefore = result.streams.filter(stream => stream.user === 'alice').length; + result.reconnect = { status: 'FAIL', missed_changes: 0 }; + const interrupted = observer.waitForEvent('requestfailed', { + predicate: request => new URL(request.url()).pathname.endsWith('/realtime/stream'), + timeout: 10000, + }); + await observer.context().setOffline(true); + const disconnect = await fetch(`${base}/__acceptance/disconnect-first`, { method: 'POST' }); + expect(disconnect.status).toBe(200); + result.reconnect.disconnected_streams = (await disconnect.json()).disconnected; + expect(result.reconnect.disconnected_streams).toBeGreaterThan(0); + const failedStream = await interrupted; + result.reconnect.stream_failure = failedStream.failure()?.errorText; + const offlineFetch = await observer.evaluate(async () => { + try { + await fetch(`/app/?offlineProbe=${Date.now()}`, { cache: 'no-store' }); + return false; + } catch { return true; } + }); + expect(offlineFetch).toBe(true); + for (const field of [newField, oldField, newField]) { + const renewed = sender.waitForResponse(response => + new URL(response.url()).pathname.endsWith('/presence/leases/renew') && + response.request().method() === 'POST' && response.status() === 200 && + response.request().postDataJSON()?.fieldName === field); + await sender.locator(`[data-qqq-id="${field}"]`).focus(); + await renewed; + result.reconnect.missed_changes++; + } + await expect(observer.locator(`[data-presence-field="${newField}"]`)).not.toContainText('Demo Administrator'); + const resumed = performance.now(); + await observer.context().setOffline(false); + await expect.poll(() => result.streams.filter(stream => stream.user === 'alice' && + stream.status === 200 && stream.type?.includes('text/event-stream')).length, + { timeout: 30000 }).toBeGreaterThan(streamsBefore); + await expect(observer.locator(`[data-presence-field="${newField}"]`)).toContainText('Demo Administrator', { timeout: 30000 }); + await expect(observer.locator(`[data-presence-field="${oldField}"]`)).not.toContainText('Demo Administrator'); + result.reconnect.recovery_ms = performance.now() - resumed; + result.reconnect.status = 'PASS'; + if (load) { + await load.stop(); + expect(load.state.errors).toEqual([]); + expect(load.state.elapsed_ms).toBeGreaterThanOrEqual(60000); + expect(load.state.nodes.first).toBeGreaterThan(0); + expect(load.state.nodes.second).toBeGreaterThan(0); + load = undefined; + } + await expect(observer.locator('[data-qqq-id="firstName"]')).toHaveValue('Retained observer draft'); + await expect(sender.locator('[data-qqq-id="firstName"]')).toHaveValue('Retained sender draft'); + for (const user of ['alice', 'admin']) { + expect(result.streams.some(stream => stream.user === user && stream.status === 200 && stream.type?.includes('text/event-stream'))).toBe(true); + } + expect(result.requests.some(request => request.user === 'alice' && request.node === 'first')).toBe(true); + expect(result.requests.some(request => request.user === 'admin' && request.node === 'second')).toBe(true); + expect(result.requests.every(request => request.status === 200)).toBe(true); + expect(result.errors).toEqual([]); + const ordered = result.samples.map(sample => sample.propagation_ms).sort((a, b) => a - b); + result.percentiles_ms = Object.fromEntries([['p50', .5], ['p95', .95], ['p99', .99], ['max', 1]] + .map(([name, quantile]) => [name, ordered[Math.ceil(ordered.length * quantile) - 1]])); + result.status = 'PASS'; +} catch (error) { + result.failure = String(error); + process.exitCode = 1; +} finally { + if (load) await load.stop(); + await browser.close(); + await writeFile(output, JSON.stringify(result, null, 2)); + console.log(JSON.stringify(result, null, 2)); +} diff --git a/docs/performance/browser_presence_load.mjs b/docs/performance/browser_presence_load.mjs new file mode 100644 index 0000000..562f0c0 --- /dev/null +++ b/docs/performance/browser_presence_load.mjs @@ -0,0 +1,75 @@ +// Bounded real HTTP load using caller-owned authenticated browser contexts. +import { performance } from 'node:perf_hooks'; +import { setTimeout as delay } from 'node:timers/promises'; + +export async function startPresenceLoad(browser, base, expect) { + const state = { workers: 16, completed: 0, errors: [], nodes: {}, elapsed_ms: 0 }; + const contexts = []; + const leases = []; + let running = true; + let stopped = false; + let tasks = []; + const began = performance.now(); + const endpoint = `${base}/qqq/v1/qbit/collaboration/presence/leases`; + async function post(context, path, data) { + const response = await context.request.post(endpoint + path, { data, timeout: 10000 }); + try { + expect(response.status()).toBe(200); + const node = response.headers()['x-acceptance-node']; + state.nodes[node] = (state.nodes[node] ?? 0) + 1; + return await response.json(); + } finally { await response.dispose(); } + } + async function stop() { + if (stopped) return; + stopped = true; + running = false; + await Promise.all(tasks); + state.elapsed_ms = performance.now() - began; + for (const lease of leases) { + try { + const response = await lease.context.request.delete(`${base}/qqq/v1/qbit/collaboration/presence`, { + data: { ...lease.target, connectionId: lease.id, sequence: ++lease.sequence }, timeout: 10000, + }); + try { expect(response.status()).toBe(200); } finally { await response.dispose(); } + } catch (error) { state.errors.push(String(error)); } + } + for (const context of contexts) await context.close(); + } + try { + for (const [index, user] of ['bob', 'carol', 'dave', 'erin'].entries()) { + const context = await browser.newContext(); + contexts.push(context); + await context.addCookies([ + { name: 'collab_dev_user', value: user, url: base }, + { name: 'presence_node', value: index % 2 ? 'second' : 'first', url: base }, + ]); + const page = await context.newPage(); + await page.goto(`${base}/app/person/1?createMockSession=true`); + await expect(page.locator('[data-qqq-id="record-view-title-person"]')).toBeVisible(); + await page.close(); + for (let tab = 0; tab < 4; tab++) { + const target = { table: 'person', pk: '1' }; + const opened = await post(context, '', target); + expect(typeof opened.connectionId).toBe('string'); + leases.push({ context, target, id: opened.connectionId, sequence: 0, completed: 0 }); + } + } + tasks = leases.map(async lease => { + try { + while (running) { + await post(lease.context, '/renew', { ...lease.target, connectionId: lease.id, + sequence: ++lease.sequence, mode: 'VIEWING' }); + lease.completed++; + state.completed++; + await delay(100); + } + } catch (error) { state.errors.push(String(error)); running = false; } + }); + await expect.poll(() => leases.filter(lease => lease.completed > 0).length).toBe(16); + return { state, stop }; + } catch (error) { + await stop(); + throw error; + } +} diff --git a/docs/performance/browser_subscription_affinity.mjs b/docs/performance/browser_subscription_affinity.mjs new file mode 100644 index 0000000..374f466 --- /dev/null +++ b/docs/performance/browser_subscription_affinity.mjs @@ -0,0 +1,60 @@ +// Verify owning-node routing with real published UI and backend artifacts. +import { createRequire } from 'node:module'; +import { resolve } from 'node:path'; +import { writeFile } from 'node:fs/promises'; + +const [frontendRepo, base, output] = process.argv.slice(2); +if (!frontendRepo || !output || !['localhost', '127.0.0.1'].includes(new URL(base).hostname)) { + throw new Error('Expected frontend repo, owned loopback URL and result path'); +} +const require = createRequire(resolve(frontendRepo, 'package.json')); +const { chromium, expect } = require('@playwright/test'); +const result = { status: 'FAIL', checks: [], errors: [] }; +const browser = await chromium.launch({ headless: true }); +try { + const context = await browser.newContext(); + await context.addCookies([ + { name: 'collab_dev_user', value: 'alice', url: base }, + { name: 'presence_node', value: 'first', url: base }, + ]); + const page = await context.newPage(); + page.on('pageerror', error => result.errors.push(error.message)); + const subscribed = page.waitForResponse(response => + new URL(response.url()).pathname.endsWith('/realtime/subscriptions') && response.status() === 200); + await page.goto(`${base}/app/person/1/edit?createMockSession=true`); + await expect(page.locator('[data-qqq-id="firstName"]')).toBeVisible(); + await page.locator('[data-qqq-id="firstName"]').fill('Keep this unsaved draft'); + const response = await subscribed; + expect(response.headers()['x-acceptance-node']).toBe('first'); + const { connectionId } = response.request().postDataJSON(); + expect(typeof connectionId).toBe('string'); + for (const [node, expected] of [['first', 200], ['second', 404], ['first', 200]]) { + // Keep the same session/user cookies and change only the test routing cookie. + const cookies = (await context.cookies(base)).map(cookie => + `${cookie.name}=${cookie.name === 'presence_node' ? node : cookie.value}`).join('; '); + const changed = await context.request.post(`${base}/qqq/v1/realtime/subscriptions`, { + headers: { Cookie: cookies }, + data: { connectionId, add: ['record:person:1'], remove: [] }, timeout: 10000, + }); + try { + result.checks.push({ node: changed.headers()['x-acceptance-node'], status: changed.status() }); + expect(changed.headers()['x-acceptance-node']).toBe(node); + expect(changed.status()).toBe(expected); + if (expected === 200) { + const body = await changed.json(); + expect(body.active).toContain('record:person:1'); + expect(body.refused ?? []).toEqual([]); + } + } finally { await changed.dispose(); } + } + await expect(page.locator('[data-qqq-id="firstName"]')).toHaveValue('Keep this unsaved draft'); + expect(result.errors).toEqual([]); + result.status = 'PASS'; +} catch (error) { + result.failure = String(error); + process.exitCode = 1; +} finally { + await browser.close(); + await writeFile(output, JSON.stringify(result, null, 2)); + console.log(JSON.stringify(result, null, 2)); +} diff --git a/docs/performance/presence_nodes.py b/docs/performance/presence_nodes.py new file mode 100644 index 0000000..6ab098a --- /dev/null +++ b/docs/performance/presence_nodes.py @@ -0,0 +1,343 @@ +"""Bounded HTTP presence workload across independent, disposable JVMs. + +This measures loopback HTTP service latency, not browser propagation or a +production capacity limit. Run test-compile and dependency:build-classpath with +test scope and output target/capacity-classpath.txt before invoking this script. +""" + +import argparse +from concurrent.futures import ThreadPoolExecutor +import json +import math +import os +from pathlib import Path +import shutil +import socket +import subprocess +import sys +import time +import urllib.error +import urllib.request +import uuid + + +def port(): + """Choose a loopback port; child startup detects a competing bind.""" + with socket.socket() as listener: + listener.bind(("127.0.0.1", 0)) + return listener.getsockname()[1] + + +def request(node, user, path, body=None, tab=0, method=None, expected=200): + """Use real HTTP and the explicit mock identity; no tenant parameter.""" + separator = "&" if "?" in path else "?" + req = urllib.request.Request( + f"http://127.0.0.1:{node}/qqq/v1/qbit/collaboration{path}{separator}user={user}", + data=None if body is None else json.dumps(body).encode(), + method=method, + headers={"Content-Type": "application/json", + "Cookie": f"sessionUUID={user}-capacity-{tab}"}) + try: + response = urllib.request.urlopen(req, timeout=10) + except urllib.error.HTTPError as error: + if error.code == expected: + with error: + return json.load(error) + with error: + detail = error.read(4096).decode(errors="replace") + raise RuntimeError(f"HTTP {error.code} node={node} user={user} path={path}: {detail}") from error + with response: + if response.status != expected: + raise AssertionError(f"Unexpected status {response.status}") + return json.load(response) + + +def distribution(values): + """Nearest-rank milliseconds, with no discarded slow or failed samples.""" + ordered = sorted(values) + return {"samples": len(ordered), **{ + name: ordered[max(0, math.ceil(len(ordered) * fraction) - 1)] + for name, fraction in (("p50_ms", .5), ("p95_ms", .95), ("p99_ms", .99), ("max_ms", 1))}} + + +def main(): + """Own child JVMs/container and retain results even when a check fails.""" + parser = argparse.ArgumentParser() + parser.add_argument("--java", required=True) + parser.add_argument("--node-image", help="Optional Linux Java21 image; isolate Redis traffic on an owned Docker network") + parser.add_argument("--nodes", type=int, default=3) + parser.add_argument("--tenants", type=int, default=2) + parser.add_argument("--users", type=int, default=6) + parser.add_argument("--tabs", type=int, default=4) + parser.add_argument("--rounds", type=int, default=10) + parser.add_argument("--workers", type=int, default=8) + parser.add_argument("--crash-round", type=int, default=0, help="Kill the first node before this round; zero disables") + parser.add_argument("--churn-every", type=int, default=0, help="Leave/rejoin the last user's tabs every N rounds; zero disables") + parser.add_argument("--pause-redis", action="store_true", help="After the workload, freeze owned Redis and verify bounded failures/recovery") + args = parser.parse_args() + for name, lower, upper in (("nodes", 2, 8), ("tenants", 2, 8), ("users", 2, 100), + ("tabs", 2, 8), ("rounds", 1, 1000), ("workers", 1, 64)): + if not lower <= getattr(args, name) <= upper: + parser.error(f"{name} must be {lower}..{upper}") + if args.crash_round and not 2 <= args.crash_round <= args.rounds: + parser.error("crash-round must be zero or 2..rounds") + if args.churn_every and not 1 <= args.churn_every <= args.rounds: + parser.error("churn-every must be zero or 1..rounds") + repo = Path(__file__).resolve().parents[2] + output = repo / "target" / ("capacity-nodes-" + uuid.uuid4().hex[:10]) + output.mkdir(parents=True) + classpath = os.pathsep.join([str(repo / "target/test-classes"), str(repo / "target/classes"), + (repo / "target/capacity-classpath.txt").read_text().strip()]) + processes, logs = [], [] + container = "qqq-capacity-" + uuid.uuid4().hex[:12] + network = container + "-net" + node_containers = [] + network_created = False + result = {"status": "FAIL", "parameters": vars(args), "scope": "loopback HTTP, mock identities, memory records"} + try: + network_args = [] + if args.node_image: + result["node_image_id"] = subprocess.check_output( + ["docker", "image", "inspect", args.node_image, "--format", "{{.Id}}"], text=True).strip() + subprocess.run(["docker", "network", "create", network], check=True, capture_output=True) + network_created = True + network_args = ["--network", network] + dependencies = output / "dependencies" + dependencies.mkdir() + linux_classpath = ["/fixture/target/test-classes", "/fixture/target/classes"] + for index, dependency in enumerate((repo / "target/capacity-classpath.txt").read_text().strip().split(os.pathsep)): + destination = dependencies / f"{index}.jar" + shutil.copyfile(dependency, destination) + linux_classpath.append("/fixture/" + destination.relative_to(repo).as_posix()) + classpath = ":".join(linux_classpath) + subprocess.run(["docker", "run", "-d", "--name", container, "-p", "127.0.0.1::6379", + *network_args, + "redis:7.4-alpine", "redis-server", "--slowlog-log-slower-than", "1000", + "--slowlog-max-len", "128", "--latency-monitor-threshold", "1"], check=True, capture_output=True) + redis_port = int(subprocess.check_output(["docker", "port", container, "6379"], text=True).strip().rsplit(":", 1)[1]) + nodes = [port() for _ in range(args.nodes)] + if len(set(nodes)) != len(nodes): + raise RuntimeError("Port allocation collision; no nodes started") + for node in nodes: + argfile = output / f"node-{node}.args" + connection_properties = (f"-Dcollab.capacity.redisHost={container}\n-Dcollab.dev.bindAddress=0.0.0.0\n" + if args.node_image else "") + argfile.write_text(f'-Xmx384m\n-Dcollab.capacity.tenants={args.tenants}\n-Dcollab.capacity.users={args.users}\n' + + connection_properties + + '-cp\n"' + classpath.replace("\\", "/") + '"\n' + 'com.kingsrook.qbits.collaboration.devserver.CapacityPresenceDevServer\n' + + (f'8080\n6379\n' if args.node_image else f'{node}\n{redis_port}\n'), encoding="utf-8") + log = (output / f"node-{node}.log").open("w") + logs.append(log) + command = [args.java, "@" + str(argfile)] + if args.node_image: + node_container = f"{container}-node-{node}" + node_containers.append(node_container) + command = ["docker", "run", "--rm", "--name", node_container, "--network", network, + "-p", f"127.0.0.1:{node}:8080", "--mount", f"type=bind,source={repo},target=/fixture,readonly", + args.node_image, "java", "@/fixture/" + argfile.relative_to(repo).as_posix()] + processes.append(subprocess.Popen(command, cwd=repo, + stdout=log, stderr=subprocess.STDOUT, + creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0))) + deadline = time.monotonic() + 120 + for node, process in zip(nodes, processes): + while "CAPACITY_PRESENCE_READY " not in (output / f"node-{node}.log").read_text(errors="replace"): + if process.poll() is not None or time.monotonic() > deadline: + raise RuntimeError(f"Node {node} failed startup; inspect {output}") + time.sleep(.2) + actors = [] + for tenant in range(args.tenants): + for user in range(args.users): + for tab in range(args.tabs): + identity = f"load-t{tenant}-u{user}" + record = str(1 + tab % 2) + node = nodes[len(actors) % len(nodes)] + lease = request(node, identity, "/presence/leases", {"table": "person", "pk": record}, tab)["connectionId"] + actors.append((identity, record, tab, lease, user)) + renewals, snapshots = [], [] + + def check_rooms(excluded=None): + """Check every tenant/record on every surviving node, including absence.""" + for tenant in range(args.tenants): + expected_users = {f"load-t{tenant}-u{user}" for user in range(args.users) if user != excluded} + for node in nodes: + for record in ("1", "2"): + before = time.perf_counter() + viewers = request(node, f"load-t{tenant}-u0", f"/presence?table=person&pk={record}")["viewers"] + snapshots.append((time.perf_counter() - before) * 1000) + actual = {viewer["userId"] for viewer in viewers} + if actual != expected_users or len(viewers) != len(expected_users): + raise AssertionError(f"Tenant isolation/deduplication mismatch: {actual} != {expected_users}") + if sum(viewer["mode"] == "EDITING" for viewer in viewers) != sum(user != excluded for user in range(2)): + raise AssertionError("Unexpected editor count") + + def renew(item, sequence): + """Keep each tab sequential while moving its writes between nodes.""" + index, (identity, record, tab, lease, user) = item + before = time.perf_counter() + request(nodes[(index + sequence) % len(nodes)], identity, "/presence/leases/renew", + {"connectionId": lease, "table": "person", "pk": record, + "mode": "EDITING" if user < 2 else "VIEWING", "sequence": sequence}, tab) + return (time.perf_counter() - before) * 1000 + + started = time.perf_counter() + with ThreadPoolExecutor(max_workers=args.workers) as workers: + for sequence in range(1, args.rounds + 1): + result["current_round"] = sequence + if sequence == args.crash_round: + crashed = nodes.pop(0) + if args.node_image: + subprocess.run(["docker", "kill", node_containers[0]], check=True, capture_output=True) + else: + processes[0].kill() + processes[0].wait(timeout=10) + result["crash"] = {"round": sequence, "http_port": crashed, + "surviving_nodes": len(nodes), "process_exit": processes[0].returncode} + renewals.extend(workers.map(lambda item: renew(item, sequence), enumerate(actors))) + check_rooms() + if args.churn_every and sequence % args.churn_every == 0: + departing = [(index, actor) for index, actor in enumerate(actors) if actor[4] == args.users - 1] + for index, (identity, record, tab, lease, user) in departing: + request(nodes[index % len(nodes)], identity, "/presence", + {"connectionId": lease, "table": "person", "pk": record, "sequence": sequence + 1}, tab, method="DELETE") + response = request(nodes[(index + 1) % len(nodes)], identity, "/presence/leases/renew", + {"connectionId": lease, "table": "person", "pk": record, + "mode": "VIEWING", "sequence": sequence + 2}, tab, expected=409) + if response.get("code") != "PRESENCE_LEASE_EXPIRED": + raise AssertionError("Departed lease was not rejected as expired") + check_rooms(excluded=args.users - 1) + for index, (identity, record, tab, lease, user) in departing: + replacement = request(nodes[(index + 1) % len(nodes)], identity, "/presence/leases", + {"table": "person", "pk": record}, tab)["connectionId"] + if replacement == lease: + raise AssertionError("Rejoin reused departed authority") + actors[index] = (identity, record, tab, replacement, user) + renew((index, actors[index]), sequence) + check_rooms() + result["churn_cycles"] = result.get("churn_cycles", 0) + 1 + result["departed_leases_rejected"] = result.get("departed_leases_rejected", 0) + len(departing) + result["completed_rounds"] = sequence + if args.pause_redis: + failures = [] + + def stalled_renew(item): + """A frozen authority must fail explicitly, never acknowledge a write.""" + index, (identity, record, tab, lease, user) = item + before = time.perf_counter() + response = request(nodes[index % len(nodes)], identity, "/presence/leases/renew", + {"connectionId": lease, "table": "person", "pk": record, + "mode": "EDITING" if user < 2 else "VIEWING", "sequence": args.rounds + 1}, + tab, expected=503) + if "viewers" in response or "temporarily unavailable" not in response.get("message", ""): + raise AssertionError("Stalled authority did not return the expected unavailable response") + duration = (time.perf_counter() - before) * 1000 + if duration > 8000: + raise AssertionError("Unavailable response exceeded the eight-second fixture bound") + return duration + + subprocess.run(["docker", "pause", container], check=True, capture_output=True) + try: + with ThreadPoolExecutor(max_workers=64) as burst: + failures.extend(burst.map(stalled_renew, enumerate(actors))) + for node in nodes: + response = request(node, "load-t0-u0", "/presence?table=person&pk=1", expected=503) + if "viewers" in response: + raise AssertionError("Stalled authority was presented as a room snapshot") + finally: + subprocess.run(["docker", "unpause", container], check=True, capture_output=True) + subprocess.run(["docker", "exec", container, "redis-cli", "PING"], check=True, capture_output=True, timeout=10) + for item in enumerate(actors): + renew(item, args.rounds + 2) + check_rooms() + result["redis_stall"] = {"rejected_renewals": len(failures), "failed_snapshot_nodes": len(nodes), + "response_times": distribution(failures), "same_leases_recovered": True} + elapsed = time.perf_counter() - started + result.update(status="PASS", seconds=elapsed, renewals=distribution(renewals), + snapshots=distribution(snapshots), renewal_throughput=len(renewals) / elapsed, + node_pids=[process.pid for process in processes], + pid_kind="docker-client" if args.node_image else "JVM", node_containers=node_containers) + except Exception as error: + result["error"] = str(error) + raise + finally: + failure_active = sys.exc_info()[0] is not None + try: + slow = json.loads(subprocess.check_output( + ["docker", "exec", container, "redis-cli", "--json", "SLOWLOG", "GET", "128"], text=True, timeout=10)) + result["redis_slow_commands"] = [{"unix_time": entry[1], "duration_us": entry[2], "command": entry[3][0]} + for entry in slow] + result["redis_latency_events"] = json.loads(subprocess.check_output( + ["docker", "exec", container, "redis-cli", "--json", "LATENCY", "LATEST"], text=True, timeout=10)) + for section in ("cpu", "memory", "commandstats", "stats"): + value = subprocess.check_output(["docker", "exec", container, "redis-cli", "INFO", section], text=True, timeout=10) + (output / f"redis-{section}.txt").write_text(value) + parsed = dict(line.split(":", 1) for line in value.splitlines() if ":" in line and not line.startswith("#")) + if section == "memory": + result["redis_memory"] = {key: int(parsed[key]) for key in + ("used_memory", "used_memory_peak", "used_memory_rss")} + if section == "cpu": + result["redis_cpu_seconds_since_start"] = {key: float(parsed[key]) for key in + ("used_cpu_sys", "used_cpu_user")} + except (subprocess.SubprocessError, ValueError, KeyError) as error: + result["redis_diagnostic_error"] = type(error).__name__ + node_cleanup = True + for node_container in node_containers: + removed = subprocess.run(["docker", "rm", "-f", node_container], capture_output=True) + if removed.returncode != 0: + remaining = subprocess.run(["docker", "ps", "-aq", "--filter", f"name=^{node_container}$"], capture_output=True, text=True) + node_cleanup = node_cleanup and remaining.returncode == 0 and not remaining.stdout.strip() + for process in processes: + if process.poll() is None: + process.terminate() + for process in processes: + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + process.kill() + process.wait(timeout=10) + for log in logs: + log.close() + resources = [] + for node_log in sorted(output.glob("node-*.log")): + samples = [json.loads(line.split("CAPACITY_METRIC ", 1)[1]) + for line in node_log.read_text(errors="replace").splitlines() + if line.startswith("CAPACITY_METRIC ")] + if samples: + first, last = samples[0], samples[-1] + wall_ms = last["uptimeMillis"] - first["uptimeMillis"] + cpu_ns = last["cpuNanos"] - first["cpuNanos"] + resources.append({"node_log": node_log.name, "samples": len(samples), + "observed_wall_ms": wall_ms, + "cpu_seconds": cpu_ns / 1e9 if first["cpuNanos"] >= 0 else None, + "cpu_core_equivalents": cpu_ns / (wall_ms * 1e6) if wall_ms > 0 and first["cpuNanos"] >= 0 else None, + **{key + "_sampled_max": max(sample[key] for sample in samples) + for key in ("heapUsedBytes", "heapCommittedBytes", "poolActive", "poolIdle", "poolWaiting", "poolMaximum")}, + "residentBytes_sampled_max": max(sample["residentBytes"] for sample in samples) + if all(sample.get("residentBytes", -1) >= 0 for sample in samples) else None}) + if any(sample["poolActive"] > sample["poolMaximum"] for sample in samples): + result.update(status="FAIL", error="Sampled pool active count exceeded configured maximum") + result["resources"] = resources + if result["status"] == "PASS" and args.node_image and any(node["residentBytes_sampled_max"] is None for node in resources): + result.update(status="FAIL", error="Linux resident-memory samples missing") + if result["status"] == "PASS" and any(key not in result for key in ("redis_memory", "redis_cpu_seconds_since_start")): + result.update(status="FAIL", error="Redis memory/CPU diagnostics missing") + if result["status"] == "PASS" and len(resources) != args.nodes: + result.update(status="FAIL", error="Missing resource samples for a node") + if result["status"] == "PASS" and args.pause_redis and not any(node["poolWaiting_sampled_max"] > 0 for node in resources): + result.update(status="FAIL", error="Redis stall did not demonstrate pool waiters") + cleanup = subprocess.run(["docker", "rm", "-f", container], capture_output=True) + network_cleanup = True + if network_created: + network_cleanup = subprocess.run(["docker", "network", "rm", network], capture_output=True).returncode == 0 + result["cleanup"] = cleanup.returncode == 0 and node_cleanup and network_cleanup and all(process.poll() is not None for process in processes) + (output / "result.json").write_text(json.dumps(result, indent=2)) + print(json.dumps(result, indent=2)) + print(output) + if result["status"] == "PASS" and not result["cleanup"]: + raise RuntimeError("Workload passed but owned resource cleanup failed") + if result["status"] == "FAIL" and not failure_active: + raise RuntimeError(result.get("error", "Capacity resource validation failed")) + + +if __name__ == "__main__": + main() diff --git a/docs/performance/presence_router.mjs b/docs/performance/presence_router.mjs new file mode 100644 index 0000000..e840b0b --- /dev/null +++ b/docs/performance/presence_router.mjs @@ -0,0 +1,42 @@ +// Disposable loopback fixture only; never deploy this fault-injection router. +import http from 'node:http'; + +const [first, second, listen] = process.argv.slice(2).map(Number); +if (![first, second, listen].every(port => Number.isInteger(port) && port > 0 && port < 65536)) { + throw new Error('Expected first-node, second-node and router ports'); +} +const firstStreams = new Set(); +const server = http.createServer((request, response) => { + if (request.method === 'POST' && request.url === '/__acceptance/disconnect-first') { + const disconnected = firstStreams.size; + for (const stream of firstStreams) stream.destroy(); + response.writeHead(200, { 'content-type': 'application/json' }); + response.end(JSON.stringify({ disconnected })); + return; + } + const isSecond = /(?:^|;\s*)presence_node=second(?:;|$)/.test(request.headers.cookie ?? ''); + const port = isSecond ? second : first; + const upstream = http.request({ + hostname: '127.0.0.1', port, path: request.url, method: request.method, + headers: { ...request.headers, host: `127.0.0.1:${port}` }, + }, incoming => { + response.writeHead(incoming.statusCode, { + ...incoming.headers, 'x-acceptance-node': isSecond ? 'second' : 'first', + }); + if (!isSecond && incoming.headers['content-type']?.includes('text/event-stream')) { + firstStreams.add(response); + } + incoming.pipe(response); + }); + upstream.on('error', () => { + if (!response.headersSent) response.writeHead(502); + response.end('Acceptance node unavailable'); + }); + response.on('close', () => { + firstStreams.delete(response); + upstream.destroy(); + }); + request.pipe(upstream); +}); +server.listen(listen, '127.0.0.1', () => console.log(`PRESENCE_ROUTER_READY ${listen}`)); +for (const signal of ['SIGINT', 'SIGTERM']) process.on(signal, () => server.close()); diff --git a/docs/performance/results/browser-focus-25.json b/docs/performance/results/browser-focus-25.json new file mode 100644 index 0000000..cf81846 --- /dev/null +++ b/docs/performance/results/browser-focus-25.json @@ -0,0 +1,285 @@ +{ + "status": "PASS", + "samples": [ + { + "field": "lastName", + "propagation_ms": 36.092099999999846 + }, + { + "field": "firstName", + "propagation_ms": 32.51870000000008 + }, + { + "field": "lastName", + "propagation_ms": 31.758600000000115 + }, + { + "field": "firstName", + "propagation_ms": 31.090299999999843 + }, + { + "field": "lastName", + "propagation_ms": 31.64980000000014 + }, + { + "field": "firstName", + "propagation_ms": 31.648199999999633 + }, + { + "field": "lastName", + "propagation_ms": 31.395500000000084 + }, + { + "field": "firstName", + "propagation_ms": 32.36750000000029 + }, + { + "field": "lastName", + "propagation_ms": 31.601799999999912 + }, + { + "field": "firstName", + "propagation_ms": 31.90599999999995 + }, + { + "field": "lastName", + "propagation_ms": 31.44150000000036 + }, + { + "field": "firstName", + "propagation_ms": 31.186699999999746 + }, + { + "field": "lastName", + "propagation_ms": 31.115800000000036 + }, + { + "field": "firstName", + "propagation_ms": 32.539400000000114 + }, + { + "field": "lastName", + "propagation_ms": 31.188000000000102 + }, + { + "field": "firstName", + "propagation_ms": 33.021200000000135 + }, + { + "field": "lastName", + "propagation_ms": 33.07759999999962 + }, + { + "field": "firstName", + "propagation_ms": 32.7176000000004 + }, + { + "field": "lastName", + "propagation_ms": 31.457300000000032 + }, + { + "field": "firstName", + "propagation_ms": 32.920900000000074 + }, + { + "field": "lastName", + "propagation_ms": 31.89130000000023 + }, + { + "field": "firstName", + "propagation_ms": 30.847200000000157 + }, + { + "field": "lastName", + "propagation_ms": 30.158300000000054 + }, + { + "field": "firstName", + "propagation_ms": 31.48239999999987 + }, + { + "field": "lastName", + "propagation_ms": 31.337600000000293 + } + ], + "streams": [ + { + "user": "alice", + "status": 200, + "type": "text/event-stream;charset=utf-8" + }, + { + "user": "admin", + "status": 200, + "type": "text/event-stream;charset=utf-8" + } + ], + "requests": [ + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + } + ], + "errors": [], + "scope": "Chromium focus-to-remote-DOM observation, including automation and UI debounce; no concurrent load", + "percentiles_ms": { + "p50": 31.648199999999633, + "p95": 33.07759999999962, + "p99": 36.092099999999846, + "max": 36.092099999999846 + }, + "frontend_jar_sha256": "c9fa2eb2381a47c8104f82f4e940d2bfa8ba67e2341298dde0de4e21ad3ff1ab", + "fixture_cleanup": true, + "backend_source": "Published core cc3981bb3/collaboration ecff347/Redis7fe5492 snapshots; two independent JVMs, mock identities and memory records." +} diff --git a/docs/performance/results/browser-load-reconnect.json b/docs/performance/results/browser-load-reconnect.json new file mode 100644 index 0000000..b2eb0cb --- /dev/null +++ b/docs/performance/results/browser-load-reconnect.json @@ -0,0 +1,357 @@ +{ + "status": "PASS", + "samples": [ + { + "field": "lastName", + "propagation_ms": 36.560600000000704 + }, + { + "field": "firstName", + "propagation_ms": 33.44050000000061 + }, + { + "field": "lastName", + "propagation_ms": 32.50890000000072 + }, + { + "field": "firstName", + "propagation_ms": 785.4951000000001 + }, + { + "field": "lastName", + "propagation_ms": 85.39069999999992 + }, + { + "field": "firstName", + "propagation_ms": 30.589599999999336 + }, + { + "field": "lastName", + "propagation_ms": 32.756099999998696 + }, + { + "field": "firstName", + "propagation_ms": 38.17399999999907 + }, + { + "field": "lastName", + "propagation_ms": 30.243099999999686 + }, + { + "field": "firstName", + "propagation_ms": 29.600799999992887 + }, + { + "field": "lastName", + "propagation_ms": 29.57220000000234 + }, + { + "field": "firstName", + "propagation_ms": 29.197800000001735 + }, + { + "field": "lastName", + "propagation_ms": 33.05130000000645 + }, + { + "field": "firstName", + "propagation_ms": 32.38390000000072 + }, + { + "field": "lastName", + "propagation_ms": 31.52850000000035 + }, + { + "field": "firstName", + "propagation_ms": 34.66169999999693 + }, + { + "field": "lastName", + "propagation_ms": 31.258300000001327 + }, + { + "field": "firstName", + "propagation_ms": 44.19029999999475 + }, + { + "field": "lastName", + "propagation_ms": 36.56189999999333 + }, + { + "field": "firstName", + "propagation_ms": 30.240600000004633 + }, + { + "field": "lastName", + "propagation_ms": 29.79899999999907 + }, + { + "field": "firstName", + "propagation_ms": 29.621500000001106 + }, + { + "field": "lastName", + "propagation_ms": 31.66189999999915 + }, + { + "field": "firstName", + "propagation_ms": 29.198399999993853 + }, + { + "field": "lastName", + "propagation_ms": 32.931200000006356 + } + ], + "streams": [ + { + "user": "alice", + "status": 200, + "type": "text/event-stream;charset=utf-8" + }, + { + "user": "admin", + "status": 200, + "type": "text/event-stream;charset=utf-8" + }, + { + "user": "alice", + "status": 200, + "type": "text/event-stream;charset=utf-8" + } + ], + "requests": [ + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + } + ], + "errors": [], + "scope": "Chromium focus-to-remote-DOM and reconnect under 16 concurrent HTTP lease workers; loopback, four background identities", + "load": { + "workers": 16, + "completed": 9046, + "errors": [], + "nodes": { + "first": 4540, + "second": 4522 + }, + "elapsed_ms": 69728.3579 + }, + "reconnect": { + "status": "PASS", + "missed_changes": 3, + "disconnected_streams": 1, + "stream_failure": "net::ERR_INCOMPLETE_CHUNKED_ENCODING", + "recovery_ms": 1855.8868999999977 + }, + "percentiles_ms": { + "p50": 32.38390000000072, + "p95": 85.39069999999992, + "p99": 785.4951000000001, + "max": 785.4951000000001 + }, + "frontend_jar_sha256": "c9fa2eb2381a47c8104f82f4e940d2bfa8ba67e2341298dde0de4e21ad3ff1ab", + "fixture_cleanup": true, + "backend_source": "Published core cc3981bb3/collaboration ecff347/Redis7fe5492 snapshots; two independent JVMs, mock identities and memory records.", + "fixture_checks": [ + "browser-six-presence", + "browser-redis-admin", + "browser-presence-latency" + ] +} diff --git a/docs/performance/results/browser-reconnect.json b/docs/performance/results/browser-reconnect.json new file mode 100644 index 0000000..16d4616 --- /dev/null +++ b/docs/performance/results/browser-reconnect.json @@ -0,0 +1,317 @@ +{ + "status": "PASS", + "samples": [ + { + "field": "lastName", + "propagation_ms": 37.25770000000011 + }, + { + "field": "firstName", + "propagation_ms": 32.14499999999998 + }, + { + "field": "lastName", + "propagation_ms": 32.40950000000021 + }, + { + "field": "firstName", + "propagation_ms": 33.52500000000009 + }, + { + "field": "lastName", + "propagation_ms": 31.228200000000015 + }, + { + "field": "firstName", + "propagation_ms": 30.430999999999585 + }, + { + "field": "lastName", + "propagation_ms": 31.493199999999888 + }, + { + "field": "firstName", + "propagation_ms": 33.14539999999988 + }, + { + "field": "lastName", + "propagation_ms": 32.11259999999993 + }, + { + "field": "firstName", + "propagation_ms": 33.23129999999992 + }, + { + "field": "lastName", + "propagation_ms": 31.06800000000021 + }, + { + "field": "firstName", + "propagation_ms": 30.109199999999873 + }, + { + "field": "lastName", + "propagation_ms": 31.534299999999803 + }, + { + "field": "firstName", + "propagation_ms": 33.770800000000236 + }, + { + "field": "lastName", + "propagation_ms": 31.654199999999946 + }, + { + "field": "firstName", + "propagation_ms": 33.31700000000001 + }, + { + "field": "lastName", + "propagation_ms": 33.108600000000024 + }, + { + "field": "firstName", + "propagation_ms": 32.27370000000019 + }, + { + "field": "lastName", + "propagation_ms": 31.452499999999873 + }, + { + "field": "firstName", + "propagation_ms": 31.86220000000003 + }, + { + "field": "lastName", + "propagation_ms": 31.734399999999823 + }, + { + "field": "firstName", + "propagation_ms": 32.04489999999987 + }, + { + "field": "lastName", + "propagation_ms": 31.051899999999932 + }, + { + "field": "firstName", + "propagation_ms": 30.876300000000356 + }, + { + "field": "lastName", + "propagation_ms": 31.56970000000001 + } + ], + "streams": [ + { + "user": "alice", + "status": 200, + "type": "text/event-stream;charset=utf-8" + }, + { + "user": "admin", + "status": 200, + "type": "text/event-stream;charset=utf-8" + }, + { + "user": "alice", + "status": 200, + "type": "text/event-stream;charset=utf-8" + } + ], + "requests": [ + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + } + ], + "errors": [], + "scope": "Chromium focus-to-remote-DOM observation, including automation and UI debounce; no concurrent load", + "reconnect": { + "status": "PASS", + "missed_changes": 3, + "disconnected_streams": 1, + "stream_failure": "net::ERR_INCOMPLETE_CHUNKED_ENCODING", + "recovery_ms": 1858.2972000000004 + }, + "percentiles_ms": { + "p50": 31.86220000000003, + "p95": 33.770800000000236, + "p99": 37.25770000000011, + "max": 37.25770000000011 + }, + "frontend_jar_sha256": "c9fa2eb2381a47c8104f82f4e940d2bfa8ba67e2341298dde0de4e21ad3ff1ab", + "fixture_cleanup": true, + "backend_source": "Published core cc3981bb3/collaboration ecff347/Redis7fe5492 snapshots; two independent JVMs, mock identities and memory records.", + "fixture_checks": [ + "browser-six-presence", + "browser-redis-admin", + "browser-presence-latency" + ] +} diff --git a/docs/performance/results/browser-subscription-affinity.json b/docs/performance/results/browser-subscription-affinity.json new file mode 100644 index 0000000..69840c8 --- /dev/null +++ b/docs/performance/results/browser-subscription-affinity.json @@ -0,0 +1,14 @@ +{ + "status": "PASS", + "checks": [ + { "node": "first", "status": 200 }, + { "node": "second", "status": 404 }, + { "node": "first", "status": 200 } + ], + "errors": [], + "fixture_cleanup": true, + "fixture_checks": ["browser-six-presence", "browser-redis-admin", "browser-subscription-affinity"], + "frontend_jar_sha256": "c9fa2eb2381a47c8104f82f4e940d2bfa8ba67e2341298dde0de4e21ad3ff1ab", + "backend_source": "Published core cc3981bb3/collaboration ecff347/Redis7fe5492 snapshots; two independent JVMs, mock identities and memory records.", + "scope": "Same browser session and live SSE connection ID; change only test-router affinity. Not deployed Traefik qualification." +} diff --git a/docs/performance/results/linux-100-users-resources.json b/docs/performance/results/linux-100-users-resources.json new file mode 100644 index 0000000..248f02a --- /dev/null +++ b/docs/performance/results/linux-100-users-resources.json @@ -0,0 +1,181 @@ +{ + "status": "PASS", + "parameters": { + "java": "/path/to/java21", + "node_image": "eclipse-temurin@sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99", + "nodes": 3, + "tenants": 2, + "users": 50, + "tabs": 4, + "rounds": 200, + "workers": 16, + "crash_round": 0, + "churn_every": 0, + "pause_redis": false + }, + "scope": "loopback HTTP, mock identities, memory records", + "node_image_id": "sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99", + "current_round": 200, + "completed_rounds": 200, + "seconds": 102.56801009993069, + "renewals": { + "samples": 80000, + "p50_ms": 11.325599974952638, + "p95_ms": 21.764500066637993, + "p99_ms": 25.881099980324507, + "max_ms": 1007.4040000326931 + }, + "snapshots": { + "samples": 2400, + "p50_ms": 15.130399959161878, + "p95_ms": 16.613700077868998, + "p99_ms": 19.423600053414702, + "max_ms": 28.864299994893372 + }, + "renewal_throughput": 779.9702843221492, + "node_pids": [ + 105696, + 4828, + 117588 + ], + "pid_kind": "docker-client", + "node_containers": [ + "qqq-capacity-f58f3ddcf5f3-node-59675", + "qqq-capacity-f58f3ddcf5f3-node-59676", + "qqq-capacity-f58f3ddcf5f3-node-59677" + ], + "redis_slow_commands": [ + { + "unix_time": 1793033347, + "duration_us": 1432, + "command": "EVALSHA" + }, + { + "unix_time": 1793033347, + "duration_us": 1612, + "command": "EVALSHA" + }, + { + "unix_time": 1793033340, + "duration_us": 1877, + "command": "EVALSHA" + }, + { + "unix_time": 1793033340, + "duration_us": 1313, + "command": "ZRANGE" + }, + { + "unix_time": 1793033336, + "duration_us": 1316, + "command": "EVALSHA" + }, + { + "unix_time": 1793033305, + "duration_us": 1459, + "command": "EVALSHA" + }, + { + "unix_time": 1793033300, + "duration_us": 1767, + "command": "EVALSHA" + }, + { + "unix_time": 1793033290, + "duration_us": 1180, + "command": "EVALSHA" + }, + { + "unix_time": 1793033283, + "duration_us": 1421, + "command": "EVALSHA" + }, + { + "unix_time": 1793033260, + "duration_us": 1291, + "command": "EVALSHA" + }, + { + "unix_time": 1793033260, + "duration_us": 1551, + "command": "EVALSHA" + }, + { + "unix_time": 1793033260, + "duration_us": 1083, + "command": "ZRANGE" + }, + { + "unix_time": 1793033255, + "duration_us": 1039, + "command": "EVALSHA" + }, + { + "unix_time": 1793033254, + "duration_us": 1215, + "command": "EVALSHA" + } + ], + "redis_latency_events": [ + [ + "command", + 1793033347, + 1, + 1 + ] + ], + "redis_cpu_seconds_since_start": { + "used_cpu_sys": 23.737706, + "used_cpu_user": 35.526214 + }, + "redis_memory": { + "used_memory": 2415952, + "used_memory_peak": 2579896, + "used_memory_rss": 8777728 + }, + "resources": [ + { + "node_log": "node-59675.log", + "samples": 1112, + "observed_wall_ms": 111099, + "cpu_seconds": 203.14, + "cpu_core_equivalents": 1.828459302063925, + "heapUsedBytes_sampled_max": 181098896, + "heapCommittedBytes_sampled_max": 211812352, + "poolActive_sampled_max": 7, + "poolIdle_sampled_max": 7, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8, + "residentBytes_sampled_max": 662786048 + }, + { + "node_log": "node-59676.log", + "samples": 1145, + "observed_wall_ms": 114399, + "cpu_seconds": 200.44, + "cpu_core_equivalents": 1.7521132177728826, + "heapUsedBytes_sampled_max": 185167416, + "heapCommittedBytes_sampled_max": 219152384, + "poolActive_sampled_max": 7, + "poolIdle_sampled_max": 7, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8, + "residentBytes_sampled_max": 662188032 + }, + { + "node_log": "node-59677.log", + "samples": 1159, + "observed_wall_ms": 115800, + "cpu_seconds": 200.16, + "cpu_core_equivalents": 1.728497409326425, + "heapUsedBytes_sampled_max": 179378632, + "heapCommittedBytes_sampled_max": 210763776, + "poolActive_sampled_max": 6, + "poolIdle_sampled_max": 7, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8, + "residentBytes_sampled_max": 703479808 + } + ], + "cleanup": true +} diff --git a/docs/performance/results/linux-1000-rounds.json b/docs/performance/results/linux-1000-rounds.json new file mode 100644 index 0000000..995a222 --- /dev/null +++ b/docs/performance/results/linux-1000-rounds.json @@ -0,0 +1,101 @@ +{ + "status": "PASS", + "parameters": { + "java": "java (Java 21)", + "node_image": "eclipse-temurin:21-jre", + "nodes": 3, + "tenants": 2, + "users": 12, + "tabs": 4, + "rounds": 1000, + "workers": 8 + }, + "scope": "loopback HTTP, mock identities, memory records", + "current_round": 1000, + "completed_rounds": 1000, + "seconds": 250.81072249996942, + "renewals": { + "samples": 96000, + "p50_ms": 6.353199947625399, + "p95_ms": 15.330899972468615, + "p99_ms": 17.760399961844087, + "max_ms": 326.8311999272555 + }, + "snapshots": { + "samples": 12000, + "p50_ms": 13.105699908919632, + "p95_ms": 14.261199976317585, + "p99_ms": 16.519399941898882, + "max_ms": 915.3468000004068 + }, + "renewal_throughput": 382.7587554595546, + "node_pids": [ + 64000, + 40240, + 115400 + ], + "pid_kind": "docker-client", + "node_containers": [ + "qqq-capacity-0a8b7956499d-node-64951", + "qqq-capacity-0a8b7956499d-node-64952", + "qqq-capacity-0a8b7956499d-node-64953" + ], + "redis_slow_commands": [ + { + "unix_time": 1793030517, + "duration_us": 1117, + "command": "EVALSHA" + } + ], + "redis_latency_events": [ + [ + "command", + 1793030517, + 1, + 1 + ] + ], + "resources": [ + { + "node_log": "node-64951.log", + "samples": 2580, + "observed_wall_ms": 257899, + "cpu_seconds": 165.55, + "cpu_core_equivalents": 0.6419179601316795, + "heapUsedBytes_sampled_max": 166667368, + "heapCommittedBytes_sampled_max": 188743680, + "poolActive_sampled_max": 4, + "poolIdle_sampled_max": 5, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-64952.log", + "samples": 2569, + "observed_wall_ms": 256799, + "cpu_seconds": 165.85, + "cpu_core_equivalents": 0.6458358482704372, + "heapUsedBytes_sampled_max": 169216944, + "heapCommittedBytes_sampled_max": 192937984, + "poolActive_sampled_max": 5, + "poolIdle_sampled_max": 4, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-64953.log", + "samples": 2608, + "observed_wall_ms": 260700, + "cpu_seconds": 164.72, + "cpu_core_equivalents": 0.631837360951285, + "heapUsedBytes_sampled_max": 166319008, + "heapCommittedBytes_sampled_max": 188743680, + "poolActive_sampled_max": 4, + "poolIdle_sampled_max": 5, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + } + ], + "cleanup": true, + "receipt_note": "Java executable path normalized; measurements unchanged. Disposable fixture identities only." +} diff --git a/docs/performance/results/linux-node-crash-churn.json b/docs/performance/results/linux-node-crash-churn.json new file mode 100644 index 0000000..9e2db3e --- /dev/null +++ b/docs/performance/results/linux-node-crash-churn.json @@ -0,0 +1,117 @@ +{ + "status": "PASS", + "parameters": { + "java": "java (Java 21)", + "node_image": "eclipse-temurin@sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99", + "nodes": 3, + "tenants": 2, + "users": 12, + "tabs": 4, + "rounds": 100, + "workers": 8, + "crash_round": 50, + "churn_every": 20 + }, + "scope": "loopback HTTP, mock identities, memory records", + "node_image_id": "sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99", + "current_round": 100, + "completed_rounds": 100, + "churn_cycles": 5, + "departed_leases_rejected": 40, + "crash": { + "round": 50, + "http_port": 52824, + "surviving_nodes": 2, + "process_exit": 137 + }, + "seconds": 28.090077199973166, + "renewals": { + "samples": 9600, + "p50_ms": 7.532799965701997, + "p95_ms": 17.53149996511638, + "p99_ms": 29.359900043345988, + "max_ms": 88.13189994543791 + }, + "snapshots": { + "samples": 1092, + "p50_ms": 13.582199928350747, + "p95_ms": 15.386099927127361, + "p99_ms": 18.502700026147068, + "max_ms": 22.144700051285326 + }, + "renewal_throughput": 341.75769371004685, + "node_pids": [ + 22844, + 66652, + 61920 + ], + "pid_kind": "docker-client", + "node_containers": [ + "qqq-capacity-bb9ba87919d2-node-52824", + "qqq-capacity-bb9ba87919d2-node-52825", + "qqq-capacity-bb9ba87919d2-node-52827" + ], + "redis_slow_commands": [ + { + "unix_time": 1793031681, + "duration_us": 1206, + "command": "EVALSHA" + }, + { + "unix_time": 1793031681, + "duration_us": 1147, + "command": "EVALSHA" + } + ], + "redis_latency_events": [ + [ + "command", + 1793031681, + 1, + 1 + ] + ], + "resources": [ + { + "node_log": "node-52824.log", + "samples": 212, + "observed_wall_ms": 21099, + "cpu_seconds": 29.48, + "cpu_core_equivalents": 1.3972226171856486, + "heapUsedBytes_sampled_max": 163641024, + "heapCommittedBytes_sampled_max": 188743680, + "poolActive_sampled_max": 3, + "poolIdle_sampled_max": 4, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-52825.log", + "samples": 354, + "observed_wall_ms": 35299, + "cpu_seconds": 43.09, + "cpu_core_equivalents": 1.22071446783195, + "heapUsedBytes_sampled_max": 168178592, + "heapCommittedBytes_sampled_max": 192937984, + "poolActive_sampled_max": 5, + "poolIdle_sampled_max": 5, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-52827.log", + "samples": 340, + "observed_wall_ms": 33900, + "cpu_seconds": 42.42, + "cpu_core_equivalents": 1.2513274336283187, + "heapUsedBytes_sampled_max": 166291160, + "heapCommittedBytes_sampled_max": 188743680, + "poolActive_sampled_max": 5, + "poolIdle_sampled_max": 5, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + } + ], + "cleanup": true, + "receipt_note": "Java executable path normalized; measurements unchanged. Disposable fixture identities only." +} diff --git a/docs/performance/results/linux-node-crash.json b/docs/performance/results/linux-node-crash.json new file mode 100644 index 0000000..bb87b25 --- /dev/null +++ b/docs/performance/results/linux-node-crash.json @@ -0,0 +1,96 @@ +{ + "status": "PASS", + "parameters": { + "java": "java (Java 21)", + "node_image": "eclipse-temurin@sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99", + "nodes": 3, + "tenants": 2, + "users": 12, + "tabs": 4, + "rounds": 100, + "workers": 8, + "crash_round": 50 + }, + "scope": "loopback HTTP, mock identities, memory records", + "node_image_id": "sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99", + "current_round": 100, + "completed_rounds": 100, + "crash": { + "round": 50, + "http_port": 54039, + "surviving_nodes": 2, + "process_exit": 137 + }, + "seconds": 25.147547999978997, + "renewals": { + "samples": 9600, + "p50_ms": 7.125200005248189, + "p95_ms": 17.407900071702898, + "p99_ms": 27.217599912546575, + "max_ms": 74.53640003222972 + }, + "snapshots": { + "samples": 996, + "p50_ms": 13.538200058974326, + "p95_ms": 15.59239998459816, + "p99_ms": 20.58999997097999, + "max_ms": 315.0679999962449 + }, + "renewal_throughput": 381.7469599822622, + "node_pids": [ + 85832, + 90280, + 67764 + ], + "pid_kind": "docker-client", + "node_containers": [ + "qqq-capacity-e599dca8e163-node-54039", + "qqq-capacity-e599dca8e163-node-54040", + "qqq-capacity-e599dca8e163-node-54041" + ], + "redis_slow_commands": [], + "redis_latency_events": [], + "resources": [ + { + "node_log": "node-54039.log", + "samples": 198, + "observed_wall_ms": 19699, + "cpu_seconds": 28.58, + "cpu_core_equivalents": 1.4508350677699375, + "heapUsedBytes_sampled_max": 163478640, + "heapCommittedBytes_sampled_max": 188743680, + "poolActive_sampled_max": 3, + "poolIdle_sampled_max": 4, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-54040.log", + "samples": 300, + "observed_wall_ms": 29900, + "cpu_seconds": 41.85, + "cpu_core_equivalents": 1.399665551839465, + "heapUsedBytes_sampled_max": 165177008, + "heapCommittedBytes_sampled_max": 188743680, + "poolActive_sampled_max": 4, + "poolIdle_sampled_max": 5, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-54041.log", + "samples": 331, + "observed_wall_ms": 33000, + "cpu_seconds": 41.52, + "cpu_core_equivalents": 1.2581818181818183, + "heapUsedBytes_sampled_max": 168546936, + "heapCommittedBytes_sampled_max": 192937984, + "poolActive_sampled_max": 4, + "poolIdle_sampled_max": 5, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + } + ], + "cleanup": true, + "receipt_note": "Java executable path normalized; measurements unchanged. Disposable fixture identities only." +} diff --git a/docs/performance/results/linux-redis-stall.json b/docs/performance/results/linux-redis-stall.json new file mode 100644 index 0000000..652b748 --- /dev/null +++ b/docs/performance/results/linux-redis-stall.json @@ -0,0 +1,117 @@ +{ + "status": "PASS", + "parameters": { + "java": "java (Java 21)", + "node_image": "eclipse-temurin@sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99", + "nodes": 3, + "tenants": 2, + "users": 12, + "tabs": 4, + "rounds": 30, + "workers": 8, + "crash_round": 0, + "churn_every": 0, + "pause_redis": true + }, + "scope": "loopback HTTP, mock identities, memory records", + "node_image_id": "sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99", + "current_round": 30, + "completed_rounds": 30, + "redis_stall": { + "rejected_renewals": 96, + "failed_snapshot_nodes": 3, + "response_times": { + "samples": 96, + "p50_ms": 1013.8123999349773, + "p95_ms": 4014.4751999760047, + "p99_ms": 4015.4822999611497, + "max_ms": 4015.4822999611497 + }, + "same_leases_recovered": true + }, + "seconds": 20.33888840000145, + "renewals": { + "samples": 2880, + "p50_ms": 8.835799992084503, + "p95_ms": 23.678299970924854, + "p99_ms": 35.42319999542087, + "max_ms": 102.84149996004999 + }, + "snapshots": { + "samples": 372, + "p50_ms": 13.98350007366389, + "p95_ms": 16.946100047789514, + "p99_ms": 20.020700059831142, + "max_ms": 20.943500101566315 + }, + "renewal_throughput": 141.6006589622565, + "node_pids": [ + 88080, + 78036, + 121352 + ], + "pid_kind": "docker-client", + "node_containers": [ + "qqq-capacity-68ad627e7be5-node-58312", + "qqq-capacity-68ad627e7be5-node-58314", + "qqq-capacity-68ad627e7be5-node-58315" + ], + "redis_slow_commands": [ + { + "unix_time": 1793032016, + "duration_us": 2198, + "command": "EVALSHA" + } + ], + "redis_latency_events": [ + [ + "command", + 1793032016, + 2, + 2 + ] + ], + "resources": [ + { + "node_log": "node-58312.log", + "samples": 280, + "observed_wall_ms": 27899, + "cpu_seconds": 23.96, + "cpu_core_equivalents": 0.8588121438044375, + "heapUsedBytes_sampled_max": 164221400, + "heapCommittedBytes_sampled_max": 188743680, + "poolActive_sampled_max": 6, + "poolIdle_sampled_max": 5, + "poolWaiting_sampled_max": 15, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-58314.log", + "samples": 289, + "observed_wall_ms": 28800, + "cpu_seconds": 23.2, + "cpu_core_equivalents": 0.8055555555555556, + "heapUsedBytes_sampled_max": 162849016, + "heapCommittedBytes_sampled_max": 192937984, + "poolActive_sampled_max": 4, + "poolIdle_sampled_max": 3, + "poolWaiting_sampled_max": 14, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-58315.log", + "samples": 272, + "observed_wall_ms": 27100, + "cpu_seconds": 23.85, + "cpu_core_equivalents": 0.8800738007380073, + "heapUsedBytes_sampled_max": 165779696, + "heapCommittedBytes_sampled_max": 188743680, + "poolActive_sampled_max": 5, + "poolIdle_sampled_max": 4, + "poolWaiting_sampled_max": 14, + "poolMaximum_sampled_max": 8 + } + ], + "cleanup": true, + "receipt_note": "Java executable path normalized; measurements unchanged. Disposable fixture identities only." +} diff --git a/docs/performance/results/windows-timeout.json b/docs/performance/results/windows-timeout.json new file mode 100644 index 0000000..9551682 --- /dev/null +++ b/docs/performance/results/windows-timeout.json @@ -0,0 +1,61 @@ +{ + "status": "FAIL", + "parameters": { + "java": "java (Java 21)", + "nodes": 3, + "tenants": 2, + "users": 12, + "tabs": 4, + "rounds": 1000, + "workers": 8 + }, + "scope": "loopback HTTP, mock identities, memory records", + "current_round": 480, + "completed_rounds": 479, + "error": "HTTP 503 node=56028 user=load-t0-u5 path=/presence/leases/renew: {\"error\":\"Presence is temporarily unavailable. Please retry.\",\"code\":\"ERROR\",\"message\":\"Presence is temporarily unavailable. Please retry.\",\"traceId\":\"cea30b82-e82b-49ec-af8a-64d8b6bd5af6\"}", + "redis_slow_commands": [], + "redis_latency_events": [], + "resources": [ + { + "node_log": "node-56027.log", + "samples": 1067, + "observed_wall_ms": 106600, + "cpu_seconds": 94.328125, + "cpu_core_equivalents": 0.8848792213883677, + "heapUsedBytes_sampled_max": 160559696, + "heapCommittedBytes_sampled_max": 178257920, + "poolActive_sampled_max": 5, + "poolIdle_sampled_max": 5, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-56028.log", + "samples": 1066, + "observed_wall_ms": 106501, + "cpu_seconds": 95.8125, + "cpu_core_equivalents": 0.8996394400052582, + "heapUsedBytes_sampled_max": 160251104, + "heapCommittedBytes_sampled_max": 178257920, + "poolActive_sampled_max": 5, + "poolIdle_sampled_max": 4, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-56029.log", + "samples": 1066, + "observed_wall_ms": 106500, + "cpu_seconds": 95.25, + "cpu_core_equivalents": 0.8943661971830986, + "heapUsedBytes_sampled_max": 157995216, + "heapCommittedBytes_sampled_max": 175112192, + "poolActive_sampled_max": 5, + "poolIdle_sampled_max": 4, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + } + ], + "cleanup": true, + "receipt_note": "Java executable path normalized; measurements unchanged. Disposable fixture identities only." +} diff --git a/pom.xml b/pom.xml index fae0d5f..25861a6 100644 --- a/pom.xml +++ b/pom.xml @@ -18,12 +18,19 @@ + + com.kingsrook.qbits.objectstorage.api,com.kingsrook.qbits.email.api,com.kingsrook.qbits.redis.api + com.kingsrook.qbits:qbit-object-storage,com.kingsrook.qbits:qbit-email,com.kingsrook.qbits:*-api,com.kingsrook.qbits:qbit-redis 0.1.0-SNAPSHOT 21 21 0.8.15 0.1.0-RC.2 + + 4.1.0-RC.22 + + 0.1.0-RC.1 @@ -68,11 +75,29 @@ HEAD + + + + com.kingsrook.qqq + qqq-bom-pom + ${qqq.version} + pom + import + + + + com.kingsrook.qqq qqq-backend-module-rdbms + + com.kingsrook.qbits + qbit-redis + ${qbit-redis.version} + true + com.kingsrook.qbits qbit-object-storage @@ -110,6 +135,47 @@ 3.27.7 test + + + com.h2database + h2 + 2.4.240 + test + + + org.testcontainers + testcontainers + 2.0.3 + test + + + org.testcontainers + testcontainers-mysql + 2.0.3 + test + + + org.postgresql + postgresql + 42.7.8 + test + + + + com.kingsrook.qqq + qqq-backend-module-postgres + test + + + com.kingsrook.qqq + qqq-backend-module-rdbms + + + com.kingsrook.qqq + qqq-backend-core + + + @@ -133,6 +199,14 @@ + + org.apache.maven.pluginsmaven-resources-plugin3.3.1 + package-license-noticesprocess-resourcescopy-resources + ${project.build.outputDirectory}/META-INF + ${project.basedir}falseLICENSENOTICE + + + org.apache.maven.plugins @@ -221,30 +295,4 @@ - - - - - - - qqq-snapshot - - true - - - 4.1.0-RC.15 - - - - - com.kingsrook.qqq - qqq-bom-pom - ${qqq.snapshot.version} - pom - import - - - - - diff --git a/src/main/java/com/kingsrook/qbits/collaboration/CollaborationQBitConfig.java b/src/main/java/com/kingsrook/qbits/collaboration/CollaborationQBitConfig.java index bcff3d4..78c79ed 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/CollaborationQBitConfig.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/CollaborationQBitConfig.java @@ -47,6 +47,7 @@ public class CollaborationQBitConfig implements QBitConfig private List mirroredSecurityKeyTypes = List.of(); private QCodeReference notifier = null; + private com.kingsrook.qbits.collaboration.presence.RedisPresenceConfig redisPresence; private Function recipientEmailResolver = null; private QCodeReference userSessionProvider = null; private String defaultTagMode = "FREE"; @@ -64,6 +65,7 @@ public class CollaborationQBitConfig implements QBitConfig private Boolean adminApp = true; private Boolean uiApi = true; private QCodeReference attachmentStorage = new QCodeReference(com.kingsrook.qbits.collaboration.attachments.AttachmentStorage.class); + private QCodeReference attachmentLifecycle = new QCodeReference(com.kingsrook.qbits.collaboration.attachments.AttachmentLifecyclePolicy.class); private Integer recentsPerUser = 200; private String attachmentsBucketName = "collaboration"; private Long defaultMaxBytes = 100L * 1024 * 1024; @@ -103,6 +105,23 @@ public class CollaborationQBitConfig implements QBitConfig return (this); } + /******************************************************************************* + ** Trusted host policy for retained attachments and access auditing. + *******************************************************************************/ + public QCodeReference getAttachmentLifecycle() + { + return (attachmentLifecycle); + } + + /******************************************************************************* + ** Configure lifecycle enforcement without coupling to a consuming qbit. + *******************************************************************************/ + public CollaborationQBitConfig withAttachmentLifecycle(QCodeReference attachmentLifecycle) + { + this.attachmentLifecycle = java.util.Objects.requireNonNull(attachmentLifecycle, "attachmentLifecycle"); + return (this); + } + /******************************************************************************* @@ -297,6 +316,10 @@ public class CollaborationQBitConfig implements QBitConfig @Override public void validate(QInstance instance, List errors) { + if(redisPresence != null) + { + redisPresence.validate(errors); + } if(assertCondition(StringUtils.hasContent(backendName), "backendName is required for Collaboration", errors)) { assertCondition(instance.getBackend(backendName) != null, "Unknown Collaboration backend: " + backendName, errors); @@ -674,4 +697,21 @@ public class CollaborationQBitConfig implements QBitConfig { this.managedIndexes = managedIndexes; } + + /*************************************************************************** + ** Optional shared Redis presence; absent retains the existing memory provider. + ***************************************************************************/ + public com.kingsrook.qbits.collaboration.presence.RedisPresenceConfig getRedisPresence() + { + return (redisPresence); + } + + /*************************************************************************** + ** Select shared presence with explicit host-derived tenant scope. + ***************************************************************************/ + public CollaborationQBitConfig withRedisPresence(com.kingsrook.qbits.collaboration.presence.RedisPresenceConfig value) + { + redisPresence = value; + return (this); + } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/CollaborationQBitProducer.java b/src/main/java/com/kingsrook/qbits/collaboration/CollaborationQBitProducer.java index d78ac37..8fabe52 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/CollaborationQBitProducer.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/CollaborationQBitProducer.java @@ -155,6 +155,10 @@ public class CollaborationQBitProducer implements QBitMetaDataProducer query(String table, QQueryFilter filter) throws QException { - QueryInput input = new QueryInput(table).withFilter(filter).withShouldFetchHeavyFields(true).withTransaction(QContext.getQBackendTransaction()); + return (query(table, filter, QContext.getQBackendTransaction())); + } + + /******************************************************************************* + ** Provider reads participate in the originating query's explicit transaction. + *******************************************************************************/ + public static List query(String table, QQueryFilter filter, QBackendTransaction transaction) throws QException + { + QueryInput input = new QueryInput(table).withFilter(filter).withShouldFetchHeavyFields(true).withTransaction(transaction); input.setTableMetaData(serviceMetadata(table)); input.withPermissionBypass("Trusted collaboration service performs parent/ownership authorization."); return (new QueryAction().execute(input).getRecords().stream().map(QRecord::new).toList()); @@ -85,8 +97,34 @@ public final class CollaborationRecords *******************************************************************************/ public static QRecord insert(String table, QRecord row) throws QException { - InsertInput input = new InsertInput(table).withRecords(List.of(row)).withTransaction(QContext.getQBackendTransaction()); - input.setTableMetaData(serviceMetadata(table)); + return (insert(serviceMetadata(table), row)); + } + + /******************************************************************************* + ** Activity belongs to the authorized source operation; timeline controls its + ** presentation. Preserve native parent checks and mirrored security keys. + *******************************************************************************/ + public static QRecord insertActivity(QRecord row, RecordFeature sourceFeature) throws QException + { + QTableMetaData metadata = serviceMetadata(CollaborationTables.ACTIVITY); + RecordReferenceTableMetaData original = RecordReferenceTableMetaData.of(metadata); + if(original == null || sourceFeature == null) + { + throw (new QException("Activity source authorization metadata is required.")); + } + RecordReferenceTableMetaData reference = (RecordReferenceTableMetaData) original.clone(); + reference.withFeature(sourceFeature); + metadata.withSupplementalMetaData(reference); + return (insert(metadata, row)); + } + + /******************************************************************************* + ** A private metadata copy retains the same transaction and validation path. + *******************************************************************************/ + private static QRecord insert(QTableMetaData metadata, QRecord row) throws QException + { + InsertInput input = new InsertInput(metadata.getName()).withRecords(List.of(row)).withTransaction(QContext.getQBackendTransaction()); + input.setTableMetaData(metadata); input.withPermissionBypass("Trusted collaboration service authorized this write."); List result = new InsertAction().execute(input).getRecords(); check(result); @@ -143,7 +181,10 @@ public final class CollaborationRecords public static QTableMetaData cleanupMetadata(String table) { QTableMetaData metadata = serviceMetadata(table); - metadata.getSupplementalMetaData().remove(com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordReferenceTableMetaData.NAME); + if(metadata.getSupplementalMetaData() != null) + { + metadata.getSupplementalMetaData().remove(com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordReferenceTableMetaData.NAME); + } metadata.setRecordSecurityLocks(null); return (metadata); } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/CollaborationSchema.java b/src/main/java/com/kingsrook/qbits/collaboration/CollaborationSchema.java index 3fb2fc3..ef3727c 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/CollaborationSchema.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/CollaborationSchema.java @@ -57,7 +57,7 @@ public final class CollaborationSchema for(QFieldMetaData field : table.getFields().values()) { String type = field.getName().equals(table.getPrimaryKeyField()) ? "INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY" : sqlType(field); - columns.add(" " + field.getBackendName() + " " + type + (field.getName().equals("parentDeleted") ? " DEFAULT FALSE" : "") + (Boolean.TRUE.equals(field.getIsRequired()) ? " NOT NULL" : "")); + columns.add(" " + field.getBackendName() + " " + type + (field.getName().equals("parentDeleted") || field.getName().equals("retained") ? " DEFAULT FALSE" : "") + (Boolean.TRUE.equals(field.getIsRequired()) ? " NOT NULL" : "")); } statements.add("CREATE TABLE IF NOT EXISTS " + table.getName() + " (\n" + String.join(",\n", columns) + "\n)"); for(QFieldMetaData field : table.getFields().values()) @@ -78,6 +78,10 @@ public final class CollaborationSchema } statements.add("CREATE INDEX IF NOT EXISTS idx_collaboration_record_tag_tag ON collaboration_record_tag (tag_id)"); statements.add("CREATE INDEX IF NOT EXISTS idx_collaboration_attachment_parent ON collaboration_attachment (table_name, record_id)"); + statements.add("ALTER TABLE collaboration_attachment ADD COLUMN IF NOT EXISTS retained BOOLEAN NOT NULL DEFAULT FALSE"); + statements.add("ALTER TABLE collaboration_attachment ADD COLUMN IF NOT EXISTS intake_key VARCHAR(64)"); + statements.add("ALTER TABLE collaboration_attachment ADD COLUMN IF NOT EXISTS intake_fingerprint VARCHAR(64)"); + statements.add("CREATE UNIQUE INDEX IF NOT EXISTS uq_collaboration_attachment_intake ON collaboration_attachment (intake_key) WHERE intake_key IS NOT NULL"); statements.add("CREATE INDEX IF NOT EXISTS idx_collaboration_comment_parent ON collaboration_comment (table_name, record_id, last_activity_at, id)"); statements.add("CREATE INDEX IF NOT EXISTS idx_collaboration_comment_created ON collaboration_comment (table_name, record_id, create_date)"); statements.add("CREATE INDEX IF NOT EXISTS idx_collaboration_comment_thread ON collaboration_comment (thread_id, create_date, id)"); @@ -95,6 +99,15 @@ public final class CollaborationSchema statements.add("CREATE INDEX IF NOT EXISTS idx_collaboration_assignment_owner ON collaboration_record_assignment (owner_user_id)"); statements.add("CREATE INDEX IF NOT EXISTS idx_collaboration_assignment_assignee ON collaboration_record_assignment (assignee_user_id)"); statements.add("ALTER TABLE collaboration_mention ADD COLUMN IF NOT EXISTS read_at TIMESTAMPTZ"); + statements.add("ALTER TABLE collaboration_attachment_cleanup ADD COLUMN IF NOT EXISTS next_attempt_at TIMESTAMPTZ"); + statements.add("ALTER TABLE collaboration_attachment_cleanup ADD COLUMN IF NOT EXISTS resolved_at TIMESTAMPTZ"); + statements.add("ALTER TABLE collaboration_attachment_cleanup ADD COLUMN IF NOT EXISTS resolved_by VARCHAR(255)"); + statements.add("ALTER TABLE collaboration_attachment_cleanup ADD COLUMN IF NOT EXISTS settlement_evidence VARCHAR(2000)"); + statements.add("ALTER TABLE collaboration_attachment_cleanup ALTER COLUMN attachment_id DROP NOT NULL"); + statements.add("DROP INDEX IF EXISTS uq_collaboration_attachment_cleanup_1"); + statements.add("CREATE INDEX IF NOT EXISTS idx_collaboration_attachment_storage_key ON collaboration_attachment USING HASH (storage_key)"); + statements.add("CREATE INDEX IF NOT EXISTS idx_collaboration_attachment_cleanup_due ON collaboration_attachment_cleanup (next_attempt_at, id)"); + statements.add("CREATE INDEX IF NOT EXISTS idx_collaboration_attachment_cleanup_pending ON collaboration_attachment_cleanup (next_attempt_at, id) WHERE resolved_at IS NULL"); statements.add("CREATE INDEX IF NOT EXISTS idx_collaboration_mention_unread ON collaboration_mention (mentioned_user_id) WHERE notified_at IS NOT NULL AND read_at IS NULL"); return (String.join(";\n", statements) + ";\n"); } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/CollaborationSummaryProvider.java b/src/main/java/com/kingsrook/qbits/collaboration/CollaborationSummaryProvider.java index 2916ced..c0e20a3 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/CollaborationSummaryProvider.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/CollaborationSummaryProvider.java @@ -49,14 +49,13 @@ public class CollaborationSummaryProvider implements RecordFeatureSummaryProvide String user = PersonalRecords.currentUserId(); if(features.isEnabled(RecordFeature.PRESENCE)) { - summary.setPresence(com.kingsrook.qbits.collaboration.presence.PresenceRegistry.get(QContext.getQInstance()).snapshot(ref.tableName(), id).stream().map(viewer -> + summary.setPresence(new com.kingsrook.qbits.collaboration.presence.PresenceService().snapshot(ref.tableName(), id).stream().map(viewer -> { java.util.Map entry = new java.util.LinkedHashMap<>(); entry.put("userId", viewer.userId()); entry.put("displayName", viewer.displayName()); entry.put("mode", viewer.mode()); entry.put("fieldName", viewer.fieldName()); - entry.put("connections", viewer.connections()); return (entry); }).toList()); } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/activity/ActivityWriter.java b/src/main/java/com/kingsrook/qbits/collaboration/activity/ActivityWriter.java index b262bbb..24c0a7d 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/activity/ActivityWriter.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/activity/ActivityWriter.java @@ -28,6 +28,7 @@ import com.kingsrook.qbits.collaboration.model.CollaborationTables; import com.kingsrook.qqq.backend.core.context.QContext; import com.kingsrook.qqq.backend.core.exceptions.QException; import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordFeature; /******************************************************************************* ** Internal activity writer; the calling service has already checked the parent. @@ -46,8 +47,28 @@ public final class ActivityWriter *******************************************************************************/ public static void write(String table, String recordId, String type, String summary, String refTable, String refId) throws QException { - CollaborationRecords.insert(CollaborationTables.ACTIVITY, new QRecord().withValue("tableName", table).withValue("recordId", recordId) + CollaborationRecords.insertActivity(new QRecord().withValue("tableName", table).withValue("recordId", recordId) .withValue("at", Instant.now()).withValue("actorUserId", CollaborationSecurity.userId(QContext.getQSession())) - .withValue("type", type).withValue("summary", summary).withValue("refTableName", refTable).withValue("refId", refId).withValue("parentDeleted", false)); + .withValue("type", type).withValue("summary", summary).withValue("refTableName", refTable).withValue("refId", refId).withValue("parentDeleted", false), sourceFeature(refTable)); + } + + /******************************************************************************* + ** Source-specific activity requires that feature; generic events retain the + ** existing timeline authorization contract. + *******************************************************************************/ + private static RecordFeature sourceFeature(String table) + { + return (switch(table == null ? "" : table) + { + case CollaborationTables.ATTACHMENT -> RecordFeature.ATTACHMENTS; + case CollaborationTables.COMMENT, CollaborationTables.COMMENT_REVISION, CollaborationTables.MENTION -> RecordFeature.COMMENTS; + case CollaborationTables.TAG, CollaborationTables.RECORD_TAG -> RecordFeature.TAGS; + case CollaborationTables.FIELD_DEFINITION, CollaborationTables.FIELD_VALUE -> RecordFeature.CUSTOM_FIELDS; + case CollaborationTables.TASK -> RecordFeature.TASKS; + case CollaborationTables.RECORD_ASSIGNMENT -> RecordFeature.ASSIGNMENT; + case CollaborationTables.FOLLOW, CollaborationTables.FOLLOW_DIGEST -> RecordFeature.FOLLOW; + case CollaborationTables.SHARE -> RecordFeature.SHARING; + default -> RecordFeature.TIMELINE; + }); } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationApi.java b/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationApi.java index 7c43429..3b69f3a 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationApi.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationApi.java @@ -52,9 +52,10 @@ public final class CollaborationApi *******************************************************************************/ public static MiddlewareSpecExtension extension() { - return (new MiddlewareSpecExtension().withName("collaboration").withSemanticVersion("1.2.0") + return (new MiddlewareSpecExtension().withName("collaboration").withSemanticVersion("1.3.0") .withDescription("Record collaboration: favorites, recents, attachments, tags, conversations, following, timeline, people and presence") .withSpec(PostPresenceSpec.class).withSpec(DeletePresenceSpec.class).withSpec(GetPresenceSpec.class) + .withSpec(OpenPresenceLeaseSpec.class).withSpec(RenewPresenceLeaseSpec.class) .withSpec(ListFavoritesSpec.class) .withSpec(StarFavoriteSpec.class) .withSpec(UnstarFavoriteSpec.class) @@ -96,6 +97,7 @@ public final class CollaborationApi .withSpec(ListMyTasksSpec.class) .withSpec(PutAssignmentSpec.class) .withSpec(GetMyWorkSpec.class) + .withSpec(MarkMentionsReadSpec.class) .withSpec(UploadAttachmentSpec.class)); } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationJsonSpec.java b/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationJsonSpec.java index 3e2a64d..accd37a 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationJsonSpec.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationJsonSpec.java @@ -116,7 +116,8 @@ abstract class CollaborationJsonSpec extends AbstractJsonEndpointSpec return (List.of(new BasicResponse(HttpStatus.BAD_REQUEST, "Invalid input", BasicErrorResponseV1.class.getSimpleName()), new BasicResponse(HttpStatus.UNAUTHORIZED, "Sign in required", BasicErrorResponseV1.class.getSimpleName()), new BasicResponse(HttpStatus.FORBIDDEN, "Permission denied", BasicErrorResponseV1.class.getSimpleName()), - new BasicResponse(HttpStatus.NOT_FOUND, "Not found (including invisible parents)", BasicErrorResponseV1.class.getSimpleName()))); + new BasicResponse(HttpStatus.NOT_FOUND, "Not found (including invisible parents)", BasicErrorResponseV1.class.getSimpleName()), + new BasicResponse(HttpStatus.SERVICE_UNAVAILABLE, "Service temporarily unavailable", BasicErrorResponseV1.class.getSimpleName()))); } /******************************************************************************* @@ -161,14 +162,13 @@ abstract class CollaborationJsonSpec extends AbstractJsonEndpointSpec case "cursor" -> "Opaque nextCursor returned by the previous page; omit for the first page."; case "before" -> "Timeline continuation token; alias for cursor. Supply only one."; case "includeDone" -> "Include completed and cancelled tasks; default false."; - case "perConnection" -> "Also return one viewer per live connection (lease) in leases; default false. Since collaboration 1.2.0."; case "status" -> "Task status: OPEN (default), DONE or CANCELLED."; case "types" -> "Comma-separated AUDIT, ACTIVITY or activity type names (for example COMMENTED)."; case "q" -> "Text to search in display names or vocabulary."; default -> name; }; return (new Parameter().withName(name).withIn(location).withRequired(required).withDescription(description) - .withSchema(new Schema().withType(name.equals("limit") ? Type.INTEGER : name.equals("includeDone") || name.equals("perConnection") ? Type.BOOLEAN : Type.STRING))); + .withSchema(new Schema().withType(name.equals("limit") ? Type.INTEGER : name.equals("includeDone") ? Type.BOOLEAN : Type.STRING))); } /******************************************************************************* @@ -214,6 +214,19 @@ abstract class CollaborationJsonSpec extends AbstractJsonEndpointSpec protected static Page page(JsonInput input, List values) throws QBadRequestException { int limit = limit(input); + PagePosition position = pagePosition(input); + int start = Math.min(position.offset(), values.size()); + int end = start + Math.min(limit, values.size() - start); + String next = end < values.size() ? position.cursor(end) : null; + return (new Page<>(values.subList(start, end), next)); + } + + /******************************************************************************* + ** Shared existing offset cursor contract, also used by bounded native lists. + ** Scope is a routing check, never authorization or a signed capability. + *******************************************************************************/ + protected static PagePosition pagePosition(JsonInput input) throws QBadRequestException + { int offset = 0; String scope = input.getSpec().defineBasicOperation().getPath() + input.getPathParams().toString() + ":" + input.getQueryParam("table") + ":" + input.getQueryParam("pk") + ":" + input.getQueryParam("q"); @@ -239,10 +252,25 @@ abstract class CollaborationJsonSpec extends AbstractJsonEndpointSpec throw (new QBadRequestException("Invalid cursor")); } } - int start = Math.min(offset, values.size()); - int end = start + Math.min(limit, values.size() - start); - String next = end < values.size() ? Base64.getUrlEncoder().withoutPadding().encodeToString((prefix + end).getBytes(StandardCharsets.UTF_8)) : null; - return (new Page<>(values.subList(start, end), next)); + return (new PagePosition(offset, prefix)); + } + + /******************************************************************************* + ** Same version, scope and offset bytes as previously emitted by page(). + *******************************************************************************/ + protected record PagePosition(int offset, String prefix) + { + /******************************************************************************* + ** Encode only a representable nonnegative continuation offset. + *******************************************************************************/ + String cursor(long next) throws QBadRequestException + { + if(next < 0 || next > Integer.MAX_VALUE) + { + throw (new QBadRequestException("Invalid cursor")); + } + return (Base64.getUrlEncoder().withoutPadding().encodeToString((prefix + next).getBytes(StandardCharsets.UTF_8))); + } } /******************************************************************************* diff --git a/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationPresenceSchemas.java b/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationPresenceSchemas.java index 55c9f56..68164ce 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationPresenceSchemas.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationPresenceSchemas.java @@ -24,7 +24,6 @@ package com.kingsrook.qbits.collaboration.api; import java.util.List; import java.util.Map; import java.util.TreeMap; -import com.fasterxml.jackson.annotation.JsonInclude; import com.kingsrook.qbits.collaboration.presence.PresenceRegistry; import com.kingsrook.qqq.middleware.javalin.schemabuilder.ToSchema; import com.kingsrook.qqq.openapi.model.Schema; @@ -42,11 +41,67 @@ public final class CollaborationPresenceSchemas { } + /******************************************************************************* + ** Target for invisible server-issued renewal authority. + *******************************************************************************/ + public record CollaborationPresenceOpenV1(String table, String pk) implements ToSchema + { + /*************************************************************************** + ** Opening alone never advertises a viewer. + ***************************************************************************/ + @Override + public Schema toSchema() + { + return (new Schema().withType(Type.OBJECT).withProperties(new TreeMap<>(Map.of( + "table", text("Parent table name.", false), "pk", text("Optional record key.", true))))); + } + } + + /******************************************************************************* + ** Opaque handle bound to the authenticated caller and target. + *******************************************************************************/ + public record CollaborationPresenceLeaseV1(String connectionId) implements ToSchema + { + /*************************************************************************** + ** Activate with a positive sequence; acquire a new handle after expiry. + ***************************************************************************/ + @Override + public Schema toSchema() + { + return (new Schema().withType(Type.OBJECT).withProperties(Map.of( + "connectionId", text("Server-issued handle. Renew before 45 seconds; departure is terminal.", false)))); + } + } + + /******************************************************************************* + ** Stable machine-readable response for missing renewal authority. + *******************************************************************************/ + public record CollaborationPresenceExpiredV1(String code) implements ToSchema + { + /*************************************************************************** + ** Clients rejoin only if their surface is still active and visible. + ***************************************************************************/ + @Override + public Schema toSchema() + { + return (new Schema().withType(Type.OBJECT).withProperties(Map.of( + "code", text("Open a fresh lease before retrying renewal.", false).withEnumValues(List.of("PRESENCE_LEASE_EXPIRED"))))); + } + } + /******************************************************************************* ** All identity information is derived from the authenticated session. *******************************************************************************/ - public record CollaborationPresenceRequestV1(String connectionId, String table, String pk, String mode, String fieldName) implements ToSchema + public record CollaborationPresenceRequestV1(String connectionId, String table, String pk, String mode, String fieldName, Long sequence) implements ToSchema { + /******************************************************************************* + ** Preserve source callers that do not yet send sequence numbers. + *******************************************************************************/ + public CollaborationPresenceRequestV1(String connectionId, String table, String pk, String mode, String fieldName) + { + this(connectionId, table, pk, mode, fieldName, null); + } + /******************************************************************************* ** Clients renew every twenty seconds; the lease expires after forty-five. *******************************************************************************/ @@ -58,15 +113,24 @@ public final class CollaborationPresenceSchemas "connectionId", text("Client connection identity.", false), "table", text("Parent table name.", false), "pk", text("Optional parent key; omit for a table view.", true), "mode", text("VIEWING or EDITING.", false).withEnumValues(List.of("VIEWING", "EDITING")), - "fieldName", text("Optional focused field on a record.", true))))); + "fieldName", text("Optional focused field on a record.", true), + "sequence", sequenceSchema())))); } } /******************************************************************************* ** Leave removes only this authenticated user's connection and target. *******************************************************************************/ - public record CollaborationPresenceLeaveV1(String connectionId, String table, String pk) implements ToSchema + public record CollaborationPresenceLeaveV1(String connectionId, String table, String pk, Long sequence) implements ToSchema { + /******************************************************************************* + ** Preserve source callers that do not yet send sequence numbers. + *******************************************************************************/ + public CollaborationPresenceLeaveV1(String connectionId, String table, String pk) + { + this(connectionId, table, pk, null); + } + /******************************************************************************* ** Leave does not need a mode or a field. *******************************************************************************/ @@ -76,27 +140,15 @@ public final class CollaborationPresenceSchemas return (new Schema().withType(Type.OBJECT).withDescription("Remove the caller's lease on this connection and target.") .withProperties(new TreeMap<>(Map.of( "connectionId", text("Client connection identity.", false), "table", text("Parent table name.", false), - "pk", text("Optional parent key.", true))))); + "pk", text("Optional parent key.", true), "sequence", sequenceSchema())))); } } /******************************************************************************* - ** The snapshot is also the data shape of presence.changed events. Viewers are - ** included even when empty, whatever inclusion the serializer defaults to. + ** The snapshot is also the data shape of presence.changed events. *******************************************************************************/ - public record CollaborationPresenceSnapshotV1(String table, String pk, @JsonInclude(JsonInclude.Include.ALWAYS) List viewers, - Integer leaseCount, @JsonInclude(JsonInclude.Include.NON_NULL) List leases) implements ToSchema + public record CollaborationPresenceSnapshotV1(String table, String pk, List viewers) implements ToSchema { - /******************************************************************************* - ** The merged snapshot without per-connection detail. - *******************************************************************************/ - public CollaborationPresenceSnapshotV1(String table, String pk, List viewers) - { - this(table, pk, viewers, null, null); - } - - - /******************************************************************************* ** A table snapshot deliberately excludes record ids and focused field names. *******************************************************************************/ @@ -106,17 +158,14 @@ public final class CollaborationPresenceSchemas return (new Schema().withType(Type.OBJECT).withDescription("Live viewers, merged across tabs; presence.changed uses this payload.") .withProperties(new TreeMap<>(Map.of( "table", text("Parent table name.", false), "pk", text("Record key; absent for table events.", true), - "viewers", new Schema().withType(Type.ARRAY).withDescription("Live viewers; always present, empty once the last viewer leaves.").withNullable(false).withItems(new Schema().withRefToSchema("CollaborationPresenceViewerV1")), - "leaseCount", new Schema().withType(Type.INTEGER).withDescription("Live connections (leases) behind viewers; a user with two terminals counts twice. Since 1.2.0.").withNullable(true), - "leases", new Schema().withType(Type.ARRAY).withDescription("One viewer per live connection, with connectionId and since. On GET only with perConnection=true; always on presence.changed. Since 1.2.0.") - .withNullable(true).withItems(new Schema().withRefToSchema("CollaborationPresenceViewerV1")))))); + "viewers", new Schema().withType(Type.ARRAY).withItems(new Schema().withRefToSchema("CollaborationPresenceViewerV1")))))); } } /******************************************************************************* ** Only public identity and focus are carried, never field contents. *******************************************************************************/ - public record CollaborationPresenceViewerV1(String userId, String displayName, String mode, String fieldName, Integer connections, String connectionId, String sessionId, String since) implements ToSchema + public record CollaborationPresenceViewerV1(String userId, String displayName, String mode, String fieldName) implements ToSchema { /******************************************************************************* ** Editing takes precedence when one user has several open tabs. @@ -124,14 +173,10 @@ public final class CollaborationPresenceSchemas @Override public Schema toSchema() { - return (new Schema().withType(Type.OBJECT).withDescription("One user merged across live tabs (EDITING takes precedence), or one live connection in leases.") + return (new Schema().withType(Type.OBJECT).withDescription("One user merged across live tabs; EDITING takes precedence.") .withProperties(new TreeMap<>(Map.of( "userId", text("Stable user identity.", false), "displayName", text("Session display name.", false), - "mode", text("VIEWING or EDITING.", false).withEnumValues(List.of("VIEWING", "EDITING")), "fieldName", text("Focused field; null on table snapshots.", true), - "connections", new Schema().withType(Type.INTEGER).withDescription("Live connections (tabs, terminals) merged into this viewer; 1 in leases. Since 1.2.0.").withNullable(true), - "connectionId", text("Opaque per-connection id, only in leases: c + the first 16 hex digits of SHA-256(userId + newline + connectionId). Since 1.2.0.", true), - "sessionId", text("Opaque id of the signed-in session behind this connection, only in leases (HMAC of the server session; never the session itself). Since 1.2.0.", true), - "since", text("When this connection's lease started (ISO-8601), only in leases. Since 1.2.0.", true).withFormat("date-time"))))); + "mode", text("VIEWING or EDITING.", false).withEnumValues(List.of("VIEWING", "EDITING")), "fieldName", text("Focused field; null on table snapshots.", true))))); } } @@ -142,4 +187,13 @@ public final class CollaborationPresenceSchemas { return (new Schema().withType(Type.STRING).withDescription(description).withNullable(nullable)); } + + /******************************************************************************* + ** Optional for legacy clients; new clients increment on every renewal and leave. + *******************************************************************************/ + private static Schema sequenceSchema() + { + return (new Schema().withType(Type.INTEGER).withNullable(true) + .withDescription("Monotonically increasing nonnegative safe integer per connection and target; orders delayed requests.")); + } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationSchemas.java b/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationSchemas.java index 911bcdc..954ce23 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationSchemas.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationSchemas.java @@ -70,8 +70,17 @@ public final class CollaborationSchemas /******************************************************************************* ** An attachment visible through its parent record; no storage coordinates are exposed. *******************************************************************************/ - public record CollaborationAttachmentV1(Integer id, String tableName, String recordId, String fileName, String contentType, Long sizeBytes, String sha256, String description, String createdByUserId, Instant createDate, Boolean canDelete, String createdByDisplayName) implements ToSchema + public record CollaborationAttachmentV1(Integer id, String tableName, String recordId, String fileName, String contentType, Long sizeBytes, String sha256, String description, String createdByUserId, Instant createDate, Boolean canDelete, String createdByDisplayName, Boolean canEdit) implements ToSchema { + /******************************************************************************* + ** Preserve construction compatibility with the earlier attachment body. + *******************************************************************************/ + public CollaborationAttachmentV1(Integer id, String tableName, String recordId, String fileName, String contentType, + Long sizeBytes, String sha256, String description, String createdByUserId, Instant createDate, Boolean canDelete, String createdByDisplayName) + { + this(id, tableName, recordId, fileName, contentType, sizeBytes, sha256, description, createdByUserId, createDate, canDelete, createdByDisplayName, canDelete); + } + /******************************************************************************* ** Describe this versioned body for generated clients. *******************************************************************************/ @@ -89,7 +98,8 @@ public final class CollaborationSchemas Map.entry("description", new Schema().withType(Type.STRING).withDescription("Optional description.").withNullable(true)), Map.entry("createdByUserId", new Schema().withType(Type.STRING).withDescription("Uploader identity.").withNullable(true)), Map.entry("createDate", new Schema().withType(Type.STRING).withFormat("date-time").withDescription("Upload time.").withNullable(true)), - Map.entry("canDelete", new Schema().withType(Type.BOOLEAN).withDescription("Whether the caller may change or delete this attachment.").withNullable(true)), + Map.entry("canDelete", new Schema().withType(Type.BOOLEAN).withDescription("Whether the caller may delete this attachment; retained files cannot be deleted.").withNullable(true)), + Map.entry("canEdit", new Schema().withType(Type.BOOLEAN).withDescription("Whether the caller may change the attachment filename or description.").withNullable(true)), Map.entry("createdByDisplayName", new Schema().withType(Type.STRING).withDescription("Uploader display name.").withNullable(true)))))); } } @@ -490,8 +500,16 @@ public final class CollaborationSchemas /******************************************************************************* ** Temporary access to file bytes. Clients follow url. *******************************************************************************/ - public record CollaborationDownloadV1(String url) implements ToSchema + public record CollaborationDownloadV1(String url, String delivery) implements ToSchema { + /*************************************************************************** + ** Existing callers retain the storage URL contract. + ***************************************************************************/ + public CollaborationDownloadV1(String url) + { + this(url, "SIGNED_URL"); + } + /******************************************************************************* ** Describe this versioned body for generated clients. *******************************************************************************/ @@ -499,7 +517,8 @@ public final class CollaborationSchemas public Schema toSchema() { return (new Schema().withType(Type.OBJECT).withDescription("Temporary access to file bytes. Clients follow url.").withProperties(new TreeMap<>(Map.ofEntries( - Map.entry("url", new Schema().withType(Type.STRING).withDescription("Presigned URL, valid for five minutes. Clients follow url; never persist it.").withNullable(true)))))); + Map.entry("url", new Schema().withType(Type.STRING).withDescription("Temporary delivery URL; never persist it.").withNullable(true)), + Map.entry("delivery", new Schema().withType(Type.STRING).withDescription("SIGNED_URL or AUTHENTICATED_FETCH. Authenticated fetch requires the trusted native API transport and a durable receipt check.").withNullable(true)))))); } } @@ -717,7 +736,10 @@ public final class CollaborationSchemas schemas.put("CollaborationPresenceRequestV1", new CollaborationPresenceSchemas.CollaborationPresenceRequestV1(null, null, null, null, null).toSchema()); schemas.put("CollaborationPresenceLeaveV1", new CollaborationPresenceSchemas.CollaborationPresenceLeaveV1(null, null, null).toSchema()); schemas.put("CollaborationPresenceSnapshotV1", new CollaborationPresenceSchemas.CollaborationPresenceSnapshotV1(null, null, null).toSchema()); - schemas.put("CollaborationPresenceViewerV1", new CollaborationPresenceSchemas.CollaborationPresenceViewerV1(null, null, null, null, null, null, null, null).toSchema()); + schemas.put("CollaborationPresenceViewerV1", new CollaborationPresenceSchemas.CollaborationPresenceViewerV1(null, null, null, null).toSchema()); + schemas.put("CollaborationPresenceOpenV1", new CollaborationPresenceSchemas.CollaborationPresenceOpenV1(null, null).toSchema()); + schemas.put("CollaborationPresenceLeaseV1", new CollaborationPresenceSchemas.CollaborationPresenceLeaseV1(null).toSchema()); + schemas.put("CollaborationPresenceExpiredV1", new CollaborationPresenceSchemas.CollaborationPresenceExpiredV1(null).toSchema()); schemas.putAll(CollaborationWorkSchemas.components()); return (schemas); } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationViews.java b/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationViews.java index 12f48b8..48f256e 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationViews.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/api/CollaborationViews.java @@ -106,7 +106,7 @@ final class CollaborationViews Map names = names(values.stream().map(AttachmentEntry::createdByUserId).toArray(String[]::new)); return (values.stream().map(value -> new CollaborationAttachmentV1( value.id(), value.tableName(), value.recordId(), value.fileName(), value.contentType(), value.sizeBytes(), value.sha256(), value.description(), value.createdByUserId(), value.createDate(), value.canDelete(), - name(names, value.createdByUserId()))).toList()); + name(names, value.createdByUserId()), value.canEdit())).toList()); } /******************************************************************************* diff --git a/src/main/java/com/kingsrook/qbits/collaboration/api/DeletePresenceSpec.java b/src/main/java/com/kingsrook/qbits/collaboration/api/DeletePresenceSpec.java index 66ab0c7..57ff0e1 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/api/DeletePresenceSpec.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/api/DeletePresenceSpec.java @@ -76,7 +76,7 @@ public class DeletePresenceSpec extends CollaborationJsonSpec protected Object handle(JsonInput input) throws Exception { var body = body(input, CollaborationPresenceSchemas.CollaborationPresenceLeaveV1.class); - new PresenceService().leave(body.connectionId(), required(body.table(), "table"), body.pk()); + new PresenceService().leave(body.connectionId(), required(body.table(), "table"), body.pk(), body.sequence()); return (success()); } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/api/DownloadAttachmentSpec.java b/src/main/java/com/kingsrook/qbits/collaboration/api/DownloadAttachmentSpec.java index 7f42cfc..c2d60f8 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/api/DownloadAttachmentSpec.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/api/DownloadAttachmentSpec.java @@ -68,6 +68,7 @@ public class DownloadAttachmentSpec extends CollaborationJsonSpec @Override protected Object handle(JsonInput input) throws Exception { - return (new CollaborationDownloadV1(new AttachmentsService().downloadUrl(id(input), false).toExternalForm())); + var delivery = new AttachmentsService().downloadDelivery(id(input)); + return (new CollaborationDownloadV1(delivery.url().toExternalForm(), delivery.authenticated() ? "AUTHENTICATED_FETCH" : "SIGNED_URL")); } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/api/GetPresenceSpec.java b/src/main/java/com/kingsrook/qbits/collaboration/api/GetPresenceSpec.java index 1ec816a..c83f55f 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/api/GetPresenceSpec.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/api/GetPresenceSpec.java @@ -23,7 +23,6 @@ package com.kingsrook.qbits.collaboration.api; import java.util.List; import com.kingsrook.qbits.collaboration.presence.PresenceService; -import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException; import com.kingsrook.qqq.middleware.javalin.specs.BasicOperation; import com.kingsrook.qqq.openapi.model.HttpMethod; import com.kingsrook.qqq.openapi.model.Parameter; @@ -49,7 +48,7 @@ public class GetPresenceSpec extends CollaborationJsonSpec @Override public List defineRequestParameters() { - return (parameters("table!", "pk", "perConnection")); + return (parameters("table!", "pk")); } /******************************************************************************* @@ -69,14 +68,6 @@ public class GetPresenceSpec extends CollaborationJsonSpec { String table = required(input.getQueryParam("table"), "table"); String pk = input.getQueryParam("pk"); - String perConnection = input.getQueryParam("perConnection"); - if(perConnection != null && !perConnection.equalsIgnoreCase("true") && !perConnection.equalsIgnoreCase("false")) - { - throw (new QBadRequestException("perConnection must be true or false")); - } - var service = new PresenceService(); - var viewers = service.snapshot(table, pk); - var leases = service.connections(table, pk); - return (new CollaborationPresenceSchemas.CollaborationPresenceSnapshotV1(table, pk, viewers, leases.size(), "true".equalsIgnoreCase(perConnection) ? leases : null)); + return (new CollaborationPresenceSchemas.CollaborationPresenceSnapshotV1(table, pk, new PresenceService().snapshot(table, pk))); } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/api/ListAttachmentsSpec.java b/src/main/java/com/kingsrook/qbits/collaboration/api/ListAttachmentsSpec.java index 5712cd4..d3a53fa 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/api/ListAttachmentsSpec.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/api/ListAttachmentsSpec.java @@ -68,7 +68,10 @@ public class ListAttachmentsSpec extends CollaborationJsonSpec @Override protected Object handle(JsonInput input) throws Exception { - var page = page(input, new AttachmentsService().list(table(input), pk(input))); - return (new CollaborationAttachmentListV1(CollaborationViews.attachments(page.items()), page.nextCursor())); + int limit = limit(input); + var position = pagePosition(input); + var page = new AttachmentsService().listPage(table(input), pk(input), position.offset(), limit); + String next = page.hasMore() ? position.cursor((long) position.offset() + page.items().size()) : null; + return (new CollaborationAttachmentListV1(CollaborationViews.attachments(page.items()), next)); } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/api/MarkMentionsReadSpec.java b/src/main/java/com/kingsrook/qbits/collaboration/api/MarkMentionsReadSpec.java new file mode 100644 index 0000000..594af5f --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/api/MarkMentionsReadSpec.java @@ -0,0 +1,71 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.api; + +import java.util.List; +import com.kingsrook.qqq.middleware.javalin.specs.BasicOperation; +import com.kingsrook.qqq.openapi.model.HttpMethod; +import com.kingsrook.qqq.openapi.model.Parameter; + +/******************************************************************************* + ** Mark all currently visible unread mentions read. + *******************************************************************************/ +public class MarkMentionsReadSpec extends CollaborationJsonSpec +{ + /******************************************************************************* + ** Describe this operation. + *******************************************************************************/ + @Override + public BasicOperation defineBasicOperation() + { + return (operation(HttpMethod.POST, "/my-work/mentions/read", "Mark my visible mentions read")); + } + + /******************************************************************************* + ** Declare inputs. + *******************************************************************************/ + @Override + public List defineRequestParameters() + { + return (parameters()); + } + + /******************************************************************************* + ** Name the response. + *******************************************************************************/ + @Override + protected String getResponseSchemaName() + { + return ("CollaborationMutationResultV1"); + } + + /******************************************************************************* + ** Run the service as the authenticated caller. + *******************************************************************************/ + @Override + protected Object handle(JsonInput input) throws Exception + { + new com.kingsrook.qbits.collaboration.mywork.MyWorkService().markMentionsRead(null); + return (success()); + } + +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/api/OpenPresenceLeaseSpec.java b/src/main/java/com/kingsrook/qbits/collaboration/api/OpenPresenceLeaseSpec.java new file mode 100644 index 0000000..06fb073 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/api/OpenPresenceLeaseSpec.java @@ -0,0 +1,81 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.api; + +import java.util.List; +import com.kingsrook.qbits.collaboration.presence.PresenceService; +import com.kingsrook.qqq.middleware.javalin.specs.BasicOperation; +import com.kingsrook.qqq.openapi.model.HttpMethod; +import com.kingsrook.qqq.openapi.model.Parameter; +import com.kingsrook.qqq.openapi.model.Schema; + +/******************************************************************************* + ** Renew presence with fresh parent access checks. + *******************************************************************************/ +public class OpenPresenceLeaseSpec extends CollaborationJsonSpec +{ + /******************************************************************************* + ** Describe the extension operation. + *******************************************************************************/ + @Override + public BasicOperation defineBasicOperation() + { + return (operation(HttpMethod.POST, "/presence/leases", "Open presence lease")); + } + + /******************************************************************************* + ** Explicit query parameters for polling. + *******************************************************************************/ + @Override + public List defineRequestParameters() + { + return (parameters()); + } + + /******************************************************************************* + ** Name the versioned response schema. + *******************************************************************************/ + @Override + protected String getResponseSchemaName() + { + return ("CollaborationPresenceLeaseV1"); + } + + /******************************************************************************* + ** Declare a typed JSON request body. + *******************************************************************************/ + @Override + protected Schema defineRequestBodySchema() + { + return (new Schema().withRefToSchema("CollaborationPresenceOpenV1")); + } + + /******************************************************************************* + ** Services derive identity from QContext and enforce visibility. + *******************************************************************************/ + @Override + protected Object handle(JsonInput input) throws Exception + { + var body = body(input, CollaborationPresenceSchemas.CollaborationPresenceOpenV1.class); + return (new CollaborationPresenceSchemas.CollaborationPresenceLeaseV1(new PresenceService().openLease(required(body.table(), "table"), body.pk()))); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/api/PostPresenceSpec.java b/src/main/java/com/kingsrook/qbits/collaboration/api/PostPresenceSpec.java index 39fbc87..a00357d 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/api/PostPresenceSpec.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/api/PostPresenceSpec.java @@ -77,6 +77,6 @@ public class PostPresenceSpec extends CollaborationJsonSpec { var body = body(input, CollaborationPresenceSchemas.CollaborationPresenceRequestV1.class); return (new CollaborationPresenceSchemas.CollaborationPresenceSnapshotV1(body.table(), body.pk(), - new PresenceService().heartbeat(body.connectionId(), required(body.table(), "table"), body.pk(), body.mode(), body.fieldName()))); + new PresenceService().heartbeat(body.connectionId(), required(body.table(), "table"), body.pk(), body.mode(), body.fieldName(), body.sequence()))); } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/api/PreviewAttachmentSpec.java b/src/main/java/com/kingsrook/qbits/collaboration/api/PreviewAttachmentSpec.java index 632dff3..faf2485 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/api/PreviewAttachmentSpec.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/api/PreviewAttachmentSpec.java @@ -68,6 +68,7 @@ public class PreviewAttachmentSpec extends CollaborationJsonSpec @Override protected Object handle(JsonInput input) throws Exception { - return (new CollaborationDownloadV1(new AttachmentsService().downloadUrl(id(input), true).toExternalForm())); + var delivery = new AttachmentsService().downloadDelivery(id(input), true); + return (new CollaborationDownloadV1(delivery.url().toExternalForm(), delivery.authenticated() ? "AUTHENTICATED_FETCH" : "SIGNED_URL")); } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/api/PutAssignmentSpec.java b/src/main/java/com/kingsrook/qbits/collaboration/api/PutAssignmentSpec.java index e8ba262..712d5c6 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/api/PutAssignmentSpec.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/api/PutAssignmentSpec.java @@ -76,7 +76,7 @@ public class PutAssignmentSpec extends CollaborationJsonSpec { var body = body(input, CollaborationWorkSchemas.CollaborationAssignmentRequestV1.class); return (CollaborationViews.assignment(new com.kingsrook.qbits.collaboration.assignment.AssignmentService() - .assign(table(input), pk(input), body.ownerUserId(), body.assigneeUserId()))); + .assignRoles(table(input), pk(input), body.ownerUserId(), body.assigneeUserId(), input.getBody().keySet()))); } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/api/RenewPresenceLeaseSpec.java b/src/main/java/com/kingsrook/qbits/collaboration/api/RenewPresenceLeaseSpec.java new file mode 100644 index 0000000..499fae1 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/api/RenewPresenceLeaseSpec.java @@ -0,0 +1,112 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.api; + +import java.util.List; +import com.kingsrook.qbits.collaboration.presence.PresenceService; +import com.kingsrook.qqq.middleware.javalin.specs.BasicOperation; +import com.kingsrook.qqq.openapi.model.HttpMethod; +import com.kingsrook.qqq.openapi.model.Parameter; +import com.kingsrook.qqq.openapi.model.Schema; + +/******************************************************************************* + ** Renew presence with fresh parent access checks. + *******************************************************************************/ +public class RenewPresenceLeaseSpec extends CollaborationJsonSpec +{ + /******************************************************************************* + ** Expiry is distinct from permissions, invalid input and service outages. + *******************************************************************************/ + @Override + public List defineAdditionalBasicResponses() + { + var responses = new java.util.ArrayList<>(super.defineAdditionalBasicResponses()); + responses.add(new com.kingsrook.qqq.middleware.javalin.specs.BasicResponse(io.javalin.http.HttpStatus.CONFLICT, + "Presence lease expired; open a new lease.", "CollaborationPresenceExpiredV1")); + return (responses); + } + + /******************************************************************************* + ** Preserve standard framework mapping for every other exception. + *******************************************************************************/ + @Override + protected void handleException(io.javalin.http.Context context, Exception exception) + { + if(com.kingsrook.qqq.backend.core.utils.ExceptionUtils.findClassInRootChain(exception, + com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class) != null) + { + context.status(409).contentType("application/json").result(com.kingsrook.qqq.backend.core.utils.JsonUtils.toJson( + new CollaborationPresenceSchemas.CollaborationPresenceExpiredV1("PRESENCE_LEASE_EXPIRED"))); + } + else + { + super.handleException(context, exception); + } + } + + /******************************************************************************* + ** Describe the extension operation. + *******************************************************************************/ + @Override + public BasicOperation defineBasicOperation() + { + return (operation(HttpMethod.POST, "/presence/leases/renew", "Renew issued presence lease")); + } + + /******************************************************************************* + ** Explicit query parameters for polling. + *******************************************************************************/ + @Override + public List defineRequestParameters() + { + return (parameters()); + } + + /******************************************************************************* + ** Name the versioned response schema. + *******************************************************************************/ + @Override + protected String getResponseSchemaName() + { + return ("CollaborationPresenceSnapshotV1"); + } + + /******************************************************************************* + ** Declare a typed JSON request body. + *******************************************************************************/ + @Override + protected Schema defineRequestBodySchema() + { + return (new Schema().withRefToSchema("CollaborationPresenceRequestV1")); + } + + /******************************************************************************* + ** Services derive identity from QContext and enforce visibility. + *******************************************************************************/ + @Override + protected Object handle(JsonInput input) throws Exception + { + var body = body(input, CollaborationPresenceSchemas.CollaborationPresenceRequestV1.class); + return (new CollaborationPresenceSchemas.CollaborationPresenceSnapshotV1(body.table(), body.pk(), + new PresenceService().renewLease(body.connectionId(), required(body.table(), "table"), body.pk(), body.mode(), body.fieldName(), body.sequence()))); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/app/CollaborationHome.java b/src/main/java/com/kingsrook/qbits/collaboration/app/CollaborationHome.java index 6018d7a..086c03e 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/app/CollaborationHome.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/app/CollaborationHome.java @@ -108,7 +108,7 @@ public final class CollaborationHome .withPermissionRules(CollaborationPermissions.permissionRules()) .withWidgets(HOME_WIDGETS) .withSection(new QAppSection().withName("collaboration").withLabel("Collaboration").withIcon(new QIcon("forum")) - .withTables(List.of(CollaborationTables.COMMENT, CollaborationTables.ATTACHMENT, CollaborationTables.TAG, CollaborationTables.FIELD_DEFINITION, CollaborationTables.TASK, CollaborationTables.SHARE, CollaborationTables.ACCESS_REQUEST, CollaborationTables.RECORD_ASSIGNMENT, CollaborationTables.ACTIVITY, CollaborationTables.FEATURE_SETTING, CollaborationTables.USER))); + .withTables(List.of(CollaborationTables.COMMENT, CollaborationTables.ATTACHMENT, CollaborationTables.ATTACHMENT_CLEANUP, CollaborationTables.TAG, CollaborationTables.FIELD_DEFINITION, CollaborationTables.TASK, CollaborationTables.SHARE, CollaborationTables.ACCESS_REQUEST, CollaborationTables.RECORD_ASSIGNMENT, CollaborationTables.ACTIVITY, CollaborationTables.FEATURE_SETTING, CollaborationTables.USER))); app.setParentAppName(config.getParentAppName()); return (app); } @@ -267,19 +267,37 @@ public final class CollaborationHome List> rows = new ArrayList<>(); if(admin) { - var registry = PresenceRegistry.get(QContext.getQInstance()); - Map connections = registry.connectionCounts(); - registry.counts().forEach((table, viewers) -> + var instance = QContext.getQInstance(); + var redis = com.kingsrook.qbits.collaboration.presence.RedisPresenceConfig.forInstance(instance); + Map counts; + if(redis == null) + { + counts = PresenceRegistry.get(instance).counts(); + } + else + { + counts = new java.util.TreeMap<>(); + var backend = new com.kingsrook.qbits.collaboration.presence.RedisPresenceBackend(redis); + // Read only known table rooms in the administrator's authenticated tenant. + // Never scan Redis keys or silently substitute local counts on failure. + for(String tableName : instance.getTables().keySet()) + { + int viewers = backend.snapshot(tableName, null).size(); + if(viewers > 0) + { + counts.put(tableName, viewers); + } + } + } + counts.forEach((table, viewers) -> { Map row = new LinkedHashMap<>(); row.put("table", tableLabel(table)); row.put("viewers", viewers); - row.put("connections", connections.getOrDefault(table, viewers)); rows.add(row); }); } - return (table("Who's here", admin ? NOBODY_HERE : PRESENCE_ADMIN_ONLY, rows, column("Table", "table", "left"), column("People", "viewers", "right"), - column("Connections", "connections", "right"))); + return (table("Who's here", admin ? NOBODY_HERE : PRESENCE_ADMIN_ONLY, rows, column("Table", "table", "left"), column("People", "viewers", "right"))); } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/assignment/AssignmentService.java b/src/main/java/com/kingsrook/qbits/collaboration/assignment/AssignmentService.java index 5cd6ee1..57074f5 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/assignment/AssignmentService.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/assignment/AssignmentService.java @@ -53,15 +53,42 @@ public class AssignmentService ** Full replacement of both roles; null clears a role. *******************************************************************************/ public QRecord assign(String table, Serializable recordId, String ownerUserId, String assigneeUserId) throws QException + { + return (assignRoles(table, recordId, ownerUserId, assigneeUserId, java.util.Set.of("ownerUserId", "assigneeUserId"))); + } + + /******************************************************************************* + ** Omitted roles remain unchanged; an explicitly supplied null clears a role. + *******************************************************************************/ + public QRecord assignRoles(String table, Serializable recordId, String ownerUserId, String assigneeUserId, java.util.Set roles) throws QException { ParentRecordAccess.requireFeature(table, recordId, "assign"); String id = FollowService.canonicalId(table, recordId); + if(roles == null || roles.isEmpty() || !java.util.Set.of("ownerUserId", "assigneeUserId").containsAll(roles)) + { + throw (new com.kingsrook.qqq.backend.core.exceptions.QBadRequestException("Supply ownerUserId or assigneeUserId; omitted roles remain unchanged.")); + } + var instance = com.kingsrook.qqq.backend.core.context.QContext.getQInstance(); + var ownership = com.kingsrook.qqq.backend.core.actions.collaboration.RecordFeaturesResolver.resolve(instance, instance.getTable(table)).getOwnership(); + if(ownership != null && ((roles.contains("ownerUserId") && ownership.getOwnerFieldName() != null) + || (roles.contains("assigneeUserId") && ownership.getAssigneeFieldName() != null))) + { + throw (new com.kingsrook.qqq.backend.core.exceptions.QBadRequestException("Update native ownership fields through the parent record's versioned update endpoint.")); + } WorkSupport.requireReader(ownerUserId, table, id); WorkSupport.requireReader(assigneeUserId, table, id); synchronized(CollaborationRecords.config()) { Optional old = get(table, id); - QRecord change = new QRecord().withValue("ownerUserId", ownerUserId).withValue("assigneeUserId", assigneeUserId); + QRecord change = new QRecord(); + if(roles.contains("ownerUserId")) + { + change.setValue("ownerUserId", ownerUserId); + } + if(roles.contains("assigneeUserId")) + { + change.setValue("assigneeUserId", assigneeUserId); + } QRecord row; if(old.isEmpty()) { @@ -73,7 +100,7 @@ public class AssignmentService row = get(table, id).orElseThrow(); } ActivityWriter.write(table, id, "ASSIGNED", "Record assignment changed", CollaborationTables.RECORD_ASSIGNMENT, row.getValueString("id")); - if(assigneeUserId != null && (old.isEmpty() || !Objects.equals(old.get().getValueString("assigneeUserId"), assigneeUserId))) + if(roles.contains("assigneeUserId") && assigneeUserId != null && (old.isEmpty() || !Objects.equals(old.get().getValueString("assigneeUserId"), assigneeUserId))) { new FollowService().ensureFollower(assigneeUserId, table, id, "ASSIGNED"); if(!assigneeUserId.equals(PersonalRecords.currentUserId())) diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AdoptRetainedAttachmentProcess.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AdoptRetainedAttachmentProcess.java new file mode 100644 index 0000000..634ed12 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AdoptRetainedAttachmentProcess.java @@ -0,0 +1,121 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + + +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qbits.collaboration.permissions.CollaborationPermissions; +import com.kingsrook.qqq.backend.core.actions.processes.BackendStep; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData; + + +/******************************************************************************* + ** A visible native adoption action for an existing original. It never + ** accepts replacement bytes, a storage path, an actor, or an intake scope. + *******************************************************************************/ +public final class AdoptRetainedAttachmentProcess +{ + public static final String NAME = "collaborationAdoptRetainedAttachment"; + + /******************************************************************************* + ** Metadata utility. + *******************************************************************************/ + private AdoptRetainedAttachmentProcess() + { + } + + /******************************************************************************* + ** Standard QQQ confirmation and result screens, attached to the attachment table. + *******************************************************************************/ + public static QProcessMetaData metadata() + { + return (new QProcessMetaData().withName(NAME).withTableName(CollaborationTables.ATTACHMENT).withLabel("Retain existing original") + .withMinInputRecords(1).withMaxInputRecords(1).withPermissionRules(CollaborationPermissions.permissionRules()) + .withStep(new com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData().withName("prepare").withCode(new QCodeReference(Prepare.class)) + .withInputData(new com.kingsrook.qqq.backend.core.model.metadata.processes.QFunctionInputMetaData() + .withRecordListMetaData(new com.kingsrook.qqq.backend.core.model.metadata.processes.QRecordListMetaData().withTableName(CollaborationTables.ATTACHMENT)))) + .withStep(new com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData().withName("confirm").withLabel("Retain this original") + .withComponent(new com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendComponentMetaData() + .withType(com.kingsrook.qqq.backend.core.model.metadata.processes.QComponentType.EDIT_FORM)) + .withFormField(new QFieldMetaData("confirmAdoption", QFieldType.BOOLEAN).withLabel("Verify and retain the existing original without copying bytes").withDefaultValue(false).withIsRequired(true))) + .withStep(new com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData().withName("adopt").withCode(new QCodeReference(Adopt.class))) + .withStep(new com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData().withName("result").withLabel("Original retained") + .withComponent(new com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendComponentMetaData() + .withType(com.kingsrook.qqq.backend.core.model.metadata.processes.QComponentType.VIEW_FORM)) + .withViewField(new QFieldMetaData("attachmentId", QFieldType.INTEGER).withLabel("Retained attachment") + .withPossibleValueSourceName(com.kingsrook.qbits.collaboration.model.CollaborationTables.ATTACHMENT)))); + } + + /******************************************************************************* + ** Resolve one visible attachment selection before displaying confirmation. + *******************************************************************************/ + public static final class Prepare implements BackendStep + { + /******************************************************************************* + ** Read-only preview under current actor and parent visibility. + *******************************************************************************/ + @Override + public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + if(input.getRecords() == null || input.getRecords().size() != 1) + { + throw (new QException("Select exactly one attachment.")); + } + var row = new AttachmentsService().requireAdoptable(input.getRecords().getFirst().getValueInteger("id")); + if(row == null) + { + throw (new QException("Attachment is unavailable.")); + } + if(row.getValueString("intakeKey") != null) + { + throw (new QException("Retry this upload through its original intake request to preserve its context.")); + } + output.addValue("attachmentId", row.getValue("id")); + output.addValue("confirmAdoption", false); + } + } + + /******************************************************************************* + ** Execution reauthorizes independently of the earlier preview. + *******************************************************************************/ + public static final class Adopt implements BackendStep + { + /******************************************************************************* + ** The owner returns only after mandatory lifecycle registration completes. + *******************************************************************************/ + @Override + public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + if(!Boolean.TRUE.equals(input.getValueBoolean("confirmAdoption"))) + { + throw (new QException("Confirm adoption of this existing original to continue.")); + } + output.addValue("attachmentId", new AttachmentsService().adoptRetainedAttachment(input.getValueInteger("attachmentId")).id()); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentAggregateGuard.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentAggregateGuard.java new file mode 100644 index 0000000..a92f6fa --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentAggregateGuard.java @@ -0,0 +1,201 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + + +import java.io.Serializable; +import java.util.List; +import com.kingsrook.qbits.collaboration.CollaborationQBitConfig; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qbits.collaboration.security.ParentRecordAccess; +import com.kingsrook.qqq.backend.core.actions.customizers.QCodeLoader; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.AggregateInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules; +import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QSupplementalTableMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; + + +/******************************************************************************* + ** Optional owner policy for parents whose dynamic visibility cannot be expressed + ** by aggregate SQL. Count/aggregate require one currently authorized parent. + ** Ordinary collaboration policies retain their existing count behavior. + *******************************************************************************/ +public final class AttachmentAggregateGuard extends AttachmentSecurity +{ + private static final String PINNED_POLICY = "collaboration.attachmentPinnedAggregateParent"; + + /******************************************************************************* + ** Hide unscoped list totals in both instance and table metadata. Native count + ** remains available to callers supplying the required authorized parent pin. + *******************************************************************************/ + public static QTableMetaData configure(QTableMetaData table, CollaborationQBitConfig config) + { + if(policy(config).requiresPinnedAggregateParent()) + { + table.withSupplementalMetaData(new PinnedParentPolicy()); + } + if(table.getSupplementalMetaData(PINNED_POLICY) != null) + { + table.withoutCapabilities(Capability.TABLE_COUNT, Capability.QUERY_STATS); + } + return (table); + } + + /******************************************************************************* + ** Preserve ordinary view/write rules, then narrow aggregate inputs only. + *******************************************************************************/ + @Override + public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException + { + super.checkPermissionsThrowing(input, metadata); + if(!(input instanceof CountInput) && !(input instanceof AggregateInput)) + { + return; + } + try + { + CollaborationQBitConfig config = (CollaborationQBitConfig) QContext.getQInstance().getQBits().values().stream() + .filter(qbit -> qbit.getConfig() instanceof CollaborationQBitConfig).findFirst().orElseThrow().getConfig(); + QTableMetaData tableMetadata = QContext.getQInstance().getTable(CollaborationTables.ATTACHMENT); + if(tableMetadata.getSupplementalMetaData(PINNED_POLICY) == null && !policy(config).requiresPinnedAggregateParent() && !hasRetainedOriginal()) + { + return; + } + QQueryFilter filter = input instanceof CountInput count ? count.getFilter() : ((AggregateInput) input).getFilter(); + var joins = input instanceof CountInput count ? count.getQueryJoins() : ((AggregateInput) input).getQueryJoins(); + if(joins != null && !joins.isEmpty() || filter == null || filter.getBooleanOperator() != QQueryFilter.BooleanOperator.AND) + { + throw (new QException("A single pinned attachment parent is required for counts and aggregates.")); + } + rejectSetOperations(filter, 0, new int[]{0}); + Serializable table = literal(filter, "tableName"); + Serializable record = literal(filter, "recordId"); + if(!(table instanceof String name) || name.isBlank() || record == null) + { + throw (new QException("A single pinned attachment parent is required for counts and aggregates.")); + } + ParentRecordAccess.requireVisibleParent(name, record); + } + catch(QException | RuntimeException denied) + { + throw (new QPermissionDeniedException("Counts and aggregates require one currently readable attachment parent.")); + } + } + + /******************************************************************************* + ** A literal top-level AND equality constrains every row, even when other + ** independent subfilters contain OR. Conflicting or computed pins are refused. + *******************************************************************************/ + private static Serializable literal(QQueryFilter filter, String field) throws QException + { + Serializable result = null; + for(var criterion : filter.getCriteria() == null ? List.of() : filter.getCriteria()) + { + if(field.equals(criterion.getFieldName()) && criterion.getOperator() == QCriteriaOperator.EQUALS) + { + if(criterion.getValues() == null || criterion.getValues().size() != 1) + { + throw (new QException("An exact parent equality is required.")); + } + Serializable value = criterion.getValues().getFirst(); + if(!(value instanceof String) && !(value instanceof Number) || result != null && !result.equals(value)) + { + throw (new QException("Computed or conflicting parent equality is not supported.")); + } + result = value; + } + } + return (result); + } + + /******************************************************************************* + ** Set-operation filters may combine independent branches; never assume their + ** SQL obeys an ordinary root AND pin. Bound traversal before SQL execution. + *******************************************************************************/ + private static void rejectSetOperations(QQueryFilter filter, int depth, int[] visited) throws QException + { + if(depth > 64 || ++visited[0] > 4096 || filter.getSubFilterSetOperator() != null) + { + throw (new QException("Unsupported aggregate filter shape.")); + } + for(QQueryFilter child : filter.getSubFilters() == null ? List.of() : filter.getSubFilters()) + { + rejectSetOperations(child, depth + 1, visited); + } + } + + /******************************************************************************* + ** Capability policy is configuration-only and must not depend on a request. + *******************************************************************************/ + private static AttachmentLifecyclePolicy policy(CollaborationQBitConfig config) + { + return (java.util.Objects.requireNonNull(QCodeLoader.getAdHoc(AttachmentLifecyclePolicy.class, config.getAttachmentLifecycle()), "Attachment lifecycle policy is unavailable.")); + } + + /******************************************************************************* + ** A fresh instance may have lost its former lifecycle configuration. Inspect + ** only existence through private owner metadata, never return retained rows. + ** Failure propagates to aggregate denial; there is no permissive fallback. + *******************************************************************************/ + private static boolean hasRetainedOriginal() throws QException + { + var input = new com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput(CollaborationTables.ATTACHMENT) + .withFilter(new QQueryFilter().withCriteria("retained", QCriteriaOperator.EQUALS, true).withLimit(1)) + .withShouldFetchHeavyFields(false).withShouldGenerateDisplayValues(false); + input.setTableMetaData(com.kingsrook.qbits.collaboration.CollaborationRecords.cleanupMetadata(CollaborationTables.ATTACHMENT)); + input.withPermissionBypass("Owner aggregate guard checks only whether retained originals require pinned-parent authorization; no row data is exposed."); + return (!new com.kingsrook.qqq.backend.core.actions.tables.QueryAction().execute(input).getRecords().isEmpty()); + } + + /******************************************************************************* + ** An installed capability restriction survives subsequent lifecycle changes + ** and metadata clones. It carries no request or authorization state. + *******************************************************************************/ + public static final class PinnedParentPolicy extends QSupplementalTableMetaData + { + /******************************************************************************* + ** Stable owner metadata identity. + *******************************************************************************/ + @Override + public String getType() + { + return (PINNED_POLICY); + } + + /******************************************************************************* + ** The marker is immutable and has no nested mutable state. + *******************************************************************************/ + @Override + protected QSupplementalTableMetaData finishClone(QSupplementalTableMetaData clone) + { + return (clone); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentCleanup.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentCleanup.java index 25d33e0..033d289 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentCleanup.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentCleanup.java @@ -66,7 +66,8 @@ public class AttachmentCleanup while(true) { List rows = com.kingsrook.qbits.collaboration.CollaborationRecords.cleanupQuery(com.kingsrook.qbits.collaboration.model.CollaborationTables.ATTACHMENT, new QQueryFilter().withCriteria("tableName", QCriteriaOperator.EQUALS, table) - .withCriteria("recordId", QCriteriaOperator.IN, ids.stream().map(Object::toString).toList()).withLimit(200)); + .withCriteria("recordId", QCriteriaOperator.IN, ids.stream().map(Object::toString).toList()) + .withCriteria("parentDeleted", QCriteriaOperator.NOT_EQUALS, true).withLimit(200)); if(rows.isEmpty()) { return; diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentCleanupQueue.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentCleanupQueue.java new file mode 100644 index 0000000..0889018 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentCleanupQueue.java @@ -0,0 +1,527 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.collaboration.attachments; + + +import java.io.Serializable; +import java.time.Instant; +import java.util.List; +import com.kingsrook.qbits.collaboration.CollaborationRecords; +import com.kingsrook.qbits.collaboration.CollaborationSecurity; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qbits.collaboration.permissions.CollaborationPermissions; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker; +import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction; +import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; +import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; +import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction; +import com.kingsrook.qqq.backend.core.context.CapturedContext; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryLockMode; +import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules; +import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession; +import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleDispatcher; + + +/******************************************************************************* + ** Durable deletion intent, committed with metadata removal before deleting bytes. + *******************************************************************************/ +public class AttachmentCleanupQueue implements CustomPermissionChecker +{ + private final AttachmentStorage storage; + + + /******************************************************************************* + ** Metadata permission checks do not create a storage client. + *******************************************************************************/ + public AttachmentCleanupQueue() + { + this(new AttachmentStorage()); + } + + + /******************************************************************************* + ** Use the same provider boundary as the authorized attachment operation. + *******************************************************************************/ + AttachmentCleanupQueue(AttachmentStorage storage) + { + this.storage = storage; + } + + + /******************************************************************************* + ** Only admins may inspect recovery locations; no direct mutations are allowed. + *******************************************************************************/ + @Override + public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException + { + if(!(input instanceof QueryInput) && !(input instanceof com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput) + && !(input instanceof com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput)) + { + throw (new QPermissionDeniedException("Attachment cleanup records are service-managed.")); + } + requireOperator(); + } + + + /******************************************************************************* + ** The caller has already authorized the parent and attachment owner. + *******************************************************************************/ + void delete(QRecord row, boolean cleanup) throws QException + { + QBackendTransaction outer = QContext.getQBackendTransaction(); + if(outer != null) + { + QRecord pending = enqueueAndDelete(row, cleanup); + CapturedContext context = new CapturedContext(QContext.getQInstance(), QContext.getQSession()); + outer.addAfterCommitCallback(() -> + { + try + { + QContext.withTemporaryContext(context, () -> finish(pending)); + } + catch(QException e) + { + throw (new IllegalStateException("Attachment cleanup remains durably queued.", e)); + } + }); + return; + } + QRecord pending; + try(QBackendTransaction transaction = QBackendTransaction.openFor(new DeleteInput(CollaborationTables.ATTACHMENT))) + { + QContext.setTransaction(transaction); + try + { + pending = enqueueAndDelete(row, cleanup); + transaction.commit(); + } + catch(QException | RuntimeException failure) + { + transaction.rollback(); + throw (failure); + } + finally + { + QContext.clearTransaction(); + } + } + finish(pending); + } + + + /******************************************************************************* + ** Store the recovery locator in the same database transaction as row deletion. + *******************************************************************************/ + private QRecord enqueueAndDelete(QRecord row, boolean cleanup) throws QException + { + QRecord intent = new QRecord().withValue("attachmentId", row.getValue("id")) + .withValue("bucketName", row.getValue("bucketName")).withValue("storageKey", row.getValue("storageKey")); + InsertInput insert = new InsertInput(CollaborationTables.ATTACHMENT_CLEANUP).withRecords(List.of(intent)) + .withTransaction(QContext.getQBackendTransaction()); + insert.withPermissionBypass("Authorized attachment deletion records durable storage cleanup."); + List inserted = new InsertAction().execute(insert).getRecords(); + AttachmentRecords.checkErrors(inserted); + DeleteInput delete = new DeleteInput(CollaborationTables.ATTACHMENT).withPrimaryKeys(List.of(row.getValue("id"))) + .withTransaction(QContext.getQBackendTransaction()); + delete.setTableMetaData(cleanup ? CollaborationRecords.cleanupMetadata(CollaborationTables.ATTACHMENT) : AttachmentRecords.storageTable()); + delete.withPermissionBypass("Attachment deletion was authorized before recording cleanup."); + AttachmentRecords.checkErrors(new DeleteAction().execute(delete).getRecordsWithErrors()); + return (inserted.getFirst()); + } + + + /******************************************************************************* + ** Durable upload ownership requires real transactional row locking. + *******************************************************************************/ + static boolean supportsUploadOwnership() throws QException + { + var backend = QContext.getQInstance().getBackendForTable(CollaborationTables.ATTACHMENT_CLEANUP); + return (new QBackendModuleDispatcher().getQBackendModule(backend).supportsQueryLockMode(backend, QueryLockMode.FOR_UPDATE_SKIP_LOCKED)); + } + + + /******************************************************************************* + ** Commit the locator independently, then lock it before any provider write. + ** If a worker claimed it first, abort without uploading any bytes. + *******************************************************************************/ + QRecord beginUpload(String bucket, String key) throws QException + { + if(!supportsUploadOwnership()) + { + return (null); + } + QBackendTransaction transaction = QContext.getQBackendTransaction(); + if(transaction == null) + { + throw (new QException("Durable attachment uploads require an active transaction.")); + } + CapturedContext context = new CapturedContext(QContext.getQInstance(), QContext.getQSession()); + QRecord[] pending = new QRecord[1]; + QContext.withTemporaryContext(context, () -> + { + InsertInput insert = new InsertInput(CollaborationTables.ATTACHMENT_CLEANUP).withRecords(List.of( + new QRecord().withValue("bucketName", bucket).withValue("storageKey", key))); + insert.withPermissionBypass("Authorized upload records its immutable locator before external I/O."); + List inserted = new InsertAction().execute(insert).getRecords(); + AttachmentRecords.checkErrors(inserted); + pending[0] = inserted.getFirst(); + }); + if(lock(pending[0], transaction) == null) + { + throw (new QException("Upload cleanup intent was already claimed; retry the upload.")); + } + Runnable cleanup = () -> + { + try + { + QContext.withTemporaryContext(context, () -> finish(pending[0])); + } + catch(QException e) + { + throw (new IllegalStateException("Upload cleanup remains durably queued.", e)); + } + }; + transaction.addAfterRollbackCallback(cleanup); + transaction.addAfterCommitCallback(cleanup); + return (pending[0]); + } + + + /******************************************************************************* + ** Publish metadata and retire its upload intent in the same transaction. + *******************************************************************************/ + void completeUpload(QRecord pending) throws QException + { + if(pending != null) + { + DeleteInput delete = new DeleteInput(CollaborationTables.ATTACHMENT_CLEANUP).withPrimaryKey(pending.getValue("id")) + .withTransaction(QContext.getQBackendTransaction()); + delete.withPermissionBypass("Attachment metadata now owns the uploaded object in this transaction."); + AttachmentRecords.checkErrors(new DeleteAction().execute(delete).getRecordsWithErrors()); + } + } + + + /******************************************************************************* + ** Skip active owners without waiting or updating their locked intent. + *******************************************************************************/ + private QRecord lock(QRecord pending, QBackendTransaction transaction) throws QException + { + QueryInput query = new QueryInput(CollaborationTables.ATTACHMENT_CLEANUP).withTransaction(transaction) + .withLockMode(QueryLockMode.FOR_UPDATE_SKIP_LOCKED) + .withFilter(new QQueryFilter().withCriteria("id", QCriteriaOperator.EQUALS, pending.getValue("id"))); + query.withPermissionBypass("Internal attachment lifecycle owns this immutable cleanup locator."); + List rows = new QueryAction().execute(query).getRecords(); + return (rows.isEmpty() ? null : rows.getFirst()); + } + + + /******************************************************************************* + ** Distinguish an active owner from a committed row requiring delayed retry. + *******************************************************************************/ + private enum CleanupOutcome + { + COMPLETED, ACTIVE, REFERENCED, UNCERTAIN_UPLOAD + } + + + /******************************************************************************* + ** Hold the same row lock as an uploader until deletion and acknowledgement end. + *******************************************************************************/ + private CleanupOutcome finish(QRecord pending) throws QException + { + if(!supportsUploadOwnership()) + { + return (finishUnlocked(pending)); + } + CleanupOutcome[] outcome = { CleanupOutcome.ACTIVE }; + CapturedContext context = new CapturedContext(QContext.getQInstance(), QContext.getQSession()); + QContext.withTemporaryContext(context, () -> + { + try(QBackendTransaction transaction = QBackendTransaction.openFor(new DeleteInput(CollaborationTables.ATTACHMENT_CLEANUP))) + { + QContext.setTransaction(transaction); + try + { + QRecord locked = lock(pending, transaction); + if(locked != null) + { + outcome[0] = finishUnlocked(locked); + } + transaction.commit(); + } + catch(QException | RuntimeException failure) + { + transaction.rollback(); + throw (failure); + } + finally + { + QContext.clearTransaction(); + } + } + }); + return (outcome[0]); + } + + + /******************************************************************************* + ** Never erase bytes while metadata still exists, including nontransactional hosts. + ** Provider or queue deletion failure retains the intent for idempotent replay. + *******************************************************************************/ + private CleanupOutcome finishUnlocked(QRecord pending) throws QException + { + if(pending.getValue("resolvedAt") != null) + { + return (CleanupOutcome.COMPLETED); + } + if(!AttachmentRecords.query(new QQueryFilter() + .withCriteria("bucketName", QCriteriaOperator.EQUALS, pending.getValue("bucketName")) + .withCriteria("storageKey", QCriteriaOperator.EQUALS, pending.getValue("storageKey"))).isEmpty()) + { + return (CleanupOutcome.REFERENCED); + } + if(pending.getValue("attachmentId") == null) + { + if(!storage.cleanupUpload(pending.getValueString("bucketName"), pending.getValueString("storageKey"))) + { + return (CleanupOutcome.UNCERTAIN_UPLOAD); + } + } + else + { + storage.delete(pending.getValueString("bucketName"), pending.getValueString("storageKey")); + } + DeleteInput delete = new DeleteInput(CollaborationTables.ATTACHMENT_CLEANUP).withPrimaryKeys(List.of(pending.getValue("id"))).withTransaction(QContext.getQBackendTransaction()); + delete.withPermissionBypass("Storage cleanup completed; remove its durable intent."); + AttachmentRecords.checkErrors(new DeleteAction().execute(delete).getRecordsWithErrors()); + return (CleanupOutcome.COMPLETED); + } + + + /******************************************************************************* + ** Rollback compensation must survive a provider outage. Persist independently + ** of the failed transaction, then attempt cleanup with the original context. + *******************************************************************************/ + void compensateUpload(String bucket, String key) + { + CapturedContext context = new CapturedContext(QContext.getQInstance(), QContext.getQSession()); + try + { + QContext.withTemporaryContext(context, () -> + { + InsertInput insert = new InsertInput(CollaborationTables.ATTACHMENT_CLEANUP).withRecords(List.of( + new QRecord().withValue("bucketName", bucket).withValue("storageKey", key))); + insert.withPermissionBypass("Failed authorized upload retains its immutable storage locator for compensation."); + List inserted = new InsertAction().execute(insert).getRecords(); + AttachmentRecords.checkErrors(inserted); + finish(inserted.getFirst()); + }); + } + catch(QException e) + { + throw (new IllegalStateException("Upload compensation could not complete.", e)); + } + } + + + /******************************************************************************* + ** Bounded recovery from committed intents, independent of the original service. + *******************************************************************************/ + public int retryPending() throws QException + { + requireOperator(); + if(QContext.getQBackendTransaction() != null) + { + throw (new QException("Attachment cleanup must run outside a caller transaction.")); + } + QueryInput query = new QueryInput(CollaborationTables.ATTACHMENT_CLEANUP).withFilter(new QQueryFilter() + .withCriteria("resolvedAt", QCriteriaOperator.IS_BLANK).withSubFilter(new QQueryFilter() + .withBooleanOperator(QQueryFilter.BooleanOperator.OR) + .withCriteria("nextAttemptAt", QCriteriaOperator.IS_BLANK) + .withCriteria("nextAttemptAt", QCriteriaOperator.LESS_THAN_OR_EQUALS, Instant.now())) + .withOrderBy(new QFilterOrderBy("id")).withLimit(100)); + query.withPermissionBypass("Authorized operator retries committed storage cleanup."); + List pending = new QueryAction().execute(query).getRecords(); + int completed = 0; + QException failure = null; + for(QRecord row : pending) + { + try + { + CleanupOutcome outcome = finish(row); + if(outcome == CleanupOutcome.COMPLETED) + { + completed++; + } + else if(outcome == CleanupOutcome.REFERENCED || outcome == CleanupOutcome.UNCERTAIN_UPLOAD) + { + defer(row); + } + } + catch(QException | RuntimeException e) + { + try + { + defer(row); + } + catch(QException | RuntimeException deferFailure) + { + e.addSuppressed(deferFailure); + } + if(failure == null) + { + failure = new QException("Attachment cleanup remains pending; retry after the provider or database recovers.", e); + } + } + } + if(failure != null) + { + throw (failure); + } + return (completed); + } + + + /******************************************************************************* + ** Persist backoff so failing entries do not monopolize the next bounded batch. + *******************************************************************************/ + private void defer(QRecord pending) throws QException + { + UpdateInput update = new UpdateInput(CollaborationTables.ATTACHMENT_CLEANUP).withRecords(List.of( + new QRecord().withValue("id", pending.getValue("id")).withValue("nextAttemptAt", Instant.now().plusSeconds(60)))); + update.withPermissionBypass("Committed cleanup failed or is not yet safe; retain it for a later attempt."); + AttachmentRecords.checkErrors(new UpdateAction().execute(update).getRecords()); + } + + + /******************************************************************************* + ** An identified administrator attests provider settlement; absence or age alone + ** is not evidence. Keep the receipt on the service-managed row permanently. + *******************************************************************************/ + public void resolveUpload(Serializable id, String evidence) throws QException + { + String actor = requireSettlementOperator(); + validateEvidence(evidence); + if(id == null || QContext.getQBackendTransaction() != null || !supportsUploadOwnership()) + { + throw (new QException("Upload settlement requires a record ID and an independent locking transaction.")); + } + try(QBackendTransaction transaction = QBackendTransaction.openFor(new UpdateInput(CollaborationTables.ATTACHMENT_CLEANUP))) + { + QContext.setTransaction(transaction); + try + { + QRecord pending = lock(new QRecord().withValue("id", id), transaction); + if(pending == null) + { + throw (new QException("Cleanup record is active or no longer exists. Retry after its owner finishes.")); + } + if(pending.getValue("resolvedAt") != null) + { + transaction.commit(); + return; + } + if(pending.getValue("attachmentId") != null) + { + throw (new QException("Use normal cleanup retry for committed attachment deletions.")); + } + if(!AttachmentRecords.query(new QQueryFilter() + .withCriteria("bucketName", QCriteriaOperator.EQUALS, pending.getValue("bucketName")) + .withCriteria("storageKey", QCriteriaOperator.EQUALS, pending.getValue("storageKey"))).isEmpty()) + { + throw (new QException("An attachment still references this object; settlement is not allowed.")); + } + storage.delete(pending.getValueString("bucketName"), pending.getValueString("storageKey")); + UpdateInput update = new UpdateInput(CollaborationTables.ATTACHMENT_CLEANUP).withTransaction(transaction) + .withRecords(List.of(new QRecord().withValue("id", pending.getValue("id")) + .withValue("resolvedAt", Instant.now()).withValue("resolvedBy", actor) + .withValue("settlementEvidence", evidence.strip()))); + update.withPermissionBypass("Identified collaboration administrator records confirmed upload settlement after storage cleanup."); + AttachmentRecords.checkErrors(new UpdateAction().execute(update).getRecords()); + transaction.commit(); + } + catch(QException | RuntimeException failure) + { + transaction.rollback(); + throw (failure); + } + finally + { + QContext.clearTransaction(); + } + } + } + + + /******************************************************************************* + ** The form and service both reject missing or oversized evidence. + *******************************************************************************/ + static void validateEvidence(String evidence) throws QException + { + if(evidence == null || evidence.isBlank() || evidence.strip().length() > 2000) + { + throw (new QException("Record provider completion or cancellation evidence (1–2000 characters).")); + } + } + + + /******************************************************************************* + ** Settlement is a deliberate, attributable admin action, never scheduled work. + *******************************************************************************/ + static String requireSettlementOperator() throws QPermissionDeniedException + { + var session = QContext.getQSession(); + String actor = CollaborationSecurity.userId(session); + if(session == null || session instanceof QSystemUserSession || !session.hasPermission(CollaborationPermissions.ADMIN) + || actor == null || actor.isBlank() || actor.length() > 255) + { + throw (new QPermissionDeniedException("An identified collaboration administrator must confirm upload settlement.")); + } + return (actor); + } + + + /******************************************************************************* + ** Scheduled system work and explicit admin recovery share one authorization rule. + *******************************************************************************/ + static void requireOperator() throws QPermissionDeniedException + { + var session = QContext.getQSession(); + if(!(session instanceof QSystemUserSession) && (session == null || !session.hasPermission(CollaborationPermissions.ADMIN))) + { + throw (new QPermissionDeniedException("This needs the collaboration.admin permission.")); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentContent.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentContent.java new file mode 100644 index 0000000..38e7f8f --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentContent.java @@ -0,0 +1,50 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + +/******************************************************************************* + ** Verified bytes returned only after authorization, integrity and audit checks. + ** Storage coordinates never cross this boundary. Content is bounded to 20 MiB. + *******************************************************************************/ +public record AttachmentContent(Integer attachmentId, String fileName, String contentType, String sha256, byte[] bytes) +{ + /******************************************************************************* + ** Own the byte array so a caller cannot change a retained receipt's content. + *******************************************************************************/ + public AttachmentContent + { + if(bytes == null || bytes.length > 20 * 1024 * 1024) + { + throw (new IllegalArgumentException("Attachment content is missing or exceeds the supported limit.")); + } + bytes = bytes.clone(); + } + + /******************************************************************************* + ** Return caller-owned bytes without exposing the immutable content buffer. + *******************************************************************************/ + @Override + public byte[] bytes() + { + return (bytes.clone()); + } +} \ No newline at end of file diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentContentIntegrityException.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentContentIntegrityException.java new file mode 100644 index 0000000..efb5d15 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentContentIntegrityException.java @@ -0,0 +1,41 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + + +import com.kingsrook.qqq.backend.core.exceptions.QException; + + +/******************************************************************************* + ** Exact stored content failed its immutable metadata check. Consumers must not + ** retry this as a transport outage or release the unverified bytes. + *******************************************************************************/ +public class AttachmentContentIntegrityException extends QException +{ + /******************************************************************************* + ** Fixed public diagnostic, with no object locator or provider details. + *******************************************************************************/ + public AttachmentContentIntegrityException() + { + super("Attachment content failed integrity verification."); + } +} \ No newline at end of file diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentDownload.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentDownload.java new file mode 100644 index 0000000..42865c7 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentDownload.java @@ -0,0 +1,54 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + +import java.net.URL; +import java.util.Objects; + +/******************************************************************************* + ** A download destination may require the application's authenticated transport. + ** The URL is sensitive and never attests that bytes were served. + *******************************************************************************/ +public record AttachmentDownload(URL url, boolean authenticated) +{ + /*************************************************************************** + ** Only ordinary HTTP destinations are eligible for attachment delivery. + ***************************************************************************/ + public AttachmentDownload + { + Objects.requireNonNull(url, "url"); + if((!"https".equalsIgnoreCase(url.getProtocol()) && !"http".equalsIgnoreCase(url.getProtocol())) + || url.getUserInfo() != null || url.getRef() != null || url.getHost() == null || url.getHost().isBlank()) + { + throw new IllegalArgumentException("Attachment download requires HTTP or HTTPS"); + } + } + + /*************************************************************************** + ** Never disclose storage signatures through ordinary object logging. + ***************************************************************************/ + @Override + public String toString() + { + return ("AttachmentDownload[authenticated=" + authenticated + ", url=]"); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentEntry.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentEntry.java index 236938a..0358346 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentEntry.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentEntry.java @@ -27,6 +27,14 @@ import java.time.Instant; ** Panel DTO. Storage locations and ephemeral signed URLs stay server-side. *******************************************************************************/ public record AttachmentEntry(Integer id, String tableName, String recordId, String fileName, String contentType, - Long sizeBytes, String sha256, String description, String createdByUserId, Instant createDate, Boolean canDelete) + Long sizeBytes, String sha256, String description, String createdByUserId, Instant createDate, Boolean canDelete, Boolean canEdit) { + /******************************************************************************* + ** Preserve source compatibility for callers using the earlier DTO constructor. + *******************************************************************************/ + public AttachmentEntry(Integer id, String tableName, String recordId, String fileName, String contentType, + Long sizeBytes, String sha256, String description, String createdByUserId, Instant createDate, Boolean canDelete) + { + this(id, tableName, recordId, fileName, contentType, sizeBytes, sha256, description, createdByUserId, createDate, canDelete, canDelete); + } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentLifecyclePolicy.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentLifecyclePolicy.java new file mode 100644 index 0000000..22de10f --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentLifecyclePolicy.java @@ -0,0 +1,214 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.data.QRecord; + +/******************************************************************************* + ** Trusted host extension for attachment lifecycle policy. The default preserves + ** ordinary Collaboration behavior. Implementations must be concurrency-safe and + ** must not retain request context. Callbacks receive defensive record copies. + ** This is not an untrusted scripting API or a new file-storage abstraction. + *******************************************************************************/ +public class AttachmentLifecyclePolicy +{ + /******************************************************************************* + ** Configuration-only capability policy, evaluated during metadata production + ** without a session. Dynamic parent visibility owners opt in to pinned counts. + *******************************************************************************/ + public boolean requiresPinnedAggregateParent() + { + return (false); + } + + /******************************************************************************* + ** An installed retention owner must authorize an exact archival reference + ** transfer in the caller's real transaction. Ordinary policies deny this. + *******************************************************************************/ + public void beforeReferenceTransfer(QRecord attachment, String tableName, String recordId, QBackendTransaction transaction) throws QException + { + throw (new QException("Retained reference transfer requires an archive policy.")); + } + + /******************************************************************************* + ** Persist the reference-transfer outcome in the same transaction. The policy + ** must not commit or close the transaction; failure requires caller rollback. + *******************************************************************************/ + public void afterReferenceTransfer(QRecord attachment, String tableName, String recordId, QBackendTransaction transaction) throws QException + { + throw (new QException("Retained reference transfer requires an atomic archive audit policy.")); + } + + /******************************************************************************* + ** Called with canonical source, actual digest/size and persisted retention + ** decision immediately before the first object write. There is no row ID yet. + ** Retained originals require an independently durable intent before storage. + *******************************************************************************/ + public void beforeStored(QRecord attachment) throws QException + { + if(Boolean.TRUE.equals(attachment.getValueBoolean("retained"))) + { + throw (new QException("Retained storage requires a durable write-intent policy.")); + } + } + + /******************************************************************************* + ** Recheck current adoption authority after external operations, without + ** creating another intent or granting access to the original's bytes. + *******************************************************************************/ + public void authorizeAdoption(QRecord attachment) throws QException + { + throw (new QException("Existing-original adoption is not configured.")); + } + + /******************************************************************************* + ** Explicit adoption of an existing owner original, not a new upload. The + ** policy must authorize the current source and commit its intent before read. + *******************************************************************************/ + public void beforeAdoption(QRecord attachment) throws QException + { + throw (new QException("Existing-original adoption requires an audited retention policy.")); + } + + /******************************************************************************* + ** Record the retention transition in the owner's transaction. Registration + ** follows separately; failure after commit leaves a retained retryable row. + *******************************************************************************/ + public void afterAdoption(QRecord attachment, QBackendTransaction transaction) throws QException + { + throw (new QException("Existing-original adoption requires an atomic retention audit.")); + } + + /******************************************************************************* + ** Called after storage and metadata insertion, before upload returns. Failure + ** leaves the original intact for reconciliation; it is not a rollback hook. + ** Enforce incomplete registration/quarantine in beforeAccess as appropriate. + *******************************************************************************/ + public void afterStored(QRecord attachment) throws QException + { + } + + /******************************************************************************* + ** Before reading an existing original for registration recovery. This is not + ** a write attempt or a user download; retained owners must audit it explicitly. + *******************************************************************************/ + public void beforeRecovery(QRecord candidate) throws QException + { + if(Boolean.TRUE.equals(candidate.getValueBoolean("retained"))) + { + throw (new QException("Retained recovery requires a durable recovery-attempt policy.")); + } + } + + /******************************************************************************* + ** Called after parent authorization and before either preview or download URL + ** signing. Throw to deny. A successful audit here means access was requested, + ** not that bytes were downloaded. Mandatory audit must commit independently + ** before returning; the caller transaction may still roll back afterwards. + *******************************************************************************/ + public void beforeAccess(QRecord attachment, boolean inline) throws QException + { + } + + /******************************************************************************* + ** Optional authenticated delivery after ordinary parent/lifecycle checks. + ** Return null to retain the existing storage URL. No alternate lookup occurs. + *******************************************************************************/ + public AttachmentDownload downloadDelivery(QRecord attachment) throws QException + { + return (null); + } + + /******************************************************************************* + ** Authorize and durably audit a server-side content read, distinct from URL + ** issuance. Retained content requires an implementation of this new contract. + *******************************************************************************/ + public void beforeContentRead(QRecord attachment) throws QException + { + if(Boolean.TRUE.equals(attachment.getValueBoolean("retained"))) + { + throw (new QException("Retained content requires an audited content-read policy.")); + } + } + + /******************************************************************************* + ** Persist the verified server-read result before content leaves the service. + ** This does not attest user download, printer spooling or physical output. + *******************************************************************************/ + public void afterContentRead(QRecord attachment, Long bytesRead) throws QException + { + if(Boolean.TRUE.equals(attachment.getValueBoolean("retained"))) + { + throw (new QException("Retained content requires an audited read-result policy.")); + } + } + + /******************************************************************************* + ** Return true to prohibit deletion and preserve bytes on parent cleanup. + ** Called during intake before storage with complete metadata but no row id; + ** true is persisted on the attachment and cannot be cleared by this API. + ** It is also queried for legacy rows during deletion/read hints. Failure + ** denies deletion. Do not depend on parent existence or caller permissions. + *******************************************************************************/ + public boolean isRetained(QRecord attachment) throws QException + { + return (false); + } + + /******************************************************************************* + ** Called after ownership/parent checks and before applying display metadata. + ** The second record contains id and proposed fields, not the complete row. + ** This records an intent only; it does not attest that a transaction committed. + *******************************************************************************/ + public void beforeMetadataChange(QRecord attachment, QRecord changes) throws QException + { + } + + /******************************************************************************* + ** Validate the actual owner transaction before applying display metadata. + ** Required providers must reject unsupported/no-op transactions and incompatible + ** audit backends before the first write. This method does not commit anything. + *******************************************************************************/ + public void validateMetadataTransaction(QRecord attachment, QBackendTransaction transaction) throws QException + { + if(Boolean.TRUE.equals(attachment.getValueBoolean("retained"))) + { + throw (new QException("Retained metadata requires an atomic audit transaction policy.")); + } + } + + /******************************************************************************* + ** Record the successful display metadata change in the supplied transaction + ** before its owner commits. Required providers must validate real transaction + ** and backend compatibility, insert their event using this transaction and + ** throw on failure. Never commit or close the supplied owner transaction. + *******************************************************************************/ + public void afterMetadataChange(QRecord attachment, QRecord changes, QBackendTransaction transaction) throws QException + { + if(Boolean.TRUE.equals(attachment.getValueBoolean("retained"))) + { + throw (new QException("Retained metadata requires an atomic audit-result policy.")); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentMediaPolicy.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentMediaPolicy.java new file mode 100644 index 0000000..3630546 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentMediaPolicy.java @@ -0,0 +1,87 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + +import java.util.Locale; +import java.util.Set; + +/******************************************************************************* + ** What an uploaded object is allowed to tell a browser. The declared media type + ** is client-controlled: it is kept as display metadata, but the object itself is + ** stored with an inert type and an inline disposition only when the type is a + ** known passive image, PDF, plain-text, audio or video format. Anything else (HTML, SVG, + ** XML, script, unknown) is stored as an opaque attachment, so a signed storage + ** link can never render or execute uploaded active content inline. + *******************************************************************************/ +final class AttachmentMediaPolicy +{ + static final String OPAQUE = "application/octet-stream"; + + private static final Set PASSIVE_INLINE = Set.of( + "image/png", "image/jpeg", "image/gif", "image/webp", "application/pdf", "text/plain", + "audio/mpeg", "audio/mp4", "audio/ogg", "audio/wav", "audio/webm", + "video/mp4", "video/webm", "video/ogg"); + + /*************************************************************************** + ** Static policy only. + ***************************************************************************/ + private AttachmentMediaPolicy() + { + } + + + + /*************************************************************************** + ** The base media type of a declared value: lowercase, without parameters. + ** Anything that is not a plain type/subtype token is treated as unknown. + ***************************************************************************/ + static String baseType(String declared) + { + if(declared == null) + { + return (OPAQUE); + } + String base = declared.split(";", 2)[0].trim().toLowerCase(Locale.ROOT); + return (base.matches("[a-z0-9][a-z0-9!#$&^_.+-]{0,126}/[a-z0-9][a-z0-9!#$&^_.+-]{0,126}") ? base : OPAQUE); + } + + + + /*************************************************************************** + ** The media type the stored object will be served with. + ***************************************************************************/ + static String storedType(String declared) + { + String base = baseType(declared); + return (PASSIVE_INLINE.contains(base) ? base : OPAQUE); + } + + + + /*************************************************************************** + ** Inline only for passive types; every other object is an attachment. + ***************************************************************************/ + static String disposition(String declared) + { + return (PASSIVE_INLINE.contains(baseType(declared)) ? "inline" : "attachment"); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentPreviewCustomizer.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentPreviewCustomizer.java index f08972d..bee456c 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentPreviewCustomizer.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentPreviewCustomizer.java @@ -46,13 +46,13 @@ public class AttachmentPreviewCustomizer implements TableCustomizerInterface @Override public List postQuery(QueryOrGetInputInterface input, List records) throws QException { - return (filter(records, input instanceof GetInput, new AttachmentsService())); + return (filter(records, input instanceof GetInput, input.getShouldGenerateDisplayValues(), new AttachmentsService())); } /******************************************************************************* ** Keep filtering every row even when the preview generation cap is reached. *******************************************************************************/ - List filter(List records, boolean isGet, AttachmentsService service) throws QException + List filter(List records, boolean isGet, boolean generateUrls, AttachmentsService service) throws QException { List visible = new ArrayList<>(); for(QRecord record : records) @@ -77,15 +77,21 @@ public class AttachmentPreviewCustomizer implements TableCustomizerInterface QRecord row = new QRecord(record); row.setValue("previewUrl", null); row.setValue("downloadUrl", null); + row.setValue("previewDelivery", null); + row.setValue("downloadDelivery", null); row.withDisplayValue("downloadUrl", null).withDisplayValue("downloadUrl:downloadUrlDynamic", null); visible.add(row); - if(visible.size() <= 200) + if(generateUrls && visible.size() <= 200) { try { - row.setValue("previewUrl", service.url(row, true).toString()); - String downloadUrl = service.url(row, false).toString(); + AttachmentDownload preview = service.delivery(row, true); + AttachmentDownload download = service.delivery(row, false); + row.setValue("previewUrl", preview.url().toString()); + row.setValue("previewDelivery", preview.authenticated() ? "AUTHENTICATED_FETCH" : "SIGNED_URL"); + String downloadUrl = download.url().toString(); row.setValue("downloadUrl", downloadUrl); + row.setValue("downloadDelivery", download.authenticated() ? "AUTHENTICATED_FETCH" : "SIGNED_URL"); row.withDisplayValue("downloadUrl", row.getValueString("fileName")) .withDisplayValue("downloadUrl:downloadUrlDynamic", downloadUrl); } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentStorage.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentStorage.java index 73e7f50..7975720 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentStorage.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentStorage.java @@ -22,13 +22,17 @@ package com.kingsrook.qbits.collaboration.attachments; import java.net.URL; +import java.util.Optional; import java.util.UUID; import com.kingsrook.qbits.objectstorage.api.DeleteMode; import com.kingsrook.qbits.objectstorage.api.DeleteRequest; import com.kingsrook.qbits.objectstorage.api.ObjectStorage; +import com.kingsrook.qbits.objectstorage.api.ObjectStream; import com.kingsrook.qbits.objectstorage.api.PresignGet; import com.kingsrook.qbits.objectstorage.api.PutRequest; +import com.kingsrook.qbits.objectstorage.api.Range; import com.kingsrook.qbits.objectstorage.api.StorageBucket; +import com.kingsrook.qbits.objectstorage.api.StoredObject; /******************************************************************************* ** Narrow storage boundary. The caller must check collaboration access first. @@ -62,6 +66,22 @@ public class AttachmentStorage bucket(name).put(request); } + /******************************************************************************* + ** Open content for an already-authorized caller. The service owns closing it. + *******************************************************************************/ + public ObjectStream open(String name, String key) + { + return (bucket(name).get(key, Range.all())); + } + + /******************************************************************************* + ** Inspect a deterministic intake key after uncertain conditional creation. + *******************************************************************************/ + public Optional head(String name, String key) + { + return (bucket(name).head(key)); + } + /******************************************************************************* ** Prefer soft deletion; allow permanent deletion only where no soft mode exists. *******************************************************************************/ @@ -73,6 +93,18 @@ public class AttachmentStorage } } + /******************************************************************************* + ** Remove failed-upload bytes and report whether no earlier PUT can still finish. + ** Generic object storage cannot prove remote settlement after a timeout. Keep + ** its durable locator for later sweeps. Adapters may return true only with an + ** authoritative completion/cancellation guarantee, never from object absence. + *******************************************************************************/ + public boolean cleanupUpload(String name, String key) + { + delete(name, key); + return (false); + } + /******************************************************************************* ** Create a temporary link, never a persistent table value. *******************************************************************************/ diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentsService.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentsService.java index c10b958..c4a3216 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentsService.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentsService.java @@ -21,6 +21,8 @@ package com.kingsrook.qbits.collaboration.attachments; +import java.io.ByteArrayOutputStream; +import java.io.DataOutputStream; import java.io.IOException; import java.io.InputStream; import java.io.OutputStream; @@ -37,14 +39,17 @@ import java.util.ArrayList; import java.util.HexFormat; import java.util.List; import java.util.Locale; +import java.util.Map; import java.util.UUID; import com.kingsrook.qbits.collaboration.CollaborationQBitConfig; import com.kingsrook.qbits.collaboration.CollaborationSecurity; import com.kingsrook.qbits.collaboration.model.CollaborationTables; import com.kingsrook.qbits.collaboration.permissions.CollaborationPermissions; import com.kingsrook.qbits.collaboration.security.ParentRecordAccess; +import com.kingsrook.qbits.objectstorage.api.ObjectStream; import com.kingsrook.qbits.objectstorage.api.PresignGet; import com.kingsrook.qbits.objectstorage.api.PutRequest; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction; import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction; @@ -68,6 +73,8 @@ public class AttachmentsService { private final CollaborationQBitConfig config; private final AttachmentStorage storage; + private final AttachmentLifecyclePolicy lifecycle; + private final com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadJournal uploadJournal; /******************************************************************************* ** Resolve the installed configuration without a global mutable singleton. @@ -77,15 +84,46 @@ public class AttachmentsService this.config = (CollaborationQBitConfig) QContext.getQInstance().getQBits().values().stream() .filter(qbit -> qbit.getConfig() instanceof CollaborationQBitConfig).findFirst().orElseThrow().getConfig(); this.storage = com.kingsrook.qqq.backend.core.actions.customizers.QCodeLoader.getAdHoc(AttachmentStorage.class, config.getAttachmentStorage()); + this.lifecycle = resolveLifecycle(config); + this.uploadJournal = new com.kingsrook.qbits.collaboration.attachments.uploadjournal.JdbcUploadJournal(); } /******************************************************************************* ** Injectable storage boundary for deterministic service tests. *******************************************************************************/ public AttachmentsService(CollaborationQBitConfig config, AttachmentStorage storage) + { + this(config, storage, resolveLifecycle(config)); + } + + /******************************************************************************* + ** Package-local deterministic lifecycle seam for service tests. + *******************************************************************************/ + AttachmentsService(CollaborationQBitConfig config, AttachmentStorage storage, AttachmentLifecyclePolicy lifecycle) + { + this(config, storage, lifecycle, new com.kingsrook.qbits.collaboration.attachments.uploadjournal.JdbcUploadJournal()); + } + + /******************************************************************************* + ** Explicit test-only package seam; public services always use the durable + ** qualified journal. It cannot be replaced through application configuration. + *******************************************************************************/ + AttachmentsService(CollaborationQBitConfig config, AttachmentStorage storage, AttachmentLifecyclePolicy lifecycle, + com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadJournal uploadJournal) { this.config = config; this.storage = storage; + this.lifecycle = java.util.Objects.requireNonNull(lifecycle, "lifecycle"); + this.uploadJournal = java.util.Objects.requireNonNull(uploadJournal, "uploadJournal"); + } + + /******************************************************************************* + ** Invalid policy configuration must fail before any storage side effect. + *******************************************************************************/ + private static AttachmentLifecyclePolicy resolveLifecycle(CollaborationQBitConfig config) + { + return (java.util.Objects.requireNonNull(com.kingsrook.qqq.backend.core.actions.customizers.QCodeLoader + .getAdHoc(AttachmentLifecyclePolicy.class, config.getAttachmentLifecycle()), "Could not load attachment lifecycle policy.")); } /******************************************************************************* @@ -93,6 +131,36 @@ public class AttachmentsService ** The caller owns the input stream. Oversize files never reach object storage. *******************************************************************************/ public AttachmentEntry upload(String tableName, Serializable recordId, String fileName, String contentType, InputStream stream, String description) throws QException + { + return (uploadInternal(tableName, recordId, fileName, contentType, stream, description, null, null)); + } + + /******************************************************************************* + ** Retryable intake for a server coordinator. Keys and manifests are SHA-256 + ** identifiers; keys are additionally scoped to actual actor and parent here. + ** This path owns persistence and refuses ambient transactions. Replays verify + ** supplied bytes and rerun registration; they never create another original. + *******************************************************************************/ + public AttachmentEntry upload(String tableName, Serializable recordId, String fileName, String contentType, InputStream stream, String description, + String intakeKey, String manifestSha256) throws QException + { + if(intakeKey == null || !intakeKey.matches("[0-9a-f]{64}") || manifestSha256 == null || !manifestSha256.matches("[0-9a-f]{64}")) + { + throw (new QBadRequestException("Intake key and manifest must be SHA-256 identifiers.")); + } + if(QContext.getQBackendTransaction() != null) + { + throw (new QException("Retryable attachment intake cannot join an ambient transaction.")); + } + return (uploadInternal(tableName, recordId, fileName, contentType, stream, description, intakeKey, manifestSha256)); + } + + /******************************************************************************* + ** Shared bounded intake; the retryable path preserves uncertain objects for + ** reconciliation under a deterministic key and never overwrites them. + *******************************************************************************/ + private AttachmentEntry uploadInternal(String tableName, Serializable recordId, String fileName, String contentType, InputStream stream, String description, + String intakeKey, String manifestSha256) throws QException { new com.kingsrook.qbits.collaboration.people.PeopleService().touch(); ParentRecordAccess.requireFeature(tableName, recordId, "attach"); @@ -100,13 +168,19 @@ public class AttachmentsService String id = canonicalId(tableName, recordId); String name = sanitize(fileName); String bucket = config.getAttachmentsBucketName(); - String key = "attachments/" + segment(tableName) + "/" + segment(id) + "/" + UUID.randomUUID() + "/" + name; + String scopedKey = intakeKey == null ? null : tupleHash("attachment-intake-v1", user, tableName, id, intakeKey); + String key = scopedKey == null ? "attachments/" + segment(tableName) + "/" + segment(id) + "/" + UUID.randomUUID() + "/" + name + : "attachments/intake/" + scopedKey; if(tableName.length() > 128 || id.length() > 64 || user.length() > 255 || bucket.length() > 255 || key.length() > 1024 || (contentType != null && contentType.length() > 255)) { throw (new QBadRequestException("Attachment metadata exceeds the supported field lengths.")); } long limit = Math.min(config.getDefaultMaxBytes(), storage.maxBytes(bucket)); + if(scopedKey != null) + { + limit = Math.min(limit, 20L * 1024 * 1024); + } Path temporary = null; try { @@ -129,30 +203,69 @@ public class AttachmentsService } } String type = contentType == null || contentType.isBlank() ? "application/octet-stream" : contentType; - try(InputStream upload = Files.newInputStream(temporary)) + QRecord row = new QRecord().withValue("tableName", tableName).withValue("recordId", id).withValue("fileName", name) + .withValue("contentType", type).withValue("sizeBytes", size).withValue("sha256", HexFormat.of().formatHex(digest.digest())) + .withValue("bucketName", bucket).withValue("storageKey", key).withValue("description", description) + .withValue("createdByUserId", user).withValue("parentDeleted", false); + row.withValue("retained", lifecycle.isRetained(new QRecord(row))); + if(scopedKey != null) { - storage.put(bucket, PutRequest.of(key, upload).withContentLength(size).withContentType(type).withContentDisposition("inline")); + String fingerprint = tupleHash("attachment-manifest-v1", manifestSha256, name, type, description, String.valueOf(size), row.getValueString("sha256")); + row.withValue("intakeKey", scopedKey).withValue("intakeFingerprint", fingerprint); + QRecord prior = findIntake(scopedKey); + if(prior != null) + { + if(Boolean.TRUE.equals(prior.getValueBoolean("retained")) && uploadJournal.findByLocation(bucket, key).isPresent()) + { + return (journaledUpload(row, temporary)); + } + return (replayIntake(prior, fingerprint)); + } + if(Boolean.TRUE.equals(row.getValueBoolean("retained"))) + { + return (journaledUpload(row, temporary)); + } + beforeObjectWrite(row); + createIntakeObject(bucket, key, Files.readAllBytes(temporary), type, fingerprint); + } + else + { + if(Boolean.TRUE.equals(row.getValueBoolean("retained"))) + { + return (journaledUpload(row, temporary)); + } + beforeObjectWrite(row); + //////////////////////////////////////////////////////////////////////// + // ordinary uploads keep the transactional cleanup-queue path (355b06d) // + //////////////////////////////////////////////////////////////////////// + return (storeUpload(row, temporary)); } boolean rowSaved = false; try { - QRecord row = new QRecord().withValue("tableName", tableName).withValue("recordId", id).withValue("fileName", name) - .withValue("contentType", type).withValue("sizeBytes", size).withValue("sha256", HexFormat.of().formatHex(digest.digest())) - .withValue("bucketName", bucket).withValue("storageKey", key).withValue("description", description) - .withValue("createdByUserId", user).withValue("parentDeleted", false); InsertInput input = new InsertInput(CollaborationTables.ATTACHMENT).withTransaction(QContext.getQBackendTransaction()).withRecords(List.of(row)); input.withPermissionBypass("AttachmentsService checked parent and attach permission before upload."); List records = new InsertAction().execute(input).getRecords(); AttachmentRecords.checkErrors(records); rowSaved = true; + lifecycle.afterStored(new QRecord(records.getFirst())); com.kingsrook.qbits.collaboration.activity.ActivityWriter.write(tableName, id, "FILE_ADDED", "File added", CollaborationTables.ATTACHMENT, records.getFirst().getValueString("id")); return (entry(records.getFirst())); } catch(QException | RuntimeException failure) { + if(scopedKey != null) + { + QRecord prior = findIntake(scopedKey); + if(!rowSaved && prior != null) + { + return (replayIntake(prior, row.getValueString("intakeFingerprint"))); + } + throw (failure); + } try { - if(!rowSaved) + if(!rowSaved && !Boolean.TRUE.equals(row.getValueBoolean("retained"))) { storage.delete(bucket, key); } @@ -184,23 +297,317 @@ public class AttachmentsService } } + /******************************************************************************* + ** On transactional locking backends, own a transaction only after spooling + ** the bounded input. An existing outer transaction retains the upload lock. + *******************************************************************************/ + private AttachmentEntry storeUpload(QRecord row, Path temporary) throws QException, IOException + { + var transaction = QContext.getQBackendTransaction(); + if(transaction == null && AttachmentCleanupQueue.supportsUploadOwnership()) + { + try(var owned = com.kingsrook.qqq.backend.core.actions.QBackendTransaction.openFor(new InsertInput(CollaborationTables.ATTACHMENT))) + { + QContext.setTransaction(owned); + try + { + AttachmentEntry result = storeUpload(row, temporary); + owned.commit(); + return (result); + } + catch(QException | IOException | RuntimeException failure) + { + try + { + owned.rollback(); + } + catch(QException rollbackFailure) + { + failure.addSuppressed(rollbackFailure); + } + throw (failure); + } + finally + { + QContext.clearTransaction(); + } + } + } + if(transaction != null && !transaction.supportsRollbackCallbacks()) + { + throw (new QException("Attachment uploads require a transaction with confirmed rollback support.")); + } + String bucket = row.getValueString("bucketName"); + String key = row.getValueString("storageKey"); + AttachmentCleanupQueue cleanup = new AttachmentCleanupQueue(storage); + QRecord pending = cleanup.beginUpload(bucket, key); + try(InputStream upload = Files.newInputStream(temporary)) + { + storage.put(bucket, PutRequest.of(key, upload).withContentLength(row.getValueLong("sizeBytes")) + .withContentType(AttachmentMediaPolicy.storedType(row.getValueString("contentType"))) + .withContentDisposition(AttachmentMediaPolicy.disposition(row.getValueString("contentType")))); + } + catch(IOException | RuntimeException failure) + { + if(pending == null) + { + try + { + cleanup.compensateUpload(bucket, key); + } + catch(RuntimeException cleanupFailure) + { + failure.addSuppressed(cleanupFailure); + } + } + throw (failure); + } + if(transaction != null && pending == null) + { + var context = new com.kingsrook.qqq.backend.core.context.CapturedContext(QContext.getQInstance(), QContext.getQSession()); + transaction.addAfterRollbackCallback(() -> QContext.withTemporaryContext(context, + () -> cleanup.compensateUpload(bucket, key))); + } + boolean rowSaved = false; + try + { + InsertInput input = new InsertInput(CollaborationTables.ATTACHMENT).withTransaction(QContext.getQBackendTransaction()).withRecords(List.of(row)); + input.withPermissionBypass("AttachmentsService checked parent and attach permission before upload."); + List records = new InsertAction().execute(input).getRecords(); + AttachmentRecords.checkErrors(records); + rowSaved = true; + cleanup.completeUpload(pending); + lifecycle.afterStored(new QRecord(records.getFirst())); + com.kingsrook.qbits.collaboration.activity.ActivityWriter.write(row.getValueString("tableName"), row.getValueString("recordId"), + "FILE_ADDED", "File added", CollaborationTables.ATTACHMENT, records.getFirst().getValueString("id")); + return (entry(records.getFirst())); + } + catch(QException | RuntimeException failure) + { + if(!rowSaved && pending == null) + { + try + { + cleanup.compensateUpload(bucket, key); + } + catch(RuntimeException cleanupFailure) + { + failure.addSuppressed(cleanupFailure); + } + } + throw (failure); + } + } + + /******************************************************************************* + ** Retained bytes cannot be written before durable intent or under an ambient + ** transaction whose later rollback could lose their metadata association. + *******************************************************************************/ + private void beforeObjectWrite(QRecord row) throws QException + { + if(Boolean.TRUE.equals(row.getValueBoolean("retained")) && QContext.getQBackendTransaction() != null) + { + throw (new QException("Retained attachment storage cannot join an ambient transaction.")); + } + lifecycle.beforeStored(new QRecord(row)); + } + + /******************************************************************************* + ** Retained originals use qualified bounded conditional creation and durable + ** owner recovery; ordinary uploads retain the established streaming behavior. + *******************************************************************************/ + private AttachmentEntry journaledUpload(QRecord row, Path temporary) throws QException, IOException + { + if(row.getValueLong("sizeBytes") > JournaledAttachmentUpload.MAX_BYTES) + { + throw (new QBadRequestException("Retained uploads are limited to 20 MiB by the qualified conditional storage path.")); + } + var receipt = new JournaledAttachmentUpload(storage, lifecycle, uploadJournal).store(row, Files.readAllBytes(temporary)); + QRecord stored = receipt.attachment(); + if(!receipt.alreadyComplete()) + { + com.kingsrook.qbits.collaboration.activity.ActivityWriter.write(stored.getValueString("tableName"), stored.getValueString("recordId"), + "FILE_ADDED", "File added", CollaborationTables.ATTACHMENT, stored.getValueString("id")); + } + return (entry(stored)); + } + + /******************************************************************************* + ** Native confirmation discovery checks the configured current adoption policy. + ** No original bytes are read and no audit success is asserted during review. + *******************************************************************************/ + public QRecord requireAdoptable(Integer attachmentId) throws QException + { + QRecord row = requireMutable(attachmentId); + lifecycle.authorizeAdoption(new QRecord(row)); + return (row); + } + + /******************************************************************************* + ** Adopt this exact visible original without uploading, copying or deleting + ** storage. Registration retries retain the same owner attachment identity. + *******************************************************************************/ + public AttachmentEntry adoptRetainedAttachment(Integer attachmentId) throws QException + { + new ExistingAttachmentAdoption(this, storage, lifecycle).adopt(attachmentId); + return (entry(requireRow(attachmentId))); + } + + /******************************************************************************* + ** Recover a visible retained upload without accepting bytes, locators or an + ** actor from the client. Original-actor and current-parent checks are mandatory. + *******************************************************************************/ + public AttachmentEntry recoverRetainedUpload(Integer journalId) throws QException + { + return (entry(new JournaledAttachmentUpload(storage, lifecycle, uploadJournal).recover(journalId))); + } + + /******************************************************************************* + ** Conditional single-request puts retain the same original across retries. + ** Byte arrays deliberately avoid a multipart provider path whose conditional + ** completion support is not qualified. The caller enforces a 20 MiB limit. + *******************************************************************************/ + private void createIntakeObject(String bucket, String key, byte[] bytes, String type, String fingerprint) throws QException + { + try + { + storage.put(bucket, PutRequest.of(key, bytes).withContentLength((long) bytes.length).withContentType(AttachmentMediaPolicy.storedType(type)) + .withContentDisposition("attachment").withIfNoneMatch(true).withMetadata(Map.of("qqq-intake-fingerprint", fingerprint))); + } + catch(RuntimeException uncertain) + { + try + { + if(storage.head(bucket, key).isEmpty()) + { + throw (uncertain); + } + try(ObjectStream existing = storage.open(bucket, key)) + { + byte[] original = existing.getInputStream().readNBytes(bytes.length + 1); + if(existing.getHeaders() == null || existing.getHeaders().getUserMetadata() == null + || !fingerprint.equals(existing.getHeaders().getUserMetadata().get("qqq-intake-fingerprint")) || !MessageDigest.isEqual(original, bytes)) + { + throw (new QException("An intake request cannot replace an existing original.")); + } + } + } + catch(IOException | RuntimeException verificationFailure) + { + throw (new QException("Could not reconcile attachment intake storage.", verificationFailure)); + } + } + } + + /******************************************************************************* + ** Internal lookup is keyed by authenticated actor and canonical parent scope. + *******************************************************************************/ + private QRecord findIntake(String scopedKey) throws QException + { + List rows = AttachmentRecords.query(new QQueryFilter().withCriteria("intakeKey", QCriteriaOperator.EQUALS, scopedKey).withLimit(1)); + return (rows.isEmpty() ? null : rows.getFirst()); + } + + /******************************************************************************* + ** Reauthorize persisted identity and retry idempotent owner registration. + *******************************************************************************/ + private AttachmentEntry replayIntake(QRecord row, String fingerprint) throws QException + { + if(Boolean.TRUE.equals(row.getValueBoolean("parentDeleted"))) + { + throw (new QNotFoundException("Not found")); + } + ParentRecordAccess.requireFeature(row.getValueString("tableName"), row.getValueString("recordId"), "attach"); + if(!fingerprint.equals(row.getValueString("intakeFingerprint"))) + { + throw (new QBadRequestException("Intake request conflicts with its original manifest.")); + } + lifecycle.afterStored(new QRecord(row)); + return (entry(row)); + } + + /******************************************************************************* + ** Length-prefix all tuple members, including null, so delimiters cannot collide. + *******************************************************************************/ + private String tupleHash(String... values) throws QException + { + try + { + ByteArrayOutputStream buffer = new ByteArrayOutputStream(); + try(DataOutputStream output = new DataOutputStream(buffer)) + { + for(String value : values) + { + byte[] bytes = value == null ? new byte[0] : value.getBytes(StandardCharsets.UTF_8); + output.writeInt(value == null ? -1 : bytes.length); + output.write(bytes); + } + } + return (HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(buffer.toByteArray()))); + } + catch(IOException | NoSuchAlgorithmException e) + { + throw (new QException("Could not identify attachment intake.", e)); + } + } + /******************************************************************************* ** Newest first, independently of collaboration.view. *******************************************************************************/ public List list(String tableName, Serializable recordId) throws QException { - new com.kingsrook.qbits.collaboration.people.PeopleService().touch(); - String id = canonicalId(tableName, recordId); List entries = new ArrayList<>(); - for(QRecord row : AttachmentRecords.query(new QQueryFilter().withCriteria("tableName", QCriteriaOperator.EQUALS, tableName) - .withCriteria("recordId", QCriteriaOperator.EQUALS, id).withCriteria("parentDeleted", QCriteriaOperator.NOT_EQUALS, true) - .withOrderBy(new QFilterOrderBy("createDate", false)).withOrderBy(new QFilterOrderBy("id", false)))) + for(QRecord row : AttachmentRecords.query(listFilter(tableName, recordId))) { entries.add(entry(row)); } return (entries); } + /******************************************************************************* + ** Standard Files-tab pages use one bounded native query, with one same-parent + ** lookahead row. Authorization is current on every request; offsets are not + ** snapshots across concurrent insertions or deletions. + *******************************************************************************/ + public AttachmentPage listPage(String tableName, Serializable recordId, int offset, int limit) throws QException + { + if(offset < 0 || limit < 1 || limit > 100) + { + throw (new QBadRequestException("Offset must be nonnegative and limit must be between 1 and 100.")); + } + List rows = AttachmentRecords.query(listFilter(tableName, recordId).withSkip(offset).withLimit(limit + 1)); + List entries = new ArrayList<>(); + for(QRecord row : rows) + { + // Preserve the same metadata-policy evaluation for the lookahead row. + AttachmentEntry visible = entry(row); + if(entries.size() < limit) + { + entries.add(visible); + } + } + return (new AttachmentPage(List.copyOf(entries), rows.size() > limit)); + } + + /******************************************************************************* + ** Only authorized display metadata and continuation, never a raw total. + *******************************************************************************/ + public record AttachmentPage(List items, boolean hasMore) + { + } + + /******************************************************************************* + ** Shared canonical source authority, deletion filter and deterministic order. + *******************************************************************************/ + private QQueryFilter listFilter(String tableName, Serializable recordId) throws QException + { + new com.kingsrook.qbits.collaboration.people.PeopleService().touch(); + String id = canonicalId(tableName, recordId); + return (new QQueryFilter().withCriteria("tableName", QCriteriaOperator.EQUALS, tableName) + .withCriteria("recordId", QCriteriaOperator.EQUALS, id).withCriteria("parentDeleted", QCriteriaOperator.NOT_EQUALS, true) + .withOrderBy(new QFilterOrderBy("createDate", false)).withOrderBy(new QFilterOrderBy("id", false))); + } + /******************************************************************************* ** Re-check the attachment's actual parent before signing a five-minute GET. ** Inline uses the stored inline header; RC.2 only overrides attachment names. @@ -211,17 +618,106 @@ public class AttachmentsService return (url(requireRow(attachmentId), inline)); } + /******************************************************************************* + ** Standard download clients preserve the selected delivery authentication mode. + *******************************************************************************/ + public AttachmentDownload downloadDelivery(Serializable attachmentId) throws QException + { + new com.kingsrook.qbits.collaboration.people.PeopleService().touch(); + return (downloadDelivery(attachmentId, false)); + } + + /*************************************************************************** + ** Preserve the explicit transport mode for preview and download consumers. + ***************************************************************************/ + public AttachmentDownload downloadDelivery(Serializable attachmentId, boolean inline) throws QException + { + return (delivery(requireRow(attachmentId), inline)); + } + + /******************************************************************************* + ** Read a bounded exact original for trusted integrations such as printing. + ** Authorize the actual parent, enforce lifecycle policy and verify persisted + ** length and SHA-256 before releasing any bytes. No signed URL is involved. + *******************************************************************************/ + public AttachmentContent readContent(Serializable attachmentId, Integer maximumBytes) throws QException + { + QRecord row = requireRow(attachmentId); + requireUser(); + requireRetainedPolicy(row); + Long size = row.getValueLong("sizeBytes"); + String sha256 = row.getValueString("sha256"); + if(maximumBytes == null || maximumBytes < 1 || maximumBytes > 20 * 1024 * 1024) + { + throw (new QBadRequestException("Content read limit must be between 1 byte and 20 MiB.")); + } + if(size == null || size < 0 || size > maximumBytes || sha256 == null || !sha256.matches("[0-9a-f]{64}")) + { + throw (new QBadRequestException("Attachment exceeds the content read limit or lacks integrity metadata.")); + } + lifecycle.beforeContentRead(new QRecord(row)); + byte[] bytes; + try(ObjectStream content = storage.open(row.getValueString("bucketName"), row.getValueString("storageKey"))) + { + if(content.getContentLength() != null && !size.equals(content.getContentLength())) + { + throw (new AttachmentContentIntegrityException()); + } + bytes = content.getInputStream().readNBytes(size.intValue() + 1); + String actualHash = HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(bytes)); + if(bytes.length != size.longValue() || !sha256.equals(actualHash)) + { + throw (new AttachmentContentIntegrityException()); + } + } + catch(IOException | NoSuchAlgorithmException | RuntimeException e) + { + throw (new QException("Could not read attachment content.", e)); + } + lifecycle.afterContentRead(new QRecord(row), size); + return (new AttachmentContent(row.getValueInteger("id"), row.getValueString("fileName"), row.getValueString("contentType"), sha256, bytes)); + } + /******************************************************************************* ** Sign an already-authorized row, also used by the preview customizer. *******************************************************************************/ - URL url(QRecord row, boolean inline) + URL url(QRecord row, boolean inline) throws QException { + return (delivery(row, inline).url()); + } + + /******************************************************************************* + ** Resolve only the configured owner policy, after exact row authorization. + *******************************************************************************/ + AttachmentDownload delivery(QRecord row, boolean inline) throws QException + { + requireRetainedPolicy(row); + try + { + lifecycle.beforeAccess(new QRecord(row), inline); + } + catch(RuntimeException e) + { + // Preview generation may ignore storage failures, never policy failures. + throw (new QException("Attachment access policy failed.", e)); + } + AttachmentDownload configured = lifecycle.downloadDelivery(new QRecord(row)); + if(configured != null) + { + return (configured); + } PresignGet request = PresignGet.of(row.getValueString("storageKey")).ttl(Duration.ofMinutes(5)); - if(!inline) + /////////////////////////////////////////////////////////////////////////////////////////// + // Objects stored before AttachmentMediaPolicy keep their original headers (possibly an // + // active type served inline). The link therefore forces an attachment disposition for // + // every type that is not a known passive format, whatever the caller asked for; passive // + // previews stay inline. // + /////////////////////////////////////////////////////////////////////////////////////////// + if(!inline || !"inline".equals(AttachmentMediaPolicy.disposition(row.getValueString("contentType")))) { request.downloadAs(row.getValueString("fileName")); } - return (storage.presign(row.getValueString("bucketName"), request)); + return (new AttachmentDownload(storage.presign(row.getValueString("bucketName"), request), false)); } /******************************************************************************* @@ -233,7 +729,7 @@ public class AttachmentsService List rows = new ArrayList<>(); for(Serializable id : attachmentIds.stream().distinct().toList()) { - rows.add(requireMutable(id)); + rows.add(requireDeletable(id)); } for(QRecord row : rows) { @@ -250,6 +746,7 @@ public class AttachmentsService { new com.kingsrook.qbits.collaboration.people.PeopleService().touch(); QRecord row = requireMutable(attachmentId); + requireRetainedPolicy(row); QRecord change = new QRecord().withValue("id", row.getValue("id")); if(fileName != null) { @@ -259,10 +756,41 @@ public class AttachmentsService { change.setValue("description", description); } + lifecycle.beforeMetadataChange(new QRecord(row), new QRecord(change)); UpdateInput input = new UpdateInput(CollaborationTables.ATTACHMENT).withTransaction(QContext.getQBackendTransaction()).withRecords(List.of(change)); input.setTableMetaData(AttachmentRecords.storageTable()); input.withPermissionBypass("AttachmentsService checked the actual parent and author/admin before metadata update."); - AttachmentRecords.checkErrors(new UpdateAction().execute(input).getRecords()); + QBackendTransaction callerTransaction = QContext.getQBackendTransaction(); + try(QBackendTransaction ownedTransaction = callerTransaction == null ? QBackendTransaction.openFor(input) : null) + { + QBackendTransaction transaction = ownedTransaction == null ? callerTransaction : ownedTransaction; + input.setTransaction(transaction); + try + { + lifecycle.validateMetadataTransaction(new QRecord(row), transaction); + AttachmentRecords.checkErrors(new UpdateAction().execute(input).getRecords()); + lifecycle.afterMetadataChange(new QRecord(row), new QRecord(change), transaction); + if(ownedTransaction != null) + { + ownedTransaction.commit(); + } + } + catch(QException | RuntimeException failure) + { + if(ownedTransaction != null) + { + try + { + ownedTransaction.rollback(); + } + catch(QException rollbackFailure) + { + failure.addSuppressed(rollbackFailure); + } + } + throw (failure); + } + } return (entry(requireRow(attachmentId))); } @@ -274,6 +802,46 @@ public class AttachmentsService return (rename(attachmentId, null, description)); } + /******************************************************************************* + ** Transfer a retained attachment's reference to an authorized archive parent. + ** The caller owns the transaction and must roll it back on any failure. Bytes, + ** hashes, author, storage coordinates and the retained flag never change. + *******************************************************************************/ + public void transferRetainedReference(Serializable attachmentId, String tableName, Serializable recordId, QBackendTransaction transaction) throws QException + { + requireUser(); + if(transaction == null || (QContext.getQBackendTransaction() != null && QContext.getQBackendTransaction() != transaction)) + { + throw (new QException("Archive transfer requires the caller's explicit transaction.")); + } + QRecord row = requireRow(attachmentId); + if(!Boolean.TRUE.equals(row.getValueBoolean("retained")) || tableName == null || recordId == null) + { + throw (new QException("Only a retained attachment can be transferred to an archive.")); + } + var candidate = uploadJournal.findByLocation(row.getValueString("bucketName"), row.getValueString("storageKey")); + if(candidate.isPresent() && candidate.get().state() != com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadJournalState.COMPLETE) + { + throw (new QException("Retained upload registration must complete before archival reference transfer.")); + } + QRecord target = com.kingsrook.qqq.backend.core.actions.collaboration.RecordReferenceSecurityHelper.visibleParent(tableName, recordId, transaction); + if(target == null) + { + throw (new QNotFoundException("Not found")); + } + String canonical = target.getValueString(QContext.getQInstance().getTable(tableName).getPrimaryKeyField()); + lifecycle.beforeReferenceTransfer(new QRecord(row), tableName, canonical, transaction); + QRecord change = new QRecord().withValue("id", row.getValue("id")).withValue("tableName", tableName).withValue("recordId", canonical); + com.kingsrook.qqq.backend.core.actions.collaboration.RecordReferenceSecurityHelper.copyMirroredLocks( + QContext.getQInstance().getTable(tableName), target, QContext.getQInstance().getTable(CollaborationTables.ATTACHMENT), change); + UpdateInput input = new UpdateInput(CollaborationTables.ATTACHMENT).withTransaction(transaction) + .withRecords(List.of(change)); + input.setTableMetaData(com.kingsrook.qbits.collaboration.CollaborationRecords.cleanupMetadata(CollaborationTables.ATTACHMENT)); + input.withPermissionBypass("Installed retention owner authorized this exact archival reference transfer and transaction."); + AttachmentRecords.checkErrors(new UpdateAction().execute(input).getRecords()); + lifecycle.afterReferenceTransfer(new QRecord(row), tableName, canonical, transaction); + } + /******************************************************************************* ** Process Review uses persisted identity rather than trusting client rows. *******************************************************************************/ @@ -282,7 +850,7 @@ public class AttachmentsService new com.kingsrook.qbits.collaboration.people.PeopleService().touch(); QRecord row = requireRow(attachmentId); requireUser(); - if(!canDelete(row)) + if(!canModify(row)) { throw (new QPermissionDeniedException("Only the author or collaboration.admin can change this attachment.")); } @@ -291,7 +859,17 @@ public class AttachmentsService } /******************************************************************************* - ** Delete storage first so a provider failure leaves a retryable row. + ** Deletion reviews must include retention as well as parent and ownership. + *******************************************************************************/ + public QRecord requireDeletable(Serializable attachmentId) throws QException + { + QRecord row = requireMutable(attachmentId); + requireNotRetained(row); + return (row); + } + + /******************************************************************************* + ** Atomically remove metadata and retain a durable storage cleanup request. *******************************************************************************/ void deleteStoredRow(QRecord row) throws QException { @@ -303,11 +881,20 @@ public class AttachmentsService *******************************************************************************/ void deleteStoredRow(QRecord row, boolean cleanup) throws QException { - storage.delete(row.getValueString("bucketName"), row.getValueString("storageKey")); - DeleteInput input = new DeleteInput(CollaborationTables.ATTACHMENT).withTransaction(QContext.getQBackendTransaction()).withPrimaryKeys(List.of(row.getValue("id"))); - input.setTableMetaData(cleanup ? com.kingsrook.qbits.collaboration.CollaborationRecords.cleanupMetadata(CollaborationTables.ATTACHMENT) : AttachmentRecords.storageTable()); - input.withPermissionBypass("Attachment service authorized deletion, or trusted parent cleanup selected this row."); - AttachmentRecords.checkErrors(new DeleteAction().execute(input).getRecordsWithErrors()); + if(isRetained(row)) + { + if(!cleanup) + { + throw (new QPermissionDeniedException("This attachment is retained and cannot be deleted.")); + } + UpdateInput mark = new UpdateInput(CollaborationTables.ATTACHMENT).withTransaction(QContext.getQBackendTransaction()) + .withRecords(List.of(new QRecord().withValue("id", row.getValue("id")).withValue("parentDeleted", true))); + mark.setTableMetaData(com.kingsrook.qbits.collaboration.CollaborationRecords.cleanupMetadata(CollaborationTables.ATTACHMENT)); + mark.withPermissionBypass("Trusted parent cleanup retains the original and marks its parent deleted."); + AttachmentRecords.checkErrors(new UpdateAction().execute(mark).getRecords()); + return; + } + new AttachmentCleanupQueue(storage).delete(row, cleanup); } /******************************************************************************* @@ -336,7 +923,7 @@ public class AttachmentsService /******************************************************************************* ** Ownership is layered on parent visibility; admins still need that visibility. *******************************************************************************/ - private boolean canDelete(QRecord row) + private boolean canModify(QRecord row) { String user = CollaborationSecurity.userId(QContext.getQSession()); return (user != null && (isAdmin() || user.equals(row.getValueString("createdByUserId")))); @@ -366,11 +953,53 @@ public class AttachmentsService /******************************************************************************* ** Return display data, without internal storage coordinates. *******************************************************************************/ - private AttachmentEntry entry(QRecord row) + private AttachmentEntry entry(QRecord row) throws QException { + boolean canEdit = canModify(row) && !(Boolean.TRUE.equals(row.getValueBoolean("retained")) && lifecycle.getClass().equals(AttachmentLifecyclePolicy.class)); + if(canEdit) + { + try + { + ParentRecordAccess.requireFeature(row.getValueString("tableName"), row.getValueString("recordId"), "attach"); + } + catch(QPermissionDeniedException e) + { + canEdit = false; + } + } return (new AttachmentEntry(row.getValueInteger("id"), row.getValueString("tableName"), row.getValueString("recordId"), row.getValueString("fileName"), row.getValueString("contentType"), row.getValueLong("sizeBytes"), row.getValueString("sha256"), - row.getValueString("description"), row.getValueString("createdByUserId"), row.getValueInstant("createDate"), canDelete(row))); + row.getValueString("description"), row.getValueString("createdByUserId"), row.getValueInstant("createDate"), canEdit && !isRetained(row), canEdit)); + } + + /******************************************************************************* + ** Preflight every selected row before deleting any storage object. + *******************************************************************************/ + private void requireNotRetained(QRecord row) throws QException + { + if(isRetained(row)) + { + throw (new QPermissionDeniedException("This attachment is retained and cannot be deleted.")); + } + } + + /******************************************************************************* + ** Persisted retention survives removal or reconfiguration of the provider. + *******************************************************************************/ + private boolean isRetained(QRecord row) throws QException + { + return (Boolean.TRUE.equals(row.getValueBoolean("retained")) || lifecycle.isRetained(new QRecord(row))); + } + + /******************************************************************************* + ** Removing the configured provider must not permit unaudited retained access. + *******************************************************************************/ + private void requireRetainedPolicy(QRecord row) throws QException + { + if(Boolean.TRUE.equals(row.getValueBoolean("retained")) && lifecycle.getClass().equals(AttachmentLifecyclePolicy.class)) + { + throw (new QException("Retained attachment access requires its lifecycle policy.")); + } } /******************************************************************************* diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/DeleteAttachmentsProcess.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/DeleteAttachmentsProcess.java index 639875c..64fe87b 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/attachments/DeleteAttachmentsProcess.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/DeleteAttachmentsProcess.java @@ -140,7 +140,7 @@ public final class DeleteAttachmentsProcess { try { - QRecord checked = service().requireMutable(row.getValue("id")); + QRecord checked = service().requireDeletable(row.getValue("id")); output.getRecords().add(checked); selected.incrementCountAndAddPrimaryKey(checked.getValue("id")); } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/ExistingAttachmentAdoption.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/ExistingAttachmentAdoption.java new file mode 100644 index 0000000..2bcfdeb --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/ExistingAttachmentAdoption.java @@ -0,0 +1,291 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + + +import java.io.IOException; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.HexFormat; +import java.util.List; +import java.util.Objects; +import com.kingsrook.qbits.collaboration.CollaborationSecurity; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qbits.objectstorage.api.ObjectStream; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; +import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSTableBackendDetails; + + +/******************************************************************************* + ** Owner-only adoption of existing bytes. A durable intent precedes verification; + ** retention and its audit commit together, then idempotent registration follows. + *******************************************************************************/ +final class ExistingAttachmentAdoption +{ + private static final long MAX_BYTES = 20L * 1024 * 1024; + private final AttachmentsService service; + private final AttachmentStorage storage; + private final AttachmentLifecyclePolicy lifecycle; + + /*************************************************************************** + ** Dependencies are the actual configured owner boundaries. + ***************************************************************************/ + ExistingAttachmentAdoption(AttachmentsService service, AttachmentStorage storage, AttachmentLifecyclePolicy lifecycle) + { + this.service = service; + this.storage = storage; + this.lifecycle = lifecycle; + } + + /*************************************************************************** + ** No external transaction, caller locator, bytes, receipt or force option. + ***************************************************************************/ + void adopt(Integer id) throws QException + { + var instance = QContext.getQInstance(); + var session = QContext.getQSession(); + String actor = CollaborationSecurity.userId(session); + Check check = () -> + { + if(QContext.getQBackendTransaction() != null || QContext.getQInstance() != instance || QContext.getQSession() != session + || actor == null || !actor.equals(CollaborationSecurity.userId(session))) + { + throw (new QException("Adoption requires the unchanged initiating context and its own transaction.")); + } + }; + check.run(); + QRecord original = service.requireMutable(id); + if(original.getValueString("intakeKey") != null) + { + throw (new QException("Coordinated intake must recover through its original request.")); + } + requireIntegrity(original); + lifecycle.authorizeAdoption(new QRecord(original)); + check.run(); + lifecycle.beforeAdoption(new QRecord(original)); + check.run(); + same(original, service.requireMutable(id)); + QRecord retained = new QRecord(original).withValue("retained", true); + UpdateInput input = new UpdateInput(CollaborationTables.ATTACHMENT) + .withRecords(List.of(new QRecord().withValue("id", id).withValue("retained", true))); + input.setTableMetaData(AttachmentRecords.storageTable()); + input.withPermissionBypass("Attachment owner rechecked current parent, author/admin and audited retention policy."); + try(QBackendTransaction transaction = QBackendTransaction.openFor(input)) + { + try + { + lifecycle.validateMetadataTransaction(retained, transaction); + check.run(); + lock(transaction, id); + same(original, read(transaction, id)); + verify(original); + check.run(); + same(original, service.requireMutable(id)); + lifecycle.authorizeAdoption(new QRecord(original)); + check.run(); + if(!Boolean.TRUE.equals(original.getValueBoolean("retained"))) + { + input.setTransaction(transaction); + AttachmentRecords.checkErrors(new UpdateAction().execute(input).getRecords()); + lifecycle.afterAdoption(retained, transaction); + } + check.run(); + service.requireMutable(id); + lifecycle.authorizeAdoption(new QRecord(retained)); + check.run(); + transaction.commit(); + } + catch(QException | RuntimeException | Error failure) + { + try + { + transaction.rollback(); + } + catch(QException | RuntimeException cleanup) + { + if(cleanup != failure) + { + failure.addSuppressed(cleanup); + } + } + throw failure; + } + } + check.run(); + same(retained, service.requireMutable(id)); + lifecycle.authorizeAdoption(new QRecord(retained)); + check.run(); + lifecycle.afterStored(new QRecord(retained)); + check.run(); + same(retained, service.requireMutable(id)); + lifecycle.authorizeAdoption(new QRecord(retained)); + check.run(); + } + + /*************************************************************************** + ** Native metadata supplies identifiers; no caller text becomes SQL. + ***************************************************************************/ + private void lock(QBackendTransaction transaction, Integer id) throws QException + { + if(!(transaction instanceof RDBMSTransaction jdbc)) + { + throw (new QException("Existing-original adoption requires a real relational transaction.")); + } + var table = AttachmentRecords.storageTable(); + String tableName = table.getBackendDetails() instanceof RDBMSTableBackendDetails details ? details.getTableName() : table.getName(); + String key = table.getField(table.getPrimaryKeyField()).getBackendName(); + key = key == null ? table.getPrimaryKeyField() : key; + if(tableName == null || !tableName.matches("[A-Za-z_][A-Za-z0-9_]*") || !key.matches("[A-Za-z_][A-Za-z0-9_]*")) + { + throw (new QException("Unsupported attachment storage mapping.")); + } + try + { + String quote = jdbc.getConnection().getMetaData().getIdentifierQuoteString().trim(); + if(!List.of("\"", "`").contains(quote)) + { + throw (new QException("Unsupported attachment identifier quoting.")); + } + try(var statement = jdbc.getConnection().prepareStatement("SELECT " + quote + key + quote + " FROM " + quote + tableName + quote + + " WHERE " + quote + key + quote + " = ? FOR UPDATE")) + { + statement.setQueryTimeout(5); + statement.setObject(1, id); + try(var result = statement.executeQuery()) + { + if(!result.next()) + { + throw (new QNotFoundException("Not found")); + } + } + } + } + catch(java.sql.SQLException failure) + { + throw (new QException("Could not lock the original attachment.", failure)); + } + } + + /*************************************************************************** + ** Read the locked original in the same transaction without URL customizers. + ***************************************************************************/ + private QRecord read(QBackendTransaction transaction, Integer id) throws QException + { + QueryInput input = new QueryInput(CollaborationTables.ATTACHMENT).withTransaction(transaction).withShouldFetchHeavyFields(true) + .withFilter(new QQueryFilter().withCriteria("id", QCriteriaOperator.EQUALS, id).withLimit(1)); + input.setTableMetaData(AttachmentRecords.storageTable()); + input.withPermissionBypass("Owner adoption already authorized the exact parent and attachment."); + var rows = new QueryAction().execute(input).getRecords(); + if(rows.size() != 1) + { + throw (new QNotFoundException("Not found")); + } + return (rows.getFirst()); + } + + /*************************************************************************** + ** Compare persisted metadata, allowing only the native modification timestamp. + ** The caller supplies the exact expected retention state at each boundary. + ***************************************************************************/ + private void same(QRecord expected, QRecord actual) throws QException + { + for(var field : AttachmentRecords.storageTable().getFields().keySet()) + { + if(!"modifyDate".equals(field) && !Objects.equals(expected.getValue(field), actual.getValue(field))) + { + throw (new QException("Attachment metadata changed during adoption.")); + } + } + } + + /*************************************************************************** + ** Reject ambiguous or oversized originals before opening storage. + ***************************************************************************/ + private void requireIntegrity(QRecord row) throws QException + { + Long size = row.getValueLong("sizeBytes"); + if(size == null || size < 0 || size > MAX_BYTES || row.getValueString("sha256") == null + || !row.getValueString("sha256").matches("[0-9a-f]{64}")) + { + throw (new QException("Original integrity metadata is missing or outside the adoption bound.")); + } + } + + /*************************************************************************** + ** Stream and hash the existing original without releasing or writing bytes. + ***************************************************************************/ + private void verify(QRecord row) throws QException + { + try(ObjectStream object = storage.open(row.getValueString("bucketName"), row.getValueString("storageKey"))) + { + long expected = row.getValueLong("sizeBytes"); + if(object.getContentLength() != null && object.getContentLength() != expected) + { + throw (new AttachmentContentIntegrityException()); + } + MessageDigest digest = MessageDigest.getInstance("SHA-256"); + byte[] buffer = new byte[8192]; + long total = 0; + int count; + while((count = object.getInputStream().read(buffer, 0, (int) Math.min(buffer.length, expected - total + 1))) != -1) + { + total += count; + if(total > expected) + { + throw (new AttachmentContentIntegrityException()); + } + digest.update(buffer, 0, count); + } + if(total != expected || !row.getValueString("sha256").equals(HexFormat.of().formatHex(digest.digest()))) + { + throw (new AttachmentContentIntegrityException()); + } + } + catch(IOException | NoSuchAlgorithmException | RuntimeException failure) + { + throw (new QException("Could not verify the existing original.", failure)); + } + } + + /*************************************************************************** + ** Checked context guard without installing a different session or authority. + ***************************************************************************/ + @FunctionalInterface + private interface Check + { + /*********************************************************************** + ** Recheck the initiating context after external callbacks. + ***********************************************************************/ + void run() throws QException; + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/JournaledAttachmentUpload.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/JournaledAttachmentUpload.java new file mode 100644 index 0000000..381ba1f --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/JournaledAttachmentUpload.java @@ -0,0 +1,305 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + + +import java.io.IOException; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.time.Duration; +import java.util.HexFormat; +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.collaboration.PersonalRecords; +import com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadCandidate; +import com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadEntry; +import com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadJournal; +import com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadJournalState; +import com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadReservation; +import com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadStorageState; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qbits.collaboration.security.ParentRecordAccess; +import com.kingsrook.qbits.objectstorage.api.ObjectStream; +import com.kingsrook.qbits.objectstorage.api.PutRequest; +import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.data.QRecord; + + +/******************************************************************************* + ** Retained-only coordination of one immutable original and its durable owner + ** manifest. All database ownership stays in Collaboration; lifecycle registration + ** may commit in its own module after the attachment checkpoint is durable. + *******************************************************************************/ +final class JournaledAttachmentUpload +{ + /******************************************************************************* + ** The caller can avoid repeating non-mandatory timeline activity on replay. + *******************************************************************************/ + record Receipt(QRecord attachment, boolean alreadyComplete) {} + + static final long MAX_BYTES = 20L * 1024 * 1024; + private static final Duration LEASE = Duration.ofMinutes(5); + private final AttachmentStorage storage; + private final AttachmentLifecyclePolicy lifecycle; + private final UploadJournal journal; + + /******************************************************************************* + ** Explicit owner dependencies; the journal never receives file bytes. + *******************************************************************************/ + JournaledAttachmentUpload(AttachmentStorage storage, AttachmentLifecyclePolicy lifecycle, UploadJournal journal) + { + this.storage = storage; + this.lifecycle = lifecycle; + this.journal = journal; + } + + /******************************************************************************* + ** Persist intent before any PUT. Conditional creation never replaces an + ** original. A late or expired PUT cannot authorize metadata registration. + *******************************************************************************/ + Receipt store(QRecord row, byte[] bytes) throws QException + { + requireIndependent(); + UploadCandidate candidate = candidate(row); + authorize(candidate); + if(bytes.length > MAX_BYTES || bytes.length != candidate.sizeBytes() || !hash(bytes).equals(candidate.sha256())) + { + throw (new QException("Retained original exceeds the qualified bound or changed before storage.")); + } + UploadEntry entry = journal.prepare(candidate); + if(entry.state() == UploadJournalState.COMPLETE) + { + QRecord existing = stored(entry); + lifecycle.afterStored(new QRecord(existing)); + return (new Receipt(existing, true)); + } + UploadReservation reservation = journal.reserve(entry.id(), LEASE); + try + { + lifecycle.beforeStored(new QRecord(row)); + reservation = journal.renew(reservation, LEASE); + if(entry.storageState() != UploadStorageState.VERIFIED) + { + journal.recordStorage(reservation, UploadStorageState.UNKNOWN); + try + { + storage.put(candidate.bucketName(), PutRequest.of(candidate.storageKey(), bytes).withContentLength(candidate.sizeBytes()) + .withContentType(candidate.contentType()).withContentDisposition("attachment").withIfNoneMatch(true) + .withMetadata(Map.of("qqq-upload-manifest", candidate.manifestHash()))); + } + catch(RuntimeException uncertain) + { + try + { + verify(candidate); + } + catch(QException verification) + { + verification.addSuppressed(uncertain); + throw (verification); + } + } + } + else + { + verify(candidate); + } + reservation = journal.renew(reservation, LEASE); + journal.recordStorage(reservation, UploadStorageState.VERIFIED); + return (new Receipt(register(reservation), false)); + } + finally + { + journal.release(reservation); + } + } + + /******************************************************************************* + ** Standard recovery reads an already stored original and never writes bytes. + ** Coordinated intake must retry through its original coordinator and scope. + *******************************************************************************/ + QRecord recover(Integer journalId) throws QException + { + requireIndependent(); + UploadEntry entry = journal.find(journalId).orElseThrow(() -> new QNotFoundException("Not found")); + if(!PersonalRecords.currentUserId().equals(entry.candidate().actorId())) + { + throw (new QNotFoundException("Not found")); + } + if(entry.state() == UploadJournalState.COMPLETE) + { + QRecord row = stored(entry); + ParentRecordAccess.requireFeature(row.getValueString("tableName"), row.getValueString("recordId"), "attach"); + return (row); + } + authorize(entry.candidate()); + if(entry.candidate().intakeKey() != null) + { + throw (new QException("Retry this upload through its original intake request to preserve its registration context.")); + } + UploadReservation reservation = journal.reserve(entry.id(), LEASE); + try + { + lifecycle.beforeRecovery(row(entry.candidate())); + verify(entry.candidate()); + reservation = journal.renew(reservation, LEASE); + journal.recordStorage(reservation, UploadStorageState.VERIFIED); + return (register(reservation)); + } + finally + { + journal.release(reservation); + } + } + + /******************************************************************************* + ** Attachment and journal checkpoint commit together. Replays use the same + ** immutable attachment id and retry the mandatory idempotent lifecycle hook. + *******************************************************************************/ + private QRecord register(UploadReservation reservation) throws QException + { + authorize(reservation.candidate()); + UploadEntry entry = journal.register(reservation, (transaction, candidate, journalId) -> + { + InsertInput input = new InsertInput(CollaborationTables.ATTACHMENT).withTransaction(transaction).withRecords(List.of(row(candidate))); + input.withPermissionBypass("Retained owner journal reauthorized the original actor and current parent attach capability."); + List records = new InsertAction().execute(input).getRecords(); + AttachmentRecords.checkErrors(records); + if(records.size() != 1) + { + throw (new QException("Retained registration did not create exactly one owner row.")); + } + return (records.getFirst().getValueInteger("id")); + }); + QRecord stored = stored(entry); + lifecycle.afterStored(new QRecord(stored)); + reservation = journal.renew(reservation, LEASE); + journal.complete(reservation); + return (stored); + } + + /******************************************************************************* + ** Exact original bytes must be readable; missing or ambiguous storage never + ** becomes a cancellation or permission to delete the source record. + *******************************************************************************/ + private void verify(UploadCandidate candidate) throws QException + { + if(candidate.sizeBytes() > MAX_BYTES) + { + throw (new QException("Retained recovery exceeds the qualified content bound.")); + } + try(ObjectStream object = storage.open(candidate.bucketName(), candidate.storageKey())) + { + byte[] bytes = object.getInputStream().readNBytes(Math.toIntExact(candidate.sizeBytes()) + 1); + if(bytes.length != candidate.sizeBytes() || !hash(bytes).equals(candidate.sha256())) + { + throw (new QException("Stored original does not match its immutable journal manifest.")); + } + } + catch(IOException | RuntimeException failure) + { + throw (new QException("Original storage remains unresolved; no absence or deletion is inferred.", failure)); + } + } + + /******************************************************************************* + ** Resolve the checkpoint without exposing storage coordinates to clients. + *******************************************************************************/ + private QRecord stored(UploadEntry entry) throws QException + { + List rows = AttachmentRecords.query(new QQueryFilter().withCriteria("id", QCriteriaOperator.EQUALS, entry.attachmentId()).withLimit(1)); + if(rows.size() != 1 || !entry.candidate().sha256().equals(rows.getFirst().getValueString("sha256")) + || !entry.candidate().sizeBytes().equals(rows.getFirst().getValueLong("sizeBytes")) + || !entry.candidate().actorId().equals(rows.getFirst().getValueString("createdByUserId"))) + { + throw (new QException("Retained journal checkpoint does not match its immutable attachment.")); + } + return (rows.getFirst()); + } + + /******************************************************************************* + ** Authorization is current and actor-specific; the manifest cannot impersonate. + *******************************************************************************/ + private void authorize(UploadCandidate candidate) throws QException + { + if(!PersonalRecords.currentUserId().equals(candidate.actorId())) + { + throw (new QNotFoundException("Not found")); + } + ParentRecordAccess.requireFeature(candidate.tableName(), candidate.recordId(), "attach"); + } + + /******************************************************************************* + ** Retained registration cannot participate in an unrelated caller transaction. + *******************************************************************************/ + private void requireIndependent() throws QException + { + if(QContext.getQBackendTransaction() != null) + { + throw (new QException("Retained upload recovery requires its own committed transactions.")); + } + } + + /******************************************************************************* + ** Server-owned values only; no client-supplied locator or actor is accepted. + *******************************************************************************/ + private UploadCandidate candidate(QRecord row) + { + return (new UploadCandidate(row.getValueString("tableName"), row.getValueString("recordId"), row.getValueString("createdByUserId"), + row.getValueString("fileName"), row.getValueString("contentType"), row.getValueString("description"), row.getValueLong("sizeBytes"), + row.getValueString("sha256"), row.getValueString("bucketName"), row.getValueString("storageKey"), row.getValueBoolean("retained"), + row.getValueString("intakeKey"), row.getValueString("intakeFingerprint"))); + } + + /******************************************************************************* + ** Reconstruct only owner metadata from the immutable saved manifest. + *******************************************************************************/ + private QRecord row(UploadCandidate candidate) + { + return (new QRecord().withValue("tableName", candidate.tableName()).withValue("recordId", candidate.recordId()) + .withValue("createdByUserId", candidate.actorId()).withValue("fileName", candidate.fileName()).withValue("contentType", candidate.contentType()) + .withValue("description", candidate.description()).withValue("sizeBytes", candidate.sizeBytes()).withValue("sha256", candidate.sha256()) + .withValue("bucketName", candidate.bucketName()).withValue("storageKey", candidate.storageKey()).withValue("retained", true) + .withValue("parentDeleted", false).withValue("intakeKey", candidate.intakeKey()).withValue("intakeFingerprint", candidate.intakeFingerprint())); + } + + /******************************************************************************* + ** SHA-256 of actual bounded bytes, never a storage metadata assertion. + *******************************************************************************/ + private String hash(byte[] bytes) throws QException + { + try + { + return (HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(bytes))); + } + catch(NoSuchAlgorithmException failure) + { + throw (new QException("SHA-256 is unavailable.", failure)); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/RecoverRetainedUploadProcess.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/RecoverRetainedUploadProcess.java new file mode 100644 index 0000000..821e1b6 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/RecoverRetainedUploadProcess.java @@ -0,0 +1,123 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + + +import com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadJournalTable; +import com.kingsrook.qbits.collaboration.permissions.CollaborationPermissions; +import com.kingsrook.qqq.backend.core.actions.processes.BackendStep; +import com.kingsrook.qqq.backend.core.actions.tables.GetAction; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput; +import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData; + + +/******************************************************************************* + ** A visible native recovery action for retained original registration. It never + ** accepts replacement bytes, a storage path, an actor, or an intake scope. + *******************************************************************************/ +public final class RecoverRetainedUploadProcess +{ + public static final String NAME = "collaborationRecoverRetainedUpload"; + + /******************************************************************************* + ** Metadata utility. + *******************************************************************************/ + private RecoverRetainedUploadProcess() + { + } + + /******************************************************************************* + ** Standard QQQ confirmation and result screens, attached to the journal table. + *******************************************************************************/ + public static QProcessMetaData metadata() + { + return (new QProcessMetaData().withName(NAME).withTableName(UploadJournalTable.NAME).withLabel("Recover retained upload") + .withMinInputRecords(1).withMaxInputRecords(1).withPermissionRules(CollaborationPermissions.permissionRules()) + .withStep(new com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData().withName("prepare").withCode(new QCodeReference(Prepare.class)) + .withInputData(new com.kingsrook.qqq.backend.core.model.metadata.processes.QFunctionInputMetaData() + .withRecordListMetaData(new com.kingsrook.qqq.backend.core.model.metadata.processes.QRecordListMetaData().withTableName(UploadJournalTable.NAME)))) + .withStep(new com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData().withName("confirm").withLabel("Recover the original upload") + .withComponent(new com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendComponentMetaData() + .withType(com.kingsrook.qqq.backend.core.model.metadata.processes.QComponentType.EDIT_FORM)) + .withFormField(new QFieldMetaData("confirmRecovery", QFieldType.BOOLEAN).withLabel("Verify the stored original and retry registration").withDefaultValue(false).withIsRequired(true))) + .withStep(new com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData().withName("recover").withCode(new QCodeReference(Recover.class))) + .withStep(new com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData().withName("result").withLabel("Original registration saved") + .withComponent(new com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendComponentMetaData() + .withType(com.kingsrook.qqq.backend.core.model.metadata.processes.QComponentType.VIEW_FORM)) + .withViewField(new QFieldMetaData("attachmentId", QFieldType.INTEGER).withLabel("Retained attachment") + .withPossibleValueSourceName(com.kingsrook.qbits.collaboration.model.CollaborationTables.ATTACHMENT)))); + } + + /******************************************************************************* + ** Resolve one visible journal selection before displaying confirmation. + *******************************************************************************/ + public static final class Prepare implements BackendStep + { + /******************************************************************************* + ** Read-only preview under current actor and parent visibility. + *******************************************************************************/ + @Override + public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + if(input.getRecords() == null || input.getRecords().size() != 1) + { + throw (new QException("Select exactly one retained upload.")); + } + var row = new GetAction().execute(new GetInput(UploadJournalTable.NAME).withPrimaryKey(input.getRecords().getFirst().getValue("id"))).getRecord(); + if(row == null) + { + throw (new QException("Retained upload is unavailable.")); + } + if(row.getValueString("intakeKey") != null) + { + throw (new QException("Retry this upload through its original intake request to preserve its context.")); + } + output.addValue("journalId", row.getValue("id")); + output.addValue("confirmRecovery", false); + } + } + + /******************************************************************************* + ** Execution reauthorizes independently of the earlier preview. + *******************************************************************************/ + public static final class Recover implements BackendStep + { + /******************************************************************************* + ** The owner returns only after mandatory lifecycle registration completes. + *******************************************************************************/ + @Override + public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + if(!Boolean.TRUE.equals(input.getValue("confirmRecovery"))) + { + throw (new QException("Confirm retained upload recovery to continue.")); + } + output.addValue("attachmentId", new AttachmentsService().recoverRetainedUpload(input.getValueInteger("journalId")).id()); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/ResolveAttachmentUploadProcess.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/ResolveAttachmentUploadProcess.java new file mode 100644 index 0000000..3106416 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/ResolveAttachmentUploadProcess.java @@ -0,0 +1,227 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + + +import java.io.Serializable; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.Map; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine; +import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput; +import com.kingsrook.qqq.backend.core.model.actions.processes.Status; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QComponentType; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendComponentMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractLoadStep; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractTransformStep; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ExtractViaQueryStep; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ProcessSummaryProviderInterface; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess; + + +/******************************************************************************* + ** Explicit provider settlement through the standard table action and Review flow. + *******************************************************************************/ +public class ResolveAttachmentUploadProcess implements CustomPermissionChecker +{ + public static final String NAME = "collaborationResolveAttachmentUpload"; + public static final String EVIDENCE = "settlementEvidence"; + + + /******************************************************************************* + ** Administrative table action, never a navigation entry or scheduled process. + *******************************************************************************/ + public static QProcessMetaData metadata() + { + Map values = new HashMap<>(); + values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, CollaborationTables.ATTACHMENT_CLEANUP); + values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, CollaborationTables.ATTACHMENT_CLEANUP); + values.put(StreamedETLWithFrontendProcess.FIELD_SUPPORTS_FULL_VALIDATION, true); + values.put(StreamedETLWithFrontendProcess.FIELD_DO_FULL_VALIDATION, true); + values.put(StreamedETLWithFrontendProcess.FIELD_PREVIEW_MESSAGE, + "Confirm that the provider has completed or cancelled every selected upload. Object absence or elapsed time alone is not proof. Current bytes will be deleted; the settlement receipt will be retained."); + return (StreamedETLWithFrontendProcess.defineProcessMetaData(ExtractViaQueryStep.class, Review.class, Apply.class, values) + .withName(NAME).withLabel("Resolve settled uploads").withTableName(CollaborationTables.ATTACHMENT_CLEANUP).withIcon(new QIcon("check")) + .withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.NOT_PROTECTED) + .withCustomPermissionChecker(new QCodeReference(ResolveAttachmentUploadProcess.class))) + .withStep(0, new QFrontendStepMetaData().withName("evidence").withLabel("Confirm provider settlement") + .withFormField(new QFieldMetaData(EVIDENCE, QFieldType.STRING).withLabel("Provider completion or cancellation evidence") + .withMaxLength(2000).withIsRequired(true)) + .withComponent(new QFrontendComponentMetaData().withType(QComponentType.EDIT_FORM)))); + } + + + /******************************************************************************* + ** Guard direct process invocation as well as menu visibility. + *******************************************************************************/ + @Override + public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException + { + AttachmentCleanupQueue.requireSettlementOperator(); + } + + + /******************************************************************************* + ** Shared standard summary language for Review and Result. + *******************************************************************************/ + private static ProcessSummaryLine line(Status status, String future, String past) + { + ProcessSummaryLine line = new ProcessSummaryLine(status); + line.setSingularFutureMessage(future); + line.setPluralFutureMessage(future); + line.setSingularPastMessage(past); + line.setPluralPastMessage(past); + return (line); + } + + + /******************************************************************************* + ** Review never changes provider state or records a completed settlement. + *******************************************************************************/ + public static class Review extends AbstractTransformStep implements ProcessSummaryProviderInterface + { + private final ProcessSummaryLine selected = line(Status.OK, "uploads will be resolved", "uploads selected"); + private final ProcessSummaryLine skipped = line(Status.WARNING, "resolved records or committed deletions will be skipped", "records skipped"); + + + /******************************************************************************* + ** Require evidence before allowing the administrator to review selected rows. + *******************************************************************************/ + @Override + public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + AttachmentCleanupQueue.requireSettlementOperator(); + AttachmentCleanupQueue.validateEvidence(input.getValueString(EVIDENCE)); + for(QRecord row : input.getRecords()) + { + if(row.getValue("attachmentId") != null || row.getValue("resolvedAt") != null) + { + skipped.incrementCountAndAddPrimaryKey(row.getValue("id")); + } + else + { + output.getRecords().add(row); + selected.incrementCountAndAddPrimaryKey(row.getValue("id")); + } + } + } + + + /******************************************************************************* + ** Keep streamed totals across pages. + *******************************************************************************/ + @Override + public ArrayList getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen) + { + ArrayList lines = new ArrayList<>(); + selected.addSelfToListIfAnyCount(lines); + skipped.addSelfToListIfAnyCount(lines); + return (lines); + } + } + + + /******************************************************************************* + ** Apply checks current authorization, row ownership and references again. + *******************************************************************************/ + public static class Apply extends AbstractLoadStep implements ProcessSummaryProviderInterface + { + private final AttachmentCleanupQueue queue; + private final ProcessSummaryLine done = line(Status.OK, "uploads will be resolved", "uploads resolved"); + private final ProcessSummaryLine failed = line(Status.ERROR, "uploads cannot be resolved", "uploads unresolved; inspect and retry"); + + + /******************************************************************************* + ** Construction is also used during context-free metadata validation. + *******************************************************************************/ + public Apply() + { + this(new AttachmentCleanupQueue()); + } + + + /******************************************************************************* + ** Use the same service with a fixture storage provider in process tests. + *******************************************************************************/ + public Apply(AttachmentCleanupQueue queue) + { + this.queue = queue; + } + + + /******************************************************************************* + ** Each locator has its own transaction and immutable settlement receipt. + *******************************************************************************/ + @Override + public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + AttachmentCleanupQueue.requireSettlementOperator(); + AttachmentCleanupQueue.validateEvidence(input.getValueString(EVIDENCE)); + for(QRecord row : input.getRecords()) + { + try + { + queue.resolveUpload(row.getValue("id"), input.getValueString(EVIDENCE)); + done.incrementCountAndAddPrimaryKey(row.getValue("id")); + } + catch(QException | RuntimeException failure) + { + failed.incrementCountAndAddPrimaryKey(row.getValue("id")); + } + } + } + + + /******************************************************************************* + ** Report actual application results and retain review exclusions. + *******************************************************************************/ + @Override + public ArrayList getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen) + { + ArrayList lines = new ArrayList<>(); + done.addSelfToListIfAnyCount(lines); + failed.addSelfToListIfAnyCount(lines); + if(getTransformStep() instanceof Review review) + { + review.skipped.addSelfToListIfAnyCount(lines); + } + return (lines); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/RetryAttachmentCleanupProcess.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/RetryAttachmentCleanupProcess.java new file mode 100644 index 0000000..c48a9bd --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/RetryAttachmentCleanupProcess.java @@ -0,0 +1,99 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + + +import com.kingsrook.qbits.collaboration.CollaborationQBitConfig; +import com.kingsrook.qbits.collaboration.permissions.CollaborationPermissions; +import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker; +import com.kingsrook.qqq.backend.core.actions.processes.BackendStep; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.QScheduleMetaData; +import com.kingsrook.qqq.backend.core.model.session.QSession; +import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession; + + +/******************************************************************************* + ** Scheduled durable attachment cleanup. Runs with its caller's session, which must be a + ** scheduler system session or collaboration admin; never elevates a user. + *******************************************************************************/ +public class RetryAttachmentCleanupProcess implements BackendStep, CustomPermissionChecker +{ + public static final String NAME = "collaborationRetryAttachmentCleanup"; + + + + /******************************************************************************* + ** Always register the backend-only process; scheduling is opt-in. + *******************************************************************************/ + public static QProcessMetaData metadata(CollaborationQBitConfig config) + { + QProcessMetaData process = new QProcessMetaData().withName(NAME).withLabel("Retry attachment cleanup").withIsHidden(true).withIcon(new QIcon("history")) + .withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.HAS_ACCESS_PERMISSION) + .withPermissionBaseName("collaboration").withCustomPermissionChecker(new QCodeReference(RetryAttachmentCleanupProcess.class))) + .withStep(new QBackendStepMetaData().withName("run").withCode(new QCodeReference(RetryAttachmentCleanupProcess.class))); + if(config.getSchedulerName() != null) + { + process.withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName()).withRepeatSeconds(60).withInitialDelaySeconds(60)); + } + return (process); + } + + + + /******************************************************************************* + ** Hidden is a presentation setting, so explicitly guard execution as well. + *******************************************************************************/ + @Override + public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException + { + QSession session = QContext.getQSession(); + if(!(session instanceof QSystemUserSession) && (session == null || !session.hasPermission(CollaborationPermissions.ADMIN))) + { + throw (new QPermissionDeniedException("This needs the collaboration.admin permission.")); + } + } + + + + /******************************************************************************* + ** Retry committed cleanup intents after provider or process recovery. + *******************************************************************************/ + @Override + public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + checkPermissionsThrowing(input, null); + new AttachmentCleanupQueue().retryPending(); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/JdbcUploadJournal.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/JdbcUploadJournal.java new file mode 100644 index 0000000..3be582d --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/JdbcUploadJournal.java @@ -0,0 +1,799 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.sql.Connection; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.sql.Statement; +import java.sql.Timestamp; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.time.temporal.ChronoUnit; +import java.util.ArrayList; +import java.util.Calendar; +import java.util.List; +import java.util.Objects; +import java.util.Optional; +import java.util.TimeZone; +import java.util.UUID; +import com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadJournalException.Reason; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.module.rdbms.RDBMSBackendModule; +import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData; + + +/******************************************************************************* + ** Real JDBC implementation for file-backed H2 with WRITE_DELAY=0 and explicitly + ** qualified MySQL 8.4/InnoDB and PostgreSQL 17 connections, with the + ** journal and the attachment table on the same backend. This is not a portable + ** production database claim; any other backend, a memory database, a no-op + ** transaction or an ambient caller transaction is rejected before any write. + ** + ** Every operation opens and owns its own transaction (commit on success, rollback + ** on any failure; no partial success). Mutations lock the row with a bounded + ** FOR UPDATE, verify the lease token, fence and expiry, and then also guard the + ** UPDATE with the same token and fence and require exactly one row. SQL uses only + ** the fixed owner table name and literal text; all values are bound parameters. + ** Nothing here ever deletes a row or an object, and it never touches bytes. + ** + ** Expiry uses this JVM's clock. Hosts must keep lease durations well above their + ** clock skew; the fence, not the clock, is what stops a stale holder from + ** writing once a newer reservation exists. + ** + ** Runtime checks cannot qualify a deployment's disks, backups or failover policy. + *******************************************************************************/ +public final class JdbcUploadJournal implements UploadJournal +{ + private static final String TABLE = CollaborationTables.UPLOAD_JOURNAL; + private static final int QUERY_TIMEOUT_SECONDS = 10; + private static final int MAXIMUM_LIST = 500; + private static final Duration MINIMUM_LEASE = Duration.ofSeconds(1); + private static final Duration MAXIMUM_LEASE = Duration.ofMinutes(30); + + private static final String COLUMNS = "id, table_name, record_id, created_by_user_id, file_name, content_type, description, size_bytes, sha256," + + " bucket_name, storage_key, retained, intake_key, intake_fingerprint, locator_hash, manifest_hash, state, storage_state, attachment_id," + + " lease_token, lease_until, fence"; + private static final String SELECT_ALL = "SELECT " + COLUMNS + " FROM " + TABLE; + private static final String BY_ID = SELECT_ALL + " WHERE id = ?"; + private static final String BY_ID_LOCKED = BY_ID + " FOR UPDATE"; + private static final String BY_LOCATOR = SELECT_ALL + " WHERE locator_hash = ?"; + private static final String BY_INTAKE = SELECT_ALL + " WHERE intake_key = ?"; + private static final String INCOMPLETE = SELECT_ALL + " WHERE state <> 'COMPLETE' ORDER BY id LIMIT ?"; + + private static final String INSERT = "INSERT INTO " + TABLE + " (create_date, modify_date, table_name, record_id, created_by_user_id, file_name," + + " content_type, description, size_bytes, sha256, bucket_name, storage_key, retained, intake_key, intake_fingerprint, locator_hash," + + " manifest_hash, state, storage_state, fence) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'PREPARED', 'PENDING', 0)"; + private static final String GRANT = "UPDATE " + TABLE + " SET lease_token = ?, lease_until = ?, fence = ?, modify_date = ? WHERE id = ? AND fence = ?"; + private static final String SET_STORAGE = "UPDATE " + TABLE + " SET storage_state = ?, modify_date = ? WHERE id = ? AND lease_token = ? AND fence = ?"; + private static final String SET_REGISTERED = "UPDATE " + TABLE + " SET state = 'REGISTERED', attachment_id = ?, modify_date = ?" + + " WHERE id = ? AND lease_token = ? AND fence = ? AND state = 'PREPARED'"; + private static final String SET_COMPLETE = "UPDATE " + TABLE + " SET state = 'COMPLETE', lease_token = NULL, lease_until = NULL, modify_date = ?" + + " WHERE id = ? AND lease_token = ? AND fence = ? AND state = 'REGISTERED'"; + private static final String CLEAR_LEASE = "UPDATE " + TABLE + " SET lease_token = NULL, lease_until = NULL, modify_date = ?" + + " WHERE id = ? AND lease_token = ? AND fence = ?"; + + private final Clock clock; + + + + /******************************************************************************* + ** Journal using the system clock. + *******************************************************************************/ + public JdbcUploadJournal() + { + this(Clock.systemUTC()); + } + + + + /******************************************************************************* + ** Package-local deterministic clock seam for lease-expiry tests. + *******************************************************************************/ + JdbcUploadJournal(Clock clock) + { + this.clock = Objects.requireNonNull(clock, "clock"); + } + + + + /******************************************************************************* + ** One database transaction's worth of work. + *******************************************************************************/ + @FunctionalInterface + private interface Work + { + /******************************************************************************* + ** Run inside the owned, qualified transaction. + *******************************************************************************/ + T run(RDBMSTransaction transaction, Connection connection) throws SQLException, QException; + } + + + + /******************************************************************************* + ** A locked or read row with the journal-private secrets. + *******************************************************************************/ + private record Row(UploadEntry entry, String leaseToken, String manifestHash) + { + } + + + + /******************************************************************************* + ** Static admission of file H2, native mysql, or the native postgres backend + ** (vendor and backend type both "postgres"). Actual engine, catalog, table + ** storage and durability checks additionally run in every transaction. + *******************************************************************************/ + public static boolean isQualifiedBackend(QBackendMetaData backend) + { + if(backend instanceof RDBMSBackendMetaData postgres && PostgresJournalDatabase.is(postgres)) + { + return (true); + } + return (backend instanceof RDBMSBackendMetaData metadata && RDBMSBackendModule.NAME.equals(metadata.getBackendType()) + && ("mysql".equals(metadata.getVendor()) || ("h2".equals(metadata.getVendor()) + && metadata.getJdbcUrl() != null && metadata.getJdbcUrl().startsWith("jdbc:h2:file:")))); + } + + + + /******************************************************************************* + ** {@inheritDoc} + *******************************************************************************/ + @Override + public UploadEntry prepare(UploadCandidate candidate) throws QException + { + Objects.requireNonNull(candidate, "candidate"); + try + { + return (inTransaction((transaction, connection) -> insertOrReplay(connection, candidate))); + } + catch(UploadJournalException e) + { + if(e.getReason() != Reason.DUPLICATE) + { + throw (e); + } + // Lost a unique-key race: the winner committed, so read its row and compare. + return (inTransaction((transaction, connection) -> + { + UploadEntry existing = existing(connection, candidate); + if(existing == null) + { + throw (new UploadJournalException(Reason.ILLEGAL_STATE, "The journal reported a duplicate but no matching row exists.")); + } + return (existing); + })); + } + } + + + + /******************************************************************************* + ** {@inheritDoc} + *******************************************************************************/ + @Override + public UploadReservation reserve(Integer journalId, Duration lease) throws QException + { + if(lease == null || lease.compareTo(MINIMUM_LEASE) < 0 || lease.compareTo(MAXIMUM_LEASE) > 0) + { + throw (new IllegalArgumentException("The lease must be between 1 second and 30 minutes.")); + } + return (inTransaction((transaction, connection) -> + { + Row row = lock(connection, journalId); + Instant now = now(); + if(row.entry().state() == UploadJournalState.COMPLETE) + { + throw (new UploadJournalException(Reason.ILLEGAL_STATE, "This upload is already complete.")); + } + if(row.entry().isLeased(now)) + { + throw (new UploadJournalException(Reason.LEASE_HELD, "Another holder has a live reservation on this upload.")); + } + String token = UUID.randomUUID().toString(); + Long fence = row.entry().fence() + 1; + Instant until = now.plus(lease); + if(update(connection, GRANT, token, Timestamp.from(until), fence, Timestamp.from(now), journalId, row.entry().fence()) != 1) + { + throw (new UploadJournalException(Reason.LEASE_HELD, "Another holder took this reservation first.")); + } + return (new UploadReservation(row.entry().leased(fence, until), token, fence, until)); + })); + } + + + + /******************************************************************************* + ** {@inheritDoc} + *******************************************************************************/ + @Override + public UploadEntry recordStorage(UploadReservation reservation, UploadStorageState storageState) throws QException + { + Objects.requireNonNull(reservation, "reservation"); + if(storageState == null || storageState == UploadStorageState.PENDING) + { + throw (new UploadJournalException(Reason.ILLEGAL_STATE, "Storage knowledge can only advance to UNKNOWN or VERIFIED.")); + } + return (inTransaction((transaction, connection) -> + { + Row row = requireHolder(connection, reservation); + UploadStorageState current = row.entry().storageState(); + if(current == storageState) + { + return (row.entry()); + } + if(current == UploadStorageState.VERIFIED) + { + throw (new UploadJournalException(Reason.ILLEGAL_STATE, "Verified storage cannot move back to UNKNOWN.")); + } + if(update(connection, SET_STORAGE, storageState.name(), Timestamp.from(now()), reservation.journalId(), reservation.leaseToken(), reservation.fence()) != 1) + { + throw (stale()); + } + return (read(connection, reservation.journalId())); + })); + } + + /******************************************************************************* + ** Renew at synchronous stage boundaries without reviving an expired holder. + *******************************************************************************/ + @Override + public UploadReservation renew(UploadReservation reservation, Duration lease) throws QException + { + if(lease == null || lease.compareTo(MINIMUM_LEASE) < 0 || lease.compareTo(MAXIMUM_LEASE) > 0) + { + throw (new IllegalArgumentException("The lease must be between 1 second and 30 minutes.")); + } + return (inTransaction((transaction, connection) -> + { + Row row = requireHolder(connection, reservation); + Instant until = now().plus(lease); + if(update(connection, "UPDATE " + TABLE + " SET lease_until=?, modify_date=? WHERE id=? AND lease_token=? AND fence=?", + Timestamp.from(until), Timestamp.from(now()), reservation.journalId(), reservation.leaseToken(), reservation.fence()) != 1) + { + throw (stale()); + } + return (new UploadReservation(row.entry().leased(reservation.fence(), until), reservation.leaseToken(), reservation.fence(), until)); + })); + } + + + + /******************************************************************************* + ** {@inheritDoc} + *******************************************************************************/ + @Override + public UploadEntry register(UploadReservation reservation, AttachmentRegistrar registrar) throws QException + { + Objects.requireNonNull(reservation, "reservation"); + Objects.requireNonNull(registrar, "registrar"); + return (inTransaction((transaction, connection) -> + { + Row row = requireHolder(connection, reservation); + if(row.entry().state() != UploadJournalState.PREPARED) + { + return (row.entry()); + } + if(row.entry().storageState() != UploadStorageState.VERIFIED) + { + throw (new UploadJournalException(Reason.STORAGE_NOT_VERIFIED, "The original object has not been verified, so it cannot be registered.")); + } + Integer attachmentId = registrar.insert(transaction, row.entry().candidate(), row.entry().id()); + if(attachmentId == null || attachmentId < 1) + { + throw (new UploadJournalException(Reason.ILLEGAL_STATE, "The registrar did not return an attachment id.")); + } + if(connection.isClosed() || connection.getAutoCommit()) + { + throw (new UploadJournalException(Reason.ILLEGAL_STATE, "The registrar altered the journal's owned transaction.")); + } + if(update(connection, SET_REGISTERED, attachmentId, Timestamp.from(now()), reservation.journalId(), reservation.leaseToken(), reservation.fence()) != 1) + { + throw (stale()); + } + return (read(connection, reservation.journalId())); + })); + } + + + + /******************************************************************************* + ** {@inheritDoc} + *******************************************************************************/ + @Override + public UploadEntry complete(UploadReservation reservation) throws QException + { + Objects.requireNonNull(reservation, "reservation"); + return (inTransaction((transaction, connection) -> + { + Row finished = lock(connection, reservation.journalId()); + if(finished.entry().state() == UploadJournalState.COMPLETE && finished.entry().fence().equals(reservation.fence())) + { + return (finished.entry()); + } + Row row = requireHolder(connection, reservation); + if(row.entry().state() != UploadJournalState.REGISTERED || row.entry().attachmentId() == null) + { + throw (new UploadJournalException(Reason.ILLEGAL_STATE, "Only a registered upload can be completed.")); + } + if(update(connection, SET_COMPLETE, Timestamp.from(now()), reservation.journalId(), reservation.leaseToken(), reservation.fence()) != 1) + { + throw (stale()); + } + return (read(connection, reservation.journalId())); + })); + } + + + + /******************************************************************************* + ** {@inheritDoc} + *******************************************************************************/ + @Override + public boolean release(UploadReservation reservation) throws QException + { + Objects.requireNonNull(reservation, "reservation"); + return (inTransaction((transaction, connection) -> + { + Row row = lock(connection, reservation.journalId()); + if(row.entry().state() == UploadJournalState.COMPLETE || !holds(row, reservation)) + { + return (false); + } + return (update(connection, CLEAR_LEASE, Timestamp.from(now()), reservation.journalId(), reservation.leaseToken(), reservation.fence()) == 1); + })); + } + + + + /******************************************************************************* + ** {@inheritDoc} + *******************************************************************************/ + @Override + public Optional find(Integer journalId) throws QException + { + return (inTransaction((transaction, connection) -> + { + List rows = select(connection, BY_ID, journalId); + return (rows.isEmpty() ? Optional.empty() : Optional.of(rows.getFirst().entry())); + })); + } + + + + /******************************************************************************* + ** {@inheritDoc} + *******************************************************************************/ + @Override + public Optional findByLocation(String bucketName, String storageKey) throws QException + { + String hash = UploadCandidate.locatorHash(bucketName, storageKey); + return (inTransaction((transaction, connection) -> + { + List rows = select(connection, BY_LOCATOR, hash); + return (rows.isEmpty() ? Optional.empty() : Optional.of(rows.getFirst().entry())); + })); + } + + + + /******************************************************************************* + ** {@inheritDoc} + *******************************************************************************/ + @Override + public List listIncomplete(int limit) throws QException + { + if(limit < 1 || limit > MAXIMUM_LIST) + { + throw (new IllegalArgumentException("The limit must be between 1 and " + MAXIMUM_LIST + ".")); + } + return (inTransaction((transaction, connection) -> + { + List entries = new ArrayList<>(); + for(Row row : select(connection, INCOMPLETE, limit)) + { + entries.add(row.entry()); + } + return (List.copyOf(entries)); + })); + } + + + + /******************************************************************************* + ** Replay an identical manifest or insert the candidate as PREPARED. + *******************************************************************************/ + private UploadEntry insertOrReplay(Connection connection, UploadCandidate candidate) throws SQLException, QException + { + UploadEntry existing = existing(connection, candidate); + if(existing != null) + { + return (existing); + } + Timestamp now = Timestamp.from(now()); + update(connection, INSERT, now, now, candidate.tableName(), candidate.recordId(), candidate.actorId(), candidate.fileName(), + candidate.contentType(), candidate.description(), candidate.sizeBytes(), candidate.sha256(), candidate.bucketName(), + candidate.storageKey(), candidate.retained(), candidate.intakeKey(), candidate.intakeFingerprint(), candidate.locatorHash(), + candidate.manifestHash()); + List rows = select(connection, BY_LOCATOR, candidate.locatorHash()); + if(rows.isEmpty()) + { + throw (new UploadJournalException(Reason.ILLEGAL_STATE, "The prepared candidate could not be read back.")); + } + return (rows.getFirst().entry()); + } + + + + /******************************************************************************* + ** The row already holding this location, if its manifest is identical. Any other + ** row sharing the location or intake key is an immutable conflict. Null if none. + *******************************************************************************/ + private UploadEntry existing(Connection connection, UploadCandidate candidate) throws SQLException, QException + { + List byLocator = select(connection, BY_LOCATOR, candidate.locatorHash()); + if(!byLocator.isEmpty()) + { + if(!candidate.manifestHash().equals(byLocator.getFirst().manifestHash())) + { + throw (new UploadJournalException(Reason.IMMUTABLE_CONFLICT, "This object location is already journaled with a different manifest.")); + } + return (byLocator.getFirst().entry()); + } + if(candidate.intakeKey() != null && !select(connection, BY_INTAKE, candidate.intakeKey()).isEmpty()) + { + throw (new UploadJournalException(Reason.IMMUTABLE_CONFLICT, "This intake key is already journaled for a different object.")); + } + return (null); + } + + + + /******************************************************************************* + ** Lock the row under the bounded database wait, or NOT_FOUND. + *******************************************************************************/ + private Row lock(Connection connection, Integer journalId) throws SQLException, QException + { + List rows = journalId == null ? List.of() : select(connection, BY_ID_LOCKED, journalId); + if(rows.isEmpty()) + { + throw (new UploadJournalException(Reason.NOT_FOUND, "No such upload journal row.")); + } + return (rows.getFirst()); + } + + + + /******************************************************************************* + ** Lock the row and require the reservation to still own a live lease on it. + *******************************************************************************/ + private Row requireHolder(Connection connection, UploadReservation reservation) throws SQLException, QException + { + Row row = lock(connection, reservation.journalId()); + if(row.entry().state() == UploadJournalState.COMPLETE || !holds(row, reservation)) + { + throw (stale()); + } + return (row); + } + + + + /******************************************************************************* + ** Token (compared in constant time), fence and unexpired lease all match. + *******************************************************************************/ + private boolean holds(Row row, UploadReservation reservation) + { + return (row.leaseToken() != null && reservation.leaseToken() != null + && MessageDigest.isEqual(row.leaseToken().getBytes(StandardCharsets.UTF_8), reservation.leaseToken().getBytes(StandardCharsets.UTF_8)) + && row.entry().fence().equals(reservation.fence()) && row.entry().isLeased(now())); + } + + + + /******************************************************************************* + ** The standard refusal for a holder that no longer owns the row. + *******************************************************************************/ + private static UploadJournalException stale() + { + return (new UploadJournalException(Reason.STALE_RESERVATION, "The reservation no longer owns this upload; reserve it again.")); + } + + + + /******************************************************************************* + ** Read one row that the transaction just changed. + *******************************************************************************/ + private UploadEntry read(Connection connection, Integer journalId) throws SQLException, QException + { + List rows = select(connection, BY_ID, journalId); + if(rows.isEmpty()) + { + throw (new UploadJournalException(Reason.NOT_FOUND, "No such upload journal row.")); + } + return (rows.getFirst().entry()); + } + + + + /******************************************************************************* + ** Bounded query with one bound parameter, verifying each row's integrity. + *******************************************************************************/ + private List select(Connection connection, String sql, Object parameter) throws SQLException, QException + { + List rows = new ArrayList<>(); + try(PreparedStatement statement = connection.prepareStatement(sql)) + { + statement.setQueryTimeout(QUERY_TIMEOUT_SECONDS); + statement.setObject(1, parameter); + try(ResultSet result = statement.executeQuery()) + { + while(result.next()) + { + rows.add(row(result)); + } + } + } + return (rows); + } + + + + /******************************************************************************* + ** Rebuild and verify a row: the preserved manifest must still hash to the + ** stored manifest and locator, or the row was altered outside this journal. + *******************************************************************************/ + private Row row(ResultSet result) throws SQLException, QException + { + UploadCandidate candidate; + try + { + candidate = new UploadCandidate(result.getString("table_name"), result.getString("record_id"), result.getString("created_by_user_id"), + result.getString("file_name"), result.getString("content_type"), result.getString("description"), result.getLong("size_bytes"), + result.getString("sha256"), result.getString("bucket_name"), result.getString("storage_key"), result.getBoolean("retained"), + result.getString("intake_key"), result.getString("intake_fingerprint")); + } + catch(IllegalArgumentException e) + { + throw (new UploadJournalException(Reason.IMMUTABLE_CONFLICT, "A journal row no longer holds a valid manifest.")); + } + String manifestHash = result.getString("manifest_hash"); + if(!candidate.manifestHash().equals(manifestHash) || !candidate.locatorHash().equals(result.getString("locator_hash"))) + { + throw (new UploadJournalException(Reason.IMMUTABLE_CONFLICT, "A journal row's manifest no longer matches its recorded hashes.")); + } + int attachment = result.getInt("attachment_id"); + Integer attachmentId = result.wasNull() ? null : attachment; + Timestamp leaseUntil = "MySQL".equals(result.getStatement().getConnection().getMetaData().getDatabaseProductName()) + ? result.getTimestamp("lease_until", Calendar.getInstance(TimeZone.getTimeZone("UTC"))) : result.getTimestamp("lease_until"); + UploadEntry entry; + try + { + entry = new UploadEntry(result.getInt("id"), candidate, UploadJournalState.valueOf(result.getString("state")), + UploadStorageState.valueOf(result.getString("storage_state")), attachmentId, result.getLong("fence"), + leaseUntil == null ? null : leaseUntil.toInstant()); + } + catch(IllegalArgumentException e) + { + throw (new UploadJournalException(Reason.ILLEGAL_STATE, "A journal row holds an unknown state.")); + } + return (new Row(entry, result.getString("lease_token"), manifestHash)); + } + + + + /******************************************************************************* + ** Execute one bound update and return its row count. + *******************************************************************************/ + private int update(Connection connection, String sql, Object... parameters) throws SQLException + { + try(PreparedStatement statement = connection.prepareStatement(sql)) + { + statement.setQueryTimeout(QUERY_TIMEOUT_SECONDS); + for(int i = 0; i < parameters.length; i++) + { + if(parameters[i] instanceof Timestamp timestamp && "MySQL".equals(connection.getMetaData().getDatabaseProductName())) + { + statement.setTimestamp(i + 1, timestamp, Calendar.getInstance(TimeZone.getTimeZone("UTC"))); + } + else + { + statement.setObject(i + 1, parameters[i]); + } + } + return (statement.executeUpdate()); + } + } + + + + /******************************************************************************* + ** Current time at the millisecond precision the database stores. + *******************************************************************************/ + private Instant now() + { + return (clock.instant().truncatedTo(ChronoUnit.MILLIS)); + } + + + + /******************************************************************************* + ** Own one real transaction: qualify it, run the work, then commit; roll back + ** every failure so there is never a partial success. + *******************************************************************************/ + private T inTransaction(Work work) throws QException + { + if(QContext.getQBackendTransaction() != null) + { + throw (new UploadJournalException(Reason.AMBIENT_TRANSACTION, "The upload journal owns its transactions and cannot join an ambient one.")); + } + RDBMSBackendMetaData backend = requireQualifiedBackends(); + try(QBackendTransaction opened = new RDBMSBackendModule().openTransaction(new InsertInput(TABLE))) + { + if(!(opened instanceof RDBMSTransaction transaction) || !transaction.getClass().equals(RDBMSTransaction.class)) + { + throw (new UploadJournalException(Reason.UNSUPPORTED_BACKEND, "The upload journal requires the database's actual transaction.")); + } + try + { + requireQualifiedConnection(transaction.getConnection(), backend); + T result = work.run(transaction, transaction.getConnection()); + transaction.commit(); + return (result); + } + catch(SQLException e) + { + rollback(transaction, e); + throw (map(e)); + } + catch(QException | RuntimeException failure) + { + rollback(transaction, failure); + throw (failure); + } + } + } + + + + /******************************************************************************* + ** Roll back, keeping any rollback failure as suppressed on the real failure. + *******************************************************************************/ + private static void rollback(RDBMSTransaction transaction, Throwable failure) + { + try + { + transaction.rollback(); + } + catch(QException | RuntimeException rollbackFailure) + { + failure.addSuppressed(rollbackFailure); + } + } + + + + /******************************************************************************* + ** Classify a database failure by SQLSTATE: unique violation, bounded-wait + ** lock timeout or deadlock, or anything else. + *******************************************************************************/ + private static UploadJournalException map(SQLException e) + { + Throwable cursor = e; + for(int depth = 0; cursor != null && depth < 8; depth++, cursor = cursor.getCause()) + { + if(cursor instanceof SQLException sql) + { + if("23505".equals(sql.getSQLState()) || ("23000".equals(sql.getSQLState()) && sql.getErrorCode() == 1062)) + { + return (new UploadJournalException(Reason.DUPLICATE, "The journal row already exists.", e)); + } + if("50200".equals(sql.getSQLState()) || "40001".equals(sql.getSQLState()) || "HYT00".equals(sql.getSQLState()) + || "55P03".equals(sql.getSQLState()) || "40P01".equals(sql.getSQLState()) + || sql.getErrorCode() == 1205 || sql.getErrorCode() == 1213) + { + return (new UploadJournalException(Reason.BUSY, "The upload journal row stayed busy past the bounded wait.", e)); + } + } + } + return (new UploadJournalException(Reason.DATABASE, "The upload journal database operation failed.", e)); + } + + + + /******************************************************************************* + ** The journal and the attachment table must share one qualified backend, so the + ** metadata insert and the checkpoint can be one real transaction. + *******************************************************************************/ + private static RDBMSBackendMetaData requireQualifiedBackends() throws QException + { + QInstance instance = QContext.getQInstance(); + QBackendMetaData journal = instance == null ? null : instance.getBackendForTable(TABLE); + QBackendMetaData attachments = instance == null ? null : instance.getBackendForTable(CollaborationTables.ATTACHMENT); + if(!isQualifiedBackend(journal) || attachments == null || !journal.getName().equals(attachments.getName())) + { + throw (new UploadJournalException(Reason.UNSUPPORTED_BACKEND, + "The upload journal requires qualified H2, MySQL 8.4 or PostgreSQL 17 shared by the journal and attachment tables.")); + } + return ((RDBMSBackendMetaData) journal); + } + + + + /******************************************************************************* + ** Qualify the actual connection, not merely its class: open, writable, + ** transactional, the configured file database, and an actual SETTINGS query + ** reporting WRITE_DELAY=0 so a commit is on disk before it returns. + *******************************************************************************/ + private static void requireQualifiedConnection(Connection connection, RDBMSBackendMetaData backend) throws SQLException, QException + { + if(PostgresJournalDatabase.is(backend)) + { + PostgresJournalDatabase.requireQualifiedConnection(connection, backend); + return; + } + if("mysql".equals(backend.getVendor())) + { + MySqlJournalDatabase.requireQualifiedConnection(connection, backend); + return; + } + String expected = backend.getJdbcUrl().split(";")[0]; + if(connection.isClosed() || connection.isReadOnly() || connection.getAutoCommit() + || !expected.equals(String.valueOf(connection.getMetaData().getURL()).split(";")[0])) + { + throw (new UploadJournalException(Reason.UNSUPPORTED_BACKEND, "The upload journal needs a writable, non-autocommit connection to its configured database.")); + } + boolean writeDelayIsZero = false; + try(Statement statement = connection.createStatement()) + { + statement.setQueryTimeout(QUERY_TIMEOUT_SECONDS); + try(ResultSet settings = statement.executeQuery("SELECT * FROM INFORMATION_SCHEMA.SETTINGS")) + { + while(settings.next()) + { + if("WRITE_DELAY".equalsIgnoreCase(settings.getString(1))) + { + writeDelayIsZero = "0".equals(settings.getString(2)); + } + } + } + } + if(!writeDelayIsZero) + { + throw (new UploadJournalException(Reason.UNSUPPORTED_BACKEND, "The upload journal requires H2 WRITE_DELAY=0 as reported by the database's SETTINGS.")); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/MySqlJournalDatabase.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/MySqlJournalDatabase.java new file mode 100644 index 0000000..3184cd3 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/MySqlJournalDatabase.java @@ -0,0 +1,184 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +import java.net.URI; +import java.sql.Connection; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.sql.Statement; +import java.util.List; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSTableBackendDetails; + + +/******************************************************************************* + ** Narrow qualification of this owner's fixed MySQL journal and attachment pair. + ** No schema migration or server-global setting is performed at runtime. + *******************************************************************************/ +final class MySqlJournalDatabase +{ + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private MySqlJournalDatabase() + { + } + + + /******************************************************************************* + ** Check the real connection, current durability settings and physical tables. + ** A host-configured session lock timeout bounds contention without mutation. + *******************************************************************************/ + static void requireQualifiedConnection(Connection connection, RDBMSBackendMetaData backend) throws SQLException, QException + { + String expected = baseUrl(ConnectionManager.getJdbcUrl(backend)); + String actual = baseUrl(connection.getMetaData().getURL()); + String catalog; + try + { + URI location = URI.create(expected.substring("jdbc:".length())); + catalog = location.getPath().substring(1); + if(location.getHost() == null || location.getUserInfo() != null || !catalog.matches("[A-Za-z0-9_]+")) + { + throw (unsupported()); + } + } + catch(IllegalArgumentException | IndexOutOfBoundsException e) + { + throw (unsupported()); + } + if(!expected.equals(actual) || !catalog.equals(connection.getCatalog()) || connection.isClosed() || connection.isReadOnly() + || connection.getAutoCommit() || !"MySQL".equals(connection.getMetaData().getDatabaseProductName()) + || !connection.getMetaData().getDatabaseProductVersion().startsWith("8.4.")) + { + throw (unsupported()); + } + try(Statement statement = connection.createStatement()) + { + statement.setQueryTimeout(10); + try(ResultSet settings = statement.executeQuery("SELECT @@GLOBAL.innodb_flush_log_at_trx_commit, @@GLOBAL.log_bin, @@GLOBAL.sync_binlog, @@GLOBAL.read_only, @@SESSION.time_zone, @@GLOBAL.system_time_zone, @@SESSION.innodb_lock_wait_timeout, @@GLOBAL.innodb_doublewrite, @@SESSION.transaction_read_only")) + { + if(!settings.next() || !List.of("ON", "DETECT_AND_RECOVER").contains(settings.getString(8)) || settings.getBoolean(9) + || settings.getInt(7) < 1 || settings.getInt(7) > 5 || settings.getInt(1) != 1 || (settings.getBoolean(2) && settings.getInt(3) != 1) || settings.getBoolean(4) + || !(List.of("+00:00", "UTC").contains(settings.getString(5)) + || ("SYSTEM".equals(settings.getString(5)) && "UTC".equals(settings.getString(6))))) + { + throw (unsupported()); + } + } + } + for(String tableName : List.of(CollaborationTables.UPLOAD_JOURNAL, CollaborationTables.ATTACHMENT)) + { + QTableMetaData table = QContext.getQInstance().getTable(tableName); + if(table.getBackendDetails() != null && (!(table.getBackendDetails() instanceof RDBMSTableBackendDetails details) + || (details.getTableName() != null && !tableName.equals(details.getTableName())))) + { + throw (unsupported()); + } + try(PreparedStatement statement = connection.prepareStatement("SELECT TABLE_NAME, ENGINE, TABLE_COLLATION FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA = ? AND TABLE_NAME = ?")) + { + statement.setQueryTimeout(10); + statement.setString(1, catalog); + statement.setString(2, tableName); + try(ResultSet tableInfo = statement.executeQuery()) + { + if(!tableInfo.next() || !tableName.equals(tableInfo.getString(1)) || !"InnoDB".equals(tableInfo.getString(2)) + || (CollaborationTables.UPLOAD_JOURNAL.equals(tableName) && !"utf8mb4_0900_bin".equals(tableInfo.getString(3))) || tableInfo.next()) + { + throw (unsupported()); + } + } + } + } + try(PreparedStatement statement = connection.prepareStatement("SELECT COLLATION_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = ? AND TABLE_NAME = ? AND COLLATION_NAME IS NOT NULL")) + { + statement.setQueryTimeout(10); + statement.setString(1, catalog); + statement.setString(2, CollaborationTables.UPLOAD_JOURNAL); + try(ResultSet columns = statement.executeQuery()) + { + int count = 0; + while(columns.next()) + { + count++; + if(!"utf8mb4_0900_bin".equals(columns.getString(1))) + { + throw (unsupported()); + } + } + if(count < 1) + { + throw (unsupported()); + } + } + } + try(PreparedStatement statement = connection.prepareStatement("SELECT INDEX_NAME, COLUMN_NAME FROM INFORMATION_SCHEMA.STATISTICS WHERE TABLE_SCHEMA = ? AND TABLE_NAME = ? AND NON_UNIQUE = 0 ORDER BY INDEX_NAME, SEQ_IN_INDEX")) + { + statement.setQueryTimeout(10); + statement.setString(1, catalog); + statement.setString(2, CollaborationTables.UPLOAD_JOURNAL); + try(ResultSet indexes = statement.executeQuery()) + { + java.util.Map> columns = new java.util.HashMap<>(); + while(indexes.next()) + { + columns.computeIfAbsent(indexes.getString(1), key -> new java.util.ArrayList<>()).add(indexes.getString(2)); + } + if(!columns.containsValue(List.of("locator_hash")) || !columns.containsValue(List.of("intake_key"))) + { + throw (unsupported()); + } + } + } + } + + + /******************************************************************************* + ** Compare the configured target without logging credentials or driver options. + *******************************************************************************/ + private static String baseUrl(String url) throws UploadJournalException + { + if(url == null || !url.startsWith("jdbc:mysql://")) + { + throw (unsupported()); + } + return (url.split("\\?", 2)[0]); + } + + + /******************************************************************************* + ** Deliberately omit connection URLs and settings from externally visible errors. + *******************************************************************************/ + private static UploadJournalException unsupported() + { + return (new UploadJournalException(UploadJournalException.Reason.UNSUPPORTED_BACKEND, + "Retained upload journaling requires its configured MySQL 8.4 catalog, InnoDB owner tables, binary journal identifiers, a UTC session, a 1-5 second lock timeout and synchronous durable commits.")); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/PostgresJournalDatabase.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/PostgresJournalDatabase.java new file mode 100644 index 0000000..11c26bf --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/PostgresJournalDatabase.java @@ -0,0 +1,239 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +import java.net.URI; +import java.sql.Connection; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.sql.Statement; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSTableBackendDetails; + + +/******************************************************************************* + ** Narrow qualification of this owner's fixed PostgreSQL 17 journal and attachment + ** pair, on the native QQQ postgres backend (vendor and backend type "postgres"). + ** Observation only: no schema migration and no session or server setting is + ** changed at runtime. + *******************************************************************************/ +final class PostgresJournalDatabase +{ + static final String VENDOR = "postgres"; + + private static final String SETTINGS = "SELECT current_setting('server_version_num')::int, current_setting('transaction_read_only'), pg_is_in_recovery()," + + " current_setting('synchronous_commit'), current_setting('fsync'), current_setting('full_page_writes'), current_setting('TimeZone')," + + " (SELECT setting::bigint FROM pg_settings WHERE name='lock_timeout'), current_setting('transaction_isolation'), current_database()"; + + private static final String TABLES = "SELECT c.relname, c.relkind, c.relpersistence FROM pg_class c JOIN pg_namespace n ON n.oid=c.relnamespace" + + " WHERE n.nspname=current_schema() AND c.relname = ANY(?)"; + + private static final String COLLATIONS = "SELECT c.relname, a.attname FROM pg_attribute a JOIN pg_class c ON c.oid=a.attrelid JOIN pg_namespace n ON n.oid=c.relnamespace" + + " JOIN pg_collation co ON co.oid=a.attcollation WHERE n.nspname=current_schema() AND c.relname = ANY(?) AND a.attnum>0 AND NOT a.attisdropped AND NOT co.collisdeterministic"; + + private static final String UNIQUES = "SELECT i.relname, a.attname FROM pg_index ix JOIN pg_class i ON i.oid=ix.indexrelid" + + " JOIN pg_class t ON t.oid=ix.indrelid JOIN pg_namespace n ON n.oid=t.relnamespace" + + " CROSS JOIN LATERAL unnest(ix.indkey::int2[]) WITH ORDINALITY AS k(attnum, ord)" + + " JOIN pg_attribute a ON a.attrelid=t.oid AND a.attnum=k.attnum" + + " WHERE n.nspname=current_schema() AND t.relname = ? AND ix.indisunique AND ix.indisvalid AND ix.indpred IS NULL AND ix.indexprs IS NULL AND k.ord<=ix.indnkeyatts" + + " ORDER BY i.relname, k.ord"; + + + + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private PostgresJournalDatabase() + { + } + + + + /******************************************************************************* + ** The native postgres module identity, not merely a vendor string: a "postgres" + ** vendor label on the generic rdbms backend type is refused. + *******************************************************************************/ + static boolean is(RDBMSBackendMetaData backend) + { + return (VENDOR.equals(backend.getVendor()) && VENDOR.equals(backend.getBackendType())); + } + + + + /******************************************************************************* + ** Check the real connection, current durability settings and physical tables. + *******************************************************************************/ + static void requireQualifiedConnection(Connection connection, RDBMSBackendMetaData backend) throws SQLException, QException + { + String expected = baseUrl(ConnectionManager.getJdbcUrl(backend)); + String actual = baseUrl(connection.getMetaData().getURL()); + String catalog; + try + { + URI location = URI.create(expected.substring("jdbc:".length())); + catalog = location.getPath() == null || !location.getPath().startsWith("/") ? "" : location.getPath().substring(1); + if(location.getHost() == null || location.getUserInfo() != null || !catalog.matches("[A-Za-z0-9_]+")) + { + throw (unsupported()); + } + } + catch(IllegalArgumentException | IndexOutOfBoundsException e) + { + throw (unsupported()); + } + if(!is(backend) || !expected.equals(actual) || !catalog.equals(connection.getCatalog()) || connection.isClosed() || connection.isReadOnly() + || connection.getAutoCommit() || !"PostgreSQL".equals(connection.getMetaData().getDatabaseProductName()) + || connection.getMetaData().getDatabaseMajorVersion() != 17) + { + throw (unsupported()); + } + try(Statement statement = connection.createStatement()) + { + statement.setQueryTimeout(10); + try(ResultSet settings = statement.executeQuery(SETTINGS)) + { + if(!settings.next() || settings.getInt(1) / 10000 != 17 || !"off".equals(settings.getString(2)) || settings.getBoolean(3) + || !Set.of("on", "remote_apply").contains(settings.getString(4)) || !"on".equals(settings.getString(5)) || !"on".equals(settings.getString(6)) + || !Set.of("UTC", "Etc/UTC").contains(settings.getString(7)) || settings.getLong(8) < 1 || settings.getLong(8) > 5000 + || !"read committed".equals(settings.getString(9)) || !catalog.equals(settings.getString(10))) + { + throw (unsupported()); + } + } + } + List names = List.of(CollaborationTables.UPLOAD_JOURNAL, CollaborationTables.ATTACHMENT); + for(String tableName : names) + { + QTableMetaData table = QContext.getQInstance().getTable(tableName); + if(table.getBackendDetails() != null && (!(table.getBackendDetails() instanceof RDBMSTableBackendDetails details) + || (details.getTableName() != null && !tableName.equals(details.getTableName())))) + { + throw (unsupported()); + } + } + requireTables(connection, names); + } + + + + /******************************************************************************* + ** Permanent (logged) ordinary tables only, deterministic text comparison, and + ** complete valid non-partial unique indexes for the journal's two identities. + *******************************************************************************/ + private static void requireTables(Connection connection, List names) throws SQLException, QException + { + java.sql.Array array = connection.createArrayOf("text", names.toArray()); + try + { + try(PreparedStatement statement = connection.prepareStatement(TABLES)) + { + statement.setQueryTimeout(10); + statement.setArray(1, array); + try(ResultSet rows = statement.executeQuery()) + { + Set observed = new java.util.HashSet<>(); + while(rows.next()) + { + if(!names.contains(rows.getString(1)) || !observed.add(rows.getString(1)) || !"r".equals(rows.getString(2)) || !"p".equals(rows.getString(3))) + { + throw (unsupported()); + } + } + if(observed.size() != names.size()) + { + throw (unsupported()); + } + } + } + try(PreparedStatement statement = connection.prepareStatement(COLLATIONS)) + { + statement.setQueryTimeout(10); + statement.setArray(1, array); + try(ResultSet rows = statement.executeQuery()) + { + if(rows.next()) + { + throw (unsupported()); + } + } + } + } + finally + { + array.free(); + } + try(PreparedStatement statement = connection.prepareStatement(UNIQUES)) + { + statement.setQueryTimeout(10); + statement.setString(1, CollaborationTables.UPLOAD_JOURNAL); + try(ResultSet rows = statement.executeQuery()) + { + Map> columns = new HashMap<>(); + while(rows.next()) + { + columns.computeIfAbsent(rows.getString(1), key -> new ArrayList<>()).add(rows.getString(2)); + } + if(!columns.containsValue(List.of("locator_hash")) || !columns.containsValue(List.of("intake_key"))) + { + throw (unsupported()); + } + } + } + } + + + + /******************************************************************************* + ** Compare the configured target without logging credentials or driver options. + *******************************************************************************/ + private static String baseUrl(String url) throws UploadJournalException + { + if(url == null || !url.startsWith("jdbc:postgresql://")) + { + throw (unsupported()); + } + return (url.split("\\?", 2)[0]); + } + + + + /******************************************************************************* + ** Deliberately omit connection URLs and settings from externally visible errors. + *******************************************************************************/ + private static UploadJournalException unsupported() + { + return (new UploadJournalException(UploadJournalException.Reason.UNSUPPORTED_BACKEND, + "Retained upload journaling requires its configured PostgreSQL 17 database, permanent owner tables, deterministic comparison, READ COMMITTED isolation, a UTC session, a bounded lock timeout and synchronous durable commits.")); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadCandidate.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadCandidate.java new file mode 100644 index 0000000..0a8c4ab --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadCandidate.java @@ -0,0 +1,198 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +import java.io.ByteArrayOutputStream; +import java.io.DataOutputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.HexFormat; + + +/******************************************************************************* + ** Immutable, bounded manifest of a retained upload, assembled by the server + ** (AttachmentsService) from the authorized source, the actual session actor, + ** the measured size/hash and the chosen storage coordinates. Nothing here is a + ** client-supplied path or journal field. All members are immutable values. + ** + ** Only retained uploads are journaled; a non-retained candidate is rejected. + ** The bucket and key are needed for recovery but are deliberately absent from + ** toString and from the QQQ table metadata; the journal exposes only their hash. + *******************************************************************************/ +public record UploadCandidate(String tableName, String recordId, String actorId, String fileName, String contentType, String description, + Long sizeBytes, String sha256, String bucketName, String storageKey, Boolean retained, + String intakeKey, String intakeFingerprint) +{ + public static final long MAX_SIZE_BYTES = 1L << 40; + private static final String HEX_64 = "[0-9a-f]{64}"; + + + + /******************************************************************************* + ** Validate every member. Failures are programming errors in the trusted + ** caller, so they are unchecked and never echo the offending value. + *******************************************************************************/ + public UploadCandidate + { + tableName = text("tableName", tableName, 128, true, false); + recordId = text("recordId", recordId, 64, true, false); + actorId = text("actorId", actorId, 255, true, false); + fileName = text("fileName", fileName, 255, true, false); + contentType = text("contentType", contentType, 255, true, false); + description = text("description", description, 4096, false, true); + bucketName = text("bucketName", bucketName, 255, true, false); + storageKey = text("storageKey", storageKey, 1024, true, false); + if(sizeBytes == null || sizeBytes < 0 || sizeBytes > MAX_SIZE_BYTES) + { + throw (new IllegalArgumentException("sizeBytes is out of range.")); + } + if(sha256 == null || !sha256.matches(HEX_64)) + { + throw (new IllegalArgumentException("sha256 must be a lowercase SHA-256 hex digest.")); + } + if(!Boolean.TRUE.equals(retained)) + { + throw (new IllegalArgumentException("Only retained uploads are journaled.")); + } + if((intakeKey == null) != (intakeFingerprint == null) + || (intakeKey != null && (!intakeKey.matches(HEX_64) || !intakeFingerprint.matches(HEX_64)))) + { + throw (new IllegalArgumentException("intakeKey and intakeFingerprint must be supplied together as SHA-256 identifiers.")); + } + } + + + + /******************************************************************************* + ** SHA-256 of the length-prefixed bucket and key; the journal's unique identity + ** for one object location. + *******************************************************************************/ + public String locatorHash() + { + return (locatorHash(bucketName, storageKey)); + } + + + + /******************************************************************************* + ** SHA-256 of the length-prefixed bucket and key. + *******************************************************************************/ + public static String locatorHash(String bucketName, String storageKey) + { + return (tupleHash("upload-journal-locator-v1", bucketName, storageKey)); + } + + + + /******************************************************************************* + ** SHA-256 over the complete manifest. A row that shares a locator or intake + ** key but not this hash is an immutable conflict. + *******************************************************************************/ + public String manifestHash() + { + return (tupleHash("upload-journal-manifest-v1", tableName, recordId, actorId, fileName, contentType, description, + String.valueOf(sizeBytes), sha256, bucketName, storageKey, String.valueOf(retained), intakeKey, intakeFingerprint)); + } + + + + /******************************************************************************* + ** Never prints the bucket, key, description or actor. + *******************************************************************************/ + @Override + public String toString() + { + return ("UploadCandidate[source=" + tableName + "/" + recordId + ", sizeBytes=" + sizeBytes + ", locatorHash=" + locatorHash() + "]"); + } + + + + /******************************************************************************* + ** Bounded text without control characters (multi-line allows CR, LF, tab). + *******************************************************************************/ + private static String text(String name, String value, int maximumLength, boolean required, boolean multiLine) + { + if(value == null) + { + if(required) + { + throw (new IllegalArgumentException(name + " is required.")); + } + return (null); + } + if((required && value.isBlank()) || value.length() > maximumLength) + { + throw (new IllegalArgumentException(name + " is blank or exceeds " + maximumLength + " characters.")); + } + for(int i = 0; i < value.length(); i++) + { + char c = value.charAt(i); + if(Character.isHighSurrogate(c)) + { + if(i + 1 >= value.length() || !Character.isLowSurrogate(value.charAt(++i))) + { + throw (new IllegalArgumentException(name + " contains malformed Unicode.")); + } + continue; + } + if(Character.isLowSurrogate(c)) + { + throw (new IllegalArgumentException(name + " contains malformed Unicode.")); + } + if(Character.isISOControl(c) && !(multiLine && (c == '\n' || c == '\r' || c == '\t'))) + { + throw (new IllegalArgumentException(name + " contains a control character.")); + } + } + return (value); + } + + + + /******************************************************************************* + ** Length-prefix every member, including null, so delimiters cannot collide. + *******************************************************************************/ + private static String tupleHash(String... values) + { + try + { + ByteArrayOutputStream buffer = new ByteArrayOutputStream(); + try(DataOutputStream output = new DataOutputStream(buffer)) + { + for(String value : values) + { + byte[] bytes = value == null ? new byte[0] : value.getBytes(StandardCharsets.UTF_8); + output.writeInt(value == null ? -1 : bytes.length); + output.write(bytes); + } + } + return (HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(buffer.toByteArray()))); + } + catch(IOException | NoSuchAlgorithmException e) + { + throw (new IllegalStateException("Could not hash the upload manifest.", e)); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadEntry.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadEntry.java new file mode 100644 index 0000000..161aae8 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadEntry.java @@ -0,0 +1,158 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +import java.time.Instant; + + +/******************************************************************************* + ** Immutable snapshot of one journal row at read time. It never carries the + ** lease token, so it cannot be used to mutate the journal. + *******************************************************************************/ +public final class UploadEntry +{ + private final Integer id; + private final UploadCandidate candidate; + private final UploadJournalState state; + private final UploadStorageState storageState; + private final Integer attachmentId; + private final Long fence; + private final Instant leaseUntil; + + + + /******************************************************************************* + ** Package-local; only the journal creates snapshots. + *******************************************************************************/ + UploadEntry(Integer id, UploadCandidate candidate, UploadJournalState state, UploadStorageState storageState, Integer attachmentId, Long fence, Instant leaseUntil) + { + this.id = id; + this.candidate = candidate; + this.state = state; + this.storageState = storageState; + this.attachmentId = attachmentId; + this.fence = fence; + this.leaseUntil = leaseUntil; + } + + + + /******************************************************************************* + ** Journal row id. + *******************************************************************************/ + public Integer id() + { + return (id); + } + + + + /******************************************************************************* + ** The preserved manifest, including the original actor and source. + *******************************************************************************/ + public UploadCandidate candidate() + { + return (candidate); + } + + + + /******************************************************************************* + ** Registration progress. + *******************************************************************************/ + public UploadJournalState state() + { + return (state); + } + + + + /******************************************************************************* + ** What is known about the original object. + *******************************************************************************/ + public UploadStorageState storageState() + { + return (storageState); + } + + + + /******************************************************************************* + ** The registered attachment id, or null before the registering commit. + *******************************************************************************/ + public Integer attachmentId() + { + return (attachmentId); + } + + + + /******************************************************************************* + ** Number of reservations ever granted; the fencing counter. + *******************************************************************************/ + public Long fence() + { + return (fence); + } + + + + /******************************************************************************* + ** When the current lease lapses, or null when no lease is held. + *******************************************************************************/ + public Instant leaseUntil() + { + return (leaseUntil); + } + + + + /******************************************************************************* + ** True while a lease is held as of the given instant. + *******************************************************************************/ + public boolean isLeased(Instant now) + { + return (leaseUntil != null && leaseUntil.isAfter(now)); + } + + + + /******************************************************************************* + ** Copy reflecting a newly granted lease. + *******************************************************************************/ + UploadEntry leased(Long newFence, Instant newLeaseUntil) + { + return (new UploadEntry(id, candidate, state, storageState, attachmentId, newFence, newLeaseUntil)); + } + + + + /******************************************************************************* + ** Never prints the manifest's bucket or key. + *******************************************************************************/ + @Override + public String toString() + { + return ("UploadEntry[id=" + id + ", state=" + state + ", storage=" + storageState + ", attachmentId=" + attachmentId + ", fence=" + fence + ", " + candidate + "]"); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournal.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournal.java new file mode 100644 index 0000000..3485edb --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournal.java @@ -0,0 +1,141 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +import java.time.Duration; +import java.util.List; +import java.util.Optional; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.exceptions.QException; + + +/******************************************************************************* + ** Durable journal of retained upload candidates. The intended order, owned by + ** AttachmentsService (this interface never reads, writes or deletes bytes): + ** + ** 1. prepare(candidate) independently committed, before the first PUT + ** 2. reserve(id, lease) exclusive fenced lease (original or recovery) + ** 3. recordStorage(UNKNOWN) before the PUT; VERIFIED once bytes are proven + ** 4. register(reservation, cb) attachment insert + checkpoint, one transaction + ** 5. (caller) lifecycle.afterStored, after the commit, idempotent and retried + ** 6. complete(reservation) only after step 5 returned + ** + ** A crash, timeout or failure at any step leaves the row visible; recovery + ** reserves it again once the lease lapses, after the caller reauthorizes the + ** original actor and the current source. Nothing here deletes an original. + ** + ** AttachmentsService owns authorization and lifecycle policy around this port. + *******************************************************************************/ +public interface UploadJournal +{ + /******************************************************************************* + ** Inserts the registering callback's attachment row. Runs inside the journal's + ** owned transaction, which the callback must use for every write and must not + ** commit, roll back or close. Returns the new attachment id. Throwing rolls + ** back the insert and the checkpoint together. + *******************************************************************************/ + @FunctionalInterface + interface AttachmentRegistrar + { + /******************************************************************************* + ** Insert the attachment metadata row for the journaled candidate. + *******************************************************************************/ + Integer insert(QBackendTransaction transaction, UploadCandidate candidate, Integer journalId) throws QException; + } + + + + /******************************************************************************* + ** Durably record the candidate in its own committed transaction. Idempotent + ** for an identical manifest; an immutable conflict for any different one. + *******************************************************************************/ + UploadEntry prepare(UploadCandidate candidate) throws QException; + + + + /******************************************************************************* + ** Take the exclusive lease, failing with LEASE_HELD while another holder's + ** lease is live. Each grant increments the fence. Lease bounded 1s to 30m. + *******************************************************************************/ + UploadReservation reserve(Integer journalId, Duration lease) throws QException; + + /******************************************************************************* + ** Extend only a still-live exact reservation. Expired holders cannot renew. + *******************************************************************************/ + UploadReservation renew(UploadReservation reservation, Duration lease) throws QException; + + + + /******************************************************************************* + ** Report original-object knowledge. Forward only: PENDING to UNKNOWN to + ** VERIFIED. Requires the live reservation. + *******************************************************************************/ + UploadEntry recordStorage(UploadReservation reservation, UploadStorageState storageState) throws QException; + + + + /******************************************************************************* + ** Requires VERIFIED storage and the live reservation. Inserts the attachment + ** through the callback and checkpoints its id and REGISTERED in one same-database + ** transaction. If already REGISTERED the callback is not called again and the + ** existing entry is returned, so the attachment is never registered twice. + *******************************************************************************/ + UploadEntry register(UploadReservation reservation, AttachmentRegistrar registrar) throws QException; + + + + /******************************************************************************* + ** Mark COMPLETE after the post-commit lifecycle registration returned. Requires + ** REGISTERED and the live reservation; repeating it for the same fence is a no-op. + *******************************************************************************/ + UploadEntry complete(UploadReservation reservation) throws QException; + + + + /******************************************************************************* + ** Give up the lease early so recovery need not wait. Changes no other data. + ** Returns false, harmlessly, if the reservation was no longer the holder. + *******************************************************************************/ + boolean release(UploadReservation reservation) throws QException; + + + + /******************************************************************************* + ** Current snapshot of one row. + *******************************************************************************/ + Optional find(Integer journalId) throws QException; + + + + /******************************************************************************* + ** Snapshot found by the object location, hashed internally. + *******************************************************************************/ + Optional findByLocation(String bucketName, String storageKey) throws QException; + + + + /******************************************************************************* + ** Oldest-first rows that are not COMPLETE, for recovery sweeps. Limit 1 to 500. + *******************************************************************************/ + List listIncomplete(int limit) throws QException; +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalException.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalException.java new file mode 100644 index 0000000..0fb45d2 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalException.java @@ -0,0 +1,118 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +import com.kingsrook.qqq.backend.core.exceptions.QException; + + +/******************************************************************************* + ** Journal failure with a machine-readable reason. Messages never contain the + ** bucket, storage key or lease token. + *******************************************************************************/ +public class UploadJournalException extends QException +{ + /******************************************************************************* + ** Why the journal refused or failed an operation. + *******************************************************************************/ + public enum Reason + { + /*************************************************************************** + ** Another transaction held the row past the bounded wait. + ***************************************************************************/ + BUSY, + /*************************************************************************** + ** A live lease belongs to another holder. + ***************************************************************************/ + LEASE_HELD, + /*************************************************************************** + ** The reservation's token, fence or lease no longer owns the row. + ***************************************************************************/ + STALE_RESERVATION, + /*************************************************************************** + ** Same locator or intake key, different immutable manifest (or tampered row). + ***************************************************************************/ + IMMUTABLE_CONFLICT, + /*************************************************************************** + ** The operation does not apply to the row's current state. + ***************************************************************************/ + ILLEGAL_STATE, + /*************************************************************************** + ** Registration requires VERIFIED original storage. + ***************************************************************************/ + STORAGE_NOT_VERIFIED, + /*************************************************************************** + ** No such journal row. + ***************************************************************************/ + NOT_FOUND, + /*************************************************************************** + ** Backend is not the qualified file-backed H2 configuration. + ***************************************************************************/ + UNSUPPORTED_BACKEND, + /*************************************************************************** + ** The journal owns its transactions and refuses an ambient one. + ***************************************************************************/ + AMBIENT_TRANSACTION, + /*************************************************************************** + ** Unique constraint hit; handled internally, not normally surfaced. + ***************************************************************************/ + DUPLICATE, + /*************************************************************************** + ** Other database failure. + ***************************************************************************/ + DATABASE + } + + private final Reason reason; + + + + /******************************************************************************* + ** Constructor. + *******************************************************************************/ + public UploadJournalException(Reason reason, String message) + { + super(message); + this.reason = reason; + } + + + + /******************************************************************************* + ** Constructor with cause. + *******************************************************************************/ + public UploadJournalException(Reason reason, String message, Throwable cause) + { + super(message, cause); + this.reason = reason; + } + + + + /******************************************************************************* + ** The refusal reason. + *******************************************************************************/ + public Reason getReason() + { + return (reason); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalReadPolicy.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalReadPolicy.java new file mode 100644 index 0000000..ff881cf --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalReadPolicy.java @@ -0,0 +1,159 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +import java.util.List; +import java.util.Set; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; + + +/******************************************************************************* + ** Private manifest fields cannot become filter or sorting oracles. This + ** table-local restriction composes with the host's existing field access policy. + *******************************************************************************/ +public final class UploadJournalReadPolicy +{ + private static final Set PRIVATE = Set.of("bucketName", "storageKey", "manifestHash", "leaseToken"); + + /******************************************************************************* + ** Utility namespace. + *******************************************************************************/ + private UploadJournalReadPolicy() + { + } + + /******************************************************************************* + ** Native read criteria are checked before SQL, including unique-key Get. + ** Joins are not an journal browsing contract and are rejected conservatively. + *******************************************************************************/ + public static void check(AbstractActionInput input, QTableMetaData table) throws QPermissionDeniedException + { + if(!UploadJournalTable.NAME.equals(table.getName())) + { + return; + } + if(input instanceof QueryInput query) + { + if(query.getQueryJoins() != null && !query.getQueryJoins().isEmpty()) + { + deny(); + } + filter(query.getFilter(), table, 0, new int[]{0}); + if(query.getFieldNamesToInclude() != null) + { + for(String name : query.getFieldNamesToInclude()) + { + field(name, table); + } + } + } + else if(input instanceof GetInput get) + { + if(get.getQueryJoins() != null && !get.getQueryJoins().isEmpty()) + { + deny(); + } + if(get.getUniqueKey() != null) + { + for(String name : get.getUniqueKey().keySet()) + { + field(name, table); + } + } + } + } + + /******************************************************************************* + ** Bound traversal and accept only literal criteria on public direct fields. + *******************************************************************************/ + private static void filter(QQueryFilter filter, QTableMetaData table, int depth, int[] visited) throws QPermissionDeniedException + { + if(filter == null) + { + return; + } + if(depth > 64 || ++visited[0] > 4096) + { + deny(); + } + for(var criterion : filter.getCriteria() == null ? List.of() : filter.getCriteria()) + { + field(criterion.getFieldName(), table); + if(criterion.getOtherFieldName() != null) + { + field(criterion.getOtherFieldName(), table); + } + if(criterion.getFieldFunction() != null) + { + deny(); + } + for(Object value : criterion.getValues() == null ? List.of() : criterion.getValues()) + { + if(value instanceof com.kingsrook.qqq.backend.core.model.actions.tables.query.expressions.CurrentUser + && "createdByUserId".equals(criterion.getFieldName()) + && criterion.getOperator() == com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator.EQUALS + && criterion.getValues().size() == 1) + { + continue; + } + if(value != null && !(value instanceof String) && !(value instanceof Number) && !(value instanceof Boolean) + && !(value instanceof java.time.temporal.TemporalAccessor) && !(value instanceof java.util.Date)) + { + deny(); + } + } + } + for(var order : filter.getOrderBys() == null ? List.of() : filter.getOrderBys()) + { + field(order.getFieldName(), table); + } + for(QQueryFilter child : filter.getSubFilters() == null ? List.of() : filter.getSubFilters()) + { + filter(child, table, depth + 1, visited); + } + } + + /******************************************************************************* + ** A public direct field cannot refer to a private projection through an alias. + *******************************************************************************/ + private static void field(String name, QTableMetaData table) throws QPermissionDeniedException + { + if(name == null || PRIVATE.contains(name) || table.getFields().get(name) == null || Boolean.TRUE.equals(table.getFields().get(name).getIsHidden())) + { + deny(); + } + } + + /******************************************************************************* + ** Uniform denial never echoes security values or a caller's probing predicate. + *******************************************************************************/ + private static void deny() throws QPermissionDeniedException + { + throw (new QPermissionDeniedException("Upload journal queries support public direct fields only.")); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalSecurity.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalSecurity.java new file mode 100644 index 0000000..f43680e --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalSecurity.java @@ -0,0 +1,118 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +import java.util.ArrayList; +import java.util.List; +import com.kingsrook.qbits.collaboration.CollaborationRecords; +import com.kingsrook.qbits.collaboration.PersonalRecords; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qbits.collaboration.permissions.CollaborationPermissionChecker; +import com.kingsrook.qbits.collaboration.security.ParentRecordAccess; +import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface; +import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.QueryOrGetInputInterface; +import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.AggregateInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules; + + +/******************************************************************************* + ** Owner journal visibility follows its original actor and its exact attachment's + ** current parent after a transfer. Immutable manifest source values never change. + *******************************************************************************/ +public class UploadJournalSecurity implements CustomPermissionChecker, TableCustomizerInterface +{ + /******************************************************************************* + ** No direct writes or counts bypass per-row dynamic parent authorization. + *******************************************************************************/ + @Override + public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException + { + if(metadata instanceof com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData table) + { + UploadJournalReadPolicy.check(input, table); + } + if(input instanceof InsertInput || input instanceof UpdateInput || input instanceof DeleteInput || input instanceof CountInput || input instanceof AggregateInput) + { + throw (new QPermissionDeniedException("The upload journal is maintained by the attachment service; unscoped totals are unavailable.")); + } + new CollaborationPermissionChecker().checkPermissionsThrowing(input, metadata); + } + + /******************************************************************************* + ** Read authoritative owner identity even if the client selected a projection. + ** Internal manifest columns are not QQQ fields and cannot be returned here. + *******************************************************************************/ + @Override + public List postQuery(QueryOrGetInputInterface input, List records) throws QException + { + List visible = new ArrayList<>(); + for(QRecord row : records) + { + if(row.getValueInteger("id") == null) + { + continue; + } + try + { + List persisted = CollaborationRecords.cleanupQuery(UploadJournalTable.NAME, + new QQueryFilter().withCriteria("id", QCriteriaOperator.EQUALS, row.getValueInteger("id")).withLimit(1)); + if(persisted.size() != 1 || !PersonalRecords.currentUserId().equals(persisted.getFirst().getValueString("createdByUserId"))) + { + continue; + } + QRecord source = persisted.getFirst(); + if(source.getValueInteger("attachmentId") != null) + { + List attachment = CollaborationRecords.cleanupQuery(CollaborationTables.ATTACHMENT, + new QQueryFilter().withCriteria("id", QCriteriaOperator.EQUALS, source.getValueInteger("attachmentId")).withLimit(1)); + if(attachment.size() != 1 || !source.getValueString("sha256").equals(attachment.getFirst().getValueString("sha256")) + || !source.getValueLong("sizeBytes").equals(attachment.getFirst().getValueLong("sizeBytes")) + || !source.getValueString("createdByUserId").equals(attachment.getFirst().getValueString("createdByUserId"))) + { + continue; + } + source = attachment.getFirst(); + } + ParentRecordAccess.requireVisibleParent(source.getValueString("tableName"), source.getValueString("recordId")); + visible.add(row); + } + catch(QNotFoundException denied) + { + // Invisible and absent owners have identical native query behavior. + } + } + return (visible); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalState.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalState.java new file mode 100644 index 0000000..a447dd3 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalState.java @@ -0,0 +1,46 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +/******************************************************************************* + ** Registration progress of a journaled retained upload. Only moves forward. + ** NOT YET WIRED: no producer, table registration or service uses this yet. + *******************************************************************************/ +public enum UploadJournalState +{ + /******************************************************************************* + ** Candidate is durable; no attachment row has been committed for it. + *******************************************************************************/ + PREPARED, + + /******************************************************************************* + ** The attachment row and its checkpoint committed together; the post-commit + ** lifecycle registration has not yet been confirmed. + *******************************************************************************/ + REGISTERED, + + /******************************************************************************* + ** Lifecycle registration returned after the registering commit. + *******************************************************************************/ + COMPLETE +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalTable.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalTable.java new file mode 100644 index 0000000..e60adf3 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalTable.java @@ -0,0 +1,222 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +import java.util.List; +import com.kingsrook.qbits.collaboration.CollaborationQBitConfig; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qbits.collaboration.permissions.CollaborationPermissions; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QFieldSection; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.tables.Tier; +import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey; + + +/******************************************************************************* + ** The one fixed journal table: native QQQ metadata for visible, read-only + ** provenance and status, and explicit H2/MySQL DDL. The bucket, storage key, + ** manifest hash and lease token are physical columns only; they are NOT QQQ + ** fields, so no UI or API surface can return a raw object locator or lease. + ** Only the hash of the locator is exposed. + ** + ** The producer registers this table; hosts apply the appropriate owner DDL + ** through migration tooling before enabling retained upload intake. + *******************************************************************************/ +public final class UploadJournalTable +{ + public static final String NAME = CollaborationTables.UPLOAD_JOURNAL; + + + + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private UploadJournalTable() + { + } + + + + /******************************************************************************* + ** Table metadata with no editable field; direct writes are denied by + ** UploadJournalSecurity. Field names line up with the parent-reference + ** metadata used by attachments (tableName, recordId, createdByUserId). + *******************************************************************************/ + public static QTableMetaData metaData(CollaborationQBitConfig config) + { + return (new QTableMetaData().withName(NAME).withLabel("Retained Upload Journal").withIcon(new QIcon("history_edu")) + .withBackendName(config.getBackendName()).withPrimaryKeyField("id") + .withPermissionRules(CollaborationPermissions.permissionRules().withCustomPermissionChecker(new QCodeReference(UploadJournalSecurity.class))) + .withCustomizer(com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers.POST_QUERY_RECORD, new QCodeReference(UploadJournalSecurity.class)) + .withoutCapabilities(com.kingsrook.qqq.backend.core.model.metadata.tables.Capability.TABLE_COUNT, com.kingsrook.qqq.backend.core.model.metadata.tables.Capability.QUERY_STATS) + .withRecordLabelFormatAndFields("%s", "fileName") + .withUniqueKey(new UniqueKey("locatorHash")) + .withField(new QFieldMetaData("id", QFieldType.INTEGER).withLabel("ID").withIsEditable(false)) + .withField(new QFieldMetaData("createDate", QFieldType.DATE_TIME).withLabel("Created").withIsEditable(false)) + .withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME).withLabel("Modified").withIsEditable(false)) + .withField(string("tableName", "Table", 128)) + .withField(string("recordId", "Record ID", 64)) + .withField(string("createdByUserId", "Original Actor", 255).withPossibleValueSourceName(CollaborationTables.USER)) + .withField(string("fileName", "File Name", 255)) + .withField(string("contentType", "Content Type", 255)) + .withField(new QFieldMetaData("description", QFieldType.TEXT).withLabel("Description").withIsEditable(false)) + .withField(new QFieldMetaData("sizeBytes", QFieldType.LONG).withLabel("Size (bytes)").withIsEditable(false)) + .withField(string("sha256", "SHA-256", 64)) + .withField(new QFieldMetaData("retained", QFieldType.BOOLEAN).withLabel("Retained").withIsEditable(false)) + .withField(string("intakeKey", "Intake Key", 64)) + .withField(string("intakeFingerprint", "Intake Fingerprint", 64)) + .withField(string("locatorHash", "Object Location Hash", 64)) + .withField(string("state", "Registration State", 16)) + .withField(string("storageState", "Original Object", 16)) + .withField(new QFieldMetaData("attachmentId", QFieldType.INTEGER).withLabel("Attachment").withIsEditable(false) + .withPossibleValueSourceName(CollaborationTables.ATTACHMENT)) + .withField(new QFieldMetaData("leaseUntil", QFieldType.DATE_TIME).withLabel("Lease Until").withIsEditable(false)) + .withField(new QFieldMetaData("fence", QFieldType.LONG).withLabel("Reservation Count").withIsEditable(false)) + .withSection(new QFieldSection("identity", new QIcon("badge"), Tier.T1, List.of("id", "fileName", "state", "storageState", "attachmentId")).withLabel("Status")) + .withSection(new QFieldSection("source", new QIcon("link"), Tier.T2, List.of("tableName", "recordId", "createdByUserId", "contentType", "sizeBytes", "sha256", "description")).withLabel("Source")) + .withSection(new QFieldSection("recovery", new QIcon("storage"), Tier.T3, List.of("locatorHash", "intakeKey", "intakeFingerprint", "leaseUntil", "fence", "retained")).withLabel("Recovery"))); + } + + + + /******************************************************************************* + ** H2 DDL for the qualified adapter only. Unquoted lowercase identifiers match + ** QQQ's inferred snake_case backend names. The constraints are database + ** guards: retained rows only, forward-only states, intake pairing, and no + ** REGISTERED/COMPLETE row without its attachment id. PostgreSQL has its own + ** script, postgresScript(). + *******************************************************************************/ + public static String h2Script() + { + return ("CREATE TABLE IF NOT EXISTS collaboration_upload_journal (\n" + + " id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,\n" + + " create_date TIMESTAMP NOT NULL,\n" + + " modify_date TIMESTAMP NOT NULL,\n" + + " table_name VARCHAR(128) NOT NULL,\n" + + " record_id VARCHAR(64) NOT NULL,\n" + + " created_by_user_id VARCHAR(255) NOT NULL,\n" + + " file_name VARCHAR(255) NOT NULL,\n" + + " content_type VARCHAR(255) NOT NULL,\n" + + " description VARCHAR(4096),\n" + + " size_bytes BIGINT NOT NULL,\n" + + " sha256 VARCHAR(64) NOT NULL,\n" + + " bucket_name VARCHAR(255) NOT NULL,\n" + + " storage_key VARCHAR(1024) NOT NULL,\n" + + " retained BOOLEAN NOT NULL,\n" + + " intake_key VARCHAR(64),\n" + + " intake_fingerprint VARCHAR(64),\n" + + " locator_hash VARCHAR(64) NOT NULL,\n" + + " manifest_hash VARCHAR(64) NOT NULL,\n" + + " state VARCHAR(16) NOT NULL,\n" + + " storage_state VARCHAR(16) NOT NULL,\n" + + " attachment_id INTEGER,\n" + + " lease_token VARCHAR(64),\n" + + " lease_until TIMESTAMP,\n" + + " fence BIGINT NOT NULL,\n" + + " CONSTRAINT ck_upload_journal_retained CHECK (retained = TRUE),\n" + + " CONSTRAINT ck_upload_journal_state CHECK (state IN ('PREPARED', 'REGISTERED', 'COMPLETE')),\n" + + " CONSTRAINT ck_upload_journal_storage CHECK (storage_state IN ('PENDING', 'UNKNOWN', 'VERIFIED')),\n" + + " CONSTRAINT ck_upload_journal_intake CHECK ((intake_key IS NULL) = (intake_fingerprint IS NULL)),\n" + + " CONSTRAINT ck_upload_journal_registered CHECK (state = 'PREPARED' OR attachment_id IS NOT NULL),\n" + + " CONSTRAINT ck_upload_journal_verified CHECK (state = 'PREPARED' OR storage_state = 'VERIFIED')\n" + + ");\n" + + "CREATE UNIQUE INDEX IF NOT EXISTS uq_collaboration_upload_journal_locator ON collaboration_upload_journal (locator_hash);\n" + + "CREATE UNIQUE INDEX IF NOT EXISTS uq_collaboration_upload_journal_intake ON collaboration_upload_journal (intake_key);\n" + + "CREATE INDEX IF NOT EXISTS idx_collaboration_upload_journal_open ON collaboration_upload_journal (state, id);\n" + + "CREATE INDEX IF NOT EXISTS idx_collaboration_upload_journal_parent ON collaboration_upload_journal (table_name, record_id);\n"); + } + + + + /******************************************************************************* + ** PostgreSQL 17 schema for the native postgres backend. Same columns, CHECK + ** guards and unique/ordinary indexes as the H2 script; instants are + ** TIMESTAMPTZ so a lease fence round-trips independent of the JVM zone. + ** Default (deterministic) collation keeps identifiers exact. Statements are + ** idempotent, so the script is safe to re-run. + *******************************************************************************/ + public static String postgresScript() + { + return (h2Script().replace("TIMESTAMP NOT NULL,", "TIMESTAMPTZ NOT NULL,").replace("lease_until TIMESTAMP,", "lease_until TIMESTAMPTZ,")); + } + + + + /******************************************************************************* + ** MySQL 8.4 schema. Binary NO PAD comparison preserves exact identifiers; + ** millisecond timestamps preserve lease fences across connection round trips. + *******************************************************************************/ + public static String mysqlScript() + { + return ("CREATE TABLE IF NOT EXISTS collaboration_upload_journal (\n" + + " id INTEGER AUTO_INCREMENT PRIMARY KEY,\n" + + " create_date TIMESTAMP(3) NOT NULL,\n" + + " modify_date TIMESTAMP(3) NOT NULL,\n" + + " table_name VARCHAR(128) NOT NULL,\n" + + " record_id VARCHAR(64) NOT NULL,\n" + + " created_by_user_id VARCHAR(255) NOT NULL,\n" + + " file_name VARCHAR(255) NOT NULL,\n" + + " content_type VARCHAR(255) NOT NULL,\n" + + " description VARCHAR(4096),\n" + + " size_bytes BIGINT NOT NULL,\n" + + " sha256 VARCHAR(64) NOT NULL,\n" + + " bucket_name VARCHAR(255) NOT NULL,\n" + + " storage_key VARCHAR(1024) NOT NULL,\n" + + " retained BOOLEAN NOT NULL,\n" + + " intake_key VARCHAR(64),\n" + + " intake_fingerprint VARCHAR(64),\n" + + " locator_hash VARCHAR(64) NOT NULL,\n" + + " manifest_hash VARCHAR(64) NOT NULL,\n" + + " state VARCHAR(16) NOT NULL,\n" + + " storage_state VARCHAR(16) NOT NULL,\n" + + " attachment_id INTEGER,\n" + + " lease_token VARCHAR(64),\n" + + " lease_until TIMESTAMP(3),\n" + + " fence BIGINT NOT NULL,\n" + + " CONSTRAINT ck_upload_journal_retained CHECK (retained = TRUE),\n" + + " CONSTRAINT ck_upload_journal_state CHECK (state IN ('PREPARED', 'REGISTERED', 'COMPLETE')),\n" + + " CONSTRAINT ck_upload_journal_storage CHECK (storage_state IN ('PENDING', 'UNKNOWN', 'VERIFIED')),\n" + + " CONSTRAINT ck_upload_journal_intake CHECK ((intake_key IS NULL) = (intake_fingerprint IS NULL)),\n" + + " CONSTRAINT ck_upload_journal_registered CHECK (state = 'PREPARED' OR attachment_id IS NOT NULL),\n" + + " CONSTRAINT ck_upload_journal_verified CHECK (state = 'PREPARED' OR storage_state = 'VERIFIED'),\n" + + " UNIQUE KEY uq_collaboration_upload_journal_locator (locator_hash),\n" + + " UNIQUE KEY uq_collaboration_upload_journal_intake (intake_key),\n" + + " KEY idx_collaboration_upload_journal_open (state, id),\n" + + " KEY idx_collaboration_upload_journal_parent (table_name, record_id)\n" + + ") ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_bin;\n"); + } + + + + /******************************************************************************* + ** A bounded read-only string field. + *******************************************************************************/ + private static QFieldMetaData string(String name, String label, Integer length) + { + return (new QFieldMetaData(name, QFieldType.STRING).withLabel(label).withMaxLength(length).withIsEditable(false)); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadRecoveryNavigation.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadRecoveryNavigation.java new file mode 100644 index 0000000..bddae32 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadRecoveryNavigation.java @@ -0,0 +1,63 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +import java.io.Serializable; +import java.util.LinkedHashMap; +import java.util.Map; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.expressions.CurrentUser; +import com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordFeature; +import com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordFeaturesMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; + + +/******************************************************************************* + ** Small metadata contract for native recovery navigation from a source's Files + ** panel. The client adds canonical source table/id pins to this server-owned view. + ** Filters aid discovery only; the journal still authorizes every row and action. + *******************************************************************************/ +public final class UploadRecoveryNavigation +{ + /******************************************************************************* + ** Metadata utility. + *******************************************************************************/ + private UploadRecoveryNavigation() + { + } + + /******************************************************************************* + ** Retention owners opt in their managed source tables after producing them. + ** No client knows or guesses the journal table or recovery process name. + *******************************************************************************/ + public static void configure(QTableMetaData source) + { + Map view = new LinkedHashMap<>(); + view.put("tableName", UploadJournalTable.NAME); + view.put("label", "Review interrupted uploads"); + view.put("filter", new QQueryFilter().withCriteria("createdByUserId", QCriteriaOperator.EQUALS, new CurrentUser()) + .withCriteria("state", QCriteriaOperator.IN, "PREPARED", "REGISTERED")); + RecordFeaturesMetaData.ofOrWithNew(source).withSetting(RecordFeature.ATTACHMENTS, "recoveryView", (Serializable) view); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadReservation.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadReservation.java new file mode 100644 index 0000000..dc88f5b --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadReservation.java @@ -0,0 +1,125 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +import java.time.Instant; + + +/******************************************************************************* + ** Immutable write reservation: an exclusive, fenced lease on one journal row. + ** Every mutating journal call re-checks the token, fence and expiry under a row + ** lock, so a holder that lost its lease cannot act. The token is generated by + ** the journal, is never persisted outside the journal row, never printed, and + ** is not accepted from clients. The snapshot is as of the grant. + *******************************************************************************/ +public final class UploadReservation +{ + private final UploadEntry entry; + private final String leaseToken; + private final Long fence; + private final Instant leaseUntil; + + + + /******************************************************************************* + ** Package-local; only the journal grants reservations. + *******************************************************************************/ + UploadReservation(UploadEntry entry, String leaseToken, Long fence, Instant leaseUntil) + { + this.entry = entry; + this.leaseToken = leaseToken; + this.fence = fence; + this.leaseUntil = leaseUntil; + } + + + + /******************************************************************************* + ** Journal row id. + *******************************************************************************/ + public Integer journalId() + { + return (entry.id()); + } + + + + /******************************************************************************* + ** The preserved manifest. + *******************************************************************************/ + public UploadCandidate candidate() + { + return (entry.candidate()); + } + + + + /******************************************************************************* + ** Journal state as of the grant; read the journal for current state. + *******************************************************************************/ + public UploadEntry entryAtGrant() + { + return (entry); + } + + + + /******************************************************************************* + ** Fencing counter of this grant. + *******************************************************************************/ + public Long fence() + { + return (fence); + } + + + + /******************************************************************************* + ** Lease expiry of this grant. + *******************************************************************************/ + public Instant leaseUntil() + { + return (leaseUntil); + } + + + + /******************************************************************************* + ** Secret proof of ownership, visible only to the journal implementation. + *******************************************************************************/ + String leaseToken() + { + return (leaseToken); + } + + + + /******************************************************************************* + ** Never prints the token. + *******************************************************************************/ + @Override + public String toString() + { + return ("UploadReservation[journalId=" + entry.id() + ", fence=" + fence + ", leaseUntil=" + leaseUntil + "]"); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadStorageState.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadStorageState.java new file mode 100644 index 0000000..08b8e46 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadStorageState.java @@ -0,0 +1,46 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +/******************************************************************************* + ** What the journal knows about the original object. The journal never reads or + ** writes bytes; the storage owner reports these facts. Never moves backwards. + *******************************************************************************/ +public enum UploadStorageState +{ + /******************************************************************************* + ** No write has been attempted that the journal knows of. + *******************************************************************************/ + PENDING, + + /******************************************************************************* + ** A write was attempted and its outcome is not known (timeout, crash). The + ** object may exist; it stays visible and reconcilable and is never deleted. + *******************************************************************************/ + UNKNOWN, + + /******************************************************************************* + ** The storage owner verified the exact bytes and hash of the original. + *******************************************************************************/ + VERIFIED +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldStorage.java b/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldStorage.java index cfc5959..f70b8b7 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldStorage.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldStorage.java @@ -83,7 +83,7 @@ public final class CustomFieldStorage } if(!postgres(table)) { - return (new Availability(false, "Backend does not support JsonPathStorage; EAV fields are read-only filters (values are not selected).")); + return (new Availability(false, "Backend does not support JsonPathStorage; EAV fields support read-only values and filters.")); } try(Connection connection = connection(table); var statement = connection.createStatement()) { @@ -92,7 +92,7 @@ public final class CustomFieldStorage } catch(SQLException | IllegalArgumentException e) { - return (new Availability(false, "JSON column unavailable; EAV fields are read-only filters. " + e.getMessage())); + return (new Availability(false, "JSON column unavailable; EAV fields support read-only values and filters. " + e.getMessage())); } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldValueProvider.java b/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldValueProvider.java new file mode 100644 index 0000000..b974680 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldValueProvider.java @@ -0,0 +1,101 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.customfields; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.collaboration.CollaborationRecords; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; +import com.kingsrook.qqq.backend.core.actions.tables.query.VirtualFieldValueProviderInterface; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordReferenceTableMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.core.utils.CollectionUtils; +import com.kingsrook.qqq.backend.core.utils.ValueUtils; + +/******************************************************************************* + ** Supply EAV values for the parent records selected by the core query pipeline. + ** Do not re-read parents here: that would recursively execute this provider. + *******************************************************************************/ +public class CustomFieldValueProvider implements VirtualFieldValueProviderInterface +{ + /******************************************************************************* + ** Bound each lookup and retain the caller's transaction through service storage. + *******************************************************************************/ + @Override + public void fill(QTableMetaData table, String fieldName, List records, QueryInput input) throws QException + { + String name = fieldName.substring(3); + var definitions = CollaborationRecords.query(CollaborationTables.FIELD_DEFINITION, new QQueryFilter() + .withCriteria("tableName", QCriteriaOperator.EQUALS, table.getName()).withCriteria("name", QCriteriaOperator.EQUALS, name) + .withCriteria("status", QCriteriaOperator.EQUALS, "ACTIVE"), input.getTransaction()); + if(definitions.size() != 1) + { + throw (new QException("Custom field is unavailable")); + } + String column = FieldCriteriaExpander.valueColumn(definitions.getFirst().getValueString("type")); + for(List page : CollectionUtils.getPages(records, 500)) + { + List ids = page.stream().map(record -> String.valueOf(record.getValue(table.getPrimaryKeyField()))).distinct().toList(); + Map values = new HashMap<>(); + for(QRecord value : valuesForSelectedParents(new QQueryFilter() + .withCriteria("tableName", QCriteriaOperator.EQUALS, table.getName()).withCriteria("fieldName", QCriteriaOperator.EQUALS, name) + .withCriteria("parentDeleted", QCriteriaOperator.EQUALS, false).withCriteria("recordId", QCriteriaOperator.IN, ids), input)) + { + if(values.put(value.getValueString("recordId"), value) != null) + { + throw (new QException("Duplicate custom field values")); + } + } + for(QRecord record : page) + { + QRecord value = values.get(String.valueOf(record.getValue(table.getPrimaryKeyField()))); + record.setValue(fieldName, value == null ? null : ValueUtils.getValueAsFieldType(table.getVirtualField(fieldName).getType(), value.getValue(column))); + } + } + } + + /******************************************************************************* + ** The enclosing query already selected these parents under its access policy. + ** Remove only recursive parent verification on a private child metadata clone; + ** preserve child locks and never install this metadata on the instance. + *******************************************************************************/ + private List valuesForSelectedParents(QQueryFilter filter, QueryInput origin) throws QException + { + QTableMetaData metadata = QContext.getQInstance().getTable(CollaborationTables.FIELD_VALUE).clone(); + metadata.withCustomizers(null); + metadata.getSupplementalMetaData().remove(RecordReferenceTableMetaData.NAME); + QueryInput input = new QueryInput(CollaborationTables.FIELD_VALUE).withFilter(filter) + .withShouldFetchHeavyFields(true).withTransaction(origin.getTransaction()); + input.setTableMetaData(metadata); + input.withPermissionBypass("EAV provider reads only parents selected by the enclosing query."); + return (new QueryAction().execute(input).getRecords()); + } + +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldsProvider.java b/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldsProvider.java index 4b36687..91ea2a7 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldsProvider.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldsProvider.java @@ -58,7 +58,10 @@ public class CustomFieldsProvider implements RuntimeMetaDataProviderInterface @Override public RuntimeMetaDataContribution getContribution(QInstance instance) throws QException { - com.kingsrook.qbits.collaboration.presence.PresenceRegistry.get(instance); + if(com.kingsrook.qbits.collaboration.presence.RedisPresenceConfig.forInstance(instance) == null) + { + com.kingsrook.qbits.collaboration.presence.PresenceRegistry.get(instance); + } var result = new RuntimeMetaDataContribution(); var overrides = new FeatureSettingsProvider().getOverrides(instance); List definitions = CollaborationRecords.query(CollaborationTables.FIELD_DEFINITION, @@ -119,7 +122,8 @@ public class CustomFieldsProvider implements RuntimeMetaDataProviderInterface String name = "cf_" + definition.getValueString("name"); QFieldType type = type(definition.getValueString("type")); QFieldMetaData field = json ? new QFieldMetaData(name, type).withBackendStorage(new JsonPathStorage().withColumnName("qqq_custom_fields").withKey(definition.getValueString("name"))) - : new QVirtualFieldMetaData(name, type).withIsEditable(false).withIsQueryCriteria(true).withIsQuerySelectable(false) + : new QVirtualFieldMetaData(name, type).withIsEditable(false).withIsQueryCriteria(true).withIsQuerySelectable(true) + .withValueProvider(new QCodeReference(CustomFieldValueProvider.class)) .withCriteriaExpander(new QCodeReference(FieldCriteriaExpander.class)); field.withLabel(definition.getValueString("label")).withIsRequired(json && Boolean.TRUE.equals(definition.getValueBoolean("required"))) .withMaxLength(definition.getValueInteger("maxLength")); diff --git a/src/main/java/com/kingsrook/qbits/collaboration/email/EmailCollaborationNotifier.java b/src/main/java/com/kingsrook/qbits/collaboration/email/EmailCollaborationNotifier.java index 1242a55..389c47a 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/email/EmailCollaborationNotifier.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/email/EmailCollaborationNotifier.java @@ -89,11 +89,20 @@ public final class EmailCollaborationNotifier implements CollaborationNotifier *******************************************************************************/ @Override public void notify(CollaborationNotification n) + { + deliver(n); + } + + /******************************************************************************* + ** A receipt means the queue accepted this event, including an existing duplicate. + *******************************************************************************/ + @Override + public boolean deliver(CollaborationNotification n) { EmailApiTemplate template = templateFor(n.category()); if(template == null) { - return; + return (false); } try { @@ -101,7 +110,7 @@ public final class EmailCollaborationNotifier implements CollaborationNotifier if(!StringUtils.hasContent(address)) { LOG.info("Collaboration email skipped: no address for recipient", logPair("userId", n.recipientUserId())); - return; + return (false); } EmailApi.ensureTemplate(template); String eventId = n.sourceId() != null ? n.sourceId() : n.sourceRecordId(); @@ -123,10 +132,12 @@ public final class EmailCollaborationNotifier implements CollaborationNotifier LOG.info("Collaboration email not queued", logPair("userId", n.recipientUserId()), logPair("status", result.status()), logPair("messageId", result.messageId())); } + return (result.status() == EmailApiStatus.QUEUED || result.status() == EmailApiStatus.DUPLICATE); } catch(QException | RuntimeException e) { LOG.warn("Collaboration email failed", e, logPair("userId", n.recipientUserId()), logPair("category", n.category())); + return (false); } } @@ -183,6 +194,7 @@ public final class EmailCollaborationNotifier implements CollaborationNotifier { case MENTION -> "mention"; case SHARE -> "share"; + case FOLLOW -> "follow"; default -> "task"; }); } @@ -199,6 +211,7 @@ public final class EmailCollaborationNotifier implements CollaborationNotifier case MENTION -> mentionTemplate(); case SHARE -> shareTemplate(); case TASK -> taskReminderTemplate(); + case FOLLOW -> definition("collaboration_follow", "Followed record changed", "Changes to {{recordLabel}}", "A record you follow has changed.\n\n[View record]({{link}})"); default -> null; }); } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/follow/FollowChangeBatcher.java b/src/main/java/com/kingsrook/qbits/collaboration/follow/FollowChangeBatcher.java index e104fad..c32f5ca 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/follow/FollowChangeBatcher.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/follow/FollowChangeBatcher.java @@ -193,12 +193,17 @@ public class FollowChangeBatcher implements RecordChangeListenerInterface String user = batch.getValueString("userId"); List follows = CollaborationRecords.query(CollaborationTables.FOLLOW, CollaborationRecords.parent(table, id).withCriteria("userId", QCriteriaOperator.EQUALS, user)); - if(!follows.isEmpty() && eligible(follows.getFirst(), now)) + if(!follows.isEmpty() && eligible(follows.getFirst(), now) + && new com.kingsrook.qbits.collaboration.security.CollaborationRecordAccessChecker().canRead(user, table, id).orElse(false)) { Notifications.send(new CollaborationNotification(user, "Changes to a followed record", batch.getValueString("changes"), - CollaborationNotification.Category.FOLLOW, table, id, Notifications.link(table, id))); + CollaborationNotification.Category.FOLLOW, table, id, Notifications.link(table, id)).withEvent(batch.getValue("id"), null), + () -> CollaborationRecords.delete(CollaborationTables.FOLLOW_DIGEST, batch.getValue("id"))); + } + else + { + CollaborationRecords.delete(CollaborationTables.FOLLOW_DIGEST, batch.getValue("id")); } - CollaborationRecords.delete(CollaborationTables.FOLLOW_DIGEST, batch.getValue("id")); } } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/model/CollaborationTables.java b/src/main/java/com/kingsrook/qbits/collaboration/model/CollaborationTables.java index 4968f3f..7d137dc 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/model/CollaborationTables.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/model/CollaborationTables.java @@ -26,8 +26,8 @@ import java.util.ArrayList; import java.util.List; import com.kingsrook.qbits.collaboration.CollaborationQBitConfig; import com.kingsrook.qbits.collaboration.CollaborationSecurity; +import com.kingsrook.qbits.collaboration.attachments.AttachmentAggregateGuard; import com.kingsrook.qbits.collaboration.attachments.AttachmentPreviewCustomizer; -import com.kingsrook.qbits.collaboration.attachments.AttachmentSecurity; import com.kingsrook.qbits.collaboration.permissions.CollaborationPermissions; import com.kingsrook.qbits.collaboration.values.FeatureScopeNamePossibleValueProvider; import com.kingsrook.qbits.collaboration.values.RecordFieldNamePossibleValueProvider; @@ -80,6 +80,8 @@ public final class CollaborationTables public static final String ACTIVITY = "collaboration_activity"; public static final String FOLLOW_DIGEST = "collaboration_follow_digest"; public static final String ATTACHMENT = "collaboration_attachment"; + public static final String ATTACHMENT_CLEANUP = "collaboration_attachment_cleanup"; + public static final String UPLOAD_JOURNAL = "collaboration_upload_journal"; public static final String FEATURE_SETTING = "collaboration_feature_setting"; public static final String FAVORITE = "collaboration_favorite"; public static final String RECENT = "collaboration_recent"; @@ -198,10 +200,26 @@ public final class CollaborationTables .withField(new QFieldMetaData("viewCount", QFieldType.INTEGER).withLabel("View Count").withIsRequired(true).withDefaultValue(1)) .withSection(new QFieldSection("details", new QIcon("history"), Tier.T2, List.of("label", "lastViewedAt", "viewCount")).withLabel("Details")); List tables = new ArrayList<>(List.of(settings, people, favorites, recents, attachment(config))); + tables.add(com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadJournalTable.metaData(config)); tables.addAll(conversationTables(config)); tables.addAll(tagTables(config)); tables.addAll(workTables(config)); tables.addAll(customFieldTables(config)); + tables.add(base(config, ATTACHMENT_CLEANUP, "Attachment Cleanup", "delete") + .withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.NOT_PROTECTED) + .withCustomPermissionChecker(new QCodeReference(com.kingsrook.qbits.collaboration.attachments.AttachmentCleanupQueue.class))) + .withRecordLabelFormatAndFields("Cleanup %s", "id") + .withField(new QFieldMetaData("attachmentId", QFieldType.INTEGER)) + .withField(string("bucketName", "Bucket", 255).withIsRequired(true)) + .withField(string("storageKey", "Storage Key", 1024).withIsRequired(true)) + .withField(new QFieldMetaData("nextAttemptAt", QFieldType.DATE_TIME).withLabel("Next Attempt")) + .withField(new QFieldMetaData("resolvedAt", QFieldType.DATE_TIME).withLabel("Resolved At")) + .withField(string("resolvedBy", "Resolved By", 255)) + .withField(string("settlementEvidence", "Provider Settlement Evidence", 2000)) + .withSection(new QFieldSection("cleanup", new QIcon("delete"), Tier.T1, + List.of("id", "attachmentId", "bucketName", "storageKey", "nextAttemptAt")).withLabel("Cleanup")) + .withSection(new QFieldSection("settlement", new QIcon("check"), Tier.T2, + List.of("resolvedAt", "resolvedBy", "settlementEvidence")).withLabel("Operator Settlement"))); for(QTableMetaData table : tables) { table.withSection(new QFieldSection("dates", new QIcon("calendar_today"), Tier.T3, List.of("createDate", "modifyDate")).withLabel("Dates")); @@ -273,9 +291,9 @@ public final class CollaborationTables *******************************************************************************/ private static QTableMetaData attachment(CollaborationQBitConfig config) { - return (base(config, ATTACHMENT, "Attachments", "attach_file") + return (AttachmentAggregateGuard.configure(base(config, ATTACHMENT, "Attachments", "attach_file") .withPermissionRules(CollaborationPermissions.permissionRules() - .withCustomPermissionChecker(new QCodeReference(AttachmentSecurity.class))) + .withCustomPermissionChecker(new QCodeReference(AttachmentAggregateGuard.class))) .withRecordLabelFormatAndFields("%s", "fileName") .withSearchFields(List.of("fileName", "description")) .withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD)) @@ -285,30 +303,39 @@ public final class CollaborationTables .withField(string("contentType", "Content Type", 255).withIsRequired(true)) .withField(new QFieldMetaData("sizeBytes", QFieldType.LONG).withLabel("Size (bytes)").withIsRequired(true)) .withField(string("sha256", "SHA-256", 64).withIsRequired(true)) + .withField(string("intakeKey", "Intake Key", 64).withIsEditable(false)) + .withField(string("intakeFingerprint", "Intake Fingerprint", 64).withIsEditable(false)) .withField(string("bucketName", "Bucket", 255).withIsRequired(true)) .withField(string("storageKey", "Storage Key", 1024).withIsRequired(true)) .withField(string("thumbnailKey", "Thumbnail Key", 1024)) .withField(new QFieldMetaData("description", QFieldType.TEXT).withLabel("Description")) .withField(user("createdByUserId", "Created By").withIsRequired(true)) .withField(new QFieldMetaData("parentDeleted", QFieldType.BOOLEAN).withLabel("Parent Deleted").withIsRequired(true).withDefaultValue(false)) + .withField(new QFieldMetaData("retained", QFieldType.BOOLEAN).withLabel("Retained").withIsRequired(true).withDefaultValue(false).withIsEditable(false)) .withVirtualField(new QVirtualFieldMetaData("previewUrl", QFieldType.STRING).withLabel("Preview").withIsEditable(false).withIsQuerySelectable(true) .withFieldAdornment(AdornmentType.FilePreviewValues.newFieldAdornment() .withValue(AdornmentType.FilePreviewValues.FILE_NAME_FIELD, "fileName") .withValue(AdornmentType.FilePreviewValues.MIME_TYPE_FIELD, "contentType") .withValue(AdornmentType.FilePreviewValues.SIZE_BYTES_FIELD, "sizeBytes") - .withValue(AdornmentType.FilePreviewValues.THUMBNAIL_MAX_SIZE, 48))) + .withValue(AdornmentType.FilePreviewValues.THUMBNAIL_MAX_SIZE, 48)) + .withFieldAdornment(new FieldAdornment(AdornmentType.FILE_DOWNLOAD) + .withValue(AdornmentType.FileDownloadValues.FILE_NAME_FIELD, "fileName") + .withValue("deliveryModeField", "previewDelivery"))) .withVirtualField(new QVirtualFieldMetaData("downloadUrl", QFieldType.STRING).withLabel("Download").withIsEditable(false).withIsQuerySelectable(true) .withFieldAdornment(new FieldAdornment(AdornmentType.FILE_DOWNLOAD) .withValue(AdornmentType.FileDownloadValues.FILE_NAME_FIELD, "fileName") - .withValue(AdornmentType.FileDownloadValues.DOWNLOAD_URL_DYNAMIC, true))) + .withValue(AdornmentType.FileDownloadValues.DOWNLOAD_URL_DYNAMIC, true) + .withValue("deliveryModeField", "downloadDelivery"))) + .withVirtualField(new QVirtualFieldMetaData("previewDelivery", QFieldType.STRING).withLabel("Preview Delivery").withIsEditable(false).withIsQuerySelectable(true)) + .withVirtualField(new QVirtualFieldMetaData("downloadDelivery", QFieldType.STRING).withLabel("Download Delivery").withIsEditable(false).withIsQuerySelectable(true)) .withCustomizer(TableCustomizers.POST_QUERY_RECORD, new QCodeReference(AttachmentPreviewCustomizer.class)) .withSection(new QFieldSection("file", new QIcon("attach_file"), Tier.T1, List.of("fileName", "contentType", "sizeBytes", "description")).withLabel("File") .withAlternative(QFieldSectionAlternativeType.RECORD_VIEW, section -> section.setFieldNames(List.of("previewUrl", "downloadUrl", "fileName", "contentType", "sizeBytes", "description")))) .withSection(new QFieldSection("parent", new QIcon("link"), Tier.T2, - List.of("tableName", "recordId", "createdByUserId", "parentDeleted")).withLabel("Parent")) + List.of("tableName", "recordId", "createdByUserId", "parentDeleted", "retained")).withLabel("Parent")) .withSection(new QFieldSection("storage", new QIcon("storage"), Tier.T3, - List.of("id", "bucketName", "storageKey", "thumbnailKey", "sha256")).withLabel("Storage"))); + List.of("id", "bucketName", "storageKey", "thumbnailKey", "sha256", "intakeKey", "intakeFingerprint")).withLabel("Storage")), config)); } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/mywork/MyWorkService.java b/src/main/java/com/kingsrook/qbits/collaboration/mywork/MyWorkService.java index 6697ba5..47bc446 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/mywork/MyWorkService.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/mywork/MyWorkService.java @@ -23,18 +23,19 @@ package com.kingsrook.qbits.collaboration.mywork; import java.time.Instant; import java.util.ArrayList; +import java.util.Comparator; +import java.util.HashMap; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import com.kingsrook.qbits.collaboration.CollaborationRecords; import com.kingsrook.qbits.collaboration.PersonalRecords; -import com.kingsrook.qbits.collaboration.assignment.AssignmentService; import com.kingsrook.qbits.collaboration.model.CollaborationTables; import com.kingsrook.qbits.collaboration.security.ParentRecordAccess; -import com.kingsrook.qbits.collaboration.tasks.TasksService; import com.kingsrook.qqq.backend.core.exceptions.QException; import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException; import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy; import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; import com.kingsrook.qqq.backend.core.model.data.QRecord; @@ -43,18 +44,94 @@ import com.kingsrook.qqq.backend.core.model.data.QRecord; *******************************************************************************/ public class MyWorkService { + private static final int PAGE_SIZE = 200; + private static final Comparator TASK_ORDER = Comparator.comparing((QRecord row) -> row.getValueInstant("dueAt"), Comparator.nullsLast(Comparator.naturalOrder())) + .thenComparing(row -> row.getValueInteger("id")); + /******************************************************************************* ** Open task count and first ten, assigned record counts, and unread mentions. *******************************************************************************/ public Summary summary() throws QException { - List tasks = new TasksService().mineRows("OPEN"); + String user = PersonalRecords.currentUserId(); + List tasks = new ArrayList<>(); + long taskCount = scanVisible(CollaborationTables.TASK, new QQueryFilter() + .withCriteria("assigneeUserId", QCriteriaOperator.EQUALS, user) + .withCriteria("status", QCriteriaOperator.EQUALS, "OPEN"), row -> + { + tasks.add(row); + tasks.sort(TASK_ORDER); + if(tasks.size() > 10) + { + tasks.removeLast(); + } + }); Map assignments = new LinkedHashMap<>(); - for(QRecord row : new AssignmentService().mineRows(null, true)) + scanVisible(CollaborationTables.RECORD_ASSIGNMENT, new QQueryFilter() + .withCriteria("assigneeUserId", QCriteriaOperator.EQUALS, user), + row -> assignments.merge(row.getValueString("tableName"), 1L, Long::sum)); + long mentions = scanVisible(CollaborationTables.MENTION, new QQueryFilter() + .withCriteria("mentionedUserId", QCriteriaOperator.EQUALS, user) + .withCriteria("notifiedAt", QCriteriaOperator.IS_NOT_BLANK).withCriteria("readAt", QCriteriaOperator.IS_BLANK), row -> + { + // Only the visible count is needed for mentions. + }); + return (new Summary(taskCount, List.copyOf(tasks), assignments, mentions)); + } + + /******************************************************************************* + ** Bound retained rows and visibility decisions to one page. A starting high + ** watermark prevents concurrent inserts from extending this scan indefinitely. + ** Counts remain exact for visible rows scanned; no cross-request auth cache. + *******************************************************************************/ + private long scanVisible(String table, QQueryFilter filter, java.util.function.Consumer consumer) throws QException + { + filter.withCriteria("parentDeleted", QCriteriaOperator.EQUALS, false); + List last = CollaborationRecords.query(table, filter.clone().withOrderBy(new QFilterOrderBy("id", false)).withLimit(1)); + if(last.isEmpty()) { - assignments.merge(row.getValueString("tableName"), 1L, Long::sum); + return (0); + } + int maximum = last.getFirst().getValueInteger("id"); + Integer cursor = null; + long count = 0; + while(true) + { + QQueryFilter pageFilter = filter.clone().withCriteria("id", QCriteriaOperator.LESS_THAN_OR_EQUALS, maximum) + .withOrderBy(new QFilterOrderBy("id")).withLimit(PAGE_SIZE); + if(cursor != null) + { + pageFilter.withCriteria("id", QCriteriaOperator.GREATER_THAN, cursor); + } + List rows = CollaborationRecords.query(table, pageFilter); + Map, Boolean> parents = new HashMap<>(); + for(QRecord row : rows) + { + List parent = java.util.Arrays.asList(row.getValueString("tableName"), row.getValueString("recordId")); + if(!parents.containsKey(parent)) + { + try + { + ParentRecordAccess.requireVisibleParent(parent.get(0), parent.get(1)); + parents.put(parent, true); + } + catch(QNotFoundException expected) + { + parents.put(parent, false); + } + } + if(parents.get(parent)) + { + consumer.accept(row); + count++; + } + } + if(rows.size() < PAGE_SIZE || rows.getLast().getValueInteger("id") >= maximum) + { + return (count); + } + cursor = rows.getLast().getValueInteger("id"); } - return (new Summary((long) tasks.size(), tasks.stream().limit(10).toList(), assignments, (long) unread().size())); } /******************************************************************************* diff --git a/src/main/java/com/kingsrook/qbits/collaboration/notify/CollaborationNotifier.java b/src/main/java/com/kingsrook/qbits/collaboration/notify/CollaborationNotifier.java index 26bcfa1..1a08338 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/notify/CollaborationNotifier.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/notify/CollaborationNotifier.java @@ -31,4 +31,13 @@ public interface CollaborationNotifier ** Deliver a recipient-authorized notification. *******************************************************************************/ void notify(CollaborationNotification n); + + /******************************************************************************* + ** Legacy transports signal failure by throwing; adapters may explicitly refuse. + *******************************************************************************/ + default boolean deliver(CollaborationNotification notification) + { + notify(notification); + return (true); + } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/notify/NoOpNotifier.java b/src/main/java/com/kingsrook/qbits/collaboration/notify/NoOpNotifier.java index bbc09bf..ca63a6c 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/notify/NoOpNotifier.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/notify/NoOpNotifier.java @@ -34,4 +34,13 @@ public class NoOpNotifier implements CollaborationNotifier public void notify(CollaborationNotification n) { } + + /******************************************************************************* + ** No transport cannot acknowledge delivery. + *******************************************************************************/ + @Override + public boolean deliver(CollaborationNotification notification) + { + return (false); + } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/notify/Notifications.java b/src/main/java/com/kingsrook/qbits/collaboration/notify/Notifications.java index 296e9ab..f092e7b 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/notify/Notifications.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/notify/Notifications.java @@ -102,7 +102,10 @@ public final class Notifications if(!checkParent || new CollaborationRecordAccessChecker().canRead(notification.recipientUserId(), notification.sourceTableName(), notification.sourceRecordId()).orElse(false)) { CollaborationNotifier notifier = notifier(); - notifier.notify(notification); + if(!notifier.deliver(notification)) + { + return; + } if(!(notifier instanceof NoOpNotifier)) { try diff --git a/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceBackend.java b/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceBackend.java new file mode 100644 index 0000000..4237241 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceBackend.java @@ -0,0 +1,56 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.presence; + +import java.util.List; +import com.kingsrook.qqq.backend.core.exceptions.QException; + +/******************************************************************************* + ** Ephemeral storage boundary behind the permission-checking presence service. + *******************************************************************************/ +public interface PresenceBackend +{ + /*************************************************************************** + ** Reserve invisible renewal authority; null means capacity refusal. + ***************************************************************************/ + String openLease(String user, String name, String table, String pk) throws QException; + + /*************************************************************************** + ** Renew existing authority or throw PresenceLeaseExpiredException. + ***************************************************************************/ + boolean renewLease(String user, String name, String connection, String table, String pk, String mode, String field, Long sequence) throws QException; + + /*************************************************************************** + ** Renew a connection's ordered lease. + ***************************************************************************/ + boolean heartbeat(String user, String name, String connection, String table, String pk, String mode, String field, Long sequence) throws QException; + + /*************************************************************************** + ** Release only the authenticated user's connection. + ***************************************************************************/ + void leave(String user, String connection, String table, String pk, Long sequence) throws QException; + + /*************************************************************************** + ** Read a snapshot for subsequent recipient-specific permission filtering. + ***************************************************************************/ + List snapshot(String table, String pk) throws QException; +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceLeaseExpiredException.java b/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceLeaseExpiredException.java new file mode 100644 index 0000000..f3bc19e --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceLeaseExpiredException.java @@ -0,0 +1,36 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.presence; + +/******************************************************************************* + ** Expired or absent renewal authority requires opening a fresh presence lease. + *******************************************************************************/ +public final class PresenceLeaseExpiredException extends IllegalStateException +{ + /*************************************************************************** + ** Never include caller identifiers or stored data in this message. + ***************************************************************************/ + public PresenceLeaseExpiredException() + { + super("Presence lease expired; open a new lease"); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceRecoveringException.java b/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceRecoveringException.java new file mode 100644 index 0000000..11d4a58 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceRecoveringException.java @@ -0,0 +1,36 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.presence; + +/******************************************************************************* + ** Shared presence is rebuilding after unknown or lost Redis state. + *******************************************************************************/ +public final class PresenceRecoveringException extends IllegalStateException +{ + /*************************************************************************** + ** Never include caller identifiers or stored data in this message. + ***************************************************************************/ + public PresenceRecoveringException() + { + super("Presence is recovering; active viewers are rejoining"); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceRegistry.java b/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceRegistry.java index 46d5a79..2e29bcb 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceRegistry.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceRegistry.java @@ -21,33 +21,18 @@ package com.kingsrook.qbits.collaboration.presence; -import javax.crypto.Mac; -import javax.crypto.spec.SecretKeySpec; -import java.io.IOException; import java.io.Serializable; -import java.nio.charset.StandardCharsets; -import java.security.InvalidKeyException; -import java.security.MessageDigest; -import java.security.NoSuchAlgorithmException; -import java.security.SecureRandom; import java.time.Clock; -import java.time.Instant; import java.util.ArrayList; import java.util.HashMap; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import java.util.Objects; -import java.util.TreeMap; import java.util.WeakHashMap; import java.util.concurrent.Executors; import java.util.concurrent.ScheduledExecutorService; import java.util.concurrent.TimeUnit; -import com.fasterxml.jackson.annotation.JsonInclude; -import com.fasterxml.jackson.core.JsonGenerator; -import com.fasterxml.jackson.databind.SerializerProvider; -import com.fasterxml.jackson.databind.annotation.JsonSerialize; -import com.fasterxml.jackson.databind.ser.std.StdSerializer; import com.kingsrook.qqq.backend.core.logging.QLogger; import com.kingsrook.qqq.backend.core.model.metadata.QInstance; import com.kingsrook.qqq.backend.core.realtime.RealtimeBus; @@ -58,13 +43,12 @@ import com.kingsrook.qqq.backend.core.realtime.RealtimeMessage; ** Ephemeral per-pod presence replicated over a channel core refuses to clients. ** Publish outside the monitor: the in-memory bus waits for listener threads. *******************************************************************************/ -public final class PresenceRegistry implements AutoCloseable +public final class PresenceRegistry implements PresenceBackend, AutoCloseable { public static final String INTERNAL_CHANNEL = "instance:presence"; public static final long TTL_MILLIS = 45000; public static final int MAX_ENTRIES = 50000; private static final QLogger LOG = QLogger.getLogger(PresenceRegistry.class); - private static final byte[] SESSION_SECRET = sessionSecret(); private static final Map REGISTRIES = new WeakHashMap<>(); private final Map entries = new HashMap<>(); private final RealtimeBusInterface bus; @@ -133,32 +117,17 @@ public final class PresenceRegistry implements AutoCloseable } /******************************************************************************* - ** Each tab has its own lease; user views merge the live leases. The lease - ** keeps an opaque id of the server-side session (never the session uuid, - ** which is a bearer credential) and the time it started (renewals keep it). + ** Each tab has its own lease; user views merge the live leases. *******************************************************************************/ - private record Entry(String displayName, String mode, String fieldName, long at, String sessionId, long since) + private record Entry(String displayName, String mode, String fieldName, long at, Long sequence) { } /******************************************************************************* ** Transport-independent snapshot entry, never a persisted directory row. - ** Merged per user, connections counts the live leases behind it; per - ** connection (connections = 1), connectionId is an opaque per-lease id, - ** sessionId an opaque id of the signed-in session (HMAC, never reversible to - ** the session uuid) and since the lease start (ISO-8601). None of the three - ** is set on merged viewers. *******************************************************************************/ - public record Viewer(String userId, String displayName, String mode, String fieldName, Integer connections, @JsonInclude(JsonInclude.Include.NON_NULL) String connectionId, - @JsonInclude(JsonInclude.Include.NON_NULL) String sessionId, @JsonInclude(JsonInclude.Include.NON_NULL) String since) implements Serializable + public record Viewer(String userId, String displayName, String mode, String fieldName) implements Serializable { - /******************************************************************************* - ** A merged viewer behind a number of live leases. - *******************************************************************************/ - public Viewer(String userId, String displayName, String mode, String fieldName, Integer connections) - { - this(userId, displayName, mode, fieldName, connections, null, null, null); - } } /******************************************************************************* @@ -166,19 +135,21 @@ public final class PresenceRegistry implements AutoCloseable *******************************************************************************/ public boolean heartbeat(String user, String displayName, String connection, String table, String pk, String mode, String fieldName) { - return (heartbeat(user, displayName, connection, null, table, pk, mode, fieldName)); + return (heartbeat(user, displayName, connection, table, pk, mode, fieldName, null)); } /******************************************************************************* - ** Renew a lease tied to the caller's server-side session. Only an opaque HMAC - ** of the session uuid is kept and replicated; the uuid itself is dropped here. + ** Client sequence orders delayed HTTP requests independently of arrival time. *******************************************************************************/ - public boolean heartbeat(String user, String displayName, String connection, String sessionUuid, String table, String pk, String mode, String fieldName) + public boolean heartbeat(String user, String displayName, String connection, String table, String pk, String mode, String fieldName, Long sequence) { + if(connection != null && connection.startsWith("lease:")) + { + throw (new PresenceLeaseExpiredException()); + } expire(); var key = new Key(user, connection, table, pk); - long now = clock.millis(); - var entry = new Entry(displayName, mode, fieldName, now, opaqueSessionId(sessionUuid), now); + var entry = new Entry(displayName, mode, fieldName, clock.millis(), sequence); if(!apply(key, entry, false)) { return (false); @@ -187,14 +158,47 @@ public final class PresenceRegistry implements AutoCloseable return (true); } + /*************************************************************************** + ** Local mode issues local authority; HA applications must use shared Redis. + ***************************************************************************/ + @Override + public String openLease(String user, String displayName, String table, String pk) + { + expire(); + String connection = "lease:" + java.util.UUID.randomUUID(); + return (apply(new Key(user, connection, table, pk), new Entry(displayName, "PENDING", null, clock.millis(), 0L), false) ? connection : null); + } + + /*************************************************************************** + ** Check renewal and update under one monitor; no late request creates a lease. + ***************************************************************************/ + @Override + public boolean renewLease(String user, String displayName, String connection, String table, String pk, String mode, String fieldName, Long sequence) + { + if(connection == null || !connection.startsWith("lease:") || sequence == null || sequence < 1 || sequence > 9007199254740991L + || !List.of("VIEWING", "EDITING").contains(mode == null ? "" : mode)) + { + throw (new IllegalArgumentException("A server-issued lease, positive safe sequence and valid mode are required")); + } + expire(); + return (apply(new Key(user, connection, table, pk), new Entry(displayName, mode, fieldName, clock.millis(), sequence), false, true)); + } + /******************************************************************************* ** Leaves use the same internal channel so every pod removes the lease. *******************************************************************************/ public void leave(String user, String connection, String table, String pk) + { + leave(user, connection, table, pk, null); + } + + /******************************************************************************* + ** Departure participates in the same ordered client lease. + *******************************************************************************/ + public void leave(String user, String connection, String table, String pk, Long sequence) { var key = new Key(user, connection, table, pk); - long now = clock.millis(); - var entry = new Entry(null, null, null, now, null, now); + var entry = new Entry(null, null, null, clock.millis(), sequence); apply(key, entry, true); publish(key, entry, true); } @@ -204,6 +208,10 @@ public final class PresenceRegistry implements AutoCloseable *******************************************************************************/ private void publish(Key key, Entry entry, boolean leave) { + if(key.connection().startsWith("lease:")) + { + return; + } Map data = new LinkedHashMap<>(); data.put("userId", key.user()); data.put("connectionId", key.connection()); @@ -213,8 +221,8 @@ public final class PresenceRegistry implements AutoCloseable data.put("mode", entry.mode()); data.put("fieldName", entry.fieldName()); data.put("at", entry.at()); - data.put("sessionId", entry.sessionId()); data.put("leave", leave); + data.put("sequence", entry.sequence()); bus.publish(new RealtimeMessage().withChannel(INTERNAL_CHANNEL).withEvent("presence.heartbeat").withData(data)); } @@ -235,51 +243,61 @@ public final class PresenceRegistry implements AutoCloseable } expire(); long receivedAt = Math.min(at.longValue(), clock.millis()); + // Best-effort replication cannot grant or restore issued renewal authority. + if(connection.startsWith("lease:")) + { + return; + } if(receivedAt + TTL_MILLIS <= clock.millis()) { return; } apply(new Key(user, connection, table, (String) data.get("pk")), - new Entry((String) data.get("displayName"), (String) data.get("mode"), (String) data.get("fieldName"), receivedAt, (String) data.get("sessionId"), receivedAt), Boolean.TRUE.equals(data.get("leave"))); + new Entry((String) data.get("displayName"), (String) data.get("mode"), (String) data.get("fieldName"), receivedAt, + data.get("sequence") instanceof Number sequence ? sequence.longValue() : null), Boolean.TRUE.equals(data.get("leave"))); } /******************************************************************************* ** Unchanged renewals extend TTL without generating client event traffic. *******************************************************************************/ private boolean apply(Key key, Entry entry, boolean leave) + { + return (apply(key, entry, leave, false)); + } + + /******************************************************************************* + ** Local issued authority is checked atomically with the mutation. + *******************************************************************************/ + private boolean apply(Key key, Entry entry, boolean leave, boolean requireLive) { boolean changed; synchronized(this) { Entry old = entries.get(key); - if(old != null && old.at() > entry.at()) + if(requireLive && (old == null || old.mode() == null || old.at() + TTL_MILLIS <= clock.millis())) + { + throw (new PresenceLeaseExpiredException()); + } + if(old != null && old.sequence() != null + && (entry.sequence() == null || entry.sequence() < old.sequence() + || (entry.sequence().equals(old.sequence()) && (!leave || old.mode() == null)))) { return (true); } - if(!leave && old == null && entries.size() >= capacity) + if(old != null && old.sequence() == null && entry.sequence() == null + && (old.at() > entry.at() || (old.at() == entry.at() && old.mode() == null && !leave))) + { + return (true); + } + if(old == null && entries.size() >= capacity) { LOG.warn("Presence capacity reached; refusing a new target"); return (false); } - List before = projection(key.table(), key.pk()); - if(leave) - { - entries.remove(key); - } - else - { - ///////////////////////////////////////////////////////////////// - // a renewal keeps the lease start (pods may differ by a little: // - // the earliest start wins) and the first known session id // - ///////////////////////////////////////////////////////////////// - if(old != null) - { - entry = new Entry(entry.displayName(), entry.mode(), entry.fieldName(), entry.at(), - old.sessionId() != null ? old.sessionId() : entry.sessionId(), Math.min(old.since(), entry.since())); - } - entries.put(key, entry); - } - changed = !before.equals(projection(key.table(), key.pk())); + List before = snapshotInternal(key.table(), key.pk()); + // Keep a bounded departure marker until old messages can no longer be fresh. + entries.put(key, entry); + changed = !before.equals(snapshotInternal(key.table(), key.pk())); } if(changed) { @@ -326,137 +344,16 @@ public final class PresenceRegistry implements AutoCloseable *******************************************************************************/ private List snapshotInternal(String table, String pk) { - Map users = new TreeMap<>(); - Map connections = new TreeMap<>(); - for(var row : live(table, pk)) - { - users.merge(row.getKey().user(), row.getValue(), (old, next) -> - (!Objects.equals(old.mode(), next.mode()) ? "EDITING".equals(next.mode()) : next.at() >= old.at()) ? next : old); - connections.merge(row.getKey().user(), 1, Integer::sum); - } - return (users.entrySet().stream().map(row -> new Viewer(row.getKey(), row.getValue().displayName(), row.getValue().mode(), - pk == null ? null : row.getValue().fieldName(), connections.get(row.getKey()))).toList()); - } - - /******************************************************************************* - ** One viewer per live lease, ordered by user, lease start and opaque id. A - ** table target counts each connection once per record it holds a lease on. - *******************************************************************************/ - private List connectionsInternal(String table, String pk) - { - return (live(table, pk).stream() - .map(row -> new Viewer(row.getKey().user(), row.getValue().displayName(), row.getValue().mode(), pk == null ? null : row.getValue().fieldName(), 1, - opaqueConnectionId(row.getKey().user(), row.getKey().connection()), row.getValue().sessionId(), Instant.ofEpochMilli(row.getValue().since()).toString())) - .sorted(java.util.Comparator.comparing(Viewer::userId).thenComparing(Viewer::since).thenComparing(Viewer::connectionId)) - .toList()); - } - - /******************************************************************************* - ** Live leases on a table (any record when pk is null) or on one record. - *******************************************************************************/ - private List> live(String table, String pk) - { - List> result = new ArrayList<>(); + Map users = new java.util.TreeMap<>(); for(var row : entries.entrySet()) { - if(row.getKey().table().equals(table) && (pk == null || Objects.equals(row.getKey().pk(), pk)) && row.getValue().at() + TTL_MILLIS > clock.millis()) + if(row.getValue().mode() != null && !"PENDING".equals(row.getValue().mode()) && row.getKey().table().equals(table) && (pk == null || Objects.equals(row.getKey().pk(), pk)) && row.getValue().at() + TTL_MILLIS > clock.millis()) { - result.add(row); + users.merge(row.getKey().user(), row.getValue(), (old, next) -> + (!Objects.equals(old.mode(), next.mode()) ? "EDITING".equals(next.mode()) : next.at() >= old.at()) ? next : old); } } - return (result); - } - - /******************************************************************************* - ** What clients can observe: merged viewers and per-connection viewers. Lease - ** renewals that change neither generate no event. - *******************************************************************************/ - private List projection(String table, String pk) - { - List result = new ArrayList<>(snapshotInternal(table, pk)); - result.addAll(connectionsInternal(table, pk)); - return (result); - } - - /******************************************************************************* - ** Opaque, stable per-lease id: "c" and the first 16 hex digits of - ** SHA-256(userId + "\n" + connectionId). A client can recognise its own - ** connection; nobody learns another client's connection id. - *******************************************************************************/ - public static String opaqueConnectionId(String userId, String connectionId) - { - try - { - byte[] digest = MessageDigest.getInstance("SHA-256").digest((userId + "\n" + connectionId).getBytes(StandardCharsets.UTF_8)); - StringBuilder result = new StringBuilder("c"); - for(int i = 0; i < 8; i++) - { - result.append(String.format("%02x", digest[i])); - } - return (result.toString()); - } - catch(NoSuchAlgorithmException e) - { - throw (new IllegalStateException("SHA-256 is unavailable", e)); - } - } - - /******************************************************************************* - ** Opaque session id: "s" and the first 16 hex digits of - ** HMAC-SHA256(secret, sessionUuid), or null without a session. The secret is - ** QQQ_PRESENCE_SESSION_SECRET (system property or environment; share it across - ** pods for ids that agree between pods) or else random per process. Session - ** uuids are bearer credentials and must never be stored or published. - *******************************************************************************/ - public static String opaqueSessionId(String sessionUuid) - { - if(sessionUuid == null || sessionUuid.isBlank()) - { - return (null); - } - try - { - Mac mac = Mac.getInstance("HmacSHA256"); - mac.init(new SecretKeySpec(SESSION_SECRET, "HmacSHA256")); - byte[] digest = mac.doFinal(sessionUuid.getBytes(StandardCharsets.UTF_8)); - StringBuilder result = new StringBuilder("s"); - for(int i = 0; i < 8; i++) - { - result.append(String.format("%02x", digest[i])); - } - return (result.toString()); - } - catch(NoSuchAlgorithmException | InvalidKeyException e) - { - throw (new IllegalStateException("HmacSHA256 is unavailable", e)); - } - } - - /******************************************************************************* - ** Configured shared secret, or 32 random bytes for this process. - *******************************************************************************/ - private static byte[] sessionSecret() - { - String configured = System.getProperty("QQQ_PRESENCE_SESSION_SECRET", System.getenv("QQQ_PRESENCE_SESSION_SECRET")); - if(configured != null && !configured.isBlank()) - { - return (configured.getBytes(StandardCharsets.UTF_8)); - } - byte[] random = new byte[32]; - new SecureRandom().nextBytes(random); - return (random); - } - - /******************************************************************************* - ** One viewer per live lease (opt-in per-connection presence, #54). - *******************************************************************************/ - public List connections(String table, String pk) - { - expire(); - synchronized(this) - { - return (connectionsInternal(table, pk)); - } + return (users.entrySet().stream().map(row -> new Viewer(row.getKey(), row.getValue().displayName(), row.getValue().mode(), pk == null ? null : row.getValue().fieldName())).toList()); } /******************************************************************************* @@ -467,27 +364,12 @@ public final class PresenceRegistry implements AutoCloseable expire(); synchronized(this) { - Map result = new TreeMap<>(); + Map result = new java.util.TreeMap<>(); entries.keySet().stream().map(Key::table).distinct().forEach(table -> result.put(table, snapshotInternal(table, null).size())); return (result); } } - /******************************************************************************* - ** Live lease (connection) counts per table, next to the user counts above. - *******************************************************************************/ - public Map connectionCounts() - { - expire(); - synchronized(this) - { - Map result = new TreeMap<>(); - entries.keySet().stream().map(Key::table).distinct().forEach(table -> result.put(table, live(table, null).size())); - result.values().removeIf(count -> count == 0); - return (result); - } - } - /******************************************************************************* ** Record focus stays on its record channel; table events contain table snapshots. *******************************************************************************/ @@ -511,97 +393,23 @@ public final class PresenceRegistry implements AutoCloseable { data.put("pk", pk); } - Viewers viewers = new Viewers(); - Viewers leases = new Viewers(); + ArrayList> viewers = new ArrayList<>(); synchronized(this) { - snapshotInternal(table, pk).forEach(viewer -> viewers.add(toMap(viewer))); - connectionsInternal(table, pk).forEach(viewer -> leases.add(toMap(viewer))); + for(Viewer viewer : snapshotInternal(table, pk)) + { + LinkedHashMap value = new LinkedHashMap<>(); + value.put("userId", viewer.userId()); + value.put("displayName", viewer.displayName()); + value.put("mode", viewer.mode()); + value.put("fieldName", viewer.fieldName()); + viewers.add(value); + } } data.put("viewers", viewers); - data.put("leaseCount", leases.size()); - data.put("leases", leases); bus.publish(new RealtimeMessage().withChannel(channel).withEvent("presence.changed").withData(data)); } - /******************************************************************************* - ** Event shape of a viewer; per-connection fields only when set. - *******************************************************************************/ - private static LinkedHashMap toMap(Viewer viewer) - { - LinkedHashMap value = new LinkedHashMap<>(); - value.put("userId", viewer.userId()); - value.put("displayName", viewer.displayName()); - value.put("mode", viewer.mode()); - value.put("fieldName", viewer.fieldName()); - value.put("connections", viewer.connections()); - if(viewer.connectionId() != null) - { - value.put("connectionId", viewer.connectionId()); - if(viewer.sessionId() != null) - { - value.put("sessionId", viewer.sessionId()); - } - value.put("since", viewer.since()); - } - return (value); - } - - /******************************************************************************* - ** Event viewers list that is never omitted: the SSE writer serializes event data - ** with NON_EMPTY inclusion, which would drop an empty list, but the schema - ** declares viewers a required array, so the last leave must send [] (#52). - *******************************************************************************/ - @JsonSerialize(using = ViewersSerializer.class) - public static final class Viewers extends ArrayList> - { - } - - - - /******************************************************************************* - ** Writes each viewer with the caller's mapper settings and reports never empty. - *******************************************************************************/ - public static final class ViewersSerializer extends StdSerializer - { - /******************************************************************************* - ** Jackson instantiates the serializer reflectively. - *******************************************************************************/ - public ViewersSerializer() - { - super(Viewers.class); - } - - - - /******************************************************************************* - ** Always an array, including the empty array after the last viewer leaves. - *******************************************************************************/ - @Override - public void serialize(Viewers viewers, JsonGenerator generator, SerializerProvider provider) throws IOException - { - generator.writeStartArray(viewers, viewers.size()); - for(LinkedHashMap viewer : viewers) - { - provider.defaultSerializeValue(viewer, generator); - } - generator.writeEndArray(); - } - - - - /******************************************************************************* - ** NON_EMPTY inclusion consults this; an empty viewer list is still a value. - *******************************************************************************/ - @Override - public boolean isEmpty(SerializerProvider provider, Viewers viewers) - { - return (false); - } - } - - - /******************************************************************************* ** Release listeners and timer on host shutdown or in tests. *******************************************************************************/ diff --git a/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceScopeProvider.java b/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceScopeProvider.java new file mode 100644 index 0000000..71f9d89 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceScopeProvider.java @@ -0,0 +1,33 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.presence; + +/******************************************************************************* + ** Host resolves tenant scope from authenticated context, never request parameters. + *******************************************************************************/ +public interface PresenceScopeProvider +{ + /*************************************************************************** + ** Return a stable nonblank scope; single-tenant hosts explicitly return one. + ***************************************************************************/ + String tenantScope(); +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceService.java b/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceService.java index bc1b9b8..1803c7e 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceService.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceService.java @@ -30,19 +30,62 @@ import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException; import com.kingsrook.qqq.backend.core.exceptions.QException; import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException; import com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordFeature; +import com.kingsrook.qqq.backend.core.realtime.PresenceFieldVisibility; /******************************************************************************* ** Authenticated, permission-checked boundary for presence; nothing is persisted. *******************************************************************************/ public class PresenceService { + /******************************************************************************* + ** Reserve invisible authority only for an authenticated, visible target. + *******************************************************************************/ + public String openLease(String table, String pk) throws QException + { + String user = PersonalRecords.currentUserId(); + String canonical = authorize(table, pk); + String name = QContext.getQSession().getUser().getFullName(); + String lease = registry().openLease(user, name == null || name.isBlank() ? user : name, table, canonical); + if(lease == null) + { + throw (new com.kingsrook.qqq.backend.core.exceptions.QServiceUnavailableException("Presence capacity reached; retry later")); + } + return (lease); + } + + /******************************************************************************* + ** Issued renewals share the same identity, parent and field access checks. + *******************************************************************************/ + public List renewLease(String connection, String table, String pk, String mode, String field, Long sequence) throws QException + { + requireIssuedLease(connection, sequence); + return (renew(connection, table, pk, mode, field, sequence, true)); + } + /******************************************************************************* ** Recheck visibility on every renewal and remove a refused caller's old lease. *******************************************************************************/ public List heartbeat(String connection, String table, String pk, String mode, String field) throws QException + { + return (heartbeat(connection, table, pk, mode, field, null)); + } + + /******************************************************************************* + ** Ordered clients supply a monotonically increasing sequence per connection. + *******************************************************************************/ + public List heartbeat(String connection, String table, String pk, String mode, String field, Long sequence) throws QException + { + return (renew(connection, table, pk, mode, field, sequence, false)); + } + + /******************************************************************************* + ** Keep legacy and issued requests on one permission and projection boundary. + *******************************************************************************/ + private List renew(String connection, String table, String pk, String mode, String field, Long sequence, boolean issued) throws QException { String user = PersonalRecords.currentUserId(); validateConnection(connection); + validateSequence(sequence); pk = canonicalKey(table, pk); try { @@ -50,34 +93,63 @@ public class PresenceService } catch(QException e) { - registry().leave(user, connection, table, pk); + try + { + registry().leave(user, connection, table, pk, sequence); + } + catch(QException cleanupFailure) + { + // Preserve the access refusal; an unreachable store expires the old lease. + } throw (e); } if(!List.of("VIEWING", "EDITING").contains(mode == null ? "" : mode)) { throw (new QBadRequestException("Mode must be VIEWING or EDITING")); } - if(field != null && (pk == null || QContext.getQInstance().getTable(table).getFields().get(field) == null)) + if(field != null && (pk == null || !PresenceFieldVisibility.visibleFieldNames(table).contains(field))) { throw (new QBadRequestException("Unknown focused field")); } - String name = QContext.getQSession().getUser().getFullName(); - if(!registry().heartbeat(user, name == null || name.isBlank() ? user : name, connection, QContext.getQSession().getUuid(), table, pk, mode, field)) + if(!issued && RedisPresenceConfig.forInstance(QContext.getQInstance()) != null) { - throw (new QBadRequestException("Presence capacity reached; retry later")); + throw (new QBadRequestException("Shared presence requires a server-issued lease; use the presence/leases endpoints")); } - return (registry().snapshot(table, pk)); + String name = QContext.getQSession().getUser().getFullName(); + String displayName = name == null || name.isBlank() ? user : name; + var backend = registry(); + boolean accepted = issued ? backend.renewLease(user, displayName, connection, table, pk, mode, field, sequence) + : backend.heartbeat(user, displayName, connection, table, pk, mode, field, sequence); + if(!accepted) + { + throw (new com.kingsrook.qqq.backend.core.exceptions.QServiceUnavailableException("Presence capacity reached; retry later")); + } + return (visibleSnapshot(table, pk)); } /******************************************************************************* ** Namespacing by user prevents one user from removing another user's connection. *******************************************************************************/ public void leave(String connection, String table, String pk) throws QException + { + leave(connection, table, pk, null); + } + + /******************************************************************************* + ** Ordered departure cannot be undone by an earlier renewal arriving late. + *******************************************************************************/ + public void leave(String connection, String table, String pk, Long sequence) throws QException { String user = PersonalRecords.currentUserId(); validateConnection(connection); - String canonical = authorize(table, pk); - registry().leave(user, connection, table, canonical); + validateSequence(sequence); + if(connection.startsWith("lease:")) + { + requireIssuedLease(connection, sequence); + } + // Removal is scoped to the caller; revoked readers must still release their lease. + String canonical = canonicalKey(table, pk); + registry().leave(user, connection, table, canonical, sequence); } /******************************************************************************* @@ -86,16 +158,17 @@ public class PresenceService public List snapshot(String table, String pk) throws QException { PersonalRecords.currentUserId(); - return (registry().snapshot(table, authorize(table, pk))); + return (visibleSnapshot(table, authorize(table, pk))); } /******************************************************************************* - ** One viewer per live lease, behind the same access boundary as snapshot. + ** Project a fresh recipient view without mutating shared registry entries. *******************************************************************************/ - public List connections(String table, String pk) throws QException + private List visibleSnapshot(String table, String pk) throws QException { - PersonalRecords.currentUserId(); - return (registry().connections(table, authorize(table, pk))); + var visible = PresenceFieldVisibility.visibleFieldNames(table); + return (registry().snapshot(table, pk).stream().map(viewer -> new PresenceRegistry.Viewer(viewer.userId(), viewer.displayName(), viewer.mode(), + viewer.fieldName() != null && visible.contains(viewer.fieldName()) ? viewer.fieldName() : null)).toList()); } /******************************************************************************* @@ -120,10 +193,6 @@ public class PresenceService *******************************************************************************/ private String canonicalKey(String tableName, String pk) throws QNotFoundException { - if(pk == null) - { - return (null); - } try { var table = QContext.getQInstance().getTable(tableName); @@ -131,6 +200,10 @@ public class PresenceService { throw (new IllegalArgumentException("Unknown table")); } + if(pk == null) + { + return (null); + } return (com.kingsrook.qqq.backend.core.utils.ValueUtils.getValueAsFieldType(table.getField(table.getPrimaryKeyField()).getType(), pk).toString()); } catch(RuntimeException e) @@ -151,10 +224,39 @@ public class PresenceService } /******************************************************************************* - ** All endpoints on a pod share the eagerly installed instance registry. + ** Issued handles require ordered activation and departure requests. *******************************************************************************/ - private PresenceRegistry registry() + private void requireIssuedLease(String connection, Long sequence) throws QBadRequestException { + validateConnection(connection); + validateSequence(sequence); + if(!connection.startsWith("lease:") || sequence == null || sequence < 1) + { + throw (new QBadRequestException("A server-issued lease and positive sequence are required")); + } + } + + /******************************************************************************* + ** Restrict ordered leases to exact nonnegative JavaScript integers. + *******************************************************************************/ + private void validateSequence(Long sequence) throws QBadRequestException + { + if(sequence != null && (sequence < 0 || sequence > 9007199254740991L)) + { + throw (new QBadRequestException("Presence sequence must be a nonnegative safe integer")); + } + } + + /******************************************************************************* + ** Select the configured backend without creating an unused local registry. + *******************************************************************************/ + private PresenceBackend registry() throws QException + { + var config = RedisPresenceConfig.forInstance(QContext.getQInstance()); + if(config != null) + { + return (new RedisPresenceBackend(config)); + } return (PresenceRegistry.get(QContext.getQInstance())); } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceAudience.java b/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceAudience.java new file mode 100644 index 0000000..824c4b4 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceAudience.java @@ -0,0 +1,54 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.presence; + +import com.kingsrook.qbits.redis.api.RedisNamespace; +import com.kingsrook.qqq.backend.core.actions.customizers.QCodeLoader; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.realtime.RealtimeAudienceProvider; + +/******************************************************************************* + ** Presence hints are visible only within the authenticated storage namespace. + *******************************************************************************/ +public class RedisPresenceAudience implements RealtimeAudienceProvider +{ + public static final String EVENT = "presence.invalidated"; + + /*************************************************************************** + ** Recompute recipient scope on every delivery and replay; never trust the hint. + ***************************************************************************/ + @Override + public boolean allows(String audience) throws Exception + { + var config = RedisPresenceConfig.forInstance(QContext.getQInstance()); + return (config != null && namespace(config).key("audience").equals(audience)); + } + + /*************************************************************************** + ** Storage and delivery must derive exactly the same host-owned tenant scope. + ***************************************************************************/ + public static RedisNamespace namespace(RedisPresenceConfig config) throws Exception + { + var scope = QCodeLoader.getAdHoc(PresenceScopeProvider.class, config.getScopeProvider()).tenantScope(); + return (new RedisNamespace(config.getDeployment(), config.getApplication(), scope, "presence:" + config.getRecoveryGeneration())); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceBackend.java b/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceBackend.java new file mode 100644 index 0000000..2bdb5cf --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceBackend.java @@ -0,0 +1,190 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.presence; + +import java.time.Duration; +import java.util.List; +import com.kingsrook.qbits.redis.api.RedisAccess; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QServiceUnavailableException; + +/******************************************************************************* + ** Request-scoped Redis adapter; tenant identity is re-derived for every request. + *******************************************************************************/ +public class RedisPresenceBackend implements PresenceBackend +{ + private final RedisPresenceStore store; + private final String audience; + + /*************************************************************************** + ** Require a running named pool and an explicit host-derived tenant namespace. + ***************************************************************************/ + public RedisPresenceBackend(RedisPresenceConfig config) throws QServiceUnavailableException + { + try + { + var namespace = RedisPresenceAudience.namespace(config); + audience = namespace.key("audience"); + var client = RedisAccess.scripts(QContext.getQInstance(), config.getConnectionName()); + store = new RedisPresenceStore(client, namespace, Duration.ofMillis(PresenceRegistry.TTL_MILLIS), config.getCapacity()); + } + catch(Exception failure) + { + throw (unavailable()); + } + } + + /*************************************************************************** + ** Reservations are invisible and do not require a client refresh hint. + ***************************************************************************/ + @Override + public String openLease(String user, String name, String table, String pk) throws QServiceUnavailableException + { + try + { + return (store.openLease(user, name, table, pk)); + } + catch(RuntimeException failure) + { + throw (unavailable()); + } + } + + /*************************************************************************** + ** Preserve expiry as a rejoin signal, distinct from infrastructure failure. + ***************************************************************************/ + @Override + public boolean renewLease(String user, String name, String connection, String table, String pk, String mode, String field, Long sequence) throws QServiceUnavailableException + { + try + { + boolean accepted = store.renewLease(user, name, connection, table, pk, mode, field, sequence); + if(accepted) + { + changed(table, pk); + } + return (accepted); + } + catch(PresenceLeaseExpiredException failure) + { + throw (failure); + } + catch(RuntimeException failure) + { + throw (unavailable()); + } + } + + /*************************************************************************** + ** Do not fall back to isolated local state when a shared store is unavailable. + ***************************************************************************/ + @Override + public boolean heartbeat(String user, String name, String connection, String table, String pk, String mode, String field, Long sequence) throws QServiceUnavailableException + { + try + { + boolean accepted = store.heartbeat(user, name, connection, table, pk, mode, field, sequence); + if(accepted) + { + changed(table, pk); + } + return (accepted); + } + catch(RuntimeException failure) + { + throw (unavailable()); + } + } + + /*************************************************************************** + ** If unavailable, the existing lease still expires using Redis server time. + ***************************************************************************/ + @Override + public void leave(String user, String connection, String table, String pk, Long sequence) throws QServiceUnavailableException + { + try + { + if(!store.leave(user, connection, table, pk, sequence)) + { + throw (unavailable()); + } + changed(table, pk); + } + catch(RuntimeException failure) + { + throw (unavailable()); + } + } + + /*************************************************************************** + ** Failure is explicit; an empty result is reserved for a successfully read room. + ***************************************************************************/ + @Override + public List snapshot(String table, String pk) throws QServiceUnavailableException + { + try + { + return (store.snapshot(table, pk)); + } + catch(PresenceRecoveringException recovering) + { + throw (new QServiceUnavailableException("Presence is recovering; active viewers are rejoining. Please retry.")); + } + catch(RuntimeException failure) + { + throw (unavailable()); + } + } + + /*************************************************************************** + ** Never expose client exceptions, endpoint details or credentials to browsers. + ***************************************************************************/ + private static QServiceUnavailableException unavailable() + { + return (new QServiceUnavailableException("Presence is temporarily unavailable. Please retry.")); + } + + /*************************************************************************** + ** Best-effort hints carry identifiers only; authorized snapshots remain truth. + ***************************************************************************/ + private void changed(String table, String pk) + { + try + { + var bus = com.kingsrook.qqq.backend.core.realtime.RealtimeBus.get(QContext.getQInstance()); + bus.publish(new com.kingsrook.qqq.backend.core.realtime.RealtimeMessage() + .withChannel("table:" + table).withEvent(RedisPresenceAudience.EVENT).withAudience(audience) + .withData(java.util.Map.of("table", table))); + if(pk != null) + { + bus.publish(new com.kingsrook.qqq.backend.core.realtime.RealtimeMessage() + .withChannel("record:" + table + ":" + pk).withEvent(RedisPresenceAudience.EVENT).withAudience(audience) + .withData(java.util.Map.of("table", table, "pk", pk))); + } + } + catch(RuntimeException failure) + { + com.kingsrook.qqq.backend.core.logging.QLogger.getLogger(RedisPresenceBackend.class) + .warn("Presence hint delivery failed; clients must reconcile through authorized reads"); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceConfig.java b/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceConfig.java new file mode 100644 index 0000000..b761c6a --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceConfig.java @@ -0,0 +1,174 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.presence; + +import java.io.Serializable; +import java.util.List; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; + +/******************************************************************************* + ** Host configuration for shared presence, including authoritative tenant scope. + *******************************************************************************/ +public class RedisPresenceConfig implements Serializable +{ + private String connectionName = null; + private String deployment = null; + private String application = null; + private QCodeReference scopeProvider = null; + private Integer capacity = 2000; + private String recoveryGeneration = "initial"; + + /*************************************************************************** + ** Inspect configuration without creating a runtime pool or local registry. + ***************************************************************************/ + public static RedisPresenceConfig forInstance(com.kingsrook.qqq.backend.core.model.metadata.QInstance instance) + { + var config = instance.getQBits().values().stream().map(bit -> bit.getConfig()) + .filter(com.kingsrook.qbits.collaboration.CollaborationQBitConfig.class::isInstance) + .map(com.kingsrook.qbits.collaboration.CollaborationQBitConfig.class::cast).findFirst().orElse(null); + return (config == null ? null : config.getRedisPresence()); + } + + /*************************************************************************** + ** Explicit scope prevents accidental sharing between deployments or tenants. + ***************************************************************************/ + public void validate(List errors) + { + if(connectionName == null || connectionName.isBlank() || deployment == null || deployment.isBlank() + || application == null || application.isBlank() || scopeProvider == null) + { + errors.add("Redis presence requires a named connection, deployment, application and tenant scope provider"); + } + if(capacity == null || capacity < 1 || capacity > 10000) + { + errors.add("Redis presence capacity must be between 1 and 10000"); + } + if(recoveryGeneration == null || !recoveryGeneration.matches("[A-Za-z0-9_-]{1,64}")) + { + errors.add("Redis presence recovery generation must contain 1 to 64 letters, digits, underscores or hyphens"); + } + } + + /*************************************************************************** + ** Return connectionName. + ***************************************************************************/ + public String getConnectionName() + { + return (connectionName); + } + + /*************************************************************************** + ** Configure connectionName. + ***************************************************************************/ + public RedisPresenceConfig withConnectionName(String value) + { + connectionName = value; + return (this); + } + + /*************************************************************************** + ** Return deployment. + ***************************************************************************/ + public String getDeployment() + { + return (deployment); + } + + /*************************************************************************** + ** Configure deployment. + ***************************************************************************/ + public RedisPresenceConfig withDeployment(String value) + { + deployment = value; + return (this); + } + + /*************************************************************************** + ** Return application. + ***************************************************************************/ + public String getApplication() + { + return (application); + } + + /*************************************************************************** + ** Configure application. + ***************************************************************************/ + public RedisPresenceConfig withApplication(String value) + { + application = value; + return (this); + } + + /*************************************************************************** + ** Return scopeProvider. + ***************************************************************************/ + public QCodeReference getScopeProvider() + { + return (scopeProvider); + } + + /*************************************************************************** + ** Configure scopeProvider. + ***************************************************************************/ + public RedisPresenceConfig withScopeProvider(QCodeReference value) + { + scopeProvider = value; + return (this); + } + + /*************************************************************************** + ** Return capacity. + ***************************************************************************/ + public Integer getCapacity() + { + return (capacity); + } + + /*************************************************************************** + ** Configure capacity. + ***************************************************************************/ + public RedisPresenceConfig withCapacity(Integer value) + { + capacity = value; + return (this); + } + + + /*************************************************************************** + ** Host-owned generation, stored outside Redis and changed after restoring data. + ***************************************************************************/ + public String getRecoveryGeneration() + { + return (recoveryGeneration); + } + + + /*************************************************************************** + ** All nodes must use the same new generation before maintenance traffic resumes. + ***************************************************************************/ + public RedisPresenceConfig withRecoveryGeneration(String value) + { + recoveryGeneration = value; + return (this); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceStore.java b/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceStore.java new file mode 100644 index 0000000..69db925 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceStore.java @@ -0,0 +1,254 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.presence; + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.time.Duration; +import java.util.ArrayList; +import java.util.Base64; +import java.util.HexFormat; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.TreeMap; +import com.kingsrook.qbits.redis.api.RedisNamespace; +import com.kingsrook.qbits.redis.api.RedisScript; +import com.kingsrook.qbits.redis.api.RedisScriptExecutor; +import com.kingsrook.qqq.backend.core.utils.JsonUtils; + +/******************************************************************************* + ** Shared ephemeral presence. Callers must authorize identity, scope and fields. + ** Scripts atomically update record/table membership using the Redis server clock. + *******************************************************************************/ +public final class RedisPresenceStore +{ + private static final RedisScript MUTATE = new RedisScript(resource("redis-recovery.lua") + "\n" + resource("redis-mutate.lua")); + private static final RedisScript SNAPSHOT = new RedisScript(resource("redis-recovery.lua") + "\n" + resource("redis-snapshot.lua")); + private final RedisScriptExecutor client; + private final RedisNamespace namespace; + private final long ttlMillis; + private final int capacity; + + /*************************************************************************** + ** Limit each indexed room, including departure markers, to bound script work. + ***************************************************************************/ + public RedisPresenceStore(RedisScriptExecutor client, RedisNamespace namespace, Duration ttl, int capacity) + { + this.client = Objects.requireNonNull(client); + this.namespace = Objects.requireNonNull(namespace); + this.ttlMillis = Objects.requireNonNull(ttl).toMillis(); + if(ttlMillis < 1 || ttlMillis > Duration.ofMinutes(5).toMillis() || capacity < 1 || capacity > 10000) + { + throw (new IllegalArgumentException("Presence requires a positive TTL of at most five minutes and capacity of 1 to 10000")); + } + this.capacity = capacity; + } + + /*************************************************************************** + ** Renew a connection; duplicate or delayed sequences never extend its lease. + ***************************************************************************/ + public boolean heartbeat(String user, String displayName, String connection, String table, String pk, String mode, String field, Long sequence) + { + if(connection != null && connection.startsWith("lease:")) + { + throw (new PresenceLeaseExpiredException()); + } + if(!List.of("VIEWING", "EDITING").contains(mode == null ? "" : mode)) + { + throw (new IllegalArgumentException("Invalid presence mode")); + } + return (mutate(user, displayName, connection, table, pk, mode, field, sequence, false, "legacy")); + } + + /*************************************************************************** + ** Reserve an invisible server-issued lease. Null means room capacity refused. + ** Only a subsequent renewal makes it visible; lost responses expire silently. + ***************************************************************************/ + public String openLease(String user, String displayName, String table, String pk) + { + String connection = "lease:" + java.util.UUID.randomUUID(); + return (mutate(user, displayName, connection, table, pk, "VIEWING", null, 0L, false, "open") ? connection : null); + } + + /*************************************************************************** + ** Activate or renew existing authority, never recreate missing/departed state. + ** Callers must reauthorize identity, parent visibility and focused fields. + ***************************************************************************/ + public boolean renewLease(String user, String displayName, String connection, String table, String pk, String mode, String field, Long sequence) + { + if(connection == null || !connection.startsWith("lease:") || sequence == null || sequence < 1) + { + throw (new IllegalArgumentException("A server-issued lease and positive sequence are required")); + } + if(!List.of("VIEWING", "EDITING").contains(mode == null ? "" : mode)) + { + throw (new IllegalArgumentException("Invalid presence mode")); + } + return (mutate(user, displayName, connection, table, pk, mode, field, sequence, false, "renew")); + } + + /*************************************************************************** + ** Retain a bounded departure marker so older HTTP renewals cannot undo leave. + ***************************************************************************/ + public boolean leave(String user, String connection, String table, String pk, Long sequence) + { + return (mutate(user, user, connection, table, pk, "VIEWING", null, sequence, true, "legacy")); + } + + /*************************************************************************** + ** Redis failures propagate; they must never be presented as an empty room. + ***************************************************************************/ + public List snapshot(String table, String pk) + { + List keys = roomKeys(table, pk, true); + Object result = client.execute(SNAPSHOT, keys, List.of(pk == null ? "" : ":" + encode(pk, 1024), + Long.toString(ttlMillis), java.util.UUID.randomUUID().toString())); + if(result instanceof Number number && number.longValue() == -2) + { + throw (new PresenceRecoveringException()); + } + Map users = new TreeMap<>(); + for(Object value : (List) result) + { + try + { + Stored entry = JsonUtils.toObject((String) value, Stored.class); + if(!entry.departed() && !entry.pending()) + { + users.merge(entry.userId(), entry, (old, next) -> + (!Objects.equals(old.mode(), next.mode()) ? "EDITING".equals(next.mode()) : next.at() >= old.at()) ? next : old); + } + } + catch(IOException e) + { + throw (new IllegalStateException("Invalid stored presence data", e)); + } + } + return (users.values().stream().map(entry -> new PresenceRegistry.Viewer(entry.userId(), entry.displayName(), entry.mode(), + pk == null ? null : entry.fieldName())).toList()); + } + + /*************************************************************************** + ** Validate bounded inputs before sending any Lua command to Redis. + ***************************************************************************/ + private boolean mutate(String user, String displayName, String connection, String table, String pk, String mode, String field, Long sequence, boolean departed, String operation) + { + String member = encode(user, 255) + ":" + encode(connection, 255) + ":" + (pk == null ? "table" : encode(pk, 1024)); + encode(displayName, 1024); + if(field != null) + { + encode(field, 255); + } + if(sequence != null && (sequence < 0 || sequence > 9007199254740991L)) + { + throw (new IllegalArgumentException("Presence sequence must be a nonnegative safe integer")); + } + Stored value = new Stored(user, displayName, mode, pk == null ? null : field, 0, sequence, departed, false, false, null); + Object result = client.execute(MUTATE, roomKeys(table, pk, true), List.of(member, JsonUtils.toJson(value), + sequence == null ? "" : sequence.toString(), departed ? "1" : "0", Long.toString(ttlMillis), Integer.toString(capacity), pk == null ? "1" : "0", operation, + java.util.UUID.randomUUID().toString())); + if(((Number) result).longValue() == -1) + { + throw (new PresenceLeaseExpiredException()); + } + return (((Number) result).longValue() == 1); + } + + /*************************************************************************** + ** One table hash tag keeps record/table updates atomic on Redis Cluster slots. + ***************************************************************************/ + private List roomKeys(String table, String pk, boolean includeTable) + { + String base = "qqq:presence:v2:{" + digest("SHA-256", namespace.key(table)) + "}"; + List keys = new ArrayList<>(); + if(pk != null) + { + String record = base + ":record:" + encode(pk, 1024); + keys.add(record + ":expiry"); + keys.add(record + ":state"); + } + if(pk == null || includeTable) + { + keys.add(base + ":table:expiry"); + keys.add(base + ":table:state"); + } + keys.add(base + ":recovery"); + return (keys); + } + + /*************************************************************************** + ** Distinguish arbitrary identifiers without allowing delimiter injection. + ***************************************************************************/ + private static String encode(String value, int maximumBytes) + { + if(value == null || value.isBlank() || value.getBytes(StandardCharsets.UTF_8).length > maximumBytes) + { + throw (new IllegalArgumentException("Invalid or oversized presence identifier")); + } + return (Base64.getUrlEncoder().withoutPadding().encodeToString(value.getBytes(StandardCharsets.UTF_8))); + } + + /*************************************************************************** + ** Hash public routing identifiers and scripts using JDK algorithms. + ***************************************************************************/ + private static String digest(String algorithm, String value) + { + try + { + return (HexFormat.of().formatHex(MessageDigest.getInstance(algorithm).digest(value.getBytes(StandardCharsets.UTF_8)))); + } + catch(NoSuchAlgorithmException e) + { + throw (new IllegalStateException("Required digest algorithm unavailable", e)); + } + } + + /*************************************************************************** + ** Load immutable scripts from the application artifact. + ***************************************************************************/ + private static String resource(String name) + { + try(var stream = RedisPresenceStore.class.getResourceAsStream("/presence/" + name)) + { + if(stream == null) + { + throw (new IllegalStateException("Missing presence script")); + } + return (new String(stream.readAllBytes(), StandardCharsets.UTF_8)); + } + catch(IOException e) + { + throw (new IllegalStateException("Cannot load presence script", e)); + } + } + + /*************************************************************************** + ** Internal lease data, never returned directly to a browser. + ***************************************************************************/ + public record Stored(String userId, String displayName, String mode, String fieldName, long at, Long sequence, boolean departed, boolean pending, boolean leased, String generation) + { + } + +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/tags/TagValueProvider.java b/src/main/java/com/kingsrook/qbits/collaboration/tags/TagValueProvider.java new file mode 100644 index 0000000..4bccd84 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/tags/TagValueProvider.java @@ -0,0 +1,90 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.tags; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.collaboration.CollaborationRecords; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; +import com.kingsrook.qqq.backend.core.actions.tables.query.VirtualFieldValueProviderInterface; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordReferenceTableMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.core.utils.CollectionUtils; + +/******************************************************************************* + ** Return approved tag labels for parents selected by the enclosing query. + *******************************************************************************/ +public class TagValueProvider implements VirtualFieldValueProviderInterface +{ + /******************************************************************************* + ** Batch both assignments and vocabulary; never recursively load parent records. + *******************************************************************************/ + @Override + public void fill(QTableMetaData table, String fieldName, List records, QueryInput origin) throws QException + { + for(List page : CollectionUtils.getPages(records, 500)) + { + List ids = page.stream().map(record -> String.valueOf(record.getValue(table.getPrimaryKeyField()))).distinct().toList(); + QTableMetaData metadata = QContext.getQInstance().getTable(CollaborationTables.RECORD_TAG).clone(); + metadata.withCustomizers(null); + metadata.getSupplementalMetaData().remove(RecordReferenceTableMetaData.NAME); + QueryInput input = new QueryInput(CollaborationTables.RECORD_TAG).withTransaction(origin.getTransaction()).withFilter(new QQueryFilter() + .withCriteria("tableName", QCriteriaOperator.EQUALS, table.getName()).withCriteria("recordId", QCriteriaOperator.IN, ids) + .withCriteria("parentDeleted", QCriteriaOperator.EQUALS, false)); + input.setTableMetaData(metadata); + input.withPermissionBypass("Tag provider reads only parents selected by the enclosing query."); + List assignments = new QueryAction().execute(input).getRecords(); + Map labels = new HashMap<>(); + List tagIds = assignments.stream().map(row -> row.getValueInteger("tagId")).distinct().toList(); + for(List tags : CollectionUtils.getPages(tagIds, 500)) + { + for(QRecord tag : CollaborationRecords.query(CollaborationTables.TAG, TagsService.vocabularyFilter(table.getName()) + .withCriteria("approved", QCriteriaOperator.EQUALS, true).withCriteria("id", QCriteriaOperator.IN, tags), origin.getTransaction())) + { + labels.put(tag.getValueInteger("id"), tag.getValueString("label")); + } + } + Map> byParent = new HashMap<>(); + for(QRecord assignment : assignments) + { + String label = labels.get(assignment.getValueInteger("tagId")); + if(label != null) + { + byParent.computeIfAbsent(assignment.getValueString("recordId"), key -> new java.util.TreeSet<>()).add(label); + } + } + for(QRecord record : page) + { + var values = byParent.get(String.valueOf(record.getValue(table.getPrimaryKeyField()))); + record.setValue(fieldName, values == null ? null : String.join(", ", values)); + } + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/tags/TagsMetaDataProvider.java b/src/main/java/com/kingsrook/qbits/collaboration/tags/TagsMetaDataProvider.java index f4ac096..51cdfa8 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/tags/TagsMetaDataProvider.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/tags/TagsMetaDataProvider.java @@ -56,7 +56,8 @@ public class TagsMetaDataProvider implements RuntimeMetaDataProviderInterface { contribution.getTables().put(table.getName(), new RuntimeTableMetaDataContribution() .withVirtualFields(List.of(new QVirtualFieldMetaData("collaborationTags", QFieldType.STRING).withLabel("Tags") - .withPossibleValueSourceName(CollaborationTables.TAG).withIsQueryCriteria(true).withIsQuerySelectable(false) + .withPossibleValueSourceName(CollaborationTables.TAG).withIsQueryCriteria(true).withIsQuerySelectable(true) + .withValueProvider(new QCodeReference(TagValueProvider.class)).withShouldTranslatePossibleValues(false) .withCriteriaExpander(new QCodeReference(TagCriteriaExpander.class)))) .withSearchFields(List.of("collaborationTags"))); } diff --git a/src/main/resources/help/qbit-collaboration.yaml b/src/main/resources/help/qbit-collaboration.yaml index 2b48923..6419469 100644 --- a/src/main/resources/help/qbit-collaboration.yaml +++ b/src/main/resources/help/qbit-collaboration.yaml @@ -134,6 +134,12 @@ entries: summary: "The person who created this entry." - key: table:collaboration_attachment;field:parentDeleted summary: "Whether the related record has been deleted." +- key: table:collaboration_attachment;field:retained + summary: "Whether the original file is protected from deletion, including after its related record is deleted." +- key: table:collaboration_attachment;field:intakeKey + summary: "The server-scoped identity used to recover a repeated file intake request." +- key: table:collaboration_attachment;field:intakeFingerprint + summary: "The fingerprint that prevents a repeated intake request from changing its original file or metadata." - key: table:collaboration_comment;slot:help summary: "Read record discussions, including replies, authors and resolution details." content: | @@ -746,3 +752,83 @@ entries: summary: "Choose the record feature you want to configure." - key: possibleValueSource:collaboration_permission_mode;slot:help summary: "Choose whether a feature accepts editing permission or requires an explicit feature permission." +- key: table:collaboration_attachment_cleanup;slot:help + summary: "Review storage cleanup waiting to finish after an attachment was deleted." + content: | + **How rows get created:** Authorized attachment operations create these records automatically. Deleted files are removed after the database commits. Uncertain uploads remain pending until the provider has finished writing. Ordinary successful cleanup removes its entry; administrator-resolved uploads retain a receipt. Administrators can inspect the records but cannot edit them directly. +- key: table:collaboration_attachment_cleanup;field:attachmentId + summary: "The deleted attachment, when known; failed uploads may have no committed attachment ID." +- key: table:collaboration_attachment_cleanup;field:bucketName + summary: "The storage bucket containing the file awaiting cleanup." +- key: table:collaboration_attachment_cleanup;field:storageKey + summary: "The immutable file location retained until cleanup succeeds." +- key: table:collaboration_attachment_cleanup;field:nextAttemptAt + summary: "Failed cleanup waits at least one minute before another scheduled attempt." +- key: table:collaboration_attachment_cleanup;field:resolvedAt + summary: "When an administrator resolved this upload. Resolved receipts are excluded from retries." +- key: table:collaboration_attachment_cleanup;field:resolvedBy + summary: "The signed-in administrator who resolved this upload. The original identity is retained on repeat requests." +- key: table:collaboration_attachment_cleanup;field:settlementEvidence + summary: "Evidence that the provider completed or cancelled the upload. Never include credentials or private file contents." +- key: table:collaboration_attachment_cleanup;section:settlement + summary: "Retained evidence and identity for an administrator-resolved upload." +- key: process:collaborationResolveAttachmentUpload;slot:help + summary: "Resolve uncertain uploads only after confirming that the provider has finished writing." + content: | + Select upload intents, record evidence of provider completion or cancellation for every selected location, and confirm the Review screen. Object absence or elapsed time alone is not proof. If completion cannot be established, leave the intent pending. The action requires an identified collaboration administrator, refuses active or referenced uploads, deletes current bytes, and retains a settlement receipt. Read Result to see which uploads remain unresolved. Existing attachment deletions use normal cleanup retry instead. +- key: process:collaborationRetryAttachmentCleanup;slot:help + summary: "Retry pending attachment file cleanup after storage or database recovery." + content: | + Processes up to 100 committed cleanup requests. Completed entries are removed; failures remain available for retry. Runs every minute when the host configures a collaboration scheduler. Manual execution requires collaboration administrator access. +- key: table:collaboration_upload_journal;slot:help + summary: "Review retained upload progress and recover interrupted registration." + content: | + **How rows get created** + + Retained uploads automatically create an immutable journal entry before original storage. PREPARED and REGISTERED entries may need recovery after an interruption; COMPLETE means mandatory registration finished, not that quarantined content was released. + Open a pending row and choose Recover retained upload while signed in as its original uploader with current access to attach files to the source. The action verifies the existing original and retries registration; it never replaces or deletes bytes. If an intake key is present, retry through the original upload workflow instead. + Missing or mismatched storage remains unresolved. Contact your operator if storage is unavailable or the original uploader is no longer active. Waiting for a reservation to expire is not permission to discard evidence. +- key: table:collaboration_upload_journal;field:tableName + summary: "Original source table. A completed archive keeps this provenance unchanged." +- key: table:collaboration_upload_journal;field:recordId + summary: "Original source record identifier. Access follows the exact retained attachment after an archive transfer." +- key: table:collaboration_upload_journal;field:createdByUserId + summary: "Person who initiated the upload. Recovery currently requires this same signed-in person and current source access." +- key: table:collaboration_upload_journal;field:fileName + summary: "Original uploaded file name captured before storing the bytes." +- key: table:collaboration_upload_journal;field:contentType + summary: "Media type captured for the original upload." +- key: table:collaboration_upload_journal;field:description + summary: "Description supplied with the original upload." +- key: table:collaboration_upload_journal;field:sizeBytes + summary: "Measured number of original bytes. Recovery requires the exact same size." +- key: table:collaboration_upload_journal;field:sha256 + summary: "SHA-256 calculated from the actual original bytes and checked again during recovery." +- key: table:collaboration_upload_journal;field:retained + summary: "This journal contains retained originals only; recovery never deletes their bytes." +- key: table:collaboration_upload_journal;field:intakeKey + summary: "A value identifies a coordinated intake. Retry through the original intake workflow rather than this recovery action." +- key: table:collaboration_upload_journal;field:intakeFingerprint + summary: "Immutable coordinated request fingerprint. Conflicting retries are rejected." +- key: table:collaboration_upload_journal;field:locatorHash + summary: "Hash identifying the stored object without revealing its storage address." +- key: table:collaboration_upload_journal;field:state + summary: "PREPARED awaits an attachment; REGISTERED awaits lifecycle completion; COMPLETE has saved registration." +- key: table:collaboration_upload_journal;field:storageState + summary: "UNKNOWN may have stored bytes; VERIFIED confirms the exact original. Missing data never proves safe cancellation." +- key: table:collaboration_upload_journal;field:attachmentId + summary: "Exact attachment checkpoint. Recovery reuses this identity rather than creating a replacement." +- key: table:collaboration_upload_journal;field:leaseUntil + summary: "Time until the current recovery reservation expires. Expiry does not delete or cancel the original." +- key: table:collaboration_upload_journal;field:fence + summary: "Monotonically increasing reservation number that prevents an expired attempt from registering metadata." +- key: process:collaborationAdoptRetainedAttachment;slot:help + summary: "Verify an existing attachment's original and register it as retained without copying bytes." + content: | + Select exactly one attachment, confirm the Review screen and read the Result. You must currently be allowed to modify the attachment and see its source record. The action verifies the existing original and completes lifecycle registration under the configured retention policy. It never accepts replacement bytes, a storage path or another person's identity, and it never copies or deletes the original. + Uploads that came from a coordinated intake request cannot be adopted here; retry them through their original workflow. If the original is missing or does not match, the attachment remains unchanged. +- key: process:collaborationRecoverRetainedUpload;slot:help + summary: "Verify an existing retained original and finish interrupted registration." + content: | + Select one journal row and confirm recovery. You must be its original uploader and currently be allowed to attach files to its source. Recovery preserves the original bytes and any committed attachment identity, and may leave the file quarantined for review. + Coordinated intake entries must be retried through their original workflow. Missing or mismatched bytes remain unresolved; the action never creates replacement content or clears a possible original by timeout. diff --git a/src/main/resources/openapi/qqq-v1/collaboration.yaml b/src/main/resources/openapi/qqq-v1/collaboration.yaml index 8b2874c..6b828d1 100644 --- a/src/main/resources/openapi/qqq-v1/collaboration.yaml +++ b/src/main/resources/openapi/qqq-v1/collaboration.yaml @@ -202,7 +202,12 @@ components: \ are exposed." properties: canDelete: - description: "Whether the caller may change or delete this attachment." + description: "Whether the caller may delete this attachment; retained files\ + \ cannot be deleted." + nullable: true + type: "boolean" + canEdit: + description: "Whether the caller may change the attachment filename or description." nullable: true type: "boolean" contentType: @@ -404,9 +409,13 @@ components: CollaborationDownloadV1: description: "Temporary access to file bytes. Clients follow url." properties: + delivery: + description: "SIGNED_URL or AUTHENTICATED_FETCH. Authenticated fetch requires\ + \ the trusted native API transport and a durable receipt check." + nullable: true + type: "string" url: - description: "Presigned URL, valid for five minutes. Clients follow url;\ - \ never persist it." + description: "Temporary delivery URL; never persist it." nullable: true type: "string" type: "object" @@ -703,6 +712,23 @@ components: nullable: true type: "string" type: "object" + CollaborationPresenceExpiredV1: + properties: + code: + description: "Open a fresh lease before retrying renewal." + enum: + - "PRESENCE_LEASE_EXPIRED" + nullable: false + type: "string" + type: "object" + CollaborationPresenceLeaseV1: + properties: + connectionId: + description: "Server-issued handle. Renew before 45 seconds; departure is\ + \ terminal." + nullable: false + type: "string" + type: "object" CollaborationPresenceLeaveV1: description: "Remove the caller's lease on this connection and target." properties: @@ -714,6 +740,22 @@ components: description: "Optional parent key." nullable: true type: "string" + sequence: + description: "Monotonically increasing nonnegative safe integer per connection\ + \ and target; orders delayed requests." + nullable: true + type: "integer" + table: + description: "Parent table name." + nullable: false + type: "string" + type: "object" + CollaborationPresenceOpenV1: + properties: + pk: + description: "Optional record key." + nullable: true + type: "string" table: description: "Parent table name." nullable: false @@ -741,6 +783,11 @@ components: description: "Optional parent key; omit for a table view." nullable: true type: "string" + sequence: + description: "Monotonically increasing nonnegative safe integer per connection\ + \ and target; orders delayed requests." + nullable: true + type: "integer" table: description: "Parent table name." nullable: false @@ -749,19 +796,6 @@ components: CollaborationPresenceSnapshotV1: description: "Live viewers, merged across tabs; presence.changed uses this payload." properties: - leaseCount: - description: "Live connections (leases) behind viewers; a user with two\ - \ terminals counts twice. Since 1.2.0." - nullable: true - type: "integer" - leases: - description: "One viewer per live connection, with connectionId and since.\ - \ On GET only with perConnection=true; always on presence.changed. Since\ - \ 1.2.0." - items: - $ref: "#/components/schemas/CollaborationPresenceViewerV1" - nullable: true - type: "array" pk: description: "Record key; absent for table events." nullable: true @@ -771,26 +805,13 @@ components: nullable: false type: "string" viewers: - description: "Live viewers; always present, empty once the last viewer leaves." items: $ref: "#/components/schemas/CollaborationPresenceViewerV1" - nullable: false type: "array" type: "object" CollaborationPresenceViewerV1: - description: "One user merged across live tabs (EDITING takes precedence), or\ - \ one live connection in leases." + description: "One user merged across live tabs; EDITING takes precedence." properties: - connectionId: - description: "Opaque per-connection id, only in leases: c + the first 16\ - \ hex digits of SHA-256(userId + newline + connectionId). Since 1.2.0." - nullable: true - type: "string" - connections: - description: "Live connections (tabs, terminals) merged into this viewer;\ - \ 1 in leases. Since 1.2.0." - nullable: true - type: "integer" displayName: description: "Session display name." nullable: false @@ -806,18 +827,6 @@ components: - "EDITING" nullable: false type: "string" - sessionId: - description: "Opaque id of the signed-in session behind this connection,\ - \ only in leases (HMAC of the server session; never the session itself).\ - \ Since 1.2.0." - nullable: true - type: "string" - since: - description: "When this connection's lease started (ISO-8601), only in leases.\ - \ Since 1.2.0." - format: "date-time" - nullable: true - type: "string" userId: description: "Stable user identity." nullable: false @@ -1288,7 +1297,7 @@ info: description: "Record collaboration: favorites, recents, attachments, tags, conversations,\ \ following, timeline, people and presence" title: "QQQ Middleware API extension: collaboration" - version: "1.2.0" + version: "1.3.0" openapi: "3.0.3" paths: /qqq/v1/qbit/collaboration/presence: @@ -1327,6 +1336,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -1375,13 +1390,6 @@ paths: required: false schema: type: "string" - - description: "Also return one viewer per live connection (lease) in leases;\ - \ default false. Since collaboration 1.2.0." - in: "query" - name: "perConnection" - required: false - schema: - type: "boolean" responses: 400: content: @@ -1407,6 +1415,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -1473,6 +1487,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -1504,6 +1524,151 @@ paths: summary: "Renew presence" tags: - "collaboration" + /qqq/v1/qbit/collaboration/presence/leases: + post: + description: "Open presence lease. Uses the signed-in caller's permissions and\ + \ record visibility." + operationId: "postQbitCollaborationPresenceLeases" + requestBody: + content: + application/json: + schema: + $ref: "#/components/schemas/CollaborationPresenceOpenV1" + required: true + responses: + 400: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Invalid input" + 401: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Sign in required" + 403: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Permission denied" + 404: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" + 200: + content: + application/json: + schema: + $ref: "#/components/schemas/CollaborationPresenceLeaseV1" + description: "Successful Response" + 409: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Conflict with the current state, e.g. an edit conflict or\ + \ an invalid transition (code CONFLICT)." + 429: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Too many requests (code RATE_LIMITED); retry later." + security: + - sessionUuidCookie: + - "N/A" + - bearerAuth: + - "N/A" + - oauth2: + - "openid" + - "profile" + - "email" + summary: "Open presence lease" + tags: + - "collaboration" + /qqq/v1/qbit/collaboration/presence/leases/renew: + post: + description: "Renew issued presence lease. Uses the signed-in caller's permissions\ + \ and record visibility." + operationId: "postQbitCollaborationPresenceLeasesRenew" + requestBody: + content: + application/json: + schema: + $ref: "#/components/schemas/CollaborationPresenceRequestV1" + required: true + responses: + 400: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Invalid input" + 401: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Sign in required" + 403: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Permission denied" + 404: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" + 200: + content: + application/json: + schema: + $ref: "#/components/schemas/CollaborationPresenceSnapshotV1" + description: "Successful Response" + 409: + content: + application/json: + schema: + $ref: "#/components/schemas/CollaborationPresenceExpiredV1" + description: "Presence lease expired; open a new lease." + 429: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Too many requests (code RATE_LIMITED); retry later." + security: + - sessionUuidCookie: + - "N/A" + - bearerAuth: + - "N/A" + - oauth2: + - "openid" + - "profile" + - "email" + summary: "Renew issued presence lease" + tags: + - "collaboration" /qqq/v1/qbit/collaboration/favorites: delete: description: "Unstar a target. Uses the signed-in caller's permissions and record\ @@ -1540,6 +1705,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -1614,6 +1785,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -1680,6 +1857,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -1755,6 +1938,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -1821,6 +2010,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -1888,6 +2083,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -1975,6 +2176,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -2054,6 +2261,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -2122,6 +2335,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -2190,6 +2409,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -2258,6 +2483,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -2331,6 +2562,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -2411,6 +2648,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -2477,6 +2720,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -2564,6 +2813,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -2643,6 +2898,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -2730,6 +2991,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -2809,6 +3076,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -2877,6 +3150,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -2950,6 +3229,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -3018,6 +3303,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -3086,6 +3377,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -3167,6 +3464,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -3260,6 +3563,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -3334,6 +3643,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -3407,6 +3722,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -3486,6 +3807,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -3573,6 +3900,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -3673,6 +4006,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -3747,6 +4086,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -3815,6 +4160,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -3889,6 +4240,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -3969,6 +4326,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -4049,6 +4412,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -4116,6 +4485,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -4191,6 +4566,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -4284,6 +4665,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -4363,6 +4750,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -4437,6 +4830,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -4518,6 +4917,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -4598,6 +5003,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -4659,6 +5070,12 @@ paths: schema: $ref: "#/components/schemas/BasicErrorResponseV1" description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" 200: content: application/json: @@ -4690,6 +5107,73 @@ paths: summary: "Get my work summary" tags: - "collaboration" + /qqq/v1/qbit/collaboration/my-work/mentions/read: + post: + description: "Mark my visible mentions read. Uses the signed-in caller's permissions\ + \ and record visibility." + operationId: "postQbitCollaborationMyWorkMentionsRead" + responses: + 400: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Invalid input" + 401: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Sign in required" + 403: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Permission denied" + 404: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Not found (including invisible parents)" + 503: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Service temporarily unavailable" + 200: + content: + application/json: + schema: + $ref: "#/components/schemas/CollaborationMutationResultV1" + description: "Successful Response" + 409: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Conflict with the current state, e.g. an edit conflict or\ + \ an invalid transition (code CONFLICT)." + 429: + content: + application/json: + schema: + $ref: "#/components/schemas/BasicErrorResponseV1" + description: "Too many requests (code RATE_LIMITED); retry later." + security: + - sessionUuidCookie: + - "N/A" + - bearerAuth: + - "N/A" + - oauth2: + - "openid" + - "profile" + - "email" + summary: "Mark my visible mentions read" + tags: + - "collaboration" servers: - description: "The QQQ server that serves this document (paths include /qqq/v1)." url: "/" diff --git a/src/main/resources/presence/redis-mutate.lua b/src/main/resources/presence/redis-mutate.lua new file mode 100644 index 0000000..797927d --- /dev/null +++ b/src/main/resources/presence/redis-mutate.lua @@ -0,0 +1,80 @@ +-- All room keys share a table hash tag. Redis TIME owns lease expiry. +local clock = redis.call('TIME') +local now = tonumber(clock[1]) * 1000 + math.floor(tonumber(clock[2]) / 1000) +local guard = presenceRecovery(KEYS, tonumber(ARGV[5]), ARGV[9], now) +local rooms = #KEYS - 1 +-- Validate every pair before changing either record or table membership. +for i = 1, rooms, 2 do + if redis.call('ZCARD', KEYS[i]) ~= redis.call('HLEN', KEYS[i + 1]) then + return redis.error_reply('Presence room state is incomplete') + end +end +local member = ARGV[1] +local ttl = tonumber(ARGV[5]) +local capacity = tonumber(ARGV[6]) +local sequence = tonumber(ARGV[3]) +local departed = ARGV[4] == '1' + +for i = 1, rooms, 2 do + local expired = redis.call('ZRANGEBYSCORE', KEYS[i], '-inf', now) + for _, id in ipairs(expired) do + redis.call('ZREM', KEYS[i], id) + redis.call('HDEL', KEYS[i + 1], id) + end +end + +local previous = redis.call('HGET', KEYS[2], member) +-- A record lease and its table projection are written atomically. If one is +-- missing while the other is live, do not erase surviving ordering/tombstones. +if rooms == 4 then + local tablePrevious = redis.call('HGET', KEYS[4], member) + if (previous and not tablePrevious) or (tablePrevious and not previous) then + return redis.error_reply('Presence room state is incomplete') + end +end +local old = previous and cjson.decode(previous) or nil +local operation = ARGV[8] or 'legacy' +-- Renewal authority is issued by open, not inferred from a caller's sequence. +-- Missing/expired/departed leases cannot be recreated by delayed HTTP requests. +if operation == 'renew' and (not old or not old.leased or old.departed or old.generation ~= guard.generation) then + return -1 +end +if operation == 'open' and old then return -1 end +if operation == 'legacy' and not departed and old and old.leased then return -1 end +if old then + local oldSequence = old.sequence + if oldSequence == cjson.null then oldSequence = nil end + if oldSequence then + if not sequence or sequence < oldSequence or (sequence == oldSequence and (not departed or old.departed)) then + return 1 + end + elseif not sequence and old.at == now and old.departed and not departed then + return 1 + end +end + +-- Check every room before writing any membership, avoiding a partial join. +for i = 1, rooms, 2 do + if redis.call('HEXISTS', KEYS[i + 1], member) == 0 and redis.call('ZCARD', KEYS[i]) >= capacity then + return 0 + end +end + +local data = cjson.decode(ARGV[2]) +data.at = now +data.sequence = sequence or cjson.null +data.departed = departed +data.pending = operation == 'open' +data.leased = operation ~= 'legacy' or (old and old.leased) or false +data.generation = guard.generation +for i = 1, rooms, 2 do + -- Record scope comes first; table scope must never carry focused-field data. + if i > 1 or ARGV[7] == '1' then data.fieldName = cjson.null end + redis.call('HSET', KEYS[i + 1], member, cjson.encode(data)) + redis.call('ZADD', KEYS[i], now + ttl, member) + redis.call('PEXPIRE', KEYS[i], ttl) + redis.call('PEXPIRE', KEYS[i + 1], ttl) +end +guard.liveUntil = now + ttl +redis.call('SET', KEYS[#KEYS], cjson.encode(guard)) +return 1 diff --git a/src/main/resources/presence/redis-recovery.lua b/src/main/resources/presence/redis-recovery.lua new file mode 100644 index 0000000..9eaeca9 --- /dev/null +++ b/src/main/resources/presence/redis-recovery.lua @@ -0,0 +1,43 @@ +-- This prelude is executed atomically with the room operation, in the same slot. +-- The guard deliberately outlives idle leases: missing means unknown, not empty. +local function presenceRecovery(keys, ttl, candidate, now) + local encoded = redis.call('GET', keys[#keys]) + local guard + local created = false + if encoded then + guard = cjson.decode(encoded) + if type(guard.generation) ~= 'string' or type(guard.readyAt) ~= 'number' or type(guard.liveUntil) ~= 'number' or type(guard.ttl) ~= 'number' then + error('Invalid presence recovery guard') + end + if guard.ttl ~= ttl then error('Presence lease lifetime differs across nodes') end + -- All volatile projections disappearing before the last issued expiry is + -- data loss too, even when a persistent guard survives an eviction policy. + if guard.liveUntil > now and redis.call('ZCARD', keys[#keys - 2]) == 0 + and redis.call('HLEN', keys[#keys - 1]) == 0 then + guard = nil + end + end + if not guard then + created = true + guard = { generation = candidate, readyAt = now + ttl, liveUntil = 0, ttl = ttl } + redis.call('SET', keys[#keys], cjson.encode(guard)) + end + -- During recovery, discard only projections from an invalidated generation. + -- The table is cleaned atomically when its generation is created. Subsequent + -- requests only need to clean the touched record; all new table writes carry + -- the current generation. Avoid rescanning the whole table on every renewal. + if now < guard.readyAt then + for i = 1, #keys - 1, 2 do + if created or i < #keys - 2 then + for _, member in ipairs(redis.call('HKEYS', keys[i + 1])) do + local data = cjson.decode(redis.call('HGET', keys[i + 1], member)) + if data.generation ~= guard.generation then + redis.call('HDEL', keys[i + 1], member) + redis.call('ZREM', keys[i], member) + end + end + end + end + end + return guard +end diff --git a/src/main/resources/presence/redis-snapshot.lua b/src/main/resources/presence/redis-snapshot.lua new file mode 100644 index 0000000..686a8cf --- /dev/null +++ b/src/main/resources/presence/redis-snapshot.lua @@ -0,0 +1,45 @@ +-- A partially evicted room is unavailable, not empty. Do not silently discard it. +local clock = redis.call('TIME') +local now = tonumber(clock[1]) * 1000 + math.floor(tonumber(clock[2]) / 1000) +local guard = presenceRecovery(KEYS, tonumber(ARGV[2]), ARGV[3], now) +local rooms = #KEYS - 1 +for i = 1, rooms, 2 do + if redis.call('ZCARD', KEYS[i]) ~= redis.call('HLEN', KEYS[i + 1]) then + return redis.error_reply('Presence room state is incomplete') + end +end +for i = 1, rooms, 2 do + local expired = redis.call('ZRANGEBYSCORE', KEYS[i], '-inf', now) + for _, id in ipairs(expired) do + redis.call('ZREM', KEYS[i], id) + redis.call('HDEL', KEYS[i + 1], id) + end +end +local members = redis.call('ZRANGE', KEYS[1], 0, -1) +-- Table membership is bounded by the configured capacity. Compare the record's +-- projection without key discovery or cross-slot reads. +if rooms == 4 then + local projected = 0 + for _, id in ipairs(redis.call('ZRANGE', KEYS[3], 0, -1)) do + if string.sub(id, -string.len(ARGV[1])) == ARGV[1] then + projected = projected + 1 + if not redis.call('ZSCORE', KEYS[1], id) then + return redis.error_reply('Presence room state is incomplete') + end + end + end + if projected ~= #members then + return redis.error_reply('Presence room state is incomplete') + end +end +local result = {} +if now < guard.readyAt then return -2 end +for _, id in ipairs(members) do + local data = redis.call('HGET', KEYS[2], id) + if not data then return redis.error_reply('Presence room state is incomplete') end + if cjson.decode(data).generation ~= guard.generation then + return redis.error_reply('Presence generation is inconsistent') + end + table.insert(result, data) +end +return result diff --git a/src/test/java/com/kingsrook/qbits/collaboration/ActivityWriterFeatureIsolationTest.java b/src/test/java/com/kingsrook/qbits/collaboration/ActivityWriterFeatureIsolationTest.java new file mode 100644 index 0000000..47f329c --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/ActivityWriterFeatureIsolationTest.java @@ -0,0 +1,97 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration; + +import com.kingsrook.qbits.collaboration.activity.ActivityWriter; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.actions.collaboration.RecordFeaturesResolver; +import com.kingsrook.qqq.backend.core.actions.collaboration.RuntimeMetaDataOverlay; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordFeature; +import com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordReferenceTableMetaData; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Service activity follows its originating feature without enabling timeline UI. + *******************************************************************************/ +class ActivityWriterFeatureIsolationTest extends PersonalServiceFixture +{ + /*************************************************************************** + ** An authorized attachment mutation must retain its event with timeline off. + ***************************************************************************/ + @Test + void attachmentActivityDoesNotRequireTimeline() throws Exception + { + features(RecordFeature.ATTACHMENTS); + assertThat(RecordFeaturesResolver.resolve(instance, instance.getTable("document")).isEnabled(RecordFeature.TIMELINE)).isFalse(); + ActivityWriter.write("document", "1", "FILE_ADDED", "File added", CollaborationTables.ATTACHMENT, "41"); + assertThat(RecordReferenceTableMetaData.of(instance.getTable(CollaborationTables.ACTIVITY)).getFeature()).isEqualTo(RecordFeature.TIMELINE); + assertThat(RecordFeaturesResolver.resolve(instance, instance.getTable("document")).isEnabled(RecordFeature.TIMELINE)).isFalse(); + + features(RecordFeature.ATTACHMENTS, RecordFeature.TIMELINE); + assertThat(CollaborationRecords.query(CollaborationTables.ACTIVITY, + CollaborationRecords.parent("document", "1"))).singleElement().satisfies(row -> + { + assertThat(row.getValueString("type")).isEqualTo("FILE_ADDED"); + assertThat(row.getValueString("actorUserId")).isEqualTo("alice"); + }); + } + + + /*************************************************************************** + ** Enabling timeline cannot authorize a disabled attachment operation. + ***************************************************************************/ + @Test + void originatingFeatureRemainsRequired() throws Exception + { + features(RecordFeature.TIMELINE); + assertThatThrownBy(() -> ActivityWriter.write("document", "1", "FILE_ADDED", "File added", CollaborationTables.ATTACHMENT, "41")) + .isInstanceOf(QException.class); + assertThat(CollaborationRecords.query(CollaborationTables.ACTIVITY, new QQueryFilter())).isEmpty(); + } + + + /*************************************************************************** + ** Service persistence must preserve native parent tenant authorization. + ***************************************************************************/ + @Test + void inaccessibleParentCannotAcquireActivity() throws Exception + { + features(RecordFeature.ATTACHMENTS); + assertThatThrownBy(() -> ActivityWriter.write("document", "3", "FILE_ADDED", "File added", CollaborationTables.ATTACHMENT, "41")) + .isInstanceOf(QException.class); + } + + + /*************************************************************************** + ** Use the supported refresh boundary so effective metadata changes atomically. + ***************************************************************************/ + private void features(RecordFeature... enabled) throws Exception + { + config.withTableFeatures("document", enabled).applyFeatures(instance); + RuntimeMetaDataOverlay.refresh(instance); + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/CollaborationApiContractTest.java b/src/test/java/com/kingsrook/qbits/collaboration/CollaborationApiContractTest.java index 30191f3..ad03688 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/CollaborationApiContractTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/CollaborationApiContractTest.java @@ -60,7 +60,7 @@ class CollaborationApiContractTest assertThat(model.getComponents().getSchemas().keySet()).allMatch(name -> name.startsWith("Collaboration") || name.equals(BasicErrorResponseV1.class.getSimpleName())); assertThat(model.getPaths().keySet()).allMatch(pathName -> pathName.startsWith("/qqq/v1/qbit/collaboration/")); assertThat(generated).contains("multipart/form-data", "format: \"binary\"", "clients follow url", "nextCursor"); - assertThat(extension.getSpecs()).hasSize(45); + assertThat(extension.getSpecs()).hasSize(48); } /******************************************************************************* diff --git a/src/test/java/com/kingsrook/qbits/collaboration/CollaborationApiHttpTest.java b/src/test/java/com/kingsrook/qbits/collaboration/CollaborationApiHttpTest.java index e789b60..41b839a 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/CollaborationApiHttpTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/CollaborationApiHttpTest.java @@ -39,6 +39,12 @@ import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.node.ObjectNode; import com.kingsrook.qbits.collaboration.attachments.AttachmentStorage; +import com.kingsrook.qbits.collaboration.customfields.FieldDefinitions; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData; import com.kingsrook.qbits.objectstorage.api.PresignGet; import com.kingsrook.qbits.objectstorage.api.PutRequest; import com.kingsrook.qqq.backend.core.actions.collaboration.RuntimeMetaDataOverlay; @@ -63,6 +69,86 @@ import static org.assertj.core.api.Assertions.assertThat; *******************************************************************************/ class CollaborationApiHttpTest extends PersonalServiceFixture { + /*************************************************************************** + ** Data loss is unavailable over HTTP until a fresh lease has rejoined. + ***************************************************************************/ + @Test + void redisLossRequiresHttpRejoin() throws Exception + { + try(var redis = RedisFixtures.redis()) + { + redis.start(); + new com.kingsrook.qbits.redis.RedisQBitProducer().withQBitConfig(new com.kingsrook.qbits.redis.RedisQBitConfig() + .withConnection("presence", new com.kingsrook.qbits.redis.RedisConnectionConfig() + .withHost(RedisFixtures.host(redis)).withPort(RedisFixtures.port(redis)).withTls(false))).produce(instance); + config.withRedisPresence(new com.kingsrook.qbits.collaboration.presence.RedisPresenceConfig() + .withConnectionName("presence").withDeployment("test").withApplication("http-recovery") + .withScopeProvider(new QCodeReference(RedisPresenceServiceTest.TenantScope.class))); + var runtime = new com.kingsrook.qbits.redis.RedisRuntimeService(); + try + { + runtime.start(instance); + String original = ok("POST", "/presence/leases", "{\"table\":\"document\",\"pk\":\"1\"}").get("connectionId").asText(); + awaitHttpPresence(original); + assertThat(redis.execInContainer("redis-cli", "FLUSHDB").getExitCode()).isZero(); + assertError(503, "GET", "/presence?table=document&pk=1", null, "alice"); + assertError(404, "GET", "/presence?table=document&pk=3", null, "alice"); + String oldRenewal = JSON.writeValueAsString(Map.of("connectionId", original, "table", "document", "pk", "1", "mode", "VIEWING", "sequence", 1000)); + assertThat(json(request("POST", "/presence/leases/renew", oldRenewal, "alice"), 409).path("code").asText()) + .isEqualTo("PRESENCE_LEASE_EXPIRED"); + assertError(404, "POST", "/presence/leases/renew", oldRenewal, "outsider"); + String replacement = ok("POST", "/presence/leases", "{\"table\":\"document\",\"pk\":\"1\"}").get("connectionId").asText(); + assertThat(replacement).isNotEqualTo(original); + awaitHttpPresence(replacement); + assertThat(ok("GET", "/presence?table=document&pk=1", null).path("viewers")).hasSize(1); + } + finally + { + runtime.stop(); + runtime.afterApplicationStop(); + } + } + } + + /*************************************************************************** + ** Keep a real HTTP lease alive while the production recovery window elapses. + ***************************************************************************/ + private void awaitHttpPresence(String lease) throws Exception + { + long deadline = System.nanoTime() + Duration.ofSeconds(55).toNanos(); + long sequence = 0; + while(true) + { + String body = JSON.writeValueAsString(Map.of("connectionId", lease, "table", "document", "pk", "1", "mode", "VIEWING", "sequence", ++sequence)); + var response = request("POST", "/presence/leases/renew", body, "alice"); + if(response.statusCode() == 200) + { + assertThat(json(response, 200).path("viewers")).hasSize(1); + return; + } + assertThat(json(response, 503).path("message").asText()).contains("recovering"); + assertThat(System.nanoTime()).as("HTTP presence recovery deadline").isLessThan(deadline); + Thread.sleep(1000); + } + } + + /*************************************************************************** + ** Shared presence has an explicit unavailable response, never a local fallback. + ***************************************************************************/ + @Test + void unavailablePresenceHttp() throws Exception + { + config.withRedisPresence(new com.kingsrook.qbits.collaboration.presence.RedisPresenceConfig() + .withConnectionName("not-started").withDeployment("test").withApplication("fixture") + .withScopeProvider(new QCodeReference(RedisPresenceServiceTest.TenantScope.class))); + assertError(503, "GET", "/presence?table=document&pk=1", null, "alice"); + assertError(503, "POST", "/presence/leases", "{\"table\":\"document\",\"pk\":\"1\"}", "alice"); + assertError(503, "POST", "/presence/leases/renew", "{\"connectionId\":\"lease:missing\",\"table\":\"document\",\"pk\":\"1\",\"mode\":\"VIEWING\",\"sequence\":1}", "alice"); + assertError(400, "POST", "/presence", "{\"connectionId\":\"tab\",\"table\":\"document\",\"pk\":\"1\",\"mode\":\"VIEWING\"}", "alice"); + assertError(503, "DELETE", "/presence", "{\"connectionId\":\"tab\",\"table\":\"document\",\"pk\":\"1\"}", "alice"); + assertError(404, "GET", "/presence?table=document&pk=3", null, "alice"); + } + private static final ObjectMapper JSON = new ObjectMapper(); private final HttpClient client = HttpClient.newBuilder().followRedirects(HttpClient.Redirect.NEVER).build(); private QApplicationJavalinServer server; @@ -115,6 +201,99 @@ class CollaborationApiHttpTest extends PersonalServiceFixture MemoryStorage.objects.clear(); } + /******************************************************************************* + ** Native record reads preserve authenticated transports and publish generic UI metadata. + *******************************************************************************/ + @Test + void standardAttachmentRecordPreservesDeliveryModes() throws Exception + { + int id = json(upload("alice", "retained.txt", "original", "/records/document/1/attachments"), 200).get("id").asInt(); + NativeDelivery.accessCalls = 0; + NativeDelivery.authenticated = true; + config.withAttachmentLifecycle(new QCodeReference(NativeDelivery.class)); + MemoryStorage.lastGet = null; + assertThat(nativeGet("/table/collaboration_attachment/" + id, "alice").statusCode()).isEqualTo(403); + assertThat(NativeDelivery.accessCalls).isZero(); + JsonNode record = json(nativeGet("/table/collaboration_attachment/" + id, "nativeview"), 200).get("record"); + assertThat(record.at("/values/previewDelivery").asText()).isEqualTo("AUTHENTICATED_FETCH"); + assertThat(record.at("/values/downloadDelivery").asText()).isEqualTo("AUTHENTICATED_FETCH"); + assertThat(record.at("/values/previewUrl").asText()).contains("/qbit/record-files/files/1/downloads/"); + assertThat(record.at("/values/downloadUrl").asText()).contains("/qbit/record-files/files/1/downloads/"); + assertThat(MemoryStorage.lastGet).isNull(); + assertThat(NativeDelivery.accessCalls).isEqualTo(2); + JsonNode metadata = json(nativeGet("/metaData/table/collaboration_attachment", "nativeview"), 200); + for(String field : List.of("preview", "download")) + { + boolean declared = false; + for(JsonNode adornment : metadata.at("/virtualFields/" + field + "Url/adornments")) + { + if("FILE_DOWNLOAD".equals(adornment.path("type").asText())) + { + assertThat(adornment.at("/values/deliveryModeField").asText()).isEqualTo(field + "Delivery"); + declared = true; + } + } + assertThat(declared).as("Native %s field declares transport mode", field).isTrue(); + } + assertThat(nativeGet("/table/collaboration_attachment/" + id, "nativeoutsider").statusCode()).isEqualTo(404); + assertThat(NativeDelivery.accessCalls).isEqualTo(2); + NativeDelivery.authenticated = false; + JsonNode signed = json(nativeGet("/table/collaboration_attachment/" + id, "nativeview"), 200).get("record"); + assertThat(signed.at("/values/previewDelivery").asText()).isEqualTo("SIGNED_URL"); + assertThat(signed.at("/values/downloadDelivery").asText()).isEqualTo("SIGNED_URL"); + assertThat(signed.at("/values/downloadUrl").asText()).startsWith("https://files.example.invalid/"); + } + + /******************************************************************************* + ** Use the supported native record and metadata routes, with the original actor cookie. + *******************************************************************************/ + private HttpResponse nativeGet(String path, String caller) throws Exception + { + String serverUrl = baseUrl.substring(0, baseUrl.indexOf("/qbit/")); + return (client.send(HttpRequest.newBuilder(URI.create(serverUrl + path)).timeout(Duration.ofSeconds(20)) + .header("Cookie", "sessionUUID=" + caller + sessionSuffix).GET().build(), HttpResponse.BodyHandlers.ofString())); + } + + /******************************************************************************* + ** A trusted transport fixture proves owner metadata without a Record Files dependency. + *******************************************************************************/ + public static class NativeDelivery extends com.kingsrook.qbits.collaboration.attachments.AttachmentLifecyclePolicy + { + private static int accessCalls; + private static boolean authenticated; + + /******************************************************************************* + ** Access must be authorized before the transport is returned. + *******************************************************************************/ + @Override + public void beforeAccess(com.kingsrook.qqq.backend.core.model.data.QRecord attachment, boolean inline) + { + accessCalls++; + } + + /******************************************************************************* + ** The authenticated descriptor remains data, never a presigned storage capability. + *******************************************************************************/ + @Override + public com.kingsrook.qbits.collaboration.attachments.AttachmentDownload downloadDelivery(com.kingsrook.qqq.backend.core.model.data.QRecord attachment) + throws com.kingsrook.qqq.backend.core.exceptions.QException + { + if(!authenticated) + { + return (null); + } + try + { + return (new com.kingsrook.qbits.collaboration.attachments.AttachmentDownload(URI.create( + "https://app.example/qqq/v1/qbit/record-files/files/1/downloads/00000000-0000-4000-8000-000000000001").toURL(), true)); + } + catch(Exception failure) + { + throw (new com.kingsrook.qqq.backend.core.exceptions.QException("Invalid fixture URL", failure)); + } + } + } + /******************************************************************************* ** Sharing validates recipients and access requests never reveal parent existence. *******************************************************************************/ @@ -223,21 +402,6 @@ class CollaborationApiHttpTest extends PersonalServiceFixture assertThat(ok("POST", "/presence", body).get("viewers")).hasSize(1); assertThat(ok("GET", "/presence?table=document&pk=1", null).at("/viewers/0/mode").asText()).isEqualTo("EDITING"); assertThat(ok("GET", "/presence?table=document", null).at("/viewers/0/fieldName").isNull()).isTrue(); - ok("POST", "/presence", body.replace("tab-one", "tab-two").replace("EDITING", "VIEWING")); - JsonNode merged = ok("GET", "/presence?table=document&pk=1", null); - assertThat(merged.get("viewers")).hasSize(1); - assertThat(merged.at("/viewers/0/connections").asInt()).isEqualTo(2); - assertThat(merged.get("leaseCount").asInt()).isEqualTo(2); - assertThat(merged.has("leases")).isFalse(); - JsonNode perConnection = ok("GET", "/presence?table=document&pk=1&perConnection=true", null); - assertThat(perConnection.get("viewers")).hasSize(1); - assertThat(perConnection.get("leases")).hasSize(2); - assertThat(perConnection.at("/leases/0/connectionId").asText()).startsWith("c").doesNotContain("tab-"); - assertThat(perConnection.at("/leases/0/since").asText()).isNotBlank(); - assertThat(perConnection.at("/leases/0/sessionId").asText()).startsWith("s"); - assertThat(perConnection.toString()).doesNotContain("alice" + sessionSuffix).doesNotContain(sessionSuffix); - assertError(400, "GET", "/presence?table=document&perConnection=maybe", null, "alice"); - ok("DELETE", "/presence", "{\"connectionId\":\"tab-two\",\"table\":\"document\",\"pk\":\"1\"}"); assertError(404, "GET", "/presence?table=document&pk=3", null, "alice"); assertError(404, "POST", "/presence", body, "outsider"); assertError(400, "POST", "/presence", "{\"connectionId\":\"x\",\"table\":\"document\",\"mode\":\"BAD\"}", "alice"); @@ -246,6 +410,27 @@ class CollaborationApiHttpTest extends PersonalServiceFixture assertThat(ok("GET", "/presence?table=document&pk=1", null).get("viewers")).isEmpty(); } + /******************************************************************************* + ** HTTP-issued leases are invisible until activated and expire explicitly. + *******************************************************************************/ + @Test + void issuedPresenceHttp() throws Exception + { + String lease = ok("POST", "/presence/leases", "{\"table\":\"document\",\"pk\":\"01\"}").get("connectionId").asText(); + assertThat(lease).startsWith("lease:"); + assertThat(ok("GET", "/presence?table=document&pk=1", null).get("viewers")).isEmpty(); + String body = new ObjectMapper().writeValueAsString(Map.of("connectionId", lease, "table", "document", "pk", "1", "mode", "EDITING", "fieldName", "title", "sequence", 1)); + assertThat(ok("POST", "/presence/leases/renew", body).get("viewers")).hasSize(1); + assertError(404, "POST", "/presence/leases", "{\"table\":\"document\",\"pk\":\"3\"}", "alice"); + assertError(404, "POST", "/presence/leases/renew", body, "outsider"); + assertError(400, "POST", "/presence/leases/renew", body.replace("\"sequence\":1", "\"sequence\":0"), "alice"); + String leave = new ObjectMapper().writeValueAsString(Map.of("connectionId", lease, "table", "document", "pk", "1", "sequence", 2)); + ok("DELETE", "/presence", leave); + var expired = json(request("POST", "/presence/leases/renew", body, "alice"), 409); + assertThat(expired.get("code").asText()).isEqualTo("PRESENCE_LEASE_EXPIRED"); + assertThat(ok("GET", "/presence?table=document&pk=1", null).get("viewers")).isEmpty(); + } + /******************************************************************************* ** Every personal operation and offset cursor runs through the HTTP boundary. *******************************************************************************/ @@ -316,6 +501,36 @@ class CollaborationApiHttpTest extends PersonalServiceFixture assertError(404, "GET", "/attachments/" + id + "/preview", null, "alice"); } + /******************************************************************************* + ** Existing offset cursors retain wire scope while native lists fetch pages. + *******************************************************************************/ + @Test + void attachmentPagesPreserveCursorScopeAndCurrentParentAccess() throws Exception + { + List ids = new java.util.ArrayList<>(); + for(int i = 0; i < 5; i++) + { + ids.add(json(upload("alice", "file-" + i + ".txt", "x", "/records/document/1/attachments"), 200).get("id").asInt()); + } + JsonNode first = ok("GET", "/records/document/1/attachments?limit=2", null); + assertThat(first.get("items")).hasSize(2); + assertThat(first.at("/items/0/id").asInt()).isEqualTo(ids.get(4)); + String cursor = first.get("nextCursor").asText(); + String decoded = new String(java.util.Base64.getUrlDecoder().decode(cursor), StandardCharsets.UTF_8); + assertThat(decoded).startsWith("1:").endsWith(":2"); + JsonNode second = ok("GET", "/records/document/1/attachments?limit=2&cursor=" + cursor, null); + assertThat(second.get("items")).hasSize(2); + assertThat(second.at("/items/0/id").asInt()).isEqualTo(ids.get(2)); + JsonNode last = ok("GET", "/records/document/1/attachments?limit=2&cursor=" + second.get("nextCursor").asText(), null); + assertThat(last.get("items")).hasSize(1); + assertThat(last.get("nextCursor").isNull()).isTrue(); + assertError(400, "GET", "/records/document/2/attachments?cursor=" + cursor, null, "alice"); + assertError(400, "GET", "/records/document/1/attachments?limit=101", null, "alice"); + assertError(400, "GET", "/records/document/1/attachments?cursor=invalid", null, "alice"); + assertError(404, "GET", "/records/document/1/attachments?cursor=" + cursor, null, "outsider"); + assertThat(ok("GET", "/records/document/2/attachments", null).get("items")).isEmpty(); + } + /******************************************************************************* ** Vocabulary creation and record associations preserve caller permissions. *******************************************************************************/ @@ -361,6 +576,80 @@ class CollaborationApiHttpTest extends PersonalServiceFixture assertThat(tagQuery("IN", "[\"alpha\"]")).containsExactlyInAnyOrder(1, 2); } + /******************************************************************************* + ** EAV choices retain their stored ids and translated labels on the HTTP wire. + *******************************************************************************/ + @Test + void customFieldValuesThroughTableReads() throws Exception + { + admin(); + instance.addBackend(new RDBMSBackendMetaData().withName("fallback").withVendor("h2")); + instance.getTable("document").setBackendName("fallback"); + QRecord definition = new InsertAction().execute(new InsertInput(CollaborationTables.FIELD_DEFINITION).withRecords(List.of( + new QRecord().withValue("tableName", "document").withValue("name", "choice").withValue("label", "Choice").withValue("type", "SELECT") + .withValue("options", "[{\"id\":\"x\",\"label\":\"Choice X\"}]")))).getRecords().getFirst(); + assertThat(definition.getErrors()).isNullOrEmpty(); + FieldDefinitions.apply(definition.getValue("id"), "Publish"); + instance.getTable("document").setBackendName("memory"); + CollaborationRecords.insert(CollaborationTables.FIELD_VALUE, new QRecord().withValue("tableName", "document").withValue("recordId", "1") + .withValue("fieldName", "choice").withValue("valueText", "x").withValue("parentDeleted", false)); + login("alice"); + String serverUrl = baseUrl.substring(0, baseUrl.indexOf("/qbit/")); + HttpResponse response = client.send(HttpRequest.newBuilder(URI.create(serverUrl + "/table/document/query")).timeout(Duration.ofSeconds(20)) + .header("Cookie", "sessionUUID=alice" + sessionSuffix).header("Content-Type", "application/json") + .POST(HttpRequest.BodyPublishers.ofString("{\"filter\":{\"orderBys\":[{\"fieldName\":\"id\"}]}}")).build(), HttpResponse.BodyHandlers.ofString()); + JsonNode records = json(response, 200).get("records"); + assertThat(records).hasSize(2); + assertThat(records.get(0).at("/values/id").asInt()).isEqualTo(1); + assertThat(records.get(0).at("/values/cf_choice").asText()).isEqualTo("x"); + assertThat(records.get(0).at("/displayValues/cf_choice").asText()).isEqualTo("Choice X"); + assertThat(records.get(1).at("/values/cf_choice").isMissingNode()).isTrue(); + + response = client.send(HttpRequest.newBuilder(URI.create(serverUrl + "/table/document/1")).timeout(Duration.ofSeconds(20)) + .header("Cookie", "sessionUUID=alice" + sessionSuffix).GET().build(), HttpResponse.BodyHandlers.ofString()); + JsonNode record = json(response, 200).get("record"); + assertThat(record.at("/values/cf_choice").asText()).isEqualTo("x"); + assertThat(record.at("/displayValues/cf_choice").asText()).isEqualTo("Choice X"); + + response = client.send(HttpRequest.newBuilder(URI.create(serverUrl + "/table/document/1")).timeout(Duration.ofSeconds(20)) + .header("Cookie", "sessionUUID=outsider" + sessionSuffix).GET().build(), HttpResponse.BodyHandlers.ofString()); + assertThat(response.statusCode()).isEqualTo(404); + assertThat(response.body()).doesNotContain("Choice X", "First document"); + } + + /******************************************************************************* + ** Table reads expose tag labels while retaining parent-record authorization. + *******************************************************************************/ + @Test + void tagValuesThroughTableReads() throws Exception + { + RuntimeMetaDataOverlay.refresh(instance); + int tag = ok("POST", "/tags", "{\"table\":\"document\",\"label\":\"Review, urgent\"}").get("id").asInt(); + ok("PUT", "/records/document/1/tags", "{\"add\":[" + tag + "]}"); + String serverUrl = baseUrl.substring(0, baseUrl.indexOf("/qbit/")); + HttpResponse response = client.send(HttpRequest.newBuilder(URI.create(serverUrl + "/table/document/query")).timeout(Duration.ofSeconds(20)) + .header("Cookie", "sessionUUID=alice" + sessionSuffix).header("Content-Type", "application/json") + .POST(HttpRequest.BodyPublishers.ofString("{\"filter\":{\"orderBys\":[{\"fieldName\":\"id\"}]}}")).build(), HttpResponse.BodyHandlers.ofString()); + JsonNode records = json(response, 200).get("records"); + assertThat(records).hasSize(2); + assertThat(records.get(0).at("/values/id").asInt()).isEqualTo(1); + assertThat(records.get(0).at("/values/collaborationTags").asText()).isEqualTo("Review, urgent"); + assertThat(records.get(0).at("/displayValues/collaborationTags").asText()).isEqualTo("Review, urgent"); + assertThat(records.get(1).at("/values/id").asInt()).isEqualTo(2); + assertThat(records.get(1).at("/values/collaborationTags").isMissingNode()).as(records.toString()).isTrue(); + + response = client.send(HttpRequest.newBuilder(URI.create(serverUrl + "/table/document/1")).timeout(Duration.ofSeconds(20)) + .header("Cookie", "sessionUUID=alice" + sessionSuffix).GET().build(), HttpResponse.BodyHandlers.ofString()); + JsonNode record = json(response, 200).get("record"); + assertThat(record.at("/values/collaborationTags").asText()).isEqualTo("Review, urgent"); + assertThat(record.at("/displayValues/collaborationTags").asText()).isEqualTo("Review, urgent"); + + response = client.send(HttpRequest.newBuilder(URI.create(serverUrl + "/table/document/1")).timeout(Duration.ofSeconds(20)) + .header("Cookie", "sessionUUID=outsider" + sessionSuffix).GET().build(), HttpResponse.BodyHandlers.ofString()); + assertThat(response.body()).doesNotContain("Review, urgent", "First document"); + assertThat(response.statusCode()).isEqualTo(404); + } + /******************************************************************************* ** Run the v1 table query filtered on the virtual tags field; return matching ids. *******************************************************************************/ @@ -417,6 +706,21 @@ class CollaborationApiHttpTest extends PersonalServiceFixture JsonNode candidate = ok("GET", "/mentions/candidates?table=document&pk=1&q=carol", null).at("/items/0"); assertThat(candidate.at("/person/userId").asText()).isEqualTo("carol"); assertThat(candidate.get("canRead").asText()).isEqualTo("true"); + json(request("POST", "/records/document/1/comments", "{\"body\":\"Review @[Alice](user:alice)\"}", "carol"), 200); + ok("POST", "/records/document/1/comments", "{\"body\":\"Review @[Carol](user:carol)\"}"); + // Delivery is external to this HTTP fixture; persist its receipt before testing read state. + for(var mention : CollaborationRecords.query(com.kingsrook.qbits.collaboration.model.CollaborationTables.MENTION, + new com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter())) + { + CollaborationRecords.update(com.kingsrook.qbits.collaboration.model.CollaborationTables.MENTION, + new com.kingsrook.qqq.backend.core.model.data.QRecord().withValue("id", mention.getValue("id")).withValue("notifiedAt", java.time.Instant.now())); + } + assertThat(ok("GET", "/my-work", null).get("unreadMentionCount").asLong()).isPositive(); + assertThat(json(request("GET", "/my-work", null, "carol"), 200).get("unreadMentionCount").asLong()).isEqualTo(1); + ok("POST", "/my-work/mentions/read", null); + assertThat(ok("GET", "/my-work", null).get("unreadMentionCount").asLong()).isZero(); + assertThat(json(request("GET", "/my-work", null, "carol"), 200).get("unreadMentionCount").asLong()).isEqualTo(1); + ok("POST", "/my-work/mentions/read", null); assertThat(ok("GET", "/records/document/1/follow", null).at("/follow/reason").asText()).isEqualTo("COMMENTED"); JsonNode state = ok("PUT", "/records/document/1/follow", "{\"notifyOnChange\":false,\"notifyOnComment\":true}"); assertThat(state.at("/follow/notifyOnChange").asBoolean()).isFalse(); @@ -524,7 +828,7 @@ class CollaborationApiHttpTest extends PersonalServiceFixture JsonNode version = json(client.send(HttpRequest.newBuilder(URI.create(serverUrl + "/qqq/v1/version")).GET().build(), HttpResponse.BodyHandlers.ofString()), 200); assertThat(version.at("/extensions/0/name").asText()).isEqualTo("collaboration"); - assertThat(version.at("/extensions/0/version").asText()).isEqualTo("1.2.0"); + assertThat(version.at("/extensions/0/version").asText()).isEqualTo("1.3.0"); } /******************************************************************************* @@ -568,7 +872,11 @@ class CollaborationApiHttpTest extends PersonalServiceFixture case "carol" -> "Carol Diaz"; default -> caller; })); - session.withPermissions("document.read").withSecurityKeyValue("tenant", caller.equals("outsider") ? "b" : "a"); + session.withPermissions("document.read").withSecurityKeyValue("tenant", (caller.equals("outsider") || caller.equals("nativeoutsider")) ? "b" : "a"); + if(caller.startsWith("native")) + { + session.withPermissions("collaboration.view"); + } if(caller.equals("alice") || caller.equals("carol")) { session.withPermissions("document.write"); diff --git a/src/test/java/com/kingsrook/qbits/collaboration/CollaborationTablesGuardrailTest.java b/src/test/java/com/kingsrook/qbits/collaboration/CollaborationTablesGuardrailTest.java index 8b3f910..e1a8572 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/CollaborationTablesGuardrailTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/CollaborationTablesGuardrailTest.java @@ -65,6 +65,8 @@ class CollaborationTablesGuardrailTest extends PersonalServiceFixture Map.entry("collaboration_favorite", "user content: a person stars something"), Map.entry("collaboration_recent", "runtime: a person opens a record"), Map.entry("collaboration_attachment", "user content: a person uploads a file"), + Map.entry("collaboration_attachment_cleanup", "runtime: authorized deletion queues durable storage cleanup"), + Map.entry("collaboration_upload_journal", "runtime: retained upload intent is committed before original storage and retained for recovery"), Map.entry("collaboration_comment", "user content: a person comments"), Map.entry("collaboration_comment_revision", "runtime: a comment is edited"), Map.entry("collaboration_mention", "runtime: a comment mentions someone"), @@ -183,7 +185,7 @@ class CollaborationTablesGuardrailTest extends PersonalServiceFixture private List collaborationTables() { List tables = instance.getTables().values().stream().map(QTableMetaData::getName).filter(n -> n.startsWith("collaboration_")).sorted().toList(); - assertThat(tables).hasSize(19); + assertThat(tables).hasSize(21); return (tables); } diff --git a/src/test/java/com/kingsrook/qbits/collaboration/CollaborationTest.java b/src/test/java/com/kingsrook/qbits/collaboration/CollaborationTest.java index 466cf3c..7399290 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/CollaborationTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/CollaborationTest.java @@ -114,7 +114,7 @@ class CollaborationTest CollaborationTables.SCOPE_TYPE, CollaborationTables.FEATURE, CollaborationTables.PERMISSION_MODE); assertThat(instance.getPossibleValueSource(CollaborationTables.FEATURE).getEnumValues()).hasSize(12); assertThat(instance.getApp("collaboration").getChildren()).extracting(child -> child.getName()) - .containsExactlyInAnyOrder(CollaborationTables.COMMENT, CollaborationTables.ATTACHMENT, CollaborationTables.TAG, CollaborationTables.FIELD_DEFINITION, CollaborationTables.TASK, CollaborationTables.SHARE, CollaborationTables.ACCESS_REQUEST, CollaborationTables.RECORD_ASSIGNMENT, CollaborationTables.ACTIVITY, CollaborationTables.FEATURE_SETTING, CollaborationTables.USER); + .containsExactlyInAnyOrder(CollaborationTables.COMMENT, CollaborationTables.ATTACHMENT, CollaborationTables.ATTACHMENT_CLEANUP, CollaborationTables.TAG, CollaborationTables.FIELD_DEFINITION, CollaborationTables.TASK, CollaborationTables.SHARE, CollaborationTables.ACCESS_REQUEST, CollaborationTables.RECORD_ASSIGNMENT, CollaborationTables.ACTIVITY, CollaborationTables.FEATURE_SETTING, CollaborationTables.USER); assertThat(instance.getApp("collaboration").getWidgets()).containsExactlyElementsOf(CollaborationHome.HOME_WIDGETS); for(String widget : CollaborationHome.HOME_WIDGETS) { @@ -221,7 +221,7 @@ class CollaborationTest QInstance headless = new QInstance(); headless.addBackend(new QBackendMetaData().withName("memory").withBackendType(MemoryBackendModule.class)); new CollaborationQBitProducer().withQBitConfig(config.withAdminApp(false)).produce(headless); - assertThat(headless.getTables()).hasSize(19); + assertThat(headless.getTables()).hasSize(21); assertThat(headless.getApps()).isEmpty(); assertThat(headless.getWidgets()).hasSize(5); } diff --git a/src/test/java/com/kingsrook/qbits/collaboration/CoreContractsTest.java b/src/test/java/com/kingsrook/qbits/collaboration/CoreContractsTest.java index d4573a0..9778328 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/CoreContractsTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/CoreContractsTest.java @@ -84,7 +84,8 @@ class CoreContractsTest extends PersonalServiceFixture RuntimeMetaDataOverlay.refresh(instance); long generation = RuntimeMetaDataOverlay.getGeneration(instance); assertThat(instance.getTable("document").getVirtualFields()).containsKey("collaborationTags"); - assertThat(instance.getTable("document").getVirtualField("collaborationTags").getIsQuerySelectable()).isFalse(); + assertThat(instance.getTable("document").getVirtualField("collaborationTags").getIsQuerySelectable()).isTrue(); + assertThat(instance.getTable("document").getVirtualField("collaborationTags").getValueProvider()).isNotNull(); QBackendTransaction tx = new QBackendTransaction(); var row = new InsertAction().execute(new InsertInput(CollaborationTables.FEATURE_SETTING).withTransaction(tx).withRecords(List.of( new QRecord().withValue("scopeType", "TABLE").withValue("scopeName", "document").withValue("feature", "TAGS") diff --git a/src/test/java/com/kingsrook/qbits/collaboration/CustomFieldValuesPostgresTest.java b/src/test/java/com/kingsrook/qbits/collaboration/CustomFieldValuesPostgresTest.java new file mode 100644 index 0000000..34c07e7 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/CustomFieldValuesPostgresTest.java @@ -0,0 +1,188 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.collaboration; + +import java.sql.Connection; +import java.sql.DriverManager; +import java.util.Set; +import com.kingsrook.qbits.collaboration.customfields.CustomFieldValueProvider; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QVirtualFieldMetaData; +import com.kingsrook.qqq.backend.module.postgres.model.metadata.PostgreSQLBackendMetaData; +import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction; +import com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.wait.strategy.Wait; +import static org.assertj.core.api.Assertions.assertThat; + +/******************************************************************************* + ** Verify provider reads using a real database and explicit caller transactions. + *******************************************************************************/ +class CustomFieldValuesPostgresTest extends PersonalServiceFixture +{ + private static GenericContainer postgres; + + /******************************************************************************* + ** Own an isolated database; never connect to an application database. + *******************************************************************************/ + @BeforeAll + static void startPostgres() + { + postgres = new GenericContainer<>("postgres:17-alpine").withEnv("POSTGRES_PASSWORD", "eav-fixture-only") + .withExposedPorts(5432).waitingFor(Wait.forLogMessage(".*database system is ready to accept connections.*", 2)); + postgres.start(); + } + + /******************************************************************************* + ** Stop only the fixture's database. + *******************************************************************************/ + @AfterAll + static void stopPostgres() + { + if(postgres != null) + { + postgres.stop(); + } + } + + /******************************************************************************* + ** Reuse the secured host metadata with real PostgreSQL storage. + *******************************************************************************/ + @Override + @BeforeEach + public void setUp() throws Exception + { + super.setUp(); + instance.getBackends().remove("memory"); + instance.addBackend(new PostgreSQLBackendMetaData().withName("memory").withHostName(postgres.getHost()) + .withPort(postgres.getMappedPort(5432)).withDatabaseName("postgres").withUsername("postgres").withPassword("eav-fixture-only")); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("DROP SCHEMA public CASCADE; CREATE SCHEMA public"); + sql.execute(CollaborationSchema.postgresScript(config)); + sql.execute("CREATE TABLE document (id INTEGER PRIMARY KEY, title TEXT, tenant TEXT)"); + sql.execute("INSERT INTO document VALUES (1, 'Visible', 'a'), (3, 'Private', 'b')"); + } + CollaborationRecords.insert(CollaborationTables.FIELD_DEFINITION, new QRecord().withValue("tableName", "document") + .withValue("name", "rating").withValue("label", "Rating").withValue("type", "INTEGER").withValue("status", "ACTIVE")); + instance.getTable("document").withVirtualField(new QVirtualFieldMetaData("cf_rating", QFieldType.INTEGER).withLabel("Rating") + .withIsQuerySelectable(true).withValueProvider(new QCodeReference(CustomFieldValueProvider.class))); + } + + /******************************************************************************* + ** Release transaction state and pooled connections before parent fixture cleanup. + *******************************************************************************/ + @Override + public void tearDown() + { + QContext.clearTransaction(); + ConnectionManager.closeConnectionProviders(instance); + super.tearDown(); + } + + /******************************************************************************* + ** A separate connection establishes committed visibility independently. + *******************************************************************************/ + private Connection connect() throws Exception + { + return (DriverManager.getConnection("jdbc:postgresql://" + postgres.getHost() + ":" + postgres.getMappedPort(5432) + + "/postgres", "postgres", "eav-fixture-only")); + } + + /******************************************************************************* + ** Both vocabulary and assignments use the explicit transaction's visibility. + *******************************************************************************/ + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void tagLabelsSeePendingVocabularyAndCompletion(boolean commit) throws Exception + { + instance.getTable("document").withVirtualField(new QVirtualFieldMetaData("collaborationTags", QFieldType.STRING).withLabel("Tags") + .withIsQuerySelectable(true).withPossibleValueSourceName(CollaborationTables.TAG).withShouldTranslatePossibleValues(false) + .withValueProvider(new QCodeReference(com.kingsrook.qbits.collaboration.tags.TagValueProvider.class))); + var tags = new com.kingsrook.qbits.collaboration.tags.TagsService(); + try(var transaction = new RDBMSTransaction(connect())) + { + QContext.setTransaction(transaction); + var tag = tags.create("document", "Pending transaction label", null); + tags.apply("document", 1, java.util.List.of(tag.id()), java.util.List.of()); + QContext.clearTransaction(); + assertThat(QContext.getQBackendTransaction()).isNull(); + var pending = new QueryAction().execute(new QueryInput("document").withFieldNamesToInclude(Set.of("collaborationTags")) + .withShouldTranslatePossibleValues(true).withShouldGenerateDisplayValues(true).withTransaction(transaction)).getRecords(); + assertThat(pending).hasSize(1); + assertThat(pending.getFirst().getValues()).containsOnlyKeys("collaborationTags").containsEntry("collaborationTags", "Pending transaction label"); + assertThat(pending.getFirst().getDisplayValue("collaborationTags")).isEqualTo("Pending transaction label"); + assertThat(new QueryAction().execute(new QueryInput("document")).getRecords().getFirst().getValue("collaborationTags")).isNull(); + if(commit) + { + transaction.commit(); + } + else + { + transaction.rollback(); + } + } + assertThat(new QueryAction().execute(new QueryInput("document")).getRecords().getFirst().getValue("collaborationTags")) + .isEqualTo(commit ? "Pending transaction label" : null); + } + + /******************************************************************************* + ** Explicit query transactions see pending values even without thread context. + *******************************************************************************/ + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void explicitTransactionSeesPendingValueAndCompletion(boolean commit) throws Exception + { + try(var transaction = new RDBMSTransaction(connect())) + { + QContext.setTransaction(transaction); + CollaborationRecords.insert(CollaborationTables.FIELD_VALUE, new QRecord().withValue("tableName", "document").withValue("recordId", "1") + .withValue("fieldName", "rating").withValue("valueNumber", 42).withValue("parentDeleted", false)); + QContext.clearTransaction(); + assertThat(QContext.getQBackendTransaction()).isNull(); + var pending = new QueryAction().execute(new QueryInput("document").withFieldNamesToInclude(Set.of("cf_rating")) + .withTransaction(transaction)).getRecords(); + assertThat(pending).hasSize(1); + assertThat(pending.getFirst().getValues()).containsOnlyKeys("cf_rating").containsEntry("cf_rating", 42); + assertThat(new QueryAction().execute(new QueryInput("document")).getRecords().getFirst().getValue("cf_rating")).isNull(); + if(commit) + { + transaction.commit(); + } + else + { + transaction.rollback(); + } + } + assertThat(new QueryAction().execute(new QueryInput("document")).getRecords().getFirst().getValue("cf_rating")).isEqualTo(commit ? 42 : null); + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/CustomFieldsTest.java b/src/test/java/com/kingsrook/qbits/collaboration/CustomFieldsTest.java index 2fde99c..d986eb4 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/CustomFieldsTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/CustomFieldsTest.java @@ -51,6 +51,9 @@ import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaDat import com.kingsrook.qbits.collaboration.app.CollaborationHome; import com.kingsrook.qqq.backend.core.model.dashboard.widgets.TableData; import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; @@ -242,6 +245,59 @@ public class CustomFieldsTest extends PersonalServiceFixture assertThat(FieldProcesses.metadata()).extracting(process -> process.getLabel()).containsExactly("Publish field", "Archive field", "Rebuild index"); } + /******************************************************************************* + ** Typed EAV storage returns the declared Java value through an actual query. + *******************************************************************************/ + @ParameterizedTest + @MethodSource("eavValueTypes") + void eavValueTypes(String type, String column, java.io.Serializable stored, java.io.Serializable expected) throws Exception + { + admin(); + instance.addBackend(new RDBMSBackendMetaData().withName("fallback").withVendor("h2")); + instance.getTable("document").setBackendName("fallback"); + QRecord draft = new QRecord().withValue("tableName", "document").withValue("name", "typed").withValue("label", "Typed").withValue("type", type); + if(List.of("SELECT", "MULTI_SELECT").contains(type)) + { + draft.withValue("options", "[{\"id\":\"x\",\"label\":\"Choice X\"}]"); + } + if("RECORD_LINK".equals(type)) + { + draft.withValue("linkTableName", CollaborationTables.USER); + } + QRecord definition = new InsertAction().execute(new InsertInput(CollaborationTables.FIELD_DEFINITION).withRecords(List.of(draft))).getRecords().getFirst(); + assertThat(definition.getErrors()).isNullOrEmpty(); + FieldDefinitions.apply(definition.getValue("id"), "Publish"); + instance.getTable("document").setBackendName("memory"); + CollaborationRecords.insert(CollaborationTables.FIELD_VALUE, new QRecord().withValue("tableName", "document").withValue("recordId", "1") + .withValue("fieldName", "typed").withValue(column, stored).withValue("parentDeleted", false)); + login("alice"); + var records = new QueryAction().execute(new QueryInput("document").withFieldNamesToInclude(java.util.Set.of("cf_typed"))).getRecords(); + assertThat(records).hasSize(2); + assertThat(records).allSatisfy(record -> assertThat(record.getValues()).containsOnlyKeys("cf_typed")); + assertThat(records).extracting(record -> record.getValue("cf_typed")).containsExactlyInAnyOrder(expected, null); + Object actual = records.stream().map(record -> record.getValue("cf_typed")).filter(java.util.Objects::nonNull).findFirst().orElseThrow(); + assertThat(actual).isExactlyInstanceOf(expected.getClass()); + } + + /******************************************************************************* + ** Keep expectations independent of the provider's column-mapping helper. + *******************************************************************************/ + static java.util.stream.Stream eavValueTypes() + { + return (java.util.stream.Stream.of( + Arguments.of("INTEGER", "valueNumber", new BigDecimal("7.0"), 7), + Arguments.of("DECIMAL", "valueNumber", new BigDecimal("12.345"), new BigDecimal("12.345")), + Arguments.of("BOOLEAN", "valueBoolean", false, false), + Arguments.of("DATE", "valueDate", java.time.LocalDate.of(2026, 10, 22), java.time.LocalDate.of(2026, 10, 22)), + Arguments.of("DATE_TIME", "valueDateTime", java.time.Instant.parse("2026-10-22T12:34:56Z"), java.time.Instant.parse("2026-10-22T12:34:56Z")), + Arguments.of("SELECT", "valueText", "x", "x"), + Arguments.of("MULTI_SELECT", "valueText", "[\"x\"]", "[\"x\"]"), + Arguments.of("USER", "valueText", "alice", "alice"), + Arguments.of("RECORD_LINK", "valueText", "alice", "alice"), + Arguments.of("STRING", "valueText", "A, B", "A, B"), + Arguments.of("TEXT", "valueText", "First\nSecond", "First\nSecond"))); + } + /******************************************************************************* ** An unsupported JSON backend gets typed, read-only subquery filters. *******************************************************************************/ @@ -256,18 +312,50 @@ public class CustomFieldsTest extends PersonalServiceFixture instance.getTable("document").setBackendName("memory"); var virtual = instance.getTable("document").getVirtualField("cf_rating"); assertThat(virtual).isNotNull(); - assertThat(virtual.getIsQuerySelectable()).isFalse(); + assertThat(virtual.getIsQuerySelectable()).isTrue(); assertThat(virtual.getIsEditable()).isFalse(); assertThat(FieldDefinitions.get(definition.getValue("id")).getValueString("storage")).isEqualTo("EAV"); CollaborationRecords.insert(CollaborationTables.FIELD_VALUE, new QRecord().withValue("tableName", "document").withValue("recordId", "1") .withValue("fieldName", "rating").withValue("valueNumber", new BigDecimal("7")).withValue("parentDeleted", false)); + QContext.getQSession().withSecurityKeyValue("allTenants", true); + CollaborationRecords.insert(CollaborationTables.FIELD_VALUE, new QRecord().withValue("tableName", "document").withValue("recordId", "3") + .withValue("fieldName", "rating").withValue("valueNumber", new BigDecimal("99")).withValue("parentDeleted", false)); login("alice"); + var selected = new QueryAction().execute(new QueryInput("document").withFieldNamesToInclude(java.util.Set.of("cf_rating"))).getRecords(); + assertThat(selected).hasSize(2); + assertThat(selected).allSatisfy(row -> assertThat(row.getValues()).containsOnlyKeys("cf_rating")); + assertThat(selected).extracting(row -> row.getValue("cf_rating")).containsExactlyInAnyOrder(7, null); assertThat(query("cf_rating", QCriteriaOperator.GREATER_THAN, 5)).extracting(row -> row.getValueInteger("id")).containsExactly(1); assertThat(query("cf_rating", QCriteriaOperator.NOT_EQUALS, 7)).extracting(row -> row.getValueInteger("id")).containsExactly(2); assertThat(new QueryAction().execute(new QueryInput("document").withFilter(new QQueryFilter().withCriteria("cf_rating", QCriteriaOperator.IS_BLANK))).getRecords()) .extracting(row -> row.getValueInteger("id")).containsExactly(2); + // Reuse the same session and query to prove results are not cached across access changes. + QueryInput projected = new QueryInput("document").withFieldNamesToInclude(java.util.Set.of("cf_rating")); + QContext.getQSession().getSecurityKeyValues().put("tenant", List.of("b")); + assertThat(new QueryAction().execute(projected).getRecords()).extracting(row -> row.getValue("cf_rating")).containsExactly(99); + QContext.getQSession().getSecurityKeyValues().put("tenant", List.of("a")); + assertThat(new QueryAction().execute(projected).getRecords()).extracting(row -> row.getValue("cf_rating")).containsExactlyInAnyOrder(7, null); + QContext.getQSession().setPermissions(java.util.Set.of()); + assertThatThrownBy(() -> new QueryAction().execute(projected)).isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + login("alice"); + assertThat(com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordReferenceTableMetaData.of(instance.getTable(CollaborationTables.FIELD_VALUE))).isNotNull(); + assertThat(new QueryAction().execute(new QueryInput(CollaborationTables.FIELD_VALUE) + .withFilter(new QQueryFilter().withCriteria("tableName", QCriteriaOperator.EQUALS, "document") + .withCriteria("recordId", QCriteriaOperator.IN, List.of("1", "3")))).getRecords()) + .extracting(row -> row.getValueString("recordId")).containsExactly("1"); + virtual.setIsHidden(true); + assertThat(new QueryAction().execute(new QueryInput("document")).getRecords()).allSatisfy(row -> assertThat(row.getValues()).doesNotContainKey("cf_rating")); + virtual.setIsHidden(false); + admin(); + QContext.getQSession().withSecurityKeyValue("allTenants", true); + assertThat(FieldDefinitions.valueCount(FieldDefinitions.get(definition.getValue("id")))).isEqualTo(2); + QRecord deleted = CollaborationRecords.insert(CollaborationTables.FIELD_VALUE, new QRecord().withValue("tableName", "document").withValue("recordId", "2") + .withValue("fieldName", "rating").withValue("valueNumber", new BigDecimal("88")).withValue("parentDeleted", true)); + assertThat(deleted.getValueBoolean("parentDeleted")).isTrue(); + login("alice"); + assertThat(new QueryAction().execute(new QueryInput("document").withFieldNamesToInclude(java.util.Set.of("cf_rating"))).getRecords()) + .extracting(row -> row.getValue("cf_rating")).containsExactlyInAnyOrder(7, null); admin(); - assertThat(FieldDefinitions.valueCount(FieldDefinitions.get(definition.getValue("id")))).isEqualTo(1); var expander = new com.kingsrook.qbits.collaboration.customfields.FieldCriteriaExpander(); assertThat(expander.expandSearchTerm(instance.getTable("document"), "text", com.kingsrook.qqq.backend.core.actions.tables.query.SearchTermKind.TEXT, null)).isNotNull(); assertThat(com.kingsrook.qbits.collaboration.customfields.FieldCriteriaExpander.valueColumn("DATE")).isEqualTo("valueDate"); diff --git a/src/test/java/com/kingsrook/qbits/collaboration/FixtureEndpoint.java b/src/test/java/com/kingsrook/qbits/collaboration/FixtureEndpoint.java new file mode 100644 index 0000000..deacc86 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/FixtureEndpoint.java @@ -0,0 +1,214 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.collaboration; + + +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.regex.Matcher; +import java.util.regex.Pattern; +import com.github.dockerjava.api.command.InspectContainerResponse; +import com.github.dockerjava.api.model.ExposedPort; +import com.github.dockerjava.api.model.PortBinding; +import com.github.dockerjava.api.model.Ports; +import org.testcontainers.DockerClientFactory; +import org.testcontainers.containers.GenericContainer; + + +/******************************************************************************* + ** Where a disposable test container is reachable FROM THIS JVM. + ** + ** The original behavior is Testcontainers' own: the host it reports (which honors + ** TESTCONTAINERS_HOST_OVERRIDE) and the published port. The bridge address of the + ** sibling is used in exactly one topology: a JVM inside a build container that the + ** SAME daemon reports on its default bridge (a local full verify through the host + ** socket). Being inside some container is not enough: on the native runner the job + ** container is not a container of the isolated test daemon at all, so a sibling's + ** bridge address (a private address of that other daemon) is unreachable and the + ** published endpoint at the configured host must be dialed. + ** + ** The choice is made from the daemon's own description of the build container's + ** networks (no connection probing). If the build container cannot be identified or + ** is not on the default bridge (a job network, host networking, a remote or + ** separate daemon), the published endpoint is used. Timeouts, scenarios and + ** assertions are unchanged. The address is re-read from the daemon on every call + ** (a restarted container may get another bridge IP or published port). + *******************************************************************************/ +final class FixtureEndpoint +{ + private static final String DEFAULT_BRIDGE = "bridge"; + + private static final Pattern MOUNTINFO_CONTAINER_ID = Pattern.compile("/docker/containers/([0-9a-f]{64})/"); + + private static volatile Boolean buildContainerOnDefaultBridge; + + + + /*************************************************************************** + ** Host and port to dial. + ***************************************************************************/ + record Address(String host, int port) + { + } + + + + /*************************************************************************** + ** Utility class. + ***************************************************************************/ + private FixtureEndpoint() + { + } + + + + /*************************************************************************** + ** Resolve the endpoint of one exposed port of a running container, from a fresh + ** inspection (not Testcontainers' cached mapping). + ***************************************************************************/ + static Address of(GenericContainer container, int internalPort) + { + InspectContainerResponse inspected = container.getDockerClient().inspectContainerCmd(container.getContainerId()).exec(); + var networks = inspected.getNetworkSettings().getNetworks(); + String bridgeIp = networks == null || networks.get(DEFAULT_BRIDGE) == null ? null : networks.get(DEFAULT_BRIDGE).getIpAddress(); + Integer publishedPort = null; + var bindings = inspected.getNetworkSettings().getPorts().getBindings().get(new ExposedPort(internalPort)); + if(bindings != null && bindings.length > 0 && bindings[0].getHostPortSpec() != null) + { + publishedPort = Integer.valueOf(bindings[0].getHostPortSpec()); + } + return (choose(onBridgeWithBuildContainer(), bridgeIp, internalPort, container.getHost(), publishedPort)); + } + + + + /*************************************************************************** + ** Pure decision, separately tested. With the build container on the default + ** bridge: the sibling's bridge address and its own port. Otherwise the published + ** endpoint exactly as Testcontainers reports it. + ***************************************************************************/ + static Address choose(boolean buildContainerOnDefaultBridge, String bridgeIp, int internalPort, String publishedHost, Integer publishedPort) + { + if(buildContainerOnDefaultBridge) + { + if(bridgeIp == null || bridgeIp.isBlank()) + { + throw new IllegalStateException("FIXTURE_ENDPOINT_NO_BRIDGE_ADDRESS: the build container cannot reach a sibling without its bridge address"); + } + return (new Address(bridgeIp, internalPort)); + } + if(publishedHost == null || publishedHost.isBlank() || publishedPort == null) + { + throw new IllegalStateException("FIXTURE_ENDPOINT_NOT_PUBLISHED: no published endpoint to dial"); + } + return (new Address(publishedHost, publishedPort)); + } + + + + /*************************************************************************** + ** How the fixture publishes its port: the historical loopback-only binding + ** wherever a loopback dial can work, otherwise the daemon's own default binding. + ***************************************************************************/ + static PortBinding publishedBinding(int internalPort) + { + boolean loopback = loopbackPublication(onBridgeWithBuildContainer(), DockerClientFactory.instance().dockerHostIpAddress()); + return (new PortBinding(loopback ? Ports.Binding.bindIpAndPort("127.0.0.1", 0) : Ports.Binding.empty(), new ExposedPort(internalPort))); + } + + + + /*************************************************************************** + ** Pure decision, separately tested. A loopback-only publication is reachable + ** when the build container dials the bridge address (publication unused) or the + ** Docker host is this machine; if Testcontainers dials another host (a remote or + ** separate daemon, TESTCONTAINERS_HOST_OVERRIDE) a loopback binding could never + ** be reached, so the daemon's configured default binding is used instead. + ***************************************************************************/ + static boolean loopbackPublication(boolean buildContainerOnDefaultBridge, String dockerHost) + { + return (buildContainerOnDefaultBridge || dockerHost == null || dockerHost.isBlank() + || dockerHost.equals("localhost") || dockerHost.equals("127.0.0.1") || dockerHost.equals("::1")); + } + + + + /*************************************************************************** + ** Whether this JVM runs in a container that the daemon reports on the default + ** bridge. Decided once per JVM from the daemon's description of this container. + ***************************************************************************/ + private static boolean onBridgeWithBuildContainer() + { + Boolean known = buildContainerOnDefaultBridge; + if(known == null) + { + known = decideBuildContainerOnDefaultBridge(); + buildContainerOnDefaultBridge = known; + } + return (known); + } + + + + /*************************************************************************** + ** Inspect the build container; anything that cannot be established means the + ** original published endpoint (false). + ***************************************************************************/ + private static boolean decideBuildContainerOnDefaultBridge() + { + if(!Files.exists(Path.of("/.dockerenv"))) + { + return (false); + } + try + { + String id = ownContainerId(Files.readString(Path.of("/proc/self/mountinfo")), System.getenv("HOSTNAME")); + if(id == null) + { + return (false); + } + var networks = DockerClientFactory.instance().client().inspectContainerCmd(id).exec().getNetworkSettings().getNetworks(); + return (networks != null && networks.containsKey(DEFAULT_BRIDGE)); + } + catch(Exception e) + { + return (false); + } + } + + + + /*************************************************************************** + ** The id of the container this JVM runs in: from the mount table when it names + ** the container directory, otherwise the host name (the container id unless + ** overridden); null when neither exists. + ***************************************************************************/ + static String ownContainerId(String mountInfo, String hostName) + { + Matcher matcher = MOUNTINFO_CONTAINER_ID.matcher(mountInfo == null ? "" : mountInfo); + if(matcher.find()) + { + return (matcher.group(1)); + } + return (hostName == null || hostName.isBlank() ? null : hostName.trim()); + } + +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/FixtureEndpointTest.java b/src/test/java/com/kingsrook/qbits/collaboration/FixtureEndpointTest.java new file mode 100644 index 0000000..736baea --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/FixtureEndpointTest.java @@ -0,0 +1,114 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.collaboration; + + +import org.junit.jupiter.api.Test; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + + +/******************************************************************************* + ** The topology decision of the sibling-container fixtures. + *******************************************************************************/ +class FixtureEndpointTest +{ + private static final String CONTAINER_ID = "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"; + + + + /*************************************************************************** + ** Build container on the default bridge: dial the sibling bridge IP and its own port, never the published port. + ***************************************************************************/ + @Test + void buildContainerOnTheBridgeUsesTheBridgeAddress() + { + assertEquals(new FixtureEndpoint.Address("172.17.0.9", 5432), FixtureEndpoint.choose(true, "172.17.0.9", 5432, "localhost", 41111)); + assertEquals(new FixtureEndpoint.Address("172.17.0.9", 5432), FixtureEndpoint.choose(true, "172.17.0.9", 5432, null, null)); + } + + + + /*************************************************************************** + ** Every other topology (JVM on the Docker host, a separate or remote daemon such as the native runner's isolated + ** test daemon, job network, host networking, unidentified build container) dials the published endpoint + ** at the host Testcontainers reports (including TESTCONTAINERS_HOST_OVERRIDE), never a private bridge address. + ***************************************************************************/ + @Test + void everyOtherTopologyDialsThePublishedEndpoint() + { + assertEquals(new FixtureEndpoint.Address("localhost", 41111), FixtureEndpoint.choose(false, "172.17.0.9", 5432, "localhost", 41111)); + assertEquals(new FixtureEndpoint.Address("172.17.0.1", 32768), FixtureEndpoint.choose(false, "172.17.0.3", 6379, "172.17.0.1", 32768)); + assertEquals(new FixtureEndpoint.Address("host.docker.internal", 40000), FixtureEndpoint.choose(false, null, 5432, "host.docker.internal", 40000)); + } + + + + /*************************************************************************** + ** An unknown endpoint is an error, never a guessed address. + ***************************************************************************/ + @Test + void missingEndpointFailsClosed() + { + assertThrows(IllegalStateException.class, () -> FixtureEndpoint.choose(true, null, 5432, "localhost", 41111)); + assertThrows(IllegalStateException.class, () -> FixtureEndpoint.choose(true, " ", 5432, "localhost", 41111)); + assertThrows(IllegalStateException.class, () -> FixtureEndpoint.choose(false, "172.17.0.9", 5432, "localhost", null)); + assertThrows(IllegalStateException.class, () -> FixtureEndpoint.choose(false, "172.17.0.9", 5432, " ", 41111)); + } + + + + /*************************************************************************** + ** The historical loopback-only publication stays wherever a loopback dial can work; when Testcontainers dials + ** another host (the native runner's separate daemon) the daemon's default binding is used so the dial can succeed. + ***************************************************************************/ + @Test + void loopbackPublicationOnlyWhereALoopbackDialCanWork() + { + assertTrue(FixtureEndpoint.loopbackPublication(false, "localhost")); + assertTrue(FixtureEndpoint.loopbackPublication(false, "127.0.0.1")); + assertTrue(FixtureEndpoint.loopbackPublication(false, "::1")); + assertTrue(FixtureEndpoint.loopbackPublication(false, null)); + assertTrue(FixtureEndpoint.loopbackPublication(true, "172.17.0.1")); + assertFalse(FixtureEndpoint.loopbackPublication(false, "172.17.0.1")); + assertFalse(FixtureEndpoint.loopbackPublication(false, "host.docker.internal")); + assertFalse(FixtureEndpoint.loopbackPublication(false, "docker.example.test")); + } + + + + /*************************************************************************** + ** The build container id comes from the mount table when present, else the host name. + ***************************************************************************/ + @Test + void ownContainerIdPrefersTheMountTableThenTheHostName() + { + String mountInfo = "700 699 0:50 /docker/containers/" + CONTAINER_ID + "/hostname /etc/hostname rw,relatime - ext4 /dev/sda1 rw"; + assertEquals(CONTAINER_ID, FixtureEndpoint.ownContainerId(mountInfo, "unrelated")); + assertEquals("abc123", FixtureEndpoint.ownContainerId("no container mounts here", " abc123 ")); + assertNull(FixtureEndpoint.ownContainerId("", null)); + assertNull(FixtureEndpoint.ownContainerId(null, " ")); + } + +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/MyWorkPostgresTest.java b/src/test/java/com/kingsrook/qbits/collaboration/MyWorkPostgresTest.java new file mode 100644 index 0000000..ae65573 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/MyWorkPostgresTest.java @@ -0,0 +1,231 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.sql.Connection; +import java.util.Arrays; +import java.util.Locale; +import java.sql.DriverManager; +import java.util.ArrayList; +import java.util.List; +import com.kingsrook.qbits.collaboration.mywork.MyWorkService; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.module.postgres.model.metadata.PostgreSQLBackendMetaData; +import com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager; +import com.kingsrook.qqq.backend.module.rdbms.jdbc.C3P0PooledConnectionProvider; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.ConnectionPoolSettings; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.wait.strategy.Wait; +import static org.assertj.core.api.Assertions.assertThat; + +/******************************************************************************* + ** Real PostgreSQL query counts for the bounded, caller-authorized My Work scan. + *******************************************************************************/ +class MyWorkPostgresTest extends PersonalServiceFixture +{ + private static GenericContainer postgres; + + /******************************************************************************* + ** Collect SQL counts in an owned disposable database, never an application DB. + *******************************************************************************/ + @BeforeAll + static void startPostgres() + { + postgres = new GenericContainer<>("postgres:17-alpine") + .withEnv("POSTGRES_PASSWORD", "mywork-fixture-only").withExposedPorts(5432) + .withCommand("postgres", "-c", "shared_preload_libraries=pg_stat_statements") + .waitingFor(Wait.forLogMessage(".*database system is ready to accept connections.*", 2)); + postgres.start(); + } + + /******************************************************************************* + ** Stop only the database owned by this class. + *******************************************************************************/ + @AfterAll + static void stopPostgres() + { + if(postgres != null) + { + postgres.stop(); + } + } + + /******************************************************************************* + ** Reuse the real authorization fixture with PostgreSQL persistence. + *******************************************************************************/ + @Override + @BeforeEach + public void setUp() throws Exception + { + super.setUp(); + instance.getBackends().remove("memory"); + instance.addBackend(new PostgreSQLBackendMetaData().withName("memory").withHostName(postgres.getHost()) + .withPort(postgres.getMappedPort(5432)).withDatabaseName("postgres") + .withUsername("postgres").withPassword("mywork-fixture-only")); + var backend = (PostgreSQLBackendMetaData) instance.getBackend("memory"); + backend.setConnectionProvider(new QCodeReference(C3P0PooledConnectionProvider.class)); + backend.setConnectionPoolSettings(new ConnectionPoolSettings().withInitialPoolSize(1).withMinPoolSize(1) + .withMaxPoolSize(3).withAcquireIncrement(1).withCheckoutTimeoutSeconds(2)); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("CREATE EXTENSION pg_stat_statements"); + sql.execute(CollaborationSchema.postgresScript(config)); + sql.execute("CREATE TABLE document (id INTEGER PRIMARY KEY, title TEXT, tenant TEXT)"); + sql.execute("INSERT INTO document VALUES (1, 'Visible', 'a'), (3, 'Private', 'b')"); + sql.execute("INSERT INTO collaboration_task (id, table_name, record_id, title, assignee_user_id, created_by_user_id, status, parent_deleted, due_at) " + + "SELECT n, 'document', CASE WHEN n=601 THEN '3' ELSE '1' END, 'Task ' || n, 'alice', 'alice', 'OPEN', false, " + + "CASE WHEN n%2=0 THEN TIMESTAMPTZ '2026-10-22 00:00:00Z' + (601-n)*INTERVAL '1 second' ELSE NULL END FROM generate_series(1,601) n"); + sql.execute("SELECT pg_stat_statements_reset()"); + } + } + + /******************************************************************************* + ** Release owner pools before container teardown. + *******************************************************************************/ + @Override + public void tearDown() + { + ConnectionManager.closeConnectionProviders(instance); + super.tearDown(); + } + + /******************************************************************************* + ** Exact counts/order and bounded SQL parent/page queries on the real backend. + *******************************************************************************/ + @Test + void summaryPreservesVisibilityWithBoundedSqlQueries() throws Exception + { + var summary = new MyWorkService().summary(); + assertThat(summary.openTaskCount()).isEqualTo(600); + assertThat(summary.tasks()).extracting(row -> row.getValueInteger("id")) + .containsExactly(600, 598, 596, 594, 592, 590, 588, 586, 584, 582); + try(Connection connection = connect(); var sql = connection.createStatement(); + var rows = sql.executeQuery("SELECT query, calls FROM pg_stat_statements WHERE query ILIKE 'SELECT%'") ) + { + long parents = 0; + long tasks = 0; + while(rows.next()) + { + String query = rows.getString(1); + if(query.contains("\"document\"")) + { + parents += rows.getLong(2); + } + if(query.contains("\"collaboration_task\"")) + { + tasks += rows.getLong(2); + } + } + assertThat(parents).as("SQL parent queries").isBetween(1L, 8L); + assertThat(tasks).as("SQL task page queries").isEqualTo(5); + } + QContext.getQSession().getSecurityKeyValues().put("tenant", new ArrayList<>(List.of("b"))); + assertThat(new MyWorkService().summary().openTaskCount()).isEqualTo(1); + if(Boolean.getBoolean("qqq.mywork.load")) + { + QContext.getQSession().getSecurityKeyValues().put("tenant", new ArrayList<>(List.of("a"))); + measureLoad(); + } + } + + /******************************************************************************* + ** Opt-in reproducible SQL work measurement, not a deployment capacity claim. + *******************************************************************************/ + private void measureLoad() throws Exception + { + final int tasks = 2000; + final int samples = 10; + List report = new ArrayList<>(); + report.add("tasks,parents,samples,p50_ms,p95_ms,p99_ms,sql_calls,sql_execution_ms"); + for(int parents : List.of(10, tasks)) + { + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("TRUNCATE collaboration_task, document"); + sql.execute("INSERT INTO document SELECT n, 'Document ' || n, CASE WHEN n%10=0 THEN 'b' ELSE 'a' END FROM generate_series(1," + parents + ") n"); + sql.execute("INSERT INTO collaboration_task (id, table_name, record_id, title, assignee_user_id, created_by_user_id, status, parent_deleted) " + + "SELECT n, 'document', (((n-1)%" + parents + ")+1)::text, 'Task ' || n, 'alice', 'alice', 'OPEN', false FROM generate_series(1," + tasks + ") n"); + } + // Warm the JVM and pool without counting these calls in the samples. + for(int warmup = 0; warmup < 2; warmup++) + { + assertThat(new MyWorkService().summary().openTaskCount()).isEqualTo(1800); + } + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("SELECT pg_stat_statements_reset()"); + } + double[] elapsed = new double[samples]; + for(int sample = 0; sample < samples; sample++) + { + long start = System.nanoTime(); + var summary = new MyWorkService().summary(); + elapsed[sample] = (System.nanoTime() - start) / 1_000_000.0; + assertThat(summary.openTaskCount()).isEqualTo(1800); + assertThat(summary.tasks()).extracting(row -> row.getValueInteger("id")) + .containsExactly(1, 2, 3, 4, 5, 6, 7, 8, 9, 11); + } + var backend = (PostgreSQLBackendMetaData) instance.getBackend("memory"); + var pool = ConnectionManager.getPoolMetrics(instance, backend); + // C3P0 returns connections asynchronously; allow a bounded check-in interval. + long deadline = System.nanoTime() + 2_000_000_000L; + while(pool.get("connectionsBusy").intValue() != 0 && System.nanoTime() < deadline) + { + Thread.sleep(10); + pool = ConnectionManager.getPoolMetrics(instance, backend); + } + assertThat(pool.get("connectionsTotal").intValue()).isBetween(1, 3); + assertThat(pool.get("connectionsBusy").intValue()).isZero(); + assertThat(pool.get("checkoutTimeouts").longValue()).isZero(); + Arrays.sort(elapsed); + try(Connection connection = connect(); var sql = connection.createStatement(); + var rows = sql.executeQuery("SELECT COALESCE(sum(calls),0), COALESCE(sum(total_exec_time),0) FROM pg_stat_statements " + + "WHERE query ILIKE 'SELECT%' AND (query LIKE '%\"document\"%' OR query LIKE '%\"collaboration_task\"%' " + + "OR query LIKE '%\"collaboration_record_assignment\"%' OR query LIKE '%\"collaboration_mention\"%')")) + { + assertThat(rows.next()).isTrue(); + report.add(String.format(Locale.ROOT, "%d,%d,%d,%.3f,%.3f,%.3f,%d,%.3f", tasks, parents, samples, + elapsed[4], elapsed[9], elapsed[9], rows.getLong(1), rows.getDouble(2))); + } + } + Path output = Path.of("target/mywork-load.csv"); + Files.createDirectories(output.getParent()); + Files.write(output, report); + System.out.println("My Work local PostgreSQL workload: " + output.toAbsolutePath()); + report.forEach(System.out::println); + } + + /******************************************************************************* + ** A fixture-owned connection for setup and independent SQL observations. + *******************************************************************************/ + private Connection connect() throws Exception + { + return (DriverManager.getConnection("jdbc:postgresql://" + postgres.getHost() + ":" + postgres.getMappedPort(5432) + "/postgres", "postgres", "mywork-fixture-only")); + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/MyWorkQueryBudgetTest.java b/src/test/java/com/kingsrook/qbits/collaboration/MyWorkQueryBudgetTest.java new file mode 100644 index 0000000..fb5bcfb --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/MyWorkQueryBudgetTest.java @@ -0,0 +1,132 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration; + +import java.time.Instant; +import java.util.ArrayList; +import java.util.List; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qbits.collaboration.mywork.MyWorkService; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; + +/******************************************************************************* + ** Measure backend queries while preserving exact, caller-visible summary counts. + *******************************************************************************/ +class MyWorkQueryBudgetTest extends PersonalServiceFixture +{ + /******************************************************************************* + ** Counts cross page boundaries without including another user's or hidden work. + *******************************************************************************/ + @Test + void summaryPagesAssignmentsAndMentionsWithoutLeakingCounts() throws Exception + { + List parents = new ArrayList<>(); + List assignments = new ArrayList<>(); + List mentions = new ArrayList<>(); + for(int id = 1; id <= 208; id++) + { + if(id > 3) + { + parents.add(new QRecord().withValue("id", id).withValue("title", "Parent " + id).withValue("tenant", id == 208 ? "b" : "a")); + } + assignments.add(new QRecord().withValue("id", id).withValue("tableName", "document").withValue("recordId", Integer.toString(id)) + .withValue("assigneeUserId", id == 207 ? "bob" : "alice").withValue("parentDeleted", false)); + } + for(int id = 1; id <= 601; id++) + { + mentions.add(new QRecord().withValue("id", id).withValue("tableName", "document").withValue("recordId", Integer.toString(id % 3 + 1)) + .withValue("mentionedUserId", "alice").withValue("notifiedAt", Instant.now()).withValue("parentDeleted", false)); + } + mentions.add(new QRecord().withValue("id", 602).withValue("tableName", "document").withValue("recordId", "1") + .withValue("mentionedUserId", "bob").withValue("notifiedAt", Instant.now()).withValue("parentDeleted", false)); + mentions.add(new QRecord().withValue("id", 603).withValue("tableName", "document").withValue("recordId", "1") + .withValue("mentionedUserId", "alice").withValue("parentDeleted", false)); + MemoryRecordStore.getInstance().insert(new InsertInput("document").withRecords(parents), true); + MemoryRecordStore.getInstance().insert(new InsertInput(CollaborationTables.RECORD_ASSIGNMENT).withRecords(assignments), true); + MemoryRecordStore.getInstance().insert(new InsertInput(CollaborationTables.MENTION).withRecords(mentions), true); + MemoryRecordStore.resetStatistics(); + MemoryRecordStore.setCollectStatistics(true); + try + { + var summary = new MyWorkService().summary(); + assertThat(summary.assignedRecordsByTable()).containsOnlyKeys("document").containsEntry("document", 205L); + assertThat(summary.unreadMentionCount()).isEqualTo(401); + assertThat(summary.openTaskCount()).isZero(); + assertThat(summary.tasks()).isEmpty(); + MemoryRecordStore.getActionInputs().get(QueryInput.class).stream().map(input -> (QueryInput) input) + .filter(input -> input.getTableName().startsWith("collaboration_")) + .forEach(input -> assertThat(input.getFilter().getLimit()).isNotNull().isLessThanOrEqualTo(200)); + } + finally + { + MemoryRecordStore.setCollectStatistics(false); + MemoryRecordStore.resetStatistics(); + } + } + + /******************************************************************************* + ** Hundreds of tasks on one parent must not cause hundreds of parent lookups. + *******************************************************************************/ + @Test + void summaryBoundsPagesAndRepeatedParentLookups() throws Exception + { + List rows = new ArrayList<>(); + for(int id = 1; id <= 601; id++) + { + rows.add(new QRecord().withValue("id", id).withValue("tableName", "document") + .withValue("recordId", id == 601 ? "3" : "1").withValue("title", "Task " + id) + .withValue("assigneeUserId", "alice").withValue("createdByUserId", "alice") + .withValue("status", "OPEN").withValue("parentDeleted", false) + .withValue("dueAt", id % 2 == 0 ? Instant.parse("2026-10-22T00:00:00Z").plusSeconds(601 - id) : null)); + } + // Populate historical rows directly so current parent authorization is tested on read. + MemoryRecordStore.getInstance().insert(new InsertInput(CollaborationTables.TASK).withRecords(rows), true); + MemoryRecordStore.resetStatistics(); + MemoryRecordStore.setCollectStatistics(true); + try + { + var summary = new MyWorkService().summary(); + assertThat(summary.openTaskCount()).isEqualTo(600); + assertThat(summary.tasks()).extracting(row -> row.getValueInteger("id")) + .containsExactly(600, 598, 596, 594, 592, 590, 588, 586, 584, 582); + List queries = MemoryRecordStore.getActionInputs().get(QueryInput.class).stream() + .map(input -> (QueryInput) input).toList(); + assertThat(queries.stream().filter(input -> "document".equals(input.getTableName())).count()).isLessThanOrEqualTo(8); + assertThat(queries.stream().filter(input -> CollaborationTables.TASK.equals(input.getTableName()))) + .allSatisfy(input -> assertThat(input.getFilter().getLimit()).isNotNull().isLessThanOrEqualTo(200)); + // A second request must recheck current permissions rather than reuse cached decisions. + QContext.getQSession().getSecurityKeyValues().put("tenant", new ArrayList<>(List.of("b"))); + assertThat(new MyWorkService().summary().openTaskCount()).isEqualTo(1); + } + finally + { + MemoryRecordStore.setCollectStatistics(false); + MemoryRecordStore.resetStatistics(); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/OptionalRedisConsumerTest.java b/src/test/java/com/kingsrook/qbits/collaboration/OptionalRedisConsumerTest.java new file mode 100644 index 0000000..49d73d6 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/OptionalRedisConsumerTest.java @@ -0,0 +1,144 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration; + +import java.net.URL; +import java.net.URLClassLoader; +import java.nio.file.Path; +import java.util.Arrays; +import org.junit.jupiter.api.Test; +import static org.junit.jupiter.api.Assertions.*; + +/******************************************************************************* + ** Exercise a consumer classpath without the optional Redis implementation/driver. + *******************************************************************************/ +class OptionalRedisConsumerTest +{ + /*************************************************************************** + ** An isolated loader must not resolve Redis from the test runner's parent. + ***************************************************************************/ + @Test + void memoryPresenceWithoutRedis() throws Exception + { + String classpath = System.getProperty("surefire.test.class.path", System.getProperty("java.class.path")); + URL[] urls = Arrays.stream(classpath.split(java.io.File.pathSeparator)) + .filter(entry -> !entry.replace('\\', '/').contains("/qbit-redis/") + && !entry.replace('\\', '/').contains("/qbit-email/") + && !Path.of(entry).getFileName().toString().startsWith("jedis-")) + .map(entry -> + { + try + { + return (Path.of(entry).toUri().toURL()); + } + catch(Exception e) + { + throw (new IllegalArgumentException(e)); + } + }).toArray(URL[]::new); + ClassLoader previous = Thread.currentThread().getContextClassLoader(); + try(var loader = new URLClassLoader(urls, ClassLoader.getPlatformClassLoader())) + { + assertThrows(ClassNotFoundException.class, () -> loader.loadClass("com.kingsrook.qbits.redis.api.RedisNamespace")); + assertThrows(ClassNotFoundException.class, () -> loader.loadClass("redis.clients.jedis.RedisClient")); + assertThrows(ClassNotFoundException.class, () -> loader.loadClass("com.kingsrook.qbits.email.api.EmailApi")); + Thread.currentThread().setContextClassLoader(loader); + loader.loadClass(Probe.class.getName()).getMethod("exercise").invoke(null); + } + finally + { + Thread.currentThread().setContextClassLoader(previous); + } + } + + /*************************************************************************** + ** Loaded in the isolated consumer, including metadata enrichment and delivery. + ***************************************************************************/ + public static class Probe extends PersonalServiceFixture + { + /************************************************************************ + ** Real host setup and authorized presence work without Redis artifacts. + ************************************************************************/ + public static void exercise() throws Exception + { + var probe = new Probe(); + try + { + probe.setUp(); + var service = new com.kingsrook.qbits.collaboration.presence.PresenceService(); + String lease = service.openLease("document", "1"); + assertTrue(service.snapshot("document", "1").isEmpty()); + assertEquals(1, service.renewLease(lease, "document", "1", "VIEWING", null, 1L).size()); + service.leave(lease, "document", "1", 2L); + assertTrue(service.snapshot("document", "1").isEmpty()); + assertFalse(new com.kingsrook.qbits.collaboration.presence.RedisPresenceAudience().allows("untrusted")); + probe.config.withUserSessionProvider(new com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference(RecipientSessions.class)); + assertEquals(java.util.Optional.of(true), new com.kingsrook.qbits.collaboration.security.CollaborationRecordAccessChecker() + .canRead("alice", "document", "1")); + var notice = new com.kingsrook.qbits.collaboration.notify.CollaborationNotification("alice", "Test", "Test", + com.kingsrook.qbits.collaboration.notify.CollaborationNotification.Category.MENTION, "document", "1", "/app/document/1"); + var receipts = new java.util.concurrent.atomic.AtomicInteger(); + com.kingsrook.qbits.collaboration.notify.Notifications.send(notice, () -> receipts.incrementAndGet()); + assertEquals(0, receipts.get(), "Missing email must not acknowledge delivery"); + probe.config.withNotifier(new com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference(AcceptingNotifier.class)); + com.kingsrook.qbits.collaboration.notify.Notifications.send(notice, () -> receipts.incrementAndGet()); + assertEquals(1, receipts.get(), "Explicit host notifier must still work without email"); + } + finally + { + probe.tearDown(); + } + } + } + + /*************************************************************************** + ** Host session lookup makes the notification eligible for actual delivery. + ***************************************************************************/ + public static class RecipientSessions implements com.kingsrook.qqq.backend.core.actions.collaboration.UserSessionProviderInterface + { + /************************************************************************ + ** Fixture recipient has access to the parent record in tenant a. + ************************************************************************/ + @Override + public java.util.Optional sessionForUser(String userId) + { + return (java.util.Optional.of(new com.kingsrook.qqq.backend.core.model.session.QSession() + .withUser(new com.kingsrook.qqq.backend.core.model.session.QUser().withIdReference(userId)) + .withPermissions("document.read").withSecurityKeyValue("tenant", "a"))); + } + } + + /*************************************************************************** + ** A host transport that accepts delivery without the optional email adapter. + ***************************************************************************/ + public static class AcceptingNotifier implements com.kingsrook.qbits.collaboration.notify.CollaborationNotifier + { + /************************************************************************ + ** Test transport has no external delivery side effects. + ************************************************************************/ + @Override + public void notify(com.kingsrook.qbits.collaboration.notify.CollaborationNotification notification) + { + assertEquals("alice", notification.recipientUserId()); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/PresenceTest.java b/src/test/java/com/kingsrook/qbits/collaboration/PresenceTest.java index 5f3fe70..21ceda4 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/PresenceTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/PresenceTest.java @@ -24,10 +24,8 @@ package com.kingsrook.qbits.collaboration; import java.time.Clock; import java.time.Instant; import java.time.ZoneId; -import java.util.LinkedHashMap; import java.util.List; import java.util.concurrent.CopyOnWriteArrayList; -import com.kingsrook.qbits.collaboration.api.CollaborationPresenceSchemas; import com.kingsrook.qbits.collaboration.presence.PresenceRegistry; import com.kingsrook.qbits.collaboration.presence.PresenceService; import com.kingsrook.qbits.collaboration.app.CollaborationHome; @@ -38,7 +36,6 @@ import com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordFeature import com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordFeaturesMetaData; import com.kingsrook.qqq.backend.core.realtime.InMemoryRealtimeBus; import com.kingsrook.qqq.backend.core.realtime.RealtimeMessage; -import com.kingsrook.qqq.backend.core.utils.JsonUtils; import org.junit.jupiter.api.Test; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; @@ -48,6 +45,163 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy; *******************************************************************************/ class PresenceTest extends PersonalServiceFixture { + /******************************************************************************* + ** Issued service operations retain identity, authorization and cleanup rules. + *******************************************************************************/ + @Test + void issuedServiceRechecksAccessAndDepartureOrder() throws Exception + { + var service = new PresenceService(); + String lease = service.openLease("document", "01"); + assertThat(service.snapshot("document", "1")).isEmpty(); + assertThatThrownBy(() -> service.renewLease(lease, "document", "1", "EDITING", "missing", 1L)) + .isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QBadRequestException.class); + assertThat(service.renewLease(lease, "document", "1", "EDITING", "title", 1L)).hasSize(1); + assertThatThrownBy(() -> service.leave(lease, "document", "1", null)) + .isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QBadRequestException.class); + login("bob"); + assertThatThrownBy(() -> service.renewLease(lease, "document", "1", "EDITING", "title", 2L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + login("alice"); + QContext.getQSession().getPermissions().remove("document.read"); + assertThatThrownBy(() -> service.openLease("document", "1")).isInstanceOf(QNotFoundException.class); + assertThatThrownBy(() -> service.renewLease(lease, "document", "1", "EDITING", "title", 3L)) + .isInstanceOf(QNotFoundException.class); + assertThat(PresenceRegistry.get(instance).snapshot("document", "1")).isEmpty(); + login("alice"); + assertThatThrownBy(() -> service.renewLease(lease, "document", "1", "EDITING", "title", 4L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + } + + /******************************************************************************* + ** Local issued authority is bounded and cannot be recreated by late requests. + *******************************************************************************/ + @Test + void localIssuedLeasesRequireExistingAuthority() throws Exception + { + var clock = new MutableClock(); + try(var bus = new InMemoryRealtimeBus(); var local = new PresenceRegistry(bus, clock, 1); var other = new PresenceRegistry(bus, clock, 1)) + { + String lease = local.openLease("alice", "Alice", "document", "1"); + assertThat(lease).startsWith("lease:"); + assertThat(local.snapshot("document", "1")).isEmpty(); + assertThat(local.openLease("bob", "Bob", "document", "2")).isNull(); + assertThatThrownBy(() -> other.renewLease("alice", "Alice", lease, "document", "1", "EDITING", "title", 1L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + assertThatThrownBy(() -> local.renewLease("bob", "Bob", lease, "document", "1", "EDITING", "title", 1L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + local.renewLease("alice", "Alice", lease, "document", "1", "EDITING", "title", 1L); + clock.advance(1000); + local.renewLease("alice", "Alice", lease, "document", "1", "VIEWING", null, 1L); + assertThat(local.snapshot("document", "1").getFirst().mode()).isEqualTo("EDITING"); + clock.advance(PresenceRegistry.TTL_MILLIS - 1000); + assertThatThrownBy(() -> local.renewLease("alice", "Alice", lease, "document", "1", "EDITING", "title", 2L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + assertThat(local.snapshot("document", "1")).isEmpty(); + String replacement = local.openLease("alice", "Alice", "document", "1"); + local.renewLease("alice", "Alice", replacement, "document", "1", "EDITING", "title", 1L); + local.leave("alice", replacement, "document", "1", 3L); + assertThatThrownBy(() -> local.renewLease("alice", "Alice", replacement, "document", "1", "EDITING", "title", 4L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + clock.advance(PresenceRegistry.TTL_MILLIS); + assertThatThrownBy(() -> local.heartbeat("alice", "Alice", replacement, "document", "1", "EDITING", "title", 5L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + assertThatThrownBy(() -> local.renewLease("alice", "Alice", replacement, "document", "1", "EDITING", "title", 5L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + assertThat(local.snapshot("document", "1")).isEmpty(); + } + } + + /******************************************************************************* + ** Delayed HTTP renewals and departures retain client order across replicas. + *******************************************************************************/ + @Test + void clientSequenceOrdersDelayedRequests() throws Exception + { + var clock = new MutableClock(); + try(var bus = new InMemoryRealtimeBus(); var first = new PresenceRegistry(bus, clock, 10); var second = new PresenceRegistry(bus, clock, 10)) + { + first.heartbeat("alice", "Alice", "tab", "document", "1", "EDITING", "title", 2L); + clock.advance(10); + second.heartbeat("alice", "Alice", "tab", "document", "1", "EDITING", "old", 1L); + assertThat(first.snapshot("document", "1").getFirst().fieldName()).isEqualTo("title"); + second.leave("alice", "tab", "document", "1", 3L); + clock.advance(10); + first.heartbeat("alice", "Alice", "tab", "document", "1", "EDITING", "title", 2L); + assertThat(first.snapshot("document", "1")).isEmpty(); + assertThat(second.snapshot("document", "1")).isEmpty(); + first.heartbeat("alice", "Alice", "tab", "document", "1", "VIEWING", null, 4L); + second.leave("alice", "tab", "document", "1", 3L); + assertThat(second.snapshot("document", "1")).hasSize(1); + } + } + + /******************************************************************************* + ** Departure keeps unknown-target errors without preventing revoked cleanup. + *******************************************************************************/ + @Test + void departureRejectsUnknownTable() throws Exception + { + var service = new PresenceService(); + assertThatThrownBy(() -> service.leave("valid-connection", "nonexistent", null)) + .isInstanceOf(QNotFoundException.class).hasMessage("Not found"); + service.heartbeat("valid-connection", "document", "1", "VIEWING", null); + QContext.getQSession().getPermissions().remove("document.read"); + service.leave("valid-connection", "document", "1"); + assertThat(PresenceRegistry.get(instance).snapshot("document", "1")).isEmpty(); + } + + /******************************************************************************* + ** A departure wins over delayed and same-timestamp renewals on every pod. + *******************************************************************************/ + @Test + void departedLeaseCannotBeResurrectedByOldHeartbeat() throws Exception + { + var clock = new MutableClock(); + try(var bus = new InMemoryRealtimeBus(); var registry = new PresenceRegistry(bus, clock, 2)) + { + List events = new CopyOnWriteArrayList<>(); + bus.subscribe(events::add); + registry.heartbeat("alice", "Alice", "one", "document", "1", "EDITING", "title"); + var old = events.stream().filter(event -> event.getEvent().equals("presence.heartbeat")).findFirst().orElseThrow(); + registry.leave("alice", "one", "document", "1"); + bus.publish(old); + assertThat(registry.snapshot("document", "1")).isEmpty(); + clock.advance(1); + registry.heartbeat("alice", "Alice", "one", "document", "1", "VIEWING", null); + assertThat(registry.snapshot("document", "1")).hasSize(1); + clock.advance(PresenceRegistry.TTL_MILLIS); + assertThat(registry.snapshot("document", "1")).isEmpty(); + } + } + + /******************************************************************************* + ** Focus is validated for the sender and projected again for each recipient. + *******************************************************************************/ + @Test + void focusedFieldsRespectVisibilityOnEveryResponse() throws Exception + { + var table = instance.getTable("document"); + table.withField(new com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData("hidden", com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType.STRING).withIsHidden(true)); + table.withField(new com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData("password", com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType.PASSWORD)); + var service = new PresenceService(); + for(String field : List.of("hidden", "password", "missing")) + { + assertThatThrownBy(() -> service.heartbeat("alice-tab", "document", "1", "EDITING", field)) + .isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QBadRequestException.class); + } + service.heartbeat("alice-tab", "document", "1", "EDITING", "title"); + login("bob"); + assertThat(service.snapshot("document", "1").getFirst().fieldName()).isEqualTo("title"); + table.getField("title").setIsHidden(true); + assertThat(service.snapshot("document", "1").getFirst().fieldName()).isNull(); + assertThat(service.heartbeat("bob-tab", "document", "1", "VIEWING", null)) + .allSatisfy(viewer -> assertThat(viewer.fieldName()).isNull()); + assertThat(PresenceRegistry.get(instance).snapshot("document", "1")).anySatisfy(viewer -> assertThat(viewer.fieldName()).isEqualTo("title")); + table.getField("title").setIsHidden(false); + assertThat(service.snapshot("document", "1")).anySatisfy(viewer -> assertThat(viewer.fieldName()).isEqualTo("title")); + } + /******************************************************************************* ** Renewals replicate even without a visible change; tabs merge until all expire. *******************************************************************************/ @@ -64,7 +218,7 @@ class PresenceTest extends PersonalServiceFixture assertThat(second.snapshot("document", "1")).hasSize(1); clock.advance(1000); second.heartbeat("alice", "Alice", "two", "document", "1", "EDITING", "title"); - assertThat(first.snapshot("document", "1")).containsExactly(new PresenceRegistry.Viewer("alice", "Alice", "EDITING", "title", 2)); + assertThat(first.snapshot("document", "1")).containsExactly(new PresenceRegistry.Viewer("alice", "Alice", "EDITING", "title")); assertThat(first.snapshot("document", null).getFirst().fieldName()).isNull(); int changes = (int) events.stream().filter(event -> event.getEvent().equals("presence.changed")).count(); clock.advance(20000); @@ -90,177 +244,6 @@ class PresenceTest extends PersonalServiceFixture } } - /******************************************************************************* - ** The last leave still sends viewers: the SSE writer and the HTTP snapshot must - ** both carry an empty array, never omit the field (#52). - *******************************************************************************/ - @Test - void lastLeaveSendsEmptyViewers() throws Exception - { - var clock = new MutableClock(); - try(var bus = new InMemoryRealtimeBus(); var registry = new PresenceRegistry(bus, clock, PresenceRegistry.MAX_ENTRIES)) - { - List events = new CopyOnWriteArrayList<>(); - bus.subscribe(events::add); - registry.heartbeat("bob", "Bob", "one", "person", null, "VIEWING", null); - registry.heartbeat("bob", "Bob", "one", "person", "7", "EDITING", "name"); - registry.leave("bob", "one", "person", null); - registry.leave("bob", "one", "person", "7"); - assertThat(registry.snapshot("person", null)).isEmpty(); - - for(String channel : List.of("table:person", "record:person:7")) - { - RealtimeMessage last = events.stream().filter(event -> event.getChannel().equals(channel) && event.getEvent().equals("presence.changed")).reduce((first, second) -> second).orElseThrow(); - assertThat(last.getData()).containsKey("viewers"); - assertThat((List) last.getData().get("viewers")).isEmpty(); - - /////////////////////////////////////////////////////////////////////// - // the realtime hub copies the data map and writes it with toJson, // - // whose NON_EMPTY inclusion used to drop the empty viewers list // - /////////////////////////////////////////////////////////////////////// - String json = JsonUtils.toJson(new LinkedHashMap<>(last.getData())); - assertThat(JsonUtils.toJSONObject(json).getJSONArray("viewers").length()).isZero(); - } - - String occupied = JsonUtils.toJson(events.stream().filter(event -> event.getChannel().equals("record:person:7")).findFirst().orElseThrow().getData()); - assertThat(JsonUtils.toJSONObject(occupied).getJSONArray("viewers").getJSONObject(0).getString("fieldName")).isEqualTo("name"); - - var snapshot = new CollaborationPresenceSchemas.CollaborationPresenceSnapshotV1("person", null, registry.snapshot("person", null)); - assertThat(JsonUtils.toJSONObject(JsonUtils.toJson(snapshot)).getJSONArray("viewers").length()).isZero(); - } - } - - /******************************************************************************* - ** Two leases of one user merge into one viewer with connections = 2; per - ** connection they are two viewers with opaque ids, modes, fields and starts. - ** Counts keep counting users; connectionCounts counts leases (#54). - *******************************************************************************/ - @Test - void perConnectionViewersAndCounts() throws Exception - { - var clock = new MutableClock(); - try(var bus = new InMemoryRealtimeBus(); var first = new PresenceRegistry(bus, clock, PresenceRegistry.MAX_ENTRIES); - var second = new PresenceRegistry(bus, clock, PresenceRegistry.MAX_ENTRIES)) - { - first.heartbeat("admin", "Admin", "terminal-1", "session-a", "person", "7", "VIEWING", null); - clock.advance(1000); - second.heartbeat("admin", "Admin", "terminal-2", "session-b", "person", "7", "EDITING", "name"); - first.heartbeat("bob", "Bob", "tab", "session-c", "person", "7", "VIEWING", null); - - assertThat(first.snapshot("person", "7")).containsExactly( - new PresenceRegistry.Viewer("admin", "Admin", "EDITING", "name", 2), - new PresenceRegistry.Viewer("bob", "Bob", "VIEWING", null, 1)); - - List connections = second.connections("person", "7"); - assertThat(connections).hasSize(3); - assertThat(connections.subList(0, 2)).extracting(PresenceRegistry.Viewer::userId, PresenceRegistry.Viewer::mode, PresenceRegistry.Viewer::fieldName, PresenceRegistry.Viewer::connections) - .containsExactly(org.assertj.core.groups.Tuple.tuple("admin", "VIEWING", null, 1), org.assertj.core.groups.Tuple.tuple("admin", "EDITING", "name", 1)); - assertThat(connections.getFirst().connectionId()).isEqualTo(PresenceRegistry.opaqueConnectionId("admin", "terminal-1")).startsWith("c").hasSize(17); - assertThat(connections.getFirst().since()).isEqualTo("2026-10-13T12:00:00Z"); - assertThat(connections.get(1).since()).isEqualTo("2026-10-13T12:00:01Z"); - - //////////////////////////////////////////////////////////////////// - // renewals keep the lease start; the session uuid replicates // - //////////////////////////////////////////////////////////////////// - clock.advance(20000); - first.heartbeat("admin", "Admin", "terminal-1", "session-a", "person", "7", "VIEWING", null); - assertThat(second.connections("person", "7").getFirst().since()).isEqualTo("2026-10-13T12:00:00Z"); - assertThat(second.connections("person", "7")).extracting(PresenceRegistry.Viewer::sessionId) - .containsExactly(PresenceRegistry.opaqueSessionId("session-a"), PresenceRegistry.opaqueSessionId("session-b"), PresenceRegistry.opaqueSessionId("session-c")); - assertThat(PresenceRegistry.opaqueSessionId("session-a")).startsWith("s").hasSize(17).isNotEqualTo(PresenceRegistry.opaqueSessionId("session-b")); - - //////////////////////////////////////////////////////////////////// - // table snapshots and counts: users vs leases // - //////////////////////////////////////////////////////////////////// - assertThat(first.snapshot("person", null)).extracting(PresenceRegistry.Viewer::connections).containsExactly(2, 1); - assertThat(first.connections("person", null)).extracting(PresenceRegistry.Viewer::fieldName).containsOnlyNulls(); - assertThat(first.counts()).containsExactly(java.util.Map.entry("person", 2)); - assertThat(first.connectionCounts()).containsExactly(java.util.Map.entry("person", 3)); - - second.leave("admin", "terminal-2", "person", "7"); - assertThat(first.snapshot("person", "7").getFirst()).isEqualTo(new PresenceRegistry.Viewer("admin", "Admin", "VIEWING", null, 1)); - } - } - - /******************************************************************************* - ** Session uuids are bearer credentials: no bus message (internal replication - ** or client event) and no snapshot carries one, only the opaque session id. - *******************************************************************************/ - @Test - void rawSessionUuidIsNeverPublished() throws Exception - { - var clock = new MutableClock(); - String sessionUuid = "7d1c2a9e-secret-session-uuid"; - try(var bus = new InMemoryRealtimeBus(); var first = new PresenceRegistry(bus, clock, PresenceRegistry.MAX_ENTRIES); - var second = new PresenceRegistry(bus, clock, PresenceRegistry.MAX_ENTRIES)) - { - List events = new CopyOnWriteArrayList<>(); - bus.subscribe(events::add); - first.heartbeat("admin", "Admin", "terminal-1", sessionUuid, "person", "7", "VIEWING", null); - first.heartbeat("admin", "Admin", "terminal-1", sessionUuid, "person", "7", "EDITING", "name"); - first.leave("admin", "terminal-1", "person", "7"); - first.heartbeat("admin", "Admin", "terminal-1", sessionUuid, "person", null, "VIEWING", null); - assertThat(events).isNotEmpty().anySatisfy(event -> assertThat(event.getChannel()).isEqualTo(PresenceRegistry.INTERNAL_CHANNEL)); - for(RealtimeMessage event : events) - { - assertThat(JsonUtils.toJson(new LinkedHashMap<>(event.getData()))).doesNotContain(sessionUuid); - } - assertThat(events).filteredOn(event -> event.getChannel().equals(PresenceRegistry.INTERNAL_CHANNEL) && !Boolean.TRUE.equals(event.getData().get("leave"))) - .allSatisfy(event -> assertThat(event.getData().get("sessionId")).isEqualTo(PresenceRegistry.opaqueSessionId(sessionUuid))); - var snapshot = new CollaborationPresenceSchemas.CollaborationPresenceSnapshotV1("person", null, second.snapshot("person", null), 1, second.connections("person", null)); - String json = JsonUtils.toJson(snapshot); - assertThat(json).doesNotContain(sessionUuid).contains(PresenceRegistry.opaqueSessionId(sessionUuid)); - } - } - - /******************************************************************************* - ** presence.changed carries connections per merged viewer, a lease count and - ** one lease entry per connection; a second terminal of one user is a change. - *******************************************************************************/ - @Test - void changedEventCarriesConnections() throws Exception - { - var clock = new MutableClock(); - try(var bus = new InMemoryRealtimeBus(); var registry = new PresenceRegistry(bus, clock, PresenceRegistry.MAX_ENTRIES)) - { - List events = new CopyOnWriteArrayList<>(); - bus.subscribe(events::add); - registry.heartbeat("admin", "Admin", "terminal-1", "person", "7", "VIEWING", null); - int before = recordEvents(events).size(); - registry.heartbeat("admin", "Admin", "terminal-2", "person", "7", "VIEWING", null); - List after = recordEvents(events); - assertThat(after).hasSize(before + 1); - - var json = JsonUtils.toJSONObject(JsonUtils.toJson(new LinkedHashMap<>(after.getLast().getData()))); - assertThat(json.getJSONArray("viewers").length()).isEqualTo(1); - assertThat(json.getJSONArray("viewers").getJSONObject(0).getInt("connections")).isEqualTo(2); - assertThat(json.getJSONArray("viewers").getJSONObject(0).has("connectionId")).isFalse(); - assertThat(json.getInt("leaseCount")).isEqualTo(2); - assertThat(json.getJSONArray("leases").length()).isEqualTo(2); - var lease = json.getJSONArray("leases").getJSONObject(1); - assertThat(lease.getString("userId")).isEqualTo("admin"); - assertThat(lease.getString("mode")).isEqualTo("VIEWING"); - assertThat(lease.getInt("connections")).isEqualTo(1); - assertThat(lease.getString("connectionId")).startsWith("c"); - assertThat(lease.getString("since")).isEqualTo("2026-10-13T12:00:00Z"); - assertThat(json.toString()).doesNotContain("terminal-"); - - registry.leave("admin", "terminal-1", "person", "7"); - registry.leave("admin", "terminal-2", "person", "7"); - var last = JsonUtils.toJSONObject(JsonUtils.toJson(new LinkedHashMap<>(recordEvents(events).getLast().getData()))); - assertThat(last.getInt("leaseCount")).isZero(); - assertThat(last.getJSONArray("leases").length()).isZero(); - } - } - - /******************************************************************************* - ** Record-channel presence.changed events so far. - *******************************************************************************/ - private static List recordEvents(List events) - { - return (events.stream().filter(event -> event.getChannel().equals("record:person:7") && event.getEvent().equals("presence.changed")).toList()); - } - /******************************************************************************* ** At capacity, existing leases renew while new targets are refused on both pods. *******************************************************************************/ @@ -291,15 +274,7 @@ class PresenceTest extends PersonalServiceFixture void accessFeatureAndSnapshot() throws Exception { var service = new PresenceService(); - QContext.getQSession().setUuid("session-uuid-alice"); assertThat(service.heartbeat("one", "document", "01", "EDITING", "title")).hasSize(1); - assertThat(service.connections("document", "1")).singleElement().satisfies(viewer -> - { - assertThat(viewer.connectionId()).isEqualTo(PresenceRegistry.opaqueConnectionId("alice", "one")); - assertThat(viewer.sessionId()).isEqualTo(PresenceRegistry.opaqueSessionId("session-uuid-alice")); - assertThat(JsonUtils.toJson(new CollaborationPresenceSchemas.CollaborationPresenceSnapshotV1("document", "1", service.snapshot("document", "1"), 1, service.connections("document", "1")))) - .doesNotContain("session-uuid-alice"); - }); assertThat(service.snapshot("document", "1")).extracting(PresenceRegistry.Viewer::userId).containsExactly("alice"); assertThat(service.snapshot("document", null)).hasSize(1); assertThatThrownBy(() -> service.heartbeat("one", "document", "3", "VIEWING", null)).isInstanceOf(QNotFoundException.class); @@ -319,7 +294,6 @@ class PresenceTest extends PersonalServiceFixture admin(); assertThat(CollaborationHome.presence().getRows()).extracting(row -> row.get("table")).containsExactly("Documents"); assertThat(CollaborationHome.presence().getRows().getFirst().get("viewers")).isEqualTo(1); - assertThat(CollaborationHome.presence().getRows().getFirst().get("connections")).isEqualTo(1); login("alice"); assertThat(CollaborationHome.presence().getRows()).isEmpty(); assertThat(CollaborationHome.presence().getNoRowsFoundHTML()).isEqualTo(CollaborationHome.PRESENCE_ADMIN_ONLY); diff --git a/src/test/java/com/kingsrook/qbits/collaboration/RedisFixtures.java b/src/test/java/com/kingsrook/qbits/collaboration/RedisFixtures.java new file mode 100644 index 0000000..5ee775c --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/RedisFixtures.java @@ -0,0 +1,80 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.collaboration; + +import java.util.List; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.utility.DockerImageName; + +/******************************************************************************* + ** Disposable Redis servers for presence tests. + ** + ** Docker Desktop publishes an unqualified random port only on the Windows side; + ** a Linux build container then reaches it through host.docker.internal, which + ** was measured stalling established connections for 300 ms to 2 s (server saw + ** no bytes during the stall; the Docker bridge and the VM-local published port + ** showed none). Setting "fixture.redis.vmLocalPort=true" asks Docker for an + ** explicit host port 0 (same ephemeral allocation on a normal Docker host), + ** which Docker Desktop binds inside the VM. The client, server, scripts and + ** every assertion are unchanged. + *******************************************************************************/ +final class RedisFixtures +{ + static final String VM_LOCAL_PORT_PROPERTY = "fixture.redis.vmLocalPort"; + + + + /*************************************************************************** + ** Create (not start) the Redis fixture used by all presence tests. + ***************************************************************************/ + static GenericContainer redis() + { + GenericContainer redis = new GenericContainer<>(DockerImageName.parse("redis:7.4-alpine")).withExposedPorts(6379); + if(Boolean.getBoolean(VM_LOCAL_PORT_PROPERTY)) + { + redis.setPortBindings(List.of("0:6379")); + } + return (redis); + } + + + + /*************************************************************************** + ** Host this JVM must dial: the sibling's bridge address only when this JVM is in a + ** container the same daemon reports on its default bridge, otherwise the + ** published endpoint at the host Testcontainers reports (see FixtureEndpoint). + ***************************************************************************/ + static String host(GenericContainer redis) + { + return (FixtureEndpoint.of(redis, 6379).host()); + } + + + + /*************************************************************************** + ** Port matching {@link #host(GenericContainer)}. + ***************************************************************************/ + static int port(GenericContainer redis) + { + return (FixtureEndpoint.of(redis, 6379).port()); + } + +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/RedisFixturesTest.java b/src/test/java/com/kingsrook/qbits/collaboration/RedisFixturesTest.java new file mode 100644 index 0000000..cf1fb1b --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/RedisFixturesTest.java @@ -0,0 +1,59 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.collaboration; + +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; + +/******************************************************************************* + ** The VM-local published-port request must reach the container definition. + *******************************************************************************/ +class RedisFixturesTest +{ + + /*************************************************************************** + ** Default is Docker's ordinary random port; the property asks for host port 0. + ***************************************************************************/ + @Test + void explicitHostPortIsRequestedOnlyWhenEnabled() + { + String previous = System.getProperty(RedisFixtures.VM_LOCAL_PORT_PROPERTY); + try + { + System.clearProperty(RedisFixtures.VM_LOCAL_PORT_PROPERTY); + assertThat(RedisFixtures.redis().getPortBindings()).isEmpty(); + System.setProperty(RedisFixtures.VM_LOCAL_PORT_PROPERTY, "true"); + assertThat(RedisFixtures.redis().getPortBindings()).containsExactly("0:6379/tcp"); + } + finally + { + if(previous == null) + { + System.clearProperty(RedisFixtures.VM_LOCAL_PORT_PROPERTY); + } + else + { + System.setProperty(RedisFixtures.VM_LOCAL_PORT_PROPERTY, previous); + } + } + } + +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceRecoveryCostProbe.java b/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceRecoveryCostProbe.java new file mode 100644 index 0000000..5fd0392 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceRecoveryCostProbe.java @@ -0,0 +1,150 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration; + +import java.time.Duration; +import java.util.Arrays; +import java.util.LinkedHashMap; +import java.util.Map; +import com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException; +import com.kingsrook.qbits.collaboration.presence.RedisPresenceStore; +import com.kingsrook.qbits.redis.RedisConnection; +import com.kingsrook.qbits.redis.api.RedisNamespace; +import com.kingsrook.qqq.backend.core.utils.JsonUtils; +import org.testcontainers.containers.GenericContainer; +import redis.clients.jedis.DefaultJedisClientConfig; +import redis.clients.jedis.HostAndPort; + +/******************************************************************************* + ** Explicit local diagnostic, not a CI latency gate or deployment capacity claim. + *******************************************************************************/ +public class RedisPresenceRecoveryCostProbe +{ + /*************************************************************************** + ** Compare issued-lease renewal cost during real recovery and after readiness. + ***************************************************************************/ + public static void main(String[] args) throws Exception + { + int count = args.length == 0 ? 1000 : Integer.parseInt(args[0]); + if(count < 10 || count > 10000 || count % 10 != 0) + { + throw (new IllegalArgumentException("Use a multiple of ten between 10 and 10000")); + } + try(var redis = new GenericContainer<>("redis:7.4-alpine").withExposedPorts(6379)) + { + redis.start(); + String host = RedisFixtures.host(redis); + try(var connection = new RedisConnection(new HostAndPort(host, RedisFixtures.port(redis)), + DefaultJedisClientConfig.builder().socketTimeoutMillis(5000).build(), 1, Duration.ofSeconds(1))) + { + var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("cost", "local", "tenant", "presence"), Duration.ofSeconds(45), count); + String[] leases = new String[count]; + long[] sequences = new long[count]; + long start = System.nanoTime(); + for(int i = 0; i < count; i++) + { + leases[i] = store.openLease("user" + i, "Person " + i, "document", Integer.toString(i / 10)); + renew(store, leases, sequences, i); + } + double populationMillis = (System.nanoTime() - start) / 1_000_000.0; + requireRecovering(store); + Map cold = sample(store, leases, sequences); + requireRecovering(store); + long deadline = System.nanoTime() + Duration.ofSeconds(60).toNanos(); + while(true) + { + for(int i = 0; i < count; i++) + { + renew(store, leases, sequences, i); + } + try + { + if(store.snapshot("document", null).size() != count) + { + throw (new AssertionError("Population did not survive recovery")); + } + break; + } + catch(PresenceRecoveringException expected) + { + if(System.nanoTime() >= deadline) + { + throw (new AssertionError("Recovery exceeded deadline")); + } + Thread.sleep(5000); + } + } + var result = new LinkedHashMap(); + result.put("entries", count); + result.put("samplesPerPhase", 200); + result.put("populationMillis", populationMillis); + result.put("recoveryRenewMillis", cold); + result.put("readyRenewMillis", sample(store, leases, sequences)); + result.put("scope", "One JVM, one connection, loopback Redis, no TLS/HTTP; diagnostic only"); + System.out.println(JsonUtils.toJson(result)); + } + } + } + + /*************************************************************************** + ** Sample the same issued leases with increasing sequence numbers. + ***************************************************************************/ + private static Map sample(RedisPresenceStore store, String[] leases, long[] sequences) + { + double[] times = new double[200]; + for(int i = 0; i < times.length; i++) + { + long before = System.nanoTime(); + renew(store, leases, sequences, i % leases.length); + times[i] = (System.nanoTime() - before) / 1_000_000.0; + } + Arrays.sort(times); + return (Map.of("p50", times[99], "p95", times[189], "p99", times[197], "max", times[199])); + } + + /*************************************************************************** + ** Renewal must not silently fail while measuring its elapsed time. + ***************************************************************************/ + private static void renew(RedisPresenceStore store, String[] leases, long[] sequences, int i) + { + if(!store.renewLease("user" + i, "Person " + i, leases[i], "document", Integer.toString(i / 10), "VIEWING", null, ++sequences[i])) + { + throw (new AssertionError("Renewal refused")); + } + } + + /*************************************************************************** + ** Do not label a measurement cold if the actual recovery window has ended. + ***************************************************************************/ + private static void requireRecovering(RedisPresenceStore store) + { + try + { + store.snapshot("document", "0"); + } + catch(PresenceRecoveringException expected) + { + return; + } + throw (new AssertionError("Recovery finished before cold measurement")); + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceRecoveryTest.java b/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceRecoveryTest.java new file mode 100644 index 0000000..7143023 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceRecoveryTest.java @@ -0,0 +1,325 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration; + + +import java.time.Duration; +import com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException; +import com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException; +import com.kingsrook.qbits.collaboration.presence.RedisPresenceStore; +import com.kingsrook.qbits.collaboration.presence.RedisPresenceAudience; +import com.kingsrook.qbits.collaboration.presence.RedisPresenceConfig; +import com.kingsrook.qbits.collaboration.presence.PresenceScopeProvider; +import com.kingsrook.qbits.redis.RedisConnection; +import com.kingsrook.qbits.redis.api.RedisNamespace; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import org.testcontainers.containers.GenericContainer; +import redis.clients.jedis.DefaultJedisClientConfig; +import redis.clients.jedis.HostAndPort; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Loss recovery must be shared across independent clients, not node-local memory. + *******************************************************************************/ +class RedisPresenceRecoveryTest extends PersonalServiceFixture +{ + /*************************************************************************** + ** Peer reads cannot prolong cold recovery; normal expiry stays known-empty. + ***************************************************************************/ + @Test + void coldRecoveryDeadlineAndIdleKnowledgeAreShared() throws Exception + { + try(var redis = RedisFixtures.redis()) + { + redis.start(); + var address = new HostAndPort(RedisFixtures.host(redis), RedisFixtures.port(redis)); + var config = DefaultJedisClientConfig.builder().build(); + try(var first = new RedisConnection(address, config, 2, Duration.ofMillis(500)); + var second = new RedisConnection(address, config, 2, Duration.ofMillis(500))) + { + var namespace = new RedisNamespace("test", "cold-idle", "tenant", "presence"); + var a = new RedisPresenceStore(first.scripts(), namespace, Duration.ofSeconds(2), 10); + var b = new RedisPresenceStore(second.scripts(), namespace, Duration.ofSeconds(2), 10); + assertThatThrownBy(() -> a.snapshot("document", "1")).isInstanceOf(PresenceRecoveringException.class); + String guardKey = first.client().keys("*:recovery").iterator().next(); + String coldGuard = first.client().get(guardKey); + assertThatThrownBy(() -> b.snapshot("document", null)).isInstanceOf(PresenceRecoveringException.class); + assertThat(first.client().get(guardKey)).isEqualTo(coldGuard); + long deadline = System.nanoTime() + Duration.ofSeconds(6).toNanos(); + while(true) + { + try + { + assertThat(b.snapshot("document", "1")).isEmpty(); + break; + } + catch(PresenceRecoveringException recovering) + { + assertThat(System.nanoTime()).isLessThan(deadline); + assertThat(first.client().get(guardKey)).isEqualTo(coldGuard); + Thread.sleep(50); + } + } + assertThat(a.snapshot("document", null)).isEmpty(); + assertThat(first.client().get(guardKey)).isEqualTo(coldGuard); + assertThat(first.client().pttl(guardKey)).isEqualTo(-1L); + + String lease = a.openLease("alice", "Alice", "document", "1"); + awaitVisible(b, lease); + String activeGuard = first.client().get(guardKey); + deadline = System.nanoTime() + Duration.ofSeconds(6).toNanos(); + while(first.client().keys("*").size() > 1) + { + assertThat(System.nanoTime()).isLessThan(deadline); + Thread.sleep(50); + } + assertThat(first.client().keys("*")).containsExactly(guardKey); + assertThat(a.snapshot("document", "1")).isEmpty(); + assertThat(b.snapshot("document", null)).isEmpty(); + assertThat(first.client().get(guardKey)).isEqualTo(activeGuard); + assertThatThrownBy(() -> b.renewLease("alice", "Alice", lease, "document", "1", "VIEWING", null, 100L)) + .isInstanceOf(PresenceLeaseExpiredException.class); + } + } + } + + + /*************************************************************************** + ** Keep a real lease alive while waiting for bounded initial recovery. + ***************************************************************************/ + private void awaitVisible(RedisPresenceStore store, String lease) throws Exception + { + long deadline = System.nanoTime() + Duration.ofSeconds(8).toNanos(); + long sequence = 1; + while(System.nanoTime() < deadline) + { + store.renewLease("alice", "Alice", lease, "document", "1", "EDITING", null, sequence++); + try + { + assertThat(store.snapshot("document", "1")).extracting("userId").containsExactly("alice"); + return; + } + catch(IllegalStateException recovering) + { + assertThat(recovering).hasMessageContaining("recover"); + Thread.sleep(50); + } + } + throw (new AssertionError("Presence did not finish bounded recovery")); + } + + + /*************************************************************************** + ** A cleared dataset is unknown, not known-empty; live clients can repopulate. + ***************************************************************************/ + @ParameterizedTest + @ValueSource(booleans = { false, true }) + void totalLossRequiresSharedRecoveryBeforeKnownEmpty(boolean retainGuard) throws Exception + { + try(GenericContainer redis = RedisFixtures.redis()) + { + redis.start(); + var address = new HostAndPort(RedisFixtures.host(redis), RedisFixtures.port(redis)); + var clientConfig = DefaultJedisClientConfig.builder().build(); + try(var first = new RedisConnection(address, clientConfig, 2, Duration.ofMillis(500)); + var second = new RedisConnection(address, clientConfig, 2, Duration.ofMillis(500))) + { + var namespace = new RedisNamespace("test", "app", "tenant-a", "presence"); + var a = new RedisPresenceStore(first.scripts(), namespace, Duration.ofSeconds(2), 10); + var b = new RedisPresenceStore(second.scripts(), namespace, Duration.ofSeconds(2), 10); + String oldLease = a.openLease("alice", "Alice", "document", "1"); + awaitVisible(b, oldLease); + if(retainGuard) + { + var projections = first.client().keys("*").stream().filter(key -> !key.endsWith(":recovery")).toArray(String[]::new); + first.client().del(projections); + } + else + { + first.client().flushDB(); + } + assertThatThrownBy(() -> b.snapshot("document", "1")) + .isInstanceOf(IllegalStateException.class).hasMessageContaining("recover"); + assertThatThrownBy(() -> a.snapshot("document", null)) + .isInstanceOf(IllegalStateException.class).hasMessageContaining("recover"); + assertThatThrownBy(() -> a.renewLease("alice", "Alice", oldLease, "document", "1", "EDITING", null, 999L)) + .isInstanceOf(PresenceLeaseExpiredException.class); + String replacement = b.openLease("alice", "Alice", "document", "1"); + assertThat(replacement).isNotNull().isNotEqualTo(oldLease); + awaitVisible(a, replacement); + assertThat(b.snapshot("document", null)).extracting("userId").containsExactly("alice"); + assertThat(b.snapshot("document", "unoccupied")).isEmpty(); + } + } + } + + + /*************************************************************************** + ** Guard loss cleans the table once and untouched record rooms on demand. + ***************************************************************************/ + @Test + void missingGuardRetiresUnvisitedRecordsWithoutRemovingNewLeases() throws Exception + { + try(var redis = RedisFixtures.redis()) + { + redis.start(); + try(var connection = new RedisConnection(new HostAndPort(RedisFixtures.host(redis), RedisFixtures.port(redis)), + DefaultJedisClientConfig.builder().build(), 2, Duration.ofMillis(500))) + { + var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "guard-loss", "tenant", "presence"), Duration.ofSeconds(2), 2); + String oldAlice = store.openLease("alice", "Alice", "document", "1"); + String oldBob = store.openLease("bob", "Bob", "document", "2"); + connection.client().del(connection.client().keys("*:recovery").toArray(String[]::new)); + assertThatThrownBy(() -> store.snapshot("document", null)).isInstanceOf(PresenceRecoveringException.class); + String alice = store.openLease("alice", "Alice", "document", "1"); + String bob = store.openLease("bob", "Bob", "document", "2"); + assertThatThrownBy(() -> store.renewLease("alice", "Alice", oldAlice, "document", "1", "VIEWING", null, 99L)) + .isInstanceOf(PresenceLeaseExpiredException.class); + assertThatThrownBy(() -> store.renewLease("bob", "Bob", oldBob, "document", "2", "VIEWING", null, 99L)) + .isInstanceOf(PresenceLeaseExpiredException.class); + assertThat(store.renewLease("bob", "Bob", bob, "document", "2", "VIEWING", null, 1L)).isTrue(); + awaitVisible(store, alice); + assertThat(store.snapshot("document", "1")).extracting("userId").containsExactly("alice"); + } + } + } + + + /*************************************************************************** + ** Restore must happen in maintenance, followed by a fresh host-owned generation. + ** Replaying both old guard and lease bytes cannot authorize the new deployment. + ***************************************************************************/ + @Test + void restoredImageCannotAuthorizeRotatedHostGeneration() throws Exception + { + try(GenericContainer redis = RedisFixtures.redis()) + { + redis.start(); + var address = new HostAndPort(RedisFixtures.host(redis), RedisFixtures.port(redis)); + var clientConfig = DefaultJedisClientConfig.builder().build(); + try(var first = new RedisConnection(address, clientConfig, 2, Duration.ofMillis(500)); + var second = new RedisConnection(address, clientConfig, 2, Duration.ofMillis(500))) + { + var oldNamespace = RedisPresenceAudience.namespace(recoveryConfig("before-restore")); + var oldStore = new RedisPresenceStore(first.scripts(), oldNamespace, Duration.ofSeconds(2), 10); + String oldLease = oldStore.openLease("alice", "Alice", "document", "1"); + awaitVisible(oldStore, oldLease); + var image = new java.util.HashMap(); + for(String key : first.client().keys("*")) + { + image.put(key, first.client().dump(key)); + } + oldStore.leave("alice", oldLease, "document", "1", 999L); + first.client().flushDB(); + for(var entry : image.entrySet()) + { + first.client().restore(entry.getKey(), 0, entry.getValue()); + } + var restoredNamespace = RedisPresenceAudience.namespace(recoveryConfig("after-restore")); + var a = new RedisPresenceStore(first.scripts(), restoredNamespace, Duration.ofSeconds(2), 10); + var b = new RedisPresenceStore(second.scripts(), restoredNamespace, Duration.ofSeconds(2), 10); + assertThatThrownBy(() -> a.renewLease("alice", "Alice", oldLease, "document", "1", "EDITING", null, 1000L)) + .isInstanceOf(PresenceLeaseExpiredException.class); + assertThatThrownBy(() -> b.snapshot("document", "1")) + .isInstanceOf(IllegalStateException.class).hasMessageContaining("recover"); + String fresh = b.openLease("alice", "Alice", "document", "1"); + awaitVisible(a, fresh); + assertThat(b.snapshot("document", "1")).extracting("userId").containsExactly("alice"); + assertThatThrownBy(() -> b.renewLease("alice", "Alice", oldLease, "document", "1", "EDITING", null, 1001L)) + .isInstanceOf(PresenceLeaseExpiredException.class); + } + } + } + + + /*************************************************************************** + ** Generation is bounded deployment configuration, never browser input. + ***************************************************************************/ + @Test + void validatesHostGeneration() + { + var errors = new java.util.ArrayList(); + recoveryConfig("valid-20261026").validate(errors); + assertThat(errors).isEmpty(); + for(String invalid : java.util.Arrays.asList(null, "", " ", "x".repeat(65), "invalid:namespace")) + { + errors.clear(); + recoveryConfig(invalid).validate(errors); + assertThat(errors).anySatisfy(error -> assertThat(error).contains("recovery generation")); + } + } + + + /*************************************************************************** + ** Recovery readiness must not depend on which node's lease policy is shorter. + ***************************************************************************/ + @Test + void refusesInconsistentNodeLeaseLifetimes() throws Exception + { + try(GenericContainer redis = RedisFixtures.redis()) + { + redis.start(); + try(var connection = new RedisConnection(new HostAndPort(RedisFixtures.host(redis), RedisFixtures.port(redis)), + DefaultJedisClientConfig.builder().build(), 2, Duration.ofMillis(500))) + { + var namespace = new RedisNamespace("test", "app", "ttl-policy", "presence"); + var first = new RedisPresenceStore(connection.scripts(), namespace, Duration.ofSeconds(2), 10); + var second = new RedisPresenceStore(connection.scripts(), namespace, Duration.ofSeconds(3), 10); + assertThatThrownBy(() -> first.snapshot("document", null)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException.class); + assertThatThrownBy(() -> second.snapshot("document", null)) + .isInstanceOf(redis.clients.jedis.exceptions.JedisException.class).hasMessageContaining("lifetime differs"); + } + } + } + + + /*************************************************************************** + ** Use the real host-configured storage/audience namespace derivation. + ***************************************************************************/ + private RedisPresenceConfig recoveryConfig(String generation) + { + return (new RedisPresenceConfig().withConnectionName("presence").withDeployment("test").withApplication("app") + .withScopeProvider(new QCodeReference(FixtureScope.class)).withRecoveryGeneration(generation)); + } + + + /*************************************************************************** + ** Tenant scope belongs to the host; this test uses one fixed authenticated scope. + ***************************************************************************/ + public static class FixtureScope implements PresenceScopeProvider + { + /*************************************************************************** + ** Return the fixture's host-selected tenant, without client-supplied fields. + ***************************************************************************/ + @Override + public String tenantScope() + { + return ("tenant-a"); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceServiceTest.java b/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceServiceTest.java new file mode 100644 index 0000000..acbef84 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceServiceTest.java @@ -0,0 +1,400 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration; + +import com.kingsrook.qbits.collaboration.presence.PresenceScopeProvider; +import com.kingsrook.qbits.collaboration.presence.PresenceService; +import com.kingsrook.qbits.collaboration.presence.RedisPresenceBackend; +import com.kingsrook.qbits.collaboration.presence.RedisPresenceConfig; +import com.kingsrook.qbits.redis.RedisConnectionConfig; +import com.kingsrook.qbits.redis.RedisQBitConfig; +import com.kingsrook.qbits.redis.RedisQBitProducer; +import com.kingsrook.qbits.redis.RedisRuntimeService; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QBadRequestException; +import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException; +import com.kingsrook.qqq.backend.core.exceptions.QServiceUnavailableException; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/******************************************************************************* + ** Exercise the authenticated service boundary against a real Redis runtime. + *******************************************************************************/ +class RedisPresenceServiceTest extends PersonalServiceFixture +{ + /*************************************************************************** + ** Separate application runtimes share leases and deliver hints through Redis. + ***************************************************************************/ + @Test + void twoNodesShareSixViewersAndTwoEditors() throws Exception + { + var second = new PersonalServiceFixture() { }; + second.setUp(); + var firstRuntime = new RedisRuntimeService(); + var secondRuntime = new RedisRuntimeService(); + var firstTransport = new com.kingsrook.qbits.redis.RedisRealtimeRuntimeService(); + var secondTransport = new com.kingsrook.qbits.redis.RedisRealtimeRuntimeService(); + try(var redis = RedisFixtures.redis()) + { + redis.start(); + var nodes = java.util.List.of(instance, second.instance); + var configs = java.util.List.of(config, second.config); + for(int i = 0; i < nodes.size(); i++) + { + com.kingsrook.qqq.backend.core.realtime.RealtimeBus.close(nodes.get(i)); + new RedisQBitProducer().withQBitConfig(new RedisQBitConfig().withConnection("presence", + new RedisConnectionConfig().withHost(RedisFixtures.host(redis)).withPort(RedisFixtures.port(redis)).withTls(false)) + .withRealtime("presence", "test", "multi-node")).produce(nodes.get(i)); + configs.get(i).withRedisPresence(new RedisPresenceConfig().withConnectionName("presence").withDeployment("test") + .withApplication("multi-node").withScopeProvider(new QCodeReference(TenantScope.class))); + } + firstRuntime.start(instance); + secondRuntime.start(second.instance); + warmScopes(instance); + firstTransport.start(instance); + secondTransport.start(second.instance); + org.junit.jupiter.api.Assertions.assertTimeoutPreemptively(java.time.Duration.ofSeconds(5), () -> + { + for(var node : nodes) + { + while(!((com.kingsrook.qbits.redis.RedisRealtimeBus) com.kingsrook.qqq.backend.core.realtime.RealtimeBus.getIfStarted(node).orElseThrow()).isListening()) + { + Thread.sleep(10); + } + } + }); + try(var hub = new com.kingsrook.qqq.middleware.javalin.realtime.RealtimeHub(second.instance)) + { + var observer = session("observer"); + QContext.init(second.instance, observer); + var stream = hub.connect(observer, null); + hub.subscriptions(observer, stream.getConnectionId(), java.util.List.of("record:document:1"), java.util.List.of()); + var otherTenant = session("other-tenant"); + otherTenant.clearSecurityKeyValues(); + otherTenant.withSecurityKeyValue("tenant", "b"); + QContext.init(second.instance, otherTenant); + var otherStream = hub.connect(otherTenant, null); + hub.subscriptions(otherTenant, otherStream.getConnectionId(), java.util.List.of("table:document"), java.util.List.of()); + var service = new PresenceService(); + var leases = new java.util.ArrayList(); + for(int i = 0; i < 6; i++) + { + QContext.init(nodes.get((i + 1) % 2), session("user" + i)); + String lease = service.openLease("document", "1"); + leases.add(lease); + assertThat(service.snapshot("document", "1")).hasSize(i); + QContext.init(nodes.get(i % 2), session("user" + i)); + service.renewLease(lease, "document", "1", i < 2 ? "EDITING" : "VIEWING", i < 2 ? "title" : null, 1L); + if(i == 0) + { + // Node one activates a lease opened on node two; its hint crosses nodes. + org.junit.jupiter.api.Assertions.assertTimeoutPreemptively(java.time.Duration.ofSeconds(5), () -> + { + while(true) + { + var frame = stream.poll(200); + if(frame != null && "presence.invalidated".equals(frame.event())) + { + assertThat(frame.data()).containsOnlyKeys("table", "pk").containsEntry("pk", "1"); + break; + } + } + }); + } + } + for(var node : nodes) + { + QContext.init(node, observer); + var viewers = service.snapshot("document", "1"); + assertThat(viewers).hasSize(6); + assertThat(viewers.stream().filter(viewer -> "EDITING".equals(viewer.mode())).count()).isEqualTo(2); + assertThat(service.snapshot("document", null)).hasSize(6); + } + // A later unscoped marker proves the other tenant's stream processed + // the earlier hints, instead of passing on an arbitrary short timeout. + com.kingsrook.qqq.backend.core.realtime.RealtimeBus.getIfStarted(instance).orElseThrow().publish( + new com.kingsrook.qqq.backend.core.realtime.RealtimeMessage().withChannel("table:document") + .withEvent("test.barrier").withData(java.util.Map.of())); + org.junit.jupiter.api.Assertions.assertTimeoutPreemptively(java.time.Duration.ofSeconds(5), () -> + { + while(true) + { + var frame = otherStream.poll(200); + if(frame != null) + { + assertThat(frame.event()).isNotEqualTo("presence.invalidated"); + if("test.barrier".equals(frame.event())) + { + break; + } + } + } + }); + for(var node : nodes) + { + QContext.init(node, otherTenant); + assertThat(service.snapshot("document", null)).isEmpty(); + assertThat(service.snapshot("document", "3")).isEmpty(); + assertThatThrownBy(() -> service.snapshot("document", "1")).isInstanceOf(QNotFoundException.class); + } + QContext.init(second.instance, session("user0")); + service.leave(leases.getFirst(), "document", "1", 2L); + QContext.init(instance, observer); + assertThat(service.snapshot("document", "1")).hasSize(5); + var firstBus = (com.kingsrook.qbits.redis.RedisRealtimeBus) com.kingsrook.qqq.backend.core.realtime.RealtimeBus.getIfStarted(instance).orElseThrow(); + var secondBus = (com.kingsrook.qbits.redis.RedisRealtimeBus) com.kingsrook.qqq.backend.core.realtime.RealtimeBus.getIfStarted(second.instance).orElseThrow(); + firstBus.publish(new com.kingsrook.qqq.backend.core.realtime.RealtimeMessage().withChannel("record:document:1") + .withEvent("test.drained").withData(java.util.Map.of())); + awaitEvent(stream, "test.drained"); + long reconnects = secondBus.metrics().reconnects(); + assertThat(redis.execInContainer("redis-cli", "CLIENT", "KILL", "TYPE", "pubsub").getExitCode()).isZero(); + assertThat(awaitEvent(stream, "resync").data()).isEmpty(); + org.junit.jupiter.api.Assertions.assertTimeoutPreemptively(java.time.Duration.ofSeconds(5), () -> + { + while(!firstBus.isListening() || !secondBus.isListening() || secondBus.metrics().reconnects() <= reconnects) + { + Thread.sleep(10); + } + }); + // The subscription failure does not erase authoritative presence leases. + QContext.init(second.instance, observer); + assertThat(service.snapshot("document", "1")).hasSize(5); + QContext.init(instance, session("user0")); + assertThatThrownBy(() -> service.renewLease(leases.getFirst(), "document", "1", "EDITING", "title", 3L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + String replacement = service.openLease("document", "1"); + service.renewLease(replacement, "document", "1", "EDITING", "title", 1L); + assertThat(awaitEvent(stream, "presence.invalidated").data()).containsEntry("pk", "1"); + QContext.init(second.instance, observer); + assertThat(service.snapshot("document", "1")).hasSize(6); + } + } + finally + { + firstTransport.stop(); + secondTransport.stop(); + firstRuntime.afterApplicationStop(); + secondRuntime.afterApplicationStop(); + second.tearDown(); + } + } + + /*************************************************************************** + ** Wait for a stream event with a bounded deadline, consuming older frames. + ***************************************************************************/ + private static com.kingsrook.qqq.middleware.javalin.realtime.RealtimeHub.Frame awaitEvent( + com.kingsrook.qqq.middleware.javalin.realtime.RealtimeHub.Attachment stream, String event) + { + return (org.junit.jupiter.api.Assertions.assertTimeoutPreemptively(java.time.Duration.ofSeconds(5), () -> + { + while(true) + { + var frame = stream.poll(200); + if(frame != null && event.equals(frame.event())) + { + return (frame); + } + } + })); + } + + /*************************************************************************** + ** Each simulated browser has an independent authenticated host session. + ***************************************************************************/ + private static com.kingsrook.qqq.backend.core.model.session.QSession session(String user) + { + return (new com.kingsrook.qqq.backend.core.model.session.QSession() + .withUser(new com.kingsrook.qqq.backend.core.model.session.QUser().withIdReference(user)) + .withPermissions("document.read", "document.write").withSecurityKeyValue("tenant", "a")); + } + + /*************************************************************************** + ** Redis preserves permission projection and fails explicitly during outages. + ***************************************************************************/ + @Test + void permissionsIsolationAndUnavailable() throws Exception + { + try(var redis = RedisFixtures.redis()) + { + redis.start(); + new RedisQBitProducer().withQBitConfig(new RedisQBitConfig().withConnection("presence", + new RedisConnectionConfig().withHost(RedisFixtures.host(redis)).withPort(RedisFixtures.port(redis)).withTls(false) + .withConnectionTimeoutMillis(200).withSocketTimeoutMillis(1000))).produce(instance); + config.withRedisPresence(new RedisPresenceConfig().withConnectionName("presence").withDeployment("test") + .withApplication("fixture").withScopeProvider(new QCodeReference(TenantScope.class))); + var runtime = new RedisRuntimeService(); + try + { + runtime.start(instance); + warmScopes(instance); + var service = new PresenceService(); + String aliceLease; + try(var hub = new com.kingsrook.qqq.middleware.javalin.realtime.RealtimeHub(instance)) + { + var viewer = hub.connect(QContext.getQSession(), null); + viewer.poll(100); + hub.subscriptions(QContext.getQSession(), viewer.getConnectionId(), java.util.List.of("table:document", "record:document:1"), java.util.List.of()); + login("other-tenant"); + QContext.getQSession().clearSecurityKeyValues(); + QContext.getQSession().withSecurityKeyValue("tenant", "b"); + var other = hub.connect(QContext.getQSession(), null); + other.poll(100); + hub.subscriptions(QContext.getQSession(), other.getConnectionId(), java.util.List.of("table:document"), java.util.List.of()); + login("alice"); + assertThatThrownBy(() -> service.heartbeat("alice-tab", "document", "01", "EDITING", "title", 1L)) + .isInstanceOf(QBadRequestException.class).hasMessageContaining("server-issued lease"); + aliceLease = service.openLease("document", "01"); + service.renewLease(aliceLease, "document", "01", "EDITING", "title", 1L); + var tableHint = viewer.poll(1000); + var recordHint = viewer.poll(1000); + assertThat(tableHint.event()).isEqualTo("presence.invalidated"); + assertThat(tableHint.data()).containsOnlyKeys("table"); + assertThat(recordHint.data()).containsOnlyKeys("table", "pk").containsEntry("pk", "1"); + assertThat(other.poll(20)).isNull(); + } + admin(); + var adminPresence = com.kingsrook.qbits.collaboration.app.CollaborationHome.presence(); + assertThat(adminPresence.getRows()).hasSize(1); + assertThat(adminPresence.getRows().getFirst()).containsEntry("table", "Documents").containsEntry("viewers", 1); + QContext.getQSession().clearSecurityKeyValues(); + QContext.getQSession().withSecurityKeyValue("tenant", "b"); + assertThat(com.kingsrook.qbits.collaboration.app.CollaborationHome.presence().getRows()).isEmpty(); + login("bob"); + assertThat(com.kingsrook.qbits.collaboration.app.CollaborationHome.presence().getRows()).isEmpty(); + assertThat(service.snapshot("document", "1")).hasSize(1); + assertThat(service.snapshot("document", "1").getFirst().userId()).isEqualTo("alice"); + assertThat(service.snapshot("document", "1").getFirst().fieldName()).isEqualTo("title"); + instance.getTable("document").getField("title").setIsHidden(true); + assertThat(service.snapshot("document", "1").getFirst().fieldName()).isNull(); + String bobLease = service.openLease("document", "1"); + assertThatThrownBy(() -> service.renewLease(bobLease, "document", "1", "EDITING", "title", 1L)) + .isInstanceOf(QBadRequestException.class).hasMessageContaining("focused field"); + assertThatThrownBy(() -> service.snapshot("document", "3")).isInstanceOf(QNotFoundException.class); + QContext.getQSession().clearSecurityKeyValues(); + QContext.getQSession().withSecurityKeyValue("tenant", "b"); + assertThat(service.snapshot("document", null)).isEmpty(); + login("alice"); + service.leave(aliceLease, "document", "1", 2L); + assertThat(service.snapshot("document", "1")).isEmpty(); + service.renewLease(service.openLease("document", "1"), "document", "1", "VIEWING", null, 1L); + var client = com.kingsrook.qbits.redis.RedisRuntimeService.connection(instance, "presence").client(); + // Test-only corruption of the disposable Redis server's record state. + client.del(client.keys("*:record:*:state").iterator().next()); + assertThatThrownBy(() -> service.snapshot("document", "1")) + .isInstanceOf(QServiceUnavailableException.class).hasMessage("Presence is temporarily unavailable. Please retry.").hasNoCause(); + redis.stop(); + assertThatThrownBy(() -> service.snapshot("document", "1")) + .isInstanceOf(QServiceUnavailableException.class).hasMessage("Presence is temporarily unavailable. Please retry.").hasNoCause(); + assertThatThrownBy(() -> service.heartbeat("tab", "document", "3", "VIEWING", null, 1L)) + .isInstanceOf(QNotFoundException.class); + admin(); + assertThatThrownBy(com.kingsrook.qbits.collaboration.app.CollaborationHome::presence) + .isInstanceOf(QServiceUnavailableException.class); + } + finally + { + runtime.stop(); + runtime.afterApplicationStop(); + } + } + } + + /*************************************************************************** + ** Wait for every dashboard table in both authenticated tenant namespaces. + ** No Redis fixture writes bypass the production cold-recovery contract. + ***************************************************************************/ + private static void warmScopes(com.kingsrook.qqq.backend.core.model.metadata.QInstance node) throws Exception + { + var previousInstance = QContext.getQInstance(); + var previousSession = QContext.getQSession(); + var a = session("warm-a"); + var b = session("warm-b"); + b.clearSecurityKeyValues(); + b.withSecurityKeyValue("tenant", "b"); + var sessions = java.util.List.of(a, b); + try + { + for(var user : sessions) + { + QContext.init(node, user); + var backend = new RedisPresenceBackend(RedisPresenceConfig.forInstance(node)); + for(String table : node.getTables().keySet()) + { + assertThatThrownBy(() -> backend.snapshot(table, null)) + .isInstanceOf(QServiceUnavailableException.class).hasMessageContaining("recovering"); + } + } + long deadline = System.nanoTime() + java.time.Duration.ofSeconds(50).toNanos(); + for(var user : sessions) + { + QContext.init(node, user); + var backend = new RedisPresenceBackend(RedisPresenceConfig.forInstance(node)); + for(String table : node.getTables().keySet()) + { + while(true) + { + try + { + assertThat(backend.snapshot(table, null)).isEmpty(); + break; + } + catch(QServiceUnavailableException recovering) + { + assertThat(recovering).hasMessageContaining("recovering"); + if(System.nanoTime() >= deadline) + { + throw (new AssertionError("Service recovery did not finish", recovering)); + } + Thread.sleep(50); + } + } + } + } + } + finally + { + QContext.init(previousInstance, previousSession); + } + } + + /*************************************************************************** + ** Tenant identity comes from the authenticated host session, never the request. + ***************************************************************************/ + public static class TenantScope implements PresenceScopeProvider + { + /************************************************************************ + ** Require exactly one authenticated tenant in this fixture. + ************************************************************************/ + @Override + public String tenantScope() + { + var tenants = QContext.getQSession().getSecurityKeyValues("tenant"); + if(tenants.size() != 1) + { + throw (new IllegalStateException("One tenant required")); + } + return (tenants.getFirst().toString()); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceStoreTest.java b/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceStoreTest.java new file mode 100644 index 0000000..52b3de0 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceStoreTest.java @@ -0,0 +1,404 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration; + +import java.time.Duration; +import com.kingsrook.qbits.collaboration.presence.RedisPresenceStore; +import com.kingsrook.qbits.redis.RedisConnection; +import com.kingsrook.qbits.redis.api.RedisNamespace; +import org.junit.jupiter.api.Test; +import org.testcontainers.containers.GenericContainer; +import redis.clients.jedis.DefaultJedisClientConfig; +import redis.clients.jedis.HostAndPort; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/******************************************************************************* + ** Independent clients exercise server-side ordering, expiry and shared snapshots. + *******************************************************************************/ +class RedisPresenceStoreTest +{ + /*************************************************************************** + ** Prime cold fixture namespaces together, using actual server readiness. + ***************************************************************************/ + private static void awaitReady(RedisPresenceStore... stores) throws Exception + { + for(RedisPresenceStore store : stores) + { + try + { + assertThat(store.snapshot("orders", null)).isEmpty(); + } + catch(com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException expected) + { + // The first read initializes the shared recovery interval. + } + } + long deadline = System.nanoTime() + Duration.ofSeconds(5).toNanos(); + for(RedisPresenceStore store : stores) + { + while(true) + { + try + { + assertThat(store.snapshot("orders", null)).isEmpty(); + break; + } + catch(com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException expected) + { + if(System.nanoTime() >= deadline) + { + throw (new AssertionError("Fixture recovery did not finish", expected)); + } + Thread.sleep(20); + } + } + } + } + + /*************************************************************************** + ** Server-issued leases cross nodes but cannot survive departure or data loss. + ***************************************************************************/ + @Test + void issuedLeaseRequiresLiveAuthority() throws Exception + { + try(GenericContainer redis = RedisFixtures.redis()) + { + redis.start(); + var address = new HostAndPort(RedisFixtures.host(redis), RedisFixtures.port(redis)); + var config = DefaultJedisClientConfig.builder().build(); + try(var first = new RedisConnection(address, config, 2, Duration.ofMillis(500)); + var second = new RedisConnection(address, config, 2, Duration.ofMillis(500))) + { + var scope = new RedisNamespace("test", "app", "issued", "presence"); + var a = new RedisPresenceStore(first.scripts(), scope, Duration.ofSeconds(2), 10); + var b = new RedisPresenceStore(second.scripts(), scope, Duration.ofSeconds(2), 10); + awaitReady(a); + String lease = a.openLease("one", "One", "orders", "1"); + assertThat(lease).startsWith("lease:"); + assertThat(b.snapshot("orders", "1")).isEmpty(); + assertThat(b.snapshot("orders", null)).isEmpty(); + assertThatThrownBy(() -> b.renewLease("other", "Other", lease, "orders", "1", "EDITING", null, 1L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + assertThatThrownBy(() -> b.renewLease("one", "One", lease, "orders", "2", "EDITING", null, 1L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + assertThat(b.renewLease("one", "One", lease, "orders", "1", "EDITING", "status", 1L)).isTrue(); + var expiryKey = first.client().keys("*:record:*:expiry").iterator().next(); + var expiryBefore = first.client().zrangeWithScores(expiryKey, 0, -1); + assertThat(a.renewLease("one", "One", lease, "orders", "1", "VIEWING", null, 1L)).isTrue(); + assertThat(first.client().zrangeWithScores(expiryKey, 0, -1)).isEqualTo(expiryBefore); + assertThat(a.snapshot("orders", "1")).hasSize(1); + assertThat(a.snapshot("orders", "1").getFirst().mode()).isEqualTo("EDITING"); + assertThat(a.snapshot("orders", null)).allSatisfy(viewer -> assertThat(viewer.fieldName()).isNull()); + a.leave("one", lease, "orders", "1", 3L); + assertThatThrownBy(() -> b.renewLease("one", "One", lease, "orders", "1", "EDITING", null, 2L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + assertThatThrownBy(() -> b.renewLease("one", "One", lease, "orders", "1", "EDITING", null, 4L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + + String replacement = b.openLease("one", "One", "orders", "1"); + assertThat(replacement).isNotEqualTo(lease); + assertThat(a.renewLease("one", "One", replacement, "orders", "1", "VIEWING", null, 1L)).isTrue(); + first.client().flushDB(); + assertThatThrownBy(() -> b.renewLease("one", "One", replacement, "orders", "1", "VIEWING", null, 2L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + assertThatThrownBy(() -> b.heartbeat("one", "One", replacement, "orders", "1", "VIEWING", null, 2L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + assertThat(first.client().dbSize()).isEqualTo(1); + assertThat(first.client().keys("*:recovery")).hasSize(1); + } + } + } + + /*************************************************************************** + ** Abandoned reservations consume bounded capacity and expire invisibly. + ***************************************************************************/ + @Test + void issuedLeaseExpiryDoesNotRecreateMembership() throws Exception + { + try(GenericContainer redis = RedisFixtures.redis()) + { + redis.start(); + try(var connection = new RedisConnection(new HostAndPort(RedisFixtures.host(redis), RedisFixtures.port(redis)), + DefaultJedisClientConfig.builder().build(), 2, Duration.ofMillis(500))) + { + var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "app", "issued-expiry", "presence"), Duration.ofMillis(150), 1); + awaitReady(store); + String pending = store.openLease("one", "One", "orders", "1"); + assertThat(store.openLease("two", "Two", "orders", "2")).isNull(); + assertThat(store.snapshot("orders", null)).isEmpty(); + Thread.sleep(250); + assertThatThrownBy(() -> store.renewLease("one", "One", pending, "orders", "1", "EDITING", null, 1L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + String active = store.openLease("two", "Two", "orders", "2"); + assertThat(store.renewLease("two", "Two", active, "orders", "2", "EDITING", null, 1L)).isTrue(); + Thread.sleep(250); + assertThatThrownBy(() -> store.renewLease("two", "Two", active, "orders", "2", "EDITING", null, 2L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + assertThat(store.snapshot("orders", null)).isEmpty(); + String departed = store.openLease("two", "Two", "orders", "2"); + assertThat(departed).isNotNull(); + store.renewLease("two", "Two", departed, "orders", "2", "EDITING", null, 1L); + store.leave("two", departed, "orders", "2", 3L); + Thread.sleep(250); + assertThatThrownBy(() -> store.renewLease("two", "Two", departed, "orders", "2", "EDITING", null, 2L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + assertThat(store.snapshot("orders", null)).isEmpty(); + } + } + } + + /*************************************************************************** + ** Total loss stays unknown even when the internal legacy API repopulates it. + ** The shared HTTP service requires server-issued leases instead of this API. + ***************************************************************************/ + @Test + void totalLossIsRecoveringEvenForLegacyHeartbeat() throws Exception + { + try(GenericContainer redis = RedisFixtures.redis()) + { + redis.start(); + var address = new HostAndPort(RedisFixtures.host(redis), RedisFixtures.port(redis)); + var config = DefaultJedisClientConfig.builder().build(); + try(var first = new RedisConnection(address, config, 2, Duration.ofMillis(500)); + var second = new RedisConnection(address, config, 2, Duration.ofMillis(500))) + { + var scope = new RedisNamespace("test", "app", "recovery", "presence"); + var a = new RedisPresenceStore(first.scripts(), scope, Duration.ofSeconds(2), 10); + var b = new RedisPresenceStore(second.scripts(), scope, Duration.ofSeconds(2), 10); + awaitReady(a); + a.heartbeat("active", "Active", "active-tab", "orders", "1", "VIEWING", null, 1L); + a.heartbeat("departed", "Departed", "old-tab", "orders", "1", "EDITING", null, 1L); + b.leave("departed", "old-tab", "orders", "1", 3L); + assertThat(b.snapshot("orders", "1")).extracting("userId").containsExactly("active"); + + // Only this disposable test server is cleared. Both clients stay alive. + first.client().flushDB(); + assertThatThrownBy(() -> b.snapshot("orders", "1")) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException.class); + assertThat(a.heartbeat("departed", "Departed", "old-tab", "orders", "1", "EDITING", null, 2L)).isTrue(); + assertThatThrownBy(() -> b.snapshot("orders", "1")) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException.class); + assertThatThrownBy(() -> b.snapshot("orders", null)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException.class); + } + } + } + + /*************************************************************************** + ** Characterize the separate ordering gap after a departure marker expires. + ** A newer room member keeps both projections alive throughout this test. + ***************************************************************************/ + @Test + void expiredDepartureCurrentlyAcceptsDelayedHeartbeat() throws Exception + { + try(GenericContainer redis = RedisFixtures.redis()) + { + redis.start(); + try(var connection = new RedisConnection(new HostAndPort(RedisFixtures.host(redis), RedisFixtures.port(redis)), + DefaultJedisClientConfig.builder().build(), 2, Duration.ofMillis(500))) + { + var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "app", "late", "presence"), Duration.ofMillis(400), 10); + awaitReady(store); + store.heartbeat("departed", "Departed", "tab", "orders", "1", "EDITING", null, 1L); + store.leave("departed", "tab", "orders", "1", 3L); + store.heartbeat("departed", "Departed", "tab", "orders", "1", "EDITING", null, 2L); + assertThat(store.snapshot("orders", "1")).isEmpty(); + + // A later renewal keeps the peer live with the same deployment TTL. + Thread.sleep(250); + var peer = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "app", "late", "presence"), Duration.ofMillis(400), 10); + peer.heartbeat("active", "Active", "peer-tab", "orders", "1", "VIEWING", null, 1L); + Thread.sleep(250); + assertThat(peer.snapshot("orders", "1")).extracting("userId").containsExactly("active"); + store.heartbeat("departed", "Departed", "tab", "orders", "1", "EDITING", null, 2L); + assertThat(peer.snapshot("orders", "1")).extracting("userId").containsExactly("active", "departed"); + } + } + } + + /*************************************************************************** + ** Six people share one record, with two editing; tabs deduplicate by user. + ***************************************************************************/ + @Test + void sharedRoomOrderingIsolationAndExpiry() throws Exception + { + try(GenericContainer redis = RedisFixtures.redis()) + { + redis.start(); + var address = new HostAndPort(RedisFixtures.host(redis), RedisFixtures.port(redis)); + var config = DefaultJedisClientConfig.builder().connectionTimeoutMillis(500).socketTimeoutMillis(500).build(); + try(var first = new RedisConnection(address, config, 4, Duration.ofMillis(500)); + var second = new RedisConnection(address, config, 4, Duration.ofMillis(500))) + { + var scope = new RedisNamespace("test", "app", "tenant-a", "presence"); + var a = new RedisPresenceStore(first.scripts(), scope, Duration.ofSeconds(2), 100); + var b = new RedisPresenceStore(second.scripts(), scope, Duration.ofSeconds(2), 100); + var tenantB = new RedisPresenceStore(second.scripts(), new RedisNamespace("test", "app", "tenant-b", "presence"), Duration.ofSeconds(2), 100); + awaitReady(a, tenantB); + for(int i = 0; i < 6; i++) + { + assertThat((i % 2 == 0 ? a : b).heartbeat("user-" + i, "Person " + i, "tab-" + i, "orders", "123", + i < 2 ? "EDITING" : "VIEWING", i < 2 ? "status" : null, 1L)).isTrue(); + } + assertThat(b.snapshot("orders", "123")).hasSize(6).isEqualTo(a.snapshot("orders", "123")); + assertThat(b.snapshot("orders", "123").stream().filter(viewer -> viewer.mode().equals("EDITING"))).hasSize(2); + assertThat(b.snapshot("orders", null)).hasSize(6).allSatisfy(viewer -> assertThat(viewer.fieldName()).isNull()); + assertThat(a.heartbeat("user-0", "Person 0", "extra-tab", "orders", "123", "VIEWING", null, 1L)).isTrue(); + assertThat(b.snapshot("orders", "123")).hasSize(6); + assertThat(b.snapshot("orders", "123").getFirst().mode()).isEqualTo("EDITING"); + assertThat(b.leave("user-0", "tab-0", "orders", "123", 3L)).isTrue(); + assertThat(a.heartbeat("user-0", "Person 0", "tab-0", "orders", "123", "EDITING", "status", 2L)).isTrue(); + assertThat(a.snapshot("orders", "123").getFirst().mode()).isEqualTo("VIEWING"); + assertThat(a.heartbeat("user-0", "Person 0", "tab-0", "orders", "123", "EDITING", "status", 3L)).isTrue(); + assertThat(a.snapshot("orders", "123").getFirst().mode()).isEqualTo("VIEWING"); + assertThat(a.heartbeat("user-0", "Person 0", "tab-0", "orders", "123", "EDITING", "status", 4L)).isTrue(); + assertThat(b.snapshot("orders", "123").getFirst().mode()).isEqualTo("EDITING"); + assertThat(tenantB.snapshot("orders", "123")).isEmpty(); + assertThat(a.snapshot("orders", "124")).isEmpty(); + + try(var writers = java.util.concurrent.Executors.newFixedThreadPool(4)) + { + var writes = new java.util.ArrayList>(); + for(long sequence = 10; sequence < 50; sequence++) + { + final long requestSequence = sequence; + writes.add(writers.submit(() -> (requestSequence % 2 == 0 ? a : b).heartbeat("racing-user", "Racing", "racing-tab", + "orders", "124", requestSequence == 49 ? "EDITING" : "VIEWING", null, requestSequence))); + } + for(var write : writes) + { + assertThat(write.get(10, java.util.concurrent.TimeUnit.SECONDS)).isTrue(); + } + } + assertThat(a.snapshot("orders", "124")).hasSize(1); + assertThat(a.snapshot("orders", "124").getFirst().mode()).isEqualTo("EDITING"); + assertThat(b.leave("racing-user", "racing-tab", "orders", "124", 50L)).isTrue(); + assertThat(a.heartbeat("racing-user", "Racing", "racing-tab", "orders", "124", "EDITING", null, 49L)).isTrue(); + assertThat(a.snapshot("orders", "124")).isEmpty(); + + var shortLease = new RedisPresenceStore(first.scripts(), new RedisNamespace("test", "app", "short", "presence"), Duration.ofMillis(150), 10); + awaitReady(shortLease); + assertThat(shortLease.heartbeat("user", "Person", "tab", "orders", "123", "EDITING", "status", 1L)).isTrue(); + long deadline = System.nanoTime() + Duration.ofSeconds(3).toNanos(); + while(!shortLease.snapshot("orders", "123").isEmpty() && System.nanoTime() < deadline) + { + Thread.sleep(20); + } + assertThat(shortLease.snapshot("orders", "123")).isEmpty(); + assertThat(shortLease.snapshot("orders", null)).isEmpty(); + redis.execInContainer("redis-cli", "SCRIPT", "FLUSH"); + assertThat(b.snapshot("orders", "123")).hasSize(6); + redis.stop(); + assertThatThrownBy(() -> a.snapshot("orders", "123")).isInstanceOf(redis.clients.jedis.exceptions.JedisException.class); + } + } + } + + /*************************************************************************** + ** A full table cannot leave a half-written record membership behind. + ***************************************************************************/ + @Test + void capacityChecksAreAtomic() throws Exception + { + try(GenericContainer redis = RedisFixtures.redis()) + { + redis.start(); + try(var connection = new RedisConnection(new HostAndPort(RedisFixtures.host(redis), RedisFixtures.port(redis)), + DefaultJedisClientConfig.builder().build(), 2, Duration.ofMillis(500))) + { + var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "app", "capacity", "presence"), Duration.ofSeconds(2), 1); + awaitReady(store); + assertThat(store.heartbeat("one", "One", "tab", "orders", "1", "VIEWING", null, 1L)).isTrue(); + assertThat(store.heartbeat("two", "Two", "tab", "orders", "2", "VIEWING", null, 1L)).isFalse(); + assertThat(store.snapshot("orders", "2")).isEmpty(); + assertThat(store.snapshot("orders", null)).hasSize(1); + assertThat(store.leave("one", "tab", "orders", "1", 2L)).isTrue(); + assertThat(store.snapshot("orders", null)).isEmpty(); + assertThat(store.heartbeat("two", "Two", "tab", "orders", "2", "VIEWING", null, 1L)).isFalse(); + assertThatThrownBy(() -> store.heartbeat("one", "One", "tab", "orders", "1", "INVALID", null, 3L)).isInstanceOf(IllegalArgumentException.class); + } + } + } + + /*************************************************************************** + ** Losing one half of a room must not report empty or partially renew a lease. + ***************************************************************************/ + @Test + void partialRoomLossIsUnavailable() throws Exception + { + try(GenericContainer redis = RedisFixtures.redis()) + { + redis.start(); + try(var connection = new RedisConnection(new HostAndPort(RedisFixtures.host(redis), RedisFixtures.port(redis)), + DefaultJedisClientConfig.builder().build(), 2, Duration.ofMillis(500))) + { + for(String lostPart : java.util.List.of("record:state", "record:expiry", "table:state", "table:expiry")) + { + String[] parts = lostPart.split(":"); + boolean record = parts[0].equals("record"); + var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "app", parts[0] + parts[1], "presence"), Duration.ofSeconds(2), 10); + awaitReady(store); + String lease = store.openLease("one", "One", "orders", "1"); + assertThat(store.renewLease("one", "One", lease, "orders", "1", "VIEWING", null, 1L)).isTrue(); + // KEYS is confined to this disposable test server, never the application path. + String lostKey = connection.client().keys(record ? "*:record:*:" + parts[1] : "*:table:" + parts[1]).iterator().next(); + String retainedState = connection.client().keys(record ? "*:table:state" : "*:record:*:state").iterator().next(); + var before = connection.client().hgetAll(retainedState); + connection.client().del(lostKey); + assertThatThrownBy(() -> store.snapshot("orders", "1")) + .isInstanceOf(redis.clients.jedis.exceptions.JedisException.class).hasMessageContaining("incomplete"); + assertThatThrownBy(() -> store.renewLease("one", "One", lease, "orders", "1", "EDITING", "status", 2L)) + .isInstanceOf(redis.clients.jedis.exceptions.JedisException.class).hasMessageContaining("incomplete"); + assertThat(connection.client().hgetAll(retainedState)).isEqualTo(before); + connection.client().flushDB(); + } + for(String room : java.util.List.of("record", "table")) + { + var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "app", room, "presence"), Duration.ofSeconds(2), 10); + awaitReady(store); + String lease = store.openLease("one", "One", "orders", "1"); + store.renewLease("one", "One", lease, "orders", "1", "VIEWING", null, 1L); + store.leave("one", lease, "orders", "1", 3L); + var lostKeys = connection.client().keys(room.equals("record") ? "*:record:*" : "*:table:*"); + connection.client().del(lostKeys.toArray(String[]::new)); + if(room.equals("record")) + { + assertThatThrownBy(() -> store.snapshot("orders", "1")) + .isInstanceOf(redis.clients.jedis.exceptions.JedisException.class).hasMessageContaining("incomplete"); + assertThatThrownBy(() -> store.renewLease("one", "One", lease, "orders", "1", "EDITING", null, 2L)) + .isInstanceOf(redis.clients.jedis.exceptions.JedisException.class).hasMessageContaining("incomplete"); + } + else + { + assertThatThrownBy(() -> store.snapshot("orders", "1")) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException.class); + assertThatThrownBy(() -> store.renewLease("one", "One", lease, "orders", "1", "EDITING", null, 2L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + } + assertThat(connection.client().keys(room.equals("record") ? "*:record:*" : "*:table:*")).isEmpty(); + connection.client().flushDB(); + } + } + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/SharingWorkTest.java b/src/test/java/com/kingsrook/qbits/collaboration/SharingWorkTest.java index 8dcbb5e..1552190 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/SharingWorkTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/SharingWorkTest.java @@ -389,11 +389,16 @@ public class SharingWorkTest extends PersonalServiceFixture @Test void assignmentsUpsertAndMyRecordsRespectParentLocks() throws Exception { + assertThatThrownBy(() -> assignments.assignRoles("document", 1, null, null, java.util.Set.of())).hasMessageContaining("Supply ownerUserId"); assertThatThrownBy(() -> assignments.assign("document", 1, "eve", null)).hasMessageContaining("can't open"); QRecord first = assignments.assign("document", 1, "alice", "bob"); QRecord again = assignments.assign("document", "01", "alice", "bob"); assertThat(again.getValue("id")).isEqualTo(first.getValue("id")); assertThat(recorder.notifications()).hasSize(1); + QRecord ownerOnly = assignments.assignRoles("document", 1, null, null, java.util.Set.of("ownerUserId")); + assertThat(ownerOnly.getValueString("ownerUserId")).isNull(); + assertThat(ownerOnly.getValueString("assigneeUserId")).isEqualTo("bob"); + assignments.assignRoles("document", 1, "alice", null, java.util.Set.of("ownerUserId")); assignments.assign("document", 2, null, "alice"); assertThat(assignments.myRecords(null, 1).get("document").count()).isEqualTo(2); assertThat(assignments.myRecords("document", 1).get("document").records()).hasSize(1); @@ -404,6 +409,21 @@ public class SharingWorkTest extends PersonalServiceFixture assertThat(instance.getTable(CollaborationTables.RECORD_ASSIGNMENT).getPermissionRules().getDenyBehavior()).isEqualTo(DenyBehavior.HIDDEN); } + /******************************************************************************* + ** The fallback API cannot create a competing owner for native ownership metadata. + *******************************************************************************/ + @Test + void fallbackAssignmentsRejectNativeOwnership() throws Exception + { + var metadata = com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordFeaturesMetaData.ofOrWithNew(instance.getTable("document")); + metadata.withOwnership(new com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordOwnership().withOwnerFieldName("title")); + com.kingsrook.qqq.backend.core.actions.collaboration.RuntimeMetaDataOverlay.refresh(instance); + assertThatThrownBy(() -> assignments.assignRoles("document", 1, "alice", null, java.util.Set.of("ownerUserId"))) + .hasMessageContaining("native ownership"); + assertThat(assignments.get("document", 1)).isEmpty(); + assertThat(assignments.assignRoles("document", 1, null, "bob", java.util.Set.of("assigneeUserId")).getValueString("assigneeUserId")).isEqualTo("bob"); + } + /******************************************************************************* ** My Work Summarizes Only Visible Work And Read Mentions. *******************************************************************************/ diff --git a/src/test/java/com/kingsrook/qbits/collaboration/TagsServiceTest.java b/src/test/java/com/kingsrook/qbits/collaboration/TagsServiceTest.java index 85ca704..63348c5 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/TagsServiceTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/TagsServiceTest.java @@ -73,6 +73,126 @@ public class TagsServiceTest extends PersonalServiceFixture private final TagsService tags = new TagsService(); + /******************************************************************************* + ** Tag labels are selectable while the ID-based filter picker remains intact. + *******************************************************************************/ + @Test + void selectsTagLabelsWithoutTranslatingThemAsIds() throws Exception + { + TagEntry beta = tags.create("document", "Beta", null); + TagEntry alpha = tags.create("document", "Alpha", null); + tags.apply("document", 1, List.of(beta.id(), alpha.id()), List.of()); + com.kingsrook.qqq.backend.core.actions.collaboration.RuntimeMetaDataOverlay.refresh(instance); + var field = instance.getTable("document").getVirtualField("collaborationTags"); + assertThat(field.getIsQuerySelectable()).isTrue(); + assertThat(field.getPossibleValueSourceName()).isEqualTo(CollaborationTables.TAG); + var rows = new QueryAction().execute(new QueryInput("document").withFieldNamesToInclude(java.util.Set.of("collaborationTags")) + .withShouldTranslatePossibleValues(true).withShouldGenerateDisplayValues(true)).getRecords(); + assertThat(rows).hasSize(2); + assertThat(rows).allSatisfy(row -> assertThat(row.getValues()).containsOnlyKeys("collaborationTags")); + assertThat(rows).extracting(row -> row.getValueString("collaborationTags")).containsExactlyInAnyOrder("Alpha, Beta", null); + assertThat(rows).filteredOn(row -> row.getValue("collaborationTags") != null) + .allSatisfy(row -> assertThat(row.getDisplayValue("collaborationTags")).isEqualTo("Alpha, Beta")); + assertThat(new QueryAction().execute(new QueryInput("document").withFilter(new QQueryFilter() + .withCriteria("collaborationTags", QCriteriaOperator.EQUALS, alpha.id()))).getRecords()) + .extracting(row -> row.getValueInteger("id")).containsExactly(1); + } + + /******************************************************************************* + ** Exported labels obey parent visibility and vocabulary approval, with CSV quoting. + *******************************************************************************/ + @Test + void tagLabelsRespectPrivacyAndExportQuoting() throws Exception + { + TagEntry visible = tags.create("document", "Needs, review", null); + TagEntry secret = tags.create("document", "Secret label", null); + tags.apply("document", 1, List.of(visible.id()), List.of()); + QContext.getQSession().withSecurityKeyValue("allTenants", true); + tags.apply("document", 3, List.of(secret.id()), List.of()); + config.withDefaultTagMode("APPROVED_ONLY"); + TagEntry pending = tags.create("document", "Pending label", null); + assertThat(pending.approved()).isFalse(); + // A stale assignment must not turn a pending suggestion into a visible label. + CollaborationRecords.insert(CollaborationTables.RECORD_TAG, new QRecord().withValue("tableName", "document") + .withValue("recordId", "1").withValue("tagId", pending.id()).withValue("createdByUserId", "alice").withValue("parentDeleted", false)); + login("alice"); + com.kingsrook.qqq.backend.core.actions.collaboration.RuntimeMetaDataOverlay.refresh(instance); + var input = new QueryInput("document").withFieldNamesToInclude(java.util.Set.of("collaborationTags")); + assertThat(new QueryAction().execute(input).getRecords()).extracting(row -> row.getValueString("collaborationTags")) + .containsExactlyInAnyOrder("Needs, review", null); + + java.io.ByteArrayOutputStream output = new java.io.ByteArrayOutputStream(); + var export = new com.kingsrook.qqq.backend.core.model.actions.reporting.ExportInput(); + export.setTableName("document"); + export.setFieldNames(List.of("collaborationTags")); + export.setReportDestination(new com.kingsrook.qqq.backend.core.model.actions.reporting.ReportDestination() + .withReportFormat(com.kingsrook.qqq.backend.core.model.actions.reporting.ReportFormat.CSV).withReportOutputStream(output)); + new com.kingsrook.qqq.backend.core.actions.reporting.ExportAction().execute(export); + String csv = output.toString(java.nio.charset.StandardCharsets.UTF_8); + assertThat(csv).doesNotContain("Secret label", "Pending label"); + try(var parser = org.apache.commons.csv.CSVParser.parse(csv, org.apache.commons.csv.CSVFormat.DEFAULT.withFirstRecordAsHeader())) + { + assertThat(parser.getHeaderNames()).containsExactly("Tags"); + assertThat(parser.getRecords()).extracting(row -> row.get("Tags")).containsExactlyInAnyOrder("Needs, review", ""); + } + QContext.getQSession().getSecurityKeyValues().put("tenant", List.of("b")); + assertThat(new QueryAction().execute(input).getRecords()).extracting(row -> row.getValueString("collaborationTags")).containsExactly("Secret label"); + login("alice"); + assertThat(new QueryAction().execute(input).getRecords()).extracting(row -> row.getValueString("collaborationTags")) + .containsExactlyInAnyOrder("Needs, review", null); + } + + /******************************************************************************* + ** A page spanning the provider's batch boundary retains its last parent values. + *******************************************************************************/ + @Test + void tagLabelsSpanMultipleParentBatches() throws Exception + { + java.util.List parents = new java.util.ArrayList<>(); + for(int id = 4; id <= 504; id++) + { + parents.add(new QRecord().withValue("id", id).withValue("title", "Document " + id).withValue("tenant", "a")); + } + assertThat(new InsertAction().execute(new InsertInput("document").withRecords(parents)).getRecords()) + .allSatisfy(row -> assertThat(row.getErrors()).isNullOrEmpty()); + TagEntry tag = tags.create("document", "Boundary", null); + tags.apply("document", 1, List.of(tag.id()), List.of()); + tags.apply("document", 504, List.of(tag.id()), List.of()); + com.kingsrook.qqq.backend.core.actions.collaboration.RuntimeMetaDataOverlay.refresh(instance); + var rows = new QueryAction().execute(new QueryInput("document").withFieldNamesToInclude(java.util.Set.of("id", "collaborationTags"))).getRecords(); + assertThat(rows).hasSize(503); + assertThat(rows).filteredOn(row -> row.getValue("collaborationTags") != null).extracting(row -> row.getValueInteger("id")) + .containsExactlyInAnyOrder(1, 504); + assertThat(rows).filteredOn(row -> row.getValue("collaborationTags") != null) + .allSatisfy(row -> assertThat(row.getValueString("collaborationTags")).isEqualTo("Boundary")); + } + + /******************************************************************************* + ** Stale links cannot expose merged vocabulary, other-table tags, or deleted links. + *******************************************************************************/ + @Test + void tagLabelsExcludeStaleAndOutOfScopeAssignments() throws Exception + { + login("admin", TagsService.MANAGE); + TagEntry global = tags.create(null, "Global", null); + TagEntry current = tags.create("document", "Current", null); + TagEntry merged = tags.create("document", "Merged old label", null); + TagEntry deleted = tags.create("document", "Deleted link label", null); + TagEntry other = tags.create("document", "Other table label", null); + CollaborationRecords.update(CollaborationTables.TAG, new QRecord().withValue("id", merged.id()).withValue("mergedIntoTagId", current.id())); + CollaborationRecords.update(CollaborationTables.TAG, new QRecord().withValue("id", other.id()).withValue("tableName", "different_table")); + for(TagEntry tag : List.of(global, current, merged, deleted, other)) + { + CollaborationRecords.insert(CollaborationTables.RECORD_TAG, new QRecord().withValue("tableName", "document").withValue("recordId", "1") + .withValue("tagId", tag.id()).withValue("createdByUserId", "admin").withValue("parentDeleted", tag.id().equals(deleted.id()))); + } + login("alice"); + com.kingsrook.qqq.backend.core.actions.collaboration.RuntimeMetaDataOverlay.refresh(instance); + var rows = new QueryAction().execute(new QueryInput("document").withFieldNamesToInclude(java.util.Set.of("collaborationTags"))).getRecords(); + assertThat(rows).extracting(row -> row.getValueString("collaborationTags")).containsExactlyInAnyOrder("Current, Global", null); + assertThat(com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordReferenceTableMetaData.of(instance.getTable(CollaborationTables.RECORD_TAG))).isNotNull(); + } + /******************************************************************************* ** Natural keys are locale-independent and validate after collapsing whitespace. *******************************************************************************/ diff --git a/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentAggregateGuardTest.java b/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentAggregateGuardTest.java new file mode 100644 index 0000000..de87518 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentAggregateGuardTest.java @@ -0,0 +1,174 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + + +import com.kingsrook.qbits.collaboration.PersonalServiceFixture; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.actions.metadata.MetaDataAction; +import com.kingsrook.qqq.backend.core.actions.tables.CountAction; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.metadata.MetaDataInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.aggregate.AggregateInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryJoin; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Actual native Count permission enforcement and metadata, plus aggregate input + ** guard semantics. Record Files separately tests archived native parent rows. + *******************************************************************************/ +public class AttachmentAggregateGuardTest extends PersonalServiceFixture +{ + /******************************************************************************* + ** Ordinary Collaboration keeps its established count and metadata contract. + *******************************************************************************/ + @Test + void defaultPolicyPreservesOrdinaryCount() throws Exception + { + login("alice", "collaboration.view"); + assertThat(new CountAction().execute(new CountInput(CollaborationTables.ATTACHMENT)).getCount()).isZero(); + assertThat(instance.getTable(CollaborationTables.ATTACHMENT).getDisabledCapabilities()).doesNotContain(Capability.TABLE_COUNT); + assertThat(new AttachmentLifecyclePolicy().requiresPinnedAggregateParent()).isFalse(); + } + + /******************************************************************************* + ** A readable literal pin is accepted; broad or invisible parents are rejected. + *******************************************************************************/ + @Test + void nativeCountRequiresCurrentlyVisiblePinnedParent() throws Exception + { + enable(); + assertThatThrownBy(() -> new CountAction().execute(new CountInput(CollaborationTables.ATTACHMENT))).isInstanceOf(QPermissionDeniedException.class); + assertThat(new CountAction().execute(new CountInput(CollaborationTables.ATTACHMENT).withFilter(pin("1"))).getCount()).isZero(); + assertThatThrownBy(() -> new CountAction().execute(new CountInput(CollaborationTables.ATTACHMENT).withFilter(pin("3")))).isInstanceOf(QPermissionDeniedException.class); + var guard = new AttachmentAggregateGuard(); + assertThatThrownBy(() -> guard.checkPermissionsThrowing(new AggregateInput(CollaborationTables.ATTACHMENT), instance.getTable(CollaborationTables.ATTACHMENT))) + .isInstanceOf(QPermissionDeniedException.class); + guard.checkPermissionsThrowing(new AggregateInput(CollaborationTables.ATTACHMENT).withFilter(pin("1")), instance.getTable(CollaborationTables.ATTACHMENT)); + } + + /******************************************************************************* + ** OR, set branches, conflicting pins and joins cannot escape the exact parent. + *******************************************************************************/ + @Test + void rejectsFilterShapesWhichDoNotGuaranteeOneParent() throws Exception + { + enable(); + for(QQueryFilter filter : java.util.List.of(pin("1").withBooleanOperator(QQueryFilter.BooleanOperator.OR), + pin("1").withSubFilterSetOperator(QQueryFilter.SubFilterSetOperator.UNION), + pin("1").withSubFilter(new QQueryFilter().withSubFilterSetOperator(QQueryFilter.SubFilterSetOperator.UNION_ALL)), + pin("1").withCriteria("recordId", QCriteriaOperator.EQUALS, "2"), + new QQueryFilter().withSubFilter(pin("1")), new QQueryFilter().withCriteria("tableName", QCriteriaOperator.EQUALS, "document"))) + { + assertThatThrownBy(() -> new CountAction().execute(new CountInput(CollaborationTables.ATTACHMENT).withFilter(filter))).isInstanceOf(QPermissionDeniedException.class); + } + assertThatThrownBy(() -> new CountAction().execute(new CountInput(CollaborationTables.ATTACHMENT).withFilter(pin("1")).withQueryJoin(new QueryJoin()))) + .isInstanceOf(QPermissionDeniedException.class); + } + + /******************************************************************************* + ** Generic list clients see no unscoped total capability in instance metadata. + *******************************************************************************/ + @Test + void metadataHidesGenericTotalsWithoutDisablingPinnedNativeCount() throws Exception + { + enable(); + var table = new MetaDataAction().execute(new MetaDataInput()).getTables().get(CollaborationTables.ATTACHMENT); + assertThat(table.getCapabilities()).doesNotContain("TABLE_COUNT", "QUERY_STATS").contains("TABLE_QUERY"); + assertThat(new CountAction().execute(new CountInput(CollaborationTables.ATTACHMENT).withFilter(pin("1"))).getCount()).isZero(); + } + + /******************************************************************************* + ** Fixture policy is installed before count actions and reflected in metadata. + *******************************************************************************/ + private void enable() + { + login("alice", "collaboration.view"); + config.withAttachmentLifecycle(new QCodeReference(PinnedPolicy.class)); + AttachmentAggregateGuard.configure(instance.getTable(CollaborationTables.ATTACHMENT), config); + } + + /******************************************************************************* + ** Removing the runtime lifecycle cannot broaden a previously installed table. + *******************************************************************************/ + @Test + void installedRestrictionSurvivesPolicyRemovalAndMetadataClone() throws Exception + { + enable(); + var clone = instance.getTable(CollaborationTables.ATTACHMENT).clone(); + config.withAttachmentLifecycle(new QCodeReference(AttachmentLifecyclePolicy.class)); + instance.getTables().put(CollaborationTables.ATTACHMENT, AttachmentAggregateGuard.configure(clone, config)); + assertThatThrownBy(() -> new CountAction().execute(new CountInput(CollaborationTables.ATTACHMENT))).isInstanceOf(QPermissionDeniedException.class); + assertThat(new CountAction().execute(new CountInput(CollaborationTables.ATTACHMENT).withFilter(pin("1"))).getCount()).isZero(); + assertThat(instance.getTable(CollaborationTables.ATTACHMENT).getDisabledCapabilities()).contains(Capability.TABLE_COUNT, Capability.QUERY_STATS); + } + + /******************************************************************************* + ** Fresh default metadata still detects retained rows of invisible parents. + *******************************************************************************/ + @Test + void persistedRetentionRequiresPinsWithoutAnInstalledPolicyMarker() throws Exception + { + login("alice", "collaboration.view"); + var input = new com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput(CollaborationTables.ATTACHMENT) + .withRecords(java.util.List.of(new com.kingsrook.qqq.backend.core.model.data.QRecord().withValue("tableName", "document") + .withValue("recordId", "3").withValue("retained", true).withValue("fileName", "retained.txt") + .withValue("contentType", "text/plain").withValue("sizeBytes", 1L).withValue("sha256", "a".repeat(64)) + .withValue("bucketName", "fixture").withValue("storageKey", "fixture-retained").withValue("createdByUserId", "alice"))); + input.setTableMetaData(com.kingsrook.qbits.collaboration.CollaborationRecords.cleanupMetadata(CollaborationTables.ATTACHMENT)); + input.withPermissionBypass("Test fixture seeds retained owner metadata without an installed lifecycle."); + assertThat(new com.kingsrook.qqq.backend.core.actions.tables.InsertAction().execute(input).getRecords()).allSatisfy(row -> assertThat(row.getErrors()).isNullOrEmpty()); + assertThatThrownBy(() -> new CountAction().execute(new CountInput(CollaborationTables.ATTACHMENT))).isInstanceOf(QPermissionDeniedException.class); + assertThat(new CountAction().execute(new CountInput(CollaborationTables.ATTACHMENT).withFilter(pin("1"))).getCount()).isZero(); + } + + /******************************************************************************* + ** Exact equality on both canonical reference fields is required. + *******************************************************************************/ + private QQueryFilter pin(String id) + { + return (new QQueryFilter().withCriteria("tableName", QCriteriaOperator.EQUALS, "document").withCriteria("recordId", QCriteriaOperator.EQUALS, id)); + } + + /******************************************************************************* + ** An ordinary generic owner can opt in without depending on Record Files. + *******************************************************************************/ + public static class PinnedPolicy extends AttachmentLifecyclePolicy + { + /******************************************************************************* + ** Constant metadata capability policy, no request context needed. + *******************************************************************************/ + @Override + public boolean requiresPinnedAggregateParent() + { + return (true); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentMediaPolicyTest.java b/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentMediaPolicyTest.java new file mode 100644 index 0000000..43b042d --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentMediaPolicyTest.java @@ -0,0 +1,60 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.collaboration.attachments; + +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; + +/******************************************************************************* + ** Declared media types are client-controlled; only passive formats may be + ** served inline, everything else is an opaque attachment. + *******************************************************************************/ +class AttachmentMediaPolicyTest +{ + /*************************************************************************** + ** Active, malformed and missing declarations are opaque attachments. + ***************************************************************************/ + @Test + void activeMalformedAndMissingTypesAreOpaqueAttachments() + { + for(String declared : new String[] { "text/html", "TEXT/HTML; charset=utf-8", "image/svg+xml", "application/xhtml+xml", "text/xml", "application/xml", + "text/javascript", "application/javascript", "weird", "", " ", "a/b/c", "text/html\r\nX-Injected: 1", null }) + { + assertThat(AttachmentMediaPolicy.storedType(declared)).as(String.valueOf(declared)).isEqualTo("application/octet-stream"); + assertThat(AttachmentMediaPolicy.disposition(declared)).as(String.valueOf(declared)).isEqualTo("attachment"); + } + } + + /*************************************************************************** + ** Passive raster, PDF, text and media types keep their normalized type inline. + ***************************************************************************/ + @Test + void passiveTypesAreNormalizedAndInline() + { + assertThat(AttachmentMediaPolicy.storedType("IMAGE/PNG; q=1")).isEqualTo("image/png"); + assertThat(AttachmentMediaPolicy.disposition("IMAGE/PNG; q=1")).isEqualTo("inline"); + for(String declared : new String[] { "image/jpeg", "image/gif", "image/webp", "application/pdf", "text/plain", "audio/mpeg", "video/mp4" }) + { + assertThat(AttachmentMediaPolicy.storedType(declared)).isEqualTo(declared); + assertThat(AttachmentMediaPolicy.disposition(declared)).isEqualTo("inline"); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentPaginationJdbcTest.java b/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentPaginationJdbcTest.java new file mode 100644 index 0000000..2007aba --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentPaginationJdbcTest.java @@ -0,0 +1,174 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + + +import java.sql.DriverManager; +import java.util.ArrayList; +import java.util.List; +import java.util.UUID; +import com.kingsrook.qbits.collaboration.PersonalServiceFixture; +import com.kingsrook.qbits.collaboration.attachments.uploadjournal.MemoryUploadJournal; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.actions.interfaces.QueryInterface; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleDispatcher; +import com.kingsrook.qqq.backend.module.rdbms.RDBMSBackendModule; +import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSQueryAction; +import com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Real native H2 attachment queries; host permissions remain the fixture's + ** actual memory-backed parent policy. No content is stored or signed here. + *******************************************************************************/ +class AttachmentPaginationJdbcTest extends PersonalServiceFixture +{ + private AttachmentsService service; + + /******************************************************************************* + ** Retain the complete owner table shape, including required fields. Observe + ** the bound at the native backend and execute the real RDBMS query unchanged. + *******************************************************************************/ + @BeforeEach + void database() throws Exception + { + ConnectionManager.closeConnectionProviders(); + var backend = new RDBMSBackendMetaData().withName("page_database").withVendor("h2").withUsername("sa") + .withJdbcUrl("jdbc:h2:mem:attachment_pages_" + UUID.randomUUID() + ";DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=FALSE"); + QBackendModuleDispatcher.registerBackendModule(new RecordingBackend()); + backend.setBackendType(RecordingBackend.class); + instance.addBackend(backend); + var table = instance.getTable(CollaborationTables.ATTACHMENT); + table.setBackendName(backend.getName()); + try(var connection = DriverManager.getConnection(backend.getJdbcUrl(), "sa", ""); var statement = connection.createStatement()) + { + List columns = new ArrayList<>(); + for(var field : table.getFields().values()) + { + String type = switch(field.getType()) + { + case INTEGER -> "INTEGER"; + case LONG -> "BIGINT"; + case BOOLEAN -> "BOOLEAN"; + case DATE_TIME -> "TIMESTAMP"; + case STRING, TEXT -> "VARCHAR(4096)"; + default -> throw new IllegalStateException("Unhandled fixture field " + field.getName()); + }; + columns.add(field.getBackendName() + " " + type + (field.getName().equals("id") ? " PRIMARY KEY" : "") + + (Boolean.TRUE.equals(field.getIsRequired()) ? " NOT NULL" : "")); + } + statement.execute("CREATE TABLE collaboration_attachment (" + String.join(",", columns) + ")"); + for(int id = 1; id <= 8; id++) + { + statement.execute("INSERT INTO collaboration_attachment (id,table_name,record_id,file_name,content_type,size_bytes,sha256,bucket_name,storage_key,created_by_user_id,parent_deleted,retained,create_date) VALUES (" + + id + ",'document','" + (id == 8 ? "2" : "1") + "','file-" + id + "','text/plain',1,'" + "a".repeat(64) + + "','test','key-" + id + "','alice'," + (id == 7) + ",TRUE,TIMESTAMP '2026-10-31 00:00:00')"); + } + } + service = new AttachmentsService(config, new AttachmentStorage(), new AttachmentLifecyclePolicy(), new MemoryUploadJournal()); + RecordingBackend.bounds.clear(); + } + + /******************************************************************************* + ** Close only fixture native providers after the query assertions. + *******************************************************************************/ + @AfterEach + void closeDatabase() + { + ConnectionManager.closeConnectionProviders(); + } + + /******************************************************************************* + ** SQL paging preserves deterministic ties, exact source scope and retained + ** metadata while returning only the requested page plus its lookahead. + *******************************************************************************/ + @Test + void boundedSqlPagesKeepTiesSourceIdentityAndRetainedMetadata() throws Exception + { + var first = service.listPage("document", "01", 0, 2); + assertThat(first.items()).extracting(AttachmentEntry::id).containsExactly(6, 5); + assertThat(first.hasMore()).isTrue(); + assertThat(first.items()).allSatisfy(item -> assertThat(item.canDelete()).isFalse()); + var second = service.listPage("document", 1, 2, 2); + assertThat(second.items()).extracting(AttachmentEntry::id).containsExactly(4, 3); + var last = service.listPage("document", 1, 4, 2); + assertThat(last.items()).extracting(AttachmentEntry::id).containsExactly(2, 1); + assertThat(last.hasMore()).isFalse(); + assertThat(service.listPage("document", 1, 6, 2).items()).isEmpty(); + assertThat(RecordingBackend.bounds).containsExactly("0:3:3", "2:3:3", "4:3:2", "6:3:0"); + assertThat(service.listPage("document", 2, 0, 100).items()).extracting(AttachmentEntry::id).containsExactly(8); + assertThat(RecordingBackend.bounds.getLast()).isEqualTo("0:101:1"); + } + + /******************************************************************************* + ** Current parent revocation and invalid bounds fail before attachment SQL. + *******************************************************************************/ + @Test + void revocationAndInvalidBoundsNeverReachAttachmentQuery() throws Exception + { + service.listPage("document", 1, 0, 1); + RecordingBackend.bounds.clear(); + QContext.getQSession().getSecurityKeyValues().put("tenant", List.of("b")); + assertThatThrownBy(() -> service.listPage("document", 1, 1, 1)).hasMessage("Not found"); + assertThatThrownBy(() -> service.listPage("document", 1, -1, 1)).hasMessageContaining("Offset"); + assertThatThrownBy(() -> service.listPage("document", 1, 0, 101)).hasMessageContaining("limit"); + assertThat(RecordingBackend.bounds).isEmpty(); + } + + /******************************************************************************* + ** Instrument only attachment query inputs/results; delegate actual SQL to QQQ. + *******************************************************************************/ + public static class RecordingBackend extends RDBMSBackendModule + { + static final List bounds = new ArrayList<>(); + + /******************************************************************************* + ** Isolated dispatcher name; never replace the production RDBMS module. + *******************************************************************************/ + @Override + public String getBackendType() + { + return ("attachment_pagination_test"); + } + + /******************************************************************************* + ** Real RDBMS execution must return no more than the requested lookahead. + *******************************************************************************/ + @Override + public QueryInterface getQueryInterface() + { + return (input -> + { + var output = new RDBMSQueryAction().execute(input); + bounds.add(input.getFilter().getSkip() + ":" + input.getFilter().getLimit() + ":" + output.getRecords().size()); + return (output); + }); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentTransactionsPostgresTest.java b/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentTransactionsPostgresTest.java new file mode 100644 index 0000000..042f15f --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentTransactionsPostgresTest.java @@ -0,0 +1,1267 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.collaboration.attachments; + + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.sql.Connection; +import java.sql.DriverManager; +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.collaboration.CollaborationSchema; +import com.kingsrook.qbits.collaboration.PersonalServiceFixture; +import com.kingsrook.qbits.objectstorage.api.PutRequest; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess; +import com.kingsrook.qqq.backend.module.postgres.model.metadata.PostgreSQLBackendMetaData; +import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction; +import com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.wait.strategy.Wait; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Real PostgreSQL transaction boundaries with deterministic storage fault injection. + *******************************************************************************/ +class AttachmentTransactionsPostgresTest extends PersonalServiceFixture +{ + private static GenericContainer postgres; + private RecordingStorage storage; + private AttachmentsService service; + private static AttachmentCleanupQueue processQueue; + + + /******************************************************************************* + ** An isolated database container is required; never use an application database. + *******************************************************************************/ + @BeforeAll + static void startPostgres() + { + postgres = new GenericContainer<>("postgres:17-alpine") + .withEnv("POSTGRES_PASSWORD", "attachment-fixture-only").withExposedPorts(5432) + .waitingFor(Wait.forLogMessage(".*database system is ready to accept connections.*", 2)); + postgres.start(); + } + + + /******************************************************************************* + ** Stop only the database owned by this fixture. + *******************************************************************************/ + @AfterAll + static void stopPostgres() + { + if(postgres != null) + { + postgres.stop(); + } + } + + + /******************************************************************************* + ** Reuse the authorization fixture, replacing its backend with real PostgreSQL. + *******************************************************************************/ + @Override + @BeforeEach + public void setUp() throws Exception + { + super.setUp(); + instance.getBackends().remove("memory"); + instance.addBackend(new PostgreSQLBackendMetaData().withName("memory").withHostName(postgres.getHost()) + .withPort(postgres.getMappedPort(5432)).withDatabaseName("postgres") + .withUsername("postgres").withPassword("attachment-fixture-only")); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("DROP SCHEMA public CASCADE; CREATE SCHEMA public"); + sql.execute(CollaborationSchema.postgresScript(config)); + sql.execute("CREATE TABLE document (id INTEGER PRIMARY KEY, title TEXT, tenant TEXT)"); + sql.execute("INSERT INTO document VALUES (1, 'Visible', 'a'), (3, 'Private', 'b')"); + } + storage = new RecordingStorage(); + service = new AttachmentsService(config, storage); + } + + + /******************************************************************************* + ** Close owner connections before the isolated schema is replaced. + *******************************************************************************/ + @Override + public void tearDown() + { + ConnectionManager.closeConnectionProviders(instance); + super.tearDown(); + } + + + /******************************************************************************* + ** An independently visible intent precedes external I/O, but active uploads + ** cannot be claimed by a cleanup worker on another database connection. + *******************************************************************************/ + @Test + void uploadIntentPrecedesPutAndProtectsActiveUpload() throws Exception + { + storage.beforePut = () -> + { + try + { + assertThat(pendingCount()).isEqualTo(1); + var context = new com.kingsrook.qqq.backend.core.context.CapturedContext(instance, + new com.kingsrook.qqq.backend.core.model.session.QSystemUserSession()); + QContext.withTemporaryContext(context, () -> assertThat(new AttachmentCleanupQueue(storage).retryPending()).isZero()); + assertThat(storage.deleteAttempts).hasValue(0); + } + catch(Exception e) + { + throw (new IllegalStateException(e)); + } + }; + upload(); + assertThat(rowCount()).isEqualTo(1); + assertThat(pendingCount()).isZero(); + assertThat(storage.objects).hasSize(1); + } + + + /******************************************************************************* + ** Rejecting durable intent creation must prevent all provider I/O. + *******************************************************************************/ + @Test + void rejectedUploadIntentPreventsProviderWrite() throws Exception + { + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("CREATE FUNCTION reject_upload_intent() RETURNS trigger LANGUAGE plpgsql AS $$ " + + "BEGIN RAISE EXCEPTION 'injected upload intent rejection'; END $$"); + sql.execute("CREATE TRIGGER reject_upload_intent BEFORE INSERT ON collaboration_attachment_cleanup " + + "FOR EACH ROW EXECUTE FUNCTION reject_upload_intent()"); + } + var puts = new java.util.concurrent.atomic.AtomicInteger(); + storage.beforePut = puts::incrementAndGet; + assertThatThrownBy(this::upload).isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QException.class); + assertThat(puts).hasValue(0); + assertThat(storage.objects).isEmpty(); + assertThat(storage.deleteAttempts).hasValue(0); + assertThat(rowCount()).isZero(); + assertThat(pendingCount()).isZero(); + assertThat(QContext.getQBackendTransaction()).isNull(); + } + + + /******************************************************************************* + ** Completed provider I/O remains protected until the outer owner commits. + *******************************************************************************/ + @Test + void outerTransactionProtectsUploadedBytesUntilCommit() throws Exception + { + try(var tx = new RDBMSTransaction(connect())) + { + QContext.setTransaction(tx); + upload(); + assertThat(rowCount()).isZero(); + assertThat(pendingCount()).isEqualTo(1); + var context = new com.kingsrook.qqq.backend.core.context.CapturedContext(instance, + new com.kingsrook.qqq.backend.core.model.session.QSystemUserSession()); + QContext.withTemporaryContext(context, () -> assertThat(new AttachmentCleanupQueue(storage).retryPending()).isZero()); + assertThat(storage.deleteAttempts).hasValue(0); + assertThat(storage.objects).hasSize(1); + tx.commit(); + QContext.clearTransaction(); + } + assertThat(rowCount()).isEqualTo(1); + assertThat(pendingCount()).isZero(); + assertThat(storage.objects).hasSize(1); + } + + + /******************************************************************************* + ** A separate JVM halts after durable provider bytes, without callbacks or finally. + *******************************************************************************/ + @Test + void abruptlyTerminatedJvmLeavesRecoverableUpload(@org.junit.jupiter.api.io.TempDir java.nio.file.Path directory) throws Exception + { + String classpath = System.getProperty("surefire.test.class.path", System.getProperty("java.class.path")); + java.nio.file.Path arguments = directory.resolve("java.args"); + java.nio.file.Files.writeString(arguments, "-cp\n\"" + classpath.replace('\\', '/') + "\"\n" + + UploadCrashProbe.class.getName() + "\n" + postgres.getHost() + "\n" + postgres.getMappedPort(5432) + + "\n\"" + directory.toString().replace('\\', '/') + "\"\n"); + java.nio.file.Path executable = java.nio.file.Path.of(System.getProperty("java.home"), "bin", + System.getProperty("os.name").startsWith("Windows") ? "java.exe" : "java"); + Process child = new ProcessBuilder(executable.toString(), "@" + arguments) + .redirectErrorStream(true).redirectOutput(directory.resolve("child.log").toFile()).start(); + try + { + assertThat(child.waitFor(45, java.util.concurrent.TimeUnit.SECONDS)).isTrue(); + assertThat(child.exitValue()).as(java.nio.file.Files.readString(directory.resolve("child.log"))).isEqualTo(73); + } + finally + { + if(child.isAlive()) + { + child.destroyForcibly(); + assertThat(child.waitFor(10, java.util.concurrent.TimeUnit.SECONDS)).isTrue(); + } + } + assertThat(rowCount()).isZero(); + assertThat(pendingCount()).isEqualTo(1); + java.nio.file.Path object = directory.resolve("uploaded-object"); + assertThat(java.nio.file.Files.readAllBytes(object)).containsExactly(1, 2, 3); + admin(); + assertThat(new AttachmentCleanupQueue(new CrashFileStorage(directory, false)).retryPending()).isEqualTo(1); + assertThat(java.nio.file.Files.exists(object)).isFalse(); + assertThat(pendingCount()).isZero(); + assertThat(new AttachmentCleanupQueue(new CrashFileStorage(directory, false)).retryPending()).isZero(); + } + + + /******************************************************************************* + ** Standalone child uses the real service against the parent's disposable DB. + *******************************************************************************/ + public static class UploadCrashProbe extends PersonalServiceFixture + { + /*************************************************************************** + ** Intentionally never returns: the provider halts this child JVM. + ***************************************************************************/ + public static void main(String[] args) throws Exception + { + var probe = new UploadCrashProbe(); + probe.setUp(); + probe.instance.getBackends().remove("memory"); + probe.instance.addBackend(new PostgreSQLBackendMetaData().withName("memory").withHostName(args[0]) + .withPort(Integer.parseInt(args[1])).withDatabaseName("postgres") + .withUsername("postgres").withPassword("attachment-fixture-only")); + new AttachmentsService(probe.config, new CrashFileStorage(java.nio.file.Path.of(args[2]), true)) + .upload("document", 1, "hello.txt", "text/plain", new ByteArrayInputStream(new byte[] { 1, 2, 3 }), null); + throw (new IllegalStateException("Crash fixture unexpectedly returned")); + } + } + + + /******************************************************************************* + ** One fixture-owned object persists across JVM death independently of SQL. + *******************************************************************************/ + private static class CrashFileStorage extends AttachmentStorage + { + private final java.nio.file.Path directory; + private final boolean haltAfterPut; + + /*************************************************************************** + ** Scope every file to the test runner's disposable directory. + ***************************************************************************/ + private CrashFileStorage(java.nio.file.Path directory, boolean haltAfterPut) + { + this.directory = directory; + this.haltAfterPut = haltAfterPut; + } + + /*************************************************************************** + ** Bound this fixture exactly as the normal injected provider. + ***************************************************************************/ + @Override + public long maxBytes(String name) + { + return (1024L); + } + + /*************************************************************************** + ** Persist bytes and their locator, then terminate without shutdown hooks. + ***************************************************************************/ + @Override + public void put(String name, PutRequest request) + { + try + { + java.nio.file.Files.writeString(directory.resolve("locator"), name + "/" + request.getKey()); + java.nio.file.Files.write(directory.resolve("uploaded-object"), request.getStream().readAllBytes()); + if(haltAfterPut) + { + Runtime.getRuntime().halt(73); + } + } + catch(IOException e) + { + throw (new IllegalStateException(e)); + } + } + + /*************************************************************************** + ** The test runner confirms the only writer JVM has exited before recovery. + ***************************************************************************/ + @Override + public boolean cleanupUpload(String name, String key) + { + delete(name, key); + return (!haltAfterPut); + } + + /*************************************************************************** + ** Verify recovery uses the original locator before deleting fixture bytes. + ***************************************************************************/ + @Override + public void delete(String name, String key) + { + try + { + assertThat(java.nio.file.Files.readString(directory.resolve("locator"))).isEqualTo(name + "/" + key); + java.nio.file.Files.deleteIfExists(directory.resolve("uploaded-object")); + } + catch(IOException e) + { + throw (new IllegalStateException(e)); + } + } + } + + + /******************************************************************************* + ** Losing the upload stack before metadata and rollback callbacks still leaves + ** independently committed intent for a later service instance to recover. + *******************************************************************************/ + @Test + void interruptedUploadSurvivesLostRollbackCallbacks() throws Exception + { + storage.afterPut = () -> { throw new AssertionError("injected interrupted upload"); }; + try(var tx = new RDBMSTransaction(connect()) + { + /*********************************************************************** + ** Simulate loss of process-local callbacks, not a killed-JVM test. + ***********************************************************************/ + @Override + public void addAfterRollbackCallback(Runnable callback) + { + } + }) + { + QContext.setTransaction(tx); + assertThatThrownBy(this::upload).isInstanceOf(AssertionError.class); + tx.rollback(); + QContext.clearTransaction(); + } + assertThat(rowCount()).isZero(); + assertThat(pendingCount()).isEqualTo(1); + assertThat(storage.objects).hasSize(1); + admin(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + assertThat(storage.objects).isEmpty(); + assertThat(pendingCount()).isZero(); + } + + + /******************************************************************************* + ** Later outer rollback removes both the SQL row and the uploaded object. + *******************************************************************************/ + @Test + void uploadOuterRollbackCompensatesObject() throws Exception + { + try(var tx = new RDBMSTransaction(connect())) + { + QContext.setTransaction(tx); + upload(); + assertThat(storage.objects).hasSize(1); + tx.rollback(); + QContext.clearTransaction(); + } + assertThat(rowCount()).isZero(); + assertThat(storage.objects).isEmpty(); + } + + + /******************************************************************************* + ** A rejected outer transaction cannot delete bytes from an existing attachment. + *******************************************************************************/ + @Test + void deleteOuterRollbackPreservesAttachment() throws Exception + { + AttachmentEntry entry = upload(); + try(var tx = new RDBMSTransaction(connect())) + { + QContext.setTransaction(tx); + service.delete(List.of(entry.id())); + tx.rollback(); + QContext.clearTransaction(); + } + assertThat(rowCount()).isEqualTo(1); + assertThat(storage.objects).hasSize(1); + assertThat(pendingCount()).isZero(); + } + + + /******************************************************************************* + ** A committed deletion removes metadata and bytes after the transaction commits. + *******************************************************************************/ + @Test + void deleteCommitRemovesAttachment() throws Exception + { + AttachmentEntry entry = upload(); + try(var tx = new RDBMSTransaction(connect())) + { + QContext.setTransaction(tx); + service.delete(List.of(entry.id())); + assertThat(storage.objects).hasSize(1); + tx.commit(); + QContext.clearTransaction(); + } + assertThat(rowCount()).isZero(); + assertThat(storage.objects).isEmpty(); + } + + + /******************************************************************************* + ** Database rejection must not leave a visible row whose bytes were already deleted. + *******************************************************************************/ + @Test + void metadataDeleteFailurePreservesDownloadableObject() throws Exception + { + AttachmentEntry entry = upload(); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("CREATE FUNCTION reject_attachment_delete() RETURNS trigger LANGUAGE plpgsql AS $$ " + + "BEGIN RAISE EXCEPTION 'injected metadata delete rejection'; END $$"); + sql.execute("CREATE TRIGGER reject_attachment_delete BEFORE DELETE ON collaboration_attachment " + + "FOR EACH ROW EXECUTE FUNCTION reject_attachment_delete()"); + } + assertThatThrownBy(() -> service.delete(List.of(entry.id()))).isInstanceOf(Exception.class); + assertThat(rowCount()).isEqualTo(1); + assertThat(storage.objects).as("a retained visible attachment must retain its bytes").hasSize(1); + assertThat(pendingCount()).isZero(); + } + + + /******************************************************************************* + ** A provider failure leaves a durable locator; a new worker can retry after restart. + *******************************************************************************/ + @Test + void storageDeleteFailureRetainsIntentAndRetrySucceeds() throws Exception + { + AttachmentEntry entry = upload(); + storage.failDelete = true; + assertThatThrownBy(() -> service.delete(List.of(entry.id()))).isInstanceOf(IllegalStateException.class); + assertThat(rowCount()).isZero(); + assertThat(pendingCount()).isEqualTo(1); + assertThat(storage.objects).hasSize(1); + storage.failDelete = false; + assertThatThrownBy(() -> new AttachmentCleanupQueue(storage).retryPending()) + .isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + admin(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + assertThat(rowCount()).isZero(); + assertThat(pendingCount()).isZero(); + assertThat(storage.objects).isEmpty(); + } + + + /******************************************************************************* + ** A database commit survives loss of every in-process post-commit callback. + *******************************************************************************/ + @Test + void committedIntentSurvivesLostCallback() throws Exception + { + AttachmentEntry entry = upload(); + try(var tx = new RDBMSTransaction(connect()) + { + /*********************************************************************** + ** Simulate process loss after commit, before any callback can execute. + ***********************************************************************/ + @Override + public void addAfterCommitCallback(Runnable callback) + { + } + }) + { + QContext.setTransaction(tx); + service.delete(List.of(entry.id())); + tx.commit(); + QContext.clearTransaction(); + } + assertThat(rowCount()).isZero(); + assertThat(pendingCount()).isEqualTo(1); + assertThat(storage.objects).hasSize(1); + admin(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + assertThat(storage.objects).isEmpty(); + assertThat(pendingCount()).isZero(); + } + + + /******************************************************************************* + ** Losing queue acknowledgement after deleting bytes must remain replayable. + *******************************************************************************/ + @Test + void queueAcknowledgementFailureIsReplayable() throws Exception + { + AttachmentEntry entry = upload(); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("CREATE FUNCTION reject_cleanup_ack() RETURNS trigger LANGUAGE plpgsql AS $$ " + + "BEGIN RAISE EXCEPTION 'injected cleanup acknowledgement rejection'; END $$"); + sql.execute("CREATE TRIGGER reject_cleanup_ack BEFORE DELETE ON collaboration_attachment_cleanup " + + "FOR EACH ROW EXECUTE FUNCTION reject_cleanup_ack()"); + } + assertThatThrownBy(() -> service.delete(List.of(entry.id()))).isInstanceOf(Exception.class); + assertThat(rowCount()).isZero(); + assertThat(storage.objects).isEmpty(); + assertThat(pendingCount()).isEqualTo(1); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("DROP TRIGGER reject_cleanup_ack ON collaboration_attachment_cleanup"); + } + admin(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isZero(); + assertThat(pendingCount()).isZero(); + } + + + /******************************************************************************* + ** Queue observations use independent committed database state. + *******************************************************************************/ + private int pendingCount() throws Exception + { + try(Connection connection = connect(); var sql = connection.createStatement(); + var rows = sql.executeQuery("SELECT COUNT(*) FROM collaboration_attachment_cleanup")) + { + rows.next(); + return (rows.getInt(1)); + } + } + + + /******************************************************************************* + ** Failed first-page entries must not starve later healthy cleanup requests. + *******************************************************************************/ + @Test + void failedBatchDoesNotStarveLaterRequests() throws Exception + { + try(Connection connection = connect(); var sql = connection.prepareStatement( + "INSERT INTO collaboration_attachment_cleanup (attachment_id, bucket_name, storage_key) VALUES (?, 'collaboration', ?)")) + { + for(int id = 1; id <= 101; id++) + { + String key = "queued/" + id; + sql.setInt(1, id); + sql.setString(2, key); + sql.executeUpdate(); + storage.objects.put("collaboration/" + key, new byte[] { 1 }); + if(id <= 100) + { + storage.failedKeys.add(key); + } + } + } + admin(); + assertThatThrownBy(() -> new AttachmentCleanupQueue(storage).retryPending()).isInstanceOf(Exception.class); + assertThat(storage.deleteAttempts.get()).isEqualTo(100); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + assertThat(storage.deleteAttempts.get()).isEqualTo(101); + assertThat(pendingCount()).isEqualTo(100); + assertThat(storage.objects).doesNotContainKey("collaboration/queued/101"); + storage.failedKeys.clear(); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("UPDATE collaboration_attachment_cleanup SET next_attempt_at = CURRENT_TIMESTAMP - INTERVAL '1 second'"); + } + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(100); + assertThat(pendingCount()).isZero(); + assertThat(storage.objects).isEmpty(); + } + + + /******************************************************************************* + ** Failed rollback compensation must preserve a durable locator for later cleanup. + *******************************************************************************/ + @Test + void failedUploadRollbackCompensationRemainsRetryable() throws Exception + { + try(var tx = new RDBMSTransaction(connect())) + { + QContext.setTransaction(tx); + upload(); + storage.failDelete = true; + tx.rollback(); + QContext.clearTransaction(); + } + assertThat(rowCount()).isZero(); + assertThat(storage.objects).hasSize(1); + assertThat(pendingCount()).isEqualTo(1); + storage.failDelete = false; + admin(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + assertThat(storage.objects).isEmpty(); + assertThat(pendingCount()).isZero(); + } + + + /******************************************************************************* + ** Upgrading the first queue shape preserves pending deletions and is repeatable. + *******************************************************************************/ + @Test + void cleanupSchemaUpgradePreservesPendingRequests() throws Exception + { + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("ALTER TABLE collaboration_attachment_cleanup DROP COLUMN next_attempt_at"); + sql.execute("ALTER TABLE collaboration_attachment_cleanup ALTER COLUMN attachment_id SET NOT NULL"); + sql.execute("CREATE UNIQUE INDEX uq_collaboration_attachment_cleanup_1 ON collaboration_attachment_cleanup (attachment_id)"); + sql.execute("INSERT INTO collaboration_attachment_cleanup (attachment_id, bucket_name, storage_key) VALUES (42, 'collaboration', 'pending/42')"); + sql.execute(CollaborationSchema.postgresScript(config)); + sql.execute(CollaborationSchema.postgresScript(config)); + sql.execute("INSERT INTO collaboration_attachment_cleanup (bucket_name, storage_key) VALUES ('collaboration', 'rollback/one'), ('collaboration', 'rollback/two')"); + } + assertThat(pendingCount()).isEqualTo(3); + } + + + /******************************************************************************* + ** A provider may store bytes and then fail before acknowledging the upload. + *******************************************************************************/ + @Test + void ambiguousUploadFailureRetainsCompensation() throws Exception + { + storage.failPutAfterWrite = true; + storage.failDelete = true; + assertThatThrownBy(this::upload).isInstanceOf(IllegalStateException.class) + .hasMessage("injected upload acknowledgement failure"); + assertThat(rowCount()).isZero(); + assertThat(storage.objects).hasSize(1); + assertThat(pendingCount()).isEqualTo(1); + storage.failDelete = false; + admin(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + assertThat(storage.objects).isEmpty(); + assertThat(pendingCount()).isZero(); + } + + + /******************************************************************************* + ** Object absence is not proof that a remote upload has stopped. + *******************************************************************************/ + @Test + void genericUploadCleanupDoesNotClaimSettlement() + { + java.util.concurrent.atomic.AtomicInteger deletes = new java.util.concurrent.atomic.AtomicInteger(); + AttachmentStorage uncertain = new AttachmentStorage() + { + @Override + public void delete(String name, String key) + { + deletes.incrementAndGet(); + } + }; + assertThat(uncertain.cleanupUpload("bucket", "key")).isFalse(); + assertThat(deletes.get()).isEqualTo(1); + } + + + /******************************************************************************* + ** Manual settlement retains an attributable receipt and does not retry it. + *******************************************************************************/ + @Test + void operatorSettlementRetainsReceiptAndStopsRetries() throws Exception + { + int id = settlementIntent(); + admin(); + new AttachmentCleanupQueue(storage).resolveUpload(id, " Provider case 123 confirms PUT completion. "); + assertThat(storage.objects).isEmpty(); + try(Connection connection = connect(); var sql = connection.createStatement(); + var row = sql.executeQuery("SELECT resolved_by, settlement_evidence, resolved_at FROM collaboration_attachment_cleanup WHERE id = " + id)) + { + assertThat(row.next()).isTrue(); + assertThat(row.getString(1)).isEqualTo("admin"); + assertThat(row.getString(2)).isEqualTo("Provider case 123 confirms PUT completion."); + assertThat(row.getTimestamp(3)).isNotNull(); + } + int attempts = storage.deleteAttempts.get(); + new AttachmentCleanupQueue(storage).resolveUpload(id, "A repeat must not overwrite the original receipt."); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isZero(); + assertThat(storage.deleteAttempts.get()).isEqualTo(attempts); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute(CollaborationSchema.postgresScript(config)); + try(var row = sql.executeQuery("SELECT settlement_evidence FROM collaboration_attachment_cleanup WHERE id = " + id)) + { + assertThat(row.next()).isTrue(); + assertThat(row.getString(1)).isEqualTo("Provider case 123 confirms PUT completion."); + } + } + } + + + /******************************************************************************* + ** Deny unauthenticated, non-admin and scheduled callers before provider I/O. + *******************************************************************************/ + @Test + void operatorSettlementRequiresIdentityPermissionAndEvidence() throws Exception + { + int id = settlementIntent(); + AttachmentCleanupQueue queue = new AttachmentCleanupQueue(storage); + assertThatThrownBy(() -> queue.resolveUpload(id, "confirmed")).isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + QContext.init(instance, new com.kingsrook.qqq.backend.core.model.session.QSystemUserSession()); + assertThatThrownBy(() -> queue.resolveUpload(id, "confirmed")).isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + admin(); + assertThatThrownBy(() -> queue.resolveUpload(id, " ")).hasMessageContaining("evidence"); + assertThatThrownBy(() -> queue.resolveUpload(id, "x".repeat(2001))).hasMessageContaining("evidence"); + assertThatThrownBy(() -> queue.resolveUpload(null, "confirmed")).hasMessageContaining("record ID"); + assertThat(storage.deleteAttempts.get()).isZero(); + assertThat(storage.objects).hasSize(1); + } + + + /******************************************************************************* + ** A locked upload cannot be settled by an administrator on another connection. + *******************************************************************************/ + @Test + void operatorSettlementSkipsActiveOwner() throws Exception + { + int id = settlementIntent(); + admin(); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + connection.setAutoCommit(false); + sql.executeQuery("SELECT id FROM collaboration_attachment_cleanup WHERE id = " + id + " FOR UPDATE").close(); + assertThatThrownBy(() -> new AttachmentCleanupQueue(storage).resolveUpload(id, "confirmed")) + .hasMessageContaining("active or no longer exists"); + connection.rollback(); + } + assertThat(storage.deleteAttempts.get()).isZero(); + assertThat(QContext.getQBackendTransaction()).isNull(); + } + + + /******************************************************************************* + ** Even an admin cannot delete a referenced locator or settle normal deletions. + *******************************************************************************/ + @Test + void operatorSettlementProtectsReferencedObjects() throws Exception + { + var attachment = upload(); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.executeUpdate("INSERT INTO collaboration_attachment_cleanup (bucket_name, storage_key) SELECT bucket_name, storage_key FROM collaboration_attachment"); + } + admin(); + int id; + try(Connection connection = connect(); var sql = connection.createStatement(); var row = sql.executeQuery("SELECT id FROM collaboration_attachment_cleanup")) + { + row.next(); + id = row.getInt(1); + } + assertThatThrownBy(() -> new AttachmentCleanupQueue(storage).resolveUpload(id, "confirmed")) + .hasMessageContaining("still references"); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.executeUpdate("UPDATE collaboration_attachment_cleanup SET attachment_id = " + attachment.id()); + } + assertThatThrownBy(() -> new AttachmentCleanupQueue(storage).resolveUpload(id, "confirmed")) + .hasMessageContaining("normal cleanup retry"); + assertThat(storage.deleteAttempts.get()).isZero(); + assertThat(storage.objects).hasSize(1); + } + + + /******************************************************************************* + ** Provider failure or rejected receipt persistence leaves a retryable locator. + *******************************************************************************/ + @Test + void operatorSettlementFailureRetainsIntent() throws Exception + { + int id = settlementIntent(); + admin(); + storage.failDelete = true; + assertThatThrownBy(() -> new AttachmentCleanupQueue(storage).resolveUpload(id, "confirmed")).isInstanceOf(Exception.class); + assertThat(storage.objects).hasSize(1); + storage.failDelete = false; + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("CREATE FUNCTION reject_receipt() RETURNS trigger LANGUAGE plpgsql AS 'BEGIN RAISE EXCEPTION ''receipt failure''; END'; CREATE TRIGGER reject_receipt BEFORE UPDATE ON collaboration_attachment_cleanup FOR EACH ROW EXECUTE FUNCTION reject_receipt()"); + } + assertThatThrownBy(() -> new AttachmentCleanupQueue(storage).resolveUpload(id, "confirmed")).isInstanceOf(Exception.class); + assertThat(storage.objects).isEmpty(); + try(Connection connection = connect(); var sql = connection.createStatement(); var row = sql.executeQuery("SELECT resolved_at FROM collaboration_attachment_cleanup WHERE id = " + id)) + { + assertThat(row.next()).isTrue(); + assertThat(row.getTimestamp(1)).isNull(); + } + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("DROP TRIGGER reject_receipt ON collaboration_attachment_cleanup"); + } + new AttachmentCleanupQueue(storage).resolveUpload(id, "confirmed after database recovery"); + assertThat(pendingCount()).isEqualTo(1); + assertThat(QContext.getQBackendTransaction()).isNull(); + } + + + /******************************************************************************* + ** Fixture-only orphan, after its writer has definitively stopped. + *******************************************************************************/ + private int settlementIntent() throws Exception + { + storage.objects.put("fixture/settled", new byte[] { 1, 2, 3 }); + try(Connection connection = connect(); var sql = connection.createStatement(); var row = sql.executeQuery( + "INSERT INTO collaboration_attachment_cleanup (bucket_name, storage_key) VALUES ('fixture', 'settled') RETURNING id")) + { + row.next(); + return (row.getInt(1)); + } + } + + + /******************************************************************************* + ** Genuine form, Review and Result boundaries; revoked access is checked again. + *******************************************************************************/ + @ParameterizedTest + @ValueSource(strings = { "success", "revoked", "storageFailure" }) + @SuppressWarnings("unchecked") + void operatorSettlementProcess(String scenario) throws Exception + { + int id = settlementIntent(); + int deletionId; + try(Connection connection = connect(); var sql = connection.createStatement(); var row = sql.executeQuery( + "INSERT INTO collaboration_attachment_cleanup (attachment_id, bucket_name, storage_key) VALUES (99, 'fixture', 'normal-deletion') RETURNING id")) + { + row.next(); + deletionId = row.getInt(1); + } + var process = instance.getProcess(ResolveAttachmentUploadProcess.NAME); + process.getBackendStep(StreamedETLWithFrontendProcess.STEP_NAME_EXECUTE).getInputMetaData() + .getFieldThrowing(StreamedETLWithFrontendProcess.FIELD_LOAD_CODE).setDefaultValue(new QCodeReference(FixtureSettlement.class)); + processQueue = new AttachmentCleanupQueue(storage); + try + { + RunProcessInput start = new RunProcessInput(); + start.setProcessName(ResolveAttachmentUploadProcess.NAME); + start.addValue("recordsParam", "recordIds"); + start.addValue("recordIds", id + "," + deletionId); + start.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.BREAK); + assertThatThrownBy(() -> new RunProcessAction().execute(start)) + .isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + admin(); + var form = new RunProcessAction().execute(start); + assertThat(form.getException()).isEmpty(); + start.setProcessUUID(form.getProcessUUID()); + start.setStartAfterStep("evidence"); + start.addValue(ResolveAttachmentUploadProcess.EVIDENCE, "Provider request 123 completed; writer stopped."); + var review = new RunProcessAction().execute(start); + assertThat(review.getException()).isEmpty(); + List reviewLines = (List) review.getValues().get(StreamedETLWithFrontendProcess.FIELD_VALIDATION_SUMMARY); + assertThat(reviewLines).hasSize(2); + assertThat(storage.deleteAttempts.get()).isZero(); + if(scenario.equals("revoked")) + { + login("admin"); + } + storage.failDelete = scenario.equals("storageFailure"); + RunProcessInput go = new RunProcessInput(); + go.setProcessName(ResolveAttachmentUploadProcess.NAME); + go.setProcessUUID(review.getProcessUUID()); + go.setStartAfterStep(StreamedETLWithFrontendProcess.STEP_NAME_REVIEW); + go.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.SKIP); + if(scenario.equals("revoked")) + { + assertThatThrownBy(() -> new RunProcessAction().execute(go)) + .isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + assertThat(storage.deleteAttempts.get()).isZero(); + } + else + { + var result = new RunProcessAction().execute(go); + assertThat(result.getException()).isEmpty(); + List lines = (List) result.getValues().get(StreamedETLWithFrontendProcess.FIELD_PROCESS_SUMMARY); + assertThat(lines).anySatisfy(line -> + { + line.pickMessage(true); + assertThat(line.getMessage()).isEqualTo(scenario.equals("success") ? "uploads resolved" : "uploads unresolved; inspect and retry"); + assertThat(line.getCount()).isEqualTo(1); + }); + } + assertThat(storage.objects).hasSize(scenario.equals("success") ? 0 : 1); + } + finally + { + processQueue = null; + } + } + + + /******************************************************************************* + ** The real process uses the real queue with deterministic fixture storage. + *******************************************************************************/ + public static class FixtureSettlement extends ResolveAttachmentUploadProcess.Apply + { + /******************************************************************************* + ** Instantiated by the standard process loader. + *******************************************************************************/ + public FixtureSettlement() + { + super(processQueue); + } + } + + + /******************************************************************************* + ** A timed-out remote PUT can finish after rollback has already attempted DELETE. + *******************************************************************************/ + @Test + void lateUploadCompletionRemainsRecoverable() throws Exception + { + storage.deferPutUntilAfterFailure = true; + assertThatThrownBy(this::upload).isInstanceOf(IllegalStateException.class) + .hasMessage("injected upload timeout"); + assertThat(rowCount()).isZero(); + assertThat(storage.objects).isEmpty(); + + admin(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isZero(); + assertThat(pendingCount()).isEqualTo(1); + int attempts = storage.deleteAttempts.get(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isZero(); + assertThat(storage.deleteAttempts.get()).as("uncertain writes use persisted backoff").isEqualTo(attempts); + + // The remote service finishes after rollback and an independent cleanup sweep. + storage.deferredPut.run(); + assertThat(storage.objects).hasSize(1); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.executeUpdate("UPDATE collaboration_attachment_cleanup SET next_attempt_at = CURRENT_TIMESTAMP - INTERVAL '1 second'"); + } + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + assertThat(storage.objects).as("late provider bytes must remain discoverable for cleanup").isEmpty(); + assertThat(pendingCount()).isZero(); + } + + + /******************************************************************************* + ** Replay real provider deletes after the database rejects queue acknowledgement. + *******************************************************************************/ + @Test + void minioDeletionReplayPreservesOtherObjects() throws Exception + { + try(var minio = new GenericContainer<>("cgr.dev/chainguard/minio@sha256:dc455cdd4db6e390b1b26514f15ff37681a777bf8dab41aac445a0259f751e56") + .withEnv("MINIO_ROOT_USER", "collaboration-test").withEnv("MINIO_ROOT_PASSWORD", "collaboration-test-password") + .withCommand("server", "/tmp/minio-data").withExposedPorts(9000) + .waitingFor(Wait.forHttp("/minio/health/live").forPort(9000))) + { + minio.start(); + String endpoint = "http://" + minio.getHost() + ":" + minio.getMappedPort(9000); + try(var client = software.amazon.awssdk.services.s3.S3Client.builder() + .endpointOverride(java.net.URI.create(endpoint)).region(software.amazon.awssdk.regions.Region.US_EAST_1) + .forcePathStyle(true).credentialsProvider(software.amazon.awssdk.auth.credentials.StaticCredentialsProvider.create( + software.amazon.awssdk.auth.credentials.AwsBasicCredentials.create("collaboration-test", "collaboration-test-password"))).build()) + { + client.createBucket(b -> b.bucket("collaboration-plain")); + client.createBucket(b -> b.bucket("collaboration-versioned")); + client.putBucketVersioning(b -> b.bucket("collaboration-versioned").versioningConfiguration(v -> + v.status(software.amazon.awssdk.services.s3.model.BucketVersioningStatus.ENABLED))); + System.setProperty("collaboration.minio.access", "collaboration-test"); + System.setProperty("collaboration.minio.secret", "collaboration-test-password"); + var storageConfig = new com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig() + .withBackendName("memory") + .withAllowedSecretRefPatterns(List.of("prop.collaboration.minio.*")) + .withBucketScoping(false) + .withConnection(new com.kingsrook.qbits.objectstorage.config.StorageConnectionConfig("fixture") + .withEndpoint(endpoint).withRegion("us-east-1") + .withAddressingStyle(com.kingsrook.qbits.objectstorage.config.AddressingStyle.PATH) + .withChecksumMode(com.kingsrook.qbits.objectstorage.config.ChecksumMode.WHEN_REQUIRED) + .withAccessKeyRef("${prop.collaboration.minio.access}").withSecretKeyRef("${prop.collaboration.minio.secret}")); + for(String alias : List.of("plain", "versioned")) + { + storageConfig.withBucket(new com.kingsrook.qbits.objectstorage.config.StorageBucketConfig(alias) + .withConnection("fixture").withBucketName("collaboration-" + alias) + .withRole(com.kingsrook.qbits.objectstorage.config.BucketRole.PRIVATE).withIndexed(false)); + } + com.kingsrook.qbits.objectstorage.api.ObjectStorage.configureStandalone(storageConfig); + for(String alias : List.of("plain", "versioned")) + { + config.withAttachmentsBucketName(alias); + service = new AttachmentsService(config, new AttachmentStorage()); + AttachmentEntry removed = upload(); + AttachmentEntry retained = upload(); + String removedKey = storedKey(removed.id()); + String retainedKey = storedKey(retained.id()); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("CREATE OR REPLACE FUNCTION reject_provider_ack() RETURNS trigger LANGUAGE plpgsql AS $$ " + + "BEGIN RAISE EXCEPTION 'injected provider acknowledgement rejection'; END $$"); + sql.execute("CREATE TRIGGER reject_provider_ack BEFORE DELETE ON collaboration_attachment_cleanup " + + "FOR EACH ROW EXECUTE FUNCTION reject_provider_ack()"); + } + assertThatThrownBy(() -> service.delete(List.of(removed.id()))).isInstanceOf(Exception.class); + assertThat(pendingCount()).isEqualTo(1); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("DROP TRIGGER reject_provider_ack ON collaboration_attachment_cleanup"); + } + admin(); + assertThat(new AttachmentCleanupQueue(new AttachmentStorage()).retryPending()).isEqualTo(1); + assertThat(pendingCount()).isZero(); + var bucket = com.kingsrook.qbits.objectstorage.api.ObjectStorage.asSystem("collaboration-test", "replay").bucket(alias); + assertThat(bucket.head(removedKey)).isEmpty(); + assertThat(bucket.head(retainedKey)).isPresent(); + assertThat(client.getObjectAsBytes(b -> b.bucket("collaboration-" + alias).key(retainedKey)).asByteArray()) + .containsExactly((byte) 1, (byte) 2, (byte) 3); + service.delete(List.of(retained.id())); + login("alice"); + } + } + } + finally + { + com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime.reset(); + com.kingsrook.qbits.objectstorage.internal.ConfigSync.reset(); + System.clearProperty("collaboration.minio.access"); + System.clearProperty("collaboration.minio.secret"); + } + } + + + /******************************************************************************* + ** Observe the immutable provider key from committed SQL metadata. + *******************************************************************************/ + private String storedKey(int attachmentId) throws Exception + { + try(Connection connection = connect(); var sql = connection.prepareStatement("SELECT storage_key FROM collaboration_attachment WHERE id = ?")) + { + sql.setInt(1, attachmentId); + try(var rows = sql.executeQuery()) + { + assertThat(rows.next()).isTrue(); + return (rows.getString(1)); + } + } + } + + + /******************************************************************************* + ** Two workers may observe the same committed intent; replay must remain safe. + *******************************************************************************/ + @Test + void concurrentWorkersCanReplayTheSameIntent() throws Exception + { + AttachmentEntry entry = upload(); + storage.failDelete = true; + assertThatThrownBy(() -> service.delete(List.of(entry.id()))).isInstanceOf(IllegalStateException.class); + storage.failDelete = false; + storage.concurrentDeletes = new java.util.concurrent.CountDownLatch(1); + storage.allowDelete = new java.util.concurrent.CountDownLatch(1); + admin(); + var session = QContext.getQSession(); + try(var executor = java.util.concurrent.Executors.newFixedThreadPool(2)) + { + java.util.concurrent.Callable retry = () -> + { + QContext.init(instance, session); + try + { + return (new AttachmentCleanupQueue(storage).retryPending()); + } + finally + { + QContext.clear(); + } + }; + var first = executor.submit(retry); + assertThat(storage.concurrentDeletes.await(10, java.util.concurrent.TimeUnit.SECONDS)).isTrue(); + try + { + var second = executor.submit(retry); + assertThat(second.get(10, java.util.concurrent.TimeUnit.SECONDS)).isZero(); + } + finally + { + storage.allowDelete.countDown(); + } + assertThat(first.get(20, java.util.concurrent.TimeUnit.SECONDS)).isEqualTo(1); + } + assertThat(rowCount()).isZero(); + assertThat(pendingCount()).isZero(); + assertThat(storage.objects).isEmpty(); + } + + + /******************************************************************************* + ** Recovery locations are admin-only and cannot be forged through table writes. + *******************************************************************************/ + @Test + void cleanupQueueIsServiceManaged() throws Exception + { + var query = new com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput( + com.kingsrook.qbits.collaboration.model.CollaborationTables.ATTACHMENT_CLEANUP); + assertThatThrownBy(() -> new com.kingsrook.qqq.backend.core.actions.tables.QueryAction().execute(query)) + .isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + admin(); + assertThat(new com.kingsrook.qqq.backend.core.actions.tables.QueryAction().execute(query).getRecords()).isEmpty(); + var insert = new com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput( + com.kingsrook.qbits.collaboration.model.CollaborationTables.ATTACHMENT_CLEANUP) + .withRecords(List.of(new com.kingsrook.qqq.backend.core.model.data.QRecord())); + assertThatThrownBy(() -> new com.kingsrook.qqq.backend.core.actions.tables.InsertAction().execute(insert)) + .isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + } + + + /******************************************************************************* + ** Use the actual service and its caller's record permissions. + *******************************************************************************/ + private AttachmentEntry upload() throws Exception + { + return (service.upload("document", 1, "hello.txt", "text/plain", new ByteArrayInputStream(new byte[] { 1, 2, 3 }), null)); + } + + + /******************************************************************************* + ** Independent JDBC observation, outside the service transaction. + *******************************************************************************/ + private int rowCount() throws Exception + { + try(Connection connection = connect(); var sql = connection.createStatement(); + var rows = sql.executeQuery("SELECT COUNT(*) FROM collaboration_attachment")) + { + rows.next(); + return (rows.getInt(1)); + } + } + + + /******************************************************************************* + ** Connect only to this disposable fixture's mapped port. + *******************************************************************************/ + private static Connection connect() throws Exception + { + return (DriverManager.getConnection("jdbc:postgresql://" + postgres.getHost() + ":" + + postgres.getMappedPort(5432) + "/postgres", "postgres", "attachment-fixture-only")); + } + + + /******************************************************************************* + ** Preserve bytes and inject provider failure independently of PostgreSQL. + *******************************************************************************/ + private static class RecordingStorage extends AttachmentStorage + { + private final Map objects = new java.util.concurrent.ConcurrentHashMap<>(); + private boolean failDelete; + private boolean failPutAfterWrite; + private boolean deferPutUntilAfterFailure; + private Runnable deferredPut; + private Runnable beforePut; + private Runnable afterPut; + private final java.util.Set failedKeys = new java.util.HashSet<>(); + private final java.util.concurrent.atomic.AtomicInteger deleteAttempts = new java.util.concurrent.atomic.AtomicInteger(); + private java.util.concurrent.CountDownLatch concurrentDeletes; + private java.util.concurrent.CountDownLatch allowDelete; + + /*************************************************************************** + ** Keep the production service's upload bound active. + ***************************************************************************/ + @Override + public long maxBytes(String name) + { + return (1024L); + } + + /*************************************************************************** + ** Consume the same upload stream a real provider receives. + ***************************************************************************/ + @Override + public void put(String name, PutRequest request) + { + try + { + if(beforePut != null) + { + beforePut.run(); + } + byte[] bytes = request.getStream().readAllBytes(); + if(deferPutUntilAfterFailure) + { + deferredPut = () -> + { + objects.put(name + "/" + request.getKey(), bytes); + deferPutUntilAfterFailure = false; + }; + throw (new IllegalStateException("injected upload timeout")); + } + objects.put(name + "/" + request.getKey(), bytes); + if(afterPut != null) + { + afterPut.run(); + } + if(failPutAfterWrite) + { + throw (new IllegalStateException("injected upload acknowledgement failure")); + } + } + catch(IOException e) + { + throw (new IllegalStateException(e)); + } + } + + /*************************************************************************** + ** This synchronous fixture knows whether its deferred PUT has completed. + ***************************************************************************/ + @Override + public boolean cleanupUpload(String name, String key) + { + delete(name, key); + return (!deferPutUntilAfterFailure); + } + + /*************************************************************************** + ** Simulate a provider refusing deletion without losing stored bytes. + ***************************************************************************/ + @Override + public void delete(String name, String key) + { + deleteAttempts.incrementAndGet(); + if(concurrentDeletes != null) + { + concurrentDeletes.countDown(); + try + { + if(!allowDelete.await(20, java.util.concurrent.TimeUnit.SECONDS)) + { + throw (new IllegalStateException("Timed out waiting to release the cleanup owner")); + } + } + catch(InterruptedException e) + { + Thread.currentThread().interrupt(); + throw (new IllegalStateException(e)); + } + } + if(failDelete || failedKeys.contains(key)) + { + throw (new IllegalStateException("injected storage delete failure")); + } + objects.remove(name + "/" + key); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentsServiceTest.java b/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentsServiceTest.java index 1b0d7f1..9f47d72 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentsServiceTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentsServiceTest.java @@ -32,16 +32,23 @@ import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.Optional; import com.kingsrook.qbits.collaboration.CollaborationQBitConfig; import com.kingsrook.qbits.collaboration.CollaborationSchema; import com.kingsrook.qbits.collaboration.HomeWidgetData; import com.kingsrook.qbits.collaboration.PersonalServiceFixture; import com.kingsrook.qbits.collaboration.app.CollaborationHome; +import com.kingsrook.qbits.collaboration.attachments.uploadjournal.MemoryUploadJournal; import com.kingsrook.qbits.collaboration.model.CollaborationTables; import com.kingsrook.qbits.collaboration.permissions.CollaborationPermissions; import com.kingsrook.qbits.collaboration.security.ParentRecordAccess; import com.kingsrook.qbits.objectstorage.api.PresignGet; +import com.kingsrook.qbits.objectstorage.api.ObjectStream; +import com.kingsrook.qbits.objectstorage.api.ObjectHeaders; +import com.kingsrook.qbits.objectstorage.api.StoredObject; +import com.kingsrook.qbits.objectstorage.api.StorageException; import com.kingsrook.qbits.objectstorage.api.PutRequest; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; import com.kingsrook.qqq.backend.core.actions.permissions.PermissionEnforcement; import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface; import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers; @@ -87,7 +94,9 @@ public class AttachmentsServiceTest extends PersonalServiceFixture { private static AttachmentsService processService; private FakeStorage storage; + private MemoryUploadJournal journal; private AttachmentsService service; + private TestLifecycle lifecycle; /******************************************************************************* ** No Docker, network, object-storage grants or collaboration.view needed. @@ -95,8 +104,566 @@ public class AttachmentsServiceTest extends PersonalServiceFixture @BeforeEach void attachments() { + lifecycle = null; storage = new FakeStorage(); - service = new AttachmentsService(config, storage); + journal = new MemoryUploadJournal(); + service = memoryService(); + } + + /******************************************************************************* + ** Standard downloads retain lifecycle checks and the authenticated descriptor. + *******************************************************************************/ + @Test + void authenticatedDeliveryPreservesPolicyAndExactUrl() throws Exception + { + AttachmentEntry file = upload("download.txt", "original"); + java.util.concurrent.atomic.AtomicBoolean deny = new java.util.concurrent.atomic.AtomicBoolean(); + URL destination = URI.create("https://app.example/qqq/v1/qbit/files/exact").toURL(); + AttachmentLifecyclePolicy policy = new AttachmentLifecyclePolicy() + { + /*********************************************************************** + ** Policy refusal must happen before a delivery is issued. + ***********************************************************************/ + @Override + public void beforeAccess(QRecord row, boolean inline) throws QException + { + if(deny.get()) + { + throw new QException("Denied by lifecycle"); + } + } + + /*********************************************************************** + ** No storage signing or alternate byte owner is needed. + ***********************************************************************/ + @Override + public AttachmentDownload downloadDelivery(QRecord row) + { + assertThat(row.getValueInteger("id")).isEqualTo(file.id()); + return (new AttachmentDownload(destination, true)); + } + }; + service = new AttachmentsService(config, storage, policy, journal); + assertThat(service.downloadDelivery(file.id())).isEqualTo(new AttachmentDownload(destination, true)); + assertThat(service.downloadUrl(file.id(), false)).isEqualTo(destination); + assertThat(service.downloadDelivery(file.id(), true)).isEqualTo(new AttachmentDownload(destination, true)); + assertThat(service.downloadUrl(file.id(), true)).isEqualTo(destination); + AttachmentDownload signed = new AttachmentDownload(URI.create("https://storage.example/file?signature=secret").toURL(), false); + assertThat(signed.toString()).doesNotContain("secret", "signature", "storage.example"); + for(String invalid : java.util.List.of("https://user:password@storage.example/file", "https://storage.example/file#fragment", "http:/missing-host")) + { + assertThatThrownBy(() -> new AttachmentDownload(URI.create(invalid).toURL(), false)).isInstanceOf(IllegalArgumentException.class); + } + deny.set(true); + assertThatThrownBy(() -> service.downloadDelivery(file.id())).hasMessageContaining("Denied by lifecycle"); + assertThatThrownBy(() -> new AttachmentDownload(URI.create("file:///tmp/bytes").toURL(), true)).isInstanceOf(IllegalArgumentException.class); + } + + /******************************************************************************* + ** Retention protects mixed bulk deletion without preventing metadata edits. + *******************************************************************************/ + @Test + void retainedFilesAreNotDeletable() throws Exception + { + withLifecycle(); + AttachmentEntry ordinary = upload("ordinary.txt", "first"); + lifecycle.retained = true; + AttachmentEntry retained = upload("retained.txt", "second"); + lifecycle.retainedId = retained.id(); + assertThat(retained.canDelete()).isFalse(); + assertThat(retained.canEdit()).isTrue(); + assertThatThrownBy(() -> service.delete(List.of(ordinary.id(), retained.id()))).hasMessageContaining("retained"); + assertThat(storage.objects).hasSize(2); + assertThatThrownBy(() -> service.requireDeletable(retained.id())).hasMessageContaining("retained"); + assertThatThrownBy(() -> service.deleteStoredRow(raw(retained.id()))).hasMessageContaining("retained"); + assertThat(service.rename(retained.id(), "renamed.txt", null).fileName()).isEqualTo("renamed.txt"); + assertThat(lifecycle.metadataCalls).isEqualTo(1); + lifecycle.metadataFails = true; + assertThatThrownBy(() -> service.rename(retained.id(), "denied.txt", null)).hasMessageContaining("Metadata audit unavailable"); + assertThat(raw(retained.id()).getValueString("fileName")).isEqualTo("renamed.txt"); + lifecycle.retentionFails = true; + assertThatThrownBy(() -> service.delete(List.of(ordinary.id()))).hasMessageContaining("Retention unavailable"); + assertThat(storage.objects).hasSize(2); + } + + /******************************************************************************* + ** Repeated cleanup advances past retained rows and never removes their bytes. + *******************************************************************************/ + @Test + void retainedParentCleanupCrossesPageBoundary() throws Exception + { + withLifecycle(); + lifecycle.retained = true; + for(int i = 0; i < 205; i++) + { + upload("photo-" + i + ".jpg", "original"); + } + new DeleteAction().execute(new DeleteInput("document").withPrimaryKeys(List.of(1))); + AttachmentCleanup cleanup = new AttachmentCleanup(service); + cleanup.onParentsDeleted("document", List.of(1)); + cleanup.onParentsDeleted("document", List.of(1)); + List rows = AttachmentRecords.query(new QQueryFilter()); + assertThat(rows).hasSize(205).allSatisfy(row -> assertThat(row.getValueBoolean("parentDeleted")).isTrue()); + assertThat(storage.objects).hasSize(205); + assertThatThrownBy(() -> service.downloadUrl(rows.getFirst().getValue("id"), false)).hasMessage("Not found"); + assertThat(lifecycle.accesses).isEmpty(); + } + + /******************************************************************************* + ** Both standard preview fields and explicit downloads pass the same policy. + *******************************************************************************/ + @Test + void accessPolicyIsStrictAndRunsAfterAuthorization() throws Exception + { + withLifecycle(); + AttachmentEntry uploaded = upload("audit.txt", "original"); + assertThat(lifecycle.storedCalls).isEqualTo(1); + service.downloadUrl(uploaded.id(), false); + new AttachmentPreviewCustomizer().filter(List.of(raw(uploaded.id())), false, true, service); + assertThat(lifecycle.accesses).containsExactly(false, true, false); + assertThat(storage.signatures).hasSize(3); + lifecycle.accessFails = true; + assertThatThrownBy(() -> service.downloadUrl(uploaded.id(), false)).hasMessageContaining("Access audit unavailable"); + assertThatThrownBy(() -> new AttachmentPreviewCustomizer().filter(List.of(raw(uploaded.id())), false, true, service)) + .hasMessageContaining("Access audit unavailable"); + assertThat(storage.signatures).hasSize(3); + lifecycle.accessRuntimeFails = true; + assertThatThrownBy(() -> new AttachmentPreviewCustomizer().filter(List.of(raw(uploaded.id())), false, true, service)) + .hasMessageContaining("Attachment access policy failed"); + assertThat(storage.signatures).hasSize(3); + lifecycle.accesses.clear(); + QContext.getQSession().getSecurityKeyValues().remove("tenant"); + assertThatThrownBy(() -> service.downloadUrl(uploaded.id(), false)).hasMessage("Not found"); + assertThat(new AttachmentPreviewCustomizer().filter(List.of(raw(uploaded.id())), false, true, service)).isEmpty(); + assertThat(lifecycle.accesses).isEmpty(); + } + + /******************************************************************************* + ** A policy registration failure preserves the original for reconciliation. + *******************************************************************************/ + @Test + void storedPolicyFailureDoesNotDeleteOriginal() throws Exception + { + withLifecycle(); + lifecycle.storedFails = true; + assertThatThrownBy(() -> upload("preserved.txt", "original")).hasMessageContaining("Registration unavailable"); + assertThat(storage.objects).hasSize(1); + assertThat(AttachmentRecords.query(new QQueryFilter())).hasSize(1); + } + + /******************************************************************************* + ** Configuration removal cannot turn a retained original into a deletable file. + *******************************************************************************/ + @Test + void persistedRetentionSurvivesProviderRemoval() throws Exception + { + withLifecycle(); + lifecycle.retained = true; + AttachmentEntry file = upload("retained.txt", "original"); + assertThat(raw(file.id()).getValueBoolean("retained")).isTrue(); + config.withAttachmentLifecycle(new QCodeReference(AttachmentLifecyclePolicy.class)); + service = memoryService(); + assertThat(service.list("document", 1).getFirst().canDelete()).isFalse(); + assertThatThrownBy(() -> service.delete(List.of(file.id()))).hasMessageContaining("retained"); + assertThatThrownBy(() -> service.downloadUrl(file.id(), false)).hasMessageContaining("requires its lifecycle policy"); + assertThat(storage.signatures).isEmpty(); + new DeleteAction().execute(new DeleteInput("document").withPrimaryKeys(List.of(1))); + new AttachmentCleanup(service).onParentsDeleted("document", List.of(1)); + assertThat(raw(file.id()).getValueBoolean("parentDeleted")).isTrue(); + assertThat(storage.objects).hasSize(1); + } + + /******************************************************************************* + ** Failed intake classification never writes object bytes or attachment rows. + *******************************************************************************/ + @Test + void unavailableRetentionPolicyDeniesIntake() throws Exception + { + withLifecycle(); + lifecycle.retentionFails = true; + assertThatThrownBy(() -> upload("blocked.txt", "bytes")).hasMessageContaining("Retention unavailable"); + assertThat(storage.objects).isEmpty(); + assertThat(AttachmentRecords.query(new QQueryFilter())).isEmpty(); + } + + /******************************************************************************* + ** Core validation reads authorize rows without generating presentation URLs. + *******************************************************************************/ + @Test + void internalReadsDoNotIssueUrlsButDisplayReadsDo() throws Exception + { + CountingLifecycle.accessCalls = 0; + PreviewStorage.signatureCalls = 0; + config.withAttachmentLifecycle(new QCodeReference(CountingLifecycle.class)) + .withAttachmentStorage(new QCodeReference(PreviewStorage.class)); + service = memoryService(); + AttachmentEntry file = upload("presentation.txt", "original"); + login("alice", CollaborationPermissions.VIEW); + QRecord internal = new GetAction().execute(new GetInput(CollaborationTables.ATTACHMENT).withPrimaryKey(file.id())).getRecord(); + assertThat(internal.getValue("previewUrl")).isNull(); + assertThat(internal.getValue("downloadUrl")).isNull(); + assertThat(internal.getValue("previewDelivery")).isNull(); + assertThat(internal.getValue("downloadDelivery")).isNull(); + QRecord stale = new QRecord(internal).withValue("previewDelivery", "AUTHENTICATED_FETCH").withValue("downloadDelivery", "SIGNED_URL"); + QRecord cleared = new AttachmentPreviewCustomizer().filter(List.of(stale), true, false, service).getFirst(); + assertThat(cleared.getValue("previewDelivery")).isNull(); + assertThat(cleared.getValue("downloadDelivery")).isNull(); + service.rename(file.id(), "renamed.txt", null); + assertThat(CountingLifecycle.accessCalls).isZero(); + assertThat(PreviewStorage.signatureCalls).isZero(); + QRecord displayed = new GetAction().execute(new GetInput(CollaborationTables.ATTACHMENT).withPrimaryKey(file.id()) + .withShouldGenerateDisplayValues(true)).getRecord(); + assertThat(displayed.getValueString("previewUrl")).startsWith("https://signed.example/"); + assertThat(displayed.getValueString("downloadUrl")).startsWith("https://signed.example/"); + assertThat(displayed.getValueString("previewDelivery")).isEqualTo("SIGNED_URL"); + assertThat(displayed.getValueString("downloadDelivery")).isEqualTo("SIGNED_URL"); + new QueryAction().execute(new QueryInput(CollaborationTables.ATTACHMENT).withShouldGenerateDisplayValues(true)); + assertThat(CountingLifecycle.accessCalls).isEqualTo(4); + assertThat(PreviewStorage.signatureCalls).isEqualTo(4); + QContext.getQSession().getSecurityKeyValues().remove("tenant"); + assertThat(new GetAction().execute(new GetInput(CollaborationTables.ATTACHMENT).withPrimaryKey(file.id())).getRecord()).isNull(); + assertThat(new QueryAction().execute(new QueryInput(CollaborationTables.ATTACHMENT)).getRecords()).isEmpty(); + assertThat(new QueryAction().execute(new QueryInput(CollaborationTables.ATTACHMENT).withShouldGenerateDisplayValues(true)).getRecords()).isEmpty(); + assertThat(CountingLifecycle.accessCalls).isEqualTo(4); + assertThat(PreviewStorage.signatureCalls).isEqualTo(4); + } + + /******************************************************************************* + ** Count policy calls across independently constructed request services. + *******************************************************************************/ + public static class CountingLifecycle extends AttachmentLifecyclePolicy + { + private static int accessCalls; + + /******************************************************************************* + ** Observe URL intent without depending on a service instance lifetime. + *******************************************************************************/ + @Override + public void beforeAccess(QRecord attachment, boolean inline) + { + accessCalls++; + } + } + + /******************************************************************************* + ** Configured preview-only storage adapter; no network or object-store access. + *******************************************************************************/ + public static class PreviewStorage extends AttachmentStorage + { + private static int signatureCalls; + + /******************************************************************************* + ** Return a synthetic capability and count actual signing calls. + *******************************************************************************/ + @Override + public URL presign(String name, PresignGet request) + { + signatureCalls++; + try + { + return (URI.create("https://signed.example/" + request.getKey()).toURL()); + } + catch(Exception e) + { + throw (new IllegalStateException(e)); + } + } + } + + /******************************************************************************* + ** Inject a per-test policy while configuring internal metadata reads too. + *******************************************************************************/ + private AttachmentsService memoryService() + { + AttachmentLifecyclePolicy selected = com.kingsrook.qqq.backend.core.actions.customizers.QCodeLoader.getAdHoc(AttachmentLifecyclePolicy.class, config.getAttachmentLifecycle()); + return (new AttachmentsService(config, storage, selected, journal)); + } + + /******************************************************************************* + ** Inject a per-test policy while configuring internal metadata reads too. + *******************************************************************************/ + private void withLifecycle() + { + config.withAttachmentLifecycle(new QCodeReference(TestLifecycle.class)); + lifecycle = new TestLifecycle(); + service = new AttachmentsService(config, storage, lifecycle, journal); + } + + /******************************************************************************* + ** Reference transfers need explicit retention authority and caller ownership. + *******************************************************************************/ + @Test + void archiveTransferRequiresPolicyAndTransaction() throws Exception + { + withLifecycle(); + lifecycle.retained = true; + AttachmentEntry file = upload("retained.txt", "original"); + assertThatThrownBy(() -> service.transferRetainedReference(file.id(), "document", 2, null)) + .hasMessageContaining("explicit transaction"); + assertThatThrownBy(() -> service.transferRetainedReference(file.id(), "document", 2, new QBackendTransaction())) + .hasMessageContaining("archive policy"); + QContext.setTransaction(new QBackendTransaction()); + try + { + assertThatThrownBy(() -> service.transferRetainedReference(file.id(), "document", 2, new QBackendTransaction())) + .hasMessageContaining("explicit transaction"); + } + finally + { + QContext.clearTransaction(); + } + assertThat(raw(file.id()).getValueString("recordId")).isEqualTo("1"); + assertThat(storage.objects).hasSize(1); + } + + /******************************************************************************* + ** A trusted policy cannot replace native target visibility with an approval. + *******************************************************************************/ + @Test + void archiveTransferChecksBothParentsBeforePolicy() throws Exception + { + withLifecycle(); + lifecycle.retained = true; + AttachmentEntry file = upload("retained.txt", "original"); + TransferPolicy policy = new TransferPolicy(); + service = new AttachmentsService(config, storage, policy, journal); + QBackendTransaction transaction = new QBackendTransaction(); + assertThatThrownBy(() -> service.transferRetainedReference(file.id(), "document", 3, transaction)).hasMessage("Not found"); + assertThatThrownBy(() -> service.transferRetainedReference(file.id(), "document", 999, transaction)).hasMessage("Not found"); + QContext.getQSession().getSecurityKeyValues().clear(); + QContext.getQSession().withSecurityKeyValue("tenant", "b"); + assertThatThrownBy(() -> service.transferRetainedReference(file.id(), "document", 3, transaction)).hasMessage("Not found"); + assertThat(policy.beforeCalls).isZero(); + assertThat(policy.afterCalls).isZero(); + assertThat(storage.objects).hasSize(1); + } + + /******************************************************************************* + ** The owner changes only the canonical reference, never the original content. + *******************************************************************************/ + @Test + void archiveTransferPreservesOriginalAndPassesExactTransaction() throws Exception + { + withLifecycle(); + lifecycle.retained = true; + AttachmentEntry file = upload("retained.txt", "original"); + QRecord prior = raw(file.id()); + Map originalObjects = new HashMap<>(storage.objects); + TransferPolicy policy = new TransferPolicy(); + service = new AttachmentsService(config, storage, policy, journal); + QBackendTransaction transaction = new QBackendTransaction(); + login("bob"); + service.transferRetainedReference(file.id(), "document", 2, transaction); + QRecord updated = raw(file.id()); + assertThat(updated.getValueString("recordId")).isEqualTo("2"); + assertThat(updated.getValueString("tableName")).isEqualTo("document"); + for(String field : prior.getValues().keySet()) + { + if(!List.of("tableName", "recordId", "modifyDate").contains(field)) + { + assertThat(updated.getValue(field)).as(field).isEqualTo(prior.getValue(field)); + } + } + assertThat(storage.objects).containsExactlyInAnyOrderEntriesOf(originalObjects); + assertThat(storage.readCount).isZero(); + assertThat(policy.beforeCalls).isEqualTo(1); + assertThat(policy.afterCalls).isEqualTo(1); + assertThat(policy.transaction).isSameAs(transaction); + assertThat(policy.targetId).isEqualTo("2"); + assertThat(policy.prior.getValueString("recordId")).isEqualTo("1"); + assertThat(service.list("document", 2)).hasSize(1); + assertThat(service.list("document", 1)).isEmpty(); + } + + /******************************************************************************* + ** Outcome failure reaches the transaction owner; memory does not prove rollback. + *******************************************************************************/ + @Test + void archiveTransferPropagatesOutcomeFailure() throws Exception + { + withLifecycle(); + lifecycle.retained = true; + AttachmentEntry file = upload("retained.txt", "original"); + TransferPolicy policy = new TransferPolicy(); + policy.failAfter = true; + service = new AttachmentsService(config, storage, policy, journal); + assertThatThrownBy(() -> service.transferRetainedReference(file.id(), "document", 2, new QBackendTransaction())) + .hasMessage("Archive audit unavailable"); + assertThat(policy.beforeCalls).isEqualTo(1); + assertThat(policy.afterCalls).isEqualTo(1); + assertThat(storage.objects).hasSize(1); + } + + /******************************************************************************* + ** An ordinary attachment cannot acquire the archive-only transfer capability. + *******************************************************************************/ + @Test + void archiveTransferRejectsOrdinaryAttachments() throws Exception + { + AttachmentEntry file = upload("ordinary.txt", "original"); + TransferPolicy policy = new TransferPolicy(); + service = new AttachmentsService(config, storage, policy, journal); + assertThatThrownBy(() -> service.transferRetainedReference(file.id(), "document", 2, new QBackendTransaction())) + .hasMessageContaining("Only a retained attachment"); + assertThat(policy.beforeCalls).isZero(); + assertThat(raw(file.id()).getValueString("recordId")).isEqualTo("1"); + } + + /******************************************************************************* + ** Test-only authority observes calls; real JDBC/archive policy lives in its QBit. + *******************************************************************************/ + private static class TransferPolicy extends AttachmentLifecyclePolicy + { + private int beforeCalls; + private int afterCalls; + private boolean failAfter; + private QBackendTransaction transaction; + private String targetId; + private QRecord prior; + + /*************************************************************************** + ** Observe the canonical target and the caller's exact transaction. + ***************************************************************************/ + @Override + public void beforeReferenceTransfer(QRecord attachment, String tableName, String recordId, QBackendTransaction transaction) + { + beforeCalls++; + this.transaction = transaction; + targetId = recordId; + prior = attachment; + } + + /*************************************************************************** + ** A failed outcome must escape so the caller can roll back its transaction. + ***************************************************************************/ + @Override + public void afterReferenceTransfer(QRecord attachment, String tableName, String recordId, QBackendTransaction transaction) throws QException + { + afterCalls++; + if(failAfter) + { + throw (new QException("Archive audit unavailable")); + } + } + } + + /******************************************************************************* + ** Per-test policy instance, exposed only to the enclosing deterministic tests. + *******************************************************************************/ + public static class TestLifecycle extends AttachmentLifecyclePolicy + { + private boolean retained; + private Integer retainedId; + private boolean retentionFails; + private boolean accessFails; + private boolean accessRuntimeFails; + private boolean storedFails; + private boolean beforeStoredFails; + private int beforeStoredCalls; + private boolean metadataFails; + private int storedCalls; + private int metadataCalls; + private final List accesses = new ArrayList<>(); + + /******************************************************************************* + ** Public no-argument constructor for host-configured policy loading. + *******************************************************************************/ + public TestLifecycle() + { + } + + /******************************************************************************* + ** Record successful registration callbacks or simulate a failed registry. + *******************************************************************************/ + @Override + public void beforeStored(QRecord attachment) throws QException + { + beforeStoredCalls++; + if(beforeStoredFails) + { + throw (new QException("Storage intent unavailable")); + } + } + + /******************************************************************************* + ** Explicit fixture-only recovery policy; real durability is tested with JDBC. + *******************************************************************************/ + @Override + public void beforeRecovery(QRecord candidate) + { + } + + /******************************************************************************* + ** Simulate registration after storage; this memory fixture is not durability proof. + *******************************************************************************/ + @Override + public void afterStored(QRecord attachment) throws QException + { + storedCalls++; + if(storedFails) + { + throw (new QException("Registration unavailable")); + } + } + + /******************************************************************************* + ** Simulate a durable access event writer failure before signing. + *******************************************************************************/ + @Override + public void beforeAccess(QRecord attachment, boolean inline) throws QException + { + accesses.add(inline); + if(accessRuntimeFails) + { + throw (new IllegalStateException("Unexpected policy failure")); + } + if(accessFails) + { + throw (new QException("Access audit unavailable")); + } + } + + /******************************************************************************* + ** Protect all rows or one specific retained row, independently of parent. + *******************************************************************************/ + @Override + public boolean isRetained(QRecord attachment) throws QException + { + if(retentionFails) + { + throw (new QException("Retention unavailable")); + } + return (retained && (retainedId == null || retainedId.equals(attachment.getValueInteger("id")))); + } + + /******************************************************************************* + ** Track proposed edits separately from successful committed writes. + *******************************************************************************/ + @Override + public void beforeMetadataChange(QRecord attachment, QRecord changes) throws QException + { + metadataCalls++; + if(metadataFails) + { + throw (new QException("Metadata audit unavailable")); + } + } + + /******************************************************************************* + ** This memory fixture tests callback ordering, not durable transactions. + *******************************************************************************/ + @Override + public void validateMetadataTransaction(QRecord attachment, QBackendTransaction transaction) + { + } + + /******************************************************************************* + ** Production providers must append their event using the owner transaction. + *******************************************************************************/ + @Override + public void afterMetadataChange(QRecord attachment, QRecord changes, QBackendTransaction transaction) + { + } } /******************************************************************************* @@ -142,6 +709,184 @@ public class AttachmentsServiceTest extends PersonalServiceFixture assertThat(service.list("document", 1)).extracting(AttachmentEntry::id).containsExactly(second.id()); } + /******************************************************************************* + ** Content is authorized and verified before release, and buffers are immutable. + *******************************************************************************/ + @Test + void verifiedContentRead() throws Exception + { + AttachmentEntry file = upload("original.txt", "hello"); + AttachmentContent content = service.readContent(file.id(), 5); + assertThat(content.attachmentId()).isEqualTo(file.id()); + assertThat(content.sha256()).isEqualTo(file.sha256()); + assertThat(content.contentType()).isEqualTo("text/plain"); + assertThat(content.bytes()).isEqualTo("hello".getBytes(StandardCharsets.UTF_8)); + byte[] copy = content.bytes(); + copy[0] = 0; + assertThat(content.bytes()[0]).isEqualTo((byte) 'h'); + assertThat(storage.lastReadClosed).isTrue(); + assertThat(storage.signatures).isEmpty(); + assertThatThrownBy(() -> service.readContent(file.id(), 4)).hasMessageContaining("limit"); + assertThatThrownBy(() -> service.readContent(file.id(), 21 * 1024 * 1024)).hasMessageContaining("20 MiB"); + assertThat(storage.readCount).isEqualTo(1); + QContext.getQSession().getSecurityKeyValues().remove("tenant"); + assertThatThrownBy(() -> service.readContent(file.id(), 5)).isInstanceOf(QNotFoundException.class); + assertThat(storage.readCount).isEqualTo(1); + } + + /******************************************************************************* + ** Changed, truncated and oversized originals never escape the integrity gate. + *******************************************************************************/ + @Test + void corruptContentRead() throws Exception + { + AttachmentEntry file = upload("original.txt", "hello"); + String key = storage.objects.keySet().iterator().next(); + storage.objects.put(key, "wrong".getBytes(StandardCharsets.UTF_8)); + assertThatThrownBy(() -> service.readContent(file.id(), 5)).hasMessageContaining("integrity"); + assertThat(storage.lastReadClosed).isTrue(); + storage.objects.put(key, "four".getBytes(StandardCharsets.UTF_8)); + assertThatThrownBy(() -> service.readContent(file.id(), 5)).isInstanceOf(AttachmentContentIntegrityException.class); + storage.reportLength = false; + assertThatThrownBy(() -> service.readContent(file.id(), 5)).hasMessageContaining("integrity"); + storage.objects.put(key, "toolong".getBytes(StandardCharsets.UTF_8)); + assertThatThrownBy(() -> service.readContent(file.id(), 5)).hasMessageContaining("integrity"); + assertThat(storage.lastReadClosed).isTrue(); + } + + /******************************************************************************* + ** Same intent reuses one original while conflicting bytes or metadata fail. + *******************************************************************************/ + @Test + void retryableIntakePreservesOneOriginal() throws Exception + { + String request = "a".repeat(64); + String manifest = "b".repeat(64); + AttachmentEntry first = service.upload("document", 1, "evidence.txt", "text/plain", stream("hello"), "caption", request, manifest); + AttachmentEntry replay = service.upload("document", "01", "evidence.txt", "text/plain", stream("hello"), "caption", request, manifest); + assertThat(replay.id()).isEqualTo(first.id()); + assertThat(storage.objects).hasSize(1); + assertThatThrownBy(() -> service.upload("document", 1, "evidence.txt", "text/plain", stream("other"), "caption", request, manifest)) + .hasMessageContaining("conflicts"); + assertThatThrownBy(() -> service.upload("document", 1, "renamed.txt", "text/plain", stream("hello"), "caption", request, manifest)) + .hasMessageContaining("conflicts"); + assertThat(storage.objects).hasSize(1); + service.upload("document", 2, "evidence.txt", "text/plain", stream("hello"), "caption", request, manifest); + assertThat(storage.objects).hasSize(2); + } + + /******************************************************************************* + ** Completed retained retries do not create another file-added timeline event. + *******************************************************************************/ + @Test + void completedRetainedReplayDoesNotRepeatTimelineActivity() throws Exception + { + withLifecycle(); + lifecycle.retained = true; + String request = "c".repeat(64); + String manifest = "d".repeat(64); + var first = service.upload("document", 1, "evidence.txt", "text/plain", stream("hello"), null, request, manifest); + var replay = service.upload("document", 1, "evidence.txt", "text/plain", stream("hello"), null, request, manifest); + assertThat(replay.id()).isEqualTo(first.id()); + assertThat(com.kingsrook.qbits.collaboration.CollaborationRecords.query(CollaborationTables.ACTIVITY, + new QQueryFilter().withCriteria("type", QCriteriaOperator.EQUALS, "FILE_ADDED"))).hasSize(1); + assertThat(lifecycle.storedCalls).as("mandatory idempotent registration runs for each coordinator replay").isEqualTo(2); + assertThat(storage.objects).hasSize(1); + } + + /******************************************************************************* + ** A metadata insert failure leaves a deterministic original recoverable. + *******************************************************************************/ + @Test + void retryableIntakeRecoversStoredObject() throws Exception + { + TestLifecycle lifecycle = new TestLifecycle(); + service = new AttachmentsService(config, storage, lifecycle, journal); + QTableMetaData table = instance.getTable(CollaborationTables.ATTACHMENT); + Map original = new HashMap<>(table.getCustomizers()); + table.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(RejectInsert.class)); + String request = "c".repeat(64); + String manifest = "d".repeat(64); + assertThatThrownBy(() -> service.upload("document", 1, "evidence.txt", "text/plain", stream("hello"), null, request, manifest)) + .hasMessageContaining("save attachment"); + assertThat(storage.objects).hasSize(1); + table.setCustomizers(original); + AttachmentEntry recovered = service.upload("document", 1, "evidence.txt", "text/plain", stream("hello"), null, request, manifest); + assertThat(recovered.sha256()).hasSize(64); + assertThat(storage.objects).hasSize(1); + assertThat(service.list("document", 1)).hasSize(1); + } + + /******************************************************************************* + ** Both audited content-read phases must succeed before releasing any bytes. + *******************************************************************************/ + @Test + void retainedContentReadRequiresExplicitAuditPolicy() throws Exception + { + AttachmentLifecyclePolicy policy = new AttachmentLifecyclePolicy() + { + /*********************************************************************** + ** Mark the fixture retained while retaining the default read denial. + ***********************************************************************/ + @Override + public boolean isRetained(QRecord attachment) + { + return (true); + } + + /*********************************************************************** + ** This fixture exercises read denial after explicitly allowed storage. + ***********************************************************************/ + @Override + public void beforeStored(QRecord attachment) + { + } + }; + service = new AttachmentsService(config, storage, policy, journal); + AttachmentEntry file = upload("retained.txt", "hello"); + assertThatThrownBy(() -> service.readContent(file.id(), 5)).hasMessageContaining("content-read policy"); + assertThat(storage.readCount).isZero(); + AttachmentLifecyclePolicy partialPolicy = new AttachmentLifecyclePolicy() + { + /*********************************************************************** + ** A before hook alone must not bypass mandatory read-result recording. + ***********************************************************************/ + @Override + public void beforeContentRead(QRecord attachment) + { + } + }; + service = new AttachmentsService(config, storage, partialPolicy, journal); + assertThatThrownBy(() -> service.readContent(file.id(), 5)).hasMessageContaining("read-result policy"); + assertThat(storage.readCount).isEqualTo(1); + assertThat(storage.lastReadClosed).isTrue(); + } + + /******************************************************************************* + ** An older retention provider cannot silently omit atomic metadata auditing. + *******************************************************************************/ + @Test + void retainedMetadataRequiresTransactionPolicyBeforeMutation() throws Exception + { + withLifecycle(); + lifecycle.retained = true; + AttachmentEntry file = upload("original.txt", "hello"); + AttachmentLifecyclePolicy legacyProvider = new AttachmentLifecyclePolicy() + { + /*********************************************************************** + ** An older provider handles intent but has no atomic result contract. + ***********************************************************************/ + @Override + public void beforeMetadataChange(QRecord attachment, QRecord changes) + { + } + }; + service = new AttachmentsService(config, storage, legacyProvider, journal); + assertThatThrownBy(() -> service.rename(file.id(), "changed.txt", null)) + .hasMessageContaining("atomic audit transaction policy"); + assertThat(raw(file.id()).getValueString("fileName")).isEqualTo("original.txt"); + } + /******************************************************************************* ** Both collaboration and bucket size limits are enforced while streaming. *******************************************************************************/ @@ -275,7 +1020,7 @@ public class AttachmentsServiceTest extends PersonalServiceFixture QRecord visible = raw(row.id()); QRecord hidden = new QRecord(visible).withValue("recordId", "3"); AttachmentPreviewCustomizer customizer = new AttachmentPreviewCustomizer(); - List results = customizer.filter(List.of(visible, hidden), false, service); + List results = customizer.filter(List.of(visible, hidden), false, true, service); assertThat(results).hasSize(1); assertThat(storage.signatures).hasSize(2); assertThat(results.getFirst().getValueString("previewUrl")).contains("signed.example"); @@ -284,7 +1029,7 @@ public class AttachmentsServiceTest extends PersonalServiceFixture assertThat(results.getFirst().getDisplayValue("downloadUrl:downloadUrlDynamic")).isEqualTo(results.getFirst().getValueString("downloadUrl")); assertThat(visible.getValue("previewUrl")).isNull(); assertThat(raw(row.id()).getValue("previewUrl")).isNull(); - assertThatThrownBy(() -> customizer.filter(List.of(hidden), true, service)).hasMessage("Not found"); + assertThatThrownBy(() -> customizer.filter(List.of(hidden), true, true, service)).hasMessage("Not found"); } /******************************************************************************* @@ -383,7 +1128,7 @@ public class AttachmentsServiceTest extends PersonalServiceFixture } /******************************************************************************* - ** Storage failures leave rows for retries; trusted cleanup works after parent deletion. + ** Storage failures leave durable cleanup; trusted cleanup works after parent deletion. *******************************************************************************/ @Test void failuresAndCleanup() throws Exception @@ -391,8 +1136,13 @@ public class AttachmentsServiceTest extends PersonalServiceFixture AttachmentEntry row = upload("file.txt", "x"); storage.failDelete = true; assertThatThrownBy(() -> service.delete(List.of(row.id()))).hasMessage("Storage unavailable"); - assertThat(service.list("document", 1)).hasSize(1); + assertThat(service.list("document", 1)).isEmpty(); + assertThat(storage.objects).hasSize(1); storage.failDelete = false; + admin(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + login("alice"); + upload("parent-cleanup.txt", "parent bytes"); new DeleteAction().execute(new DeleteInput("document").withPrimaryKeys(List.of(1))); assertThatThrownBy(() -> service.list("document", 1)).hasMessage("Not found"); QContext.getQSession().getPermissions().clear(); @@ -400,10 +1150,56 @@ public class AttachmentsServiceTest extends PersonalServiceFixture cleanup.onParentsDeleted("document", List.of()); cleanup.onParentsDeleted("document", List.of(1)); cleanup.onParentsDeleted("document", List.of(1)); + admin(); + new AttachmentCleanupQueue(storage).retryPending(); assertThat(storage.objects).isEmpty(); assertThat(AttachmentRecords.query(new QQueryFilter())).isEmpty(); } + /******************************************************************************* + ** A required intent failure prevents even the first object write. + *******************************************************************************/ + @Test + void retainedStorageRequiresIntentBeforePut() throws Exception + { + withLifecycle(); + lifecycle.retained = true; + lifecycle.beforeStoredFails = true; + assertThatThrownBy(() -> upload("evidence.txt", "original")).hasMessageContaining("Storage intent unavailable"); + assertThat(storage.objects).isEmpty(); + assertThat(AttachmentRecords.query(new QQueryFilter())).isEmpty(); + assertThat(lifecycle.storedCalls).isZero(); + assertThatThrownBy(() -> new AttachmentLifecyclePolicy().beforeStored(new QRecord().withValue("retained", true))) + .hasMessageContaining("durable write-intent"); + lifecycle.beforeStoredFails = false; + QContext.setTransaction(new QBackendTransaction()); + try + { + assertThatThrownBy(() -> upload("evidence.txt", "original")).isInstanceOf(QException.class).hasMessageContaining("own committed transactions"); + } + finally + { + QContext.clearTransaction(); + } + assertThat(storage.objects).isEmpty(); + } + + /******************************************************************************* + ** A stored retained original survives a metadata insertion failure. + *******************************************************************************/ + @Test + void retainedFailedInsertNeverDeletesOriginal() throws Exception + { + withLifecycle(); + lifecycle.retained = true; + instance.getTable(CollaborationTables.ATTACHMENT).withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(RejectInsert.class)); + assertThatThrownBy(() -> upload("evidence.txt", "original")).hasMessageContaining("Could not save attachment"); + assertThat(storage.objects).hasSize(1); + assertThat(AttachmentRecords.query(new QQueryFilter())).isEmpty(); + assertThat(lifecycle.beforeStoredCalls).isEqualTo(1); + assertThat(lifecycle.storedCalls).isZero(); + } + /******************************************************************************* ** A rejected row compensates the put; a broken upload never creates a row. *******************************************************************************/ @@ -458,7 +1254,8 @@ public class AttachmentsServiceTest extends PersonalServiceFixture assertThat(CollaborationHome.humanSize(3L * 1024 * 1024 * 1024)).isEqualTo("3.0 GB"); String ddl = CollaborationSchema.postgresScript(config); assertThat(ddl).contains("CREATE TABLE IF NOT EXISTS collaboration_attachment", "size_bytes BIGINT", "parent_deleted BOOLEAN DEFAULT FALSE NOT NULL", - "ON collaboration_attachment (table_name, record_id)").doesNotContain("preview_url", "download_url"); + "ON collaboration_attachment (table_name, record_id)", "retained BOOLEAN DEFAULT FALSE NOT NULL", + "ALTER TABLE collaboration_attachment ADD COLUMN IF NOT EXISTS retained BOOLEAN NOT NULL DEFAULT FALSE").doesNotContain("preview_url", "download_url"); QTableMetaData table = instance.getTable(CollaborationTables.ATTACHMENT); assertThat(table.getField("sizeBytes").getType()).isEqualTo(QFieldType.LONG); assertThat(table.getVirtualField("previewUrl")).isNotNull(); @@ -498,11 +1295,50 @@ public class AttachmentsServiceTest extends PersonalServiceFixture private static class FakeStorage extends AttachmentStorage { private final Map objects = new HashMap<>(); + private final Map> objectMetadata = new HashMap<>(); private final List signatures = new ArrayList<>(); private long limit = Long.MAX_VALUE; private PutRequest lastPut; private PresignGet lastGet; private boolean failDelete; + private Integer readCount = 0; + private Boolean lastReadClosed = false; + private Boolean reportLength = true; + + /*************************************************************************** + ** Simulate a closeable object response with independently reported length. + ***************************************************************************/ + @Override + public ObjectStream open(String name, String key) + { + readCount++; + lastReadClosed = false; + byte[] bytes = objects.get(name + "/" + key); + ByteArrayInputStream input = new ByteArrayInputStream(bytes) + { + /******************************************************************* + ** Observe cleanup on success and on every integrity failure path. + *******************************************************************/ + @Override + public void close() throws IOException + { + lastReadClosed = true; + super.close(); + } + }; + return (new ObjectStream(input, reportLength ? (long) bytes.length : null, "text/plain", null, Instant.now(), + new ObjectHeaders().withUserMetadata(objectMetadata.get(name + "/" + key)))); + } + + /*************************************************************************** + ** Report presence for retry reconciliation without exposing content. + ***************************************************************************/ + @Override + public Optional head(String name, String key) + { + byte[] bytes = objects.get(name + "/" + key); + return (bytes == null ? Optional.empty() : Optional.of(new StoredObject().withSize((long) bytes.length))); + } @Override public long maxBytes(String name) @@ -514,9 +1350,15 @@ public class AttachmentsServiceTest extends PersonalServiceFixture public void put(String name, PutRequest request) { lastPut = request; + String key = name + "/" + request.getKey(); + if(Boolean.TRUE.equals(request.getIfNoneMatch()) && objects.containsKey(key)) + { + throw (new StorageException(StorageException.Reason.PRECONDITION_FAILED, "Object already exists")); + } try { - objects.put(name + "/" + request.getKey(), request.getStream().readAllBytes()); + objects.put(key, request.getBytes() == null ? request.getStream().readAllBytes() : request.getBytes().clone()); + objectMetadata.put(key, request.getMetadata()); } catch(IOException e) { diff --git a/src/test/java/com/kingsrook/qbits/collaboration/attachments/ExistingAttachmentAdoptionTest.java b/src/test/java/com/kingsrook/qbits/collaboration/attachments/ExistingAttachmentAdoptionTest.java new file mode 100644 index 0000000..6fa4c70 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/attachments/ExistingAttachmentAdoptionTest.java @@ -0,0 +1,248 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + + +import java.io.ByteArrayInputStream; +import java.sql.DriverManager; +import java.util.ArrayList; +import java.util.List; +import java.util.UUID; +import com.kingsrook.qbits.collaboration.PersonalServiceFixture; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qbits.objectstorage.api.ObjectStream; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Owner transaction/control tests with real H2 and fixed existing bytes. The + ** injected lifecycle is a test policy, not a durability qualification claim. + *******************************************************************************/ +class ExistingAttachmentAdoptionTest extends PersonalServiceFixture +{ + private static final byte[] BYTES = new byte[] {0, 1, 2, 3}; + private RDBMSBackendMetaData database; + private Policy policy; + private AttachmentsService service; + private int reads; + + /*************************************************************************** + ** Native metadata controls physical field mappings; no validation bypass. + ***************************************************************************/ + @BeforeEach + void database() throws Exception + { + ConnectionManager.closeConnectionProviders(); + database = new RDBMSBackendMetaData().withName("adoption_database").withVendor("h2").withUsername("sa") + .withJdbcUrl("jdbc:h2:mem:adoption_" + UUID.randomUUID() + ";DB_CLOSE_DELAY=-1;DATABASE_TO_UPPER=FALSE"); + instance.addBackend(database); + var table = instance.getTable(CollaborationTables.ATTACHMENT); + table.setBackendName(database.getName()); + try(var connection = DriverManager.getConnection(database.getJdbcUrl(), "sa", ""); var statement = connection.createStatement()) + { + List columns = new ArrayList<>(); + for(var field : table.getFields().values()) + { + String type = switch(field.getType()) + { + case INTEGER -> "INTEGER"; + case LONG -> "BIGINT"; + case BOOLEAN -> "BOOLEAN"; + case DATE_TIME -> "TIMESTAMP"; + case STRING, TEXT -> "VARCHAR(4096)"; + default -> throw new IllegalStateException("Unexpected fixture field"); + }; + columns.add(field.getBackendName() + " " + type + (field.getName().equals("id") ? " PRIMARY KEY" : "") + + (Boolean.TRUE.equals(field.getIsRequired()) ? " NOT NULL" : "")); + } + statement.execute("CREATE TABLE collaboration_attachment (" + String.join(",", columns) + ")"); + String hash = java.util.HexFormat.of().formatHex(java.security.MessageDigest.getInstance("SHA-256").digest(BYTES)); + statement.execute("INSERT INTO collaboration_attachment (id,table_name,record_id,file_name,content_type,size_bytes,sha256,bucket_name,storage_key,created_by_user_id,parent_deleted,retained,create_date) VALUES (" + + "9,'document','1','original.bin','application/octet-stream',4,'" + hash + "','test','original-key','alice',FALSE,FALSE,TIMESTAMP '2026-10-31 00:00:00')"); + } + policy = new Policy(); + service = new AttachmentsService(config, new AttachmentStorage() + { + /******************************************************************* + ** Fixed existing storage: any put/delete falls through to refusal. + *******************************************************************/ + @Override + public ObjectStream open(String bucket, String key) + { + assertThat(bucket).isEqualTo("test"); + assertThat(key).isEqualTo("original-key"); + reads++; + return (new ObjectStream(new ByteArrayInputStream(BYTES), 4L, "application/octet-stream", null, java.time.Instant.now(), null)); + } + }, policy); + } + + /*************************************************************************** + ** Replay keeps the same row and does not repeat the retention transition. + ***************************************************************************/ + @Test + void sameIdentityAndBytesAreAdoptedWithoutUpload() throws Exception + { + assertThat(service.adoptRetainedAttachment(9).id()).isEqualTo(9); + assertThat(service.adoptRetainedAttachment(9).id()).isEqualTo(9); + assertThat(reads).isEqualTo(2); + assertThat(policy.transitions).isEqualTo(1); + assertThat(policy.registrations).isEqualTo(2); + assertThat(retained()).isTrue(); + } + + /*************************************************************************** + ** Failure before read cannot transition retention or register anything. + ***************************************************************************/ + @Test + void refusedIntentDoesNotReadOrChangeTheOwner() throws Exception + { + policy.failIntent = true; + assertThatThrownBy(() -> service.adoptRetainedAttachment(9)).hasMessage("intent refused"); + assertThat(reads).isZero(); + assertThat(retained()).isFalse(); + assertThat(policy.registrations).isZero(); + } + + /*************************************************************************** + ** The retention result callback is part of the owner's actual transaction. + ***************************************************************************/ + @Test + void outcomeFailureRollsBackRetention() throws Exception + { + policy.failOutcome = true; + assertThatThrownBy(() -> service.adoptRetainedAttachment(9)).hasMessage("outcome refused"); + assertThat(reads).isEqualTo(1); + assertThat(retained()).isFalse(); + assertThat(policy.registrations).isZero(); + } + + /*************************************************************************** + ** Later registration failure cannot pretend the earlier retention rolled back. + ***************************************************************************/ + @Test + void registrationFailureRemainsRetainedAndRetriesSameRow() throws Exception + { + policy.failRegistration = true; + assertThatThrownBy(() -> service.adoptRetainedAttachment(9)).hasMessage("registration refused"); + assertThat(retained()).isTrue(); + policy.failRegistration = false; + assertThat(service.adoptRetainedAttachment(9).id()).isEqualTo(9); + assertThat(policy.transitions).isEqualTo(1); + } + + /*************************************************************************** + ** Ordinary/default policy does not silently become an adoption authority. + ***************************************************************************/ + @Test + void absentPolicyAndDifferentAuthorRefuse() throws Exception + { + assertThatThrownBy(() -> new AttachmentsService(config, new AttachmentStorage(), new AttachmentLifecyclePolicy()) + .adoptRetainedAttachment(9)).hasMessageContaining("not configured"); + login("bob"); + assertThatThrownBy(() -> service.adoptRetainedAttachment(9)).hasMessageContaining("author"); + assertThat(reads).isZero(); + } + + /*************************************************************************** + ** Independent connection proves committed state rather than in-memory values. + ***************************************************************************/ + private boolean retained() throws Exception + { + try(var connection = DriverManager.getConnection(database.getJdbcUrl(), "sa", ""); + var statement = connection.createStatement(); var result = statement.executeQuery("SELECT retained FROM collaboration_attachment WHERE id=9")) + { + assertThat(result.next()).isTrue(); + return (result.getBoolean(1)); + } + } + + /*************************************************************************** + ** Explicit fixture hooks isolate owner transaction behavior from RF auditing. + ***************************************************************************/ + private static final class Policy extends AttachmentLifecyclePolicy + { + private boolean failIntent; + private boolean failOutcome; + private boolean failRegistration; + private int transitions; + private int registrations; + + /*********************************************************************** + ** Fixture current-authority grant only. + ***********************************************************************/ + @Override + public void authorizeAdoption(QRecord attachment) {} + + /*********************************************************************** + ** Fault before the owner opens existing bytes. + ***********************************************************************/ + @Override + public void beforeAdoption(QRecord attachment) throws QException + { + if(failIntent) + { + throw (new QException("intent refused")); + } + } + + /*********************************************************************** + ** The fixture admits its real H2 transaction without claiming RF durability. + ***********************************************************************/ + @Override + public void validateMetadataTransaction(QRecord attachment, QBackendTransaction transaction) {} + + /*********************************************************************** + ** Failure must roll back the actual retained column. + ***********************************************************************/ + @Override + public void afterAdoption(QRecord attachment, QBackendTransaction transaction) throws QException + { + if(failOutcome) + { + throw (new QException("outcome refused")); + } + transitions++; + } + + /*********************************************************************** + ** Failure occurs after the owner's retention commit. + ***********************************************************************/ + @Override + public void afterStored(QRecord attachment) throws QException + { + if(failRegistration) + { + throw (new QException("registration refused")); + } + registrations++; + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/attachments/UploadJournalVisibilityTest.java b/src/test/java/com/kingsrook/qbits/collaboration/attachments/UploadJournalVisibilityTest.java new file mode 100644 index 0000000..d31363b --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/attachments/UploadJournalVisibilityTest.java @@ -0,0 +1,140 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + + +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.collaboration.CollaborationRecords; +import com.kingsrook.qbits.collaboration.PersonalServiceFixture; +import com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadJournalTable; +import com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadRecoveryNavigation; +import com.kingsrook.qqq.backend.core.actions.tables.CountAction; +import com.kingsrook.qqq.backend.core.actions.tables.GetAction; +import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.expressions.CurrentUser; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordFeature; +import com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordFeaturesMetaData; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Native visibility and predicate tests. Memory is only the ordinary query + ** backend here; no memory durability or retained upload claim is made. + *******************************************************************************/ +public class UploadJournalVisibilityTest extends PersonalServiceFixture +{ + @Test + void journalRequiresOriginalActorAndCurrentParentVisibility() throws Exception + { + login("alice", "collaboration.view"); + QRecord row = seed(); + assertThat(new QueryAction().execute(new QueryInput(UploadJournalTable.NAME)).getRecords()).hasSize(1); + assertThat(new GetAction().execute(new GetInput(UploadJournalTable.NAME).withPrimaryKey(row.getValue("id"))).getRecord()).isNotNull(); + login("bob", "collaboration.view"); + assertThat(new QueryAction().execute(new QueryInput(UploadJournalTable.NAME)).getRecords()).isEmpty(); + login("alice", "collaboration.view"); + com.kingsrook.qqq.backend.core.context.QContext.getQSession().getSecurityKeyValues().remove("tenant"); + assertThat(new QueryAction().execute(new QueryInput(UploadJournalTable.NAME)).getRecords()).isEmpty(); + } + + @Test + void privatePhysicalColumnsCannotBeProbedOrProjected() throws Exception + { + login("alice", "collaboration.view"); + seed(); + for(String field : List.of("storageKey", "storage_key", "bucket_name", "manifest_hash", "lease_token")) + { + assertThatThrownBy(() -> new QueryAction().execute(new QueryInput(UploadJournalTable.NAME).withFilter( + new QQueryFilter().withCriteria(field, QCriteriaOperator.EQUALS, "probe")))).isInstanceOf(QPermissionDeniedException.class); + assertThatThrownBy(() -> new QueryAction().execute(new QueryInput(UploadJournalTable.NAME).withFilter( + new QQueryFilter().withOrderBy(new QFilterOrderBy(field, true))))).isInstanceOf(QPermissionDeniedException.class); + assertThatThrownBy(() -> new GetAction().execute(new GetInput(UploadJournalTable.NAME).withUniqueKey(Map.of(field, "probe")))) + .isInstanceOf(QPermissionDeniedException.class); + } + assertThatThrownBy(() -> new CountAction().execute(new CountInput(UploadJournalTable.NAME))).isInstanceOf(QPermissionDeniedException.class); + } + + @Test + void recoveryProcessRequiresOneVisibleSelectionAndExplicitConfirmation() throws Exception + { + login("alice", "collaboration.view"); + var metadata = RecoverRetainedUploadProcess.metadata(); + var prepare = new RecoverRetainedUploadProcess.Prepare(); + var input = new com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput().withProcessName(metadata.getName()); + var output = new com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput(); + assertThatThrownBy(() -> prepare.run(input, output)).hasMessageContaining("exactly one"); + input.setRecords(List.of(seed())); + prepare.run(input, output); + assertThat(output.getValue("journalId")).isNotNull(); + assertThat(output.getValue("confirmRecovery")).isEqualTo(false); + assertThatThrownBy(() -> new RecoverRetainedUploadProcess.Recover().run(input, new com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput())) + .hasMessageContaining("Confirm"); + } + + @Test + void recoveryNavigationUsesCurrentActorAndCanonicalSourcePins() throws Exception + { + login("alice", "collaboration.view"); + seed(); + var source = com.kingsrook.qqq.backend.core.context.QContext.getQInstance().getTable("document"); + UploadRecoveryNavigation.configure(source); + Object descriptor = RecordFeaturesMetaData.of(source).getSettings().get(RecordFeature.ATTACHMENTS).get("recoveryView"); + assertThat(descriptor).isInstanceOf(Map.class); + Map view = (Map) descriptor; + assertThat(view.get("tableName")).isEqualTo(UploadJournalTable.NAME); + assertThat(view.get("label")).isEqualTo("Review interrupted uploads"); + QQueryFilter filter = (QQueryFilter) view.get("filter"); + filter.withCriteria("tableName", QCriteriaOperator.EQUALS, "document").withCriteria("recordId", QCriteriaOperator.EQUALS, "1"); + assertThat(new QueryAction().execute(new QueryInput(UploadJournalTable.NAME).withFilter(filter)).getRecords()).hasSize(1); + login("bob", "collaboration.view"); + assertThat(new QueryAction().execute(new QueryInput(UploadJournalTable.NAME).withFilter(filter)).getRecords()).isEmpty(); + login("alice", "collaboration.view"); + for(String field : List.of("recordId", "state")) + { + assertThatThrownBy(() -> new QueryAction().execute(new QueryInput(UploadJournalTable.NAME).withFilter( + new QQueryFilter().withCriteria(field, QCriteriaOperator.EQUALS, new CurrentUser())))).isInstanceOf(QPermissionDeniedException.class); + } + assertThatThrownBy(() -> new QueryAction().execute(new QueryInput(UploadJournalTable.NAME).withFilter( + new QQueryFilter().withCriteria("createdByUserId", QCriteriaOperator.IN, new CurrentUser())))).isInstanceOf(QPermissionDeniedException.class); + } + + /******************************************************************************* + ** Public manifest projection only; physical storage secrets are not fields. + *******************************************************************************/ + private QRecord seed() throws Exception + { + return (CollaborationRecords.insert(UploadJournalTable.NAME, new QRecord().withValue("tableName", "document").withValue("recordId", "1") + .withValue("createdByUserId", "alice").withValue("fileName", "pending.txt").withValue("state", "PREPARED") + .withValue("storageState", "UNKNOWN").withValue("sha256", "a".repeat(64)).withValue("sizeBytes", 5L).withValue("locatorHash", "b".repeat(64)))); + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/MemoryUploadJournal.java b/src/test/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/MemoryUploadJournal.java new file mode 100644 index 0000000..5a29bf9 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/MemoryUploadJournal.java @@ -0,0 +1,263 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + +import java.time.Duration; +import java.time.Instant; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.UUID; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.exceptions.QException; + +/******************************************************************************* + ** Explicit test-only journal for existing memory-backend service behavior tests. + ** It provides no durability or rollback evidence. Production constructors use + ** JdbcUploadJournal; actual transaction/restart tests exercise that implementation. + *******************************************************************************/ +public final class MemoryUploadJournal implements UploadJournal +{ + private final Map rows = new LinkedHashMap<>(); + private final Map tokens = new LinkedHashMap<>(); + + /******************************************************************************* + ** Test-only implementation; does not establish durability. + *******************************************************************************/ + @Override + public synchronized UploadEntry prepare(UploadCandidate candidate) throws QException + { + for(UploadEntry entry : rows.values()) + { + if(entry.candidate().locatorHash().equals(candidate.locatorHash()) + || (candidate.intakeKey() != null && candidate.intakeKey().equals(entry.candidate().intakeKey()))) + { + if(!entry.candidate().equals(candidate)) + { + throw failure(UploadJournalException.Reason.IMMUTABLE_CONFLICT); + } + return (entry); + } + } + UploadEntry entry = new UploadEntry(rows.size() + 1, candidate, UploadJournalState.PREPARED, UploadStorageState.PENDING, null, 0L, null); + rows.put(entry.id(), entry); + return (entry); + } + + /******************************************************************************* + ** Test-only implementation; does not establish durability. + *******************************************************************************/ + @Override + public synchronized UploadReservation reserve(Integer journalId, Duration lease) throws QException + { + duration(lease); + UploadEntry entry = require(journalId); + if(entry.isLeased(Instant.now())) + { + throw failure(UploadJournalException.Reason.LEASE_HELD); + } + entry = entry.leased(entry.fence() + 1, Instant.now().plus(lease)); + rows.put(journalId, entry); + String token = UUID.randomUUID().toString(); + tokens.put(journalId, token); + return (new UploadReservation(entry, token, entry.fence(), entry.leaseUntil())); + } + + /******************************************************************************* + ** Test-only implementation; does not establish durability. + *******************************************************************************/ + @Override + public synchronized UploadReservation renew(UploadReservation reservation, Duration lease) throws QException + { + duration(lease); + UploadEntry entry = holder(reservation).leased(reservation.fence(), Instant.now().plus(lease)); + rows.put(entry.id(), entry); + return (new UploadReservation(entry, reservation.leaseToken(), entry.fence(), entry.leaseUntil())); + } + + /******************************************************************************* + ** Test-only implementation; does not establish durability. + *******************************************************************************/ + @Override + public synchronized UploadEntry recordStorage(UploadReservation reservation, UploadStorageState storageState) throws QException + { + UploadEntry entry = holder(reservation); + if(storageState == null || storageState.ordinal() < entry.storageState().ordinal()) + { + throw failure(UploadJournalException.Reason.ILLEGAL_STATE); + } + return (replace(entry, entry.state(), storageState, entry.attachmentId(), entry.leaseUntil())); + } + + /******************************************************************************* + ** Test-only implementation; does not establish durability. + *******************************************************************************/ + @Override + public synchronized UploadEntry register(UploadReservation reservation, AttachmentRegistrar registrar) throws QException + { + UploadEntry entry = holder(reservation); + if(entry.attachmentId() != null) + { + return (entry); + } + if(entry.storageState() != UploadStorageState.VERIFIED) + { + throw failure(UploadJournalException.Reason.STORAGE_NOT_VERIFIED); + } + // Memory-backend fixtures do not prove atomic rollback. The JDBC tests do. + Integer attachment = registrar.insert(new QBackendTransaction(), entry.candidate(), entry.id()); + return (replace(entry, UploadJournalState.REGISTERED, entry.storageState(), attachment, entry.leaseUntil())); + } + + /******************************************************************************* + ** Test-only implementation; does not establish durability. + *******************************************************************************/ + @Override + public synchronized UploadEntry complete(UploadReservation reservation) throws QException + { + UploadEntry entry = require(reservation.journalId()); + if(entry.state() == UploadJournalState.COMPLETE && entry.fence().equals(reservation.fence())) + { + return (entry); + } + entry = holder(reservation); + if(entry.state() != UploadJournalState.REGISTERED || entry.attachmentId() == null) + { + throw failure(UploadJournalException.Reason.ILLEGAL_STATE); + } + tokens.remove(entry.id()); + return (replace(entry, UploadJournalState.COMPLETE, entry.storageState(), entry.attachmentId(), null)); + } + + /******************************************************************************* + ** Test-only implementation; does not establish durability. + *******************************************************************************/ + @Override + public synchronized boolean release(UploadReservation reservation) throws QException + { + UploadEntry entry = require(reservation.journalId()); + if(entry.state() == UploadJournalState.COMPLETE || !matches(entry, reservation)) + { + return (false); + } + tokens.remove(entry.id()); + replace(entry, entry.state(), entry.storageState(), entry.attachmentId(), null); + return (true); + } + + /******************************************************************************* + ** Test-only implementation; does not establish durability. + *******************************************************************************/ + @Override + public synchronized Optional find(Integer journalId) + { + return (Optional.ofNullable(rows.get(journalId))); + } + + /******************************************************************************* + ** Test-only implementation; does not establish durability. + *******************************************************************************/ + @Override + public synchronized Optional findByLocation(String bucketName, String storageKey) + { + String locator = UploadCandidate.locatorHash(bucketName, storageKey); + return (rows.values().stream().filter(entry -> entry.candidate().locatorHash().equals(locator)).findFirst()); + } + + /******************************************************************************* + ** Test-only implementation; does not establish durability. + *******************************************************************************/ + @Override + public synchronized List listIncomplete(int limit) + { + if(limit < 1 || limit > 500) + { + throw new IllegalArgumentException("Invalid fixture limit."); + } + return (rows.values().stream().filter(entry -> entry.state() != UploadJournalState.COMPLETE).limit(limit).toList()); + } + + /******************************************************************************* + ** Enforce fixture state without claiming production transaction guarantees. + *******************************************************************************/ + private UploadEntry require(Integer id) throws QException + { + UploadEntry entry = rows.get(id); + if(entry == null) + { + throw failure(UploadJournalException.Reason.NOT_FOUND); + } + return (entry); + } + + /******************************************************************************* + ** Enforce fixture state without claiming production transaction guarantees. + *******************************************************************************/ + private UploadEntry holder(UploadReservation reservation) throws QException + { + UploadEntry entry = require(reservation.journalId()); + if(!matches(entry, reservation)) + { + throw failure(UploadJournalException.Reason.STALE_RESERVATION); + } + return (entry); + } + + /******************************************************************************* + ** Enforce fixture state without claiming production transaction guarantees. + *******************************************************************************/ + private boolean matches(UploadEntry entry, UploadReservation reservation) + { + return (entry.fence().equals(reservation.fence()) && entry.isLeased(Instant.now()) + && reservation.leaseToken().equals(tokens.get(entry.id()))); + } + + /******************************************************************************* + ** Enforce fixture state without claiming production transaction guarantees. + *******************************************************************************/ + private UploadEntry replace(UploadEntry entry, UploadJournalState state, UploadStorageState storage, Integer attachment, Instant leaseUntil) + { + UploadEntry changed = new UploadEntry(entry.id(), entry.candidate(), state, storage, attachment, entry.fence(), leaseUntil); + rows.put(entry.id(), changed); + return (changed); + } + + /******************************************************************************* + ** Enforce fixture state without claiming production transaction guarantees. + *******************************************************************************/ + private static void duration(Duration lease) + { + if(lease == null || lease.compareTo(Duration.ofSeconds(1)) < 0 || lease.compareTo(Duration.ofMinutes(30)) > 0) + { + throw new IllegalArgumentException("Invalid fixture lease."); + } + } + + /******************************************************************************* + ** Enforce fixture state without claiming production transaction guarantees. + *******************************************************************************/ + private static UploadJournalException failure(UploadJournalException.Reason reason) + { + return (new UploadJournalException(reason, "Memory test journal refused: " + reason)); + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/MySqlUploadJournalTest.java b/src/test/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/MySqlUploadJournalTest.java new file mode 100644 index 0000000..dc6133b --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/MySqlUploadJournalTest.java @@ -0,0 +1,697 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.sql.Statement; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.TimeZone; +import java.util.UUID; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import com.github.dockerjava.api.model.ExposedPort; +import com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadJournalException.Reason; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.core.model.session.QSession; +import com.kingsrook.qqq.backend.core.model.session.QUser; +import com.kingsrook.qqq.backend.module.rdbms.RDBMSBackendModule; +import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction; +import com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSTableBackendDetails; +import org.apache.logging.log4j.Level; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.core.config.Configurator; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfSystemProperty; +import org.junit.jupiter.api.extension.ExtensionContext; +import org.junit.jupiter.api.extension.RegisterExtension; +import org.junit.jupiter.api.extension.TestWatcher; +import org.testcontainers.mysql.MySQLContainer; +import org.testcontainers.utility.DockerImageName; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Explicit opt-in real MySQL 8.4 qualification. The container and anonymous data + ** volume belong only to this suite; no existing server, database or credentials + ** are used. The attachment row is a minimal stand-in, not a full intake fixture. + *******************************************************************************/ +@EnabledIfSystemProperty(named = "mysqlJournalTests", matches = "true") +class MySqlUploadJournalTest +{ + private static final String IMAGE = "mysql@sha256:a0225d1ea6631c8bc2385dc4c2faba17bbf1834ecf369e1659a9a382bd8d5dd7"; + private static final Instant START = Instant.parse("2026-10-30T12:00:00.123Z"); + private static MySQLContainer mysql; + private static int currentPort; + private static final Map LOG_LEVELS = new HashMap<>(); + private RDBMSBackendMetaData backend; + private QInstance instance; + private JdbcUploadJournal journal; + + @RegisterExtension + static final TestWatcher FAILURE_DIAGNOSTICS = new TestWatcher() + { + /******************************************************************************* + ** Capture only bounded health/counts after failure, before container teardown. + *******************************************************************************/ + @Override + public void testFailed(ExtensionContext context, Throwable cause) + { + diagnoseFailure(cause); + } + }; + + + /******************************************************************************* + ** Never log exception messages, SQL, credentials, row values or session IDs. + ** Independent diagnostic deadlines do not alter application/test deadlines. + *******************************************************************************/ + private static void diagnoseFailure(Throwable failure) + { + diagnosticCause(failure); + try + { + var inspection = mysql.getDockerClient().inspectContainerCmd(mysql.getContainerId()).exec(); + var state = inspection.getState(); + System.out.println("MYSQL_FIXTURE_HEALTH running=" + state.getRunning() + " oomKilled=" + state.getOOMKilled() + + " exitCode=" + state.getExitCodeLong() + " restarts=" + inspection.getRestartCount()); + } + catch(Exception unavailable) + { + diagnosticCause(unavailable); + } + try(Connection connection = DriverManager.getConnection("jdbc:mysql://" + mysql.getHost() + ":" + currentPort + "/" + mysql.getDatabaseName() + + "?useSSL=false&allowPublicKeyRetrieval=true&connectTimeout=1000&socketTimeout=1000", mysql.getUsername(), mysql.getPassword()); + Statement statement = connection.createStatement()) + { + statement.setQueryTimeout(1); + try(ResultSet result = statement.executeQuery("SELECT 1")) + { + System.out.println("MYSQL_FIXTURE_SELECT_ONE success=" + (result.next() && result.getInt(1) == 1)); + } + try(ResultSet result = statement.executeQuery("SELECT COUNT(*), SUM(COMMAND = 'Sleep') FROM information_schema.PROCESSLIST")) + { + if(result.next()) + { + System.out.println("MYSQL_FIXTURE_CONNECTIONS total=" + result.getLong(1) + " sleeping=" + result.getLong(2)); + } + } + try(ResultSet result = statement.executeQuery("SELECT COUNT(*), SUM(TRX_STATE = 'LOCK WAIT') FROM information_schema.INNODB_TRX")) + { + if(result.next()) + { + System.out.println("MYSQL_FIXTURE_TRANSACTIONS total=" + result.getLong(1) + " lockWaiting=" + result.getLong(2)); + } + } + try(ResultSet result = statement.executeQuery("SELECT COUNT(*), SUM(LOCK_STATUS = 'PENDING') FROM performance_schema.metadata_locks")) + { + if(result.next()) + { + System.out.println("MYSQL_FIXTURE_METADATA_LOCKS total=" + result.getLong(1) + " pending=" + result.getLong(2)); + } + } + } + catch(Exception unavailable) + { + diagnosticCause(unavailable); + } + } + + + /******************************************************************************* + ** A bounded class/state chain is safe even when driver messages contain secrets. + *******************************************************************************/ + private static void diagnosticCause(Throwable failure) + { + for(int depth = 0; failure != null && depth < 12; depth++, failure = failure.getCause()) + { + String detail = ""; + if(failure instanceof SQLException sql) + { + String state = sql.getSQLState(); + detail = " state=" + (state != null && state.matches("[A-Z0-9]{5}") ? state : "unknown") + " code=" + sql.getErrorCode(); + } + System.out.println("MYSQL_FIXTURE_FAILURE class=" + failure.getClass().getName() + detail); + } + } + + + + /******************************************************************************* + ** Start only the pinned disposable provider, with a random synthetic password. + *******************************************************************************/ + @BeforeAll + static void start() + { + for(String logger : List.of("org.apache.hc", "org.apache.http", "org.testcontainers", "com.github.dockerjava")) + { + LOG_LEVELS.put(logger, LogManager.getLogger(logger).getLevel()); + Configurator.setLevel(logger, Level.WARN); + } + mysql = new MySQLContainer(DockerImageName.parse(IMAGE)).withDatabaseName("journal_test").withUsername("root") + .withPassword(UUID.randomUUID().toString()).withCommand("--innodb-flush-log-at-trx-commit=1", "--sync-binlog=1", "--innodb-lock-wait-timeout=5"); + mysql.start(); + currentPort = mysql.getMappedPort(3306); + } + + + /******************************************************************************* + ** Destroy only the provider created by this suite and its private volume. + *******************************************************************************/ + @AfterAll + static void stop() + { + if(mysql != null) + { + mysql.stop(); + } + LOG_LEVELS.forEach(Configurator::setLevel); + } + + + /******************************************************************************* + ** Exercise native URL construction, rather than configuring a custom provider. + *******************************************************************************/ + @BeforeEach + void setUp() throws Exception + { + ConnectionManager.resetConnectionProviders(); + backend = new RDBMSBackendMetaData().withName("mysql_journal").withVendor("mysql").withHostName(mysql.getHost()) + .withPort(currentPort).withDatabaseName(mysql.getDatabaseName()).withUsername(mysql.getUsername()).withPassword(mysql.getPassword()); + instance = new QInstance().withInstanceDefaultAuthentication(new QAuthenticationMetaData().withName("anonymous").withType(QAuthenticationType.FULLY_ANONYMOUS)); + instance.addBackend(backend); + for(String name : List.of(UploadJournalTable.NAME, CollaborationTables.ATTACHMENT)) + { + instance.addTable(new QTableMetaData().withName(name).withBackendName(backend.getName()).withPrimaryKeyField("id") + .withField(new QFieldMetaData("id", QFieldType.INTEGER))); + } + context(); + execute("DROP TABLE IF EXISTS collaboration_upload_journal"); + execute("DROP TABLE IF EXISTS collaboration_attachment"); + execute(UploadJournalTable.mysqlScript()); + execute("CREATE TABLE collaboration_attachment (id INTEGER AUTO_INCREMENT PRIMARY KEY, file_name VARCHAR(255)) ENGINE=InnoDB"); + journal = new JdbcUploadJournal(Clock.fixed(START, ZoneOffset.UTC)); + } + + + /******************************************************************************* + ** No connection or thread-local authority survives a fixture. + *******************************************************************************/ + @AfterEach + void close() throws Exception + { + QContext.clearTransaction(); + QContext.clear(); + ConnectionManager.resetConnectionProviders(); + } + + + /******************************************************************************* + ** Acknowledged metadata survives a real database process restart, not merely + ** an open pooled connection. The same candidate replays its original row. + *******************************************************************************/ + @Test + void committedCandidateSurvivesDatabaseRestart() throws Exception + { + UploadEntry original = journal.prepare(candidate("restart")); + ConnectionManager.resetConnectionProviders(); + mysql.getDockerClient().restartContainerCmd(mysql.getContainerId()).exec(); + currentPort = Integer.parseInt(mysql.getCurrentContainerInfo().getNetworkSettings().getPorts().getBindings().get(new ExposedPort(3306))[0].getHostPortSpec()); + backend.setPort(currentPort); + long deadline = System.nanoTime() + Duration.ofSeconds(60).toNanos(); + boolean ready = false; + while(System.nanoTime() < deadline) + { + try(Connection connection = connection(); Statement statement = connection.createStatement()) + { + statement.execute("SELECT 1"); + ready = true; + break; + } + catch(SQLException transientStartup) + { + Thread.sleep(250); + } + } + assertThat(ready).isTrue(); + assertThat(journal.prepare(candidate("restart")).id()).isEqualTo(original.id()); + assertThat(count("collaboration_upload_journal")).isEqualTo(1); + } + + + /******************************************************************************* + ** A registrar failure rolls back both its attachment and the owner checkpoint. + *******************************************************************************/ + @Test + void registrationAndCheckpointCommitOrRollbackTogether() throws Exception + { + UploadReservation reservation = verified("rollback"); + assertThatThrownBy(() -> journal.register(reservation, (transaction, candidate, id) -> + { + insertAttachment(transaction); + throw (new QException("Synthetic registrar failure.")); + })).isInstanceOf(QException.class); + assertThat(count("collaboration_attachment")).isZero(); + assertThat(journal.find(reservation.journalId()).orElseThrow().state()).isEqualTo(UploadJournalState.PREPARED); + UploadEntry registered = journal.register(reservation, (transaction, candidate, id) -> insertAttachment(transaction)); + assertThat(registered.state()).isEqualTo(UploadJournalState.REGISTERED); + assertThat(journal.complete(reservation).state()).isEqualTo(UploadJournalState.COMPLETE); + assertThat(journal.complete(reservation).attachmentId()).isEqualTo(registered.attachmentId()); + assertThat(count("collaboration_attachment")).isEqualTo(1); + } + + + /******************************************************************************* + ** Unique preparation and exclusive leases hold across real simultaneous JDBC + ** connections. A new fence rejects the old holder after expiry. + *******************************************************************************/ + @Test + void concurrentPrepareAndReservationFence() throws Exception + { + try(var executor = Executors.newFixedThreadPool(4)) + { + CountDownLatch start = new CountDownLatch(1); + List> prepares = new ArrayList<>(); + for(int i = 0; i < 4; i++) + { + prepares.add(executor.submit(() -> + { + context(); + start.await(); + return (journal.prepare(candidate("race"))); + })); + } + start.countDown(); + Integer id = prepares.getFirst().get(30, TimeUnit.SECONDS).id(); + for(Future prepared : prepares) + { + assertThat(prepared.get(30, TimeUnit.SECONDS).id()).isEqualTo(id); + } + assertThat(count("collaboration_upload_journal")).isEqualTo(1); + List> reservations = new ArrayList<>(); + CountDownLatch reserveStart = new CountDownLatch(1); + for(int i = 0; i < 4; i++) + { + reservations.add(executor.submit(() -> + { + context(); + reserveStart.await(); + try + { + return (journal.reserve(id, Duration.ofSeconds(5))); + } + catch(UploadJournalException denied) + { + return (denied.getReason()); + } + })); + } + reserveStart.countDown(); + List outcomes = new ArrayList<>(); + for(Future reservation : reservations) + { + outcomes.add(reservation.get(30, TimeUnit.SECONDS)); + } + assertThat(outcomes.stream().filter(UploadReservation.class::isInstance)).hasSize(1); + assertThat(outcomes.stream().filter(Reason.class::isInstance)).containsOnly(Reason.LEASE_HELD); + UploadReservation old = (UploadReservation) outcomes.stream().filter(UploadReservation.class::isInstance).findFirst().orElseThrow(); + JdbcUploadJournal later = new JdbcUploadJournal(Clock.fixed(START.plusSeconds(6), ZoneOffset.UTC)); + UploadReservation next = later.reserve(id, Duration.ofSeconds(5)); + assertThat(next.fence()).isGreaterThan(old.fence()); + assertThatThrownBy(() -> later.recordStorage(old, UploadStorageState.VERIFIED)).isInstanceOf(UploadJournalException.class); + } + } + + + /******************************************************************************* + ** Wrong metadata mappings, catalogs, and ambient transactions fail before a + ** manifest write. Aurora is not an alias for the tested MySQL provider. + *******************************************************************************/ + @Test + void rejectsWrongBackendCatalogMappingAndAmbientTransaction() throws Exception + { + instance.addBackend(new RDBMSBackendMetaData().withName("other_backend").withVendor("mysql")); + instance.getTable(CollaborationTables.ATTACHMENT).setBackendName("other_backend"); + assertUnsupported(); + instance.getTable(CollaborationTables.ATTACHMENT).setBackendName(backend.getName()); + instance.getTable(CollaborationTables.ATTACHMENT).setBackendDetails(new RDBMSTableBackendDetails().withTableName("other_attachment")); + assertUnsupported(); + instance.getTable(CollaborationTables.ATTACHMENT).setBackendDetails(null); + try(Connection connection = connection()) + { + connection.setAutoCommit(false); + connection.setCatalog("mysql"); + assertThatThrownBy(() -> MySqlJournalDatabase.requireQualifiedConnection(connection, backend)).isInstanceOf(UploadJournalException.class); + } + backend.setVendor("aurora-mysql"); + assertUnsupported(); + backend.setVendor("mysql"); + try(QBackendTransaction transaction = new RDBMSBackendModule().openTransaction(new InsertInput(UploadJournalTable.NAME))) + { + QContext.setTransaction(transaction); + assertThatThrownBy(() -> journal.prepare(candidate("denied"))).isInstanceOf(UploadJournalException.class) + .extracting(e -> ((UploadJournalException) e).getReason()).isEqualTo(Reason.AMBIENT_TRANSACTION); + } + QContext.clearTransaction(); + assertThat(count("collaboration_upload_journal")).isZero(); + } + + + /******************************************************************************* + ** Runtime settings and the real attachment engine are authoritative. Restore + ** all disposable-server globals even when an assertion fails. + *******************************************************************************/ + @Test + void rejectsNonDurableSettingsAndNonTransactionalAttachment() throws Exception + { + try + { + execute("SET GLOBAL innodb_flush_log_at_trx_commit=2"); + assertUnsupported(); + execute("SET GLOBAL innodb_flush_log_at_trx_commit=1"); + execute("SET GLOBAL sync_binlog=0"); + assertUnsupported(); + } + finally + { + execute("SET GLOBAL innodb_flush_log_at_trx_commit=1"); + execute("SET GLOBAL sync_binlog=1"); + } + execute("ALTER TABLE collaboration_attachment ENGINE=MyISAM"); + assertUnsupported(); + assertThat(count("collaboration_upload_journal")).isZero(); + } + + + /******************************************************************************* + ** The SQL schema itself preserves case and trailing spaces in identifiers, + ** and native generic writes cannot bypass journal ownership. + *******************************************************************************/ + @Test + void binaryIdentifiersAndNativeWritePolicy() throws Exception + { + journal.prepare(candidate("binary")); + try(Connection connection = connection(); Statement statement = connection.createStatement(); + ResultSet result = statement.executeQuery("SELECT COUNT(*) FROM collaboration_upload_journal WHERE created_by_user_id IN ('Alice', 'alice ')")) + { + result.next(); + assertThat(result.getInt(1)).isZero(); + } + assertThatThrownBy(() -> new UploadJournalSecurity().checkPermissionsThrowing(new InsertInput(UploadJournalTable.NAME), instance.getTable(UploadJournalTable.NAME))) + .isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + execute("ALTER TABLE collaboration_upload_journal DEFAULT COLLATE=utf8mb4_0900_ai_ci"); + assertUnsupported(); + } + + + /******************************************************************************* + ** The actual server lock timeout maps to BUSY and leaves no granted lease. + *******************************************************************************/ + @Test + void lockTimeoutIsBoundedAndDoesNotGrantLease() throws Exception + { + Integer id = journal.prepare(candidate("locked")).id(); + try(Connection connection = connection(); PreparedStatement lock = connection.prepareStatement("SELECT id FROM collaboration_upload_journal WHERE id=? FOR UPDATE")) + { + connection.setAutoCommit(false); + lock.setInt(1, id); + try(ResultSet row = lock.executeQuery()) + { + assertThat(row.next()).isTrue(); + long started = System.nanoTime(); + assertThatThrownBy(() -> journal.reserve(id, Duration.ofSeconds(5))).isInstanceOf(UploadJournalException.class) + .extracting(e -> ((UploadJournalException) e).getReason()).isEqualTo(Reason.BUSY); + System.out.println("MYSQL_FIXTURE_PHASE reserveBusyMillis=" + TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - started)); + } + System.out.println("MYSQL_FIXTURE_PHASE beforeIndependentRollback"); + diagnoseFailure(null); + connection.rollback(); + System.out.println("MYSQL_FIXTURE_PHASE afterIndependentRollback"); + } + assertThat(journal.find(id).orElseThrow().fence()).isZero(); + assertThat(journal.reserve(id, Duration.ofSeconds(5)).fence()).isEqualTo(1L); + } + + + /******************************************************************************* + ** Actual column/index drift is refused even if the table default still looks + ** correct. A changed candidate at the original location remains a conflict. + *******************************************************************************/ + @Test + void columnAndIndexDriftAndManifestConflictFailClosed() throws Exception + { + UploadCandidate original = candidate("immutable"); + journal.prepare(original); + UploadCandidate changed = new UploadCandidate(original.tableName(), original.recordId(), original.actorId(), "changed.pdf", original.contentType(), original.description(), + original.sizeBytes(), original.sha256(), original.bucketName(), original.storageKey(), true, null, null); + assertThatThrownBy(() -> journal.prepare(changed)).isInstanceOf(UploadJournalException.class) + .extracting(e -> ((UploadJournalException) e).getReason()).isEqualTo(Reason.IMMUTABLE_CONFLICT); + execute("ALTER TABLE collaboration_upload_journal MODIFY created_by_user_id VARCHAR(255) COLLATE utf8mb4_0900_ai_ci NOT NULL"); + assertUnsupported(); + execute("ALTER TABLE collaboration_upload_journal MODIFY created_by_user_id VARCHAR(255) COLLATE utf8mb4_0900_bin NOT NULL"); + execute("DROP INDEX uq_collaboration_upload_journal_locator ON collaboration_upload_journal"); + assertUnsupported(); + assertThat(count("collaboration_upload_journal")).isEqualTo(1); + } + + + /******************************************************************************* + ** Lease instants are stable when app hosts use different JVM default zones. + ** A non-UTC database session is refused rather than silently shifting leases. + *******************************************************************************/ + @Test + void leaseInstantsUseExplicitUtcAcrossJvmZones() throws Exception + { + TimeZone prior = TimeZone.getDefault(); + try + { + TimeZone.setDefault(TimeZone.getTimeZone("Pacific/Honolulu")); + UploadReservation reservation = verified("timezone"); + assertThat(reservation.leaseUntil()).isEqualTo(START.plusSeconds(60)); + ConnectionManager.resetConnectionProviders(); + TimeZone.setDefault(TimeZone.getTimeZone("Pacific/Kiritimati")); + assertThat(journal.find(reservation.journalId()).orElseThrow().leaseUntil()).isEqualTo(START.plusSeconds(60)); + assertThat(journal.renew(reservation, Duration.ofMinutes(2)).leaseUntil()).isEqualTo(START.plusSeconds(120)); + try(Connection connection = connection(); Statement statement = connection.createStatement()) + { + connection.setAutoCommit(false); + statement.execute("SET SESSION time_zone='+01:00'"); + assertThatThrownBy(() -> MySqlJournalDatabase.requireQualifiedConnection(connection, backend)).isInstanceOf(UploadJournalException.class); + } + } + finally + { + ConnectionManager.resetConnectionProviders(); + TimeZone.setDefault(prior); + } + } + + + /******************************************************************************* + ** Qualification never mutates a host's pooled session. Accepted settings are + ** preserved, while an excessive wait is refused and also left unchanged. + *******************************************************************************/ + @Test + void lockTimeoutQualificationPreservesHostSession() throws Exception + { + try(Connection connection = connection(); Statement statement = connection.createStatement()) + { + connection.setAutoCommit(false); + statement.execute("SET SESSION innodb_lock_wait_timeout=3"); + MySqlJournalDatabase.requireQualifiedConnection(connection, backend); + try(ResultSet value = statement.executeQuery("SELECT @@SESSION.innodb_lock_wait_timeout")) + { + value.next(); + assertThat(value.getInt(1)).isEqualTo(3); + } + statement.execute("SET SESSION innodb_lock_wait_timeout=6"); + assertThatThrownBy(() -> MySqlJournalDatabase.requireQualifiedConnection(connection, backend)).isInstanceOf(UploadJournalException.class) + .extracting(e -> ((UploadJournalException) e).getReason()).isEqualTo(Reason.UNSUPPORTED_BACKEND); + try(ResultSet value = statement.executeQuery("SELECT @@SESSION.innodb_lock_wait_timeout")) + { + value.next(); + assertThat(value.getInt(1)).isEqualTo(6); + } + } + } + + + /******************************************************************************* + ** Detect-only page protection and read-only sessions cannot promise durable + ** journal writes. Qualification must leave the host session unchanged. + *******************************************************************************/ + @Test + void doublewriteAndSessionReadOnlyAreRequired() throws Exception + { + try + { + execute("SET GLOBAL innodb_doublewrite=DETECT_ONLY"); + assertUnsupported(); + } + finally + { + execute("SET GLOBAL innodb_doublewrite=ON"); + } + try(Connection connection = connection(); Statement statement = connection.createStatement()) + { + connection.setAutoCommit(false); + statement.execute("SET SESSION transaction_read_only=1"); + assertThatThrownBy(() -> MySqlJournalDatabase.requireQualifiedConnection(connection, backend)).isInstanceOf(UploadJournalException.class) + .extracting(e -> ((UploadJournalException) e).getReason()).isEqualTo(Reason.UNSUPPORTED_BACKEND); + try(ResultSet value = statement.executeQuery("SELECT @@SESSION.transaction_read_only")) + { + value.next(); + assertThat(value.getBoolean(1)).isTrue(); + } + statement.execute("SET SESSION transaction_read_only=0"); + MySqlJournalDatabase.requireQualifiedConnection(connection, backend); + } + assertThat(journal.prepare(candidate("restored-durability")).id()).isPositive(); + } + + + /******************************************************************************* + ** Assert a typed refusal, with no connection URL in its public message. + *******************************************************************************/ + private void assertUnsupported() + { + assertThatThrownBy(() -> journal.prepare(candidate("denied"))).isInstanceOf(UploadJournalException.class) + .extracting(e -> ((UploadJournalException) e).getReason()).isEqualTo(Reason.UNSUPPORTED_BACKEND); + } + + + /******************************************************************************* + ** Prepare a leased manifest whose stored bytes have been verified by owner. + *******************************************************************************/ + private UploadReservation verified(String label) throws Exception + { + UploadReservation reservation = journal.reserve(journal.prepare(candidate(label)).id(), Duration.ofMinutes(1)); + journal.recordStorage(reservation, UploadStorageState.VERIFIED); + return (reservation); + } + + + /******************************************************************************* + ** Insert only through the owner's exact transaction. + *******************************************************************************/ + private static Integer insertAttachment(QBackendTransaction transaction) throws QException + { + try(PreparedStatement statement = ((RDBMSTransaction) transaction).getConnection() + .prepareStatement("INSERT INTO collaboration_attachment(file_name) VALUES ('fixture.pdf')", Statement.RETURN_GENERATED_KEYS)) + { + statement.executeUpdate(); + try(ResultSet keys = statement.getGeneratedKeys()) + { + keys.next(); + return (keys.getInt(1)); + } + } + catch(SQLException e) + { + throw (new QException("Synthetic attachment insert failed.", e)); + } + } + + + /******************************************************************************* + ** Independent connection for committed-state assertions and fixture setup. + *******************************************************************************/ + private Connection connection() throws SQLException + { + return (DriverManager.getConnection("jdbc:mysql://" + mysql.getHost() + ":" + currentPort + "/" + mysql.getDatabaseName() + + "?useSSL=false&allowPublicKeyRetrieval=true&connectTimeout=1000&socketTimeout=10000", mysql.getUsername(), mysql.getPassword())); + } + + + /******************************************************************************* + ** Execute fixed trusted fixture SQL, never application-supplied identifiers. + *******************************************************************************/ + private void execute(String sql) throws SQLException + { + try(Connection connection = connection(); Statement statement = connection.createStatement()) + { + statement.execute(sql); + } + } + + + /******************************************************************************* + ** Count a fixed owner table through a new committed connection. + *******************************************************************************/ + private long count(String table) throws SQLException + { + try(Connection connection = connection(); Statement statement = connection.createStatement(); ResultSet result = statement.executeQuery("SELECT COUNT(*) FROM " + table)) + { + result.next(); + return (result.getLong(1)); + } + } + + + /******************************************************************************* + ** Establish the same synthetic actor independently on each test thread. + *******************************************************************************/ + private void context() + { + QContext.init(instance, new QSession().withUser(new QUser().withIdReference("alice"))); + } + + + /******************************************************************************* + ** Server-shaped immutable candidate; no real customer data or storage is used. + *******************************************************************************/ + private static UploadCandidate candidate(String label) + { + return (new UploadCandidate("document", "1", "alice", label + ".pdf", "application/pdf", "Synthetic fixture", 5L, "a".repeat(64), + "fixture-bucket", "retained/" + label, true, null, null)); + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/PostgresUploadJournalTest.java b/src/test/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/PostgresUploadJournalTest.java new file mode 100644 index 0000000..2b68742 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/PostgresUploadJournalTest.java @@ -0,0 +1,689 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.sql.Statement; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.List; +import java.util.TimeZone; +import java.util.UUID; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import com.github.dockerjava.api.model.ExposedPort; +import com.github.dockerjava.api.model.PortBinding; +import com.github.dockerjava.api.model.Ports; +import com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadJournalException.Reason; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.core.model.session.QSession; +import com.kingsrook.qqq.backend.core.model.session.QUser; +import com.kingsrook.qqq.backend.module.postgres.model.metadata.PostgreSQLBackendMetaData; +import com.kingsrook.qqq.backend.module.rdbms.RDBMSBackendModule; +import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction; +import com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSTableBackendDetails; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.condition.EnabledIfSystemProperty; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.wait.strategy.Wait; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Explicit opt-in real PostgreSQL 17 qualification of the upload journal on the + ** native postgres backend. The container belongs only to this suite (loopback, + ** dynamic port, random synthetic passwords); the application connects as a + ** DML-only role, and the superuser is used only for fixture DDL and fault + ** injection. The attachment row is a minimal stand-in, not a full intake fixture. + *******************************************************************************/ +@EnabledIfSystemProperty(named = "postgresJournalTests", matches = "true") +class PostgresUploadJournalTest +{ + private static final Instant START = Instant.parse("2026-10-30T12:00:00.123Z"); + private static final String DATABASE = "journal_test"; + private static final String APP_ROLE = "journal_app"; + private static final String ROOT_PASSWORD = UUID.randomUUID().toString(); + private static final String APP_PASSWORD = UUID.randomUUID().toString(); + private static final String OPTIONS = "?connectTimeout=5&socketTimeout=30&options=-c%20TimeZone%3DUTC"; + private static GenericContainer postgres; + private static TimeZone previousZone; + private static int currentPort; + // opt-in: reach the disposable server by its container bridge address (no host port published) - for the + // sibling-container Linux gate, where the host loopback is not the test JVM's loopback + private static final boolean BRIDGE_ADDRESS = Boolean.getBoolean("postgresJournalTests.bridgeAddress"); + private static String currentHost = "127.0.0.1"; + private PostgreSQLBackendMetaData backend; + private QInstance instance; + private JdbcUploadJournal journal; + + + + /******************************************************************************* + ** Start only the disposable server with explicit durable settings. pgjdbc + ** announces the JVM zone as the session TimeZone, so the suite runs in UTC. + *******************************************************************************/ + @BeforeAll + static void start() throws Exception + { + previousZone = TimeZone.getDefault(); + TimeZone.setDefault(TimeZone.getTimeZone("UTC")); + postgres = new GenericContainer<>("postgres:17") + .withEnv("POSTGRES_PASSWORD", ROOT_PASSWORD).withEnv("POSTGRES_DB", DATABASE) + .withCommand("postgres", "-c", "fsync=on", "-c", "synchronous_commit=on", "-c", "full_page_writes=on") + .withExposedPorts(5432) + .withCreateContainerCmdModifier(command -> + { + command.getHostConfig().withMemory(768L * 1024 * 1024); + if(!BRIDGE_ADDRESS) + { + command.getHostConfig().withPortBindings(new PortBinding(Ports.Binding.bindIpAndPort("127.0.0.1", 0), new ExposedPort(5432))); + } + }) + .waitingFor(Wait.forLogMessage(".*database system is ready to accept connections.*\\s", 2).withStartupTimeout(Duration.ofMinutes(2))); + postgres.start(); + locateServer(); + try(Connection connection = root(); Statement statement = connection.createStatement()) + { + statement.execute("CREATE ROLE " + APP_ROLE + " LOGIN PASSWORD '" + APP_PASSWORD + "' NOSUPERUSER NOCREATEDB NOCREATEROLE"); + statement.execute("ALTER ROLE " + APP_ROLE + " SET lock_timeout='5s'"); + statement.execute("DROP SCHEMA public CASCADE"); + statement.execute("CREATE SCHEMA public"); + statement.execute("GRANT USAGE ON SCHEMA public TO " + APP_ROLE); + statement.execute("ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO " + APP_ROLE); + statement.execute("ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT USAGE, SELECT ON SEQUENCES TO " + APP_ROLE); + } + } + + + + /******************************************************************************* + ** Destroy only the container created by this suite and restore the JVM zone. + *******************************************************************************/ + @AfterAll + static void stop() + { + ConnectionManager.closeConnectionProviders(); + if(postgres != null) + { + postgres.close(); + } + if(previousZone != null) + { + TimeZone.setDefault(previousZone); + } + } + + + + /******************************************************************************* + ** Native postgres backend metadata with the DML-only identity, a fresh journal + ** and attachment pair, and the shipped owner script. + *******************************************************************************/ + @BeforeEach + void setUp() throws Exception + { + ConnectionManager.closeConnectionProviders(); + TimeZone.setDefault(TimeZone.getTimeZone("UTC")); + execute("ALTER ROLE " + APP_ROLE + " SET lock_timeout='5s'"); + execute("ALTER ROLE " + APP_ROLE + " RESET synchronous_commit"); + execute("ALTER ROLE " + APP_ROLE + " RESET default_transaction_isolation"); + execute("ALTER ROLE " + APP_ROLE + " RESET default_transaction_read_only"); + backend = new PostgreSQLBackendMetaData(); + backend.setName("postgres_journal"); + backend.setDatabaseName(DATABASE); + backend.setUsername(APP_ROLE); + backend.setPassword(APP_PASSWORD); + backend.setJdbcUrl(appUrl()); + instance = new QInstance().withInstanceDefaultAuthentication(new QAuthenticationMetaData().withName("anonymous").withType(QAuthenticationType.FULLY_ANONYMOUS)); + instance.addBackend(backend); + for(String name : List.of(UploadJournalTable.NAME, CollaborationTables.ATTACHMENT)) + { + instance.addTable(new QTableMetaData().withName(name).withBackendName(backend.getName()).withPrimaryKeyField("id") + .withField(new QFieldMetaData("id", QFieldType.INTEGER))); + } + context(); + execute("DROP TABLE IF EXISTS collaboration_upload_journal"); + execute("DROP TABLE IF EXISTS collaboration_attachment"); + execute("DROP COLLATION IF EXISTS journal_case_insensitive"); + execute(UploadJournalTable.postgresScript()); + execute("CREATE TABLE collaboration_attachment (id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, file_name VARCHAR(255))"); + journal = new JdbcUploadJournal(Clock.fixed(START, ZoneOffset.UTC)); + } + + + + /******************************************************************************* + ** No connection or thread-local authority survives a fixture. + *******************************************************************************/ + @AfterEach + void close() throws Exception + { + QContext.clearTransaction(); + QContext.clear(); + ConnectionManager.closeConnectionProviders(); + TimeZone.setDefault(TimeZone.getTimeZone("UTC")); + } + + + + /******************************************************************************* + ** Only the native postgres module with both strings is admitted; the generic + ** rdbms backend type with a postgres vendor label is not. + *******************************************************************************/ + @Test + void admitsOnlyTheNativePostgresBackend() + { + assertThat(JdbcUploadJournal.isQualifiedBackend(backend)).isTrue(); + backend.setBackendType("rdbms"); + assertThat(JdbcUploadJournal.isQualifiedBackend(backend)).isFalse(); + assertUnsupported(); + backend.setBackendType("postgres"); + backend.setVendor("aurora-postgresql"); + assertThat(JdbcUploadJournal.isQualifiedBackend(backend)).isFalse(); + backend.setVendor("postgres"); + assertThat(JdbcUploadJournal.isQualifiedBackend(backend)).isTrue(); + } + + + + /******************************************************************************* + ** The shipped script is re-runnable, and its CHECK guards stand on the server: + ** REGISTERED or COMPLETE without an attachment id, and a half intake pair, are refused. + *******************************************************************************/ + @Test + void scriptIsRerunnableAndConstraintsHold() throws Exception + { + execute(UploadJournalTable.postgresScript()); + String insert = "INSERT INTO collaboration_upload_journal (create_date, modify_date, table_name, record_id, created_by_user_id, file_name, content_type," + + " size_bytes, sha256, bucket_name, storage_key, retained, intake_key, intake_fingerprint, locator_hash, manifest_hash, state, storage_state, attachment_id, fence)" + + " VALUES (now(), now(), 't', '1', 'a', 'f', 'c', 1, 's', 'b', 'k', TRUE, %s, %s, 'L%s', 'm', '%s', '%s', %s, 0)"; + assertThatThrownBy(() -> execute(String.format(insert, "NULL", "NULL", "1", "REGISTERED", "VERIFIED", "NULL"))).isInstanceOf(SQLException.class) + .extracting(e -> ((SQLException) e).getSQLState()).isEqualTo("23514"); + assertThatThrownBy(() -> execute(String.format(insert, "'k'", "NULL", "2", "PREPARED", "PENDING", "NULL"))).isInstanceOf(SQLException.class) + .extracting(e -> ((SQLException) e).getSQLState()).isEqualTo("23514"); + assertThatThrownBy(() -> execute(String.format(insert, "NULL", "NULL", "3", "REGISTERED", "PENDING", "1"))).isInstanceOf(SQLException.class) + .extracting(e -> ((SQLException) e).getSQLState()).isEqualTo("23514"); + execute(String.format(insert, "NULL", "NULL", "4", "PREPARED", "PENDING", "NULL")); + assertThatThrownBy(() -> execute(String.format(insert, "NULL", "NULL", "4", "PREPARED", "PENDING", "NULL"))).isInstanceOf(SQLException.class) + .extracting(e -> ((SQLException) e).getSQLState()).isEqualTo("23505"); + assertThat(count("collaboration_upload_journal")).isEqualTo(1); + } + + + + /******************************************************************************* + ** Acknowledged metadata survives a real database process restart, not merely + ** an open pooled connection. The same candidate replays its original row. + *******************************************************************************/ + @Test + void committedCandidateSurvivesDatabaseRestart() throws Exception + { + UploadEntry original = journal.prepare(candidate("restart")); + ConnectionManager.closeConnectionProviders(); + postgres.getDockerClient().restartContainerCmd(postgres.getContainerId()).exec(); + locateServer(); + backend.setJdbcUrl(appUrl()); + long deadline = System.nanoTime() + Duration.ofSeconds(60).toNanos(); + boolean ready = false; + while(System.nanoTime() < deadline) + { + try(Connection connection = root(); Statement statement = connection.createStatement()) + { + statement.execute("SELECT 1"); + ready = true; + break; + } + catch(SQLException transientStartup) + { + Thread.sleep(250); + } + } + assertThat(ready).isTrue(); + assertThat(journal.prepare(candidate("restart")).id()).isEqualTo(original.id()); + assertThat(count("collaboration_upload_journal")).isEqualTo(1); + } + + + + /******************************************************************************* + ** A registrar failure rolls back both its attachment and the owner checkpoint. + *******************************************************************************/ + @Test + void registrationAndCheckpointCommitOrRollbackTogether() throws Exception + { + UploadReservation reservation = verified("rollback"); + assertThatThrownBy(() -> journal.register(reservation, (transaction, candidate, id) -> + { + insertAttachment(transaction); + throw (new QException("Synthetic registrar failure.")); + })).isInstanceOf(QException.class); + assertThat(count("collaboration_attachment")).isZero(); + assertThat(journal.find(reservation.journalId()).orElseThrow().state()).isEqualTo(UploadJournalState.PREPARED); + UploadEntry registered = journal.register(reservation, (transaction, candidate, id) -> insertAttachment(transaction)); + assertThat(registered.state()).isEqualTo(UploadJournalState.REGISTERED); + assertThat(journal.complete(reservation).state()).isEqualTo(UploadJournalState.COMPLETE); + assertThat(journal.complete(reservation).attachmentId()).isEqualTo(registered.attachmentId()); + assertThat(count("collaboration_attachment")).isEqualTo(1); + } + + + + /******************************************************************************* + ** Unique preparation and exclusive leases hold across real simultaneous JDBC + ** connections. A new fence rejects the old holder after expiry. + *******************************************************************************/ + @Test + void concurrentPrepareAndReservationFence() throws Exception + { + try(var executor = Executors.newFixedThreadPool(4)) + { + CountDownLatch start = new CountDownLatch(1); + List> prepares = new ArrayList<>(); + for(int i = 0; i < 4; i++) + { + prepares.add(executor.submit(() -> + { + context(); + start.await(); + return (journal.prepare(candidate("race"))); + })); + } + start.countDown(); + Integer id = prepares.getFirst().get(30, TimeUnit.SECONDS).id(); + for(Future prepared : prepares) + { + assertThat(prepared.get(30, TimeUnit.SECONDS).id()).isEqualTo(id); + } + assertThat(count("collaboration_upload_journal")).isEqualTo(1); + List> reservations = new ArrayList<>(); + CountDownLatch reserveStart = new CountDownLatch(1); + for(int i = 0; i < 4; i++) + { + reservations.add(executor.submit(() -> + { + context(); + reserveStart.await(); + try + { + return (journal.reserve(id, Duration.ofSeconds(5))); + } + catch(UploadJournalException denied) + { + return (denied.getReason()); + } + })); + } + reserveStart.countDown(); + List outcomes = new ArrayList<>(); + for(Future reservation : reservations) + { + outcomes.add(reservation.get(30, TimeUnit.SECONDS)); + } + assertThat(outcomes.stream().filter(UploadReservation.class::isInstance)).hasSize(1); + assertThat(outcomes.stream().filter(Reason.class::isInstance)).containsOnly(Reason.LEASE_HELD); + UploadReservation old = (UploadReservation) outcomes.stream().filter(UploadReservation.class::isInstance).findFirst().orElseThrow(); + JdbcUploadJournal later = new JdbcUploadJournal(Clock.fixed(START.plusSeconds(6), ZoneOffset.UTC)); + UploadReservation next = later.reserve(id, Duration.ofSeconds(5)); + assertThat(next.fence()).isGreaterThan(old.fence()); + assertThatThrownBy(() -> later.recordStorage(old, UploadStorageState.VERIFIED)).isInstanceOf(UploadJournalException.class); + } + } + + + + /******************************************************************************* + ** Wrong metadata mappings, databases and ambient transactions fail before a + ** manifest write. + *******************************************************************************/ + @Test + void rejectsWrongBackendDatabaseMappingAndAmbientTransaction() throws Exception + { + PostgreSQLBackendMetaData other = new PostgreSQLBackendMetaData(); + other.setName("other_backend"); + instance.addBackend(other); + instance.getTable(CollaborationTables.ATTACHMENT).setBackendName("other_backend"); + assertUnsupported(); + instance.getTable(CollaborationTables.ATTACHMENT).setBackendName(backend.getName()); + instance.getTable(CollaborationTables.ATTACHMENT).setBackendDetails(new RDBMSTableBackendDetails().withTableName("other_attachment")); + assertUnsupported(); + instance.getTable(CollaborationTables.ATTACHMENT).setBackendDetails(null); + try(Connection connection = DriverManager.getConnection("jdbc:postgresql://" + currentHost + ":" + currentPort + "/postgres" + OPTIONS, "postgres", ROOT_PASSWORD)) + { + connection.setAutoCommit(false); + assertThatThrownBy(() -> PostgresJournalDatabase.requireQualifiedConnection(connection, backend)).isInstanceOf(UploadJournalException.class); + } + try(QBackendTransaction transaction = new RDBMSBackendModule().openTransaction(new InsertInput(UploadJournalTable.NAME))) + { + QContext.setTransaction(transaction); + assertThatThrownBy(() -> journal.prepare(candidate("denied"))).isInstanceOf(UploadJournalException.class) + .extracting(e -> ((UploadJournalException) e).getReason()).isEqualTo(Reason.AMBIENT_TRANSACTION); + } + QContext.clearTransaction(); + assertThat(count("collaboration_upload_journal")).isZero(); + } + + + + /******************************************************************************* + ** Runtime settings are authoritative: weak commit durability, an unbounded or + ** missing lock timeout, a stronger isolation level and a read-only default + ** each refuse before a write. Role-level settings are restored in setUp. + *******************************************************************************/ + @Test + void rejectsUnsafeSessionSettings() throws Exception + { + journal.prepare(candidate("baseline")); + execute("ALTER ROLE " + APP_ROLE + " SET synchronous_commit='off'"); + assertUnsupported(); + execute("ALTER ROLE " + APP_ROLE + " RESET synchronous_commit"); + execute("ALTER ROLE " + APP_ROLE + " SET lock_timeout='60s'"); + assertUnsupported(); + execute("ALTER ROLE " + APP_ROLE + " RESET lock_timeout"); + assertUnsupported(); + execute("ALTER ROLE " + APP_ROLE + " SET lock_timeout='5s'"); + execute("ALTER ROLE " + APP_ROLE + " SET default_transaction_isolation='repeatable read'"); + assertUnsupported(); + execute("ALTER ROLE " + APP_ROLE + " RESET default_transaction_isolation"); + execute("ALTER ROLE " + APP_ROLE + " SET default_transaction_read_only=on"); + assertUnsupported(); + execute("ALTER ROLE " + APP_ROLE + " RESET default_transaction_read_only"); + ConnectionManager.closeConnectionProviders(); + assertThat(journal.prepare(candidate("restored")).id()).isPositive(); + assertThat(count("collaboration_upload_journal")).isEqualTo(2); + } + + + + /******************************************************************************* + ** Physical drift is refused: an unlogged owner table (lost in crash recovery) + ** and a non-deterministic collation on a journal column. + *******************************************************************************/ + @Test + void rejectsUnloggedTableAndNonDeterministicCollation() throws Exception + { + execute("ALTER TABLE collaboration_attachment SET UNLOGGED"); + assertUnsupported(); + execute("ALTER TABLE collaboration_attachment SET LOGGED"); + execute("ALTER TABLE collaboration_upload_journal SET UNLOGGED"); + assertUnsupported(); + execute("ALTER TABLE collaboration_upload_journal SET LOGGED"); + execute("CREATE COLLATION journal_case_insensitive (provider = icu, locale = 'und-u-ks-level2', deterministic = false)"); + execute("ALTER TABLE collaboration_upload_journal ALTER COLUMN created_by_user_id TYPE VARCHAR(255) COLLATE journal_case_insensitive"); + assertUnsupported(); + execute("ALTER TABLE collaboration_upload_journal ALTER COLUMN created_by_user_id TYPE VARCHAR(255) COLLATE \"default\""); + assertThat(journal.prepare(candidate("restored-drift")).id()).isPositive(); + } + + + + /******************************************************************************* + ** A missing or partial unique index would let identities duplicate, so each is + ** refused until the complete index is back. A changed candidate at the original + ** location stays an immutable conflict. + *******************************************************************************/ + @Test + void rejectsMissingAndPartialUniqueIndexesAndManifestConflict() throws Exception + { + UploadCandidate original = candidate("immutable"); + journal.prepare(original); + UploadCandidate changed = new UploadCandidate(original.tableName(), original.recordId(), original.actorId(), "changed.pdf", original.contentType(), original.description(), + original.sizeBytes(), original.sha256(), original.bucketName(), original.storageKey(), true, null, null); + assertThatThrownBy(() -> journal.prepare(changed)).isInstanceOf(UploadJournalException.class) + .extracting(e -> ((UploadJournalException) e).getReason()).isEqualTo(Reason.IMMUTABLE_CONFLICT); + execute("DROP INDEX uq_collaboration_upload_journal_locator"); + assertUnsupported(); + execute("CREATE UNIQUE INDEX uq_collaboration_upload_journal_locator ON collaboration_upload_journal (locator_hash) WHERE locator_hash <> ''"); + assertUnsupported(); + execute("DROP INDEX uq_collaboration_upload_journal_locator"); + execute("CREATE UNIQUE INDEX uq_collaboration_upload_journal_locator ON collaboration_upload_journal (locator_hash)"); + execute("DROP INDEX uq_collaboration_upload_journal_intake"); + assertUnsupported(); + execute("CREATE UNIQUE INDEX uq_collaboration_upload_journal_intake ON collaboration_upload_journal (intake_key)"); + assertThat(journal.prepare(original).id()).isPositive(); + assertThat(count("collaboration_upload_journal")).isEqualTo(1); + } + + + + /******************************************************************************* + ** Native generic writes cannot bypass journal ownership. + *******************************************************************************/ + @Test + void nativeWritePolicyDeniesDirectInsert() + { + assertThatThrownBy(() -> new UploadJournalSecurity().checkPermissionsThrowing(new InsertInput(UploadJournalTable.NAME), instance.getTable(UploadJournalTable.NAME))) + .isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + } + + + + /******************************************************************************* + ** The actual server lock timeout (SQLSTATE 55P03) maps to BUSY and leaves no + ** granted lease. + *******************************************************************************/ + @Test + void lockTimeoutIsBoundedAndDoesNotGrantLease() throws Exception + { + Integer id = journal.prepare(candidate("locked")).id(); + try(Connection connection = root(); PreparedStatement lock = connection.prepareStatement("SELECT id FROM collaboration_upload_journal WHERE id=? FOR UPDATE")) + { + connection.setAutoCommit(false); + lock.setInt(1, id); + try(ResultSet row = lock.executeQuery()) + { + assertThat(row.next()).isTrue(); + long started = System.nanoTime(); + assertThatThrownBy(() -> journal.reserve(id, Duration.ofSeconds(5))).isInstanceOf(UploadJournalException.class) + .extracting(e -> ((UploadJournalException) e).getReason()).isEqualTo(Reason.BUSY); + assertThat(TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - started)).isBetween(3000L, 20000L); + } + connection.rollback(); + } + assertThat(journal.find(id).orElseThrow().fence()).isZero(); + assertThat(journal.reserve(id, Duration.ofSeconds(5)).fence()).isEqualTo(1L); + } + + + + /******************************************************************************* + ** Lease instants round-trip as TIMESTAMPTZ. A host whose JVM zone is not UTC + ** is refused (pgjdbc would announce that zone to the session) rather than + ** silently shifting leases, and the instant is exact once the zone is UTC. + *******************************************************************************/ + @Test + void leaseInstantsRoundTripAndNonUtcHostIsRefused() throws Exception + { + UploadReservation reservation = verified("timezone"); + assertThat(reservation.leaseUntil()).isEqualTo(START.plusSeconds(60)); + TimeZone.setDefault(TimeZone.getTimeZone("Pacific/Honolulu")); + ConnectionManager.closeConnectionProviders(); + assertUnsupported(); + TimeZone.setDefault(TimeZone.getTimeZone("UTC")); + ConnectionManager.closeConnectionProviders(); + assertThat(journal.find(reservation.journalId()).orElseThrow().leaseUntil()).isEqualTo(START.plusSeconds(60)); + assertThat(journal.renew(reservation, Duration.ofMinutes(2)).leaseUntil()).isEqualTo(START.plusSeconds(120)); + } + + + + /******************************************************************************* + ** Assert a typed refusal on a fresh pooled session, so role-level setting + ** changes are actually observed. + *******************************************************************************/ + private void assertUnsupported() + { + ConnectionManager.closeConnectionProviders(); + assertThatThrownBy(() -> journal.prepare(candidate("denied"))).isInstanceOf(UploadJournalException.class) + .extracting(e -> ((UploadJournalException) e).getReason()).isEqualTo(Reason.UNSUPPORTED_BACKEND); + } + + + + /******************************************************************************* + ** Prepare a leased manifest whose stored bytes have been verified by owner. + *******************************************************************************/ + private UploadReservation verified(String label) throws Exception + { + UploadReservation reservation = journal.reserve(journal.prepare(candidate(label)).id(), Duration.ofMinutes(1)); + journal.recordStorage(reservation, UploadStorageState.VERIFIED); + return (reservation); + } + + + + /******************************************************************************* + ** Insert only through the owner's exact transaction. + *******************************************************************************/ + private static Integer insertAttachment(QBackendTransaction transaction) throws QException + { + try(PreparedStatement statement = ((RDBMSTransaction) transaction).getConnection() + .prepareStatement("INSERT INTO collaboration_attachment(file_name) VALUES ('fixture.pdf')", new String[] { "id" })) + { + statement.executeUpdate(); + try(ResultSet keys = statement.getGeneratedKeys()) + { + keys.next(); + return (keys.getInt(1)); + } + } + catch(SQLException e) + { + throw (new QException("Synthetic attachment insert failed.", e)); + } + } + + + + /******************************************************************************* + ** The application's JDBC URL for the current dynamic port. + *******************************************************************************/ + private static String appUrl() + { + return ("jdbc:postgresql://" + currentHost + ":" + currentPort + "/" + DATABASE + OPTIONS); + } + + + + /******************************************************************************* + ** Find the disposable server: the published loopback port by default, or (opt-in + ** bridge mode) the container's own bridge address and the server port. + *******************************************************************************/ + private static void locateServer() + { + if(BRIDGE_ADDRESS) + { + currentHost = postgres.getContainerInfo().getNetworkSettings().getNetworks().values().iterator().next().getIpAddress(); + currentPort = 5432; + } + else + { + currentHost = "127.0.0.1"; + currentPort = Integer.parseInt(postgres.getCurrentContainerInfo().getNetworkSettings().getPorts().getBindings().get(new ExposedPort(5432))[0].getHostPortSpec()); + } + } + + + + /******************************************************************************* + ** Independent superuser connection for fixture DDL, fault injection and + ** committed-state assertions. + *******************************************************************************/ + private static Connection root() throws SQLException + { + return (DriverManager.getConnection("jdbc:postgresql://" + currentHost + ":" + currentPort + "/" + DATABASE + OPTIONS, "postgres", ROOT_PASSWORD)); + } + + + + /******************************************************************************* + ** Execute fixed trusted fixture SQL, never application-supplied identifiers. + *******************************************************************************/ + private void execute(String sql) throws SQLException + { + try(Connection connection = root(); Statement statement = connection.createStatement()) + { + statement.execute(sql); + } + } + + + + /******************************************************************************* + ** Count a fixed owner table through a new committed connection. + *******************************************************************************/ + private long count(String table) throws SQLException + { + try(Connection connection = root(); Statement statement = connection.createStatement(); ResultSet result = statement.executeQuery("SELECT COUNT(*) FROM " + table)) + { + result.next(); + return (result.getLong(1)); + } + } + + + + /******************************************************************************* + ** Establish the same synthetic actor independently on each test thread. + *******************************************************************************/ + private void context() + { + QContext.init(instance, new QSession().withUser(new QUser().withIdReference("alice"))); + } + + + + /******************************************************************************* + ** Server-shaped immutable candidate; no real customer data or storage is used. + *******************************************************************************/ + private static UploadCandidate candidate(String label) + { + return (new UploadCandidate("document", "1", "alice", label + ".pdf", "application/pdf", "Synthetic fixture", 5L, "a".repeat(64), + "fixture-bucket", "retained/" + label, true, null, null)); + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadCandidateTest.java b/src/test/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadCandidateTest.java new file mode 100644 index 0000000..3b95316 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadCandidateTest.java @@ -0,0 +1,98 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Pure validation and hashing of the immutable candidate; no database. + ** NOT YET RUN: written without a build. + *******************************************************************************/ +public class UploadCandidateTest +{ + private static final String HASH = "a".repeat(64); + + + + /******************************************************************************* + ** A server-shaped candidate with overridable pieces. + *******************************************************************************/ + private static UploadCandidate make(String fileName, String description, Long size, Boolean retained, String intakeKey, String fingerprint) + { + return (new UploadCandidate("document", "7", "alice", fileName, "application/pdf", description, size, HASH, "bucket", "attachments/key", retained, + intakeKey, fingerprint)); + } + + + + /******************************************************************************* + ** Only retained uploads are journaled. + *******************************************************************************/ + @Test + void nonRetainedUploadsAreRejected() + { + assertThatThrownBy(() -> make("a.pdf", null, 1L, false, null, null)).isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> make("a.pdf", null, 1L, null, null, null)).isInstanceOf(IllegalArgumentException.class); + } + + + + /******************************************************************************* + ** Bounds, control characters, hashes and the intake pair are enforced. + *******************************************************************************/ + @Test + void membersAreBounded() + { + assertThatThrownBy(() -> make("x".repeat(256), null, 1L, true, null, null)).isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> make("bad\nname", null, 1L, true, null, null)).isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> make(" ", null, 1L, true, null, null)).isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> make("a.pdf", "d".repeat(4097), 1L, true, null, null)).isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> make("a.pdf", null, -1L, true, null, null)).isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> make("a.pdf", null, UploadCandidate.MAX_SIZE_BYTES + 1, true, null, null)).isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> make("a.pdf", null, 1L, true, HASH, null)).isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> make("a.pdf", null, 1L, true, "short", HASH)).isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> new UploadCandidate("document", "7", "alice", "a.pdf", "application/pdf", null, 1L, "NOT-HEX", "bucket", "key", true, null, null)) + .isInstanceOf(IllegalArgumentException.class); + assertThat(make("a.pdf", "line one\nline two", 0L, true, HASH, HASH).description()).contains("\n"); + } + + + + /******************************************************************************* + ** The locator hash covers bucket and key only; the manifest hash covers all. + ** toString never prints the bucket, key, description or actor. + *******************************************************************************/ + @Test + void hashesAreStableAndRedacted() + { + UploadCandidate one = make("a.pdf", "secret description", 1L, true, null, null); + UploadCandidate sameLocationOtherName = make("b.pdf", "secret description", 1L, true, null, null); + assertThat(one.locatorHash()).hasSize(64).isEqualTo(sameLocationOtherName.locatorHash()).isEqualTo(UploadCandidate.locatorHash("bucket", "attachments/key")); + assertThat(one.manifestHash()).hasSize(64).isNotEqualTo(sameLocationOtherName.manifestHash()); + assertThat(UploadCandidate.locatorHash("ab", "c")).isNotEqualTo(UploadCandidate.locatorHash("a", "bc")); + assertThat(one.toString()).doesNotContain("bucket", "attachments/key", "secret description", "alice").contains(one.locatorHash()); + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalTest.java b/src/test/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalTest.java new file mode 100644 index 0000000..82e08a8 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/attachments/uploadjournal/UploadJournalTest.java @@ -0,0 +1,703 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments.uploadjournal; + + +import java.nio.file.Path; +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.PreparedStatement; +import java.sql.ResultSet; +import java.sql.SQLException; +import java.sql.Statement; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.time.ZoneId; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.UUID; +import java.util.concurrent.Callable; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.Future; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; +import com.kingsrook.qbits.collaboration.CollaborationQBitConfig; +import com.kingsrook.qbits.collaboration.attachments.uploadjournal.UploadJournalException.Reason; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; +import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.core.model.session.QSession; +import com.kingsrook.qqq.backend.core.model.session.QUser; +import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction; +import com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Real file-backed H2 (WRITE_DELAY=0), real separate JDBC connections and the + ** real QQQ RDBMS transaction. Only the attachment table is a synthetic stand-in + ** (id and file_name) registered manually; no existing fixture is changed. + ** + ** These regressions accompany the separate opt-in real MySQL provider suite. + *******************************************************************************/ +public class UploadJournalTest +{ + private static final String ATTACHMENT_DDL = "CREATE TABLE collaboration_attachment (id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, file_name VARCHAR(255))"; + + @TempDir + Path directory; + + private QInstance instance; + private RDBMSBackendMetaData database; + private MutableClock clock; + private JdbcUploadJournal journal; + + + + /******************************************************************************* + ** A fresh file database with the journal and a stand-in attachment table. + *******************************************************************************/ + @BeforeEach + void setUp() throws Exception + { + ConnectionManager.resetConnectionProviders(); + database = fileBackend("journal_" + UUID.randomUUID(), "journal", ";WRITE_DELAY=0"); + instance = instanceFor(database, database); + initContext(instance); + try(Connection connection = DriverManager.getConnection(database.getJdbcUrl(), "sa", ""); Statement statement = connection.createStatement()) + { + for(String sql : UploadJournalTable.h2Script().split(";")) + { + if(!sql.isBlank()) + { + statement.execute(sql); + } + } + statement.execute(ATTACHMENT_DDL); + } + clock = new MutableClock(Instant.parse("2026-01-01T00:00:00Z")); + journal = new JdbcUploadJournal(clock); + } + + + + /******************************************************************************* + ** No context or pooled connection survives a test. + *******************************************************************************/ + @AfterEach + void tearDown() + { + QContext.clearTransaction(); + QContext.clear(); + ConnectionManager.resetConnectionProviders(); + } + + + + /******************************************************************************* + ** Committed PREPARED data survives a crashed lock holder and a full database + ** shutdown and reopen; nothing uncommitted leaks. + *******************************************************************************/ + @Test + void preparedCandidateSurvivesCrashStyleReopen() throws Exception + { + UploadCandidate candidate = candidate("one"); + UploadEntry prepared = journal.prepare(candidate); + assertThat(prepared.state()).isEqualTo(UploadJournalState.PREPARED); + assertThat(prepared.storageState()).isEqualTo(UploadStorageState.PENDING); + + try(Connection crashing = DriverManager.getConnection(database.getJdbcUrl(), "sa", ""); Statement statement = crashing.createStatement()) + { + crashing.setAutoCommit(false); + statement.execute("UPDATE collaboration_upload_journal SET fence = 99 WHERE id = " + prepared.id()); + // closed with the row locked and never committed: the crash + } + ConnectionManager.resetConnectionProviders(); + try(Connection connection = DriverManager.getConnection(database.getJdbcUrl(), "sa", ""); Statement statement = connection.createStatement()) + { + statement.execute("SHUTDOWN"); + } + ConnectionManager.resetConnectionProviders(); + + JdbcUploadJournal reopened = new JdbcUploadJournal(clock); + UploadEntry found = reopened.find(prepared.id()).orElseThrow(); + assertThat(found.candidate()).isEqualTo(candidate); + assertThat(found.state()).isEqualTo(UploadJournalState.PREPARED); + assertThat(found.fence()).isZero(); + assertThat(reopened.listIncomplete(10)).extracting(UploadEntry::id).containsExactly(prepared.id()); + assertThat(reopened.findByLocation("bucket", "attachments/one")).isPresent(); + assertThat(reopened.reserve(prepared.id(), Duration.ofMinutes(1)).fence()).isEqualTo(1L); + } + + + + /******************************************************************************* + ** A failure after the attachment insert rolls back the insert and the + ** checkpoint together; a clean retry then registers exactly once. + *******************************************************************************/ + @Test + void failedMetadataInsertRollsBackAttachmentAndCheckpoint() throws Exception + { + UploadReservation reservation = verified(candidate("rollback")); + AtomicInteger calls = new AtomicInteger(); + UploadJournal.AttachmentRegistrar failing = (transaction, candidate, journalId) -> + { + insertAttachment(transaction, candidate); + calls.incrementAndGet(); + throw (new IllegalStateException("simulated failure after insert")); + }; + + assertThatThrownBy(() -> journal.register(reservation, failing)).isInstanceOf(IllegalStateException.class); + assertThat(count("SELECT COUNT(*) FROM collaboration_attachment")).isZero(); + UploadEntry afterFailure = journal.find(reservation.journalId()).orElseThrow(); + assertThat(afterFailure.state()).isEqualTo(UploadJournalState.PREPARED); + assertThat(afterFailure.attachmentId()).isNull(); + assertThat(afterFailure.storageState()).isEqualTo(UploadStorageState.VERIFIED); + + UploadEntry registered = journal.register(reservation, inserting(calls)); + assertThat(registered.state()).isEqualTo(UploadJournalState.REGISTERED); + assertThat(registered.attachmentId()).isNotNull(); + assertThat(count("SELECT COUNT(*) FROM collaboration_attachment")).isEqualTo(1); + assertThat(count("SELECT COUNT(*) FROM collaboration_attachment WHERE id = " + registered.attachmentId())).isEqualTo(1); + } + + + + /******************************************************************************* + ** Unverified original storage is never registered, and the callback never runs. + *******************************************************************************/ + @Test + void registrationRequiresVerifiedStorage() throws Exception + { + UploadEntry prepared = journal.prepare(candidate("unverified")); + UploadReservation reservation = journal.reserve(prepared.id(), Duration.ofMinutes(1)); + journal.recordStorage(reservation, UploadStorageState.UNKNOWN); + AtomicInteger calls = new AtomicInteger(); + + assertThatThrownBy(() -> journal.register(reservation, inserting(calls))).isInstanceOfSatisfying(UploadJournalException.class, + e -> assertThat(e.getReason()).isEqualTo(Reason.STORAGE_NOT_VERIFIED)); + assertThat(calls).hasValue(0); + assertThat(count("SELECT COUNT(*) FROM collaboration_attachment")).isZero(); + assertThatThrownBy(() -> journal.recordStorage(reservation, UploadStorageState.PENDING)).isInstanceOf(UploadJournalException.class); + journal.recordStorage(reservation, UploadStorageState.VERIFIED); + assertThatThrownBy(() -> journal.recordStorage(reservation, UploadStorageState.UNKNOWN)).isInstanceOfSatisfying(UploadJournalException.class, + e -> assertThat(e.getReason()).isEqualTo(Reason.ILLEGAL_STATE)); + } + + + + /******************************************************************************* + ** An expired or superseded holder cannot mutate; a newer fence takes over. + *******************************************************************************/ + @Test + void staleLeaseIsRejected() throws Exception + { + UploadEntry prepared = journal.prepare(candidate("stale")); + UploadReservation first = journal.reserve(prepared.id(), Duration.ofMinutes(1)); + assertThatThrownBy(() -> journal.reserve(prepared.id(), Duration.ofMinutes(1))).isInstanceOfSatisfying(UploadJournalException.class, + e -> assertThat(e.getReason()).isEqualTo(Reason.LEASE_HELD)); + + clock.advance(Duration.ofMinutes(2)); + assertThatThrownBy(() -> journal.recordStorage(first, UploadStorageState.UNKNOWN)).isInstanceOfSatisfying(UploadJournalException.class, + e -> assertThat(e.getReason()).isEqualTo(Reason.STALE_RESERVATION)); + + UploadReservation second = journal.reserve(prepared.id(), Duration.ofMinutes(1)); + assertThat(second.fence()).isEqualTo(2L); + AtomicInteger calls = new AtomicInteger(); + assertThatThrownBy(() -> journal.recordStorage(first, UploadStorageState.VERIFIED)).isInstanceOfSatisfying(UploadJournalException.class, + e -> assertThat(e.getReason()).isEqualTo(Reason.STALE_RESERVATION)); + assertThatThrownBy(() -> journal.register(first, inserting(calls))).isInstanceOfSatisfying(UploadJournalException.class, + e -> assertThat(e.getReason()).isEqualTo(Reason.STALE_RESERVATION)); + assertThatThrownBy(() -> journal.complete(first)).isInstanceOf(UploadJournalException.class); + assertThat(journal.release(first)).isFalse(); + assertThat(calls).hasValue(0); + assertThat(count("SELECT COUNT(*) FROM collaboration_attachment")).isZero(); + + journal.recordStorage(second, UploadStorageState.VERIFIED); + assertThat(journal.release(second)).isTrue(); + assertThat(journal.find(prepared.id()).orElseThrow().leaseUntil()).isNull(); + assertThat(journal.reserve(prepared.id(), Duration.ofMinutes(1)).fence()).isEqualTo(3L); + } + + + + /******************************************************************************* + ** Renewal extends only the live exact holder and cannot revive expired work. + *******************************************************************************/ + @Test + void renewPreservesFenceAndRefusesExpiredOrReplacedHolders() throws Exception + { + UploadEntry prepared = journal.prepare(candidate("renew")); + UploadReservation first = journal.reserve(prepared.id(), Duration.ofMinutes(1)); + clock.advance(Duration.ofSeconds(45)); + UploadReservation renewed = journal.renew(first, Duration.ofMinutes(1)); + assertThat(renewed.fence()).isEqualTo(first.fence()); + clock.advance(Duration.ofSeconds(30)); + journal.recordStorage(renewed, UploadStorageState.VERIFIED); + clock.advance(Duration.ofMinutes(1)); + assertThatThrownBy(() -> journal.renew(renewed, Duration.ofMinutes(1))).isInstanceOfSatisfying(UploadJournalException.class, + error -> assertThat(error.getReason()).isEqualTo(Reason.STALE_RESERVATION)); + UploadReservation replacement = journal.reserve(prepared.id(), Duration.ofMinutes(1)); + assertThat(replacement.fence()).isGreaterThan(first.fence()); + assertThatThrownBy(() -> journal.renew(first, Duration.ofMinutes(1))).isInstanceOf(UploadJournalException.class); + assertThatThrownBy(() -> journal.renew(replacement, Duration.ZERO)).isInstanceOf(IllegalArgumentException.class); + } + + /******************************************************************************* + ** UTF-8 replacement must not alias distinct malformed manifests into one hash. + *******************************************************************************/ + @Test + void rejectsUnpairedSurrogatesBeforeManifestHashing() + { + UploadCandidate valid = candidate("unicode"); + for(String malformed : List.of("\uD800", "\uDC00", "x\uD800y")) + { + assertThatThrownBy(() -> new UploadCandidate(valid.tableName(), valid.recordId(), valid.actorId(), malformed, + valid.contentType(), valid.description(), valid.sizeBytes(), valid.sha256(), valid.bucketName(), valid.storageKey(), true, null, null)) + .isInstanceOf(IllegalArgumentException.class).hasMessageContaining("malformed Unicode"); + } + } + + /******************************************************************************* + ** Many real connections race for one reservation; exactly one wins. + *******************************************************************************/ + @Test + void concurrentClaimsGrantExactlyOneReservation() throws Exception + { + UploadEntry prepared = journal.prepare(candidate("race")); + List outcomes = race(8, () -> journal.reserve(prepared.id(), Duration.ofMinutes(1))); + + List winners = outcomes.stream().filter(UploadReservation.class::isInstance).toList(); + assertThat(winners).hasSize(1); + for(Object outcome : outcomes) + { + if(!(outcome instanceof UploadReservation)) + { + assertThat(outcome).isInstanceOf(UploadJournalException.class); + assertThat(((UploadJournalException) outcome).getReason()).isIn(Reason.LEASE_HELD, Reason.BUSY); + } + } + assertThat(journal.find(prepared.id()).orElseThrow().fence()).isEqualTo(1L); + } + + + + /******************************************************************************* + ** Concurrent identical prepares converge on one row; differing manifests, + ** altered rows and reused intake keys are immutable conflicts. + *******************************************************************************/ + @Test + void immutableManifestAndUniqueRaceRecovery() throws Exception + { + UploadCandidate candidate = candidate("immutable"); + List outcomes = race(6, () -> journal.prepare(candidate)); + Set ids = new HashSet<>(); + for(Object outcome : outcomes) + { + assertThat(outcome).isInstanceOf(UploadEntry.class); + ids.add(((UploadEntry) outcome).id()); + } + assertThat(ids).hasSize(1); + assertThat(count("SELECT COUNT(*) FROM collaboration_upload_journal")).isEqualTo(1); + assertThat(journal.prepare(candidate).id()).isEqualTo(ids.iterator().next()); + + UploadCandidate changed = new UploadCandidate(candidate.tableName(), candidate.recordId(), "mallory", candidate.fileName(), candidate.contentType(), + candidate.description(), candidate.sizeBytes(), candidate.sha256(), candidate.bucketName(), candidate.storageKey(), true, null, null); + assertThatThrownBy(() -> journal.prepare(changed)).isInstanceOfSatisfying(UploadJournalException.class, + e -> assertThat(e.getReason()).isEqualTo(Reason.IMMUTABLE_CONFLICT)); + + String intake = "b".repeat(64); + journal.prepare(withIntake(candidate("intake-a"), intake)); + assertThatThrownBy(() -> journal.prepare(withIntake(candidate("intake-b"), intake))).isInstanceOfSatisfying(UploadJournalException.class, + e -> assertThat(e.getReason()).isEqualTo(Reason.IMMUTABLE_CONFLICT)); + + try(Connection connection = DriverManager.getConnection(database.getJdbcUrl(), "sa", ""); Statement statement = connection.createStatement()) + { + statement.execute("UPDATE collaboration_upload_journal SET file_name = 'evil.pdf' WHERE id = " + ids.iterator().next()); + } + assertThatThrownBy(() -> journal.find(ids.iterator().next())).isInstanceOfSatisfying(UploadJournalException.class, + e -> assertThat(e.getReason()).isEqualTo(Reason.IMMUTABLE_CONFLICT)); + } + + + + /******************************************************************************* + ** COMPLETE only follows registration; a registered row recovers without a + ** second attachment once the lifecycle callback is retried. + *******************************************************************************/ + @Test + void lifecycleCompletionCheckpointAndIdempotentRecovery() throws Exception + { + UploadReservation reservation = verified(candidate("lifecycle")); + AtomicInteger calls = new AtomicInteger(); + assertThatThrownBy(() -> journal.complete(reservation)).isInstanceOfSatisfying(UploadJournalException.class, + e -> assertThat(e.getReason()).isEqualTo(Reason.ILLEGAL_STATE)); + + UploadEntry registered = journal.register(reservation, inserting(calls)); + assertThat(registered.state()).isEqualTo(UploadJournalState.REGISTERED); + assertThat(journal.register(reservation, inserting(calls)).attachmentId()).isEqualTo(registered.attachmentId()); + assertThat(calls).hasValue(1); + + // The post-commit lifecycle callback failed: free the lease and recover later. + assertThat(journal.release(reservation)).isTrue(); + UploadReservation recovery = journal.reserve(reservation.journalId(), Duration.ofMinutes(1)); + assertThat(journal.register(recovery, inserting(calls)).attachmentId()).isEqualTo(registered.attachmentId()); + assertThat(calls).hasValue(1); + assertThat(count("SELECT COUNT(*) FROM collaboration_attachment")).isEqualTo(1); + + UploadEntry complete = journal.complete(recovery); + assertThat(complete.state()).isEqualTo(UploadJournalState.COMPLETE); + assertThat(complete.leaseUntil()).isNull(); + assertThat(count("SELECT COUNT(*) FROM collaboration_upload_journal WHERE lease_token IS NULL AND state = 'COMPLETE'")).isEqualTo(1); + assertThat(journal.complete(recovery).state()).isEqualTo(UploadJournalState.COMPLETE); + assertThat(journal.listIncomplete(10)).isEmpty(); + assertThatThrownBy(() -> journal.reserve(reservation.journalId(), Duration.ofMinutes(1))).isInstanceOfSatisfying(UploadJournalException.class, + e -> assertThat(e.getReason()).isEqualTo(Reason.ILLEGAL_STATE)); + } + + + + /******************************************************************************* + ** Memory, delayed-write, mismatched and ambient-transaction setups are refused. + *******************************************************************************/ + @Test + void unsupportedBackendsAndAmbientTransactionsAreRejected() throws Exception + { + RDBMSBackendMetaData memory = fileBackend("memory_" + UUID.randomUUID(), "unused", ";WRITE_DELAY=0"); + memory.setJdbcUrl("jdbc:h2:mem:" + UUID.randomUUID() + ";WRITE_DELAY=0"); + assertThat(JdbcUploadJournal.isQualifiedBackend(memory)).isFalse(); + assertThat(JdbcUploadJournal.isQualifiedBackend(database)).isTrue(); + initContext(instanceFor(memory, memory)); + assertThatThrownBy(() -> journal.find(1)).isInstanceOfSatisfying(UploadJournalException.class, + e -> assertThat(e.getReason()).isEqualTo(Reason.UNSUPPORTED_BACKEND)); + + RDBMSBackendMetaData delayed = fileBackend("delayed_" + UUID.randomUUID(), "delayed", ""); + assertThat(JdbcUploadJournal.isQualifiedBackend(delayed)).isTrue(); + initContext(instanceFor(delayed, delayed)); + assertThatThrownBy(() -> journal.find(1)).isInstanceOfSatisfying(UploadJournalException.class, + e -> assertThat(e.getReason()).isEqualTo(Reason.UNSUPPORTED_BACKEND)); + + RDBMSBackendMetaData other = fileBackend("other_" + UUID.randomUUID(), "other", ";WRITE_DELAY=0"); + initContext(instanceFor(database, other)); + assertThatThrownBy(() -> journal.find(1)).isInstanceOfSatisfying(UploadJournalException.class, + e -> assertThat(e.getReason()).isEqualTo(Reason.UNSUPPORTED_BACKEND)); + + initContext(instance); + QContext.setTransaction(new QBackendTransaction()); + assertThatThrownBy(() -> journal.prepare(candidate("ambient"))).isInstanceOfSatisfying(UploadJournalException.class, + e -> assertThat(e.getReason()).isEqualTo(Reason.AMBIENT_TRANSACTION)); + QContext.clearTransaction(); + assertThat(count("SELECT COUNT(*) FROM collaboration_upload_journal")).isZero(); + } + + + + /******************************************************************************* + ** Metadata exposes provenance and status but no raw locator or lease secret, + ** and denies every direct write. + *******************************************************************************/ + @Test + void metadataHidesLocatorAndDeniesDirectWrites() throws Exception + { + QTableMetaData table = UploadJournalTable.metaData(new CollaborationQBitConfig().withBackendName("backend")); + assertThat(table.getName()).isEqualTo(UploadJournalTable.NAME); + assertThat(table.getFields()).containsKeys("tableName", "recordId", "createdByUserId", "sha256", "state", "storageState", "attachmentId", "locatorHash"); + assertThat(table.getFields()).doesNotContainKeys("bucketName", "storageKey", "leaseToken", "manifestHash"); + assertThat(table.getFields().values()).allMatch(field -> Boolean.FALSE.equals(field.getIsEditable())); + + UploadJournalSecurity security = new UploadJournalSecurity(); + assertThatThrownBy(() -> security.checkPermissionsThrowing(new InsertInput(UploadJournalTable.NAME), table)).isInstanceOf(QPermissionDeniedException.class); + assertThatThrownBy(() -> security.checkPermissionsThrowing(new UpdateInput(UploadJournalTable.NAME), table)).isInstanceOf(QPermissionDeniedException.class); + assertThatThrownBy(() -> security.checkPermissionsThrowing(new DeleteInput(UploadJournalTable.NAME), table)).isInstanceOf(QPermissionDeniedException.class); + } + + + + /******************************************************************************* + ** Prepare a candidate and bring it to a live reservation with VERIFIED storage. + *******************************************************************************/ + private UploadReservation verified(UploadCandidate candidate) throws QException + { + UploadEntry prepared = journal.prepare(candidate); + UploadReservation reservation = journal.reserve(prepared.id(), Duration.ofMinutes(5)); + journal.recordStorage(reservation, UploadStorageState.UNKNOWN); + journal.recordStorage(reservation, UploadStorageState.VERIFIED); + return (reservation); + } + + + + /******************************************************************************* + ** Run the same call on N threads, each with its own QQQ context and connection. + ** Each outcome is the result or the thrown exception. + *******************************************************************************/ + private List race(int threads, Callable call) throws Exception + { + ExecutorService pool = Executors.newFixedThreadPool(threads); + try + { + CountDownLatch start = new CountDownLatch(1); + List> futures = new ArrayList<>(); + for(int i = 0; i < threads; i++) + { + futures.add(pool.submit(() -> + { + initContext(instance); + start.await(); + try + { + return (call.call()); + } + catch(UploadJournalException e) + { + return (e); + } + })); + } + start.countDown(); + List outcomes = new ArrayList<>(); + for(Future future : futures) + { + outcomes.add(future.get(60, TimeUnit.SECONDS)); + } + return (outcomes); + } + finally + { + pool.shutdownNow(); + } + } + + + + /******************************************************************************* + ** A registrar that inserts a stand-in attachment row in the journal's transaction. + *******************************************************************************/ + private UploadJournal.AttachmentRegistrar inserting(AtomicInteger calls) + { + return ((transaction, candidate, journalId) -> + { + calls.incrementAndGet(); + return (insertAttachment(transaction, candidate)); + }); + } + + + + /******************************************************************************* + ** Insert through the supplied transaction's own connection and return the id. + *******************************************************************************/ + private static Integer insertAttachment(QBackendTransaction transaction, UploadCandidate candidate) throws QException + { + try(PreparedStatement statement = ((RDBMSTransaction) transaction).getConnection() + .prepareStatement("INSERT INTO collaboration_attachment (file_name) VALUES (?)", Statement.RETURN_GENERATED_KEYS)) + { + statement.setString(1, candidate.fileName()); + statement.executeUpdate(); + try(ResultSet keys = statement.getGeneratedKeys()) + { + keys.next(); + return (keys.getInt(1)); + } + } + catch(SQLException e) + { + throw (new QException("Could not insert the stand-in attachment.", e)); + } + } + + + + /******************************************************************************* + ** Count rows using a separate real connection, so only committed data counts. + *******************************************************************************/ + private long count(String sql) throws SQLException + { + try(Connection connection = DriverManager.getConnection(database.getJdbcUrl(), "sa", ""); Statement statement = connection.createStatement(); + ResultSet result = statement.executeQuery(sql)) + { + result.next(); + return (result.getLong(1)); + } + } + + + + /******************************************************************************* + ** A server-shaped retained candidate whose location is unique to the label. + *******************************************************************************/ + private static UploadCandidate candidate(String label) + { + return (new UploadCandidate("document", "1", "alice", "scan-" + label + ".pdf", "application/pdf", "A scanned form", 5L, "a".repeat(64), + "bucket", "attachments/" + label, true, null, null)); + } + + + + /******************************************************************************* + ** The same candidate carrying an intake key and fingerprint. + *******************************************************************************/ + private static UploadCandidate withIntake(UploadCandidate base, String intakeKey) + { + return (new UploadCandidate(base.tableName(), base.recordId(), base.actorId(), base.fileName(), base.contentType(), base.description(), + base.sizeBytes(), base.sha256(), base.bucketName(), base.storageKey(), true, intakeKey, "c".repeat(64))); + } + + + + /******************************************************************************* + ** An H2 file backend; settings are appended to the URL verbatim. + *******************************************************************************/ + private RDBMSBackendMetaData fileBackend(String name, String file, String settings) + { + RDBMSBackendMetaData backend = new RDBMSBackendMetaData().withName(name).withVendor("h2").withHostName("mem").withDatabaseName(file).withUsername("sa"); + backend.setJdbcUrl("jdbc:h2:file:" + directory.resolve(file).toAbsolutePath().toString().replace('\\', '/') + ";DATABASE_TO_UPPER=FALSE;LOCK_TIMEOUT=5000" + settings); + return (backend); + } + + + + /******************************************************************************* + ** A synthetic instance with the journal and attachment tables registered on + ** the given backends. + *******************************************************************************/ + private QInstance instanceFor(RDBMSBackendMetaData journalBackend, RDBMSBackendMetaData attachmentBackend) + { + QInstance built = new QInstance().withInstanceDefaultAuthentication(new QAuthenticationMetaData().withName("anonymous").withType(QAuthenticationType.FULLY_ANONYMOUS)); + built.addBackend(journalBackend); + if(!attachmentBackend.getName().equals(journalBackend.getName())) + { + built.addBackend(attachmentBackend); + } + built.addTable(stand(UploadJournalTable.NAME, journalBackend)); + built.addTable(stand(CollaborationTables.ATTACHMENT, attachmentBackend)); + return (built); + } + + + + /******************************************************************************* + ** Minimal table registration; the journal itself uses only fixed SQL. + *******************************************************************************/ + private static QTableMetaData stand(String name, RDBMSBackendMetaData backend) + { + return (new QTableMetaData().withName(name).withBackendName(backend.getName()).withPrimaryKeyField("id") + .withField(new QFieldMetaData("id", QFieldType.INTEGER))); + } + + + + /******************************************************************************* + ** Bind a QQQ context with a signed-in user on the current thread. + *******************************************************************************/ + private static void initContext(QInstance context) + { + QContext.init(context, new QSession().withUser(new QUser().withIdReference("alice"))); + } + + + + /******************************************************************************* + ** A clock a test can move forward, safe across threads. + *******************************************************************************/ + private static final class MutableClock extends Clock + { + private final AtomicReference now; + + + + /******************************************************************************* + ** Start at the given instant. + *******************************************************************************/ + MutableClock(Instant start) + { + this.now = new AtomicReference<>(start); + } + + + + /******************************************************************************* + ** Move time forward. + *******************************************************************************/ + void advance(Duration duration) + { + now.updateAndGet(current -> current.plus(duration)); + } + + + + /******************************************************************************* + ** UTC. + *******************************************************************************/ + @Override + public ZoneId getZone() + { + return (ZoneOffset.UTC); + } + + + + /******************************************************************************* + ** Fixed zone only. + *******************************************************************************/ + @Override + public Clock withZone(ZoneId zone) + { + return (this); + } + + + + /******************************************************************************* + ** Current test time. + *******************************************************************************/ + @Override + public Instant instant() + { + return (now.get()); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/devserver/CapacityPresenceDevServer.java b/src/test/java/com/kingsrook/qbits/collaboration/devserver/CapacityPresenceDevServer.java new file mode 100644 index 0000000..1d2752d --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/devserver/CapacityPresenceDevServer.java @@ -0,0 +1,174 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.devserver; + +import com.kingsrook.qbits.collaboration.CollaborationQBitConfig; +import com.kingsrook.qbits.collaboration.presence.PresenceRegistry; +import com.kingsrook.qbits.collaboration.presence.PresenceScopeProvider; +import com.kingsrook.qbits.collaboration.presence.RedisPresenceConfig; +import com.kingsrook.qbits.redis.RedisConnectionConfig; +import com.kingsrook.qbits.redis.RedisQBitConfig; +import com.kingsrook.qbits.redis.RedisQBitProducer; +import com.kingsrook.qbits.redis.RedisRealtimeRuntimeService; +import com.kingsrook.qbits.redis.RedisRuntimeService; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.realtime.RealtimeBus; + +/******************************************************************************* + ** Loopback-only browser acceptance node. Run separate JVMs against one disposable + ** Redis endpoint. Record data is independently seeded, not a shared database. + *******************************************************************************/ +public class CapacityPresenceDevServer +{ + /*************************************************************************** + ** Arguments are HTTP port and disposable local Redis port. Never deploy this + ** fixture: it uses the demo's mock authentication and plaintext loopback Redis. + ***************************************************************************/ + public static void main(String[] args) throws Exception + { + if(args.length != 2) + { + throw (new IllegalArgumentException("Expected HTTP port and local Redis port")); + } + int port = Integer.parseInt(args[0]); + int redisPort = Integer.parseInt(args[1]); + var instance = CollaborationDevServer.buildInstance(port); + CollaborationDevServer.seed(instance); + instance.withInstanceDefaultAuthentication(new com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData() + .withName("mock").withType(com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType.MOCK) + .withCustomizer(new QCodeReference(AcceptanceSessions.class))); + PresenceRegistry.close(instance); + RealtimeBus.close(instance); + new RedisQBitProducer().withQBitConfig(new RedisQBitConfig().withConnection("presence", + new RedisConnectionConfig().withHost(System.getProperty("collab.capacity.redisHost", "127.0.0.1")).withPort(redisPort).withTls(false).withMaximumConnections(8).withMaximumWaitMillis(500)) + .withRealtime("presence", "acceptance", "collaboration")).produce(instance); + instance.getQBits().values().stream().map(bit -> bit.getConfig()) + .filter(CollaborationQBitConfig.class::isInstance).map(CollaborationQBitConfig.class::cast) + .findFirst().orElseThrow().withRedisPresence(new RedisPresenceConfig().withConnectionName("presence") + .withDeployment("acceptance").withApplication("collaboration") + .withScopeProvider(new QCodeReference(FixtureTenant.class))); + var redis = new RedisRuntimeService(); + var transport = new RedisRealtimeRuntimeService(); + new com.kingsrook.qqq.backend.core.health.HealthMetaDataProvider().defineAll(instance, "memory", null); + var health = new com.kingsrook.qqq.backend.core.health.HealthRunner(instance, "acceptance-" + port, java.time.Clock.systemUTC()); + var server = CollaborationDevServer.server(instance, port, false); + var sampler = java.util.concurrent.Executors.newSingleThreadScheduledExecutor(); + try + { + redis.start(instance); + transport.start(instance); + health.start(); + server.start(); + sampler.scheduleAtFixedRate(() -> + { + var pool = RedisRuntimeService.connection(instance, "presence").metrics(); + var heap = java.lang.management.ManagementFactory.getMemoryMXBean().getHeapMemoryUsage(); + var values = new java.util.LinkedHashMap(); + values.put("uptimeMillis", java.lang.management.ManagementFactory.getRuntimeMXBean().getUptime()); + values.put("cpuNanos", ProcessHandle.current().info().totalCpuDuration().map(java.time.Duration::toNanos).orElse(-1L)); + values.put("heapUsedBytes", heap.getUsed()); + values.put("heapCommittedBytes", heap.getCommitted()); + values.put("residentBytes", residentBytes()); + values.put("poolActive", pool.active()); + values.put("poolIdle", pool.idle()); + values.put("poolWaiting", pool.waiting()); + values.put("poolMaximum", pool.maximum()); + System.out.println("CAPACITY_METRIC " + com.kingsrook.qqq.backend.core.utils.JsonUtils.toJson(values)); + }, 0, 100, java.util.concurrent.TimeUnit.MILLISECONDS); + System.out.println("CAPACITY_PRESENCE_READY " + port); + Thread.currentThread().join(); + } + finally + { + sampler.shutdownNow(); + server.stop(); + health.stop(); + transport.stop(); + PresenceRegistry.close(instance); + RealtimeBus.close(instance); + redis.stop(); + redis.afterApplicationStop(); + } + } + + /*************************************************************************** + ** Read Linux process resident memory; unsupported platforms report -1. + ***************************************************************************/ + private static long residentBytes() + { + try + { + for(String line : java.nio.file.Files.readAllLines(java.nio.file.Path.of("/proc/self/status"))) + { + if(line.startsWith("VmRSS:")) + { + return (Long.parseLong(line.trim().split("\\s+")[1]) * 1024L); + } + } + } + catch(java.io.IOException | NumberFormatException e) + { + return (-1L); + } + return (-1L); + } + + /*************************************************************************** + ** Tenant comes from the configured mock identity, never a request tenant parameter. + ***************************************************************************/ + public static class FixtureTenant implements PresenceScopeProvider + { + /************************************************************************ + ** Identical record IDs are isolated by the authenticated fixture identity. + ************************************************************************/ + @Override + public String tenantScope() + { + String identity = com.kingsrook.qqq.backend.core.context.QContext.getQSession().getUser().getIdReference(); + if(!identity.matches("load-t[0-7]-u[0-9]{1,2}")) + { + throw (new IllegalStateException("Capacity presence requires a configured load identity")); + } + return (identity.substring(0, identity.indexOf("-u"))); + } + } + + /*************************************************************************** + ** Grant Redis dashboard access only to the fixture's existing administrator. + ***************************************************************************/ + public static class AcceptanceSessions extends CollaborationDevServer.DemoSessions + { + /************************************************************************ + ** Keep normal demo identity/grants and add the service-view permission. + ************************************************************************/ + @Override + public void customizeSession(com.kingsrook.qqq.backend.core.model.metadata.QInstance instance, + com.kingsrook.qqq.backend.core.model.session.QSession target, java.util.Map context) + { + super.customizeSession(instance, target, context); + if("admin".equals(target.getUser().getIdReference())) + { + target.withPermissions("redisView.hasAccess", "health.read"); + } + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/devserver/CollaborationDevServer.java b/src/test/java/com/kingsrook/qbits/collaboration/devserver/CollaborationDevServer.java index 9c552be..16bed25 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/devserver/CollaborationDevServer.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/devserver/CollaborationDevServer.java @@ -123,11 +123,35 @@ import io.javalin.config.JavalinConfig; public class CollaborationDevServer extends AbstractQQQApplication { private final QInstance instance; - static final List USERS = List.of("alice", "bob", "admin"); + static final List USERS = fixtureUsers(); static final String USER_COOKIE = "collab_dev_user"; private static final ThreadLocal REQUEST_USER = new ThreadLocal<>(); + /******************************************************************************* + ** Opt-in bounded identities for the disposable capacity fixture only. + *******************************************************************************/ + private static List fixtureUsers() + { + int tenants = Integer.getInteger("collab.capacity.tenants", 0); + int users = Integer.getInteger("collab.capacity.users", 0); + if(tenants < 0 || tenants > 8 || users < 0 || users > 100) + { + throw (new IllegalArgumentException("Capacity fixture permits at most 8 tenants and 100 users each")); + } + var identities = new java.util.ArrayList<>(List.of("alice", "bob", "admin")); + for(int tenant = 0; tenant < tenants; tenant++) + { + for(int user = 0; user < users; user++) + { + identities.add("load-t" + tenant + "-u" + user); + } + } + return (List.copyOf(identities)); + } + + + /******************************************************************************* ** Keep the already seeded instance when Javalin initializes the application. @@ -186,7 +210,7 @@ public class CollaborationDevServer extends AbstractQQQApplication .withServeLegacyUnversionedMiddlewareAPI(false).withStrictUiApiRoutes(true) .withJavalinConfigCustomizer(config -> { - config.jetty.host = "127.0.0.1"; + config.jetty.host = System.getProperty("collab.dev.bindAddress", "127.0.0.1"); userSelection(config); MemoryStorage.routes(config); })); @@ -310,6 +334,9 @@ public class CollaborationDevServer extends AbstractQQQApplication { return (new QTableMetaData().withName(name).withLabel(label).withIcon(new QIcon(icon)).withBackendName("memory") .withPrimaryKeyField("id").withField(new QFieldMetaData("id", QFieldType.INTEGER)) + .withField(new QFieldMetaData("modifyDate", QFieldType.DATE_TIME) + .withBehavior(com.kingsrook.qqq.backend.core.model.metadata.fields.DynamicDefaultValueBehavior.MODIFY_DATE)) + .withSection(new QFieldSection("audit", new QIcon("history"), Tier.T3, List.of("modifyDate")).withLabel("Updated")) .withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.READ_WRITE_PERMISSIONS)) .withAuditRules(new QAuditRules().withAuditLevel(AuditLevel.FIELD))); } @@ -329,7 +356,7 @@ public class CollaborationDevServer extends AbstractQQQApplication { case "alice" -> "Alice Morgan"; case "bob" -> "Bob Chen"; - default -> "Demo Administrator"; + default -> user.equals("admin") ? "Demo Administrator" : user; }; QSession session = new QSession().withUser(new QUser().withIdReference(user).withFullName(name)) .withPermissions("person.read", "person.comment") diff --git a/src/test/java/com/kingsrook/qbits/collaboration/devserver/SharedPresenceDevServer.java b/src/test/java/com/kingsrook/qbits/collaboration/devserver/SharedPresenceDevServer.java new file mode 100644 index 0000000..8530b32 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/devserver/SharedPresenceDevServer.java @@ -0,0 +1,130 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.devserver; + +import com.kingsrook.qbits.collaboration.CollaborationQBitConfig; +import com.kingsrook.qbits.collaboration.presence.PresenceRegistry; +import com.kingsrook.qbits.collaboration.presence.PresenceScopeProvider; +import com.kingsrook.qbits.collaboration.presence.RedisPresenceConfig; +import com.kingsrook.qbits.redis.RedisConnectionConfig; +import com.kingsrook.qbits.redis.RedisQBitConfig; +import com.kingsrook.qbits.redis.RedisQBitProducer; +import com.kingsrook.qbits.redis.RedisRealtimeRuntimeService; +import com.kingsrook.qbits.redis.RedisRuntimeService; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.realtime.RealtimeBus; + +/******************************************************************************* + ** Loopback-only browser acceptance node. Run separate JVMs against one disposable + ** Redis endpoint. Record data is independently seeded, not a shared database. + *******************************************************************************/ +public class SharedPresenceDevServer +{ + /*************************************************************************** + ** Arguments are HTTP port and disposable local Redis port. Never deploy this + ** fixture: it uses the demo's mock authentication and plaintext loopback Redis. + ***************************************************************************/ + public static void main(String[] args) throws Exception + { + if(args.length != 2) + { + throw (new IllegalArgumentException("Expected HTTP port and local Redis port")); + } + int port = Integer.parseInt(args[0]); + int redisPort = Integer.parseInt(args[1]); + var instance = CollaborationDevServer.buildInstance(port); + CollaborationDevServer.seed(instance); + instance.withInstanceDefaultAuthentication(new com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData() + .withName("mock").withType(com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType.MOCK) + .withCustomizer(new QCodeReference(AcceptanceSessions.class))); + PresenceRegistry.close(instance); + RealtimeBus.close(instance); + new RedisQBitProducer().withQBitConfig(new RedisQBitConfig().withConnection("presence", + new RedisConnectionConfig().withHost("127.0.0.1").withPort(redisPort).withTls(false)) + .withRealtime("presence", "acceptance", "collaboration")).produce(instance); + instance.getQBits().values().stream().map(bit -> bit.getConfig()) + .filter(CollaborationQBitConfig.class::isInstance).map(CollaborationQBitConfig.class::cast) + .findFirst().orElseThrow().withRedisPresence(new RedisPresenceConfig().withConnectionName("presence") + .withDeployment("acceptance").withApplication("collaboration") + .withRecoveryGeneration(System.getProperty("collab.dev.recoveryGeneration", "initial")) + .withScopeProvider(new QCodeReference(SingleTenant.class))); + var redis = new RedisRuntimeService(); + var transport = new RedisRealtimeRuntimeService(); + new com.kingsrook.qqq.backend.core.health.HealthMetaDataProvider().defineAll(instance, "memory", null); + var health = new com.kingsrook.qqq.backend.core.health.HealthRunner(instance, "acceptance-" + port, java.time.Clock.systemUTC()); + var server = CollaborationDevServer.server(instance, port, Boolean.getBoolean("collab.dev.frontend")); + try + { + redis.start(instance); + transport.start(instance); + health.start(); + server.start(); + System.out.println("SHARED_PRESENCE_READY " + port); + Thread.currentThread().join(); + } + finally + { + server.stop(); + health.stop(); + transport.stop(); + PresenceRegistry.close(instance); + RealtimeBus.close(instance); + redis.stop(); + redis.afterApplicationStop(); + } + } + + /*************************************************************************** + ** Explicit single-tenant fixture; no scope comes from browser request input. + ***************************************************************************/ + public static class SingleTenant implements PresenceScopeProvider + { + /************************************************************************ + ** The two acceptance nodes share this configured tenant. + ************************************************************************/ + @Override + public String tenantScope() + { + return ("acceptance"); + } + } + + /*************************************************************************** + ** Grant Redis dashboard access only to the fixture's existing administrator. + ***************************************************************************/ + public static class AcceptanceSessions extends CollaborationDevServer.DemoSessions + { + /************************************************************************ + ** Keep normal demo identity/grants and add the service-view permission. + ************************************************************************/ + @Override + public void customizeSession(com.kingsrook.qqq.backend.core.model.metadata.QInstance instance, + com.kingsrook.qqq.backend.core.model.session.QSession target, java.util.Map context) + { + super.customizeSession(instance, target, context); + if("admin".equals(target.getUser().getIdReference())) + { + target.withPermissions("redisView.hasAccess", "health.read"); + } + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/email/EmailCollaborationNotifierTest.java b/src/test/java/com/kingsrook/qbits/collaboration/email/EmailCollaborationNotifierTest.java index 545433b..2b4e3e3 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/email/EmailCollaborationNotifierTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/email/EmailCollaborationNotifierTest.java @@ -200,7 +200,7 @@ class EmailCollaborationNotifierTest extends PersonalServiceFixture /******************************************************************************* ** Each recipient gets their own message; the share and task kinds use their - ** own templates and key kinds; follow digests send no email. + ** own templates and key kinds, including follow digests. *******************************************************************************/ @Test void oneMessagePerRecipientAndKinds() throws Exception @@ -211,7 +211,7 @@ class EmailCollaborationNotifierTest extends PersonalServiceFixture Notifications.send(notification("bob", CollaborationNotification.Category.TASK, "9:remindAt")); Notifications.send(notification("bob", CollaborationNotification.Category.FOLLOW, "1")); assertThat(messages()).extracting(m -> m.getValueString("idempotencyKey")) - .containsExactlyInAnyOrder("collaboration:share:3:bob", "collaboration:share:3:eve", "collaboration:task:9:remindAt:bob"); + .containsExactlyInAnyOrder("collaboration:share:3:bob", "collaboration:share:3:eve", "collaboration:task:9:remindAt:bob", "collaboration:follow:1:bob"); assertThat(messages()).allSatisfy(m -> assertThat(m.getValueString("toAddresses")).doesNotContain(",")); assertThat(EmailApi.findMessagesByIdempotencyKeyPrefix("collaboration:", 10)).extracting(EmailApiMessage::templateName) .contains("collaboration_share", "collaboration_task_reminder"); @@ -228,7 +228,11 @@ class EmailCollaborationNotifierTest extends PersonalServiceFixture withEmail(); config.withRecipientEmailResolver(user -> "eve".equals(user) ? "other@example.org" : null); Notifications.send(notification("eve", CollaborationNotification.Category.MENTION, "1")); - Notifications.send(notification("bob", CollaborationNotification.Category.MENTION, "2")); + boolean[] receipt = {false}; + Notifications.send(notification("bob", CollaborationNotification.Category.MENTION, "2"), () -> receipt[0] = true); + assertThat(receipt[0]).isFalse(); + Notifications.send(notification("eve", CollaborationNotification.Category.MENTION, "1"), () -> receipt[0] = true); + assertThat(receipt[0]).isTrue(); assertThat(messages()).singleElement().satisfies(m -> assertThat(m.getValueString("toAddresses")).contains("other@example.org")); } diff --git a/src/test/resources/log4j2-test.xml b/src/test/resources/log4j2-test.xml new file mode 100644 index 0000000..14d941d --- /dev/null +++ b/src/test/resources/log4j2-test.xml @@ -0,0 +1,11 @@ + + + + + + + + + + +