diff --git a/README.md b/README.md index 1cdb8d3..69af815 100644 --- a/README.md +++ b/README.md @@ -408,7 +408,7 @@ size, pending suggestions, and the five most-used approved tags. `collaborationTags` is a virtual STRING field labeled Tags with possible-value source `collaboration_tag`. `TagCriteriaExpander` supports IN, NOT_IN, EQUALS, NOT_EQUALS, IS_BLANK, IS_NOT_BLANK and CONTAINS_ALL using core `QSubQuery` membership filters. Ordinary quick search matches labels; `#name` matches normalized names. Numeric criteria are tag IDs; string criteria are names. The old ID materializer is removed. Core owns SQL execution or bounded cross-backend materialization; its bound applies to inner rows, not final parent matches. -The runtime provider adds this field and its search entry where TAGS is enabled. It is queryable but not selectable: the overlay has no virtual-field post-query hook and existing host customizers must be preserved. +The runtime provider adds this field and its search entry where TAGS is enabled. With the coordinated core value-provider API, it is selectable in queries, record reads and exports. Labels are sorted, deduplicated and joined with a comma and space; the picker still uses tag IDs. The provider runs after the host post-query customizer without replacing it. This is an unreleased source change: see [virtual-field upgrade status](docs/VIRTUAL-FIELD-UPGRADE.md) before building or adopting this branch. ## UI API @@ -683,8 +683,9 @@ metadata. Memory uses core `JsonPathStorage`. PostgreSQL uses it after a cached `SELECT qqq_custom_fields FROM LIMIT 0` probe succeeds. Missing columns and unsupported backends get read-only EAV virtual filters over `collaboration_field_value`, with -typed `QSubQuery` criteria. These fields are not query-selectable; EAV value writes and projection -are outside this version. `statusMessage` records the fallback reason. After adding a missing +typed `QSubQuery` criteria. The coordinated value-provider API also makes existing EAV values +selectable in queries, record reads and exports, with their declared types and possible-value +labels. EAV writes remain unsupported. See [virtual-field upgrade status](docs/VIRTUAL-FIELD-UPGRADE.md). `statusMessage` records the fallback reason. After adding a missing column, call `CustomFieldStorage.invalidate(instance, tableName)` and refresh the overlay. ### PostgreSQL setup and indexes diff --git a/docs/ATTACHMENT-RECOVERY.md b/docs/ATTACHMENT-RECOVERY.md new file mode 100644 index 0000000..be5459c --- /dev/null +++ b/docs/ATTACHMENT-RECOVERY.md @@ -0,0 +1,37 @@ +# Attachment cleanup recovery + +Deletion writes a standard `collaboration_attachment_cleanup` row and removes attachment metadata in the same backend transaction. Only after commit does it delete the stored object, then acknowledge the cleanup row. PostgreSQL rejection of metadata deletion leaves both the original row and bytes intact. A provider failure or lost callback leaves the cleanup locator available after restart. The service returns a failure when immediate cleanup fails; the logical attachment deletion may already have committed. + +The cleanup table is visible in the Collaboration app to administrators. Direct table mutations are denied, including to administrators, so callers cannot forge deletion targets. Recovery runs through `collaborationRetryAttachmentCleanup`, guarded for system sessions or `collaboration.admin`. With the host's configured scheduler, it retries a bounded batch of 100 every minute. Kof22 already supplies a scheduler name. Without a scheduler, operators must invoke the recovery process; it is not safe to assume background retries exist. + +Apply the generated PostgreSQL schema before enabling this version; it creates the additional cleanup table and due-time index. Attachment ID is optional because rejected uploads may never have a committed ID. The upgrade preserves existing cleanup rows and removes the earlier attachment-ID uniqueness constraint. Do not remove that table during application rollback while cleanup remains pending. Existing releases do not know how to drain it. + +Verification covers real PostgreSQL metadata rejection, outer upload/delete rollback, commit, provider failure followed by a new recovery service, loss of post-commit callbacks, failed queue acknowledgement and repeat replay. Deterministic storage faults also cover a provider that writes bytes before reporting upload failure and two concurrent cleanup workers. A real MinIO fixture, accessed through the published object-storage QBit, verifies deletion replay in plain and versioned buckets while preserving another object's exact bytes. This is not evidence for every object-storage provider. The separate abrupt-JVM test described below uses fixture file storage. + +Failed upload rollback compensation now records its locator independently of the rolled-back transaction before attempting storage cleanup. Failed worker attempts persist a next-attempt time at least 60 seconds later; this lets healthy requests beyond a failing batch proceed. Tests verify later recovery once the failed requests become due. + +Upload I/O and runtime failures attempt durable compensation and preserve the original exception, adding cleanup failure as a suppressed exception. Deferred provider completion is covered by the settlement protocol below. + +Full Maven verification including operator settlement passes 206 tests, zero failures, errors or skips, plus packaging, help coverage, architecture guardrails and JaCoCo checks. PostgreSQL tests cover the retained receipt, repeated requests, active owners, existing references, rejected receipt writes, provider failure and the real form/Review/Result flow with permission revocation. Coordinated host adoption and staging rollback still require separate verification. + +On backends supporting transactional FOR UPDATE SKIP LOCKED (including the PostgreSQL host), upload now commits a cleanup locator before calling storage and holds its row lock for the upload transaction. Metadata insertion and removal of that locator share the same transaction. Recovery workers skip active uploads and hold the same lock while deleting abandoned bytes and acknowledging cleanup. An existing caller transaction retains ownership until it ends; otherwise the service starts its own transaction after spooling the input. Backends without this locking capability retain callback-based compensation and are not covered by this ownership guarantee. + +Focused PostgreSQL tests verify the locator exists before provider I/O, cleanup skips an active upload, and recovery succeeds after an interrupted upload whose rollback callbacks were deliberately suppressed. That test simulates callback loss. A separate child JVM executes the actual upload service against the disposable PostgreSQL database, persists fixture bytes, then calls Runtime.halt before returning from the provider. The parent verifies exit code 73, absent metadata, retained intent and bytes, and successful idempotent recovery using the original locator. This proves abrupt JVM termination recovery, not power-loss durability or late remote-provider completion. Another test rejects intent insertion in PostgreSQL and verifies zero provider writes or deletes; a completed upload in an open outer transaction remains protected from cleanup until commit. Concurrent workers verify that only one worker enters provider deletion for a locator. + +Issue #1289 remains open for coordinated host adoption and application rollback acceptance. Row locking does not make external object storage and the database one atomic transaction. + +## Unreleased late-upload settlement change + +A PostgreSQL regression reproduced a PUT finishing after timeout rollback had deleted the still-empty location and acknowledged its cleanup row. Upload intents (no attachment ID) now use AttachmentStorage.cleanupUpload, which removes current bytes and reports whether an earlier PUT can still complete. The generic adapter cannot prove settlement and returns false. Recovery retains the locator for later sweeps, with the existing 60-second backoff and 100-row batch. Ordinary committed deletion and successful upload acknowledgement remain unchanged. + +Adapters may report settlement only from authoritative completion/cancellation information, never merely object absence. A deterministic deferred-write fixture verifies the late-completion ordering; this does not establish remote S3 settlement. Generic uncertain-upload records remain pending until the provider proves settlement or an administrator performs the explicit operation below. There is no age-based expiry. + +## Resolve a settled upload + +In the Attachment Cleanup table, select upload intents and run **Resolve settled uploads**. An identified user with `collaboration.admin` must supply provider completion or cancellation evidence covering every selected locator, then confirm the standard Review screen. A support case or request receipt establishing completion is suitable; object absence, an elapsed timeout, or an application process stopping does not establish that an external provider has stopped writing. If settlement cannot be established, leave the intent pending. + +Apply rechecks permission, locks each row, rejects active ownership and existing attachment references, and deletes current bytes before saving completion. Provider or database failures leave the intent unresolved for retry. Committed attachment deletions use the ordinary cleanup retry operation instead. + +The service-managed row retains `resolvedAt`, `resolvedBy` and `settlementEvidence`, together with the original locator. Repeated settlement does not overwrite that receipt; scheduled retries exclude resolved rows. Direct edits and deletes remain denied even to administrators. These fields provide the durable operational receipt without relying on the framework's best-effort audit writer. Store a reference to provider evidence, never credentials or private object contents. + +Apply the additive settlement columns and pending-row index before this code runs. Kof22's corresponding migration is `2026-10-26-collaboration-upload-settlement.sql`. Retain receipts and pending intents during rollback. Do not start an earlier cleanup worker that ignores `resolvedAt`: it could consume retained receipts. Roll back to the pre-cleanup application with cleanup processing disabled, or use a version that understands settlement receipts. diff --git a/docs/CANDIDATE-RELEASE.md b/docs/CANDIDATE-RELEASE.md index 0eee83b..dba8ed7 100644 --- a/docs/CANDIDATE-RELEASE.md +++ b/docs/CANDIDATE-RELEASE.md @@ -1,6 +1,41 @@ # Coordinated collaboration snapshot -The main-line candidate is published. Use these versions together: +## Combined integration follow-up under qualification + +PR #51 combines current attachment settlement, My Work, presence recovery, +contributed-field and capacity follow-ups. Its source inventory and gate history +are in [INTEGRATION-20261029.md](INTEGRATION-20261029.md). The candidate selects +verified published Redis `0.1.0-redis-integration-20261029-551f1d5-SNAPSHOT` and +the existing core `4.1.0-virtual-field-values-1196-cc3981bb3-SNAPSHOT`. +The combined collaboration artifact is not yet published. Historical snapshot +and acceptance evidence below is not proof of this candidate's deployment. + +## Selectable-field follow-up snapshot + +The following focused upgrade is published for QQQ #1196: + +| Component | Version | +| --- | --- | +| QQQ BOM and modules | `4.1.0-virtual-field-values-1196-cc3981bb3-SNAPSHOT` | +| qbit-collaboration | `0.1.0-field-values-1196-ecff347-SNAPSHOT` | +| qbit-redis (unchanged prerequisite) | `0.1.0-redis-7fe5492-SNAPSHOT` | + +Import this QQQ BOM in the consuming host. Collaboration publisher 168 passed +from `ecff347efe2bb84949b8452a3383e3fdeb4bcede`; its POM and JAR resolve to +timestamp `20261026.133203-1`. Full collaboration verification against the +published core passed 198 tests with no failures, errors or skips, plus packaging +and coverage gates. An independent empty-cache Java 21 consumer downloaded the +pair and compiled both collaboration value providers against the core API. +See [the upgrade evidence](VIRTUAL-FIELD-UPGRADE.md) for checksums and scope. + +This is not a combined release of the separate attachment, My Work and presence +recovery follow-ups, nor a Kof22 deployment. The earlier candidate's browser +acceptance below must not be treated as browser qualification of this new pair. +A compatible packaged frontend and host staging qualification remain required. + +## Earlier main-line candidate + +The earlier main-line candidate is published. Its coordinated versions are: | Component | Version | | --- | --- | diff --git a/docs/INTEGRATION-20261029.md b/docs/INTEGRATION-20261029.md new file mode 100644 index 0000000..183fb05 --- /dev/null +++ b/docs/INTEGRATION-20261029.md @@ -0,0 +1,51 @@ +# Coordinated collaboration integration candidate + +This branch combines the current heads of the attachment, My Work, presence +recovery, virtual-field and capacity PRs. It is a qualification candidate, not +a published release or a deployed host. + +| PR | Included source | +| --- | --- | +| #45 attachment settlement | `dc65a826ce362416d55d91136fa39225f4a7cf94` | +| #46 bounded My Work | `cdcf7ee36509c7a1fc6f22fa864209eb80a6ff34` | +| #47 presence recovery | `776ebe3fcb6492881bc9d7ec3f6ca6adfa320200` | +| #48 contributed field values | `1da538cf43ba1ed80e3f8fd599729b5691563cbf` | +| #49 capacity qualification | `56a8b760d9c1a1862cb11e80fad07cf5cdc20b3a` | + +All five heads are ancestors of integration commit +`44a89d77b1ad3a39d26c512ada78af0884872013`. The stacked My Work and capacity +branches did not contain the latest heads of their base PRs, so those base +heads were merged explicitly as well. No production source was rewritten during +integration. The core prerequisite remains +`4.1.0-virtual-field-values-1196-cc3981bb3-SNAPSHOT`. The candidate now selects +published Redis `0.1.0-redis-integration-20261029-551f1d5-SNAPSHOT` after native +publisher job42 succeeded and independent timestamped POM/JAR downloads matched +their declared version and repository checksums. The timestamp is +`20261029.165102-1`; JAR SHA256 is +`385f69c5d84362dbab40a3044261fb9820e25126785d7f22901121302e6ce076`. + +The caller and manifest both select released `kingsrook/qqq-orb@0.8`. The +canonical validator extracted from released 0.8.32 and CircleCI schema validation +pass; no unpublished orb changes are used. + +Local `./mvnw.cmd verify -Dqqq.mywork.load=true` ran 238 tests with no skips, but +reported three failures and four errors. Failures include Redis command read +timeouts/rejoin assertions and PostgreSQL connection I/O during the attachment +fixture. Docker Desktop's published-port path has independently exhibited +300–900 ms replies after idle, compared with 0–58 ms on a direct Docker network. +This observation is not proof of every failure's cause. No timeout, assertion, +security boundary, or quality gate was weakened. Native hosted Linux results +are required before publication; the local run is **not** a passing gate. + +Native CircleCI job192 subsequently passed the full verification at integration +head `02085baff928c374e3b0d77367fbcc8182f81f93`; CodeQL also passed. Those checks +used the earlier Redis pin. The new published Redis pin above requires its own +current-head native verification before this collaboration artifact is published. +Use an explicit short feature branch (`feature/collab-int-20261029`) and dry-run +the released version calculator: generated help-bundle versions must remain at +most 50 characters. The longer `collaboration-integration-20261029` name would +produce 57 characters and is not a valid publication route. + +The successful earlier combined-source checks remain historical evidence only. +Host TLS/ACL, actual identity/proxy/provider acceptance, maintenance/restore +enforcement, artifact publication, and deployment are separate gates. diff --git a/docs/MY-WORK-QUERY-BOUNDS.md b/docs/MY-WORK-QUERY-BOUNDS.md new file mode 100644 index 0000000..0c27f99 --- /dev/null +++ b/docs/MY-WORK-QUERY-BOUNDS.md @@ -0,0 +1,92 @@ +# My Work summary query bounds + +The summary previously materialized every matching task, assignment and mention. +A regression fixture measured 601 parent lookups for 601 tasks on two parents. + +The summary now scans each category in primary-key pages of at most 200 rows. +It retains at most ten display tasks and sorts them using the existing due-date +order (earliest first, null last, then ID). Counts still include every visible +matching row, rather than silently truncating at a display limit. Assignment and +mention filters remain scoped to the signed-in user. + +Each page caches parent visibility only for that page, using the existing caller +permission and record-lock checks. Missing or revoked parents contribute neither +rows nor counts; unexpected authorization/backend failures propagate. No decision +is shared across requests. The starting maximum ID bounds each category scan so +concurrent inserts cannot extend it indefinitely. Like the earlier summary, this +is not a transactionally consistent snapshot across the three categories. + +This bounds row retention, not total work independently of data size. For N +matching rows, each category needs one high-watermark query plus approximately +ceil(N/200) page queries, and one parent lookup per distinct parent per page. +Many unique parents still require linear authorization work. Per-record heavy +field size is also outside this row-count bound. Database and browser latency, +N-node presence, saturation and production capacity remain part of #1291/#1302. + +Regression evidence: the 601-task case passes a budget of at most eight parent +queries, with no task query above 200 rows; correct top-ten ordering and revoked +access on a subsequent request are checked. Another case crosses assignment and +mention page boundaries and excludes hidden parents, other users and undelivered +mentions. These tests use the framework's real memory backend instrumentation. + +`MyWorkPostgresTest` repeats the 601-row case using PostgreSQL 17 and the generated +production schema. Independent `pg_stat_statements` observations verify exactly +five task queries and between one and eight parent queries. It also verifies the +same top ten, visible count and next-request permission revocation against SQL. +The fixture passes; these query counts are not sustained throughput or +deployment-capacity measurements. + +## Reproducible local workload + +Run `mvn -Dtest=MyWorkPostgresTest -Dqqq.mywork.load=true test` with Java21 and +Docker. The fixture owns a disposable PostgreSQL17 container, uses the generated +schema and real authorization checks, and writes `target/mywork-load.csv`. +It never connects to an application database. The optional workload runs after +the normal query-budget regression so ordinary CI keeps the smaller test. + +Each scenario has 2,000 tasks: one uses ten shared parents, the other 2,000 distinct +parents. Every tenth parent is inaccessible. Two warmups precede ten measured +sequential summaries. Every sample verifies 1,800 visible tasks and the exact top +ten. SQL calls and server execution time are collected independently using +pg_stat_statements after warmup. Latency includes service execution and database +round trips, excluding fixture setup and assertions. + +CSV percentiles use nearest rank. With only ten samples, p95 and p99 both equal +the maximum observed sample; they are descriptive, not stable tail estimates. +SQL figures are totals for all ten samples. This is a single-JVM local workload, +not HTTP/browser latency, sustained concurrency, pool saturation, multi-node +presence or a deployment-capacity claim. Do not derive production alert thresholds +from these numbers. No wall-clock performance assertion is added to CI. + +The fixture explicitly uses the framework C3P0 connection provider, matching the +host's provider, with initial/minimum1, maximum3 and a two-second checkout timeout. +It verifies the pool stays within that cap, has no checkout timeouts and returns +all borrowed connections after each scenario. The original fixture used the +unpooled default; the expanded workload failed with a Windows socket-allocation +error after repeatedly opening connections. This was a fixture mismatch, not +proof of a Kof22 outage. The same workload completed with pooling. + +Observed local run (Java21, PostgreSQL17 in Docker on Windows, ten sequential +samples after two warmups): shared-parent p50=91.192ms, max=112.105ms; distinct-parent +p50=960.565ms, max=1100.260ms. SQL calls per summary were113 and2013 respectively. +This confirms linear distinct-parent authorization work remains despite bounded +row retention. The existing exact-count semantics still require examining every +matching task. No runtime optimization or production-capacity claim follows from +this fixture measurement alone. + +## Compatibility with the later attachment changes + +On October 26, an isolated source export combined attachment commit +`dc65a826ce362416d55d91136fa39225f4a7cf94` with the four My Work files from +`2ac067bea4af199486f90e2dc5212baf8910b8fb`. Both branches share base +`15cb8cbf72aa9a5e033280cf5a38384d1295fc7d`; their changed files do not overlap. +This was a source-overlay verification, not a branch merge or publication. + +Java 21 `./mvnw.cmd verify -Dqqq.mywork.load=true` passed all 209 tests with zero +failures, errors or skips, including the 29 attachment transaction tests. +Packaging and JaCoCo gates passed. The optional PostgreSQL workload retained the +same visible-count, top-ten, query-budget and pool assertions. Shared-parent +p50/max were 76.301/96.044 ms; distinct-parent p50/max were 939.896/1035.402 ms. +SQL counts remained 113 and 2013 per summary. These local measurements preserve +the limitations above; they do not qualify an integrated published artifact or +Kof22 deployment. The eventual integrated branch still needs its own CI gates. diff --git a/docs/PRESENCE-RECOVERY.md b/docs/PRESENCE-RECOVERY.md new file mode 100644 index 0000000..43035ae --- /dev/null +++ b/docs/PRESENCE-RECOVERY.md @@ -0,0 +1,179 @@ +# Presence recovery candidate — QQQ #1300 + +This is an unreleased implementation candidate in collaboration draft PR47. +The existing store/service suites and CI pass; host adoption and enforced +maintenance remain incomplete. It is not part of the published snapshots. + +## Contract under qualification + +Each deployment/application/tenant/table has a persistent recovery guard in the +same Redis hash slot as its bounded room projections. Each Lua operation reads +that guard atomically with its room work. The guard contains a generation, a +readiness deadline and the latest issued expiry. A missing guard, or complete +loss of table projections before that expiry, starts a fresh generation and a +recovery interval of one lease lifetime. Existing live browsers can reserve and +renew replacement leases during that interval. Snapshots report recovery rather +than a successful empty room until the interval finishes. + +Lease entries carry the guard generation. Recovery removes invalid-generation +projections from touched rooms so they cannot consume all capacity and prevent +re-registration. Existing authorization remains in PresenceService; the guard +does not grant record, tenant or field access. The shared adapter maps recovery +to a sanitized service-unavailable response. Record-version conflict checks are +independent of this advisory state. + +An untouched new namespace is conservatively unknown too. Its first snapshot +therefore starts a recovery interval. Normal idle expiry retains the guard so +later empty snapshots can be known-empty. The guard has no TTL; tenant-retirement +operations and long-term guard storage must be included in operational capacity +qualification. + +## Controlled restore + +The host configuration has a bounded `recoveryGeneration`, default `initial`, +included in both the storage namespace and realtime audience. It belongs in +deployment configuration outside the Redis backup. All nodes must use the same +value. A restored guard can otherwise restore old authority; Redis process IDs +cannot detect arbitrary same-process data restoration. + +The required maintenance procedure is: stop admission and drain all old web +nodes; restore Redis; choose a never-reused generation outside Redis; start only +nodes with that generation; verify recovery and authenticated re-registration; +then reopen traffic. Do not roll back the generation when rolling back an app. +The host tooling that enforces and verifies this sequence is **not implemented** +yet. The branch does not claim transparent arbitrary restore or split-brain +recovery. Native Sentinel/Cluster routing remains the separate Redis issue #1301. + +The storage prefix is v2 so old executables cannot treat generation-bound entries +as old-format authority. This requires a coordinated deployment, not a mixed +version rolling deployment. Existing published v1 snapshots are unchanged. + +## Evidence and remaining work + +- Before the patch, a two-client disposable Redis test failed because total loss + returned an empty snapshot. +- Seven focused tests now pass: full loss, loss of volatile projections while the + guard survives, restored guard/lease bytes followed by a host generation + change, bounded generation validation, and rejection of inconsistent lease + lifetimes across nodes, guard loss across previously unvisited records, and + shared cold readiness followed by normal idle expiry. The latter verifies + that peer reads do not extend the deadline, the guard remains persistent, + expired projections remain known-empty and expired leases cannot renew. + These use independent clients in + one JVM, not a deployed HA cluster. +- Full wrapper verification passed on October 26: 184 tests, no failures, + errors or skips; packaging and coverage gates passed. This includes the + original seven store tests and two-node six-viewer/two-editor scenario. + The admin dashboard reads every table, so the fixture initializes every + dashboard table for both tenants through the real backend and waits for + recovery without writing Redis guard state. +- The service fixture socket read timeout is one second; its previous 200 ms + limit caused a Redis EVALSHA read timeout. Production timeouts are unchanged. +- The full run includes a real HTTP recovery test: real Redis + FLUSHDB produces 503 during recovery, the former lease returns 409 with + PRESENCE_LEASE_EXPIRED, hidden records and unauthorized viewers remain + inaccessible, and a new lease rejoins through the production recovery window. + This uses a loopback HTTP server with fixture authentication, not a browser + or a deployed host. The focused test and the combined full suite both pass. +- The store suite checks loss of either state or expiry data in both record + and table projections. Server-issued lease renewal must fail unavailable + without changing the surviving projection. It also covers complete loss + of either projection pair and stale departure ordering. +- Still required: deployed-host qualification and enforced host maintenance/reset. Local + per-operation cost measurements at 1,000 and 10,000 leases are recorded in + `acceptance/RECOVERY-COST.md`; sustained production capacity is unqualified. + The implementation uses no INFO call or lifecycle polling cache. +- This is an unreleased draft candidate. Publication and production deployment + have not occurred; the remaining acceptance work above is still required. + +## Independent-process acceptance + +`docs/acceptance/presence_recovery_processes.py` passed on October 26 using +two independent `SharedPresenceDevServer` JVMs and one disposable Redis container. +Each user renews through the opposite node. After FLUSHDB, both nodes return 503 +for presence and 409 for the former leases; fresh leases restore both viewers on +both nodes. The harness terminates only its owned processes and container. +This is fixture authentication with independently seeded memory records, not a +deployed HA application or a browser acceptance result. + +Reproduce from the repository root after `./mvnw test-compile`: + +```text +./mvnw dependency:build-classpath -Dmdep.includeScope=test -Dmdep.outputFile=target/recovery-process-classpath.txt +python docs/acceptance/presence_recovery_processes.py --java /path/to/java --repo /path/to/repository +``` + +Requires Python 3, Java 21 and Docker. JVM output and argument files stay under +`target/recovery-processes-*` for diagnosis. The harness uses the production +45-second recovery window, with bounded startup and recovery deadlines. + +## Browser recovery acceptance + +The same harness accepts `--frontend-repo /path/to/qqq-frontend-next`. It copies +that checkout's existing `out` export into the owned test output directory and +serves it through the real middleware on both JVMs. Playwright is loaded from +the frontend checkout; no API mocks are installed. Both browser users first +see the other viewer, then Redis FLUSHDB triggers the unavailable message and +removes stale people. Both viewer badges return without reloading either page. + +The final fixed-candidate run passed all five profiles on October 26 against +commit `4b0d6a32d6bd8ecba023857d3661d9f3f5980975`: Chromium, Firefox, WebKit, +Pixel 7 emulation and iPad emulation. Source/compiled Lua hashes were checked +before the run and after each profile. Traces and screenshots remain under +each `target/recovery-processes-*` directory. The earlier Chromium screenshot +in `target/recovery-processes-229f305e83` was also visually inspected. + +Select profiles using `--browser-profile chromium|firefox|webkit|mobile|tablet`. +These tests use the local export and do not establish provenance of a newly +published frontend artifact, physical-device behavior or deployed Kof22 +acceptance. Fixture frontend serving is opt-in via `collab.dev.frontend`; +its default remains false. + +## Controlled restore acceptance + +`acceptance/presence_restore_processes.py` exercises a disposable Redis RDB +restore with two independent JVMs. It saves the populated dataset, verifies +both old JVMs have terminated and their listeners have closed, clears the +dataset, reloads the saved image and starts both nodes with a shared fresh +generation supplied outside Redis. It checks recovery responses, rejects old +leases on both nodes, and requires two fresh viewers to rejoin across nodes. +The final run passed on October 26; logs are retained in +`target/restore-processes-aec5288289`. Both leases are explicitly renewed before +saving, and rejection must complete within 40 seconds of that renewal, before +their normal 45-second expiry. Old leases are rejected again after fresh +viewers have rejoined. + +The fixture enables Redis DEBUG RELOAD only on its owned loopback container. +Run after test compilation and classpath preparation: + +```text +python docs/acceptance/presence_restore_processes.py --java /path/to/java --repo /path/to/repository +``` + +This is a protocol acceptance fixture, not host maintenance automation. Actual +host admission closure, replacement prevention and generation configuration +still require implementation and staging validation. + +To qualify a packaged backend, add `--backend-jar /path/to/qbit-collaboration.jar`. +This excludes production `target/classes` from the JVM classpath, rejects a +duplicate collaboration dependency JAR, and records the artifact SHA-256 in +`target/restore-processes-*/result.json`. The receipt includes the elapsed time +for rejection of old leases (required to remain below their normal expiry), +the restore/rejoin checks and fixture cleanup. Cleanup failure fails the run. +The test repository still supplies compiled fixture classes and its dependency +classpath; this is not a standalone production launcher. + +## Packaged browser qualification + +The process harness accepts `--backend-jar` and `--frontend-jar` for artifact +verification. With a backend JAR, production `target/classes` is omitted from the +runtime classpath; test classes provide only the disposable server fixture. The +frontend JAR is served directly instead of copying a local `out` export. +`--frontend-repo` remains required for Playwright tooling. Dependency classpaths +must not contain a second collaboration or frontend artifact. + +Each run writes `result.json` with the selected artifact SHA-256 hashes, browser +profile, result and cleanup outcome. A successful test with failed fixture cleanup +exits unsuccessfully. This records local artifact qualification, not publication +or deployment provenance. Supply `--browser-profile` to qualify each required +browser independently. diff --git a/docs/VIRTUAL-FIELD-UPGRADE.md b/docs/VIRTUAL-FIELD-UPGRADE.md new file mode 100644 index 0000000..fda061e --- /dev/null +++ b/docs/VIRTUAL-FIELD-UPGRADE.md @@ -0,0 +1,76 @@ +# Selectable collaboration fields: published feature snapshot + +Tracks [QQQ #1196](https://github.com/QRun-IO/qqq/issues/1196). + +This source branch requires the new core virtual-field value-provider API. Its +default BOM pin is now the published +`4.1.0-virtual-field-values-1196-cc3981bb3-SNAPSHOT`. Core publisher job 9716 +passed from commit `cc3981bb3c95e8f2dd7ef2a02e2233db7d0e2a79`; an independent +consumer with an initially empty Maven repository downloaded its BOM and core +and compiled the provider interface on October 26, 2026. Both artifacts resolve +to timestamp `20261026.130631-1` from Sonatype's snapshot repository. + +Published artifact SHA-256 receipts: + +| Artifact | SHA-256 | +| --- | --- | +| BOM POM | `d68cf837d7e07513c4e70977829f99e6236cb3e89f8af85fc75239d5b52564a5` | +| Core POM | `d89d491aaa9312051ec462e1cf165b948ee3e88c9aec2721ff25a00e956c431b` | +| Core JAR | `ae6dbeba8d0e4bc55f2714303ee3482dbf6a0cd34c03663c6009d8fe2c1da54d` | + +The collaboration follow-up is published as +`0.1.0-field-values-1196-ecff347-SNAPSHOT`, timestamp +`20261026.133203-1`. Publisher job 168 passed from commit +`ecff347efe2bb84949b8452a3383e3fdeb4bcede`, using signed tag +`publish-values1196-20261026-ecff347`. + +| Collaboration artifact | SHA-256 | +| --- | --- | +| POM | `cc12f59cbdfd4bc53ba545a9f2b6d7c14a316034b107ef5541ddc48df45fe6af` | +| JAR | `284fa370504dd82cee8c87d2b8f5c0cecba8c4b15b32a4f9d3d62a0a062e549d` | + +On October 26, an independent Java 21 consumer downloaded this pair into an +initially empty Maven repository and compiled both `TagValueProvider` and +`CustomFieldValueProvider` as `VirtualFieldValueProviderInterface` instances. +Maven recorded Sonatype `central-portal-snapshots` as the artifact source. The +downloaded collaboration POM pins the core version above; the independently +downloaded core JAR matches the receipt above. This checks published dependency +resolution and API compatibility; runtime behavior is covered by the 198-test +verification below, not by this compilation probe. + +This snapshot does not include the separate attachment, My Work or presence +recovery follow-up PRs. It has not been deployed to Kof22. + +Earlier local verification used core artifacts installed as +`4.1.0-values1196-SNAPSHOT`, with +`./mvnw.cmd -Dqqq.snapshot.version=4.1.0-values1196-SNAPSHOT verify`. +That version exists only in the local Maven repository; it is not an adoption pin. + +## Behavior + +- `collaborationTags` returns sorted, distinct approved labels from global or + matching-table vocabulary. Merged tags and deleted assignments are excluded. + The picker and filters retain their tag IDs; exports contain one Tags column. +- EAV custom fields return typed existing values and ordinary possible-value + display labels. Missing values are null internally and omitted from HTTP value + maps. This change does not enable EAV writes. +- Providers process batches and use the originating transaction. Parent record + authorization and field visibility remain enforced by core. Existing host + post-query customizers remain installed. +- SQL DISTINCT on provider values is explicitly unsupported. Backend primary-key + identity is required; joined heavy primary keys must be fetched explicitly. + +## Release sequence + +1. Publish and verify the coordinated core artifact containing the provider API (done). +2. Replace this branch's core BOM pin with that published version (done). +3. Run full collaboration verification against the published prerequisite + (done October 26: 198 tests, no failures/errors/skips; packaging and coverage + passed, including HTTP, PostgreSQL transaction and Redis tests). +4. Publish the collaboration snapshot and verify an independent consumer using + published artifacts (done; exact pair recorded in CANDIDATE-RELEASE.md). +5. Update Kof22 Website-Backend adoption work and qualify it in the selected + isolated staging environment before requesting production deployment. + +The published pair contains this implementation. Kof22 adoption, a compatible +packaged frontend and staging qualification remain separate release steps. diff --git a/docs/acceptance/RECOVERY-COST.md b/docs/acceptance/RECOVERY-COST.md new file mode 100644 index 0000000..74121a0 --- /dev/null +++ b/docs/acceptance/RECOVERY-COST.md @@ -0,0 +1,56 @@ +# Recovery cost diagnostic — October 26, 2026 + +One Java 21 process, one Redis connection, disposable loopback Redis 7.4-alpine, +1,000 issued leases, ten viewers per record, no TLS or HTTP. The probe keeps +all leases alive through the real 45-second recovery window and verifies that +the final table snapshot contains all 1,000 viewers. Each phase samples 200 +renewals with increasing sequence numbers. This is not a capacity or SLA claim. + +| Renewal latency (ms) | During recovery | Ready | +|---|---:|---:| +| p50 | 3.6779 | 0.5134 | +| p95 | 4.1956 | 0.7226 | +| p99 | 4.5447 | 0.9122 | +| Maximum | 6.8271 | 1.4886 | + +Population took 4,640.799 ms. The run passed all correctness checks. This first +implementation scanned touched projections on every request during recovery. + +After changing table cleanup to occur atomically only when creating its recovery +generation (individual record cleanup remains lazy), the same diagnostic passed: + +| Renewal latency (ms) | Optimized recovery | Ready | +|---|---:|---:| +| p50 | 0.6375 | 0.5029 | +| p95 | 0.8150 | 0.5762 | +| p99 | 0.9406 | 0.7500 | +| Maximum | 4.2280 | 1.2251 | + +Optimized population took 1,634.486 ms. All 1,000 leases survived. The focused +recovery/store suite also passed 13 tests, including guard loss with previously +unvisited record rooms. Full optimized verification passed 184 tests with no failures/errors/skips and coverage gates passed. These are +individual local runs and need further qualification before claiming supported +production capacity. They do not isolate every source +of latency, compare against an earlier release, or measure sustained concurrent +load, saturation, TLS, HTTP or browser propagation. + +At 10,000 entries the optimized diagnostic also passed, retaining all leases. +Population took 10,687.2105 ms; the same 200-sample phases measured: + +| Renewal latency (ms) | Optimized recovery, 10,000 entries | Ready | +|---|---:|---:| +| p50 | 0.4978 | 0.5341 | +| p95 | 0.5914 | 0.7108 | +| p99 | 0.7255 | 1.2964 | +| Maximum | 0.7544 | 1.8799 | + +Do not infer that larger tables are faster from these separate local runs. +JIT warmup, scheduling and concurrent machine activity were not controlled. +The useful result is that the optimization avoids a repeated whole-table scan +during recovery and both populated-table runs preserved lease correctness. + +The explicit `RedisPresenceRecoveryCostProbe` test-source main class is not +automatically run by the unit suite and imposes no machine-dependent CI gate. +After test compilation, run it on the test classpath with argument `1000`. +It fails if the real recovery window ends before the cold measurement, rather +than silently reporting ready-state latency as recovery latency. diff --git a/docs/acceptance/presence_recovery_browser.mjs b/docs/acceptance/presence_recovery_browser.mjs new file mode 100644 index 0000000..5171454 --- /dev/null +++ b/docs/acceptance/presence_recovery_browser.mjs @@ -0,0 +1,52 @@ +// Real browser recovery against the disposable two-JVM harness. No API mocks. +import { createRequire } from 'node:module' +import { execFileSync } from 'node:child_process' +import path from 'node:path' + +const [frontend, first, second, container, output, profile = 'chromium'] = process.argv.slice(2) +const require = createRequire(path.join(frontend, 'package.json')) +const { chromium, firefox, webkit, devices, expect } = require('@playwright/test') +const profiles = { + chromium: [chromium, devices['Desktop Chrome']], + firefox: [firefox, devices['Desktop Firefox']], + webkit: [webkit, devices['Desktop Safari']], + mobile: [chromium, devices['Pixel 7']], + tablet: [webkit, devices['iPad (gen 7)']], +} +if (!profiles[profile]) throw new Error(`Unknown browser profile: ${profile}`) +const [engine, device] = profiles[profile] +const browser = await engine.launch({ headless: true }) +const contexts = [] +try { + const pages = [] + for (const [port, user] of [[first, 'alice'], [second, 'bob']]) { + const context = await browser.newContext(device) + contexts.push(context) + await context.tracing.start({ screenshots: true, snapshots: true }) + const page = await context.newPage() + await page.goto(`http://127.0.0.1:${port}/?createMockSession=true&user=${user}`) + // The root export redirects after hydration; finish it before record navigation. + await page.waitForURL(url => /^\/app\/?$/.test(url.pathname), { waitUntil: 'domcontentloaded' }) + await page.goto(`http://127.0.0.1:${port}/app/person/1`) + pages.push(page) + } + const unavailable = page => page.getByText('Presence temporarily unavailable', { exact: true }) + await expect(pages[0].locator('[data-qqq-id="presence-bob"]')).toBeVisible({ timeout: 90000 }) + await expect(pages[1].locator('[data-qqq-id="presence-alice"]')).toBeVisible({ timeout: 90000 }) + execFileSync('docker', ['exec', container, 'redis-cli', 'FLUSHDB']) + for (const page of pages) await expect(unavailable(page)).toBeVisible({ timeout: 35000 }) + await expect(pages[0].locator('[data-qqq-id="presence-bob"]')).toHaveCount(0) + await expect(pages[1].locator('[data-qqq-id="presence-alice"]')).toHaveCount(0) + await expect(pages[0].locator('[data-qqq-id="presence-bob"]')).toBeVisible({ timeout: 90000 }) + await expect(pages[1].locator('[data-qqq-id="presence-alice"]')).toBeVisible({ timeout: 90000 }) + for (const [index, page] of pages.entries()) { + await expect(unavailable(page)).toHaveCount(0) + await page.screenshot({ path: path.join(output, `recovered-${index}.png`) }) + } + console.log(`PASS: ${profile} shows recovery unavailable, clears stale people, and rejoins both JVMs without reload.`) +} finally { + for (const [index, context] of contexts.entries()) { + await context.tracing.stop({ path: path.join(output, `browser-${index}.zip`) }) + } + await browser.close() +} diff --git a/docs/acceptance/presence_recovery_processes.py b/docs/acceptance/presence_recovery_processes.py new file mode 100644 index 0000000..e406f7a --- /dev/null +++ b/docs/acceptance/presence_recovery_processes.py @@ -0,0 +1,172 @@ +"""Exercise Redis loss through two independent SharedPresenceDevServer JVMs. + +Run after test-compile and dependency:build-classpath with test scope and output +target/recovery-process-classpath.txt. Uses only disposable loopback fixtures. +""" + +import argparse +import hashlib +import json +import os +from pathlib import Path +import socket +import shutil +import subprocess +import time +import urllib.error +import urllib.request +import uuid + + +def free_port(): + """Reserve a candidate loopback port; startup detects any bind race.""" + with socket.socket() as listener: + listener.bind(("127.0.0.1", 0)) + return listener.getsockname()[1] + + +def request(port, user, method, path, body=None): + """Use the fixture's mock session through the actual HTTP boundary.""" + separator = "&" if "?" in path else "?" + url = f"http://127.0.0.1:{port}/qqq/v1/qbit/collaboration{path}{separator}user={user}" + data = None if body is None else json.dumps(body).encode() + req = urllib.request.Request(url, data=data, method=method, headers={ + "Cookie": f"sessionUUID={user}-recovery-processes", + "Content-Type": "application/json", + }) + try: + response = urllib.request.urlopen(req, timeout=10) + except urllib.error.HTTPError as error: + response = error + with response: + return response.status, json.load(response) + + +def require(status, result): + """Fail with fixture-only response evidence when transport differs.""" + assert result[0] == status, result + return result[1] + + +def settle(ports, leases): + """Renew on the opposite node until both nodes report both people.""" + deadline = time.monotonic() + 60 + sequence = 0 + while time.monotonic() < deadline: + sequence += 1 + for index, (user, lease) in enumerate(leases.items()): + result = request(ports[1 - index], user, "POST", "/presence/leases/renew", { + "connectionId": lease, "table": "person", "pk": "1", + "mode": "VIEWING", "sequence": sequence, + }) + assert result[0] in (200, 503), result + if result[0] == 503: + assert "recovering" in result[1]["message"], result + snapshots = [request(port, "alice", "GET", "/presence?table=person&pk=1") for port in ports] + if all(status == 200 for status, _ in snapshots): + assert all(len(body["viewers"]) == 2 for _, body in snapshots), snapshots + return + assert all(status == 503 for status, _ in snapshots), snapshots + time.sleep(1) + raise AssertionError("Two-process recovery exceeded 60 seconds") + + +def main(): + """Own and clean up the exact child JVMs and disposable Redis container.""" + parser = argparse.ArgumentParser() + parser.add_argument("--java", required=True) + parser.add_argument("--repo", type=Path, required=True) + parser.add_argument("--frontend-repo", type=Path) + parser.add_argument("--frontend-jar", type=Path, help="Serve this packaged frontend instead of a local export") + parser.add_argument("--backend-jar", type=Path, help="Use this backend artifact instead of target/classes") + parser.add_argument("--browser-profile", choices=("chromium", "firefox", "webkit", "mobile", "tablet"), default="chromium") + args = parser.parse_args() + repo = args.repo.resolve() + output = repo / "target" / ("recovery-processes-" + uuid.uuid4().hex[:10]) + output.mkdir() + if args.frontend_jar and not args.frontend_repo: + parser.error("frontend-jar requires frontend-repo for browser tooling") + dependencies = (repo / "target/recovery-process-classpath.txt").read_text().strip().split(os.pathsep) + assert not any(Path(item).name.startswith(("qbit-collaboration-", "qqq-frontend-next-")) for item in dependencies) + backend = args.backend_jar.resolve(strict=True) if args.backend_jar else repo / "target/classes" + classpath = os.pathsep.join([str(repo / "target/test-classes"), str(backend), *dependencies]) + evidence = {"status": "FAIL", "browser_profile": args.browser_profile, "cleanup": False} + if args.backend_jar: + evidence["backend_jar_sha256"] = hashlib.sha256(backend.read_bytes()).hexdigest() + if args.frontend_jar: + frontend = args.frontend_jar.resolve(strict=True) + evidence["frontend_jar_sha256"] = hashlib.sha256(frontend.read_bytes()).hexdigest() + classpath = str(frontend) + os.pathsep + classpath + elif args.frontend_repo: + shutil.copytree(args.frontend_repo.resolve() / "out", output / "next-dashboard") + classpath = str(output) + os.pathsep + classpath + container = "qqq-recovery-" + uuid.uuid4().hex[:12] + processes, logs = [], [] + try: + subprocess.run(["docker", "run", "-d", "--name", container, "-p", "127.0.0.1::6379", "redis:7.4-alpine"], check=True, capture_output=True) + binding = subprocess.check_output(["docker", "port", container, "6379"], text=True).strip() + redis_port = int(binding.rsplit(":", 1)[1]) + ports = [free_port(), free_port()] + for port in ports: + argfile = output / f"node-{port}.args" + argfile.write_text('-Dcollab.dev.frontend=' + str(bool(args.frontend_repo)).lower() + '\n-cp\n"' + classpath.replace("\\", "/") + '"\n' + 'com.kingsrook.qbits.collaboration.devserver.SharedPresenceDevServer\n' + f'{port}\n{redis_port}\n', encoding="utf-8") + log = (output / f"node-{port}.log").open("w") + logs.append(log) + processes.append(subprocess.Popen([args.java, "@" + str(argfile)], cwd=repo, + stdout=log, stderr=subprocess.STDOUT, + creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0))) + deadline = time.monotonic() + 60 + for port, process in zip(ports, processes): + while f"SHARED_PRESENCE_READY {port}" not in (output / f"node-{port}.log").read_text(errors="replace"): + assert process.poll() is None, f"Node exited; see {output}" + assert time.monotonic() < deadline, f"Startup timed out; see {output}" + time.sleep(0.2) + if args.frontend_repo: + subprocess.run(["node", str(repo / "docs/acceptance/presence_recovery_browser.mjs"), + str(args.frontend_repo.resolve()), str(ports[0]), str(ports[1]), container, str(output), args.browser_profile], check=True) + evidence["status"] = "PASS" + return + def join(): + """Acquire each person's new lease on a different node.""" + return {user: require(200, request(port, user, "POST", "/presence/leases", { + "table": "person", "pk": "1"}))["connectionId"] for port, user in zip(ports, ("alice", "bob"))} + original = join() + settle(ports, original) + subprocess.run(["docker", "exec", container, "redis-cli", "FLUSHDB"], check=True, capture_output=True) + for port in ports: + require(503, request(port, "alice", "GET", "/presence?table=person&pk=1")) + for user, lease in original.items(): + error = require(409, request(port, user, "POST", "/presence/leases/renew", { + "connectionId": lease, "table": "person", "pk": "1", "mode": "VIEWING", "sequence": 1000})) + assert error["code"] == "PRESENCE_LEASE_EXPIRED", error + replacement = join() + assert all(replacement[user] != original[user] for user in original) + settle(ports, replacement) + print(f"PASS: two independent JVMs recovered two viewers after Redis loss. Logs: {output}") + evidence["status"] = "PASS" + except Exception as error: + evidence["failure"] = str(error) + raise + finally: + for process in processes: + process.terminate() + for process in processes: + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + process.kill() + process.wait(timeout=10) + for log in logs: + log.close() + cleanup = subprocess.run(["docker", "rm", "-f", container], capture_output=True, check=False) + evidence["cleanup"] = cleanup.returncode == 0 and all(process.poll() is not None for process in processes) + (output / "result.json").write_text(json.dumps(evidence, indent=2), encoding="utf-8") + print(output, flush=True) + if evidence["status"] == "PASS" and not evidence["cleanup"]: + raise RuntimeError("Recovery passed but owned fixture cleanup failed") + + +if __name__ == "__main__": + main() diff --git a/docs/acceptance/presence_restore_processes.py b/docs/acceptance/presence_restore_processes.py new file mode 100644 index 0000000..8b22710 --- /dev/null +++ b/docs/acceptance/presence_restore_processes.py @@ -0,0 +1,166 @@ +"""Restore a disposable Redis image while both fixture JVMs are stopped. + +This verifies the controlled-restore protocol, not production orchestration. +Requires the same compiled test classpath as presence_recovery_processes.py. +""" + +import argparse +import hashlib +import json +import os +from pathlib import Path +import socket +import subprocess +import time +import uuid + +from presence_recovery_processes import free_port, request, require, settle + + +def main(): + """Own a Redis fixture and two generations of independently running nodes.""" + parser = argparse.ArgumentParser() + parser.add_argument("--java", required=True) + parser.add_argument("--repo", type=Path, required=True) + parser.add_argument("--backend-jar", type=Path, help="Verify this artifact without production target/classes") + args = parser.parse_args() + repo = args.repo.resolve() + output = repo / "target" / ("restore-processes-" + uuid.uuid4().hex[:10]) + output.mkdir() + dependencies = (repo / "target/recovery-process-classpath.txt").read_text().strip().split(os.pathsep) + assert all(Path(item).is_file() and item.endswith('.jar') for item in dependencies) + assert not any(Path(item).name.startswith("qbit-collaboration-") for item in dependencies) + backend = args.backend_jar.resolve(strict=True) if args.backend_jar else repo / "target/classes" + classpath = os.pathsep.join([str(repo / "target/test-classes"), str(backend), *dependencies]) + evidence = {"status": "FAIL", "cleanup": False, "checks": []} + if args.backend_jar: + evidence["backend_jar_sha256"] = hashlib.sha256(backend.read_bytes()).hexdigest() + container = "qqq-restore-" + uuid.uuid4().hex[:12] + container_created = False + processes, logs = [], [] + ports = [free_port(), free_port()] + + def redis(*command): + """Run commands only against this owned, disposable Redis container.""" + return subprocess.check_output(["docker", "exec", container, "redis-cli", *command], text=True).strip() + + def stop_nodes(): + """Require process termination and closed old listeners before restore.""" + for process in processes: + if process.poll() is None: + process.terminate() + for process in processes: + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + process.kill() + process.wait(timeout=10) + processes.clear() + for port in ports: + with socket.socket() as connection: + connection.settimeout(1) + assert connection.connect_ex(("127.0.0.1", port)) != 0, "Old listener remains open" + + def start_nodes(generation, redis_port): + """Supply the shared generation outside the Redis image, on both JVMs.""" + for port in ports: + argfile = output / f"{generation}-{port}.args" + argfile.write_text(f'-Dcollab.dev.recoveryGeneration={generation}\n-cp\n"' + + classpath.replace("\\", "/") + '"\n' + 'com.kingsrook.qbits.collaboration.devserver.SharedPresenceDevServer\n' + f'{port}\n{redis_port}\n', encoding="utf-8") + logfile = output / f"{generation}-{port}.log" + log = logfile.open("w") + logs.append(log) + processes.append(subprocess.Popen([args.java, "@" + str(argfile)], cwd=repo, + stdout=log, stderr=subprocess.STDOUT, + creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0))) + deadline = time.monotonic() + 60 + for port, process in zip(ports, processes): + logfile = output / f"{generation}-{port}.log" + while f"SHARED_PRESENCE_READY {port}" not in logfile.read_text(errors="replace"): + assert process.poll() is None, f"Node exited; see {output}" + assert time.monotonic() < deadline, f"Startup timed out; see {output}" + time.sleep(0.2) + + def join(): + """Issue each person's lease on a separate node.""" + return {user: require(200, request(port, user, "POST", "/presence/leases", { + "table": "person", "pk": "1"}))["connectionId"] for port, user in zip(ports, ("alice", "bob"))} + + def reject_old(leases, sequence): + """Restored lease authority must fail on both new-generation nodes.""" + for port in ports: + for user, lease in leases.items(): + error = require(409, request(port, user, "POST", "/presence/leases/renew", { + "connectionId": lease, "table": "person", "pk": "1", + "mode": "VIEWING", "sequence": sequence})) + assert error["code"] == "PRESENCE_LEASE_EXPIRED", error + + try: + # DEBUG RELOAD is enabled only on this disposable loopback fixture. + subprocess.run(["docker", "run", "-d", "--name", container, "-p", "127.0.0.1::6379", + "redis:7.4-alpine", "redis-server", "--enable-debug-command", "yes", "--save", ""], + check=True, capture_output=True) + container_created = True + binding = subprocess.check_output(["docker", "port", container, "6379"], text=True).strip() + redis_port = int(binding.rsplit(":", 1)[1]) + start_nodes("before-restore", redis_port) + original = join() + settle(ports, original) + last_renewal = time.monotonic() + for port, (user, lease) in zip(ports, original.items()): + require(200, request(port, user, "POST", "/presence/leases/renew", { + "connectionId": lease, "table": "person", "pk": "1", + "mode": "VIEWING", "sequence": 100})) + assert redis("SAVE") == "OK" + before = int(redis("DBSIZE")) + assert before > 0 + stop_nodes() + evidence["checks"].append("old JVMs stopped and listeners closed before restore") + assert redis("FLUSHDB") == "OK" + assert redis("DBSIZE") == "0" + assert redis("DEBUG", "RELOAD", "NOSAVE") == "OK" + assert int(redis("DBSIZE")) > 0, "Saved Redis image was not restored" + evidence["checks"].append("nonempty saved Redis image restored") + start_nodes("restored-" + uuid.uuid4().hex, redis_port) + reject_old(original, 1000) + # Avoid a false pass caused only by the old 45-second lease expiring. + evidence["old_lease_rejection_seconds"] = time.monotonic() - last_renewal + assert evidence["old_lease_rejection_seconds"] < 40, "Restore exceeded the live-lease proof window" + evidence["checks"].append("both nodes reject old leases before normal expiry") + for port in ports: + result = require(503, request(port, "alice", "GET", "/presence?table=person&pk=1")) + assert "recovering" in result["message"], result + replacement = join() + assert all(replacement[user] != original[user] for user in original) + settle(ports, replacement) + reject_old(original, 1001) + evidence["checks"].append("both nodes recover with fresh viewers and still reject old leases") + if args.backend_jar: + assert hashlib.sha256(backend.read_bytes()).hexdigest() == evidence["backend_jar_sha256"] + evidence["status"] = "PASS" + print(f"PASS: restored Redis image cannot authorize old leases on either new-generation JVM. Logs: {output}") + except Exception as error: + evidence["failure"] = str(error) + raise + finally: + try: + try: + stop_nodes() + finally: + for log in logs: + log.close() + if container_created: + subprocess.run(["docker", "rm", "-f", container], capture_output=True, check=True) + evidence["cleanup"] = True + except Exception as error: + evidence["status"] = "FAIL" + evidence["cleanup_failure"] = str(error) + raise + finally: + (output / "result.json").write_text(json.dumps(evidence, indent=2), encoding="utf-8") + + +if __name__ == "__main__": + main() diff --git a/docs/performance/MULTI-NODE.md b/docs/performance/MULTI-NODE.md new file mode 100644 index 0000000..db3d1f0 --- /dev/null +++ b/docs/performance/MULTI-NODE.md @@ -0,0 +1,264 @@ +# Independent-node presence workload + +This fixture targets QQQ #1302/#1291. It runs 2–8 independent JVM HTTP servers +against an owned loopback Redis container. Every node serves the same configured +mock identities across 2–8 tenant scopes. The scope comes from the authenticated +fixture identity, never a supplied tenant parameter. Records are independently +seeded in memory; this does not qualify shared database or production IAM behavior. + +Build with Java 21: + +```powershell +./mvnw.cmd test-compile dependency:build-classpath '-Dmdep.includeScope=test' '-Dmdep.outputFile=target/capacity-classpath.txt' +python docs/performance/presence_nodes.py --java /path/to/java --nodes 3 --tenants 2 --users 6 --tabs 4 --rounds 100 +``` + +Four tabs per user means two duplicate tabs on each of two records. Every round +renews each issued lease on a rotating node, with sequential per-tab sequences. +All nodes must then return exactly the expected tenant's distinct people and +exactly two editors on both records. Any mismatch or request failure fails the +run. Requests use bounded worker concurrency and ten-second timeouts. Each node +has eight pooled Redis connections, a 500 ms pool wait and a 384 MiB heap ceiling. + +Results and per-node logs are retained under `target/capacity-nodes-`. +The result records parameters, node PIDs, elapsed time, sample counts and +nearest-rank HTTP p50/p95/p99/max. Renewal throughput includes snapshot checks +and is not maximum server throughput. All owned JVMs and the Redis container are +removed in `finally`; the receipt reports cleanup separately from test success. + +Each fixture JVM emits resource samples every 100 ms using Java management APIs +and the Redis QBit's existing pool metrics. The receipt summarizes sampled heap +used/committed bytes, active/idle/waiting pool counts and the configured maximum. +CPU core equivalents are process CPU time divided by observed wall time; they +can exceed one for a multithreaded JVM. Observation begins after server startup +and includes lease setup and workload execution. Unsupported CPU counters are +reported as null. Linux fixtures also read process `VmRSS` from `/proc/self/status` +every 100 ms and record `residentBytes_sampled_max`; unsupported platforms report +null. Linux-container runs require resident-memory samples. Heap is not total +resident memory, and process RSS is not the container's complete cgroup footprint. +Sampled peaks may miss +short bursts and must not be represented as exact maxima. Raw samples remain in +node logs; missing node samples or a sampled pool-bound violation fail the run. + +The final Redis INFO capture records `used_memory`, lifetime `used_memory_peak`, +and current `used_memory_rss`, plus process user/system CPU seconds since Redis +startup. Redis RSS is an end-of-run observation, not a sampled peak. These fresh, +owned Redis instances include fixture setup and diagnostics in their lifetime; +do not interpret those CPU values as workload-only measurements or add independent +process peaks and call the sum a measured simultaneous maximum. Missing Redis +memory/CPU diagnostics fail acceptance while cleanup still runs. + +The [100-user resource run](results/linux-100-users-resources.json) used three +Linux JVMs, two tenants with 50 people each, four leases per person, 16 request +workers and 200 rounds. All 80,000 renewals and 2,400 exact-membership snapshots +passed in 102.568 workload seconds. Renewal p50/p95/p99/max were +11.326/21.765/25.881/1,007.404 ms; snapshot p99 was 19.424 ms. The one-second +renewal maximum is retained, not discarded. Cleanup passed. + +Sampled JVM RSS peaks were 662,786,048 / 662,188,032 / 703,479,808 bytes +(approximately 632–671 MiB), while used-heap peaks were about 171–177 MiB. +CPU observations were 1.73–1.83 core equivalents per JVM; sampled pool-active +peaks were 7/7/6 of eight with no sampled waiters. Redis peak allocated memory +was 2,579,896 bytes; final RSS was 8,777,728 bytes. Redis process CPU since +startup was 23.738 system plus 35.526 user seconds. These measurements include +fixture overhead, omit browser load, and do not establish deployment limits. + +Initial smoke run: three JVMs, two tenants, six users each, four tabs per user, +three rounds: 144 renewals and 36 snapshots passed with tenant isolation, +duplicate-tab deduplication and two editors retained. Cleanup passed. This short +run validates the harness, not sustained capacity or operational thresholds. + +A second run with 12 users per tenant and 100 rounds passed 9,600 renewals and +1,200 snapshots in 23.110 seconds. Renewal p50/p95/p99 were 7.246/16.454/21.853 ms; +snapshot p50/p95/p99 were 12.908/15.776/18.239 ms. Every round checked exact +tenant membership, deduplication and two editors on both records at every node. +Cleanup passed. These are measurements from one local Windows/Docker run; +they do not establish deployment limits or browser propagation latency. + +## Sustained-run limitation under investigation + +The 1,000-round workload has intermittently failed with HTTP 503 on lease +renewal. Four runs failed; one completed 96,000 renewals and 12,000 snapshots. +The passing run does not resolve the failures. Temporary local diagnostics on +the third failure identified `JedisConnectionException` caused by +`SocketTimeoutException` while reading the `EVALSHA` response in +`RedisPresenceStore.mutate`. The configured socket timeout remained two seconds. +The temporary production-file diagnostics were removed after identifying this +path. No timeout increase, hidden request retry or production fix is included. + +Resource samples did not show pool waiters, but cannot rule out short bursts. +Redis slow-command timings and latency events are now captured before fixture +cleanup, alongside Redis CPU, memory and command counters, to distinguish server +execution cost from transport or scheduling delays. Slow-command summaries omit +command arguments. These diagnostics belong only to the disposable fixture. +Until the cause is established and qualified, this workload establishes no +supported sustained throughput or production alert thresholds. + +The timing-instrumented Windows-JVM run failed at round 480, with no commands +at or above the configured one-millisecond Redis slow-log threshold and no +latency-monitor events. Redis reported about 93 microseconds per `EVALSHA` on +average. This weakens a slow-Lua explanation but does not identify the delayed +component or exclude time spent outside command execution. + +## Linux comparison + +`--node-image eclipse-temurin:21-jre` optionally runs each JVM in its own Linux +container on an owned Docker network shared with Redis. The HTTP driver remains +on the host. Only loopback HTTP ports are published. Compiled fixture classes +and copied dependency JARs are mounted read-only; Redis remains disposable. +The existing loopback defaults remain unchanged when this option is absent. +Container mode records Docker client PIDs separately from container identities. +It removes the Windows-JVM-to-published-Redis-port path, but also changes the JVM +platform and scheduling environment; it is a diagnostic comparison, not proof +of a specific operating-system or Docker defect. Cleanup includes every owned +node container and the owned network. + +`--churn-every 20` departs all tabs belonging to the last user in each tenant +every 20 rounds. Every departed lease must reject a later renewal with HTTP 409 +and `PRESENCE_LEASE_EXPIRED` on another node. All surviving nodes must show that +user absent on both records, then show the original exact membership after +fresh, distinct leases are acquired and activated. Only this explicit rejoin +path acquires new leases; the node-crash path preserves existing ones. + +`--pause-redis` adds a controlled failure phase after the ordinary rounds. It +freezes only the owned Redis container, sends up to 64 concurrent renewal +requests, and requires HTTP 503 within an eight-second fixture bound. Snapshots +must also fail explicitly rather than return an empty room. Redis is unpaused +in `finally`; once reachable, higher-sequence renewals must recover the same +leases and exact membership. The test requires observed pool waiters and rejects +sampled active counts above the configured maximum. It does not retry failed +requests invisibly or raise production timeouts. + +The [Linux Redis-stall receipt](results/linux-redis-stall.json) passed with three +nodes, two tenants, 24 users and 96 tab leases. All 96 stalled renewals and all +three node snapshots returned the expected unavailable response. Existing +leases recovered after Redis resumed, and cleanup passed. This proves bounded +failure and recovery for an injected stall; it does not explain the intermittent +Windows timeout or establish a saturation throughput threshold. + +```powershell +python docs/performance/presence_nodes.py --java java --node-image eclipse-temurin:21-jre --nodes 3 --tenants 2 --users 12 --tabs 4 --rounds 1000 +``` + +The first Linux comparison completed 1,000 rounds in 250.811 seconds: 96,000 +renewals and 12,000 snapshots, with exact membership/editor assertions and +cleanup passing. Renewal p95/p99 were 15.331/17.760 ms. The sampled active pool +peak was five of eight, with no sampled waiters. This one run is separate from +the unresolved Windows failures and does not establish a production SLA. + +`--crash-round 50` abruptly kills the first node before that round, then sends +all existing leases through surviving nodes. It does not acquire replacement +leases or hide failed requests with retries. Exact tenant membership, duplicate +tabs and editor counts must remain correct on every surviving node. This tests +HTTP lease continuity after process loss, not browser/SSE reconnect, ingress +failover, database durability or a Redis outage. + +The Linux crash run passed 100 rounds (9,600 renewals and 996 snapshots) in +25.148 seconds. The first JVM exited with code 137 at round 50; the remaining +two retained all issued leases and expected membership. Cleanup passed. + +A [combined churn and node-loss run](results/linux-node-crash-churn.json) passed +100 rounds with the first node killed at round 50 and churn every 20 rounds. +Five cycles rejected all 40 departed handles and checked absence/restored +membership across both tenants and records on every surviving node. Cleanup +passed. This is bounded HTTP lifecycle qualification; browser reconnect and +missed-event behavior remain separate requirements. + +Recorded receipts retain both successful and failing evidence: + +- [Windows timeout with Redis diagnostics](results/windows-timeout.json) +- [Linux 1,000-round run](results/linux-1000-rounds.json) +- [Linux node crash](results/linux-node-crash.json) + +Only the Java executable path is normalized in these receipts. The Linux image +used for these runs was +`eclipse-temurin@sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99`. + +## Browser propagation baseline + +`browser_subscription_affinity.mjs FRONTEND_REPO BASE_URL RESULT_PATH` verifies +the current owning-node subscription contract against the two-node fixture. It +uses the connection ID from the real browser's successful subscription request, +then preserves the same session cookies while changing only the test routing +cookie. The [published-artifact receipt](results/browser-subscription-affinity.json) +records owner 200, other node 404, owner 200, with the requested channel active +on both successful responses and the unsaved draft retained. This proves why +subscription requests need owning-node routing even when presence state is shared. +It does not test the deployed Traefik cookie, HTTPS, buffering or failover. +The packaged runner accepts this script through `--affinity-script`. + +`browser_presence_latency.mjs` measures Chromium field-focus changes until a +different editor sees the corresponding remote marker. Supply a caller-owned +two-node loopback fixture serving the published frontend with `alice`/`admin` +mock identities and the acceptance router's `presence_node=first/second` +affinity cookies: + +```powershell +node docs/performance/browser_presence_latency.mjs /path/to/frontend http://localhost:PORT result.json 25 +``` + +Use `presence_router.mjs FIRST_NODE_PORT SECOND_NODE_PORT ROUTER_PORT` for this +test. It binds only to loopback and adds a test-only POST endpoint that severs +first-node SSE responses. Never deploy this fault-injection router. The packaged +acceptance fixture accepts it through `--router-script` alongside `--latency-script`. + +The test verifies the target marker is absent before every change, the old +marker disappears afterward, both real SSE streams opened successfully, +issued-lease writes reached distinct nodes, and unsaved drafts survive. API +responses are not mocked. The fixture owns backend/router/Redis cleanup; this +script owns and closes its browser. + +After timing the normal transitions, the script takes the observer offline and +explicitly severs its existing SSE response through the fixture router. It requires +an observed stream failure, a failed offline fetch, three acknowledged sender focus +changes, and absence of the new marker while disconnected. After network restoration +it requires a new successful SSE response and the current marker with the previous +marker removed, without reload or loss of either unsaved draft. Recovery may use +the shipped reconnect and polling behavior; this does not prove event replay. + +The [disconnect/recovery receipt](results/browser-reconnect.json) passed against +the public RC37 JAR: one stream severed, three missed changes, recovery in 1,858 ms, +both drafts preserved. The accompanying six-user and Redis admin checks passed; +fixture cleanup passed. This is one Chromium disconnect case without concurrent +load, not a reconnect storm, lease-expiry test, or production recovery objective. + +Append `load` to the browser command (or use `--latency-load` on the packaged +fixture) to add 16 concurrent HTTP lease workers: four mock identities with four +leases each on the same record, split between both nodes. Each worker renews +sequentially with a 100 ms pause after each response. The browser observations +are spread over at least 60 seconds; every observation interval must include +completed background renewals. The workload continues during disconnect/reconnect. +Any request or lease-cleanup failure fails the test. Responses are disposed after +reading, and workers stop before the contexts close. The receipt reports completed +renewals, observed node counts, elapsed time and any errors. These are HTTP load +workers, not 16 additional browser tabs or distinct people. + +The [bounded load/reconnect receipt](results/browser-load-reconnect.json) passed +9,046 renewals over 69.728 seconds, with no worker or lease-release errors. +Across 25 observed focus transitions, p50/p95/p99 were 32.384/85.391/785.495 ms; +p99 is the maximum of this small sample, not a reliable tail-latency estimate. +The observer recovered in 1,855.887 ms after missing three updates while load +continued. Both drafts survived. Six-user and Redis admin acceptance also passed +against the public RC37 JAR; fixture cleanup passed. An initial run failed only +because the new load helper used the wrong lease-release endpoint; that receipt +is retained in the local packaged fixture history, not counted as a passing run. + +The [25-transition receipt](results/browser-focus-25.json) passed against the +downloaded public RC37 JAR and published backend snapshots. Focus-to-observed-DOM +p50/p95/p99 were 31.648/33.078/36.092 ms; p99 is the maximum in this small sample. +There were no page errors or failed lease responses. Six-user presence and Redis +admin browser checks also passed in the same fixture run; fixture cleanup passed. + +This baseline includes browser automation, UI debounce, network, invalidation +and rendering. It is not a server-only transport measurement, an SSE-only causal +proof, an under-load result, or a production latency objective. The separate load +receipt above covers one bounded workload and disconnect scenario; larger loads, +reconnect storms and deployment-specific behavior remain unqualified. + +Remaining qualification includes longer soak and larger workloads, container +resource limits, reconnect storms and the actual deployment's capacity and alert +thresholds. The receipts above cover specific bounded HTTP, browser, saturation, +churn, missed-update and process-crash cases, not every combination. The native +Windows intermittent Redis read timeout remains unresolved. Loopback HTTP latency must not be reported +as browser propagation or a production SLA. No deployed capacity threshold is +established by this fixture. diff --git a/docs/performance/browser_presence_latency.mjs b/docs/performance/browser_presence_latency.mjs new file mode 100644 index 0000000..be5944b --- /dev/null +++ b/docs/performance/browser_presence_latency.mjs @@ -0,0 +1,145 @@ +// Real UI observations against a caller-owned two-node acceptance fixture. +import { createRequire } from 'node:module'; +import { resolve } from 'node:path'; +import { writeFile } from 'node:fs/promises'; +import { performance } from 'node:perf_hooks'; +import { setTimeout as delay } from 'node:timers/promises'; +import { startPresenceLoad } from './browser_presence_load.mjs'; + +const [frontendRepo, base, output, countValue = '25', loadMode] = process.argv.slice(2); +if (loadMode && loadMode !== 'load') throw new Error('Optional final argument must be load'); +const samples = Number(countValue); +if (!frontendRepo || !base || !output || !Number.isInteger(samples) || samples < 5 || samples > 100) { + throw new Error('Expected frontend repo, loopback base URL, result path and 5..100 samples'); +} +if (!['localhost', '127.0.0.1'].includes(new URL(base).hostname)) { + throw new Error('Only the disposable loopback fixture is supported'); +} +const require = createRequire(resolve(frontendRepo, 'package.json')); +const { chromium, expect } = require('@playwright/test'); +const result = { status: 'FAIL', samples: [], streams: [], requests: [], errors: [], + scope: 'Chromium focus-to-remote-DOM observation, including automation and UI debounce; no concurrent load' }; +const browser = await chromium.launch({ headless: true }); +let load; +try { + async function editor(user, node) { + const context = await browser.newContext(); + await context.addCookies([ + { name: 'collab_dev_user', value: user, url: base }, + { name: 'presence_node', value: node, url: base }, + ]); + const page = await context.newPage(); + page.setDefaultTimeout(15000); + page.on('pageerror', error => result.errors.push(error.message)); + page.on('response', response => { + const path = new URL(response.url()).pathname; + if (path.endsWith('/realtime/stream')) { + result.streams.push({ user, status: response.status(), type: response.headers()['content-type'] }); + } + if (path.includes('/presence/leases')) { + result.requests.push({ user, status: response.status(), node: response.headers()['x-acceptance-node'] }); + } + }); + await page.goto(`${base}/app/person/1/edit?createMockSession=true`); + await expect(page.locator('[data-qqq-id="firstName"]')).toBeVisible(); + return page; + } + const observer = await editor('alice', 'first'); + const sender = await editor('admin', 'second'); + await observer.locator('[data-qqq-id="firstName"]').fill('Retained observer draft'); + await sender.locator('[data-qqq-id="firstName"]').fill('Retained sender draft'); + await expect(observer.locator('[data-presence-field="firstName"]')).toContainText('Demo Administrator'); + if (loadMode) { + load = await startPresenceLoad(browser, base, expect); + result.load = load.state; + result.scope = 'Chromium focus-to-remote-DOM and reconnect under 16 concurrent HTTP lease workers; loopback, four background identities'; + } + for (let index = 0; index < samples; index++) { + const previousCompleted = load?.state.completed; + if (load) await delay(60000 / samples); + const field = index % 2 === 0 ? 'lastName' : 'firstName'; + const previous = field === 'firstName' ? 'lastName' : 'firstName'; + await expect(observer.locator(`[data-presence-field="${field}"]`)).not.toContainText('Demo Administrator'); + const before = performance.now(); + await sender.locator(`[data-qqq-id="${field}"]`).focus(); + await expect(observer.locator(`[data-presence-field="${field}"]`)).toContainText('Demo Administrator'); + result.samples.push({ field, propagation_ms: performance.now() - before }); + await expect(observer.locator(`[data-presence-field="${previous}"]`)).not.toContainText('Demo Administrator'); + if (load) { + expect(load.state.errors).toEqual([]); + expect(load.state.completed).toBeGreaterThan(previousCompleted); + } + } + // Drop a real browser connection, miss multiple focus updates, then converge + // through the shipped reconnect/polling behavior without reloading the form. + const oldField = samples % 2 === 0 ? 'firstName' : 'lastName'; + const newField = oldField === 'firstName' ? 'lastName' : 'firstName'; + const streamsBefore = result.streams.filter(stream => stream.user === 'alice').length; + result.reconnect = { status: 'FAIL', missed_changes: 0 }; + const interrupted = observer.waitForEvent('requestfailed', { + predicate: request => new URL(request.url()).pathname.endsWith('/realtime/stream'), + timeout: 10000, + }); + await observer.context().setOffline(true); + const disconnect = await fetch(`${base}/__acceptance/disconnect-first`, { method: 'POST' }); + expect(disconnect.status).toBe(200); + result.reconnect.disconnected_streams = (await disconnect.json()).disconnected; + expect(result.reconnect.disconnected_streams).toBeGreaterThan(0); + const failedStream = await interrupted; + result.reconnect.stream_failure = failedStream.failure()?.errorText; + const offlineFetch = await observer.evaluate(async () => { + try { + await fetch(`/app/?offlineProbe=${Date.now()}`, { cache: 'no-store' }); + return false; + } catch { return true; } + }); + expect(offlineFetch).toBe(true); + for (const field of [newField, oldField, newField]) { + const renewed = sender.waitForResponse(response => + new URL(response.url()).pathname.endsWith('/presence/leases/renew') && + response.request().method() === 'POST' && response.status() === 200 && + response.request().postDataJSON()?.fieldName === field); + await sender.locator(`[data-qqq-id="${field}"]`).focus(); + await renewed; + result.reconnect.missed_changes++; + } + await expect(observer.locator(`[data-presence-field="${newField}"]`)).not.toContainText('Demo Administrator'); + const resumed = performance.now(); + await observer.context().setOffline(false); + await expect.poll(() => result.streams.filter(stream => stream.user === 'alice' && + stream.status === 200 && stream.type?.includes('text/event-stream')).length, + { timeout: 30000 }).toBeGreaterThan(streamsBefore); + await expect(observer.locator(`[data-presence-field="${newField}"]`)).toContainText('Demo Administrator', { timeout: 30000 }); + await expect(observer.locator(`[data-presence-field="${oldField}"]`)).not.toContainText('Demo Administrator'); + result.reconnect.recovery_ms = performance.now() - resumed; + result.reconnect.status = 'PASS'; + if (load) { + await load.stop(); + expect(load.state.errors).toEqual([]); + expect(load.state.elapsed_ms).toBeGreaterThanOrEqual(60000); + expect(load.state.nodes.first).toBeGreaterThan(0); + expect(load.state.nodes.second).toBeGreaterThan(0); + load = undefined; + } + await expect(observer.locator('[data-qqq-id="firstName"]')).toHaveValue('Retained observer draft'); + await expect(sender.locator('[data-qqq-id="firstName"]')).toHaveValue('Retained sender draft'); + for (const user of ['alice', 'admin']) { + expect(result.streams.some(stream => stream.user === user && stream.status === 200 && stream.type?.includes('text/event-stream'))).toBe(true); + } + expect(result.requests.some(request => request.user === 'alice' && request.node === 'first')).toBe(true); + expect(result.requests.some(request => request.user === 'admin' && request.node === 'second')).toBe(true); + expect(result.requests.every(request => request.status === 200)).toBe(true); + expect(result.errors).toEqual([]); + const ordered = result.samples.map(sample => sample.propagation_ms).sort((a, b) => a - b); + result.percentiles_ms = Object.fromEntries([['p50', .5], ['p95', .95], ['p99', .99], ['max', 1]] + .map(([name, quantile]) => [name, ordered[Math.ceil(ordered.length * quantile) - 1]])); + result.status = 'PASS'; +} catch (error) { + result.failure = String(error); + process.exitCode = 1; +} finally { + if (load) await load.stop(); + await browser.close(); + await writeFile(output, JSON.stringify(result, null, 2)); + console.log(JSON.stringify(result, null, 2)); +} diff --git a/docs/performance/browser_presence_load.mjs b/docs/performance/browser_presence_load.mjs new file mode 100644 index 0000000..562f0c0 --- /dev/null +++ b/docs/performance/browser_presence_load.mjs @@ -0,0 +1,75 @@ +// Bounded real HTTP load using caller-owned authenticated browser contexts. +import { performance } from 'node:perf_hooks'; +import { setTimeout as delay } from 'node:timers/promises'; + +export async function startPresenceLoad(browser, base, expect) { + const state = { workers: 16, completed: 0, errors: [], nodes: {}, elapsed_ms: 0 }; + const contexts = []; + const leases = []; + let running = true; + let stopped = false; + let tasks = []; + const began = performance.now(); + const endpoint = `${base}/qqq/v1/qbit/collaboration/presence/leases`; + async function post(context, path, data) { + const response = await context.request.post(endpoint + path, { data, timeout: 10000 }); + try { + expect(response.status()).toBe(200); + const node = response.headers()['x-acceptance-node']; + state.nodes[node] = (state.nodes[node] ?? 0) + 1; + return await response.json(); + } finally { await response.dispose(); } + } + async function stop() { + if (stopped) return; + stopped = true; + running = false; + await Promise.all(tasks); + state.elapsed_ms = performance.now() - began; + for (const lease of leases) { + try { + const response = await lease.context.request.delete(`${base}/qqq/v1/qbit/collaboration/presence`, { + data: { ...lease.target, connectionId: lease.id, sequence: ++lease.sequence }, timeout: 10000, + }); + try { expect(response.status()).toBe(200); } finally { await response.dispose(); } + } catch (error) { state.errors.push(String(error)); } + } + for (const context of contexts) await context.close(); + } + try { + for (const [index, user] of ['bob', 'carol', 'dave', 'erin'].entries()) { + const context = await browser.newContext(); + contexts.push(context); + await context.addCookies([ + { name: 'collab_dev_user', value: user, url: base }, + { name: 'presence_node', value: index % 2 ? 'second' : 'first', url: base }, + ]); + const page = await context.newPage(); + await page.goto(`${base}/app/person/1?createMockSession=true`); + await expect(page.locator('[data-qqq-id="record-view-title-person"]')).toBeVisible(); + await page.close(); + for (let tab = 0; tab < 4; tab++) { + const target = { table: 'person', pk: '1' }; + const opened = await post(context, '', target); + expect(typeof opened.connectionId).toBe('string'); + leases.push({ context, target, id: opened.connectionId, sequence: 0, completed: 0 }); + } + } + tasks = leases.map(async lease => { + try { + while (running) { + await post(lease.context, '/renew', { ...lease.target, connectionId: lease.id, + sequence: ++lease.sequence, mode: 'VIEWING' }); + lease.completed++; + state.completed++; + await delay(100); + } + } catch (error) { state.errors.push(String(error)); running = false; } + }); + await expect.poll(() => leases.filter(lease => lease.completed > 0).length).toBe(16); + return { state, stop }; + } catch (error) { + await stop(); + throw error; + } +} diff --git a/docs/performance/browser_subscription_affinity.mjs b/docs/performance/browser_subscription_affinity.mjs new file mode 100644 index 0000000..374f466 --- /dev/null +++ b/docs/performance/browser_subscription_affinity.mjs @@ -0,0 +1,60 @@ +// Verify owning-node routing with real published UI and backend artifacts. +import { createRequire } from 'node:module'; +import { resolve } from 'node:path'; +import { writeFile } from 'node:fs/promises'; + +const [frontendRepo, base, output] = process.argv.slice(2); +if (!frontendRepo || !output || !['localhost', '127.0.0.1'].includes(new URL(base).hostname)) { + throw new Error('Expected frontend repo, owned loopback URL and result path'); +} +const require = createRequire(resolve(frontendRepo, 'package.json')); +const { chromium, expect } = require('@playwright/test'); +const result = { status: 'FAIL', checks: [], errors: [] }; +const browser = await chromium.launch({ headless: true }); +try { + const context = await browser.newContext(); + await context.addCookies([ + { name: 'collab_dev_user', value: 'alice', url: base }, + { name: 'presence_node', value: 'first', url: base }, + ]); + const page = await context.newPage(); + page.on('pageerror', error => result.errors.push(error.message)); + const subscribed = page.waitForResponse(response => + new URL(response.url()).pathname.endsWith('/realtime/subscriptions') && response.status() === 200); + await page.goto(`${base}/app/person/1/edit?createMockSession=true`); + await expect(page.locator('[data-qqq-id="firstName"]')).toBeVisible(); + await page.locator('[data-qqq-id="firstName"]').fill('Keep this unsaved draft'); + const response = await subscribed; + expect(response.headers()['x-acceptance-node']).toBe('first'); + const { connectionId } = response.request().postDataJSON(); + expect(typeof connectionId).toBe('string'); + for (const [node, expected] of [['first', 200], ['second', 404], ['first', 200]]) { + // Keep the same session/user cookies and change only the test routing cookie. + const cookies = (await context.cookies(base)).map(cookie => + `${cookie.name}=${cookie.name === 'presence_node' ? node : cookie.value}`).join('; '); + const changed = await context.request.post(`${base}/qqq/v1/realtime/subscriptions`, { + headers: { Cookie: cookies }, + data: { connectionId, add: ['record:person:1'], remove: [] }, timeout: 10000, + }); + try { + result.checks.push({ node: changed.headers()['x-acceptance-node'], status: changed.status() }); + expect(changed.headers()['x-acceptance-node']).toBe(node); + expect(changed.status()).toBe(expected); + if (expected === 200) { + const body = await changed.json(); + expect(body.active).toContain('record:person:1'); + expect(body.refused ?? []).toEqual([]); + } + } finally { await changed.dispose(); } + } + await expect(page.locator('[data-qqq-id="firstName"]')).toHaveValue('Keep this unsaved draft'); + expect(result.errors).toEqual([]); + result.status = 'PASS'; +} catch (error) { + result.failure = String(error); + process.exitCode = 1; +} finally { + await browser.close(); + await writeFile(output, JSON.stringify(result, null, 2)); + console.log(JSON.stringify(result, null, 2)); +} diff --git a/docs/performance/presence_nodes.py b/docs/performance/presence_nodes.py new file mode 100644 index 0000000..6ab098a --- /dev/null +++ b/docs/performance/presence_nodes.py @@ -0,0 +1,343 @@ +"""Bounded HTTP presence workload across independent, disposable JVMs. + +This measures loopback HTTP service latency, not browser propagation or a +production capacity limit. Run test-compile and dependency:build-classpath with +test scope and output target/capacity-classpath.txt before invoking this script. +""" + +import argparse +from concurrent.futures import ThreadPoolExecutor +import json +import math +import os +from pathlib import Path +import shutil +import socket +import subprocess +import sys +import time +import urllib.error +import urllib.request +import uuid + + +def port(): + """Choose a loopback port; child startup detects a competing bind.""" + with socket.socket() as listener: + listener.bind(("127.0.0.1", 0)) + return listener.getsockname()[1] + + +def request(node, user, path, body=None, tab=0, method=None, expected=200): + """Use real HTTP and the explicit mock identity; no tenant parameter.""" + separator = "&" if "?" in path else "?" + req = urllib.request.Request( + f"http://127.0.0.1:{node}/qqq/v1/qbit/collaboration{path}{separator}user={user}", + data=None if body is None else json.dumps(body).encode(), + method=method, + headers={"Content-Type": "application/json", + "Cookie": f"sessionUUID={user}-capacity-{tab}"}) + try: + response = urllib.request.urlopen(req, timeout=10) + except urllib.error.HTTPError as error: + if error.code == expected: + with error: + return json.load(error) + with error: + detail = error.read(4096).decode(errors="replace") + raise RuntimeError(f"HTTP {error.code} node={node} user={user} path={path}: {detail}") from error + with response: + if response.status != expected: + raise AssertionError(f"Unexpected status {response.status}") + return json.load(response) + + +def distribution(values): + """Nearest-rank milliseconds, with no discarded slow or failed samples.""" + ordered = sorted(values) + return {"samples": len(ordered), **{ + name: ordered[max(0, math.ceil(len(ordered) * fraction) - 1)] + for name, fraction in (("p50_ms", .5), ("p95_ms", .95), ("p99_ms", .99), ("max_ms", 1))}} + + +def main(): + """Own child JVMs/container and retain results even when a check fails.""" + parser = argparse.ArgumentParser() + parser.add_argument("--java", required=True) + parser.add_argument("--node-image", help="Optional Linux Java21 image; isolate Redis traffic on an owned Docker network") + parser.add_argument("--nodes", type=int, default=3) + parser.add_argument("--tenants", type=int, default=2) + parser.add_argument("--users", type=int, default=6) + parser.add_argument("--tabs", type=int, default=4) + parser.add_argument("--rounds", type=int, default=10) + parser.add_argument("--workers", type=int, default=8) + parser.add_argument("--crash-round", type=int, default=0, help="Kill the first node before this round; zero disables") + parser.add_argument("--churn-every", type=int, default=0, help="Leave/rejoin the last user's tabs every N rounds; zero disables") + parser.add_argument("--pause-redis", action="store_true", help="After the workload, freeze owned Redis and verify bounded failures/recovery") + args = parser.parse_args() + for name, lower, upper in (("nodes", 2, 8), ("tenants", 2, 8), ("users", 2, 100), + ("tabs", 2, 8), ("rounds", 1, 1000), ("workers", 1, 64)): + if not lower <= getattr(args, name) <= upper: + parser.error(f"{name} must be {lower}..{upper}") + if args.crash_round and not 2 <= args.crash_round <= args.rounds: + parser.error("crash-round must be zero or 2..rounds") + if args.churn_every and not 1 <= args.churn_every <= args.rounds: + parser.error("churn-every must be zero or 1..rounds") + repo = Path(__file__).resolve().parents[2] + output = repo / "target" / ("capacity-nodes-" + uuid.uuid4().hex[:10]) + output.mkdir(parents=True) + classpath = os.pathsep.join([str(repo / "target/test-classes"), str(repo / "target/classes"), + (repo / "target/capacity-classpath.txt").read_text().strip()]) + processes, logs = [], [] + container = "qqq-capacity-" + uuid.uuid4().hex[:12] + network = container + "-net" + node_containers = [] + network_created = False + result = {"status": "FAIL", "parameters": vars(args), "scope": "loopback HTTP, mock identities, memory records"} + try: + network_args = [] + if args.node_image: + result["node_image_id"] = subprocess.check_output( + ["docker", "image", "inspect", args.node_image, "--format", "{{.Id}}"], text=True).strip() + subprocess.run(["docker", "network", "create", network], check=True, capture_output=True) + network_created = True + network_args = ["--network", network] + dependencies = output / "dependencies" + dependencies.mkdir() + linux_classpath = ["/fixture/target/test-classes", "/fixture/target/classes"] + for index, dependency in enumerate((repo / "target/capacity-classpath.txt").read_text().strip().split(os.pathsep)): + destination = dependencies / f"{index}.jar" + shutil.copyfile(dependency, destination) + linux_classpath.append("/fixture/" + destination.relative_to(repo).as_posix()) + classpath = ":".join(linux_classpath) + subprocess.run(["docker", "run", "-d", "--name", container, "-p", "127.0.0.1::6379", + *network_args, + "redis:7.4-alpine", "redis-server", "--slowlog-log-slower-than", "1000", + "--slowlog-max-len", "128", "--latency-monitor-threshold", "1"], check=True, capture_output=True) + redis_port = int(subprocess.check_output(["docker", "port", container, "6379"], text=True).strip().rsplit(":", 1)[1]) + nodes = [port() for _ in range(args.nodes)] + if len(set(nodes)) != len(nodes): + raise RuntimeError("Port allocation collision; no nodes started") + for node in nodes: + argfile = output / f"node-{node}.args" + connection_properties = (f"-Dcollab.capacity.redisHost={container}\n-Dcollab.dev.bindAddress=0.0.0.0\n" + if args.node_image else "") + argfile.write_text(f'-Xmx384m\n-Dcollab.capacity.tenants={args.tenants}\n-Dcollab.capacity.users={args.users}\n' + + connection_properties + + '-cp\n"' + classpath.replace("\\", "/") + '"\n' + 'com.kingsrook.qbits.collaboration.devserver.CapacityPresenceDevServer\n' + + (f'8080\n6379\n' if args.node_image else f'{node}\n{redis_port}\n'), encoding="utf-8") + log = (output / f"node-{node}.log").open("w") + logs.append(log) + command = [args.java, "@" + str(argfile)] + if args.node_image: + node_container = f"{container}-node-{node}" + node_containers.append(node_container) + command = ["docker", "run", "--rm", "--name", node_container, "--network", network, + "-p", f"127.0.0.1:{node}:8080", "--mount", f"type=bind,source={repo},target=/fixture,readonly", + args.node_image, "java", "@/fixture/" + argfile.relative_to(repo).as_posix()] + processes.append(subprocess.Popen(command, cwd=repo, + stdout=log, stderr=subprocess.STDOUT, + creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0))) + deadline = time.monotonic() + 120 + for node, process in zip(nodes, processes): + while "CAPACITY_PRESENCE_READY " not in (output / f"node-{node}.log").read_text(errors="replace"): + if process.poll() is not None or time.monotonic() > deadline: + raise RuntimeError(f"Node {node} failed startup; inspect {output}") + time.sleep(.2) + actors = [] + for tenant in range(args.tenants): + for user in range(args.users): + for tab in range(args.tabs): + identity = f"load-t{tenant}-u{user}" + record = str(1 + tab % 2) + node = nodes[len(actors) % len(nodes)] + lease = request(node, identity, "/presence/leases", {"table": "person", "pk": record}, tab)["connectionId"] + actors.append((identity, record, tab, lease, user)) + renewals, snapshots = [], [] + + def check_rooms(excluded=None): + """Check every tenant/record on every surviving node, including absence.""" + for tenant in range(args.tenants): + expected_users = {f"load-t{tenant}-u{user}" for user in range(args.users) if user != excluded} + for node in nodes: + for record in ("1", "2"): + before = time.perf_counter() + viewers = request(node, f"load-t{tenant}-u0", f"/presence?table=person&pk={record}")["viewers"] + snapshots.append((time.perf_counter() - before) * 1000) + actual = {viewer["userId"] for viewer in viewers} + if actual != expected_users or len(viewers) != len(expected_users): + raise AssertionError(f"Tenant isolation/deduplication mismatch: {actual} != {expected_users}") + if sum(viewer["mode"] == "EDITING" for viewer in viewers) != sum(user != excluded for user in range(2)): + raise AssertionError("Unexpected editor count") + + def renew(item, sequence): + """Keep each tab sequential while moving its writes between nodes.""" + index, (identity, record, tab, lease, user) = item + before = time.perf_counter() + request(nodes[(index + sequence) % len(nodes)], identity, "/presence/leases/renew", + {"connectionId": lease, "table": "person", "pk": record, + "mode": "EDITING" if user < 2 else "VIEWING", "sequence": sequence}, tab) + return (time.perf_counter() - before) * 1000 + + started = time.perf_counter() + with ThreadPoolExecutor(max_workers=args.workers) as workers: + for sequence in range(1, args.rounds + 1): + result["current_round"] = sequence + if sequence == args.crash_round: + crashed = nodes.pop(0) + if args.node_image: + subprocess.run(["docker", "kill", node_containers[0]], check=True, capture_output=True) + else: + processes[0].kill() + processes[0].wait(timeout=10) + result["crash"] = {"round": sequence, "http_port": crashed, + "surviving_nodes": len(nodes), "process_exit": processes[0].returncode} + renewals.extend(workers.map(lambda item: renew(item, sequence), enumerate(actors))) + check_rooms() + if args.churn_every and sequence % args.churn_every == 0: + departing = [(index, actor) for index, actor in enumerate(actors) if actor[4] == args.users - 1] + for index, (identity, record, tab, lease, user) in departing: + request(nodes[index % len(nodes)], identity, "/presence", + {"connectionId": lease, "table": "person", "pk": record, "sequence": sequence + 1}, tab, method="DELETE") + response = request(nodes[(index + 1) % len(nodes)], identity, "/presence/leases/renew", + {"connectionId": lease, "table": "person", "pk": record, + "mode": "VIEWING", "sequence": sequence + 2}, tab, expected=409) + if response.get("code") != "PRESENCE_LEASE_EXPIRED": + raise AssertionError("Departed lease was not rejected as expired") + check_rooms(excluded=args.users - 1) + for index, (identity, record, tab, lease, user) in departing: + replacement = request(nodes[(index + 1) % len(nodes)], identity, "/presence/leases", + {"table": "person", "pk": record}, tab)["connectionId"] + if replacement == lease: + raise AssertionError("Rejoin reused departed authority") + actors[index] = (identity, record, tab, replacement, user) + renew((index, actors[index]), sequence) + check_rooms() + result["churn_cycles"] = result.get("churn_cycles", 0) + 1 + result["departed_leases_rejected"] = result.get("departed_leases_rejected", 0) + len(departing) + result["completed_rounds"] = sequence + if args.pause_redis: + failures = [] + + def stalled_renew(item): + """A frozen authority must fail explicitly, never acknowledge a write.""" + index, (identity, record, tab, lease, user) = item + before = time.perf_counter() + response = request(nodes[index % len(nodes)], identity, "/presence/leases/renew", + {"connectionId": lease, "table": "person", "pk": record, + "mode": "EDITING" if user < 2 else "VIEWING", "sequence": args.rounds + 1}, + tab, expected=503) + if "viewers" in response or "temporarily unavailable" not in response.get("message", ""): + raise AssertionError("Stalled authority did not return the expected unavailable response") + duration = (time.perf_counter() - before) * 1000 + if duration > 8000: + raise AssertionError("Unavailable response exceeded the eight-second fixture bound") + return duration + + subprocess.run(["docker", "pause", container], check=True, capture_output=True) + try: + with ThreadPoolExecutor(max_workers=64) as burst: + failures.extend(burst.map(stalled_renew, enumerate(actors))) + for node in nodes: + response = request(node, "load-t0-u0", "/presence?table=person&pk=1", expected=503) + if "viewers" in response: + raise AssertionError("Stalled authority was presented as a room snapshot") + finally: + subprocess.run(["docker", "unpause", container], check=True, capture_output=True) + subprocess.run(["docker", "exec", container, "redis-cli", "PING"], check=True, capture_output=True, timeout=10) + for item in enumerate(actors): + renew(item, args.rounds + 2) + check_rooms() + result["redis_stall"] = {"rejected_renewals": len(failures), "failed_snapshot_nodes": len(nodes), + "response_times": distribution(failures), "same_leases_recovered": True} + elapsed = time.perf_counter() - started + result.update(status="PASS", seconds=elapsed, renewals=distribution(renewals), + snapshots=distribution(snapshots), renewal_throughput=len(renewals) / elapsed, + node_pids=[process.pid for process in processes], + pid_kind="docker-client" if args.node_image else "JVM", node_containers=node_containers) + except Exception as error: + result["error"] = str(error) + raise + finally: + failure_active = sys.exc_info()[0] is not None + try: + slow = json.loads(subprocess.check_output( + ["docker", "exec", container, "redis-cli", "--json", "SLOWLOG", "GET", "128"], text=True, timeout=10)) + result["redis_slow_commands"] = [{"unix_time": entry[1], "duration_us": entry[2], "command": entry[3][0]} + for entry in slow] + result["redis_latency_events"] = json.loads(subprocess.check_output( + ["docker", "exec", container, "redis-cli", "--json", "LATENCY", "LATEST"], text=True, timeout=10)) + for section in ("cpu", "memory", "commandstats", "stats"): + value = subprocess.check_output(["docker", "exec", container, "redis-cli", "INFO", section], text=True, timeout=10) + (output / f"redis-{section}.txt").write_text(value) + parsed = dict(line.split(":", 1) for line in value.splitlines() if ":" in line and not line.startswith("#")) + if section == "memory": + result["redis_memory"] = {key: int(parsed[key]) for key in + ("used_memory", "used_memory_peak", "used_memory_rss")} + if section == "cpu": + result["redis_cpu_seconds_since_start"] = {key: float(parsed[key]) for key in + ("used_cpu_sys", "used_cpu_user")} + except (subprocess.SubprocessError, ValueError, KeyError) as error: + result["redis_diagnostic_error"] = type(error).__name__ + node_cleanup = True + for node_container in node_containers: + removed = subprocess.run(["docker", "rm", "-f", node_container], capture_output=True) + if removed.returncode != 0: + remaining = subprocess.run(["docker", "ps", "-aq", "--filter", f"name=^{node_container}$"], capture_output=True, text=True) + node_cleanup = node_cleanup and remaining.returncode == 0 and not remaining.stdout.strip() + for process in processes: + if process.poll() is None: + process.terminate() + for process in processes: + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + process.kill() + process.wait(timeout=10) + for log in logs: + log.close() + resources = [] + for node_log in sorted(output.glob("node-*.log")): + samples = [json.loads(line.split("CAPACITY_METRIC ", 1)[1]) + for line in node_log.read_text(errors="replace").splitlines() + if line.startswith("CAPACITY_METRIC ")] + if samples: + first, last = samples[0], samples[-1] + wall_ms = last["uptimeMillis"] - first["uptimeMillis"] + cpu_ns = last["cpuNanos"] - first["cpuNanos"] + resources.append({"node_log": node_log.name, "samples": len(samples), + "observed_wall_ms": wall_ms, + "cpu_seconds": cpu_ns / 1e9 if first["cpuNanos"] >= 0 else None, + "cpu_core_equivalents": cpu_ns / (wall_ms * 1e6) if wall_ms > 0 and first["cpuNanos"] >= 0 else None, + **{key + "_sampled_max": max(sample[key] for sample in samples) + for key in ("heapUsedBytes", "heapCommittedBytes", "poolActive", "poolIdle", "poolWaiting", "poolMaximum")}, + "residentBytes_sampled_max": max(sample["residentBytes"] for sample in samples) + if all(sample.get("residentBytes", -1) >= 0 for sample in samples) else None}) + if any(sample["poolActive"] > sample["poolMaximum"] for sample in samples): + result.update(status="FAIL", error="Sampled pool active count exceeded configured maximum") + result["resources"] = resources + if result["status"] == "PASS" and args.node_image and any(node["residentBytes_sampled_max"] is None for node in resources): + result.update(status="FAIL", error="Linux resident-memory samples missing") + if result["status"] == "PASS" and any(key not in result for key in ("redis_memory", "redis_cpu_seconds_since_start")): + result.update(status="FAIL", error="Redis memory/CPU diagnostics missing") + if result["status"] == "PASS" and len(resources) != args.nodes: + result.update(status="FAIL", error="Missing resource samples for a node") + if result["status"] == "PASS" and args.pause_redis and not any(node["poolWaiting_sampled_max"] > 0 for node in resources): + result.update(status="FAIL", error="Redis stall did not demonstrate pool waiters") + cleanup = subprocess.run(["docker", "rm", "-f", container], capture_output=True) + network_cleanup = True + if network_created: + network_cleanup = subprocess.run(["docker", "network", "rm", network], capture_output=True).returncode == 0 + result["cleanup"] = cleanup.returncode == 0 and node_cleanup and network_cleanup and all(process.poll() is not None for process in processes) + (output / "result.json").write_text(json.dumps(result, indent=2)) + print(json.dumps(result, indent=2)) + print(output) + if result["status"] == "PASS" and not result["cleanup"]: + raise RuntimeError("Workload passed but owned resource cleanup failed") + if result["status"] == "FAIL" and not failure_active: + raise RuntimeError(result.get("error", "Capacity resource validation failed")) + + +if __name__ == "__main__": + main() diff --git a/docs/performance/presence_router.mjs b/docs/performance/presence_router.mjs new file mode 100644 index 0000000..e840b0b --- /dev/null +++ b/docs/performance/presence_router.mjs @@ -0,0 +1,42 @@ +// Disposable loopback fixture only; never deploy this fault-injection router. +import http from 'node:http'; + +const [first, second, listen] = process.argv.slice(2).map(Number); +if (![first, second, listen].every(port => Number.isInteger(port) && port > 0 && port < 65536)) { + throw new Error('Expected first-node, second-node and router ports'); +} +const firstStreams = new Set(); +const server = http.createServer((request, response) => { + if (request.method === 'POST' && request.url === '/__acceptance/disconnect-first') { + const disconnected = firstStreams.size; + for (const stream of firstStreams) stream.destroy(); + response.writeHead(200, { 'content-type': 'application/json' }); + response.end(JSON.stringify({ disconnected })); + return; + } + const isSecond = /(?:^|;\s*)presence_node=second(?:;|$)/.test(request.headers.cookie ?? ''); + const port = isSecond ? second : first; + const upstream = http.request({ + hostname: '127.0.0.1', port, path: request.url, method: request.method, + headers: { ...request.headers, host: `127.0.0.1:${port}` }, + }, incoming => { + response.writeHead(incoming.statusCode, { + ...incoming.headers, 'x-acceptance-node': isSecond ? 'second' : 'first', + }); + if (!isSecond && incoming.headers['content-type']?.includes('text/event-stream')) { + firstStreams.add(response); + } + incoming.pipe(response); + }); + upstream.on('error', () => { + if (!response.headersSent) response.writeHead(502); + response.end('Acceptance node unavailable'); + }); + response.on('close', () => { + firstStreams.delete(response); + upstream.destroy(); + }); + request.pipe(upstream); +}); +server.listen(listen, '127.0.0.1', () => console.log(`PRESENCE_ROUTER_READY ${listen}`)); +for (const signal of ['SIGINT', 'SIGTERM']) process.on(signal, () => server.close()); diff --git a/docs/performance/results/browser-focus-25.json b/docs/performance/results/browser-focus-25.json new file mode 100644 index 0000000..cf81846 --- /dev/null +++ b/docs/performance/results/browser-focus-25.json @@ -0,0 +1,285 @@ +{ + "status": "PASS", + "samples": [ + { + "field": "lastName", + "propagation_ms": 36.092099999999846 + }, + { + "field": "firstName", + "propagation_ms": 32.51870000000008 + }, + { + "field": "lastName", + "propagation_ms": 31.758600000000115 + }, + { + "field": "firstName", + "propagation_ms": 31.090299999999843 + }, + { + "field": "lastName", + "propagation_ms": 31.64980000000014 + }, + { + "field": "firstName", + "propagation_ms": 31.648199999999633 + }, + { + "field": "lastName", + "propagation_ms": 31.395500000000084 + }, + { + "field": "firstName", + "propagation_ms": 32.36750000000029 + }, + { + "field": "lastName", + "propagation_ms": 31.601799999999912 + }, + { + "field": "firstName", + "propagation_ms": 31.90599999999995 + }, + { + "field": "lastName", + "propagation_ms": 31.44150000000036 + }, + { + "field": "firstName", + "propagation_ms": 31.186699999999746 + }, + { + "field": "lastName", + "propagation_ms": 31.115800000000036 + }, + { + "field": "firstName", + "propagation_ms": 32.539400000000114 + }, + { + "field": "lastName", + "propagation_ms": 31.188000000000102 + }, + { + "field": "firstName", + "propagation_ms": 33.021200000000135 + }, + { + "field": "lastName", + "propagation_ms": 33.07759999999962 + }, + { + "field": "firstName", + "propagation_ms": 32.7176000000004 + }, + { + "field": "lastName", + "propagation_ms": 31.457300000000032 + }, + { + "field": "firstName", + "propagation_ms": 32.920900000000074 + }, + { + "field": "lastName", + "propagation_ms": 31.89130000000023 + }, + { + "field": "firstName", + "propagation_ms": 30.847200000000157 + }, + { + "field": "lastName", + "propagation_ms": 30.158300000000054 + }, + { + "field": "firstName", + "propagation_ms": 31.48239999999987 + }, + { + "field": "lastName", + "propagation_ms": 31.337600000000293 + } + ], + "streams": [ + { + "user": "alice", + "status": 200, + "type": "text/event-stream;charset=utf-8" + }, + { + "user": "admin", + "status": 200, + "type": "text/event-stream;charset=utf-8" + } + ], + "requests": [ + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + } + ], + "errors": [], + "scope": "Chromium focus-to-remote-DOM observation, including automation and UI debounce; no concurrent load", + "percentiles_ms": { + "p50": 31.648199999999633, + "p95": 33.07759999999962, + "p99": 36.092099999999846, + "max": 36.092099999999846 + }, + "frontend_jar_sha256": "c9fa2eb2381a47c8104f82f4e940d2bfa8ba67e2341298dde0de4e21ad3ff1ab", + "fixture_cleanup": true, + "backend_source": "Published core cc3981bb3/collaboration ecff347/Redis7fe5492 snapshots; two independent JVMs, mock identities and memory records." +} diff --git a/docs/performance/results/browser-load-reconnect.json b/docs/performance/results/browser-load-reconnect.json new file mode 100644 index 0000000..b2eb0cb --- /dev/null +++ b/docs/performance/results/browser-load-reconnect.json @@ -0,0 +1,357 @@ +{ + "status": "PASS", + "samples": [ + { + "field": "lastName", + "propagation_ms": 36.560600000000704 + }, + { + "field": "firstName", + "propagation_ms": 33.44050000000061 + }, + { + "field": "lastName", + "propagation_ms": 32.50890000000072 + }, + { + "field": "firstName", + "propagation_ms": 785.4951000000001 + }, + { + "field": "lastName", + "propagation_ms": 85.39069999999992 + }, + { + "field": "firstName", + "propagation_ms": 30.589599999999336 + }, + { + "field": "lastName", + "propagation_ms": 32.756099999998696 + }, + { + "field": "firstName", + "propagation_ms": 38.17399999999907 + }, + { + "field": "lastName", + "propagation_ms": 30.243099999999686 + }, + { + "field": "firstName", + "propagation_ms": 29.600799999992887 + }, + { + "field": "lastName", + "propagation_ms": 29.57220000000234 + }, + { + "field": "firstName", + "propagation_ms": 29.197800000001735 + }, + { + "field": "lastName", + "propagation_ms": 33.05130000000645 + }, + { + "field": "firstName", + "propagation_ms": 32.38390000000072 + }, + { + "field": "lastName", + "propagation_ms": 31.52850000000035 + }, + { + "field": "firstName", + "propagation_ms": 34.66169999999693 + }, + { + "field": "lastName", + "propagation_ms": 31.258300000001327 + }, + { + "field": "firstName", + "propagation_ms": 44.19029999999475 + }, + { + "field": "lastName", + "propagation_ms": 36.56189999999333 + }, + { + "field": "firstName", + "propagation_ms": 30.240600000004633 + }, + { + "field": "lastName", + "propagation_ms": 29.79899999999907 + }, + { + "field": "firstName", + "propagation_ms": 29.621500000001106 + }, + { + "field": "lastName", + "propagation_ms": 31.66189999999915 + }, + { + "field": "firstName", + "propagation_ms": 29.198399999993853 + }, + { + "field": "lastName", + "propagation_ms": 32.931200000006356 + } + ], + "streams": [ + { + "user": "alice", + "status": 200, + "type": "text/event-stream;charset=utf-8" + }, + { + "user": "admin", + "status": 200, + "type": "text/event-stream;charset=utf-8" + }, + { + "user": "alice", + "status": 200, + "type": "text/event-stream;charset=utf-8" + } + ], + "requests": [ + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + } + ], + "errors": [], + "scope": "Chromium focus-to-remote-DOM and reconnect under 16 concurrent HTTP lease workers; loopback, four background identities", + "load": { + "workers": 16, + "completed": 9046, + "errors": [], + "nodes": { + "first": 4540, + "second": 4522 + }, + "elapsed_ms": 69728.3579 + }, + "reconnect": { + "status": "PASS", + "missed_changes": 3, + "disconnected_streams": 1, + "stream_failure": "net::ERR_INCOMPLETE_CHUNKED_ENCODING", + "recovery_ms": 1855.8868999999977 + }, + "percentiles_ms": { + "p50": 32.38390000000072, + "p95": 85.39069999999992, + "p99": 785.4951000000001, + "max": 785.4951000000001 + }, + "frontend_jar_sha256": "c9fa2eb2381a47c8104f82f4e940d2bfa8ba67e2341298dde0de4e21ad3ff1ab", + "fixture_cleanup": true, + "backend_source": "Published core cc3981bb3/collaboration ecff347/Redis7fe5492 snapshots; two independent JVMs, mock identities and memory records.", + "fixture_checks": [ + "browser-six-presence", + "browser-redis-admin", + "browser-presence-latency" + ] +} diff --git a/docs/performance/results/browser-reconnect.json b/docs/performance/results/browser-reconnect.json new file mode 100644 index 0000000..16d4616 --- /dev/null +++ b/docs/performance/results/browser-reconnect.json @@ -0,0 +1,317 @@ +{ + "status": "PASS", + "samples": [ + { + "field": "lastName", + "propagation_ms": 37.25770000000011 + }, + { + "field": "firstName", + "propagation_ms": 32.14499999999998 + }, + { + "field": "lastName", + "propagation_ms": 32.40950000000021 + }, + { + "field": "firstName", + "propagation_ms": 33.52500000000009 + }, + { + "field": "lastName", + "propagation_ms": 31.228200000000015 + }, + { + "field": "firstName", + "propagation_ms": 30.430999999999585 + }, + { + "field": "lastName", + "propagation_ms": 31.493199999999888 + }, + { + "field": "firstName", + "propagation_ms": 33.14539999999988 + }, + { + "field": "lastName", + "propagation_ms": 32.11259999999993 + }, + { + "field": "firstName", + "propagation_ms": 33.23129999999992 + }, + { + "field": "lastName", + "propagation_ms": 31.06800000000021 + }, + { + "field": "firstName", + "propagation_ms": 30.109199999999873 + }, + { + "field": "lastName", + "propagation_ms": 31.534299999999803 + }, + { + "field": "firstName", + "propagation_ms": 33.770800000000236 + }, + { + "field": "lastName", + "propagation_ms": 31.654199999999946 + }, + { + "field": "firstName", + "propagation_ms": 33.31700000000001 + }, + { + "field": "lastName", + "propagation_ms": 33.108600000000024 + }, + { + "field": "firstName", + "propagation_ms": 32.27370000000019 + }, + { + "field": "lastName", + "propagation_ms": 31.452499999999873 + }, + { + "field": "firstName", + "propagation_ms": 31.86220000000003 + }, + { + "field": "lastName", + "propagation_ms": 31.734399999999823 + }, + { + "field": "firstName", + "propagation_ms": 32.04489999999987 + }, + { + "field": "lastName", + "propagation_ms": 31.051899999999932 + }, + { + "field": "firstName", + "propagation_ms": 30.876300000000356 + }, + { + "field": "lastName", + "propagation_ms": 31.56970000000001 + } + ], + "streams": [ + { + "user": "alice", + "status": 200, + "type": "text/event-stream;charset=utf-8" + }, + { + "user": "admin", + "status": 200, + "type": "text/event-stream;charset=utf-8" + }, + { + "user": "alice", + "status": 200, + "type": "text/event-stream;charset=utf-8" + } + ], + "requests": [ + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "alice", + "status": 200, + "node": "first" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + }, + { + "user": "admin", + "status": 200, + "node": "second" + } + ], + "errors": [], + "scope": "Chromium focus-to-remote-DOM observation, including automation and UI debounce; no concurrent load", + "reconnect": { + "status": "PASS", + "missed_changes": 3, + "disconnected_streams": 1, + "stream_failure": "net::ERR_INCOMPLETE_CHUNKED_ENCODING", + "recovery_ms": 1858.2972000000004 + }, + "percentiles_ms": { + "p50": 31.86220000000003, + "p95": 33.770800000000236, + "p99": 37.25770000000011, + "max": 37.25770000000011 + }, + "frontend_jar_sha256": "c9fa2eb2381a47c8104f82f4e940d2bfa8ba67e2341298dde0de4e21ad3ff1ab", + "fixture_cleanup": true, + "backend_source": "Published core cc3981bb3/collaboration ecff347/Redis7fe5492 snapshots; two independent JVMs, mock identities and memory records.", + "fixture_checks": [ + "browser-six-presence", + "browser-redis-admin", + "browser-presence-latency" + ] +} diff --git a/docs/performance/results/browser-subscription-affinity.json b/docs/performance/results/browser-subscription-affinity.json new file mode 100644 index 0000000..69840c8 --- /dev/null +++ b/docs/performance/results/browser-subscription-affinity.json @@ -0,0 +1,14 @@ +{ + "status": "PASS", + "checks": [ + { "node": "first", "status": 200 }, + { "node": "second", "status": 404 }, + { "node": "first", "status": 200 } + ], + "errors": [], + "fixture_cleanup": true, + "fixture_checks": ["browser-six-presence", "browser-redis-admin", "browser-subscription-affinity"], + "frontend_jar_sha256": "c9fa2eb2381a47c8104f82f4e940d2bfa8ba67e2341298dde0de4e21ad3ff1ab", + "backend_source": "Published core cc3981bb3/collaboration ecff347/Redis7fe5492 snapshots; two independent JVMs, mock identities and memory records.", + "scope": "Same browser session and live SSE connection ID; change only test-router affinity. Not deployed Traefik qualification." +} diff --git a/docs/performance/results/linux-100-users-resources.json b/docs/performance/results/linux-100-users-resources.json new file mode 100644 index 0000000..248f02a --- /dev/null +++ b/docs/performance/results/linux-100-users-resources.json @@ -0,0 +1,181 @@ +{ + "status": "PASS", + "parameters": { + "java": "/path/to/java21", + "node_image": "eclipse-temurin@sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99", + "nodes": 3, + "tenants": 2, + "users": 50, + "tabs": 4, + "rounds": 200, + "workers": 16, + "crash_round": 0, + "churn_every": 0, + "pause_redis": false + }, + "scope": "loopback HTTP, mock identities, memory records", + "node_image_id": "sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99", + "current_round": 200, + "completed_rounds": 200, + "seconds": 102.56801009993069, + "renewals": { + "samples": 80000, + "p50_ms": 11.325599974952638, + "p95_ms": 21.764500066637993, + "p99_ms": 25.881099980324507, + "max_ms": 1007.4040000326931 + }, + "snapshots": { + "samples": 2400, + "p50_ms": 15.130399959161878, + "p95_ms": 16.613700077868998, + "p99_ms": 19.423600053414702, + "max_ms": 28.864299994893372 + }, + "renewal_throughput": 779.9702843221492, + "node_pids": [ + 105696, + 4828, + 117588 + ], + "pid_kind": "docker-client", + "node_containers": [ + "qqq-capacity-f58f3ddcf5f3-node-59675", + "qqq-capacity-f58f3ddcf5f3-node-59676", + "qqq-capacity-f58f3ddcf5f3-node-59677" + ], + "redis_slow_commands": [ + { + "unix_time": 1793033347, + "duration_us": 1432, + "command": "EVALSHA" + }, + { + "unix_time": 1793033347, + "duration_us": 1612, + "command": "EVALSHA" + }, + { + "unix_time": 1793033340, + "duration_us": 1877, + "command": "EVALSHA" + }, + { + "unix_time": 1793033340, + "duration_us": 1313, + "command": "ZRANGE" + }, + { + "unix_time": 1793033336, + "duration_us": 1316, + "command": "EVALSHA" + }, + { + "unix_time": 1793033305, + "duration_us": 1459, + "command": "EVALSHA" + }, + { + "unix_time": 1793033300, + "duration_us": 1767, + "command": "EVALSHA" + }, + { + "unix_time": 1793033290, + "duration_us": 1180, + "command": "EVALSHA" + }, + { + "unix_time": 1793033283, + "duration_us": 1421, + "command": "EVALSHA" + }, + { + "unix_time": 1793033260, + "duration_us": 1291, + "command": "EVALSHA" + }, + { + "unix_time": 1793033260, + "duration_us": 1551, + "command": "EVALSHA" + }, + { + "unix_time": 1793033260, + "duration_us": 1083, + "command": "ZRANGE" + }, + { + "unix_time": 1793033255, + "duration_us": 1039, + "command": "EVALSHA" + }, + { + "unix_time": 1793033254, + "duration_us": 1215, + "command": "EVALSHA" + } + ], + "redis_latency_events": [ + [ + "command", + 1793033347, + 1, + 1 + ] + ], + "redis_cpu_seconds_since_start": { + "used_cpu_sys": 23.737706, + "used_cpu_user": 35.526214 + }, + "redis_memory": { + "used_memory": 2415952, + "used_memory_peak": 2579896, + "used_memory_rss": 8777728 + }, + "resources": [ + { + "node_log": "node-59675.log", + "samples": 1112, + "observed_wall_ms": 111099, + "cpu_seconds": 203.14, + "cpu_core_equivalents": 1.828459302063925, + "heapUsedBytes_sampled_max": 181098896, + "heapCommittedBytes_sampled_max": 211812352, + "poolActive_sampled_max": 7, + "poolIdle_sampled_max": 7, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8, + "residentBytes_sampled_max": 662786048 + }, + { + "node_log": "node-59676.log", + "samples": 1145, + "observed_wall_ms": 114399, + "cpu_seconds": 200.44, + "cpu_core_equivalents": 1.7521132177728826, + "heapUsedBytes_sampled_max": 185167416, + "heapCommittedBytes_sampled_max": 219152384, + "poolActive_sampled_max": 7, + "poolIdle_sampled_max": 7, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8, + "residentBytes_sampled_max": 662188032 + }, + { + "node_log": "node-59677.log", + "samples": 1159, + "observed_wall_ms": 115800, + "cpu_seconds": 200.16, + "cpu_core_equivalents": 1.728497409326425, + "heapUsedBytes_sampled_max": 179378632, + "heapCommittedBytes_sampled_max": 210763776, + "poolActive_sampled_max": 6, + "poolIdle_sampled_max": 7, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8, + "residentBytes_sampled_max": 703479808 + } + ], + "cleanup": true +} diff --git a/docs/performance/results/linux-1000-rounds.json b/docs/performance/results/linux-1000-rounds.json new file mode 100644 index 0000000..995a222 --- /dev/null +++ b/docs/performance/results/linux-1000-rounds.json @@ -0,0 +1,101 @@ +{ + "status": "PASS", + "parameters": { + "java": "java (Java 21)", + "node_image": "eclipse-temurin:21-jre", + "nodes": 3, + "tenants": 2, + "users": 12, + "tabs": 4, + "rounds": 1000, + "workers": 8 + }, + "scope": "loopback HTTP, mock identities, memory records", + "current_round": 1000, + "completed_rounds": 1000, + "seconds": 250.81072249996942, + "renewals": { + "samples": 96000, + "p50_ms": 6.353199947625399, + "p95_ms": 15.330899972468615, + "p99_ms": 17.760399961844087, + "max_ms": 326.8311999272555 + }, + "snapshots": { + "samples": 12000, + "p50_ms": 13.105699908919632, + "p95_ms": 14.261199976317585, + "p99_ms": 16.519399941898882, + "max_ms": 915.3468000004068 + }, + "renewal_throughput": 382.7587554595546, + "node_pids": [ + 64000, + 40240, + 115400 + ], + "pid_kind": "docker-client", + "node_containers": [ + "qqq-capacity-0a8b7956499d-node-64951", + "qqq-capacity-0a8b7956499d-node-64952", + "qqq-capacity-0a8b7956499d-node-64953" + ], + "redis_slow_commands": [ + { + "unix_time": 1793030517, + "duration_us": 1117, + "command": "EVALSHA" + } + ], + "redis_latency_events": [ + [ + "command", + 1793030517, + 1, + 1 + ] + ], + "resources": [ + { + "node_log": "node-64951.log", + "samples": 2580, + "observed_wall_ms": 257899, + "cpu_seconds": 165.55, + "cpu_core_equivalents": 0.6419179601316795, + "heapUsedBytes_sampled_max": 166667368, + "heapCommittedBytes_sampled_max": 188743680, + "poolActive_sampled_max": 4, + "poolIdle_sampled_max": 5, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-64952.log", + "samples": 2569, + "observed_wall_ms": 256799, + "cpu_seconds": 165.85, + "cpu_core_equivalents": 0.6458358482704372, + "heapUsedBytes_sampled_max": 169216944, + "heapCommittedBytes_sampled_max": 192937984, + "poolActive_sampled_max": 5, + "poolIdle_sampled_max": 4, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-64953.log", + "samples": 2608, + "observed_wall_ms": 260700, + "cpu_seconds": 164.72, + "cpu_core_equivalents": 0.631837360951285, + "heapUsedBytes_sampled_max": 166319008, + "heapCommittedBytes_sampled_max": 188743680, + "poolActive_sampled_max": 4, + "poolIdle_sampled_max": 5, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + } + ], + "cleanup": true, + "receipt_note": "Java executable path normalized; measurements unchanged. Disposable fixture identities only." +} diff --git a/docs/performance/results/linux-node-crash-churn.json b/docs/performance/results/linux-node-crash-churn.json new file mode 100644 index 0000000..9e2db3e --- /dev/null +++ b/docs/performance/results/linux-node-crash-churn.json @@ -0,0 +1,117 @@ +{ + "status": "PASS", + "parameters": { + "java": "java (Java 21)", + "node_image": "eclipse-temurin@sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99", + "nodes": 3, + "tenants": 2, + "users": 12, + "tabs": 4, + "rounds": 100, + "workers": 8, + "crash_round": 50, + "churn_every": 20 + }, + "scope": "loopback HTTP, mock identities, memory records", + "node_image_id": "sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99", + "current_round": 100, + "completed_rounds": 100, + "churn_cycles": 5, + "departed_leases_rejected": 40, + "crash": { + "round": 50, + "http_port": 52824, + "surviving_nodes": 2, + "process_exit": 137 + }, + "seconds": 28.090077199973166, + "renewals": { + "samples": 9600, + "p50_ms": 7.532799965701997, + "p95_ms": 17.53149996511638, + "p99_ms": 29.359900043345988, + "max_ms": 88.13189994543791 + }, + "snapshots": { + "samples": 1092, + "p50_ms": 13.582199928350747, + "p95_ms": 15.386099927127361, + "p99_ms": 18.502700026147068, + "max_ms": 22.144700051285326 + }, + "renewal_throughput": 341.75769371004685, + "node_pids": [ + 22844, + 66652, + 61920 + ], + "pid_kind": "docker-client", + "node_containers": [ + "qqq-capacity-bb9ba87919d2-node-52824", + "qqq-capacity-bb9ba87919d2-node-52825", + "qqq-capacity-bb9ba87919d2-node-52827" + ], + "redis_slow_commands": [ + { + "unix_time": 1793031681, + "duration_us": 1206, + "command": "EVALSHA" + }, + { + "unix_time": 1793031681, + "duration_us": 1147, + "command": "EVALSHA" + } + ], + "redis_latency_events": [ + [ + "command", + 1793031681, + 1, + 1 + ] + ], + "resources": [ + { + "node_log": "node-52824.log", + "samples": 212, + "observed_wall_ms": 21099, + "cpu_seconds": 29.48, + "cpu_core_equivalents": 1.3972226171856486, + "heapUsedBytes_sampled_max": 163641024, + "heapCommittedBytes_sampled_max": 188743680, + "poolActive_sampled_max": 3, + "poolIdle_sampled_max": 4, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-52825.log", + "samples": 354, + "observed_wall_ms": 35299, + "cpu_seconds": 43.09, + "cpu_core_equivalents": 1.22071446783195, + "heapUsedBytes_sampled_max": 168178592, + "heapCommittedBytes_sampled_max": 192937984, + "poolActive_sampled_max": 5, + "poolIdle_sampled_max": 5, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-52827.log", + "samples": 340, + "observed_wall_ms": 33900, + "cpu_seconds": 42.42, + "cpu_core_equivalents": 1.2513274336283187, + "heapUsedBytes_sampled_max": 166291160, + "heapCommittedBytes_sampled_max": 188743680, + "poolActive_sampled_max": 5, + "poolIdle_sampled_max": 5, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + } + ], + "cleanup": true, + "receipt_note": "Java executable path normalized; measurements unchanged. Disposable fixture identities only." +} diff --git a/docs/performance/results/linux-node-crash.json b/docs/performance/results/linux-node-crash.json new file mode 100644 index 0000000..bb87b25 --- /dev/null +++ b/docs/performance/results/linux-node-crash.json @@ -0,0 +1,96 @@ +{ + "status": "PASS", + "parameters": { + "java": "java (Java 21)", + "node_image": "eclipse-temurin@sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99", + "nodes": 3, + "tenants": 2, + "users": 12, + "tabs": 4, + "rounds": 100, + "workers": 8, + "crash_round": 50 + }, + "scope": "loopback HTTP, mock identities, memory records", + "node_image_id": "sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99", + "current_round": 100, + "completed_rounds": 100, + "crash": { + "round": 50, + "http_port": 54039, + "surviving_nodes": 2, + "process_exit": 137 + }, + "seconds": 25.147547999978997, + "renewals": { + "samples": 9600, + "p50_ms": 7.125200005248189, + "p95_ms": 17.407900071702898, + "p99_ms": 27.217599912546575, + "max_ms": 74.53640003222972 + }, + "snapshots": { + "samples": 996, + "p50_ms": 13.538200058974326, + "p95_ms": 15.59239998459816, + "p99_ms": 20.58999997097999, + "max_ms": 315.0679999962449 + }, + "renewal_throughput": 381.7469599822622, + "node_pids": [ + 85832, + 90280, + 67764 + ], + "pid_kind": "docker-client", + "node_containers": [ + "qqq-capacity-e599dca8e163-node-54039", + "qqq-capacity-e599dca8e163-node-54040", + "qqq-capacity-e599dca8e163-node-54041" + ], + "redis_slow_commands": [], + "redis_latency_events": [], + "resources": [ + { + "node_log": "node-54039.log", + "samples": 198, + "observed_wall_ms": 19699, + "cpu_seconds": 28.58, + "cpu_core_equivalents": 1.4508350677699375, + "heapUsedBytes_sampled_max": 163478640, + "heapCommittedBytes_sampled_max": 188743680, + "poolActive_sampled_max": 3, + "poolIdle_sampled_max": 4, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-54040.log", + "samples": 300, + "observed_wall_ms": 29900, + "cpu_seconds": 41.85, + "cpu_core_equivalents": 1.399665551839465, + "heapUsedBytes_sampled_max": 165177008, + "heapCommittedBytes_sampled_max": 188743680, + "poolActive_sampled_max": 4, + "poolIdle_sampled_max": 5, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-54041.log", + "samples": 331, + "observed_wall_ms": 33000, + "cpu_seconds": 41.52, + "cpu_core_equivalents": 1.2581818181818183, + "heapUsedBytes_sampled_max": 168546936, + "heapCommittedBytes_sampled_max": 192937984, + "poolActive_sampled_max": 4, + "poolIdle_sampled_max": 5, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + } + ], + "cleanup": true, + "receipt_note": "Java executable path normalized; measurements unchanged. Disposable fixture identities only." +} diff --git a/docs/performance/results/linux-redis-stall.json b/docs/performance/results/linux-redis-stall.json new file mode 100644 index 0000000..652b748 --- /dev/null +++ b/docs/performance/results/linux-redis-stall.json @@ -0,0 +1,117 @@ +{ + "status": "PASS", + "parameters": { + "java": "java (Java 21)", + "node_image": "eclipse-temurin@sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99", + "nodes": 3, + "tenants": 2, + "users": 12, + "tabs": 4, + "rounds": 30, + "workers": 8, + "crash_round": 0, + "churn_every": 0, + "pause_redis": true + }, + "scope": "loopback HTTP, mock identities, memory records", + "node_image_id": "sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99", + "current_round": 30, + "completed_rounds": 30, + "redis_stall": { + "rejected_renewals": 96, + "failed_snapshot_nodes": 3, + "response_times": { + "samples": 96, + "p50_ms": 1013.8123999349773, + "p95_ms": 4014.4751999760047, + "p99_ms": 4015.4822999611497, + "max_ms": 4015.4822999611497 + }, + "same_leases_recovered": true + }, + "seconds": 20.33888840000145, + "renewals": { + "samples": 2880, + "p50_ms": 8.835799992084503, + "p95_ms": 23.678299970924854, + "p99_ms": 35.42319999542087, + "max_ms": 102.84149996004999 + }, + "snapshots": { + "samples": 372, + "p50_ms": 13.98350007366389, + "p95_ms": 16.946100047789514, + "p99_ms": 20.020700059831142, + "max_ms": 20.943500101566315 + }, + "renewal_throughput": 141.6006589622565, + "node_pids": [ + 88080, + 78036, + 121352 + ], + "pid_kind": "docker-client", + "node_containers": [ + "qqq-capacity-68ad627e7be5-node-58312", + "qqq-capacity-68ad627e7be5-node-58314", + "qqq-capacity-68ad627e7be5-node-58315" + ], + "redis_slow_commands": [ + { + "unix_time": 1793032016, + "duration_us": 2198, + "command": "EVALSHA" + } + ], + "redis_latency_events": [ + [ + "command", + 1793032016, + 2, + 2 + ] + ], + "resources": [ + { + "node_log": "node-58312.log", + "samples": 280, + "observed_wall_ms": 27899, + "cpu_seconds": 23.96, + "cpu_core_equivalents": 0.8588121438044375, + "heapUsedBytes_sampled_max": 164221400, + "heapCommittedBytes_sampled_max": 188743680, + "poolActive_sampled_max": 6, + "poolIdle_sampled_max": 5, + "poolWaiting_sampled_max": 15, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-58314.log", + "samples": 289, + "observed_wall_ms": 28800, + "cpu_seconds": 23.2, + "cpu_core_equivalents": 0.8055555555555556, + "heapUsedBytes_sampled_max": 162849016, + "heapCommittedBytes_sampled_max": 192937984, + "poolActive_sampled_max": 4, + "poolIdle_sampled_max": 3, + "poolWaiting_sampled_max": 14, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-58315.log", + "samples": 272, + "observed_wall_ms": 27100, + "cpu_seconds": 23.85, + "cpu_core_equivalents": 0.8800738007380073, + "heapUsedBytes_sampled_max": 165779696, + "heapCommittedBytes_sampled_max": 188743680, + "poolActive_sampled_max": 5, + "poolIdle_sampled_max": 4, + "poolWaiting_sampled_max": 14, + "poolMaximum_sampled_max": 8 + } + ], + "cleanup": true, + "receipt_note": "Java executable path normalized; measurements unchanged. Disposable fixture identities only." +} diff --git a/docs/performance/results/windows-timeout.json b/docs/performance/results/windows-timeout.json new file mode 100644 index 0000000..9551682 --- /dev/null +++ b/docs/performance/results/windows-timeout.json @@ -0,0 +1,61 @@ +{ + "status": "FAIL", + "parameters": { + "java": "java (Java 21)", + "nodes": 3, + "tenants": 2, + "users": 12, + "tabs": 4, + "rounds": 1000, + "workers": 8 + }, + "scope": "loopback HTTP, mock identities, memory records", + "current_round": 480, + "completed_rounds": 479, + "error": "HTTP 503 node=56028 user=load-t0-u5 path=/presence/leases/renew: {\"error\":\"Presence is temporarily unavailable. Please retry.\",\"code\":\"ERROR\",\"message\":\"Presence is temporarily unavailable. Please retry.\",\"traceId\":\"cea30b82-e82b-49ec-af8a-64d8b6bd5af6\"}", + "redis_slow_commands": [], + "redis_latency_events": [], + "resources": [ + { + "node_log": "node-56027.log", + "samples": 1067, + "observed_wall_ms": 106600, + "cpu_seconds": 94.328125, + "cpu_core_equivalents": 0.8848792213883677, + "heapUsedBytes_sampled_max": 160559696, + "heapCommittedBytes_sampled_max": 178257920, + "poolActive_sampled_max": 5, + "poolIdle_sampled_max": 5, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-56028.log", + "samples": 1066, + "observed_wall_ms": 106501, + "cpu_seconds": 95.8125, + "cpu_core_equivalents": 0.8996394400052582, + "heapUsedBytes_sampled_max": 160251104, + "heapCommittedBytes_sampled_max": 178257920, + "poolActive_sampled_max": 5, + "poolIdle_sampled_max": 4, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + }, + { + "node_log": "node-56029.log", + "samples": 1066, + "observed_wall_ms": 106500, + "cpu_seconds": 95.25, + "cpu_core_equivalents": 0.8943661971830986, + "heapUsedBytes_sampled_max": 157995216, + "heapCommittedBytes_sampled_max": 175112192, + "poolActive_sampled_max": 5, + "poolIdle_sampled_max": 4, + "poolWaiting_sampled_max": 0, + "poolMaximum_sampled_max": 8 + } + ], + "cleanup": true, + "receipt_note": "Java executable path normalized; measurements unchanged. Disposable fixture identities only." +} diff --git a/pom.xml b/pom.xml index 32a601d..1fcaa40 100644 --- a/pom.xml +++ b/pom.xml @@ -72,10 +72,15 @@ + + com.kingsrook.qqq + qqq-backend-module-postgres + test + com.kingsrook.qbits qbit-redis - 0.1.0-redis-7fe5492-SNAPSHOT + 0.1.0-redis-integration-20261029-551f1d5-SNAPSHOT true @@ -247,7 +252,7 @@ true - 4.1.0-collab-33464d37e-SNAPSHOT + 4.1.0-virtual-field-values-1196-cc3981bb3-SNAPSHOT diff --git a/src/main/java/com/kingsrook/qbits/collaboration/CollaborationQBitProducer.java b/src/main/java/com/kingsrook/qbits/collaboration/CollaborationQBitProducer.java index 5cb6bcf..b389749 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/CollaborationQBitProducer.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/CollaborationQBitProducer.java @@ -187,6 +187,8 @@ public class CollaborationQBitProducer implements QBitMetaDataProducer query(String table, QQueryFilter filter) throws QException { - QueryInput input = new QueryInput(table).withFilter(filter).withShouldFetchHeavyFields(true).withTransaction(QContext.getQBackendTransaction()); + return (query(table, filter, QContext.getQBackendTransaction())); + } + + /******************************************************************************* + ** Provider reads participate in the originating query's explicit transaction. + *******************************************************************************/ + public static List query(String table, QQueryFilter filter, QBackendTransaction transaction) throws QException + { + QueryInput input = new QueryInput(table).withFilter(filter).withShouldFetchHeavyFields(true).withTransaction(transaction); input.setTableMetaData(serviceMetadata(table)); input.withPermissionBypass("Trusted collaboration service performs parent/ownership authorization."); return (new QueryAction().execute(input).getRecords().stream().map(QRecord::new).toList()); diff --git a/src/main/java/com/kingsrook/qbits/collaboration/CollaborationSchema.java b/src/main/java/com/kingsrook/qbits/collaboration/CollaborationSchema.java index 3fb2fc3..5be972d 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/CollaborationSchema.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/CollaborationSchema.java @@ -95,6 +95,15 @@ public final class CollaborationSchema statements.add("CREATE INDEX IF NOT EXISTS idx_collaboration_assignment_owner ON collaboration_record_assignment (owner_user_id)"); statements.add("CREATE INDEX IF NOT EXISTS idx_collaboration_assignment_assignee ON collaboration_record_assignment (assignee_user_id)"); statements.add("ALTER TABLE collaboration_mention ADD COLUMN IF NOT EXISTS read_at TIMESTAMPTZ"); + statements.add("ALTER TABLE collaboration_attachment_cleanup ADD COLUMN IF NOT EXISTS next_attempt_at TIMESTAMPTZ"); + statements.add("ALTER TABLE collaboration_attachment_cleanup ADD COLUMN IF NOT EXISTS resolved_at TIMESTAMPTZ"); + statements.add("ALTER TABLE collaboration_attachment_cleanup ADD COLUMN IF NOT EXISTS resolved_by VARCHAR(255)"); + statements.add("ALTER TABLE collaboration_attachment_cleanup ADD COLUMN IF NOT EXISTS settlement_evidence VARCHAR(2000)"); + statements.add("ALTER TABLE collaboration_attachment_cleanup ALTER COLUMN attachment_id DROP NOT NULL"); + statements.add("DROP INDEX IF EXISTS uq_collaboration_attachment_cleanup_1"); + statements.add("CREATE INDEX IF NOT EXISTS idx_collaboration_attachment_storage_key ON collaboration_attachment USING HASH (storage_key)"); + statements.add("CREATE INDEX IF NOT EXISTS idx_collaboration_attachment_cleanup_due ON collaboration_attachment_cleanup (next_attempt_at, id)"); + statements.add("CREATE INDEX IF NOT EXISTS idx_collaboration_attachment_cleanup_pending ON collaboration_attachment_cleanup (next_attempt_at, id) WHERE resolved_at IS NULL"); statements.add("CREATE INDEX IF NOT EXISTS idx_collaboration_mention_unread ON collaboration_mention (mentioned_user_id) WHERE notified_at IS NOT NULL AND read_at IS NULL"); return (String.join(";\n", statements) + ";\n"); } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/app/CollaborationHome.java b/src/main/java/com/kingsrook/qbits/collaboration/app/CollaborationHome.java index 7899579..086c03e 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/app/CollaborationHome.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/app/CollaborationHome.java @@ -108,7 +108,7 @@ public final class CollaborationHome .withPermissionRules(CollaborationPermissions.permissionRules()) .withWidgets(HOME_WIDGETS) .withSection(new QAppSection().withName("collaboration").withLabel("Collaboration").withIcon(new QIcon("forum")) - .withTables(List.of(CollaborationTables.COMMENT, CollaborationTables.ATTACHMENT, CollaborationTables.TAG, CollaborationTables.FIELD_DEFINITION, CollaborationTables.TASK, CollaborationTables.SHARE, CollaborationTables.ACCESS_REQUEST, CollaborationTables.RECORD_ASSIGNMENT, CollaborationTables.ACTIVITY, CollaborationTables.FEATURE_SETTING, CollaborationTables.USER))); + .withTables(List.of(CollaborationTables.COMMENT, CollaborationTables.ATTACHMENT, CollaborationTables.ATTACHMENT_CLEANUP, CollaborationTables.TAG, CollaborationTables.FIELD_DEFINITION, CollaborationTables.TASK, CollaborationTables.SHARE, CollaborationTables.ACCESS_REQUEST, CollaborationTables.RECORD_ASSIGNMENT, CollaborationTables.ACTIVITY, CollaborationTables.FEATURE_SETTING, CollaborationTables.USER))); app.setParentAppName(config.getParentAppName()); return (app); } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentCleanupQueue.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentCleanupQueue.java new file mode 100644 index 0000000..0889018 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentCleanupQueue.java @@ -0,0 +1,527 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.collaboration.attachments; + + +import java.io.Serializable; +import java.time.Instant; +import java.util.List; +import com.kingsrook.qbits.collaboration.CollaborationRecords; +import com.kingsrook.qbits.collaboration.CollaborationSecurity; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qbits.collaboration.permissions.CollaborationPermissions; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker; +import com.kingsrook.qqq.backend.core.actions.tables.DeleteAction; +import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; +import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; +import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction; +import com.kingsrook.qqq.backend.core.context.CapturedContext; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryLockMode; +import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules; +import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession; +import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleDispatcher; + + +/******************************************************************************* + ** Durable deletion intent, committed with metadata removal before deleting bytes. + *******************************************************************************/ +public class AttachmentCleanupQueue implements CustomPermissionChecker +{ + private final AttachmentStorage storage; + + + /******************************************************************************* + ** Metadata permission checks do not create a storage client. + *******************************************************************************/ + public AttachmentCleanupQueue() + { + this(new AttachmentStorage()); + } + + + /******************************************************************************* + ** Use the same provider boundary as the authorized attachment operation. + *******************************************************************************/ + AttachmentCleanupQueue(AttachmentStorage storage) + { + this.storage = storage; + } + + + /******************************************************************************* + ** Only admins may inspect recovery locations; no direct mutations are allowed. + *******************************************************************************/ + @Override + public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException + { + if(!(input instanceof QueryInput) && !(input instanceof com.kingsrook.qqq.backend.core.model.actions.tables.get.GetInput) + && !(input instanceof com.kingsrook.qqq.backend.core.model.actions.tables.count.CountInput)) + { + throw (new QPermissionDeniedException("Attachment cleanup records are service-managed.")); + } + requireOperator(); + } + + + /******************************************************************************* + ** The caller has already authorized the parent and attachment owner. + *******************************************************************************/ + void delete(QRecord row, boolean cleanup) throws QException + { + QBackendTransaction outer = QContext.getQBackendTransaction(); + if(outer != null) + { + QRecord pending = enqueueAndDelete(row, cleanup); + CapturedContext context = new CapturedContext(QContext.getQInstance(), QContext.getQSession()); + outer.addAfterCommitCallback(() -> + { + try + { + QContext.withTemporaryContext(context, () -> finish(pending)); + } + catch(QException e) + { + throw (new IllegalStateException("Attachment cleanup remains durably queued.", e)); + } + }); + return; + } + QRecord pending; + try(QBackendTransaction transaction = QBackendTransaction.openFor(new DeleteInput(CollaborationTables.ATTACHMENT))) + { + QContext.setTransaction(transaction); + try + { + pending = enqueueAndDelete(row, cleanup); + transaction.commit(); + } + catch(QException | RuntimeException failure) + { + transaction.rollback(); + throw (failure); + } + finally + { + QContext.clearTransaction(); + } + } + finish(pending); + } + + + /******************************************************************************* + ** Store the recovery locator in the same database transaction as row deletion. + *******************************************************************************/ + private QRecord enqueueAndDelete(QRecord row, boolean cleanup) throws QException + { + QRecord intent = new QRecord().withValue("attachmentId", row.getValue("id")) + .withValue("bucketName", row.getValue("bucketName")).withValue("storageKey", row.getValue("storageKey")); + InsertInput insert = new InsertInput(CollaborationTables.ATTACHMENT_CLEANUP).withRecords(List.of(intent)) + .withTransaction(QContext.getQBackendTransaction()); + insert.withPermissionBypass("Authorized attachment deletion records durable storage cleanup."); + List inserted = new InsertAction().execute(insert).getRecords(); + AttachmentRecords.checkErrors(inserted); + DeleteInput delete = new DeleteInput(CollaborationTables.ATTACHMENT).withPrimaryKeys(List.of(row.getValue("id"))) + .withTransaction(QContext.getQBackendTransaction()); + delete.setTableMetaData(cleanup ? CollaborationRecords.cleanupMetadata(CollaborationTables.ATTACHMENT) : AttachmentRecords.storageTable()); + delete.withPermissionBypass("Attachment deletion was authorized before recording cleanup."); + AttachmentRecords.checkErrors(new DeleteAction().execute(delete).getRecordsWithErrors()); + return (inserted.getFirst()); + } + + + /******************************************************************************* + ** Durable upload ownership requires real transactional row locking. + *******************************************************************************/ + static boolean supportsUploadOwnership() throws QException + { + var backend = QContext.getQInstance().getBackendForTable(CollaborationTables.ATTACHMENT_CLEANUP); + return (new QBackendModuleDispatcher().getQBackendModule(backend).supportsQueryLockMode(backend, QueryLockMode.FOR_UPDATE_SKIP_LOCKED)); + } + + + /******************************************************************************* + ** Commit the locator independently, then lock it before any provider write. + ** If a worker claimed it first, abort without uploading any bytes. + *******************************************************************************/ + QRecord beginUpload(String bucket, String key) throws QException + { + if(!supportsUploadOwnership()) + { + return (null); + } + QBackendTransaction transaction = QContext.getQBackendTransaction(); + if(transaction == null) + { + throw (new QException("Durable attachment uploads require an active transaction.")); + } + CapturedContext context = new CapturedContext(QContext.getQInstance(), QContext.getQSession()); + QRecord[] pending = new QRecord[1]; + QContext.withTemporaryContext(context, () -> + { + InsertInput insert = new InsertInput(CollaborationTables.ATTACHMENT_CLEANUP).withRecords(List.of( + new QRecord().withValue("bucketName", bucket).withValue("storageKey", key))); + insert.withPermissionBypass("Authorized upload records its immutable locator before external I/O."); + List inserted = new InsertAction().execute(insert).getRecords(); + AttachmentRecords.checkErrors(inserted); + pending[0] = inserted.getFirst(); + }); + if(lock(pending[0], transaction) == null) + { + throw (new QException("Upload cleanup intent was already claimed; retry the upload.")); + } + Runnable cleanup = () -> + { + try + { + QContext.withTemporaryContext(context, () -> finish(pending[0])); + } + catch(QException e) + { + throw (new IllegalStateException("Upload cleanup remains durably queued.", e)); + } + }; + transaction.addAfterRollbackCallback(cleanup); + transaction.addAfterCommitCallback(cleanup); + return (pending[0]); + } + + + /******************************************************************************* + ** Publish metadata and retire its upload intent in the same transaction. + *******************************************************************************/ + void completeUpload(QRecord pending) throws QException + { + if(pending != null) + { + DeleteInput delete = new DeleteInput(CollaborationTables.ATTACHMENT_CLEANUP).withPrimaryKey(pending.getValue("id")) + .withTransaction(QContext.getQBackendTransaction()); + delete.withPermissionBypass("Attachment metadata now owns the uploaded object in this transaction."); + AttachmentRecords.checkErrors(new DeleteAction().execute(delete).getRecordsWithErrors()); + } + } + + + /******************************************************************************* + ** Skip active owners without waiting or updating their locked intent. + *******************************************************************************/ + private QRecord lock(QRecord pending, QBackendTransaction transaction) throws QException + { + QueryInput query = new QueryInput(CollaborationTables.ATTACHMENT_CLEANUP).withTransaction(transaction) + .withLockMode(QueryLockMode.FOR_UPDATE_SKIP_LOCKED) + .withFilter(new QQueryFilter().withCriteria("id", QCriteriaOperator.EQUALS, pending.getValue("id"))); + query.withPermissionBypass("Internal attachment lifecycle owns this immutable cleanup locator."); + List rows = new QueryAction().execute(query).getRecords(); + return (rows.isEmpty() ? null : rows.getFirst()); + } + + + /******************************************************************************* + ** Distinguish an active owner from a committed row requiring delayed retry. + *******************************************************************************/ + private enum CleanupOutcome + { + COMPLETED, ACTIVE, REFERENCED, UNCERTAIN_UPLOAD + } + + + /******************************************************************************* + ** Hold the same row lock as an uploader until deletion and acknowledgement end. + *******************************************************************************/ + private CleanupOutcome finish(QRecord pending) throws QException + { + if(!supportsUploadOwnership()) + { + return (finishUnlocked(pending)); + } + CleanupOutcome[] outcome = { CleanupOutcome.ACTIVE }; + CapturedContext context = new CapturedContext(QContext.getQInstance(), QContext.getQSession()); + QContext.withTemporaryContext(context, () -> + { + try(QBackendTransaction transaction = QBackendTransaction.openFor(new DeleteInput(CollaborationTables.ATTACHMENT_CLEANUP))) + { + QContext.setTransaction(transaction); + try + { + QRecord locked = lock(pending, transaction); + if(locked != null) + { + outcome[0] = finishUnlocked(locked); + } + transaction.commit(); + } + catch(QException | RuntimeException failure) + { + transaction.rollback(); + throw (failure); + } + finally + { + QContext.clearTransaction(); + } + } + }); + return (outcome[0]); + } + + + /******************************************************************************* + ** Never erase bytes while metadata still exists, including nontransactional hosts. + ** Provider or queue deletion failure retains the intent for idempotent replay. + *******************************************************************************/ + private CleanupOutcome finishUnlocked(QRecord pending) throws QException + { + if(pending.getValue("resolvedAt") != null) + { + return (CleanupOutcome.COMPLETED); + } + if(!AttachmentRecords.query(new QQueryFilter() + .withCriteria("bucketName", QCriteriaOperator.EQUALS, pending.getValue("bucketName")) + .withCriteria("storageKey", QCriteriaOperator.EQUALS, pending.getValue("storageKey"))).isEmpty()) + { + return (CleanupOutcome.REFERENCED); + } + if(pending.getValue("attachmentId") == null) + { + if(!storage.cleanupUpload(pending.getValueString("bucketName"), pending.getValueString("storageKey"))) + { + return (CleanupOutcome.UNCERTAIN_UPLOAD); + } + } + else + { + storage.delete(pending.getValueString("bucketName"), pending.getValueString("storageKey")); + } + DeleteInput delete = new DeleteInput(CollaborationTables.ATTACHMENT_CLEANUP).withPrimaryKeys(List.of(pending.getValue("id"))).withTransaction(QContext.getQBackendTransaction()); + delete.withPermissionBypass("Storage cleanup completed; remove its durable intent."); + AttachmentRecords.checkErrors(new DeleteAction().execute(delete).getRecordsWithErrors()); + return (CleanupOutcome.COMPLETED); + } + + + /******************************************************************************* + ** Rollback compensation must survive a provider outage. Persist independently + ** of the failed transaction, then attempt cleanup with the original context. + *******************************************************************************/ + void compensateUpload(String bucket, String key) + { + CapturedContext context = new CapturedContext(QContext.getQInstance(), QContext.getQSession()); + try + { + QContext.withTemporaryContext(context, () -> + { + InsertInput insert = new InsertInput(CollaborationTables.ATTACHMENT_CLEANUP).withRecords(List.of( + new QRecord().withValue("bucketName", bucket).withValue("storageKey", key))); + insert.withPermissionBypass("Failed authorized upload retains its immutable storage locator for compensation."); + List inserted = new InsertAction().execute(insert).getRecords(); + AttachmentRecords.checkErrors(inserted); + finish(inserted.getFirst()); + }); + } + catch(QException e) + { + throw (new IllegalStateException("Upload compensation could not complete.", e)); + } + } + + + /******************************************************************************* + ** Bounded recovery from committed intents, independent of the original service. + *******************************************************************************/ + public int retryPending() throws QException + { + requireOperator(); + if(QContext.getQBackendTransaction() != null) + { + throw (new QException("Attachment cleanup must run outside a caller transaction.")); + } + QueryInput query = new QueryInput(CollaborationTables.ATTACHMENT_CLEANUP).withFilter(new QQueryFilter() + .withCriteria("resolvedAt", QCriteriaOperator.IS_BLANK).withSubFilter(new QQueryFilter() + .withBooleanOperator(QQueryFilter.BooleanOperator.OR) + .withCriteria("nextAttemptAt", QCriteriaOperator.IS_BLANK) + .withCriteria("nextAttemptAt", QCriteriaOperator.LESS_THAN_OR_EQUALS, Instant.now())) + .withOrderBy(new QFilterOrderBy("id")).withLimit(100)); + query.withPermissionBypass("Authorized operator retries committed storage cleanup."); + List pending = new QueryAction().execute(query).getRecords(); + int completed = 0; + QException failure = null; + for(QRecord row : pending) + { + try + { + CleanupOutcome outcome = finish(row); + if(outcome == CleanupOutcome.COMPLETED) + { + completed++; + } + else if(outcome == CleanupOutcome.REFERENCED || outcome == CleanupOutcome.UNCERTAIN_UPLOAD) + { + defer(row); + } + } + catch(QException | RuntimeException e) + { + try + { + defer(row); + } + catch(QException | RuntimeException deferFailure) + { + e.addSuppressed(deferFailure); + } + if(failure == null) + { + failure = new QException("Attachment cleanup remains pending; retry after the provider or database recovers.", e); + } + } + } + if(failure != null) + { + throw (failure); + } + return (completed); + } + + + /******************************************************************************* + ** Persist backoff so failing entries do not monopolize the next bounded batch. + *******************************************************************************/ + private void defer(QRecord pending) throws QException + { + UpdateInput update = new UpdateInput(CollaborationTables.ATTACHMENT_CLEANUP).withRecords(List.of( + new QRecord().withValue("id", pending.getValue("id")).withValue("nextAttemptAt", Instant.now().plusSeconds(60)))); + update.withPermissionBypass("Committed cleanup failed or is not yet safe; retain it for a later attempt."); + AttachmentRecords.checkErrors(new UpdateAction().execute(update).getRecords()); + } + + + /******************************************************************************* + ** An identified administrator attests provider settlement; absence or age alone + ** is not evidence. Keep the receipt on the service-managed row permanently. + *******************************************************************************/ + public void resolveUpload(Serializable id, String evidence) throws QException + { + String actor = requireSettlementOperator(); + validateEvidence(evidence); + if(id == null || QContext.getQBackendTransaction() != null || !supportsUploadOwnership()) + { + throw (new QException("Upload settlement requires a record ID and an independent locking transaction.")); + } + try(QBackendTransaction transaction = QBackendTransaction.openFor(new UpdateInput(CollaborationTables.ATTACHMENT_CLEANUP))) + { + QContext.setTransaction(transaction); + try + { + QRecord pending = lock(new QRecord().withValue("id", id), transaction); + if(pending == null) + { + throw (new QException("Cleanup record is active or no longer exists. Retry after its owner finishes.")); + } + if(pending.getValue("resolvedAt") != null) + { + transaction.commit(); + return; + } + if(pending.getValue("attachmentId") != null) + { + throw (new QException("Use normal cleanup retry for committed attachment deletions.")); + } + if(!AttachmentRecords.query(new QQueryFilter() + .withCriteria("bucketName", QCriteriaOperator.EQUALS, pending.getValue("bucketName")) + .withCriteria("storageKey", QCriteriaOperator.EQUALS, pending.getValue("storageKey"))).isEmpty()) + { + throw (new QException("An attachment still references this object; settlement is not allowed.")); + } + storage.delete(pending.getValueString("bucketName"), pending.getValueString("storageKey")); + UpdateInput update = new UpdateInput(CollaborationTables.ATTACHMENT_CLEANUP).withTransaction(transaction) + .withRecords(List.of(new QRecord().withValue("id", pending.getValue("id")) + .withValue("resolvedAt", Instant.now()).withValue("resolvedBy", actor) + .withValue("settlementEvidence", evidence.strip()))); + update.withPermissionBypass("Identified collaboration administrator records confirmed upload settlement after storage cleanup."); + AttachmentRecords.checkErrors(new UpdateAction().execute(update).getRecords()); + transaction.commit(); + } + catch(QException | RuntimeException failure) + { + transaction.rollback(); + throw (failure); + } + finally + { + QContext.clearTransaction(); + } + } + } + + + /******************************************************************************* + ** The form and service both reject missing or oversized evidence. + *******************************************************************************/ + static void validateEvidence(String evidence) throws QException + { + if(evidence == null || evidence.isBlank() || evidence.strip().length() > 2000) + { + throw (new QException("Record provider completion or cancellation evidence (1–2000 characters).")); + } + } + + + /******************************************************************************* + ** Settlement is a deliberate, attributable admin action, never scheduled work. + *******************************************************************************/ + static String requireSettlementOperator() throws QPermissionDeniedException + { + var session = QContext.getQSession(); + String actor = CollaborationSecurity.userId(session); + if(session == null || session instanceof QSystemUserSession || !session.hasPermission(CollaborationPermissions.ADMIN) + || actor == null || actor.isBlank() || actor.length() > 255) + { + throw (new QPermissionDeniedException("An identified collaboration administrator must confirm upload settlement.")); + } + return (actor); + } + + + /******************************************************************************* + ** Scheduled system work and explicit admin recovery share one authorization rule. + *******************************************************************************/ + static void requireOperator() throws QPermissionDeniedException + { + var session = QContext.getQSession(); + if(!(session instanceof QSystemUserSession) && (session == null || !session.hasPermission(CollaborationPermissions.ADMIN))) + { + throw (new QPermissionDeniedException("This needs the collaboration.admin permission.")); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentStorage.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentStorage.java index 73e7f50..f08ab44 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentStorage.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentStorage.java @@ -73,6 +73,18 @@ public class AttachmentStorage } } + /******************************************************************************* + ** Remove failed-upload bytes and report whether no earlier PUT can still finish. + ** Generic object storage cannot prove remote settlement after a timeout. Keep + ** its durable locator for later sweeps. Adapters may return true only with an + ** authoritative completion/cancellation guarantee, never from object absence. + *******************************************************************************/ + public boolean cleanupUpload(String name, String key) + { + delete(name, key); + return (false); + } + /******************************************************************************* ** Create a temporary link, never a persistent table value. *******************************************************************************/ diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentsService.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentsService.java index fc77167..95294f2 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentsService.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/AttachmentsService.java @@ -129,50 +129,11 @@ public class AttachmentsService } } String type = contentType == null || contentType.isBlank() ? "application/octet-stream" : contentType; - var transaction = QContext.getQBackendTransaction(); - if(transaction != null && !transaction.supportsRollbackCallbacks()) - { - throw (new QException("Attachment uploads require a transaction with confirmed rollback support.")); - } - try(InputStream upload = Files.newInputStream(temporary)) - { - storage.put(bucket, PutRequest.of(key, upload).withContentLength(size).withContentType(type).withContentDisposition("inline")); - } - if(transaction != null) - { - var context = new com.kingsrook.qqq.backend.core.context.CapturedContext(QContext.getQInstance(), QContext.getQSession()); - transaction.addAfterRollbackCallback(() -> QContext.withTemporaryContext(context, () -> storage.delete(bucket, key))); - } - boolean rowSaved = false; - try - { - QRecord row = new QRecord().withValue("tableName", tableName).withValue("recordId", id).withValue("fileName", name) - .withValue("contentType", type).withValue("sizeBytes", size).withValue("sha256", HexFormat.of().formatHex(digest.digest())) - .withValue("bucketName", bucket).withValue("storageKey", key).withValue("description", description) - .withValue("createdByUserId", user).withValue("parentDeleted", false); - InsertInput input = new InsertInput(CollaborationTables.ATTACHMENT).withTransaction(QContext.getQBackendTransaction()).withRecords(List.of(row)); - input.withPermissionBypass("AttachmentsService checked parent and attach permission before upload."); - List records = new InsertAction().execute(input).getRecords(); - AttachmentRecords.checkErrors(records); - rowSaved = true; - com.kingsrook.qbits.collaboration.activity.ActivityWriter.write(tableName, id, "FILE_ADDED", "File added", CollaborationTables.ATTACHMENT, records.getFirst().getValueString("id")); - return (entry(records.getFirst())); - } - catch(QException | RuntimeException failure) - { - try - { - if(!rowSaved) - { - storage.delete(bucket, key); - } - } - catch(RuntimeException cleanupFailure) - { - failure.addSuppressed(cleanupFailure); - } - throw (failure); - } + QRecord row = new QRecord().withValue("tableName", tableName).withValue("recordId", id).withValue("fileName", name) + .withValue("contentType", type).withValue("sizeBytes", size).withValue("sha256", HexFormat.of().formatHex(digest.digest())) + .withValue("bucketName", bucket).withValue("storageKey", key).withValue("description", description) + .withValue("createdByUserId", user).withValue("parentDeleted", false); + return (storeUpload(row, temporary)); } catch(IOException | NoSuchAlgorithmException e) { @@ -194,6 +155,107 @@ public class AttachmentsService } } + /******************************************************************************* + ** On transactional locking backends, own a transaction only after spooling + ** the bounded input. An existing outer transaction retains the upload lock. + *******************************************************************************/ + private AttachmentEntry storeUpload(QRecord row, Path temporary) throws QException, IOException + { + var transaction = QContext.getQBackendTransaction(); + if(transaction == null && AttachmentCleanupQueue.supportsUploadOwnership()) + { + try(var owned = com.kingsrook.qqq.backend.core.actions.QBackendTransaction.openFor(new InsertInput(CollaborationTables.ATTACHMENT))) + { + QContext.setTransaction(owned); + try + { + AttachmentEntry result = storeUpload(row, temporary); + owned.commit(); + return (result); + } + catch(QException | IOException | RuntimeException failure) + { + try + { + owned.rollback(); + } + catch(QException rollbackFailure) + { + failure.addSuppressed(rollbackFailure); + } + throw (failure); + } + finally + { + QContext.clearTransaction(); + } + } + } + if(transaction != null && !transaction.supportsRollbackCallbacks()) + { + throw (new QException("Attachment uploads require a transaction with confirmed rollback support.")); + } + String bucket = row.getValueString("bucketName"); + String key = row.getValueString("storageKey"); + AttachmentCleanupQueue cleanup = new AttachmentCleanupQueue(storage); + QRecord pending = cleanup.beginUpload(bucket, key); + try(InputStream upload = Files.newInputStream(temporary)) + { + storage.put(bucket, PutRequest.of(key, upload).withContentLength(row.getValueLong("sizeBytes")) + .withContentType(row.getValueString("contentType")).withContentDisposition("inline")); + } + catch(IOException | RuntimeException failure) + { + if(pending == null) + { + try + { + cleanup.compensateUpload(bucket, key); + } + catch(RuntimeException cleanupFailure) + { + failure.addSuppressed(cleanupFailure); + } + } + throw (failure); + } + if(transaction != null && pending == null) + { + var context = new com.kingsrook.qqq.backend.core.context.CapturedContext(QContext.getQInstance(), QContext.getQSession()); + transaction.addAfterRollbackCallback(() -> QContext.withTemporaryContext(context, + () -> cleanup.compensateUpload(bucket, key))); + } + boolean rowSaved = false; + try + { + InsertInput input = new InsertInput(CollaborationTables.ATTACHMENT).withTransaction(QContext.getQBackendTransaction()).withRecords(List.of(row)); + input.withPermissionBypass("AttachmentsService checked parent and attach permission before upload."); + List records = new InsertAction().execute(input).getRecords(); + AttachmentRecords.checkErrors(records); + rowSaved = true; + cleanup.completeUpload(pending); + com.kingsrook.qbits.collaboration.activity.ActivityWriter.write(row.getValueString("tableName"), row.getValueString("recordId"), + "FILE_ADDED", "File added", CollaborationTables.ATTACHMENT, records.getFirst().getValueString("id")); + return (entry(records.getFirst())); + } + catch(QException | RuntimeException failure) + { + if(!rowSaved && pending == null) + { + try + { + cleanup.compensateUpload(bucket, key); + } + catch(RuntimeException cleanupFailure) + { + failure.addSuppressed(cleanupFailure); + } + } + throw (failure); + } + } + + /******************************************************************************* ** Newest first, independently of collaboration.view. *******************************************************************************/ @@ -301,7 +363,7 @@ public class AttachmentsService } /******************************************************************************* - ** Delete storage first so a provider failure leaves a retryable row. + ** Atomically remove metadata and retain a durable storage cleanup request. *******************************************************************************/ void deleteStoredRow(QRecord row) throws QException { @@ -313,27 +375,7 @@ public class AttachmentsService *******************************************************************************/ void deleteStoredRow(QRecord row, boolean cleanup) throws QException { - if(QContext.getQBackendTransaction() != null) - { - var context = new com.kingsrook.qqq.backend.core.context.CapturedContext(QContext.getQInstance(), QContext.getQSession()); - QContext.getQBackendTransaction().addAfterCommitCallback(() -> - { - try - { - QContext.withTemporaryContext(context, () -> deleteStoredRow(row, cleanup)); - } - catch(QException e) - { - throw (new IllegalStateException("Attachment deletion requires retry", e)); - } - }); - return; - } - storage.delete(row.getValueString("bucketName"), row.getValueString("storageKey")); - DeleteInput input = new DeleteInput(CollaborationTables.ATTACHMENT).withTransaction(QContext.getQBackendTransaction()).withPrimaryKeys(List.of(row.getValue("id"))); - input.setTableMetaData(cleanup ? com.kingsrook.qbits.collaboration.CollaborationRecords.cleanupMetadata(CollaborationTables.ATTACHMENT) : AttachmentRecords.storageTable()); - input.withPermissionBypass("Attachment service authorized deletion, or trusted parent cleanup selected this row."); - AttachmentRecords.checkErrors(new DeleteAction().execute(input).getRecordsWithErrors()); + new AttachmentCleanupQueue(storage).delete(row, cleanup); } /******************************************************************************* diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/ResolveAttachmentUploadProcess.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/ResolveAttachmentUploadProcess.java new file mode 100644 index 0000000..3106416 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/ResolveAttachmentUploadProcess.java @@ -0,0 +1,227 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + + +import java.io.Serializable; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.Map; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine; +import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput; +import com.kingsrook.qqq.backend.core.model.actions.processes.Status; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QComponentType; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendComponentMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QFrontendStepMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractLoadStep; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractTransformStep; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ExtractViaQueryStep; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ProcessSummaryProviderInterface; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess; + + +/******************************************************************************* + ** Explicit provider settlement through the standard table action and Review flow. + *******************************************************************************/ +public class ResolveAttachmentUploadProcess implements CustomPermissionChecker +{ + public static final String NAME = "collaborationResolveAttachmentUpload"; + public static final String EVIDENCE = "settlementEvidence"; + + + /******************************************************************************* + ** Administrative table action, never a navigation entry or scheduled process. + *******************************************************************************/ + public static QProcessMetaData metadata() + { + Map values = new HashMap<>(); + values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, CollaborationTables.ATTACHMENT_CLEANUP); + values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, CollaborationTables.ATTACHMENT_CLEANUP); + values.put(StreamedETLWithFrontendProcess.FIELD_SUPPORTS_FULL_VALIDATION, true); + values.put(StreamedETLWithFrontendProcess.FIELD_DO_FULL_VALIDATION, true); + values.put(StreamedETLWithFrontendProcess.FIELD_PREVIEW_MESSAGE, + "Confirm that the provider has completed or cancelled every selected upload. Object absence or elapsed time alone is not proof. Current bytes will be deleted; the settlement receipt will be retained."); + return (StreamedETLWithFrontendProcess.defineProcessMetaData(ExtractViaQueryStep.class, Review.class, Apply.class, values) + .withName(NAME).withLabel("Resolve settled uploads").withTableName(CollaborationTables.ATTACHMENT_CLEANUP).withIcon(new QIcon("check")) + .withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.NOT_PROTECTED) + .withCustomPermissionChecker(new QCodeReference(ResolveAttachmentUploadProcess.class))) + .withStep(0, new QFrontendStepMetaData().withName("evidence").withLabel("Confirm provider settlement") + .withFormField(new QFieldMetaData(EVIDENCE, QFieldType.STRING).withLabel("Provider completion or cancellation evidence") + .withMaxLength(2000).withIsRequired(true)) + .withComponent(new QFrontendComponentMetaData().withType(QComponentType.EDIT_FORM)))); + } + + + /******************************************************************************* + ** Guard direct process invocation as well as menu visibility. + *******************************************************************************/ + @Override + public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException + { + AttachmentCleanupQueue.requireSettlementOperator(); + } + + + /******************************************************************************* + ** Shared standard summary language for Review and Result. + *******************************************************************************/ + private static ProcessSummaryLine line(Status status, String future, String past) + { + ProcessSummaryLine line = new ProcessSummaryLine(status); + line.setSingularFutureMessage(future); + line.setPluralFutureMessage(future); + line.setSingularPastMessage(past); + line.setPluralPastMessage(past); + return (line); + } + + + /******************************************************************************* + ** Review never changes provider state or records a completed settlement. + *******************************************************************************/ + public static class Review extends AbstractTransformStep implements ProcessSummaryProviderInterface + { + private final ProcessSummaryLine selected = line(Status.OK, "uploads will be resolved", "uploads selected"); + private final ProcessSummaryLine skipped = line(Status.WARNING, "resolved records or committed deletions will be skipped", "records skipped"); + + + /******************************************************************************* + ** Require evidence before allowing the administrator to review selected rows. + *******************************************************************************/ + @Override + public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + AttachmentCleanupQueue.requireSettlementOperator(); + AttachmentCleanupQueue.validateEvidence(input.getValueString(EVIDENCE)); + for(QRecord row : input.getRecords()) + { + if(row.getValue("attachmentId") != null || row.getValue("resolvedAt") != null) + { + skipped.incrementCountAndAddPrimaryKey(row.getValue("id")); + } + else + { + output.getRecords().add(row); + selected.incrementCountAndAddPrimaryKey(row.getValue("id")); + } + } + } + + + /******************************************************************************* + ** Keep streamed totals across pages. + *******************************************************************************/ + @Override + public ArrayList getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen) + { + ArrayList lines = new ArrayList<>(); + selected.addSelfToListIfAnyCount(lines); + skipped.addSelfToListIfAnyCount(lines); + return (lines); + } + } + + + /******************************************************************************* + ** Apply checks current authorization, row ownership and references again. + *******************************************************************************/ + public static class Apply extends AbstractLoadStep implements ProcessSummaryProviderInterface + { + private final AttachmentCleanupQueue queue; + private final ProcessSummaryLine done = line(Status.OK, "uploads will be resolved", "uploads resolved"); + private final ProcessSummaryLine failed = line(Status.ERROR, "uploads cannot be resolved", "uploads unresolved; inspect and retry"); + + + /******************************************************************************* + ** Construction is also used during context-free metadata validation. + *******************************************************************************/ + public Apply() + { + this(new AttachmentCleanupQueue()); + } + + + /******************************************************************************* + ** Use the same service with a fixture storage provider in process tests. + *******************************************************************************/ + public Apply(AttachmentCleanupQueue queue) + { + this.queue = queue; + } + + + /******************************************************************************* + ** Each locator has its own transaction and immutable settlement receipt. + *******************************************************************************/ + @Override + public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + AttachmentCleanupQueue.requireSettlementOperator(); + AttachmentCleanupQueue.validateEvidence(input.getValueString(EVIDENCE)); + for(QRecord row : input.getRecords()) + { + try + { + queue.resolveUpload(row.getValue("id"), input.getValueString(EVIDENCE)); + done.incrementCountAndAddPrimaryKey(row.getValue("id")); + } + catch(QException | RuntimeException failure) + { + failed.incrementCountAndAddPrimaryKey(row.getValue("id")); + } + } + } + + + /******************************************************************************* + ** Report actual application results and retain review exclusions. + *******************************************************************************/ + @Override + public ArrayList getProcessSummary(RunBackendStepOutput output, boolean isForResultScreen) + { + ArrayList lines = new ArrayList<>(); + done.addSelfToListIfAnyCount(lines); + failed.addSelfToListIfAnyCount(lines); + if(getTransformStep() instanceof Review review) + { + review.skipped.addSelfToListIfAnyCount(lines); + } + return (lines); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/attachments/RetryAttachmentCleanupProcess.java b/src/main/java/com/kingsrook/qbits/collaboration/attachments/RetryAttachmentCleanupProcess.java new file mode 100644 index 0000000..c48a9bd --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/attachments/RetryAttachmentCleanupProcess.java @@ -0,0 +1,99 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.attachments; + + +import com.kingsrook.qbits.collaboration.CollaborationQBitConfig; +import com.kingsrook.qbits.collaboration.permissions.CollaborationPermissions; +import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker; +import com.kingsrook.qqq.backend.core.actions.processes.BackendStep; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.QScheduleMetaData; +import com.kingsrook.qqq.backend.core.model.session.QSession; +import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession; + + +/******************************************************************************* + ** Scheduled durable attachment cleanup. Runs with its caller's session, which must be a + ** scheduler system session or collaboration admin; never elevates a user. + *******************************************************************************/ +public class RetryAttachmentCleanupProcess implements BackendStep, CustomPermissionChecker +{ + public static final String NAME = "collaborationRetryAttachmentCleanup"; + + + + /******************************************************************************* + ** Always register the backend-only process; scheduling is opt-in. + *******************************************************************************/ + public static QProcessMetaData metadata(CollaborationQBitConfig config) + { + QProcessMetaData process = new QProcessMetaData().withName(NAME).withLabel("Retry attachment cleanup").withIsHidden(true).withIcon(new QIcon("history")) + .withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.HAS_ACCESS_PERMISSION) + .withPermissionBaseName("collaboration").withCustomPermissionChecker(new QCodeReference(RetryAttachmentCleanupProcess.class))) + .withStep(new QBackendStepMetaData().withName("run").withCode(new QCodeReference(RetryAttachmentCleanupProcess.class))); + if(config.getSchedulerName() != null) + { + process.withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName()).withRepeatSeconds(60).withInitialDelaySeconds(60)); + } + return (process); + } + + + + /******************************************************************************* + ** Hidden is a presentation setting, so explicitly guard execution as well. + *******************************************************************************/ + @Override + public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException + { + QSession session = QContext.getQSession(); + if(!(session instanceof QSystemUserSession) && (session == null || !session.hasPermission(CollaborationPermissions.ADMIN))) + { + throw (new QPermissionDeniedException("This needs the collaboration.admin permission.")); + } + } + + + + /******************************************************************************* + ** Retry committed cleanup intents after provider or process recovery. + *******************************************************************************/ + @Override + public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + checkPermissionsThrowing(input, null); + new AttachmentCleanupQueue().retryPending(); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldStorage.java b/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldStorage.java index cfc5959..f70b8b7 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldStorage.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldStorage.java @@ -83,7 +83,7 @@ public final class CustomFieldStorage } if(!postgres(table)) { - return (new Availability(false, "Backend does not support JsonPathStorage; EAV fields are read-only filters (values are not selected).")); + return (new Availability(false, "Backend does not support JsonPathStorage; EAV fields support read-only values and filters.")); } try(Connection connection = connection(table); var statement = connection.createStatement()) { @@ -92,7 +92,7 @@ public final class CustomFieldStorage } catch(SQLException | IllegalArgumentException e) { - return (new Availability(false, "JSON column unavailable; EAV fields are read-only filters. " + e.getMessage())); + return (new Availability(false, "JSON column unavailable; EAV fields support read-only values and filters. " + e.getMessage())); } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldValueProvider.java b/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldValueProvider.java new file mode 100644 index 0000000..b974680 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldValueProvider.java @@ -0,0 +1,101 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.customfields; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.collaboration.CollaborationRecords; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; +import com.kingsrook.qqq.backend.core.actions.tables.query.VirtualFieldValueProviderInterface; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordReferenceTableMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.core.utils.CollectionUtils; +import com.kingsrook.qqq.backend.core.utils.ValueUtils; + +/******************************************************************************* + ** Supply EAV values for the parent records selected by the core query pipeline. + ** Do not re-read parents here: that would recursively execute this provider. + *******************************************************************************/ +public class CustomFieldValueProvider implements VirtualFieldValueProviderInterface +{ + /******************************************************************************* + ** Bound each lookup and retain the caller's transaction through service storage. + *******************************************************************************/ + @Override + public void fill(QTableMetaData table, String fieldName, List records, QueryInput input) throws QException + { + String name = fieldName.substring(3); + var definitions = CollaborationRecords.query(CollaborationTables.FIELD_DEFINITION, new QQueryFilter() + .withCriteria("tableName", QCriteriaOperator.EQUALS, table.getName()).withCriteria("name", QCriteriaOperator.EQUALS, name) + .withCriteria("status", QCriteriaOperator.EQUALS, "ACTIVE"), input.getTransaction()); + if(definitions.size() != 1) + { + throw (new QException("Custom field is unavailable")); + } + String column = FieldCriteriaExpander.valueColumn(definitions.getFirst().getValueString("type")); + for(List page : CollectionUtils.getPages(records, 500)) + { + List ids = page.stream().map(record -> String.valueOf(record.getValue(table.getPrimaryKeyField()))).distinct().toList(); + Map values = new HashMap<>(); + for(QRecord value : valuesForSelectedParents(new QQueryFilter() + .withCriteria("tableName", QCriteriaOperator.EQUALS, table.getName()).withCriteria("fieldName", QCriteriaOperator.EQUALS, name) + .withCriteria("parentDeleted", QCriteriaOperator.EQUALS, false).withCriteria("recordId", QCriteriaOperator.IN, ids), input)) + { + if(values.put(value.getValueString("recordId"), value) != null) + { + throw (new QException("Duplicate custom field values")); + } + } + for(QRecord record : page) + { + QRecord value = values.get(String.valueOf(record.getValue(table.getPrimaryKeyField()))); + record.setValue(fieldName, value == null ? null : ValueUtils.getValueAsFieldType(table.getVirtualField(fieldName).getType(), value.getValue(column))); + } + } + } + + /******************************************************************************* + ** The enclosing query already selected these parents under its access policy. + ** Remove only recursive parent verification on a private child metadata clone; + ** preserve child locks and never install this metadata on the instance. + *******************************************************************************/ + private List valuesForSelectedParents(QQueryFilter filter, QueryInput origin) throws QException + { + QTableMetaData metadata = QContext.getQInstance().getTable(CollaborationTables.FIELD_VALUE).clone(); + metadata.withCustomizers(null); + metadata.getSupplementalMetaData().remove(RecordReferenceTableMetaData.NAME); + QueryInput input = new QueryInput(CollaborationTables.FIELD_VALUE).withFilter(filter) + .withShouldFetchHeavyFields(true).withTransaction(origin.getTransaction()); + input.setTableMetaData(metadata); + input.withPermissionBypass("EAV provider reads only parents selected by the enclosing query."); + return (new QueryAction().execute(input).getRecords()); + } + +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldsProvider.java b/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldsProvider.java index 96df5f6..91ea2a7 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldsProvider.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/customfields/CustomFieldsProvider.java @@ -122,7 +122,8 @@ public class CustomFieldsProvider implements RuntimeMetaDataProviderInterface String name = "cf_" + definition.getValueString("name"); QFieldType type = type(definition.getValueString("type")); QFieldMetaData field = json ? new QFieldMetaData(name, type).withBackendStorage(new JsonPathStorage().withColumnName("qqq_custom_fields").withKey(definition.getValueString("name"))) - : new QVirtualFieldMetaData(name, type).withIsEditable(false).withIsQueryCriteria(true).withIsQuerySelectable(false) + : new QVirtualFieldMetaData(name, type).withIsEditable(false).withIsQueryCriteria(true).withIsQuerySelectable(true) + .withValueProvider(new QCodeReference(CustomFieldValueProvider.class)) .withCriteriaExpander(new QCodeReference(FieldCriteriaExpander.class)); field.withLabel(definition.getValueString("label")).withIsRequired(json && Boolean.TRUE.equals(definition.getValueBoolean("required"))) .withMaxLength(definition.getValueInteger("maxLength")); diff --git a/src/main/java/com/kingsrook/qbits/collaboration/model/CollaborationTables.java b/src/main/java/com/kingsrook/qbits/collaboration/model/CollaborationTables.java index 4968f3f..b1ddaf4 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/model/CollaborationTables.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/model/CollaborationTables.java @@ -80,6 +80,7 @@ public final class CollaborationTables public static final String ACTIVITY = "collaboration_activity"; public static final String FOLLOW_DIGEST = "collaboration_follow_digest"; public static final String ATTACHMENT = "collaboration_attachment"; + public static final String ATTACHMENT_CLEANUP = "collaboration_attachment_cleanup"; public static final String FEATURE_SETTING = "collaboration_feature_setting"; public static final String FAVORITE = "collaboration_favorite"; public static final String RECENT = "collaboration_recent"; @@ -202,6 +203,21 @@ public final class CollaborationTables tables.addAll(tagTables(config)); tables.addAll(workTables(config)); tables.addAll(customFieldTables(config)); + tables.add(base(config, ATTACHMENT_CLEANUP, "Attachment Cleanup", "delete") + .withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.NOT_PROTECTED) + .withCustomPermissionChecker(new QCodeReference(com.kingsrook.qbits.collaboration.attachments.AttachmentCleanupQueue.class))) + .withRecordLabelFormatAndFields("Cleanup %s", "id") + .withField(new QFieldMetaData("attachmentId", QFieldType.INTEGER)) + .withField(string("bucketName", "Bucket", 255).withIsRequired(true)) + .withField(string("storageKey", "Storage Key", 1024).withIsRequired(true)) + .withField(new QFieldMetaData("nextAttemptAt", QFieldType.DATE_TIME).withLabel("Next Attempt")) + .withField(new QFieldMetaData("resolvedAt", QFieldType.DATE_TIME).withLabel("Resolved At")) + .withField(string("resolvedBy", "Resolved By", 255)) + .withField(string("settlementEvidence", "Provider Settlement Evidence", 2000)) + .withSection(new QFieldSection("cleanup", new QIcon("delete"), Tier.T1, + List.of("id", "attachmentId", "bucketName", "storageKey", "nextAttemptAt")).withLabel("Cleanup")) + .withSection(new QFieldSection("settlement", new QIcon("check"), Tier.T2, + List.of("resolvedAt", "resolvedBy", "settlementEvidence")).withLabel("Operator Settlement"))); for(QTableMetaData table : tables) { table.withSection(new QFieldSection("dates", new QIcon("calendar_today"), Tier.T3, List.of("createDate", "modifyDate")).withLabel("Dates")); diff --git a/src/main/java/com/kingsrook/qbits/collaboration/mywork/MyWorkService.java b/src/main/java/com/kingsrook/qbits/collaboration/mywork/MyWorkService.java index 6697ba5..47bc446 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/mywork/MyWorkService.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/mywork/MyWorkService.java @@ -23,18 +23,19 @@ package com.kingsrook.qbits.collaboration.mywork; import java.time.Instant; import java.util.ArrayList; +import java.util.Comparator; +import java.util.HashMap; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import com.kingsrook.qbits.collaboration.CollaborationRecords; import com.kingsrook.qbits.collaboration.PersonalRecords; -import com.kingsrook.qbits.collaboration.assignment.AssignmentService; import com.kingsrook.qbits.collaboration.model.CollaborationTables; import com.kingsrook.qbits.collaboration.security.ParentRecordAccess; -import com.kingsrook.qbits.collaboration.tasks.TasksService; import com.kingsrook.qqq.backend.core.exceptions.QException; import com.kingsrook.qqq.backend.core.exceptions.QNotFoundException; import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy; import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; import com.kingsrook.qqq.backend.core.model.data.QRecord; @@ -43,18 +44,94 @@ import com.kingsrook.qqq.backend.core.model.data.QRecord; *******************************************************************************/ public class MyWorkService { + private static final int PAGE_SIZE = 200; + private static final Comparator TASK_ORDER = Comparator.comparing((QRecord row) -> row.getValueInstant("dueAt"), Comparator.nullsLast(Comparator.naturalOrder())) + .thenComparing(row -> row.getValueInteger("id")); + /******************************************************************************* ** Open task count and first ten, assigned record counts, and unread mentions. *******************************************************************************/ public Summary summary() throws QException { - List tasks = new TasksService().mineRows("OPEN"); + String user = PersonalRecords.currentUserId(); + List tasks = new ArrayList<>(); + long taskCount = scanVisible(CollaborationTables.TASK, new QQueryFilter() + .withCriteria("assigneeUserId", QCriteriaOperator.EQUALS, user) + .withCriteria("status", QCriteriaOperator.EQUALS, "OPEN"), row -> + { + tasks.add(row); + tasks.sort(TASK_ORDER); + if(tasks.size() > 10) + { + tasks.removeLast(); + } + }); Map assignments = new LinkedHashMap<>(); - for(QRecord row : new AssignmentService().mineRows(null, true)) + scanVisible(CollaborationTables.RECORD_ASSIGNMENT, new QQueryFilter() + .withCriteria("assigneeUserId", QCriteriaOperator.EQUALS, user), + row -> assignments.merge(row.getValueString("tableName"), 1L, Long::sum)); + long mentions = scanVisible(CollaborationTables.MENTION, new QQueryFilter() + .withCriteria("mentionedUserId", QCriteriaOperator.EQUALS, user) + .withCriteria("notifiedAt", QCriteriaOperator.IS_NOT_BLANK).withCriteria("readAt", QCriteriaOperator.IS_BLANK), row -> + { + // Only the visible count is needed for mentions. + }); + return (new Summary(taskCount, List.copyOf(tasks), assignments, mentions)); + } + + /******************************************************************************* + ** Bound retained rows and visibility decisions to one page. A starting high + ** watermark prevents concurrent inserts from extending this scan indefinitely. + ** Counts remain exact for visible rows scanned; no cross-request auth cache. + *******************************************************************************/ + private long scanVisible(String table, QQueryFilter filter, java.util.function.Consumer consumer) throws QException + { + filter.withCriteria("parentDeleted", QCriteriaOperator.EQUALS, false); + List last = CollaborationRecords.query(table, filter.clone().withOrderBy(new QFilterOrderBy("id", false)).withLimit(1)); + if(last.isEmpty()) { - assignments.merge(row.getValueString("tableName"), 1L, Long::sum); + return (0); + } + int maximum = last.getFirst().getValueInteger("id"); + Integer cursor = null; + long count = 0; + while(true) + { + QQueryFilter pageFilter = filter.clone().withCriteria("id", QCriteriaOperator.LESS_THAN_OR_EQUALS, maximum) + .withOrderBy(new QFilterOrderBy("id")).withLimit(PAGE_SIZE); + if(cursor != null) + { + pageFilter.withCriteria("id", QCriteriaOperator.GREATER_THAN, cursor); + } + List rows = CollaborationRecords.query(table, pageFilter); + Map, Boolean> parents = new HashMap<>(); + for(QRecord row : rows) + { + List parent = java.util.Arrays.asList(row.getValueString("tableName"), row.getValueString("recordId")); + if(!parents.containsKey(parent)) + { + try + { + ParentRecordAccess.requireVisibleParent(parent.get(0), parent.get(1)); + parents.put(parent, true); + } + catch(QNotFoundException expected) + { + parents.put(parent, false); + } + } + if(parents.get(parent)) + { + consumer.accept(row); + count++; + } + } + if(rows.size() < PAGE_SIZE || rows.getLast().getValueInteger("id") >= maximum) + { + return (count); + } + cursor = rows.getLast().getValueInteger("id"); } - return (new Summary((long) tasks.size(), tasks.stream().limit(10).toList(), assignments, (long) unread().size())); } /******************************************************************************* diff --git a/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceRecoveringException.java b/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceRecoveringException.java new file mode 100644 index 0000000..11d4a58 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/presence/PresenceRecoveringException.java @@ -0,0 +1,36 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.presence; + +/******************************************************************************* + ** Shared presence is rebuilding after unknown or lost Redis state. + *******************************************************************************/ +public final class PresenceRecoveringException extends IllegalStateException +{ + /*************************************************************************** + ** Never include caller identifiers or stored data in this message. + ***************************************************************************/ + public PresenceRecoveringException() + { + super("Presence is recovering; active viewers are rejoining"); + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceAudience.java b/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceAudience.java index e767049..824c4b4 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceAudience.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceAudience.java @@ -49,6 +49,6 @@ public class RedisPresenceAudience implements RealtimeAudienceProvider public static RedisNamespace namespace(RedisPresenceConfig config) throws Exception { var scope = QCodeLoader.getAdHoc(PresenceScopeProvider.class, config.getScopeProvider()).tenantScope(); - return (new RedisNamespace(config.getDeployment(), config.getApplication(), scope, "presence")); + return (new RedisNamespace(config.getDeployment(), config.getApplication(), scope, "presence:" + config.getRecoveryGeneration())); } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceBackend.java b/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceBackend.java index 199428f..2bdb5cf 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceBackend.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceBackend.java @@ -145,6 +145,10 @@ public class RedisPresenceBackend implements PresenceBackend { return (store.snapshot(table, pk)); } + catch(PresenceRecoveringException recovering) + { + throw (new QServiceUnavailableException("Presence is recovering; active viewers are rejoining. Please retry.")); + } catch(RuntimeException failure) { throw (unavailable()); diff --git a/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceConfig.java b/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceConfig.java index 8724a8d..b761c6a 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceConfig.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceConfig.java @@ -35,6 +35,7 @@ public class RedisPresenceConfig implements Serializable private String application = null; private QCodeReference scopeProvider = null; private Integer capacity = 2000; + private String recoveryGeneration = "initial"; /*************************************************************************** ** Inspect configuration without creating a runtime pool or local registry. @@ -61,6 +62,10 @@ public class RedisPresenceConfig implements Serializable { errors.add("Redis presence capacity must be between 1 and 10000"); } + if(recoveryGeneration == null || !recoveryGeneration.matches("[A-Za-z0-9_-]{1,64}")) + { + errors.add("Redis presence recovery generation must contain 1 to 64 letters, digits, underscores or hyphens"); + } } /*************************************************************************** @@ -147,4 +152,23 @@ public class RedisPresenceConfig implements Serializable capacity = value; return (this); } + + + /*************************************************************************** + ** Host-owned generation, stored outside Redis and changed after restoring data. + ***************************************************************************/ + public String getRecoveryGeneration() + { + return (recoveryGeneration); + } + + + /*************************************************************************** + ** All nodes must use the same new generation before maintenance traffic resumes. + ***************************************************************************/ + public RedisPresenceConfig withRecoveryGeneration(String value) + { + recoveryGeneration = value; + return (this); + } } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceStore.java b/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceStore.java index 9a8c208..69db925 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceStore.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/presence/RedisPresenceStore.java @@ -44,8 +44,8 @@ import com.kingsrook.qqq.backend.core.utils.JsonUtils; *******************************************************************************/ public final class RedisPresenceStore { - private static final RedisScript MUTATE = new RedisScript(resource("redis-mutate.lua")); - private static final RedisScript SNAPSHOT = new RedisScript(resource("redis-snapshot.lua")); + private static final RedisScript MUTATE = new RedisScript(resource("redis-recovery.lua") + "\n" + resource("redis-mutate.lua")); + private static final RedisScript SNAPSHOT = new RedisScript(resource("redis-recovery.lua") + "\n" + resource("redis-snapshot.lua")); private final RedisScriptExecutor client; private final RedisNamespace namespace; private final long ttlMillis; @@ -123,7 +123,12 @@ public final class RedisPresenceStore public List snapshot(String table, String pk) { List keys = roomKeys(table, pk, true); - Object result = client.execute(SNAPSHOT, keys, List.of(pk == null ? "" : ":" + encode(pk, 1024))); + Object result = client.execute(SNAPSHOT, keys, List.of(pk == null ? "" : ":" + encode(pk, 1024), + Long.toString(ttlMillis), java.util.UUID.randomUUID().toString())); + if(result instanceof Number number && number.longValue() == -2) + { + throw (new PresenceRecoveringException()); + } Map users = new TreeMap<>(); for(Object value : (List) result) { @@ -160,9 +165,10 @@ public final class RedisPresenceStore { throw (new IllegalArgumentException("Presence sequence must be a nonnegative safe integer")); } - Stored value = new Stored(user, displayName, mode, pk == null ? null : field, 0, sequence, departed, false, false); + Stored value = new Stored(user, displayName, mode, pk == null ? null : field, 0, sequence, departed, false, false, null); Object result = client.execute(MUTATE, roomKeys(table, pk, true), List.of(member, JsonUtils.toJson(value), - sequence == null ? "" : sequence.toString(), departed ? "1" : "0", Long.toString(ttlMillis), Integer.toString(capacity), pk == null ? "1" : "0", operation)); + sequence == null ? "" : sequence.toString(), departed ? "1" : "0", Long.toString(ttlMillis), Integer.toString(capacity), pk == null ? "1" : "0", operation, + java.util.UUID.randomUUID().toString())); if(((Number) result).longValue() == -1) { throw (new PresenceLeaseExpiredException()); @@ -175,7 +181,7 @@ public final class RedisPresenceStore ***************************************************************************/ private List roomKeys(String table, String pk, boolean includeTable) { - String base = "qqq:presence:{" + digest("SHA-256", namespace.key(table)) + "}"; + String base = "qqq:presence:v2:{" + digest("SHA-256", namespace.key(table)) + "}"; List keys = new ArrayList<>(); if(pk != null) { @@ -188,6 +194,7 @@ public final class RedisPresenceStore keys.add(base + ":table:expiry"); keys.add(base + ":table:state"); } + keys.add(base + ":recovery"); return (keys); } @@ -240,7 +247,7 @@ public final class RedisPresenceStore /*************************************************************************** ** Internal lease data, never returned directly to a browser. ***************************************************************************/ - public record Stored(String userId, String displayName, String mode, String fieldName, long at, Long sequence, boolean departed, boolean pending, boolean leased) + public record Stored(String userId, String displayName, String mode, String fieldName, long at, Long sequence, boolean departed, boolean pending, boolean leased, String generation) { } diff --git a/src/main/java/com/kingsrook/qbits/collaboration/tags/TagValueProvider.java b/src/main/java/com/kingsrook/qbits/collaboration/tags/TagValueProvider.java new file mode 100644 index 0000000..4bccd84 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/collaboration/tags/TagValueProvider.java @@ -0,0 +1,90 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.tags; + +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.collaboration.CollaborationRecords; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; +import com.kingsrook.qqq.backend.core.actions.tables.query.VirtualFieldValueProviderInterface; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordReferenceTableMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.core.utils.CollectionUtils; + +/******************************************************************************* + ** Return approved tag labels for parents selected by the enclosing query. + *******************************************************************************/ +public class TagValueProvider implements VirtualFieldValueProviderInterface +{ + /******************************************************************************* + ** Batch both assignments and vocabulary; never recursively load parent records. + *******************************************************************************/ + @Override + public void fill(QTableMetaData table, String fieldName, List records, QueryInput origin) throws QException + { + for(List page : CollectionUtils.getPages(records, 500)) + { + List ids = page.stream().map(record -> String.valueOf(record.getValue(table.getPrimaryKeyField()))).distinct().toList(); + QTableMetaData metadata = QContext.getQInstance().getTable(CollaborationTables.RECORD_TAG).clone(); + metadata.withCustomizers(null); + metadata.getSupplementalMetaData().remove(RecordReferenceTableMetaData.NAME); + QueryInput input = new QueryInput(CollaborationTables.RECORD_TAG).withTransaction(origin.getTransaction()).withFilter(new QQueryFilter() + .withCriteria("tableName", QCriteriaOperator.EQUALS, table.getName()).withCriteria("recordId", QCriteriaOperator.IN, ids) + .withCriteria("parentDeleted", QCriteriaOperator.EQUALS, false)); + input.setTableMetaData(metadata); + input.withPermissionBypass("Tag provider reads only parents selected by the enclosing query."); + List assignments = new QueryAction().execute(input).getRecords(); + Map labels = new HashMap<>(); + List tagIds = assignments.stream().map(row -> row.getValueInteger("tagId")).distinct().toList(); + for(List tags : CollectionUtils.getPages(tagIds, 500)) + { + for(QRecord tag : CollaborationRecords.query(CollaborationTables.TAG, TagsService.vocabularyFilter(table.getName()) + .withCriteria("approved", QCriteriaOperator.EQUALS, true).withCriteria("id", QCriteriaOperator.IN, tags), origin.getTransaction())) + { + labels.put(tag.getValueInteger("id"), tag.getValueString("label")); + } + } + Map> byParent = new HashMap<>(); + for(QRecord assignment : assignments) + { + String label = labels.get(assignment.getValueInteger("tagId")); + if(label != null) + { + byParent.computeIfAbsent(assignment.getValueString("recordId"), key -> new java.util.TreeSet<>()).add(label); + } + } + for(QRecord record : page) + { + var values = byParent.get(String.valueOf(record.getValue(table.getPrimaryKeyField()))); + record.setValue(fieldName, values == null ? null : String.join(", ", values)); + } + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/collaboration/tags/TagsMetaDataProvider.java b/src/main/java/com/kingsrook/qbits/collaboration/tags/TagsMetaDataProvider.java index f4ac096..51cdfa8 100644 --- a/src/main/java/com/kingsrook/qbits/collaboration/tags/TagsMetaDataProvider.java +++ b/src/main/java/com/kingsrook/qbits/collaboration/tags/TagsMetaDataProvider.java @@ -56,7 +56,8 @@ public class TagsMetaDataProvider implements RuntimeMetaDataProviderInterface { contribution.getTables().put(table.getName(), new RuntimeTableMetaDataContribution() .withVirtualFields(List.of(new QVirtualFieldMetaData("collaborationTags", QFieldType.STRING).withLabel("Tags") - .withPossibleValueSourceName(CollaborationTables.TAG).withIsQueryCriteria(true).withIsQuerySelectable(false) + .withPossibleValueSourceName(CollaborationTables.TAG).withIsQueryCriteria(true).withIsQuerySelectable(true) + .withValueProvider(new QCodeReference(TagValueProvider.class)).withShouldTranslatePossibleValues(false) .withCriteriaExpander(new QCodeReference(TagCriteriaExpander.class)))) .withSearchFields(List.of("collaborationTags"))); } diff --git a/src/main/resources/help/qbit-collaboration.yaml b/src/main/resources/help/qbit-collaboration.yaml index 2b48923..d1e282d 100644 --- a/src/main/resources/help/qbit-collaboration.yaml +++ b/src/main/resources/help/qbit-collaboration.yaml @@ -746,3 +746,31 @@ entries: summary: "Choose the record feature you want to configure." - key: possibleValueSource:collaboration_permission_mode;slot:help summary: "Choose whether a feature accepts editing permission or requires an explicit feature permission." +- key: table:collaboration_attachment_cleanup;slot:help + summary: "Review storage cleanup waiting to finish after an attachment was deleted." + content: | + **How rows get created:** Authorized attachment operations create these records automatically. Deleted files are removed after the database commits. Uncertain uploads remain pending until the provider has finished writing. Ordinary successful cleanup removes its entry; administrator-resolved uploads retain a receipt. Administrators can inspect the records but cannot edit them directly. +- key: table:collaboration_attachment_cleanup;field:attachmentId + summary: "The deleted attachment, when known; failed uploads may have no committed attachment ID." +- key: table:collaboration_attachment_cleanup;field:bucketName + summary: "The storage bucket containing the file awaiting cleanup." +- key: table:collaboration_attachment_cleanup;field:storageKey + summary: "The immutable file location retained until cleanup succeeds." +- key: table:collaboration_attachment_cleanup;field:nextAttemptAt + summary: "Failed cleanup waits at least one minute before another scheduled attempt." +- key: table:collaboration_attachment_cleanup;field:resolvedAt + summary: "When an administrator resolved this upload. Resolved receipts are excluded from retries." +- key: table:collaboration_attachment_cleanup;field:resolvedBy + summary: "The signed-in administrator who resolved this upload. The original identity is retained on repeat requests." +- key: table:collaboration_attachment_cleanup;field:settlementEvidence + summary: "Evidence that the provider completed or cancelled the upload. Never include credentials or private file contents." +- key: table:collaboration_attachment_cleanup;section:settlement + summary: "Retained evidence and identity for an administrator-resolved upload." +- key: process:collaborationResolveAttachmentUpload;slot:help + summary: "Resolve uncertain uploads only after confirming that the provider has finished writing." + content: | + Select upload intents, record evidence of provider completion or cancellation for every selected location, and confirm the Review screen. Object absence or elapsed time alone is not proof. If completion cannot be established, leave the intent pending. The action requires an identified collaboration administrator, refuses active or referenced uploads, deletes current bytes, and retains a settlement receipt. Read Result to see which uploads remain unresolved. Existing attachment deletions use normal cleanup retry instead. +- key: process:collaborationRetryAttachmentCleanup;slot:help + summary: "Retry pending attachment file cleanup after storage or database recovery." + content: | + Processes up to 100 committed cleanup requests. Completed entries are removed; failures remain available for retry. Runs every minute when the host configures a collaboration scheduler. Manual execution requires collaboration administrator access. diff --git a/src/main/resources/presence/redis-mutate.lua b/src/main/resources/presence/redis-mutate.lua index 9102888..797927d 100644 --- a/src/main/resources/presence/redis-mutate.lua +++ b/src/main/resources/presence/redis-mutate.lua @@ -1,19 +1,21 @@ -- All room keys share a table hash tag. Redis TIME owns lease expiry. +local clock = redis.call('TIME') +local now = tonumber(clock[1]) * 1000 + math.floor(tonumber(clock[2]) / 1000) +local guard = presenceRecovery(KEYS, tonumber(ARGV[5]), ARGV[9], now) +local rooms = #KEYS - 1 -- Validate every pair before changing either record or table membership. -for i = 1, #KEYS, 2 do +for i = 1, rooms, 2 do if redis.call('ZCARD', KEYS[i]) ~= redis.call('HLEN', KEYS[i + 1]) then return redis.error_reply('Presence room state is incomplete') end end -local clock = redis.call('TIME') -local now = tonumber(clock[1]) * 1000 + math.floor(tonumber(clock[2]) / 1000) local member = ARGV[1] local ttl = tonumber(ARGV[5]) local capacity = tonumber(ARGV[6]) local sequence = tonumber(ARGV[3]) local departed = ARGV[4] == '1' -for i = 1, #KEYS, 2 do +for i = 1, rooms, 2 do local expired = redis.call('ZRANGEBYSCORE', KEYS[i], '-inf', now) for _, id in ipairs(expired) do redis.call('ZREM', KEYS[i], id) @@ -24,7 +26,7 @@ end local previous = redis.call('HGET', KEYS[2], member) -- A record lease and its table projection are written atomically. If one is -- missing while the other is live, do not erase surviving ordering/tombstones. -if #KEYS == 4 then +if rooms == 4 then local tablePrevious = redis.call('HGET', KEYS[4], member) if (previous and not tablePrevious) or (tablePrevious and not previous) then return redis.error_reply('Presence room state is incomplete') @@ -34,7 +36,7 @@ local old = previous and cjson.decode(previous) or nil local operation = ARGV[8] or 'legacy' -- Renewal authority is issued by open, not inferred from a caller's sequence. -- Missing/expired/departed leases cannot be recreated by delayed HTTP requests. -if operation == 'renew' and (not old or not old.leased or old.departed) then +if operation == 'renew' and (not old or not old.leased or old.departed or old.generation ~= guard.generation) then return -1 end if operation == 'open' and old then return -1 end @@ -52,7 +54,7 @@ if old then end -- Check every room before writing any membership, avoiding a partial join. -for i = 1, #KEYS, 2 do +for i = 1, rooms, 2 do if redis.call('HEXISTS', KEYS[i + 1], member) == 0 and redis.call('ZCARD', KEYS[i]) >= capacity then return 0 end @@ -64,7 +66,8 @@ data.sequence = sequence or cjson.null data.departed = departed data.pending = operation == 'open' data.leased = operation ~= 'legacy' or (old and old.leased) or false -for i = 1, #KEYS, 2 do +data.generation = guard.generation +for i = 1, rooms, 2 do -- Record scope comes first; table scope must never carry focused-field data. if i > 1 or ARGV[7] == '1' then data.fieldName = cjson.null end redis.call('HSET', KEYS[i + 1], member, cjson.encode(data)) @@ -72,4 +75,6 @@ for i = 1, #KEYS, 2 do redis.call('PEXPIRE', KEYS[i], ttl) redis.call('PEXPIRE', KEYS[i + 1], ttl) end +guard.liveUntil = now + ttl +redis.call('SET', KEYS[#KEYS], cjson.encode(guard)) return 1 diff --git a/src/main/resources/presence/redis-recovery.lua b/src/main/resources/presence/redis-recovery.lua new file mode 100644 index 0000000..9eaeca9 --- /dev/null +++ b/src/main/resources/presence/redis-recovery.lua @@ -0,0 +1,43 @@ +-- This prelude is executed atomically with the room operation, in the same slot. +-- The guard deliberately outlives idle leases: missing means unknown, not empty. +local function presenceRecovery(keys, ttl, candidate, now) + local encoded = redis.call('GET', keys[#keys]) + local guard + local created = false + if encoded then + guard = cjson.decode(encoded) + if type(guard.generation) ~= 'string' or type(guard.readyAt) ~= 'number' or type(guard.liveUntil) ~= 'number' or type(guard.ttl) ~= 'number' then + error('Invalid presence recovery guard') + end + if guard.ttl ~= ttl then error('Presence lease lifetime differs across nodes') end + -- All volatile projections disappearing before the last issued expiry is + -- data loss too, even when a persistent guard survives an eviction policy. + if guard.liveUntil > now and redis.call('ZCARD', keys[#keys - 2]) == 0 + and redis.call('HLEN', keys[#keys - 1]) == 0 then + guard = nil + end + end + if not guard then + created = true + guard = { generation = candidate, readyAt = now + ttl, liveUntil = 0, ttl = ttl } + redis.call('SET', keys[#keys], cjson.encode(guard)) + end + -- During recovery, discard only projections from an invalidated generation. + -- The table is cleaned atomically when its generation is created. Subsequent + -- requests only need to clean the touched record; all new table writes carry + -- the current generation. Avoid rescanning the whole table on every renewal. + if now < guard.readyAt then + for i = 1, #keys - 1, 2 do + if created or i < #keys - 2 then + for _, member in ipairs(redis.call('HKEYS', keys[i + 1])) do + local data = cjson.decode(redis.call('HGET', keys[i + 1], member)) + if data.generation ~= guard.generation then + redis.call('HDEL', keys[i + 1], member) + redis.call('ZREM', keys[i], member) + end + end + end + end + end + return guard +end diff --git a/src/main/resources/presence/redis-snapshot.lua b/src/main/resources/presence/redis-snapshot.lua index 46e32d0..686a8cf 100644 --- a/src/main/resources/presence/redis-snapshot.lua +++ b/src/main/resources/presence/redis-snapshot.lua @@ -1,12 +1,14 @@ -- A partially evicted room is unavailable, not empty. Do not silently discard it. -for i = 1, #KEYS, 2 do +local clock = redis.call('TIME') +local now = tonumber(clock[1]) * 1000 + math.floor(tonumber(clock[2]) / 1000) +local guard = presenceRecovery(KEYS, tonumber(ARGV[2]), ARGV[3], now) +local rooms = #KEYS - 1 +for i = 1, rooms, 2 do if redis.call('ZCARD', KEYS[i]) ~= redis.call('HLEN', KEYS[i + 1]) then return redis.error_reply('Presence room state is incomplete') end end -local clock = redis.call('TIME') -local now = tonumber(clock[1]) * 1000 + math.floor(tonumber(clock[2]) / 1000) -for i = 1, #KEYS, 2 do +for i = 1, rooms, 2 do local expired = redis.call('ZRANGEBYSCORE', KEYS[i], '-inf', now) for _, id in ipairs(expired) do redis.call('ZREM', KEYS[i], id) @@ -16,7 +18,7 @@ end local members = redis.call('ZRANGE', KEYS[1], 0, -1) -- Table membership is bounded by the configured capacity. Compare the record's -- projection without key discovery or cross-slot reads. -if #KEYS == 4 then +if rooms == 4 then local projected = 0 for _, id in ipairs(redis.call('ZRANGE', KEYS[3], 0, -1)) do if string.sub(id, -string.len(ARGV[1])) == ARGV[1] then @@ -31,9 +33,13 @@ if #KEYS == 4 then end end local result = {} +if now < guard.readyAt then return -2 end for _, id in ipairs(members) do local data = redis.call('HGET', KEYS[2], id) if not data then return redis.error_reply('Presence room state is incomplete') end + if cjson.decode(data).generation ~= guard.generation then + return redis.error_reply('Presence generation is inconsistent') + end table.insert(result, data) end return result diff --git a/src/test/java/com/kingsrook/qbits/collaboration/CollaborationApiHttpTest.java b/src/test/java/com/kingsrook/qbits/collaboration/CollaborationApiHttpTest.java index 6d86d75..dc5b61b 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/CollaborationApiHttpTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/CollaborationApiHttpTest.java @@ -39,6 +39,12 @@ import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.node.ObjectNode; import com.kingsrook.qbits.collaboration.attachments.AttachmentStorage; +import com.kingsrook.qbits.collaboration.customfields.FieldDefinitions; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData; import com.kingsrook.qbits.objectstorage.api.PresignGet; import com.kingsrook.qbits.objectstorage.api.PutRequest; import com.kingsrook.qqq.backend.core.actions.collaboration.RuntimeMetaDataOverlay; @@ -63,6 +69,69 @@ import static org.assertj.core.api.Assertions.assertThat; *******************************************************************************/ class CollaborationApiHttpTest extends PersonalServiceFixture { + /*************************************************************************** + ** Data loss is unavailable over HTTP until a fresh lease has rejoined. + ***************************************************************************/ + @Test + void redisLossRequiresHttpRejoin() throws Exception + { + try(var redis = new org.testcontainers.containers.GenericContainer<>(org.testcontainers.utility.DockerImageName.parse("redis:7.4-alpine")).withExposedPorts(6379)) + { + redis.start(); + new com.kingsrook.qbits.redis.RedisQBitProducer().withQBitConfig(new com.kingsrook.qbits.redis.RedisQBitConfig() + .withConnection("presence", new com.kingsrook.qbits.redis.RedisConnectionConfig() + .withHost(redis.getHost()).withPort(redis.getMappedPort(6379)).withTls(false))).produce(instance); + config.withRedisPresence(new com.kingsrook.qbits.collaboration.presence.RedisPresenceConfig() + .withConnectionName("presence").withDeployment("test").withApplication("http-recovery") + .withScopeProvider(new QCodeReference(RedisPresenceServiceTest.TenantScope.class))); + var runtime = new com.kingsrook.qbits.redis.RedisRuntimeService(); + try + { + runtime.start(instance); + String original = ok("POST", "/presence/leases", "{\"table\":\"document\",\"pk\":\"1\"}").get("connectionId").asText(); + awaitHttpPresence(original); + assertThat(redis.execInContainer("redis-cli", "FLUSHDB").getExitCode()).isZero(); + assertError(503, "GET", "/presence?table=document&pk=1", null, "alice"); + assertError(404, "GET", "/presence?table=document&pk=3", null, "alice"); + String oldRenewal = JSON.writeValueAsString(Map.of("connectionId", original, "table", "document", "pk", "1", "mode", "VIEWING", "sequence", 1000)); + assertThat(json(request("POST", "/presence/leases/renew", oldRenewal, "alice"), 409).path("code").asText()) + .isEqualTo("PRESENCE_LEASE_EXPIRED"); + assertError(404, "POST", "/presence/leases/renew", oldRenewal, "outsider"); + String replacement = ok("POST", "/presence/leases", "{\"table\":\"document\",\"pk\":\"1\"}").get("connectionId").asText(); + assertThat(replacement).isNotEqualTo(original); + awaitHttpPresence(replacement); + assertThat(ok("GET", "/presence?table=document&pk=1", null).path("viewers")).hasSize(1); + } + finally + { + runtime.stop(); + runtime.afterApplicationStop(); + } + } + } + + /*************************************************************************** + ** Keep a real HTTP lease alive while the production recovery window elapses. + ***************************************************************************/ + private void awaitHttpPresence(String lease) throws Exception + { + long deadline = System.nanoTime() + Duration.ofSeconds(55).toNanos(); + long sequence = 0; + while(true) + { + String body = JSON.writeValueAsString(Map.of("connectionId", lease, "table", "document", "pk", "1", "mode", "VIEWING", "sequence", ++sequence)); + var response = request("POST", "/presence/leases/renew", body, "alice"); + if(response.statusCode() == 200) + { + assertThat(json(response, 200).path("viewers")).hasSize(1); + return; + } + assertThat(json(response, 503).path("message").asText()).contains("recovering"); + assertThat(System.nanoTime()).as("HTTP presence recovery deadline").isLessThan(deadline); + Thread.sleep(1000); + } + } + /*************************************************************************** ** Shared presence has an explicit unavailable response, never a local fallback. ***************************************************************************/ @@ -384,6 +453,80 @@ class CollaborationApiHttpTest extends PersonalServiceFixture assertThat(tagQuery("IN", "[\"alpha\"]")).containsExactlyInAnyOrder(1, 2); } + /******************************************************************************* + ** EAV choices retain their stored ids and translated labels on the HTTP wire. + *******************************************************************************/ + @Test + void customFieldValuesThroughTableReads() throws Exception + { + admin(); + instance.addBackend(new RDBMSBackendMetaData().withName("fallback").withVendor("h2")); + instance.getTable("document").setBackendName("fallback"); + QRecord definition = new InsertAction().execute(new InsertInput(CollaborationTables.FIELD_DEFINITION).withRecords(List.of( + new QRecord().withValue("tableName", "document").withValue("name", "choice").withValue("label", "Choice").withValue("type", "SELECT") + .withValue("options", "[{\"id\":\"x\",\"label\":\"Choice X\"}]")))).getRecords().getFirst(); + assertThat(definition.getErrors()).isNullOrEmpty(); + FieldDefinitions.apply(definition.getValue("id"), "Publish"); + instance.getTable("document").setBackendName("memory"); + CollaborationRecords.insert(CollaborationTables.FIELD_VALUE, new QRecord().withValue("tableName", "document").withValue("recordId", "1") + .withValue("fieldName", "choice").withValue("valueText", "x").withValue("parentDeleted", false)); + login("alice"); + String serverUrl = baseUrl.substring(0, baseUrl.indexOf("/qbit/")); + HttpResponse response = client.send(HttpRequest.newBuilder(URI.create(serverUrl + "/table/document/query")).timeout(Duration.ofSeconds(20)) + .header("Cookie", "sessionUUID=alice" + sessionSuffix).header("Content-Type", "application/json") + .POST(HttpRequest.BodyPublishers.ofString("{\"filter\":{\"orderBys\":[{\"fieldName\":\"id\"}]}}")).build(), HttpResponse.BodyHandlers.ofString()); + JsonNode records = json(response, 200).get("records"); + assertThat(records).hasSize(2); + assertThat(records.get(0).at("/values/id").asInt()).isEqualTo(1); + assertThat(records.get(0).at("/values/cf_choice").asText()).isEqualTo("x"); + assertThat(records.get(0).at("/displayValues/cf_choice").asText()).isEqualTo("Choice X"); + assertThat(records.get(1).at("/values/cf_choice").isMissingNode()).isTrue(); + + response = client.send(HttpRequest.newBuilder(URI.create(serverUrl + "/table/document/1")).timeout(Duration.ofSeconds(20)) + .header("Cookie", "sessionUUID=alice" + sessionSuffix).GET().build(), HttpResponse.BodyHandlers.ofString()); + JsonNode record = json(response, 200).get("record"); + assertThat(record.at("/values/cf_choice").asText()).isEqualTo("x"); + assertThat(record.at("/displayValues/cf_choice").asText()).isEqualTo("Choice X"); + + response = client.send(HttpRequest.newBuilder(URI.create(serverUrl + "/table/document/1")).timeout(Duration.ofSeconds(20)) + .header("Cookie", "sessionUUID=outsider" + sessionSuffix).GET().build(), HttpResponse.BodyHandlers.ofString()); + assertThat(response.statusCode()).isEqualTo(404); + assertThat(response.body()).doesNotContain("Choice X", "First document"); + } + + /******************************************************************************* + ** Table reads expose tag labels while retaining parent-record authorization. + *******************************************************************************/ + @Test + void tagValuesThroughTableReads() throws Exception + { + RuntimeMetaDataOverlay.refresh(instance); + int tag = ok("POST", "/tags", "{\"table\":\"document\",\"label\":\"Review, urgent\"}").get("id").asInt(); + ok("PUT", "/records/document/1/tags", "{\"add\":[" + tag + "]}"); + String serverUrl = baseUrl.substring(0, baseUrl.indexOf("/qbit/")); + HttpResponse response = client.send(HttpRequest.newBuilder(URI.create(serverUrl + "/table/document/query")).timeout(Duration.ofSeconds(20)) + .header("Cookie", "sessionUUID=alice" + sessionSuffix).header("Content-Type", "application/json") + .POST(HttpRequest.BodyPublishers.ofString("{\"filter\":{\"orderBys\":[{\"fieldName\":\"id\"}]}}")).build(), HttpResponse.BodyHandlers.ofString()); + JsonNode records = json(response, 200).get("records"); + assertThat(records).hasSize(2); + assertThat(records.get(0).at("/values/id").asInt()).isEqualTo(1); + assertThat(records.get(0).at("/values/collaborationTags").asText()).isEqualTo("Review, urgent"); + assertThat(records.get(0).at("/displayValues/collaborationTags").asText()).isEqualTo("Review, urgent"); + assertThat(records.get(1).at("/values/id").asInt()).isEqualTo(2); + assertThat(records.get(1).at("/values/collaborationTags").isMissingNode()).as(records.toString()).isTrue(); + + response = client.send(HttpRequest.newBuilder(URI.create(serverUrl + "/table/document/1")).timeout(Duration.ofSeconds(20)) + .header("Cookie", "sessionUUID=alice" + sessionSuffix).GET().build(), HttpResponse.BodyHandlers.ofString()); + JsonNode record = json(response, 200).get("record"); + assertThat(record.at("/values/collaborationTags").asText()).isEqualTo("Review, urgent"); + assertThat(record.at("/displayValues/collaborationTags").asText()).isEqualTo("Review, urgent"); + + response = client.send(HttpRequest.newBuilder(URI.create(serverUrl + "/table/document/1")).timeout(Duration.ofSeconds(20)) + .header("Cookie", "sessionUUID=outsider" + sessionSuffix).GET().build(), HttpResponse.BodyHandlers.ofString()); + assertThat(response.body()).doesNotContain("Review, urgent", "First document"); + assertThat(response.statusCode()).isEqualTo(404); + } + /******************************************************************************* ** Run the v1 table query filtered on the virtual tags field; return matching ids. *******************************************************************************/ diff --git a/src/test/java/com/kingsrook/qbits/collaboration/CollaborationTablesGuardrailTest.java b/src/test/java/com/kingsrook/qbits/collaboration/CollaborationTablesGuardrailTest.java index 8b3f910..cc0b464 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/CollaborationTablesGuardrailTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/CollaborationTablesGuardrailTest.java @@ -65,6 +65,7 @@ class CollaborationTablesGuardrailTest extends PersonalServiceFixture Map.entry("collaboration_favorite", "user content: a person stars something"), Map.entry("collaboration_recent", "runtime: a person opens a record"), Map.entry("collaboration_attachment", "user content: a person uploads a file"), + Map.entry("collaboration_attachment_cleanup", "runtime: authorized deletion queues durable storage cleanup"), Map.entry("collaboration_comment", "user content: a person comments"), Map.entry("collaboration_comment_revision", "runtime: a comment is edited"), Map.entry("collaboration_mention", "runtime: a comment mentions someone"), @@ -183,7 +184,7 @@ class CollaborationTablesGuardrailTest extends PersonalServiceFixture private List collaborationTables() { List tables = instance.getTables().values().stream().map(QTableMetaData::getName).filter(n -> n.startsWith("collaboration_")).sorted().toList(); - assertThat(tables).hasSize(19); + assertThat(tables).hasSize(20); return (tables); } diff --git a/src/test/java/com/kingsrook/qbits/collaboration/CollaborationTest.java b/src/test/java/com/kingsrook/qbits/collaboration/CollaborationTest.java index 466cf3c..0057ea6 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/CollaborationTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/CollaborationTest.java @@ -114,7 +114,7 @@ class CollaborationTest CollaborationTables.SCOPE_TYPE, CollaborationTables.FEATURE, CollaborationTables.PERMISSION_MODE); assertThat(instance.getPossibleValueSource(CollaborationTables.FEATURE).getEnumValues()).hasSize(12); assertThat(instance.getApp("collaboration").getChildren()).extracting(child -> child.getName()) - .containsExactlyInAnyOrder(CollaborationTables.COMMENT, CollaborationTables.ATTACHMENT, CollaborationTables.TAG, CollaborationTables.FIELD_DEFINITION, CollaborationTables.TASK, CollaborationTables.SHARE, CollaborationTables.ACCESS_REQUEST, CollaborationTables.RECORD_ASSIGNMENT, CollaborationTables.ACTIVITY, CollaborationTables.FEATURE_SETTING, CollaborationTables.USER); + .containsExactlyInAnyOrder(CollaborationTables.COMMENT, CollaborationTables.ATTACHMENT, CollaborationTables.ATTACHMENT_CLEANUP, CollaborationTables.TAG, CollaborationTables.FIELD_DEFINITION, CollaborationTables.TASK, CollaborationTables.SHARE, CollaborationTables.ACCESS_REQUEST, CollaborationTables.RECORD_ASSIGNMENT, CollaborationTables.ACTIVITY, CollaborationTables.FEATURE_SETTING, CollaborationTables.USER); assertThat(instance.getApp("collaboration").getWidgets()).containsExactlyElementsOf(CollaborationHome.HOME_WIDGETS); for(String widget : CollaborationHome.HOME_WIDGETS) { @@ -221,7 +221,7 @@ class CollaborationTest QInstance headless = new QInstance(); headless.addBackend(new QBackendMetaData().withName("memory").withBackendType(MemoryBackendModule.class)); new CollaborationQBitProducer().withQBitConfig(config.withAdminApp(false)).produce(headless); - assertThat(headless.getTables()).hasSize(19); + assertThat(headless.getTables()).hasSize(20); assertThat(headless.getApps()).isEmpty(); assertThat(headless.getWidgets()).hasSize(5); } diff --git a/src/test/java/com/kingsrook/qbits/collaboration/CoreContractsTest.java b/src/test/java/com/kingsrook/qbits/collaboration/CoreContractsTest.java index d4573a0..9778328 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/CoreContractsTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/CoreContractsTest.java @@ -84,7 +84,8 @@ class CoreContractsTest extends PersonalServiceFixture RuntimeMetaDataOverlay.refresh(instance); long generation = RuntimeMetaDataOverlay.getGeneration(instance); assertThat(instance.getTable("document").getVirtualFields()).containsKey("collaborationTags"); - assertThat(instance.getTable("document").getVirtualField("collaborationTags").getIsQuerySelectable()).isFalse(); + assertThat(instance.getTable("document").getVirtualField("collaborationTags").getIsQuerySelectable()).isTrue(); + assertThat(instance.getTable("document").getVirtualField("collaborationTags").getValueProvider()).isNotNull(); QBackendTransaction tx = new QBackendTransaction(); var row = new InsertAction().execute(new InsertInput(CollaborationTables.FEATURE_SETTING).withTransaction(tx).withRecords(List.of( new QRecord().withValue("scopeType", "TABLE").withValue("scopeName", "document").withValue("feature", "TAGS") diff --git a/src/test/java/com/kingsrook/qbits/collaboration/CustomFieldValuesPostgresTest.java b/src/test/java/com/kingsrook/qbits/collaboration/CustomFieldValuesPostgresTest.java new file mode 100644 index 0000000..34c07e7 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/CustomFieldValuesPostgresTest.java @@ -0,0 +1,188 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.collaboration; + +import java.sql.Connection; +import java.sql.DriverManager; +import java.util.Set; +import com.kingsrook.qbits.collaboration.customfields.CustomFieldValueProvider; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QVirtualFieldMetaData; +import com.kingsrook.qqq.backend.module.postgres.model.metadata.PostgreSQLBackendMetaData; +import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction; +import com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.wait.strategy.Wait; +import static org.assertj.core.api.Assertions.assertThat; + +/******************************************************************************* + ** Verify provider reads using a real database and explicit caller transactions. + *******************************************************************************/ +class CustomFieldValuesPostgresTest extends PersonalServiceFixture +{ + private static GenericContainer postgres; + + /******************************************************************************* + ** Own an isolated database; never connect to an application database. + *******************************************************************************/ + @BeforeAll + static void startPostgres() + { + postgres = new GenericContainer<>("postgres:17-alpine").withEnv("POSTGRES_PASSWORD", "eav-fixture-only") + .withExposedPorts(5432).waitingFor(Wait.forLogMessage(".*database system is ready to accept connections.*", 2)); + postgres.start(); + } + + /******************************************************************************* + ** Stop only the fixture's database. + *******************************************************************************/ + @AfterAll + static void stopPostgres() + { + if(postgres != null) + { + postgres.stop(); + } + } + + /******************************************************************************* + ** Reuse the secured host metadata with real PostgreSQL storage. + *******************************************************************************/ + @Override + @BeforeEach + public void setUp() throws Exception + { + super.setUp(); + instance.getBackends().remove("memory"); + instance.addBackend(new PostgreSQLBackendMetaData().withName("memory").withHostName(postgres.getHost()) + .withPort(postgres.getMappedPort(5432)).withDatabaseName("postgres").withUsername("postgres").withPassword("eav-fixture-only")); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("DROP SCHEMA public CASCADE; CREATE SCHEMA public"); + sql.execute(CollaborationSchema.postgresScript(config)); + sql.execute("CREATE TABLE document (id INTEGER PRIMARY KEY, title TEXT, tenant TEXT)"); + sql.execute("INSERT INTO document VALUES (1, 'Visible', 'a'), (3, 'Private', 'b')"); + } + CollaborationRecords.insert(CollaborationTables.FIELD_DEFINITION, new QRecord().withValue("tableName", "document") + .withValue("name", "rating").withValue("label", "Rating").withValue("type", "INTEGER").withValue("status", "ACTIVE")); + instance.getTable("document").withVirtualField(new QVirtualFieldMetaData("cf_rating", QFieldType.INTEGER).withLabel("Rating") + .withIsQuerySelectable(true).withValueProvider(new QCodeReference(CustomFieldValueProvider.class))); + } + + /******************************************************************************* + ** Release transaction state and pooled connections before parent fixture cleanup. + *******************************************************************************/ + @Override + public void tearDown() + { + QContext.clearTransaction(); + ConnectionManager.closeConnectionProviders(instance); + super.tearDown(); + } + + /******************************************************************************* + ** A separate connection establishes committed visibility independently. + *******************************************************************************/ + private Connection connect() throws Exception + { + return (DriverManager.getConnection("jdbc:postgresql://" + postgres.getHost() + ":" + postgres.getMappedPort(5432) + + "/postgres", "postgres", "eav-fixture-only")); + } + + /******************************************************************************* + ** Both vocabulary and assignments use the explicit transaction's visibility. + *******************************************************************************/ + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void tagLabelsSeePendingVocabularyAndCompletion(boolean commit) throws Exception + { + instance.getTable("document").withVirtualField(new QVirtualFieldMetaData("collaborationTags", QFieldType.STRING).withLabel("Tags") + .withIsQuerySelectable(true).withPossibleValueSourceName(CollaborationTables.TAG).withShouldTranslatePossibleValues(false) + .withValueProvider(new QCodeReference(com.kingsrook.qbits.collaboration.tags.TagValueProvider.class))); + var tags = new com.kingsrook.qbits.collaboration.tags.TagsService(); + try(var transaction = new RDBMSTransaction(connect())) + { + QContext.setTransaction(transaction); + var tag = tags.create("document", "Pending transaction label", null); + tags.apply("document", 1, java.util.List.of(tag.id()), java.util.List.of()); + QContext.clearTransaction(); + assertThat(QContext.getQBackendTransaction()).isNull(); + var pending = new QueryAction().execute(new QueryInput("document").withFieldNamesToInclude(Set.of("collaborationTags")) + .withShouldTranslatePossibleValues(true).withShouldGenerateDisplayValues(true).withTransaction(transaction)).getRecords(); + assertThat(pending).hasSize(1); + assertThat(pending.getFirst().getValues()).containsOnlyKeys("collaborationTags").containsEntry("collaborationTags", "Pending transaction label"); + assertThat(pending.getFirst().getDisplayValue("collaborationTags")).isEqualTo("Pending transaction label"); + assertThat(new QueryAction().execute(new QueryInput("document")).getRecords().getFirst().getValue("collaborationTags")).isNull(); + if(commit) + { + transaction.commit(); + } + else + { + transaction.rollback(); + } + } + assertThat(new QueryAction().execute(new QueryInput("document")).getRecords().getFirst().getValue("collaborationTags")) + .isEqualTo(commit ? "Pending transaction label" : null); + } + + /******************************************************************************* + ** Explicit query transactions see pending values even without thread context. + *******************************************************************************/ + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void explicitTransactionSeesPendingValueAndCompletion(boolean commit) throws Exception + { + try(var transaction = new RDBMSTransaction(connect())) + { + QContext.setTransaction(transaction); + CollaborationRecords.insert(CollaborationTables.FIELD_VALUE, new QRecord().withValue("tableName", "document").withValue("recordId", "1") + .withValue("fieldName", "rating").withValue("valueNumber", 42).withValue("parentDeleted", false)); + QContext.clearTransaction(); + assertThat(QContext.getQBackendTransaction()).isNull(); + var pending = new QueryAction().execute(new QueryInput("document").withFieldNamesToInclude(Set.of("cf_rating")) + .withTransaction(transaction)).getRecords(); + assertThat(pending).hasSize(1); + assertThat(pending.getFirst().getValues()).containsOnlyKeys("cf_rating").containsEntry("cf_rating", 42); + assertThat(new QueryAction().execute(new QueryInput("document")).getRecords().getFirst().getValue("cf_rating")).isNull(); + if(commit) + { + transaction.commit(); + } + else + { + transaction.rollback(); + } + } + assertThat(new QueryAction().execute(new QueryInput("document")).getRecords().getFirst().getValue("cf_rating")).isEqualTo(commit ? 42 : null); + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/CustomFieldsTest.java b/src/test/java/com/kingsrook/qbits/collaboration/CustomFieldsTest.java index 2fde99c..d986eb4 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/CustomFieldsTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/CustomFieldsTest.java @@ -51,6 +51,9 @@ import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaDat import com.kingsrook.qbits.collaboration.app.CollaborationHome; import com.kingsrook.qqq.backend.core.model.dashboard.widgets.TableData; import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.Arguments; +import org.junit.jupiter.params.provider.MethodSource; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; @@ -242,6 +245,59 @@ public class CustomFieldsTest extends PersonalServiceFixture assertThat(FieldProcesses.metadata()).extracting(process -> process.getLabel()).containsExactly("Publish field", "Archive field", "Rebuild index"); } + /******************************************************************************* + ** Typed EAV storage returns the declared Java value through an actual query. + *******************************************************************************/ + @ParameterizedTest + @MethodSource("eavValueTypes") + void eavValueTypes(String type, String column, java.io.Serializable stored, java.io.Serializable expected) throws Exception + { + admin(); + instance.addBackend(new RDBMSBackendMetaData().withName("fallback").withVendor("h2")); + instance.getTable("document").setBackendName("fallback"); + QRecord draft = new QRecord().withValue("tableName", "document").withValue("name", "typed").withValue("label", "Typed").withValue("type", type); + if(List.of("SELECT", "MULTI_SELECT").contains(type)) + { + draft.withValue("options", "[{\"id\":\"x\",\"label\":\"Choice X\"}]"); + } + if("RECORD_LINK".equals(type)) + { + draft.withValue("linkTableName", CollaborationTables.USER); + } + QRecord definition = new InsertAction().execute(new InsertInput(CollaborationTables.FIELD_DEFINITION).withRecords(List.of(draft))).getRecords().getFirst(); + assertThat(definition.getErrors()).isNullOrEmpty(); + FieldDefinitions.apply(definition.getValue("id"), "Publish"); + instance.getTable("document").setBackendName("memory"); + CollaborationRecords.insert(CollaborationTables.FIELD_VALUE, new QRecord().withValue("tableName", "document").withValue("recordId", "1") + .withValue("fieldName", "typed").withValue(column, stored).withValue("parentDeleted", false)); + login("alice"); + var records = new QueryAction().execute(new QueryInput("document").withFieldNamesToInclude(java.util.Set.of("cf_typed"))).getRecords(); + assertThat(records).hasSize(2); + assertThat(records).allSatisfy(record -> assertThat(record.getValues()).containsOnlyKeys("cf_typed")); + assertThat(records).extracting(record -> record.getValue("cf_typed")).containsExactlyInAnyOrder(expected, null); + Object actual = records.stream().map(record -> record.getValue("cf_typed")).filter(java.util.Objects::nonNull).findFirst().orElseThrow(); + assertThat(actual).isExactlyInstanceOf(expected.getClass()); + } + + /******************************************************************************* + ** Keep expectations independent of the provider's column-mapping helper. + *******************************************************************************/ + static java.util.stream.Stream eavValueTypes() + { + return (java.util.stream.Stream.of( + Arguments.of("INTEGER", "valueNumber", new BigDecimal("7.0"), 7), + Arguments.of("DECIMAL", "valueNumber", new BigDecimal("12.345"), new BigDecimal("12.345")), + Arguments.of("BOOLEAN", "valueBoolean", false, false), + Arguments.of("DATE", "valueDate", java.time.LocalDate.of(2026, 10, 22), java.time.LocalDate.of(2026, 10, 22)), + Arguments.of("DATE_TIME", "valueDateTime", java.time.Instant.parse("2026-10-22T12:34:56Z"), java.time.Instant.parse("2026-10-22T12:34:56Z")), + Arguments.of("SELECT", "valueText", "x", "x"), + Arguments.of("MULTI_SELECT", "valueText", "[\"x\"]", "[\"x\"]"), + Arguments.of("USER", "valueText", "alice", "alice"), + Arguments.of("RECORD_LINK", "valueText", "alice", "alice"), + Arguments.of("STRING", "valueText", "A, B", "A, B"), + Arguments.of("TEXT", "valueText", "First\nSecond", "First\nSecond"))); + } + /******************************************************************************* ** An unsupported JSON backend gets typed, read-only subquery filters. *******************************************************************************/ @@ -256,18 +312,50 @@ public class CustomFieldsTest extends PersonalServiceFixture instance.getTable("document").setBackendName("memory"); var virtual = instance.getTable("document").getVirtualField("cf_rating"); assertThat(virtual).isNotNull(); - assertThat(virtual.getIsQuerySelectable()).isFalse(); + assertThat(virtual.getIsQuerySelectable()).isTrue(); assertThat(virtual.getIsEditable()).isFalse(); assertThat(FieldDefinitions.get(definition.getValue("id")).getValueString("storage")).isEqualTo("EAV"); CollaborationRecords.insert(CollaborationTables.FIELD_VALUE, new QRecord().withValue("tableName", "document").withValue("recordId", "1") .withValue("fieldName", "rating").withValue("valueNumber", new BigDecimal("7")).withValue("parentDeleted", false)); + QContext.getQSession().withSecurityKeyValue("allTenants", true); + CollaborationRecords.insert(CollaborationTables.FIELD_VALUE, new QRecord().withValue("tableName", "document").withValue("recordId", "3") + .withValue("fieldName", "rating").withValue("valueNumber", new BigDecimal("99")).withValue("parentDeleted", false)); login("alice"); + var selected = new QueryAction().execute(new QueryInput("document").withFieldNamesToInclude(java.util.Set.of("cf_rating"))).getRecords(); + assertThat(selected).hasSize(2); + assertThat(selected).allSatisfy(row -> assertThat(row.getValues()).containsOnlyKeys("cf_rating")); + assertThat(selected).extracting(row -> row.getValue("cf_rating")).containsExactlyInAnyOrder(7, null); assertThat(query("cf_rating", QCriteriaOperator.GREATER_THAN, 5)).extracting(row -> row.getValueInteger("id")).containsExactly(1); assertThat(query("cf_rating", QCriteriaOperator.NOT_EQUALS, 7)).extracting(row -> row.getValueInteger("id")).containsExactly(2); assertThat(new QueryAction().execute(new QueryInput("document").withFilter(new QQueryFilter().withCriteria("cf_rating", QCriteriaOperator.IS_BLANK))).getRecords()) .extracting(row -> row.getValueInteger("id")).containsExactly(2); + // Reuse the same session and query to prove results are not cached across access changes. + QueryInput projected = new QueryInput("document").withFieldNamesToInclude(java.util.Set.of("cf_rating")); + QContext.getQSession().getSecurityKeyValues().put("tenant", List.of("b")); + assertThat(new QueryAction().execute(projected).getRecords()).extracting(row -> row.getValue("cf_rating")).containsExactly(99); + QContext.getQSession().getSecurityKeyValues().put("tenant", List.of("a")); + assertThat(new QueryAction().execute(projected).getRecords()).extracting(row -> row.getValue("cf_rating")).containsExactlyInAnyOrder(7, null); + QContext.getQSession().setPermissions(java.util.Set.of()); + assertThatThrownBy(() -> new QueryAction().execute(projected)).isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + login("alice"); + assertThat(com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordReferenceTableMetaData.of(instance.getTable(CollaborationTables.FIELD_VALUE))).isNotNull(); + assertThat(new QueryAction().execute(new QueryInput(CollaborationTables.FIELD_VALUE) + .withFilter(new QQueryFilter().withCriteria("tableName", QCriteriaOperator.EQUALS, "document") + .withCriteria("recordId", QCriteriaOperator.IN, List.of("1", "3")))).getRecords()) + .extracting(row -> row.getValueString("recordId")).containsExactly("1"); + virtual.setIsHidden(true); + assertThat(new QueryAction().execute(new QueryInput("document")).getRecords()).allSatisfy(row -> assertThat(row.getValues()).doesNotContainKey("cf_rating")); + virtual.setIsHidden(false); + admin(); + QContext.getQSession().withSecurityKeyValue("allTenants", true); + assertThat(FieldDefinitions.valueCount(FieldDefinitions.get(definition.getValue("id")))).isEqualTo(2); + QRecord deleted = CollaborationRecords.insert(CollaborationTables.FIELD_VALUE, new QRecord().withValue("tableName", "document").withValue("recordId", "2") + .withValue("fieldName", "rating").withValue("valueNumber", new BigDecimal("88")).withValue("parentDeleted", true)); + assertThat(deleted.getValueBoolean("parentDeleted")).isTrue(); + login("alice"); + assertThat(new QueryAction().execute(new QueryInput("document").withFieldNamesToInclude(java.util.Set.of("cf_rating"))).getRecords()) + .extracting(row -> row.getValue("cf_rating")).containsExactlyInAnyOrder(7, null); admin(); - assertThat(FieldDefinitions.valueCount(FieldDefinitions.get(definition.getValue("id")))).isEqualTo(1); var expander = new com.kingsrook.qbits.collaboration.customfields.FieldCriteriaExpander(); assertThat(expander.expandSearchTerm(instance.getTable("document"), "text", com.kingsrook.qqq.backend.core.actions.tables.query.SearchTermKind.TEXT, null)).isNotNull(); assertThat(com.kingsrook.qbits.collaboration.customfields.FieldCriteriaExpander.valueColumn("DATE")).isEqualTo("valueDate"); diff --git a/src/test/java/com/kingsrook/qbits/collaboration/MyWorkPostgresTest.java b/src/test/java/com/kingsrook/qbits/collaboration/MyWorkPostgresTest.java new file mode 100644 index 0000000..ae65573 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/MyWorkPostgresTest.java @@ -0,0 +1,231 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration; + +import java.nio.file.Files; +import java.nio.file.Path; +import java.sql.Connection; +import java.util.Arrays; +import java.util.Locale; +import java.sql.DriverManager; +import java.util.ArrayList; +import java.util.List; +import com.kingsrook.qbits.collaboration.mywork.MyWorkService; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.module.postgres.model.metadata.PostgreSQLBackendMetaData; +import com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager; +import com.kingsrook.qqq.backend.module.rdbms.jdbc.C3P0PooledConnectionProvider; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.ConnectionPoolSettings; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.wait.strategy.Wait; +import static org.assertj.core.api.Assertions.assertThat; + +/******************************************************************************* + ** Real PostgreSQL query counts for the bounded, caller-authorized My Work scan. + *******************************************************************************/ +class MyWorkPostgresTest extends PersonalServiceFixture +{ + private static GenericContainer postgres; + + /******************************************************************************* + ** Collect SQL counts in an owned disposable database, never an application DB. + *******************************************************************************/ + @BeforeAll + static void startPostgres() + { + postgres = new GenericContainer<>("postgres:17-alpine") + .withEnv("POSTGRES_PASSWORD", "mywork-fixture-only").withExposedPorts(5432) + .withCommand("postgres", "-c", "shared_preload_libraries=pg_stat_statements") + .waitingFor(Wait.forLogMessage(".*database system is ready to accept connections.*", 2)); + postgres.start(); + } + + /******************************************************************************* + ** Stop only the database owned by this class. + *******************************************************************************/ + @AfterAll + static void stopPostgres() + { + if(postgres != null) + { + postgres.stop(); + } + } + + /******************************************************************************* + ** Reuse the real authorization fixture with PostgreSQL persistence. + *******************************************************************************/ + @Override + @BeforeEach + public void setUp() throws Exception + { + super.setUp(); + instance.getBackends().remove("memory"); + instance.addBackend(new PostgreSQLBackendMetaData().withName("memory").withHostName(postgres.getHost()) + .withPort(postgres.getMappedPort(5432)).withDatabaseName("postgres") + .withUsername("postgres").withPassword("mywork-fixture-only")); + var backend = (PostgreSQLBackendMetaData) instance.getBackend("memory"); + backend.setConnectionProvider(new QCodeReference(C3P0PooledConnectionProvider.class)); + backend.setConnectionPoolSettings(new ConnectionPoolSettings().withInitialPoolSize(1).withMinPoolSize(1) + .withMaxPoolSize(3).withAcquireIncrement(1).withCheckoutTimeoutSeconds(2)); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("CREATE EXTENSION pg_stat_statements"); + sql.execute(CollaborationSchema.postgresScript(config)); + sql.execute("CREATE TABLE document (id INTEGER PRIMARY KEY, title TEXT, tenant TEXT)"); + sql.execute("INSERT INTO document VALUES (1, 'Visible', 'a'), (3, 'Private', 'b')"); + sql.execute("INSERT INTO collaboration_task (id, table_name, record_id, title, assignee_user_id, created_by_user_id, status, parent_deleted, due_at) " + + "SELECT n, 'document', CASE WHEN n=601 THEN '3' ELSE '1' END, 'Task ' || n, 'alice', 'alice', 'OPEN', false, " + + "CASE WHEN n%2=0 THEN TIMESTAMPTZ '2026-10-22 00:00:00Z' + (601-n)*INTERVAL '1 second' ELSE NULL END FROM generate_series(1,601) n"); + sql.execute("SELECT pg_stat_statements_reset()"); + } + } + + /******************************************************************************* + ** Release owner pools before container teardown. + *******************************************************************************/ + @Override + public void tearDown() + { + ConnectionManager.closeConnectionProviders(instance); + super.tearDown(); + } + + /******************************************************************************* + ** Exact counts/order and bounded SQL parent/page queries on the real backend. + *******************************************************************************/ + @Test + void summaryPreservesVisibilityWithBoundedSqlQueries() throws Exception + { + var summary = new MyWorkService().summary(); + assertThat(summary.openTaskCount()).isEqualTo(600); + assertThat(summary.tasks()).extracting(row -> row.getValueInteger("id")) + .containsExactly(600, 598, 596, 594, 592, 590, 588, 586, 584, 582); + try(Connection connection = connect(); var sql = connection.createStatement(); + var rows = sql.executeQuery("SELECT query, calls FROM pg_stat_statements WHERE query ILIKE 'SELECT%'") ) + { + long parents = 0; + long tasks = 0; + while(rows.next()) + { + String query = rows.getString(1); + if(query.contains("\"document\"")) + { + parents += rows.getLong(2); + } + if(query.contains("\"collaboration_task\"")) + { + tasks += rows.getLong(2); + } + } + assertThat(parents).as("SQL parent queries").isBetween(1L, 8L); + assertThat(tasks).as("SQL task page queries").isEqualTo(5); + } + QContext.getQSession().getSecurityKeyValues().put("tenant", new ArrayList<>(List.of("b"))); + assertThat(new MyWorkService().summary().openTaskCount()).isEqualTo(1); + if(Boolean.getBoolean("qqq.mywork.load")) + { + QContext.getQSession().getSecurityKeyValues().put("tenant", new ArrayList<>(List.of("a"))); + measureLoad(); + } + } + + /******************************************************************************* + ** Opt-in reproducible SQL work measurement, not a deployment capacity claim. + *******************************************************************************/ + private void measureLoad() throws Exception + { + final int tasks = 2000; + final int samples = 10; + List report = new ArrayList<>(); + report.add("tasks,parents,samples,p50_ms,p95_ms,p99_ms,sql_calls,sql_execution_ms"); + for(int parents : List.of(10, tasks)) + { + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("TRUNCATE collaboration_task, document"); + sql.execute("INSERT INTO document SELECT n, 'Document ' || n, CASE WHEN n%10=0 THEN 'b' ELSE 'a' END FROM generate_series(1," + parents + ") n"); + sql.execute("INSERT INTO collaboration_task (id, table_name, record_id, title, assignee_user_id, created_by_user_id, status, parent_deleted) " + + "SELECT n, 'document', (((n-1)%" + parents + ")+1)::text, 'Task ' || n, 'alice', 'alice', 'OPEN', false FROM generate_series(1," + tasks + ") n"); + } + // Warm the JVM and pool without counting these calls in the samples. + for(int warmup = 0; warmup < 2; warmup++) + { + assertThat(new MyWorkService().summary().openTaskCount()).isEqualTo(1800); + } + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("SELECT pg_stat_statements_reset()"); + } + double[] elapsed = new double[samples]; + for(int sample = 0; sample < samples; sample++) + { + long start = System.nanoTime(); + var summary = new MyWorkService().summary(); + elapsed[sample] = (System.nanoTime() - start) / 1_000_000.0; + assertThat(summary.openTaskCount()).isEqualTo(1800); + assertThat(summary.tasks()).extracting(row -> row.getValueInteger("id")) + .containsExactly(1, 2, 3, 4, 5, 6, 7, 8, 9, 11); + } + var backend = (PostgreSQLBackendMetaData) instance.getBackend("memory"); + var pool = ConnectionManager.getPoolMetrics(instance, backend); + // C3P0 returns connections asynchronously; allow a bounded check-in interval. + long deadline = System.nanoTime() + 2_000_000_000L; + while(pool.get("connectionsBusy").intValue() != 0 && System.nanoTime() < deadline) + { + Thread.sleep(10); + pool = ConnectionManager.getPoolMetrics(instance, backend); + } + assertThat(pool.get("connectionsTotal").intValue()).isBetween(1, 3); + assertThat(pool.get("connectionsBusy").intValue()).isZero(); + assertThat(pool.get("checkoutTimeouts").longValue()).isZero(); + Arrays.sort(elapsed); + try(Connection connection = connect(); var sql = connection.createStatement(); + var rows = sql.executeQuery("SELECT COALESCE(sum(calls),0), COALESCE(sum(total_exec_time),0) FROM pg_stat_statements " + + "WHERE query ILIKE 'SELECT%' AND (query LIKE '%\"document\"%' OR query LIKE '%\"collaboration_task\"%' " + + "OR query LIKE '%\"collaboration_record_assignment\"%' OR query LIKE '%\"collaboration_mention\"%')")) + { + assertThat(rows.next()).isTrue(); + report.add(String.format(Locale.ROOT, "%d,%d,%d,%.3f,%.3f,%.3f,%d,%.3f", tasks, parents, samples, + elapsed[4], elapsed[9], elapsed[9], rows.getLong(1), rows.getDouble(2))); + } + } + Path output = Path.of("target/mywork-load.csv"); + Files.createDirectories(output.getParent()); + Files.write(output, report); + System.out.println("My Work local PostgreSQL workload: " + output.toAbsolutePath()); + report.forEach(System.out::println); + } + + /******************************************************************************* + ** A fixture-owned connection for setup and independent SQL observations. + *******************************************************************************/ + private Connection connect() throws Exception + { + return (DriverManager.getConnection("jdbc:postgresql://" + postgres.getHost() + ":" + postgres.getMappedPort(5432) + "/postgres", "postgres", "mywork-fixture-only")); + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/MyWorkQueryBudgetTest.java b/src/test/java/com/kingsrook/qbits/collaboration/MyWorkQueryBudgetTest.java new file mode 100644 index 0000000..fb5bcfb --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/MyWorkQueryBudgetTest.java @@ -0,0 +1,132 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration; + +import java.time.Instant; +import java.util.ArrayList; +import java.util.List; +import com.kingsrook.qbits.collaboration.model.CollaborationTables; +import com.kingsrook.qbits.collaboration.mywork.MyWorkService; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryRecordStore; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; + +/******************************************************************************* + ** Measure backend queries while preserving exact, caller-visible summary counts. + *******************************************************************************/ +class MyWorkQueryBudgetTest extends PersonalServiceFixture +{ + /******************************************************************************* + ** Counts cross page boundaries without including another user's or hidden work. + *******************************************************************************/ + @Test + void summaryPagesAssignmentsAndMentionsWithoutLeakingCounts() throws Exception + { + List parents = new ArrayList<>(); + List assignments = new ArrayList<>(); + List mentions = new ArrayList<>(); + for(int id = 1; id <= 208; id++) + { + if(id > 3) + { + parents.add(new QRecord().withValue("id", id).withValue("title", "Parent " + id).withValue("tenant", id == 208 ? "b" : "a")); + } + assignments.add(new QRecord().withValue("id", id).withValue("tableName", "document").withValue("recordId", Integer.toString(id)) + .withValue("assigneeUserId", id == 207 ? "bob" : "alice").withValue("parentDeleted", false)); + } + for(int id = 1; id <= 601; id++) + { + mentions.add(new QRecord().withValue("id", id).withValue("tableName", "document").withValue("recordId", Integer.toString(id % 3 + 1)) + .withValue("mentionedUserId", "alice").withValue("notifiedAt", Instant.now()).withValue("parentDeleted", false)); + } + mentions.add(new QRecord().withValue("id", 602).withValue("tableName", "document").withValue("recordId", "1") + .withValue("mentionedUserId", "bob").withValue("notifiedAt", Instant.now()).withValue("parentDeleted", false)); + mentions.add(new QRecord().withValue("id", 603).withValue("tableName", "document").withValue("recordId", "1") + .withValue("mentionedUserId", "alice").withValue("parentDeleted", false)); + MemoryRecordStore.getInstance().insert(new InsertInput("document").withRecords(parents), true); + MemoryRecordStore.getInstance().insert(new InsertInput(CollaborationTables.RECORD_ASSIGNMENT).withRecords(assignments), true); + MemoryRecordStore.getInstance().insert(new InsertInput(CollaborationTables.MENTION).withRecords(mentions), true); + MemoryRecordStore.resetStatistics(); + MemoryRecordStore.setCollectStatistics(true); + try + { + var summary = new MyWorkService().summary(); + assertThat(summary.assignedRecordsByTable()).containsOnlyKeys("document").containsEntry("document", 205L); + assertThat(summary.unreadMentionCount()).isEqualTo(401); + assertThat(summary.openTaskCount()).isZero(); + assertThat(summary.tasks()).isEmpty(); + MemoryRecordStore.getActionInputs().get(QueryInput.class).stream().map(input -> (QueryInput) input) + .filter(input -> input.getTableName().startsWith("collaboration_")) + .forEach(input -> assertThat(input.getFilter().getLimit()).isNotNull().isLessThanOrEqualTo(200)); + } + finally + { + MemoryRecordStore.setCollectStatistics(false); + MemoryRecordStore.resetStatistics(); + } + } + + /******************************************************************************* + ** Hundreds of tasks on one parent must not cause hundreds of parent lookups. + *******************************************************************************/ + @Test + void summaryBoundsPagesAndRepeatedParentLookups() throws Exception + { + List rows = new ArrayList<>(); + for(int id = 1; id <= 601; id++) + { + rows.add(new QRecord().withValue("id", id).withValue("tableName", "document") + .withValue("recordId", id == 601 ? "3" : "1").withValue("title", "Task " + id) + .withValue("assigneeUserId", "alice").withValue("createdByUserId", "alice") + .withValue("status", "OPEN").withValue("parentDeleted", false) + .withValue("dueAt", id % 2 == 0 ? Instant.parse("2026-10-22T00:00:00Z").plusSeconds(601 - id) : null)); + } + // Populate historical rows directly so current parent authorization is tested on read. + MemoryRecordStore.getInstance().insert(new InsertInput(CollaborationTables.TASK).withRecords(rows), true); + MemoryRecordStore.resetStatistics(); + MemoryRecordStore.setCollectStatistics(true); + try + { + var summary = new MyWorkService().summary(); + assertThat(summary.openTaskCount()).isEqualTo(600); + assertThat(summary.tasks()).extracting(row -> row.getValueInteger("id")) + .containsExactly(600, 598, 596, 594, 592, 590, 588, 586, 584, 582); + List queries = MemoryRecordStore.getActionInputs().get(QueryInput.class).stream() + .map(input -> (QueryInput) input).toList(); + assertThat(queries.stream().filter(input -> "document".equals(input.getTableName())).count()).isLessThanOrEqualTo(8); + assertThat(queries.stream().filter(input -> CollaborationTables.TASK.equals(input.getTableName()))) + .allSatisfy(input -> assertThat(input.getFilter().getLimit()).isNotNull().isLessThanOrEqualTo(200)); + // A second request must recheck current permissions rather than reuse cached decisions. + QContext.getQSession().getSecurityKeyValues().put("tenant", new ArrayList<>(List.of("b"))); + assertThat(new MyWorkService().summary().openTaskCount()).isEqualTo(1); + } + finally + { + MemoryRecordStore.setCollectStatistics(false); + MemoryRecordStore.resetStatistics(); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceRecoveryCostProbe.java b/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceRecoveryCostProbe.java new file mode 100644 index 0000000..90b0db6 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceRecoveryCostProbe.java @@ -0,0 +1,150 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration; + +import java.time.Duration; +import java.util.Arrays; +import java.util.LinkedHashMap; +import java.util.Map; +import com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException; +import com.kingsrook.qbits.collaboration.presence.RedisPresenceStore; +import com.kingsrook.qbits.redis.RedisConnection; +import com.kingsrook.qbits.redis.api.RedisNamespace; +import com.kingsrook.qqq.backend.core.utils.JsonUtils; +import org.testcontainers.containers.GenericContainer; +import redis.clients.jedis.DefaultJedisClientConfig; +import redis.clients.jedis.HostAndPort; + +/******************************************************************************* + ** Explicit local diagnostic, not a CI latency gate or deployment capacity claim. + *******************************************************************************/ +public class RedisPresenceRecoveryCostProbe +{ + /*************************************************************************** + ** Compare issued-lease renewal cost during real recovery and after readiness. + ***************************************************************************/ + public static void main(String[] args) throws Exception + { + int count = args.length == 0 ? 1000 : Integer.parseInt(args[0]); + if(count < 10 || count > 10000 || count % 10 != 0) + { + throw (new IllegalArgumentException("Use a multiple of ten between 10 and 10000")); + } + try(var redis = new GenericContainer<>("redis:7.4-alpine").withExposedPorts(6379)) + { + redis.start(); + String host = "localhost".equals(redis.getHost()) ? "127.0.0.1" : redis.getHost(); + try(var connection = new RedisConnection(new HostAndPort(host, redis.getMappedPort(6379)), + DefaultJedisClientConfig.builder().socketTimeoutMillis(5000).build(), 1, Duration.ofSeconds(1))) + { + var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("cost", "local", "tenant", "presence"), Duration.ofSeconds(45), count); + String[] leases = new String[count]; + long[] sequences = new long[count]; + long start = System.nanoTime(); + for(int i = 0; i < count; i++) + { + leases[i] = store.openLease("user" + i, "Person " + i, "document", Integer.toString(i / 10)); + renew(store, leases, sequences, i); + } + double populationMillis = (System.nanoTime() - start) / 1_000_000.0; + requireRecovering(store); + Map cold = sample(store, leases, sequences); + requireRecovering(store); + long deadline = System.nanoTime() + Duration.ofSeconds(60).toNanos(); + while(true) + { + for(int i = 0; i < count; i++) + { + renew(store, leases, sequences, i); + } + try + { + if(store.snapshot("document", null).size() != count) + { + throw (new AssertionError("Population did not survive recovery")); + } + break; + } + catch(PresenceRecoveringException expected) + { + if(System.nanoTime() >= deadline) + { + throw (new AssertionError("Recovery exceeded deadline")); + } + Thread.sleep(5000); + } + } + var result = new LinkedHashMap(); + result.put("entries", count); + result.put("samplesPerPhase", 200); + result.put("populationMillis", populationMillis); + result.put("recoveryRenewMillis", cold); + result.put("readyRenewMillis", sample(store, leases, sequences)); + result.put("scope", "One JVM, one connection, loopback Redis, no TLS/HTTP; diagnostic only"); + System.out.println(JsonUtils.toJson(result)); + } + } + } + + /*************************************************************************** + ** Sample the same issued leases with increasing sequence numbers. + ***************************************************************************/ + private static Map sample(RedisPresenceStore store, String[] leases, long[] sequences) + { + double[] times = new double[200]; + for(int i = 0; i < times.length; i++) + { + long before = System.nanoTime(); + renew(store, leases, sequences, i % leases.length); + times[i] = (System.nanoTime() - before) / 1_000_000.0; + } + Arrays.sort(times); + return (Map.of("p50", times[99], "p95", times[189], "p99", times[197], "max", times[199])); + } + + /*************************************************************************** + ** Renewal must not silently fail while measuring its elapsed time. + ***************************************************************************/ + private static void renew(RedisPresenceStore store, String[] leases, long[] sequences, int i) + { + if(!store.renewLease("user" + i, "Person " + i, leases[i], "document", Integer.toString(i / 10), "VIEWING", null, ++sequences[i])) + { + throw (new AssertionError("Renewal refused")); + } + } + + /*************************************************************************** + ** Do not label a measurement cold if the actual recovery window has ended. + ***************************************************************************/ + private static void requireRecovering(RedisPresenceStore store) + { + try + { + store.snapshot("document", "0"); + } + catch(PresenceRecoveringException expected) + { + return; + } + throw (new AssertionError("Recovery finished before cold measurement")); + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceRecoveryTest.java b/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceRecoveryTest.java new file mode 100644 index 0000000..a276c2d --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceRecoveryTest.java @@ -0,0 +1,325 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration; + + +import java.time.Duration; +import com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException; +import com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException; +import com.kingsrook.qbits.collaboration.presence.RedisPresenceStore; +import com.kingsrook.qbits.collaboration.presence.RedisPresenceAudience; +import com.kingsrook.qbits.collaboration.presence.RedisPresenceConfig; +import com.kingsrook.qbits.collaboration.presence.PresenceScopeProvider; +import com.kingsrook.qbits.redis.RedisConnection; +import com.kingsrook.qbits.redis.api.RedisNamespace; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import org.testcontainers.containers.GenericContainer; +import redis.clients.jedis.DefaultJedisClientConfig; +import redis.clients.jedis.HostAndPort; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Loss recovery must be shared across independent clients, not node-local memory. + *******************************************************************************/ +class RedisPresenceRecoveryTest extends PersonalServiceFixture +{ + /*************************************************************************** + ** Peer reads cannot prolong cold recovery; normal expiry stays known-empty. + ***************************************************************************/ + @Test + void coldRecoveryDeadlineAndIdleKnowledgeAreShared() throws Exception + { + try(var redis = new GenericContainer<>("redis:7.4-alpine").withExposedPorts(6379)) + { + redis.start(); + var address = new HostAndPort(redis.getHost(), redis.getMappedPort(6379)); + var config = DefaultJedisClientConfig.builder().build(); + try(var first = new RedisConnection(address, config, 2, Duration.ofMillis(500)); + var second = new RedisConnection(address, config, 2, Duration.ofMillis(500))) + { + var namespace = new RedisNamespace("test", "cold-idle", "tenant", "presence"); + var a = new RedisPresenceStore(first.scripts(), namespace, Duration.ofSeconds(2), 10); + var b = new RedisPresenceStore(second.scripts(), namespace, Duration.ofSeconds(2), 10); + assertThatThrownBy(() -> a.snapshot("document", "1")).isInstanceOf(PresenceRecoveringException.class); + String guardKey = first.client().keys("*:recovery").iterator().next(); + String coldGuard = first.client().get(guardKey); + assertThatThrownBy(() -> b.snapshot("document", null)).isInstanceOf(PresenceRecoveringException.class); + assertThat(first.client().get(guardKey)).isEqualTo(coldGuard); + long deadline = System.nanoTime() + Duration.ofSeconds(6).toNanos(); + while(true) + { + try + { + assertThat(b.snapshot("document", "1")).isEmpty(); + break; + } + catch(PresenceRecoveringException recovering) + { + assertThat(System.nanoTime()).isLessThan(deadline); + assertThat(first.client().get(guardKey)).isEqualTo(coldGuard); + Thread.sleep(50); + } + } + assertThat(a.snapshot("document", null)).isEmpty(); + assertThat(first.client().get(guardKey)).isEqualTo(coldGuard); + assertThat(first.client().pttl(guardKey)).isEqualTo(-1L); + + String lease = a.openLease("alice", "Alice", "document", "1"); + awaitVisible(b, lease); + String activeGuard = first.client().get(guardKey); + deadline = System.nanoTime() + Duration.ofSeconds(6).toNanos(); + while(first.client().keys("*").size() > 1) + { + assertThat(System.nanoTime()).isLessThan(deadline); + Thread.sleep(50); + } + assertThat(first.client().keys("*")).containsExactly(guardKey); + assertThat(a.snapshot("document", "1")).isEmpty(); + assertThat(b.snapshot("document", null)).isEmpty(); + assertThat(first.client().get(guardKey)).isEqualTo(activeGuard); + assertThatThrownBy(() -> b.renewLease("alice", "Alice", lease, "document", "1", "VIEWING", null, 100L)) + .isInstanceOf(PresenceLeaseExpiredException.class); + } + } + } + + + /*************************************************************************** + ** Keep a real lease alive while waiting for bounded initial recovery. + ***************************************************************************/ + private void awaitVisible(RedisPresenceStore store, String lease) throws Exception + { + long deadline = System.nanoTime() + Duration.ofSeconds(8).toNanos(); + long sequence = 1; + while(System.nanoTime() < deadline) + { + store.renewLease("alice", "Alice", lease, "document", "1", "EDITING", null, sequence++); + try + { + assertThat(store.snapshot("document", "1")).extracting("userId").containsExactly("alice"); + return; + } + catch(IllegalStateException recovering) + { + assertThat(recovering).hasMessageContaining("recover"); + Thread.sleep(50); + } + } + throw (new AssertionError("Presence did not finish bounded recovery")); + } + + + /*************************************************************************** + ** A cleared dataset is unknown, not known-empty; live clients can repopulate. + ***************************************************************************/ + @ParameterizedTest + @ValueSource(booleans = { false, true }) + void totalLossRequiresSharedRecoveryBeforeKnownEmpty(boolean retainGuard) throws Exception + { + try(GenericContainer redis = new GenericContainer<>("redis:7.4-alpine").withExposedPorts(6379)) + { + redis.start(); + var address = new HostAndPort(redis.getHost(), redis.getMappedPort(6379)); + var clientConfig = DefaultJedisClientConfig.builder().build(); + try(var first = new RedisConnection(address, clientConfig, 2, Duration.ofMillis(500)); + var second = new RedisConnection(address, clientConfig, 2, Duration.ofMillis(500))) + { + var namespace = new RedisNamespace("test", "app", "tenant-a", "presence"); + var a = new RedisPresenceStore(first.scripts(), namespace, Duration.ofSeconds(2), 10); + var b = new RedisPresenceStore(second.scripts(), namespace, Duration.ofSeconds(2), 10); + String oldLease = a.openLease("alice", "Alice", "document", "1"); + awaitVisible(b, oldLease); + if(retainGuard) + { + var projections = first.client().keys("*").stream().filter(key -> !key.endsWith(":recovery")).toArray(String[]::new); + first.client().del(projections); + } + else + { + first.client().flushDB(); + } + assertThatThrownBy(() -> b.snapshot("document", "1")) + .isInstanceOf(IllegalStateException.class).hasMessageContaining("recover"); + assertThatThrownBy(() -> a.snapshot("document", null)) + .isInstanceOf(IllegalStateException.class).hasMessageContaining("recover"); + assertThatThrownBy(() -> a.renewLease("alice", "Alice", oldLease, "document", "1", "EDITING", null, 999L)) + .isInstanceOf(PresenceLeaseExpiredException.class); + String replacement = b.openLease("alice", "Alice", "document", "1"); + assertThat(replacement).isNotNull().isNotEqualTo(oldLease); + awaitVisible(a, replacement); + assertThat(b.snapshot("document", null)).extracting("userId").containsExactly("alice"); + assertThat(b.snapshot("document", "unoccupied")).isEmpty(); + } + } + } + + + /*************************************************************************** + ** Guard loss cleans the table once and untouched record rooms on demand. + ***************************************************************************/ + @Test + void missingGuardRetiresUnvisitedRecordsWithoutRemovingNewLeases() throws Exception + { + try(var redis = new GenericContainer<>("redis:7.4-alpine").withExposedPorts(6379)) + { + redis.start(); + try(var connection = new RedisConnection(new HostAndPort(redis.getHost(), redis.getMappedPort(6379)), + DefaultJedisClientConfig.builder().build(), 2, Duration.ofMillis(500))) + { + var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "guard-loss", "tenant", "presence"), Duration.ofSeconds(2), 2); + String oldAlice = store.openLease("alice", "Alice", "document", "1"); + String oldBob = store.openLease("bob", "Bob", "document", "2"); + connection.client().del(connection.client().keys("*:recovery").toArray(String[]::new)); + assertThatThrownBy(() -> store.snapshot("document", null)).isInstanceOf(PresenceRecoveringException.class); + String alice = store.openLease("alice", "Alice", "document", "1"); + String bob = store.openLease("bob", "Bob", "document", "2"); + assertThatThrownBy(() -> store.renewLease("alice", "Alice", oldAlice, "document", "1", "VIEWING", null, 99L)) + .isInstanceOf(PresenceLeaseExpiredException.class); + assertThatThrownBy(() -> store.renewLease("bob", "Bob", oldBob, "document", "2", "VIEWING", null, 99L)) + .isInstanceOf(PresenceLeaseExpiredException.class); + assertThat(store.renewLease("bob", "Bob", bob, "document", "2", "VIEWING", null, 1L)).isTrue(); + awaitVisible(store, alice); + assertThat(store.snapshot("document", "1")).extracting("userId").containsExactly("alice"); + } + } + } + + + /*************************************************************************** + ** Restore must happen in maintenance, followed by a fresh host-owned generation. + ** Replaying both old guard and lease bytes cannot authorize the new deployment. + ***************************************************************************/ + @Test + void restoredImageCannotAuthorizeRotatedHostGeneration() throws Exception + { + try(GenericContainer redis = new GenericContainer<>("redis:7.4-alpine").withExposedPorts(6379)) + { + redis.start(); + var address = new HostAndPort(redis.getHost(), redis.getMappedPort(6379)); + var clientConfig = DefaultJedisClientConfig.builder().build(); + try(var first = new RedisConnection(address, clientConfig, 2, Duration.ofMillis(500)); + var second = new RedisConnection(address, clientConfig, 2, Duration.ofMillis(500))) + { + var oldNamespace = RedisPresenceAudience.namespace(recoveryConfig("before-restore")); + var oldStore = new RedisPresenceStore(first.scripts(), oldNamespace, Duration.ofSeconds(2), 10); + String oldLease = oldStore.openLease("alice", "Alice", "document", "1"); + awaitVisible(oldStore, oldLease); + var image = new java.util.HashMap(); + for(String key : first.client().keys("*")) + { + image.put(key, first.client().dump(key)); + } + oldStore.leave("alice", oldLease, "document", "1", 999L); + first.client().flushDB(); + for(var entry : image.entrySet()) + { + first.client().restore(entry.getKey(), 0, entry.getValue()); + } + var restoredNamespace = RedisPresenceAudience.namespace(recoveryConfig("after-restore")); + var a = new RedisPresenceStore(first.scripts(), restoredNamespace, Duration.ofSeconds(2), 10); + var b = new RedisPresenceStore(second.scripts(), restoredNamespace, Duration.ofSeconds(2), 10); + assertThatThrownBy(() -> a.renewLease("alice", "Alice", oldLease, "document", "1", "EDITING", null, 1000L)) + .isInstanceOf(PresenceLeaseExpiredException.class); + assertThatThrownBy(() -> b.snapshot("document", "1")) + .isInstanceOf(IllegalStateException.class).hasMessageContaining("recover"); + String fresh = b.openLease("alice", "Alice", "document", "1"); + awaitVisible(a, fresh); + assertThat(b.snapshot("document", "1")).extracting("userId").containsExactly("alice"); + assertThatThrownBy(() -> b.renewLease("alice", "Alice", oldLease, "document", "1", "EDITING", null, 1001L)) + .isInstanceOf(PresenceLeaseExpiredException.class); + } + } + } + + + /*************************************************************************** + ** Generation is bounded deployment configuration, never browser input. + ***************************************************************************/ + @Test + void validatesHostGeneration() + { + var errors = new java.util.ArrayList(); + recoveryConfig("valid-20261026").validate(errors); + assertThat(errors).isEmpty(); + for(String invalid : java.util.Arrays.asList(null, "", " ", "x".repeat(65), "invalid:namespace")) + { + errors.clear(); + recoveryConfig(invalid).validate(errors); + assertThat(errors).anySatisfy(error -> assertThat(error).contains("recovery generation")); + } + } + + + /*************************************************************************** + ** Recovery readiness must not depend on which node's lease policy is shorter. + ***************************************************************************/ + @Test + void refusesInconsistentNodeLeaseLifetimes() throws Exception + { + try(GenericContainer redis = new GenericContainer<>("redis:7.4-alpine").withExposedPorts(6379)) + { + redis.start(); + try(var connection = new RedisConnection(new HostAndPort(redis.getHost(), redis.getMappedPort(6379)), + DefaultJedisClientConfig.builder().build(), 2, Duration.ofMillis(500))) + { + var namespace = new RedisNamespace("test", "app", "ttl-policy", "presence"); + var first = new RedisPresenceStore(connection.scripts(), namespace, Duration.ofSeconds(2), 10); + var second = new RedisPresenceStore(connection.scripts(), namespace, Duration.ofSeconds(3), 10); + assertThatThrownBy(() -> first.snapshot("document", null)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException.class); + assertThatThrownBy(() -> second.snapshot("document", null)) + .isInstanceOf(redis.clients.jedis.exceptions.JedisException.class).hasMessageContaining("lifetime differs"); + } + } + } + + + /*************************************************************************** + ** Use the real host-configured storage/audience namespace derivation. + ***************************************************************************/ + private RedisPresenceConfig recoveryConfig(String generation) + { + return (new RedisPresenceConfig().withConnectionName("presence").withDeployment("test").withApplication("app") + .withScopeProvider(new QCodeReference(FixtureScope.class)).withRecoveryGeneration(generation)); + } + + + /*************************************************************************** + ** Tenant scope belongs to the host; this test uses one fixed authenticated scope. + ***************************************************************************/ + public static class FixtureScope implements PresenceScopeProvider + { + /*************************************************************************** + ** Return the fixture's host-selected tenant, without client-supplied fields. + ***************************************************************************/ + @Override + public String tenantScope() + { + return ("tenant-a"); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceServiceTest.java b/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceServiceTest.java index 785359f..83e8862 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceServiceTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceServiceTest.java @@ -23,6 +23,7 @@ package com.kingsrook.qbits.collaboration; import com.kingsrook.qbits.collaboration.presence.PresenceScopeProvider; import com.kingsrook.qbits.collaboration.presence.PresenceService; +import com.kingsrook.qbits.collaboration.presence.RedisPresenceBackend; import com.kingsrook.qbits.collaboration.presence.RedisPresenceConfig; import com.kingsrook.qbits.redis.RedisConnectionConfig; import com.kingsrook.qbits.redis.RedisQBitConfig; @@ -72,6 +73,7 @@ class RedisPresenceServiceTest extends PersonalServiceFixture } firstRuntime.start(instance); secondRuntime.start(second.instance); + warmScopes(instance); firstTransport.start(instance); secondTransport.start(second.instance); org.junit.jupiter.api.Assertions.assertTimeoutPreemptively(java.time.Duration.ofSeconds(5), () -> @@ -240,13 +242,14 @@ class RedisPresenceServiceTest extends PersonalServiceFixture redis.start(); new RedisQBitProducer().withQBitConfig(new RedisQBitConfig().withConnection("presence", new RedisConnectionConfig().withHost(redis.getHost()).withPort(redis.getMappedPort(6379)).withTls(false) - .withConnectionTimeoutMillis(200).withSocketTimeoutMillis(200))).produce(instance); + .withConnectionTimeoutMillis(200).withSocketTimeoutMillis(1000))).produce(instance); config.withRedisPresence(new RedisPresenceConfig().withConnectionName("presence").withDeployment("test") .withApplication("fixture").withScopeProvider(new QCodeReference(TenantScope.class))); var runtime = new RedisRuntimeService(); try { runtime.start(instance); + warmScopes(instance); var service = new PresenceService(); String aliceLease; try(var hub = new com.kingsrook.qqq.middleware.javalin.realtime.RealtimeHub(instance)) @@ -319,6 +322,64 @@ class RedisPresenceServiceTest extends PersonalServiceFixture } } + /*************************************************************************** + ** Wait for every dashboard table in both authenticated tenant namespaces. + ** No Redis fixture writes bypass the production cold-recovery contract. + ***************************************************************************/ + private static void warmScopes(com.kingsrook.qqq.backend.core.model.metadata.QInstance node) throws Exception + { + var previousInstance = QContext.getQInstance(); + var previousSession = QContext.getQSession(); + var a = session("warm-a"); + var b = session("warm-b"); + b.clearSecurityKeyValues(); + b.withSecurityKeyValue("tenant", "b"); + var sessions = java.util.List.of(a, b); + try + { + for(var user : sessions) + { + QContext.init(node, user); + var backend = new RedisPresenceBackend(RedisPresenceConfig.forInstance(node)); + for(String table : node.getTables().keySet()) + { + assertThatThrownBy(() -> backend.snapshot(table, null)) + .isInstanceOf(QServiceUnavailableException.class).hasMessageContaining("recovering"); + } + } + long deadline = System.nanoTime() + java.time.Duration.ofSeconds(50).toNanos(); + for(var user : sessions) + { + QContext.init(node, user); + var backend = new RedisPresenceBackend(RedisPresenceConfig.forInstance(node)); + for(String table : node.getTables().keySet()) + { + while(true) + { + try + { + assertThat(backend.snapshot(table, null)).isEmpty(); + break; + } + catch(QServiceUnavailableException recovering) + { + assertThat(recovering).hasMessageContaining("recovering"); + if(System.nanoTime() >= deadline) + { + throw (new AssertionError("Service recovery did not finish", recovering)); + } + Thread.sleep(50); + } + } + } + } + } + finally + { + QContext.init(previousInstance, previousSession); + } + } + /*************************************************************************** ** Tenant identity comes from the authenticated host session, never the request. ***************************************************************************/ diff --git a/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceStoreTest.java b/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceStoreTest.java index 7817587..3a1401d 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceStoreTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceStoreTest.java @@ -38,6 +38,44 @@ import static org.assertj.core.api.Assertions.assertThatThrownBy; *******************************************************************************/ class RedisPresenceStoreTest { + /*************************************************************************** + ** Prime cold fixture namespaces together, using actual server readiness. + ***************************************************************************/ + private static void awaitReady(RedisPresenceStore... stores) throws Exception + { + for(RedisPresenceStore store : stores) + { + try + { + assertThat(store.snapshot("orders", null)).isEmpty(); + } + catch(com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException expected) + { + // The first read initializes the shared recovery interval. + } + } + long deadline = System.nanoTime() + Duration.ofSeconds(5).toNanos(); + for(RedisPresenceStore store : stores) + { + while(true) + { + try + { + assertThat(store.snapshot("orders", null)).isEmpty(); + break; + } + catch(com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException expected) + { + if(System.nanoTime() >= deadline) + { + throw (new AssertionError("Fixture recovery did not finish", expected)); + } + Thread.sleep(20); + } + } + } + } + /*************************************************************************** ** Server-issued leases cross nodes but cannot survive departure or data loss. ***************************************************************************/ @@ -53,8 +91,9 @@ class RedisPresenceStoreTest var second = new RedisConnection(address, config, 2, Duration.ofMillis(500))) { var scope = new RedisNamespace("test", "app", "issued", "presence"); - var a = new RedisPresenceStore(first.scripts(), scope, Duration.ofSeconds(45), 10); - var b = new RedisPresenceStore(second.scripts(), scope, Duration.ofSeconds(45), 10); + var a = new RedisPresenceStore(first.scripts(), scope, Duration.ofSeconds(2), 10); + var b = new RedisPresenceStore(second.scripts(), scope, Duration.ofSeconds(2), 10); + awaitReady(a); String lease = a.openLease("one", "One", "orders", "1"); assertThat(lease).startsWith("lease:"); assertThat(b.snapshot("orders", "1")).isEmpty(); @@ -85,7 +124,8 @@ class RedisPresenceStoreTest .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); assertThatThrownBy(() -> b.heartbeat("one", "One", replacement, "orders", "1", "VIEWING", null, 2L)) .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); - assertThat(first.client().dbSize()).isZero(); + assertThat(first.client().dbSize()).isEqualTo(1); + assertThat(first.client().keys("*:recovery")).hasSize(1); } } } @@ -103,6 +143,7 @@ class RedisPresenceStoreTest DefaultJedisClientConfig.builder().build(), 2, Duration.ofMillis(500))) { var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "app", "issued-expiry", "presence"), Duration.ofMillis(150), 1); + awaitReady(store); String pending = store.openLease("one", "One", "orders", "1"); assertThat(store.openLease("two", "Two", "orders", "2")).isNull(); assertThat(store.snapshot("orders", null)).isEmpty(); @@ -128,11 +169,11 @@ class RedisPresenceStoreTest } /*************************************************************************** - ** Characterize the open recovery gap: all projections lost together look new. - ** This is evidence for #1295, not an assertion of the desired final contract. + ** Total loss stays unknown even when the internal legacy API repopulates it. + ** The shared HTTP service requires server-issued leases instead of this API. ***************************************************************************/ @Test - void completeDataLossCurrentlyAcceptsDelayedHeartbeat() throws Exception + void totalLossIsRecoveringEvenForLegacyHeartbeat() throws Exception { try(GenericContainer redis = new GenericContainer<>(DockerImageName.parse("redis:7.4-alpine")).withExposedPorts(6379)) { @@ -143,8 +184,9 @@ class RedisPresenceStoreTest var second = new RedisConnection(address, config, 2, Duration.ofMillis(500))) { var scope = new RedisNamespace("test", "app", "recovery", "presence"); - var a = new RedisPresenceStore(first.scripts(), scope, Duration.ofSeconds(45), 10); - var b = new RedisPresenceStore(second.scripts(), scope, Duration.ofSeconds(45), 10); + var a = new RedisPresenceStore(first.scripts(), scope, Duration.ofSeconds(2), 10); + var b = new RedisPresenceStore(second.scripts(), scope, Duration.ofSeconds(2), 10); + awaitReady(a); a.heartbeat("active", "Active", "active-tab", "orders", "1", "VIEWING", null, 1L); a.heartbeat("departed", "Departed", "old-tab", "orders", "1", "EDITING", null, 1L); b.leave("departed", "old-tab", "orders", "1", 3L); @@ -152,10 +194,13 @@ class RedisPresenceStoreTest // Only this disposable test server is cleared. Both clients stay alive. first.client().flushDB(); - assertThat(b.snapshot("orders", "1")).isEmpty(); + assertThatThrownBy(() -> b.snapshot("orders", "1")) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException.class); assertThat(a.heartbeat("departed", "Departed", "old-tab", "orders", "1", "EDITING", null, 2L)).isTrue(); - assertThat(b.snapshot("orders", "1")).extracting("userId").containsExactly("departed"); - assertThat(b.snapshot("orders", null)).extracting("userId").containsExactly("departed"); + assertThatThrownBy(() -> b.snapshot("orders", "1")) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException.class); + assertThatThrownBy(() -> b.snapshot("orders", null)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException.class); } } } @@ -173,14 +218,16 @@ class RedisPresenceStoreTest try(var connection = new RedisConnection(new HostAndPort(redis.getHost(), redis.getMappedPort(6379)), DefaultJedisClientConfig.builder().build(), 2, Duration.ofMillis(500))) { - var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "app", "late", "presence"), Duration.ofMillis(150), 10); + var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "app", "late", "presence"), Duration.ofMillis(400), 10); + awaitReady(store); store.heartbeat("departed", "Departed", "tab", "orders", "1", "EDITING", null, 1L); store.leave("departed", "tab", "orders", "1", 3L); store.heartbeat("departed", "Departed", "tab", "orders", "1", "EDITING", null, 2L); assertThat(store.snapshot("orders", "1")).isEmpty(); - // Longer-lived peer proves this does not depend on losing entire keys. - var peer = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "app", "late", "presence"), Duration.ofSeconds(45), 10); + // A later renewal keeps the peer live with the same deployment TTL. + Thread.sleep(250); + var peer = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "app", "late", "presence"), Duration.ofMillis(400), 10); peer.heartbeat("active", "Active", "peer-tab", "orders", "1", "VIEWING", null, 1L); Thread.sleep(250); assertThat(peer.snapshot("orders", "1")).extracting("userId").containsExactly("active"); @@ -205,8 +252,10 @@ class RedisPresenceStoreTest var second = new RedisConnection(address, config, 4, Duration.ofMillis(500))) { var scope = new RedisNamespace("test", "app", "tenant-a", "presence"); - var a = new RedisPresenceStore(first.scripts(), scope, Duration.ofSeconds(45), 100); - var b = new RedisPresenceStore(second.scripts(), scope, Duration.ofSeconds(45), 100); + var a = new RedisPresenceStore(first.scripts(), scope, Duration.ofSeconds(2), 100); + var b = new RedisPresenceStore(second.scripts(), scope, Duration.ofSeconds(2), 100); + var tenantB = new RedisPresenceStore(second.scripts(), new RedisNamespace("test", "app", "tenant-b", "presence"), Duration.ofSeconds(2), 100); + awaitReady(a, tenantB); for(int i = 0; i < 6; i++) { assertThat((i % 2 == 0 ? a : b).heartbeat("user-" + i, "Person " + i, "tab-" + i, "orders", "123", @@ -225,7 +274,6 @@ class RedisPresenceStoreTest assertThat(a.snapshot("orders", "123").getFirst().mode()).isEqualTo("VIEWING"); assertThat(a.heartbeat("user-0", "Person 0", "tab-0", "orders", "123", "EDITING", "status", 4L)).isTrue(); assertThat(b.snapshot("orders", "123").getFirst().mode()).isEqualTo("EDITING"); - var tenantB = new RedisPresenceStore(second.scripts(), new RedisNamespace("test", "app", "tenant-b", "presence"), Duration.ofSeconds(45), 100); assertThat(tenantB.snapshot("orders", "123")).isEmpty(); assertThat(a.snapshot("orders", "124")).isEmpty(); @@ -250,6 +298,7 @@ class RedisPresenceStoreTest assertThat(a.snapshot("orders", "124")).isEmpty(); var shortLease = new RedisPresenceStore(first.scripts(), new RedisNamespace("test", "app", "short", "presence"), Duration.ofMillis(150), 10); + awaitReady(shortLease); assertThat(shortLease.heartbeat("user", "Person", "tab", "orders", "123", "EDITING", "status", 1L)).isTrue(); long deadline = System.nanoTime() + Duration.ofSeconds(3).toNanos(); while(!shortLease.snapshot("orders", "123").isEmpty() && System.nanoTime() < deadline) @@ -278,7 +327,8 @@ class RedisPresenceStoreTest try(var connection = new RedisConnection(new HostAndPort(redis.getHost(), redis.getMappedPort(6379)), DefaultJedisClientConfig.builder().build(), 2, Duration.ofMillis(500))) { - var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "app", "capacity", "presence"), Duration.ofSeconds(45), 1); + var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "app", "capacity", "presence"), Duration.ofSeconds(2), 1); + awaitReady(store); assertThat(store.heartbeat("one", "One", "tab", "orders", "1", "VIEWING", null, 1L)).isTrue(); assertThat(store.heartbeat("two", "Two", "tab", "orders", "2", "VIEWING", null, 1L)).isFalse(); assertThat(store.snapshot("orders", "2")).isEmpty(); @@ -303,33 +353,49 @@ class RedisPresenceStoreTest try(var connection = new RedisConnection(new HostAndPort(redis.getHost(), redis.getMappedPort(6379)), DefaultJedisClientConfig.builder().build(), 2, Duration.ofMillis(500))) { - for(String suffix : java.util.List.of("state", "expiry")) + for(String lostPart : java.util.List.of("record:state", "record:expiry", "table:state", "table:expiry")) { - var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "app", suffix, "presence"), Duration.ofSeconds(45), 10); - assertThat(store.heartbeat("one", "One", "tab", "orders", "1", "VIEWING", null, 1L)).isTrue(); + String[] parts = lostPart.split(":"); + boolean record = parts[0].equals("record"); + var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "app", parts[0] + parts[1], "presence"), Duration.ofSeconds(2), 10); + awaitReady(store); + String lease = store.openLease("one", "One", "orders", "1"); + assertThat(store.renewLease("one", "One", lease, "orders", "1", "VIEWING", null, 1L)).isTrue(); // KEYS is confined to this disposable test server, never the application path. - String recordKey = connection.client().keys("*:record:*:" + suffix).iterator().next(); - String tableKey = recordKey.substring(0, recordKey.indexOf(":record:")) + ":table:state"; - var before = connection.client().hgetAll(tableKey); - connection.client().del(recordKey); + String lostKey = connection.client().keys(record ? "*:record:*:" + parts[1] : "*:table:" + parts[1]).iterator().next(); + String retainedState = connection.client().keys(record ? "*:table:state" : "*:record:*:state").iterator().next(); + var before = connection.client().hgetAll(retainedState); + connection.client().del(lostKey); assertThatThrownBy(() -> store.snapshot("orders", "1")) .isInstanceOf(redis.clients.jedis.exceptions.JedisException.class).hasMessageContaining("incomplete"); - assertThatThrownBy(() -> store.heartbeat("one", "One", "tab", "orders", "1", "EDITING", "status", 2L)) + assertThatThrownBy(() -> store.renewLease("one", "One", lease, "orders", "1", "EDITING", "status", 2L)) .isInstanceOf(redis.clients.jedis.exceptions.JedisException.class).hasMessageContaining("incomplete"); - assertThat(connection.client().hgetAll(tableKey)).isEqualTo(before); + assertThat(connection.client().hgetAll(retainedState)).isEqualTo(before); connection.client().flushDB(); } for(String room : java.util.List.of("record", "table")) { - var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "app", room, "presence"), Duration.ofSeconds(45), 10); - store.heartbeat("one", "One", "tab", "orders", "1", "VIEWING", null, 1L); - store.leave("one", "tab", "orders", "1", 3L); + var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "app", room, "presence"), Duration.ofSeconds(2), 10); + awaitReady(store); + String lease = store.openLease("one", "One", "orders", "1"); + store.renewLease("one", "One", lease, "orders", "1", "VIEWING", null, 1L); + store.leave("one", lease, "orders", "1", 3L); var lostKeys = connection.client().keys(room.equals("record") ? "*:record:*" : "*:table:*"); connection.client().del(lostKeys.toArray(String[]::new)); - assertThatThrownBy(() -> store.snapshot("orders", "1")) - .isInstanceOf(redis.clients.jedis.exceptions.JedisException.class).hasMessageContaining("incomplete"); - assertThatThrownBy(() -> store.heartbeat("one", "One", "tab", "orders", "1", "EDITING", null, 2L)) - .isInstanceOf(redis.clients.jedis.exceptions.JedisException.class).hasMessageContaining("incomplete"); + if(room.equals("record")) + { + assertThatThrownBy(() -> store.snapshot("orders", "1")) + .isInstanceOf(redis.clients.jedis.exceptions.JedisException.class).hasMessageContaining("incomplete"); + assertThatThrownBy(() -> store.renewLease("one", "One", lease, "orders", "1", "EDITING", null, 2L)) + .isInstanceOf(redis.clients.jedis.exceptions.JedisException.class).hasMessageContaining("incomplete"); + } + else + { + assertThatThrownBy(() -> store.snapshot("orders", "1")) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException.class); + assertThatThrownBy(() -> store.renewLease("one", "One", lease, "orders", "1", "EDITING", null, 2L)) + .isInstanceOf(com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException.class); + } assertThat(connection.client().keys(room.equals("record") ? "*:record:*" : "*:table:*")).isEmpty(); connection.client().flushDB(); } diff --git a/src/test/java/com/kingsrook/qbits/collaboration/TagsServiceTest.java b/src/test/java/com/kingsrook/qbits/collaboration/TagsServiceTest.java index 85ca704..63348c5 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/TagsServiceTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/TagsServiceTest.java @@ -73,6 +73,126 @@ public class TagsServiceTest extends PersonalServiceFixture private final TagsService tags = new TagsService(); + /******************************************************************************* + ** Tag labels are selectable while the ID-based filter picker remains intact. + *******************************************************************************/ + @Test + void selectsTagLabelsWithoutTranslatingThemAsIds() throws Exception + { + TagEntry beta = tags.create("document", "Beta", null); + TagEntry alpha = tags.create("document", "Alpha", null); + tags.apply("document", 1, List.of(beta.id(), alpha.id()), List.of()); + com.kingsrook.qqq.backend.core.actions.collaboration.RuntimeMetaDataOverlay.refresh(instance); + var field = instance.getTable("document").getVirtualField("collaborationTags"); + assertThat(field.getIsQuerySelectable()).isTrue(); + assertThat(field.getPossibleValueSourceName()).isEqualTo(CollaborationTables.TAG); + var rows = new QueryAction().execute(new QueryInput("document").withFieldNamesToInclude(java.util.Set.of("collaborationTags")) + .withShouldTranslatePossibleValues(true).withShouldGenerateDisplayValues(true)).getRecords(); + assertThat(rows).hasSize(2); + assertThat(rows).allSatisfy(row -> assertThat(row.getValues()).containsOnlyKeys("collaborationTags")); + assertThat(rows).extracting(row -> row.getValueString("collaborationTags")).containsExactlyInAnyOrder("Alpha, Beta", null); + assertThat(rows).filteredOn(row -> row.getValue("collaborationTags") != null) + .allSatisfy(row -> assertThat(row.getDisplayValue("collaborationTags")).isEqualTo("Alpha, Beta")); + assertThat(new QueryAction().execute(new QueryInput("document").withFilter(new QQueryFilter() + .withCriteria("collaborationTags", QCriteriaOperator.EQUALS, alpha.id()))).getRecords()) + .extracting(row -> row.getValueInteger("id")).containsExactly(1); + } + + /******************************************************************************* + ** Exported labels obey parent visibility and vocabulary approval, with CSV quoting. + *******************************************************************************/ + @Test + void tagLabelsRespectPrivacyAndExportQuoting() throws Exception + { + TagEntry visible = tags.create("document", "Needs, review", null); + TagEntry secret = tags.create("document", "Secret label", null); + tags.apply("document", 1, List.of(visible.id()), List.of()); + QContext.getQSession().withSecurityKeyValue("allTenants", true); + tags.apply("document", 3, List.of(secret.id()), List.of()); + config.withDefaultTagMode("APPROVED_ONLY"); + TagEntry pending = tags.create("document", "Pending label", null); + assertThat(pending.approved()).isFalse(); + // A stale assignment must not turn a pending suggestion into a visible label. + CollaborationRecords.insert(CollaborationTables.RECORD_TAG, new QRecord().withValue("tableName", "document") + .withValue("recordId", "1").withValue("tagId", pending.id()).withValue("createdByUserId", "alice").withValue("parentDeleted", false)); + login("alice"); + com.kingsrook.qqq.backend.core.actions.collaboration.RuntimeMetaDataOverlay.refresh(instance); + var input = new QueryInput("document").withFieldNamesToInclude(java.util.Set.of("collaborationTags")); + assertThat(new QueryAction().execute(input).getRecords()).extracting(row -> row.getValueString("collaborationTags")) + .containsExactlyInAnyOrder("Needs, review", null); + + java.io.ByteArrayOutputStream output = new java.io.ByteArrayOutputStream(); + var export = new com.kingsrook.qqq.backend.core.model.actions.reporting.ExportInput(); + export.setTableName("document"); + export.setFieldNames(List.of("collaborationTags")); + export.setReportDestination(new com.kingsrook.qqq.backend.core.model.actions.reporting.ReportDestination() + .withReportFormat(com.kingsrook.qqq.backend.core.model.actions.reporting.ReportFormat.CSV).withReportOutputStream(output)); + new com.kingsrook.qqq.backend.core.actions.reporting.ExportAction().execute(export); + String csv = output.toString(java.nio.charset.StandardCharsets.UTF_8); + assertThat(csv).doesNotContain("Secret label", "Pending label"); + try(var parser = org.apache.commons.csv.CSVParser.parse(csv, org.apache.commons.csv.CSVFormat.DEFAULT.withFirstRecordAsHeader())) + { + assertThat(parser.getHeaderNames()).containsExactly("Tags"); + assertThat(parser.getRecords()).extracting(row -> row.get("Tags")).containsExactlyInAnyOrder("Needs, review", ""); + } + QContext.getQSession().getSecurityKeyValues().put("tenant", List.of("b")); + assertThat(new QueryAction().execute(input).getRecords()).extracting(row -> row.getValueString("collaborationTags")).containsExactly("Secret label"); + login("alice"); + assertThat(new QueryAction().execute(input).getRecords()).extracting(row -> row.getValueString("collaborationTags")) + .containsExactlyInAnyOrder("Needs, review", null); + } + + /******************************************************************************* + ** A page spanning the provider's batch boundary retains its last parent values. + *******************************************************************************/ + @Test + void tagLabelsSpanMultipleParentBatches() throws Exception + { + java.util.List parents = new java.util.ArrayList<>(); + for(int id = 4; id <= 504; id++) + { + parents.add(new QRecord().withValue("id", id).withValue("title", "Document " + id).withValue("tenant", "a")); + } + assertThat(new InsertAction().execute(new InsertInput("document").withRecords(parents)).getRecords()) + .allSatisfy(row -> assertThat(row.getErrors()).isNullOrEmpty()); + TagEntry tag = tags.create("document", "Boundary", null); + tags.apply("document", 1, List.of(tag.id()), List.of()); + tags.apply("document", 504, List.of(tag.id()), List.of()); + com.kingsrook.qqq.backend.core.actions.collaboration.RuntimeMetaDataOverlay.refresh(instance); + var rows = new QueryAction().execute(new QueryInput("document").withFieldNamesToInclude(java.util.Set.of("id", "collaborationTags"))).getRecords(); + assertThat(rows).hasSize(503); + assertThat(rows).filteredOn(row -> row.getValue("collaborationTags") != null).extracting(row -> row.getValueInteger("id")) + .containsExactlyInAnyOrder(1, 504); + assertThat(rows).filteredOn(row -> row.getValue("collaborationTags") != null) + .allSatisfy(row -> assertThat(row.getValueString("collaborationTags")).isEqualTo("Boundary")); + } + + /******************************************************************************* + ** Stale links cannot expose merged vocabulary, other-table tags, or deleted links. + *******************************************************************************/ + @Test + void tagLabelsExcludeStaleAndOutOfScopeAssignments() throws Exception + { + login("admin", TagsService.MANAGE); + TagEntry global = tags.create(null, "Global", null); + TagEntry current = tags.create("document", "Current", null); + TagEntry merged = tags.create("document", "Merged old label", null); + TagEntry deleted = tags.create("document", "Deleted link label", null); + TagEntry other = tags.create("document", "Other table label", null); + CollaborationRecords.update(CollaborationTables.TAG, new QRecord().withValue("id", merged.id()).withValue("mergedIntoTagId", current.id())); + CollaborationRecords.update(CollaborationTables.TAG, new QRecord().withValue("id", other.id()).withValue("tableName", "different_table")); + for(TagEntry tag : List.of(global, current, merged, deleted, other)) + { + CollaborationRecords.insert(CollaborationTables.RECORD_TAG, new QRecord().withValue("tableName", "document").withValue("recordId", "1") + .withValue("tagId", tag.id()).withValue("createdByUserId", "admin").withValue("parentDeleted", tag.id().equals(deleted.id()))); + } + login("alice"); + com.kingsrook.qqq.backend.core.actions.collaboration.RuntimeMetaDataOverlay.refresh(instance); + var rows = new QueryAction().execute(new QueryInput("document").withFieldNamesToInclude(java.util.Set.of("collaborationTags"))).getRecords(); + assertThat(rows).extracting(row -> row.getValueString("collaborationTags")).containsExactlyInAnyOrder("Current, Global", null); + assertThat(com.kingsrook.qqq.backend.core.model.metadata.collaboration.RecordReferenceTableMetaData.of(instance.getTable(CollaborationTables.RECORD_TAG))).isNotNull(); + } + /******************************************************************************* ** Natural keys are locale-independent and validate after collapsing whitespace. *******************************************************************************/ diff --git a/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentTransactionsPostgresTest.java b/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentTransactionsPostgresTest.java new file mode 100644 index 0000000..042f15f --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentTransactionsPostgresTest.java @@ -0,0 +1,1267 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.collaboration.attachments; + + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.sql.Connection; +import java.sql.DriverManager; +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.collaboration.CollaborationSchema; +import com.kingsrook.qbits.collaboration.PersonalServiceFixture; +import com.kingsrook.qbits.objectstorage.api.PutRequest; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.actions.processes.RunProcessAction; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunProcessInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess; +import com.kingsrook.qqq.backend.module.postgres.model.metadata.PostgreSQLBackendMetaData; +import com.kingsrook.qqq.backend.module.rdbms.actions.RDBMSTransaction; +import com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.BeforeAll; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import org.testcontainers.containers.GenericContainer; +import org.testcontainers.containers.wait.strategy.Wait; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Real PostgreSQL transaction boundaries with deterministic storage fault injection. + *******************************************************************************/ +class AttachmentTransactionsPostgresTest extends PersonalServiceFixture +{ + private static GenericContainer postgres; + private RecordingStorage storage; + private AttachmentsService service; + private static AttachmentCleanupQueue processQueue; + + + /******************************************************************************* + ** An isolated database container is required; never use an application database. + *******************************************************************************/ + @BeforeAll + static void startPostgres() + { + postgres = new GenericContainer<>("postgres:17-alpine") + .withEnv("POSTGRES_PASSWORD", "attachment-fixture-only").withExposedPorts(5432) + .waitingFor(Wait.forLogMessage(".*database system is ready to accept connections.*", 2)); + postgres.start(); + } + + + /******************************************************************************* + ** Stop only the database owned by this fixture. + *******************************************************************************/ + @AfterAll + static void stopPostgres() + { + if(postgres != null) + { + postgres.stop(); + } + } + + + /******************************************************************************* + ** Reuse the authorization fixture, replacing its backend with real PostgreSQL. + *******************************************************************************/ + @Override + @BeforeEach + public void setUp() throws Exception + { + super.setUp(); + instance.getBackends().remove("memory"); + instance.addBackend(new PostgreSQLBackendMetaData().withName("memory").withHostName(postgres.getHost()) + .withPort(postgres.getMappedPort(5432)).withDatabaseName("postgres") + .withUsername("postgres").withPassword("attachment-fixture-only")); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("DROP SCHEMA public CASCADE; CREATE SCHEMA public"); + sql.execute(CollaborationSchema.postgresScript(config)); + sql.execute("CREATE TABLE document (id INTEGER PRIMARY KEY, title TEXT, tenant TEXT)"); + sql.execute("INSERT INTO document VALUES (1, 'Visible', 'a'), (3, 'Private', 'b')"); + } + storage = new RecordingStorage(); + service = new AttachmentsService(config, storage); + } + + + /******************************************************************************* + ** Close owner connections before the isolated schema is replaced. + *******************************************************************************/ + @Override + public void tearDown() + { + ConnectionManager.closeConnectionProviders(instance); + super.tearDown(); + } + + + /******************************************************************************* + ** An independently visible intent precedes external I/O, but active uploads + ** cannot be claimed by a cleanup worker on another database connection. + *******************************************************************************/ + @Test + void uploadIntentPrecedesPutAndProtectsActiveUpload() throws Exception + { + storage.beforePut = () -> + { + try + { + assertThat(pendingCount()).isEqualTo(1); + var context = new com.kingsrook.qqq.backend.core.context.CapturedContext(instance, + new com.kingsrook.qqq.backend.core.model.session.QSystemUserSession()); + QContext.withTemporaryContext(context, () -> assertThat(new AttachmentCleanupQueue(storage).retryPending()).isZero()); + assertThat(storage.deleteAttempts).hasValue(0); + } + catch(Exception e) + { + throw (new IllegalStateException(e)); + } + }; + upload(); + assertThat(rowCount()).isEqualTo(1); + assertThat(pendingCount()).isZero(); + assertThat(storage.objects).hasSize(1); + } + + + /******************************************************************************* + ** Rejecting durable intent creation must prevent all provider I/O. + *******************************************************************************/ + @Test + void rejectedUploadIntentPreventsProviderWrite() throws Exception + { + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("CREATE FUNCTION reject_upload_intent() RETURNS trigger LANGUAGE plpgsql AS $$ " + + "BEGIN RAISE EXCEPTION 'injected upload intent rejection'; END $$"); + sql.execute("CREATE TRIGGER reject_upload_intent BEFORE INSERT ON collaboration_attachment_cleanup " + + "FOR EACH ROW EXECUTE FUNCTION reject_upload_intent()"); + } + var puts = new java.util.concurrent.atomic.AtomicInteger(); + storage.beforePut = puts::incrementAndGet; + assertThatThrownBy(this::upload).isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QException.class); + assertThat(puts).hasValue(0); + assertThat(storage.objects).isEmpty(); + assertThat(storage.deleteAttempts).hasValue(0); + assertThat(rowCount()).isZero(); + assertThat(pendingCount()).isZero(); + assertThat(QContext.getQBackendTransaction()).isNull(); + } + + + /******************************************************************************* + ** Completed provider I/O remains protected until the outer owner commits. + *******************************************************************************/ + @Test + void outerTransactionProtectsUploadedBytesUntilCommit() throws Exception + { + try(var tx = new RDBMSTransaction(connect())) + { + QContext.setTransaction(tx); + upload(); + assertThat(rowCount()).isZero(); + assertThat(pendingCount()).isEqualTo(1); + var context = new com.kingsrook.qqq.backend.core.context.CapturedContext(instance, + new com.kingsrook.qqq.backend.core.model.session.QSystemUserSession()); + QContext.withTemporaryContext(context, () -> assertThat(new AttachmentCleanupQueue(storage).retryPending()).isZero()); + assertThat(storage.deleteAttempts).hasValue(0); + assertThat(storage.objects).hasSize(1); + tx.commit(); + QContext.clearTransaction(); + } + assertThat(rowCount()).isEqualTo(1); + assertThat(pendingCount()).isZero(); + assertThat(storage.objects).hasSize(1); + } + + + /******************************************************************************* + ** A separate JVM halts after durable provider bytes, without callbacks or finally. + *******************************************************************************/ + @Test + void abruptlyTerminatedJvmLeavesRecoverableUpload(@org.junit.jupiter.api.io.TempDir java.nio.file.Path directory) throws Exception + { + String classpath = System.getProperty("surefire.test.class.path", System.getProperty("java.class.path")); + java.nio.file.Path arguments = directory.resolve("java.args"); + java.nio.file.Files.writeString(arguments, "-cp\n\"" + classpath.replace('\\', '/') + "\"\n" + + UploadCrashProbe.class.getName() + "\n" + postgres.getHost() + "\n" + postgres.getMappedPort(5432) + + "\n\"" + directory.toString().replace('\\', '/') + "\"\n"); + java.nio.file.Path executable = java.nio.file.Path.of(System.getProperty("java.home"), "bin", + System.getProperty("os.name").startsWith("Windows") ? "java.exe" : "java"); + Process child = new ProcessBuilder(executable.toString(), "@" + arguments) + .redirectErrorStream(true).redirectOutput(directory.resolve("child.log").toFile()).start(); + try + { + assertThat(child.waitFor(45, java.util.concurrent.TimeUnit.SECONDS)).isTrue(); + assertThat(child.exitValue()).as(java.nio.file.Files.readString(directory.resolve("child.log"))).isEqualTo(73); + } + finally + { + if(child.isAlive()) + { + child.destroyForcibly(); + assertThat(child.waitFor(10, java.util.concurrent.TimeUnit.SECONDS)).isTrue(); + } + } + assertThat(rowCount()).isZero(); + assertThat(pendingCount()).isEqualTo(1); + java.nio.file.Path object = directory.resolve("uploaded-object"); + assertThat(java.nio.file.Files.readAllBytes(object)).containsExactly(1, 2, 3); + admin(); + assertThat(new AttachmentCleanupQueue(new CrashFileStorage(directory, false)).retryPending()).isEqualTo(1); + assertThat(java.nio.file.Files.exists(object)).isFalse(); + assertThat(pendingCount()).isZero(); + assertThat(new AttachmentCleanupQueue(new CrashFileStorage(directory, false)).retryPending()).isZero(); + } + + + /******************************************************************************* + ** Standalone child uses the real service against the parent's disposable DB. + *******************************************************************************/ + public static class UploadCrashProbe extends PersonalServiceFixture + { + /*************************************************************************** + ** Intentionally never returns: the provider halts this child JVM. + ***************************************************************************/ + public static void main(String[] args) throws Exception + { + var probe = new UploadCrashProbe(); + probe.setUp(); + probe.instance.getBackends().remove("memory"); + probe.instance.addBackend(new PostgreSQLBackendMetaData().withName("memory").withHostName(args[0]) + .withPort(Integer.parseInt(args[1])).withDatabaseName("postgres") + .withUsername("postgres").withPassword("attachment-fixture-only")); + new AttachmentsService(probe.config, new CrashFileStorage(java.nio.file.Path.of(args[2]), true)) + .upload("document", 1, "hello.txt", "text/plain", new ByteArrayInputStream(new byte[] { 1, 2, 3 }), null); + throw (new IllegalStateException("Crash fixture unexpectedly returned")); + } + } + + + /******************************************************************************* + ** One fixture-owned object persists across JVM death independently of SQL. + *******************************************************************************/ + private static class CrashFileStorage extends AttachmentStorage + { + private final java.nio.file.Path directory; + private final boolean haltAfterPut; + + /*************************************************************************** + ** Scope every file to the test runner's disposable directory. + ***************************************************************************/ + private CrashFileStorage(java.nio.file.Path directory, boolean haltAfterPut) + { + this.directory = directory; + this.haltAfterPut = haltAfterPut; + } + + /*************************************************************************** + ** Bound this fixture exactly as the normal injected provider. + ***************************************************************************/ + @Override + public long maxBytes(String name) + { + return (1024L); + } + + /*************************************************************************** + ** Persist bytes and their locator, then terminate without shutdown hooks. + ***************************************************************************/ + @Override + public void put(String name, PutRequest request) + { + try + { + java.nio.file.Files.writeString(directory.resolve("locator"), name + "/" + request.getKey()); + java.nio.file.Files.write(directory.resolve("uploaded-object"), request.getStream().readAllBytes()); + if(haltAfterPut) + { + Runtime.getRuntime().halt(73); + } + } + catch(IOException e) + { + throw (new IllegalStateException(e)); + } + } + + /*************************************************************************** + ** The test runner confirms the only writer JVM has exited before recovery. + ***************************************************************************/ + @Override + public boolean cleanupUpload(String name, String key) + { + delete(name, key); + return (!haltAfterPut); + } + + /*************************************************************************** + ** Verify recovery uses the original locator before deleting fixture bytes. + ***************************************************************************/ + @Override + public void delete(String name, String key) + { + try + { + assertThat(java.nio.file.Files.readString(directory.resolve("locator"))).isEqualTo(name + "/" + key); + java.nio.file.Files.deleteIfExists(directory.resolve("uploaded-object")); + } + catch(IOException e) + { + throw (new IllegalStateException(e)); + } + } + } + + + /******************************************************************************* + ** Losing the upload stack before metadata and rollback callbacks still leaves + ** independently committed intent for a later service instance to recover. + *******************************************************************************/ + @Test + void interruptedUploadSurvivesLostRollbackCallbacks() throws Exception + { + storage.afterPut = () -> { throw new AssertionError("injected interrupted upload"); }; + try(var tx = new RDBMSTransaction(connect()) + { + /*********************************************************************** + ** Simulate loss of process-local callbacks, not a killed-JVM test. + ***********************************************************************/ + @Override + public void addAfterRollbackCallback(Runnable callback) + { + } + }) + { + QContext.setTransaction(tx); + assertThatThrownBy(this::upload).isInstanceOf(AssertionError.class); + tx.rollback(); + QContext.clearTransaction(); + } + assertThat(rowCount()).isZero(); + assertThat(pendingCount()).isEqualTo(1); + assertThat(storage.objects).hasSize(1); + admin(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + assertThat(storage.objects).isEmpty(); + assertThat(pendingCount()).isZero(); + } + + + /******************************************************************************* + ** Later outer rollback removes both the SQL row and the uploaded object. + *******************************************************************************/ + @Test + void uploadOuterRollbackCompensatesObject() throws Exception + { + try(var tx = new RDBMSTransaction(connect())) + { + QContext.setTransaction(tx); + upload(); + assertThat(storage.objects).hasSize(1); + tx.rollback(); + QContext.clearTransaction(); + } + assertThat(rowCount()).isZero(); + assertThat(storage.objects).isEmpty(); + } + + + /******************************************************************************* + ** A rejected outer transaction cannot delete bytes from an existing attachment. + *******************************************************************************/ + @Test + void deleteOuterRollbackPreservesAttachment() throws Exception + { + AttachmentEntry entry = upload(); + try(var tx = new RDBMSTransaction(connect())) + { + QContext.setTransaction(tx); + service.delete(List.of(entry.id())); + tx.rollback(); + QContext.clearTransaction(); + } + assertThat(rowCount()).isEqualTo(1); + assertThat(storage.objects).hasSize(1); + assertThat(pendingCount()).isZero(); + } + + + /******************************************************************************* + ** A committed deletion removes metadata and bytes after the transaction commits. + *******************************************************************************/ + @Test + void deleteCommitRemovesAttachment() throws Exception + { + AttachmentEntry entry = upload(); + try(var tx = new RDBMSTransaction(connect())) + { + QContext.setTransaction(tx); + service.delete(List.of(entry.id())); + assertThat(storage.objects).hasSize(1); + tx.commit(); + QContext.clearTransaction(); + } + assertThat(rowCount()).isZero(); + assertThat(storage.objects).isEmpty(); + } + + + /******************************************************************************* + ** Database rejection must not leave a visible row whose bytes were already deleted. + *******************************************************************************/ + @Test + void metadataDeleteFailurePreservesDownloadableObject() throws Exception + { + AttachmentEntry entry = upload(); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("CREATE FUNCTION reject_attachment_delete() RETURNS trigger LANGUAGE plpgsql AS $$ " + + "BEGIN RAISE EXCEPTION 'injected metadata delete rejection'; END $$"); + sql.execute("CREATE TRIGGER reject_attachment_delete BEFORE DELETE ON collaboration_attachment " + + "FOR EACH ROW EXECUTE FUNCTION reject_attachment_delete()"); + } + assertThatThrownBy(() -> service.delete(List.of(entry.id()))).isInstanceOf(Exception.class); + assertThat(rowCount()).isEqualTo(1); + assertThat(storage.objects).as("a retained visible attachment must retain its bytes").hasSize(1); + assertThat(pendingCount()).isZero(); + } + + + /******************************************************************************* + ** A provider failure leaves a durable locator; a new worker can retry after restart. + *******************************************************************************/ + @Test + void storageDeleteFailureRetainsIntentAndRetrySucceeds() throws Exception + { + AttachmentEntry entry = upload(); + storage.failDelete = true; + assertThatThrownBy(() -> service.delete(List.of(entry.id()))).isInstanceOf(IllegalStateException.class); + assertThat(rowCount()).isZero(); + assertThat(pendingCount()).isEqualTo(1); + assertThat(storage.objects).hasSize(1); + storage.failDelete = false; + assertThatThrownBy(() -> new AttachmentCleanupQueue(storage).retryPending()) + .isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + admin(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + assertThat(rowCount()).isZero(); + assertThat(pendingCount()).isZero(); + assertThat(storage.objects).isEmpty(); + } + + + /******************************************************************************* + ** A database commit survives loss of every in-process post-commit callback. + *******************************************************************************/ + @Test + void committedIntentSurvivesLostCallback() throws Exception + { + AttachmentEntry entry = upload(); + try(var tx = new RDBMSTransaction(connect()) + { + /*********************************************************************** + ** Simulate process loss after commit, before any callback can execute. + ***********************************************************************/ + @Override + public void addAfterCommitCallback(Runnable callback) + { + } + }) + { + QContext.setTransaction(tx); + service.delete(List.of(entry.id())); + tx.commit(); + QContext.clearTransaction(); + } + assertThat(rowCount()).isZero(); + assertThat(pendingCount()).isEqualTo(1); + assertThat(storage.objects).hasSize(1); + admin(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + assertThat(storage.objects).isEmpty(); + assertThat(pendingCount()).isZero(); + } + + + /******************************************************************************* + ** Losing queue acknowledgement after deleting bytes must remain replayable. + *******************************************************************************/ + @Test + void queueAcknowledgementFailureIsReplayable() throws Exception + { + AttachmentEntry entry = upload(); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("CREATE FUNCTION reject_cleanup_ack() RETURNS trigger LANGUAGE plpgsql AS $$ " + + "BEGIN RAISE EXCEPTION 'injected cleanup acknowledgement rejection'; END $$"); + sql.execute("CREATE TRIGGER reject_cleanup_ack BEFORE DELETE ON collaboration_attachment_cleanup " + + "FOR EACH ROW EXECUTE FUNCTION reject_cleanup_ack()"); + } + assertThatThrownBy(() -> service.delete(List.of(entry.id()))).isInstanceOf(Exception.class); + assertThat(rowCount()).isZero(); + assertThat(storage.objects).isEmpty(); + assertThat(pendingCount()).isEqualTo(1); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("DROP TRIGGER reject_cleanup_ack ON collaboration_attachment_cleanup"); + } + admin(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isZero(); + assertThat(pendingCount()).isZero(); + } + + + /******************************************************************************* + ** Queue observations use independent committed database state. + *******************************************************************************/ + private int pendingCount() throws Exception + { + try(Connection connection = connect(); var sql = connection.createStatement(); + var rows = sql.executeQuery("SELECT COUNT(*) FROM collaboration_attachment_cleanup")) + { + rows.next(); + return (rows.getInt(1)); + } + } + + + /******************************************************************************* + ** Failed first-page entries must not starve later healthy cleanup requests. + *******************************************************************************/ + @Test + void failedBatchDoesNotStarveLaterRequests() throws Exception + { + try(Connection connection = connect(); var sql = connection.prepareStatement( + "INSERT INTO collaboration_attachment_cleanup (attachment_id, bucket_name, storage_key) VALUES (?, 'collaboration', ?)")) + { + for(int id = 1; id <= 101; id++) + { + String key = "queued/" + id; + sql.setInt(1, id); + sql.setString(2, key); + sql.executeUpdate(); + storage.objects.put("collaboration/" + key, new byte[] { 1 }); + if(id <= 100) + { + storage.failedKeys.add(key); + } + } + } + admin(); + assertThatThrownBy(() -> new AttachmentCleanupQueue(storage).retryPending()).isInstanceOf(Exception.class); + assertThat(storage.deleteAttempts.get()).isEqualTo(100); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + assertThat(storage.deleteAttempts.get()).isEqualTo(101); + assertThat(pendingCount()).isEqualTo(100); + assertThat(storage.objects).doesNotContainKey("collaboration/queued/101"); + storage.failedKeys.clear(); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("UPDATE collaboration_attachment_cleanup SET next_attempt_at = CURRENT_TIMESTAMP - INTERVAL '1 second'"); + } + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(100); + assertThat(pendingCount()).isZero(); + assertThat(storage.objects).isEmpty(); + } + + + /******************************************************************************* + ** Failed rollback compensation must preserve a durable locator for later cleanup. + *******************************************************************************/ + @Test + void failedUploadRollbackCompensationRemainsRetryable() throws Exception + { + try(var tx = new RDBMSTransaction(connect())) + { + QContext.setTransaction(tx); + upload(); + storage.failDelete = true; + tx.rollback(); + QContext.clearTransaction(); + } + assertThat(rowCount()).isZero(); + assertThat(storage.objects).hasSize(1); + assertThat(pendingCount()).isEqualTo(1); + storage.failDelete = false; + admin(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + assertThat(storage.objects).isEmpty(); + assertThat(pendingCount()).isZero(); + } + + + /******************************************************************************* + ** Upgrading the first queue shape preserves pending deletions and is repeatable. + *******************************************************************************/ + @Test + void cleanupSchemaUpgradePreservesPendingRequests() throws Exception + { + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("ALTER TABLE collaboration_attachment_cleanup DROP COLUMN next_attempt_at"); + sql.execute("ALTER TABLE collaboration_attachment_cleanup ALTER COLUMN attachment_id SET NOT NULL"); + sql.execute("CREATE UNIQUE INDEX uq_collaboration_attachment_cleanup_1 ON collaboration_attachment_cleanup (attachment_id)"); + sql.execute("INSERT INTO collaboration_attachment_cleanup (attachment_id, bucket_name, storage_key) VALUES (42, 'collaboration', 'pending/42')"); + sql.execute(CollaborationSchema.postgresScript(config)); + sql.execute(CollaborationSchema.postgresScript(config)); + sql.execute("INSERT INTO collaboration_attachment_cleanup (bucket_name, storage_key) VALUES ('collaboration', 'rollback/one'), ('collaboration', 'rollback/two')"); + } + assertThat(pendingCount()).isEqualTo(3); + } + + + /******************************************************************************* + ** A provider may store bytes and then fail before acknowledging the upload. + *******************************************************************************/ + @Test + void ambiguousUploadFailureRetainsCompensation() throws Exception + { + storage.failPutAfterWrite = true; + storage.failDelete = true; + assertThatThrownBy(this::upload).isInstanceOf(IllegalStateException.class) + .hasMessage("injected upload acknowledgement failure"); + assertThat(rowCount()).isZero(); + assertThat(storage.objects).hasSize(1); + assertThat(pendingCount()).isEqualTo(1); + storage.failDelete = false; + admin(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + assertThat(storage.objects).isEmpty(); + assertThat(pendingCount()).isZero(); + } + + + /******************************************************************************* + ** Object absence is not proof that a remote upload has stopped. + *******************************************************************************/ + @Test + void genericUploadCleanupDoesNotClaimSettlement() + { + java.util.concurrent.atomic.AtomicInteger deletes = new java.util.concurrent.atomic.AtomicInteger(); + AttachmentStorage uncertain = new AttachmentStorage() + { + @Override + public void delete(String name, String key) + { + deletes.incrementAndGet(); + } + }; + assertThat(uncertain.cleanupUpload("bucket", "key")).isFalse(); + assertThat(deletes.get()).isEqualTo(1); + } + + + /******************************************************************************* + ** Manual settlement retains an attributable receipt and does not retry it. + *******************************************************************************/ + @Test + void operatorSettlementRetainsReceiptAndStopsRetries() throws Exception + { + int id = settlementIntent(); + admin(); + new AttachmentCleanupQueue(storage).resolveUpload(id, " Provider case 123 confirms PUT completion. "); + assertThat(storage.objects).isEmpty(); + try(Connection connection = connect(); var sql = connection.createStatement(); + var row = sql.executeQuery("SELECT resolved_by, settlement_evidence, resolved_at FROM collaboration_attachment_cleanup WHERE id = " + id)) + { + assertThat(row.next()).isTrue(); + assertThat(row.getString(1)).isEqualTo("admin"); + assertThat(row.getString(2)).isEqualTo("Provider case 123 confirms PUT completion."); + assertThat(row.getTimestamp(3)).isNotNull(); + } + int attempts = storage.deleteAttempts.get(); + new AttachmentCleanupQueue(storage).resolveUpload(id, "A repeat must not overwrite the original receipt."); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isZero(); + assertThat(storage.deleteAttempts.get()).isEqualTo(attempts); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute(CollaborationSchema.postgresScript(config)); + try(var row = sql.executeQuery("SELECT settlement_evidence FROM collaboration_attachment_cleanup WHERE id = " + id)) + { + assertThat(row.next()).isTrue(); + assertThat(row.getString(1)).isEqualTo("Provider case 123 confirms PUT completion."); + } + } + } + + + /******************************************************************************* + ** Deny unauthenticated, non-admin and scheduled callers before provider I/O. + *******************************************************************************/ + @Test + void operatorSettlementRequiresIdentityPermissionAndEvidence() throws Exception + { + int id = settlementIntent(); + AttachmentCleanupQueue queue = new AttachmentCleanupQueue(storage); + assertThatThrownBy(() -> queue.resolveUpload(id, "confirmed")).isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + QContext.init(instance, new com.kingsrook.qqq.backend.core.model.session.QSystemUserSession()); + assertThatThrownBy(() -> queue.resolveUpload(id, "confirmed")).isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + admin(); + assertThatThrownBy(() -> queue.resolveUpload(id, " ")).hasMessageContaining("evidence"); + assertThatThrownBy(() -> queue.resolveUpload(id, "x".repeat(2001))).hasMessageContaining("evidence"); + assertThatThrownBy(() -> queue.resolveUpload(null, "confirmed")).hasMessageContaining("record ID"); + assertThat(storage.deleteAttempts.get()).isZero(); + assertThat(storage.objects).hasSize(1); + } + + + /******************************************************************************* + ** A locked upload cannot be settled by an administrator on another connection. + *******************************************************************************/ + @Test + void operatorSettlementSkipsActiveOwner() throws Exception + { + int id = settlementIntent(); + admin(); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + connection.setAutoCommit(false); + sql.executeQuery("SELECT id FROM collaboration_attachment_cleanup WHERE id = " + id + " FOR UPDATE").close(); + assertThatThrownBy(() -> new AttachmentCleanupQueue(storage).resolveUpload(id, "confirmed")) + .hasMessageContaining("active or no longer exists"); + connection.rollback(); + } + assertThat(storage.deleteAttempts.get()).isZero(); + assertThat(QContext.getQBackendTransaction()).isNull(); + } + + + /******************************************************************************* + ** Even an admin cannot delete a referenced locator or settle normal deletions. + *******************************************************************************/ + @Test + void operatorSettlementProtectsReferencedObjects() throws Exception + { + var attachment = upload(); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.executeUpdate("INSERT INTO collaboration_attachment_cleanup (bucket_name, storage_key) SELECT bucket_name, storage_key FROM collaboration_attachment"); + } + admin(); + int id; + try(Connection connection = connect(); var sql = connection.createStatement(); var row = sql.executeQuery("SELECT id FROM collaboration_attachment_cleanup")) + { + row.next(); + id = row.getInt(1); + } + assertThatThrownBy(() -> new AttachmentCleanupQueue(storage).resolveUpload(id, "confirmed")) + .hasMessageContaining("still references"); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.executeUpdate("UPDATE collaboration_attachment_cleanup SET attachment_id = " + attachment.id()); + } + assertThatThrownBy(() -> new AttachmentCleanupQueue(storage).resolveUpload(id, "confirmed")) + .hasMessageContaining("normal cleanup retry"); + assertThat(storage.deleteAttempts.get()).isZero(); + assertThat(storage.objects).hasSize(1); + } + + + /******************************************************************************* + ** Provider failure or rejected receipt persistence leaves a retryable locator. + *******************************************************************************/ + @Test + void operatorSettlementFailureRetainsIntent() throws Exception + { + int id = settlementIntent(); + admin(); + storage.failDelete = true; + assertThatThrownBy(() -> new AttachmentCleanupQueue(storage).resolveUpload(id, "confirmed")).isInstanceOf(Exception.class); + assertThat(storage.objects).hasSize(1); + storage.failDelete = false; + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("CREATE FUNCTION reject_receipt() RETURNS trigger LANGUAGE plpgsql AS 'BEGIN RAISE EXCEPTION ''receipt failure''; END'; CREATE TRIGGER reject_receipt BEFORE UPDATE ON collaboration_attachment_cleanup FOR EACH ROW EXECUTE FUNCTION reject_receipt()"); + } + assertThatThrownBy(() -> new AttachmentCleanupQueue(storage).resolveUpload(id, "confirmed")).isInstanceOf(Exception.class); + assertThat(storage.objects).isEmpty(); + try(Connection connection = connect(); var sql = connection.createStatement(); var row = sql.executeQuery("SELECT resolved_at FROM collaboration_attachment_cleanup WHERE id = " + id)) + { + assertThat(row.next()).isTrue(); + assertThat(row.getTimestamp(1)).isNull(); + } + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("DROP TRIGGER reject_receipt ON collaboration_attachment_cleanup"); + } + new AttachmentCleanupQueue(storage).resolveUpload(id, "confirmed after database recovery"); + assertThat(pendingCount()).isEqualTo(1); + assertThat(QContext.getQBackendTransaction()).isNull(); + } + + + /******************************************************************************* + ** Fixture-only orphan, after its writer has definitively stopped. + *******************************************************************************/ + private int settlementIntent() throws Exception + { + storage.objects.put("fixture/settled", new byte[] { 1, 2, 3 }); + try(Connection connection = connect(); var sql = connection.createStatement(); var row = sql.executeQuery( + "INSERT INTO collaboration_attachment_cleanup (bucket_name, storage_key) VALUES ('fixture', 'settled') RETURNING id")) + { + row.next(); + return (row.getInt(1)); + } + } + + + /******************************************************************************* + ** Genuine form, Review and Result boundaries; revoked access is checked again. + *******************************************************************************/ + @ParameterizedTest + @ValueSource(strings = { "success", "revoked", "storageFailure" }) + @SuppressWarnings("unchecked") + void operatorSettlementProcess(String scenario) throws Exception + { + int id = settlementIntent(); + int deletionId; + try(Connection connection = connect(); var sql = connection.createStatement(); var row = sql.executeQuery( + "INSERT INTO collaboration_attachment_cleanup (attachment_id, bucket_name, storage_key) VALUES (99, 'fixture', 'normal-deletion') RETURNING id")) + { + row.next(); + deletionId = row.getInt(1); + } + var process = instance.getProcess(ResolveAttachmentUploadProcess.NAME); + process.getBackendStep(StreamedETLWithFrontendProcess.STEP_NAME_EXECUTE).getInputMetaData() + .getFieldThrowing(StreamedETLWithFrontendProcess.FIELD_LOAD_CODE).setDefaultValue(new QCodeReference(FixtureSettlement.class)); + processQueue = new AttachmentCleanupQueue(storage); + try + { + RunProcessInput start = new RunProcessInput(); + start.setProcessName(ResolveAttachmentUploadProcess.NAME); + start.addValue("recordsParam", "recordIds"); + start.addValue("recordIds", id + "," + deletionId); + start.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.BREAK); + assertThatThrownBy(() -> new RunProcessAction().execute(start)) + .isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + admin(); + var form = new RunProcessAction().execute(start); + assertThat(form.getException()).isEmpty(); + start.setProcessUUID(form.getProcessUUID()); + start.setStartAfterStep("evidence"); + start.addValue(ResolveAttachmentUploadProcess.EVIDENCE, "Provider request 123 completed; writer stopped."); + var review = new RunProcessAction().execute(start); + assertThat(review.getException()).isEmpty(); + List reviewLines = (List) review.getValues().get(StreamedETLWithFrontendProcess.FIELD_VALIDATION_SUMMARY); + assertThat(reviewLines).hasSize(2); + assertThat(storage.deleteAttempts.get()).isZero(); + if(scenario.equals("revoked")) + { + login("admin"); + } + storage.failDelete = scenario.equals("storageFailure"); + RunProcessInput go = new RunProcessInput(); + go.setProcessName(ResolveAttachmentUploadProcess.NAME); + go.setProcessUUID(review.getProcessUUID()); + go.setStartAfterStep(StreamedETLWithFrontendProcess.STEP_NAME_REVIEW); + go.setFrontendStepBehavior(RunProcessInput.FrontendStepBehavior.SKIP); + if(scenario.equals("revoked")) + { + assertThatThrownBy(() -> new RunProcessAction().execute(go)) + .isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + assertThat(storage.deleteAttempts.get()).isZero(); + } + else + { + var result = new RunProcessAction().execute(go); + assertThat(result.getException()).isEmpty(); + List lines = (List) result.getValues().get(StreamedETLWithFrontendProcess.FIELD_PROCESS_SUMMARY); + assertThat(lines).anySatisfy(line -> + { + line.pickMessage(true); + assertThat(line.getMessage()).isEqualTo(scenario.equals("success") ? "uploads resolved" : "uploads unresolved; inspect and retry"); + assertThat(line.getCount()).isEqualTo(1); + }); + } + assertThat(storage.objects).hasSize(scenario.equals("success") ? 0 : 1); + } + finally + { + processQueue = null; + } + } + + + /******************************************************************************* + ** The real process uses the real queue with deterministic fixture storage. + *******************************************************************************/ + public static class FixtureSettlement extends ResolveAttachmentUploadProcess.Apply + { + /******************************************************************************* + ** Instantiated by the standard process loader. + *******************************************************************************/ + public FixtureSettlement() + { + super(processQueue); + } + } + + + /******************************************************************************* + ** A timed-out remote PUT can finish after rollback has already attempted DELETE. + *******************************************************************************/ + @Test + void lateUploadCompletionRemainsRecoverable() throws Exception + { + storage.deferPutUntilAfterFailure = true; + assertThatThrownBy(this::upload).isInstanceOf(IllegalStateException.class) + .hasMessage("injected upload timeout"); + assertThat(rowCount()).isZero(); + assertThat(storage.objects).isEmpty(); + + admin(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isZero(); + assertThat(pendingCount()).isEqualTo(1); + int attempts = storage.deleteAttempts.get(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isZero(); + assertThat(storage.deleteAttempts.get()).as("uncertain writes use persisted backoff").isEqualTo(attempts); + + // The remote service finishes after rollback and an independent cleanup sweep. + storage.deferredPut.run(); + assertThat(storage.objects).hasSize(1); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.executeUpdate("UPDATE collaboration_attachment_cleanup SET next_attempt_at = CURRENT_TIMESTAMP - INTERVAL '1 second'"); + } + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + assertThat(storage.objects).as("late provider bytes must remain discoverable for cleanup").isEmpty(); + assertThat(pendingCount()).isZero(); + } + + + /******************************************************************************* + ** Replay real provider deletes after the database rejects queue acknowledgement. + *******************************************************************************/ + @Test + void minioDeletionReplayPreservesOtherObjects() throws Exception + { + try(var minio = new GenericContainer<>("cgr.dev/chainguard/minio@sha256:dc455cdd4db6e390b1b26514f15ff37681a777bf8dab41aac445a0259f751e56") + .withEnv("MINIO_ROOT_USER", "collaboration-test").withEnv("MINIO_ROOT_PASSWORD", "collaboration-test-password") + .withCommand("server", "/tmp/minio-data").withExposedPorts(9000) + .waitingFor(Wait.forHttp("/minio/health/live").forPort(9000))) + { + minio.start(); + String endpoint = "http://" + minio.getHost() + ":" + minio.getMappedPort(9000); + try(var client = software.amazon.awssdk.services.s3.S3Client.builder() + .endpointOverride(java.net.URI.create(endpoint)).region(software.amazon.awssdk.regions.Region.US_EAST_1) + .forcePathStyle(true).credentialsProvider(software.amazon.awssdk.auth.credentials.StaticCredentialsProvider.create( + software.amazon.awssdk.auth.credentials.AwsBasicCredentials.create("collaboration-test", "collaboration-test-password"))).build()) + { + client.createBucket(b -> b.bucket("collaboration-plain")); + client.createBucket(b -> b.bucket("collaboration-versioned")); + client.putBucketVersioning(b -> b.bucket("collaboration-versioned").versioningConfiguration(v -> + v.status(software.amazon.awssdk.services.s3.model.BucketVersioningStatus.ENABLED))); + System.setProperty("collaboration.minio.access", "collaboration-test"); + System.setProperty("collaboration.minio.secret", "collaboration-test-password"); + var storageConfig = new com.kingsrook.qbits.objectstorage.ObjectStorageQBitConfig() + .withBackendName("memory") + .withAllowedSecretRefPatterns(List.of("prop.collaboration.minio.*")) + .withBucketScoping(false) + .withConnection(new com.kingsrook.qbits.objectstorage.config.StorageConnectionConfig("fixture") + .withEndpoint(endpoint).withRegion("us-east-1") + .withAddressingStyle(com.kingsrook.qbits.objectstorage.config.AddressingStyle.PATH) + .withChecksumMode(com.kingsrook.qbits.objectstorage.config.ChecksumMode.WHEN_REQUIRED) + .withAccessKeyRef("${prop.collaboration.minio.access}").withSecretKeyRef("${prop.collaboration.minio.secret}")); + for(String alias : List.of("plain", "versioned")) + { + storageConfig.withBucket(new com.kingsrook.qbits.objectstorage.config.StorageBucketConfig(alias) + .withConnection("fixture").withBucketName("collaboration-" + alias) + .withRole(com.kingsrook.qbits.objectstorage.config.BucketRole.PRIVATE).withIndexed(false)); + } + com.kingsrook.qbits.objectstorage.api.ObjectStorage.configureStandalone(storageConfig); + for(String alias : List.of("plain", "versioned")) + { + config.withAttachmentsBucketName(alias); + service = new AttachmentsService(config, new AttachmentStorage()); + AttachmentEntry removed = upload(); + AttachmentEntry retained = upload(); + String removedKey = storedKey(removed.id()); + String retainedKey = storedKey(retained.id()); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("CREATE OR REPLACE FUNCTION reject_provider_ack() RETURNS trigger LANGUAGE plpgsql AS $$ " + + "BEGIN RAISE EXCEPTION 'injected provider acknowledgement rejection'; END $$"); + sql.execute("CREATE TRIGGER reject_provider_ack BEFORE DELETE ON collaboration_attachment_cleanup " + + "FOR EACH ROW EXECUTE FUNCTION reject_provider_ack()"); + } + assertThatThrownBy(() -> service.delete(List.of(removed.id()))).isInstanceOf(Exception.class); + assertThat(pendingCount()).isEqualTo(1); + try(Connection connection = connect(); var sql = connection.createStatement()) + { + sql.execute("DROP TRIGGER reject_provider_ack ON collaboration_attachment_cleanup"); + } + admin(); + assertThat(new AttachmentCleanupQueue(new AttachmentStorage()).retryPending()).isEqualTo(1); + assertThat(pendingCount()).isZero(); + var bucket = com.kingsrook.qbits.objectstorage.api.ObjectStorage.asSystem("collaboration-test", "replay").bucket(alias); + assertThat(bucket.head(removedKey)).isEmpty(); + assertThat(bucket.head(retainedKey)).isPresent(); + assertThat(client.getObjectAsBytes(b -> b.bucket("collaboration-" + alias).key(retainedKey)).asByteArray()) + .containsExactly((byte) 1, (byte) 2, (byte) 3); + service.delete(List.of(retained.id())); + login("alice"); + } + } + } + finally + { + com.kingsrook.qbits.objectstorage.internal.ObjectStorageRuntime.reset(); + com.kingsrook.qbits.objectstorage.internal.ConfigSync.reset(); + System.clearProperty("collaboration.minio.access"); + System.clearProperty("collaboration.minio.secret"); + } + } + + + /******************************************************************************* + ** Observe the immutable provider key from committed SQL metadata. + *******************************************************************************/ + private String storedKey(int attachmentId) throws Exception + { + try(Connection connection = connect(); var sql = connection.prepareStatement("SELECT storage_key FROM collaboration_attachment WHERE id = ?")) + { + sql.setInt(1, attachmentId); + try(var rows = sql.executeQuery()) + { + assertThat(rows.next()).isTrue(); + return (rows.getString(1)); + } + } + } + + + /******************************************************************************* + ** Two workers may observe the same committed intent; replay must remain safe. + *******************************************************************************/ + @Test + void concurrentWorkersCanReplayTheSameIntent() throws Exception + { + AttachmentEntry entry = upload(); + storage.failDelete = true; + assertThatThrownBy(() -> service.delete(List.of(entry.id()))).isInstanceOf(IllegalStateException.class); + storage.failDelete = false; + storage.concurrentDeletes = new java.util.concurrent.CountDownLatch(1); + storage.allowDelete = new java.util.concurrent.CountDownLatch(1); + admin(); + var session = QContext.getQSession(); + try(var executor = java.util.concurrent.Executors.newFixedThreadPool(2)) + { + java.util.concurrent.Callable retry = () -> + { + QContext.init(instance, session); + try + { + return (new AttachmentCleanupQueue(storage).retryPending()); + } + finally + { + QContext.clear(); + } + }; + var first = executor.submit(retry); + assertThat(storage.concurrentDeletes.await(10, java.util.concurrent.TimeUnit.SECONDS)).isTrue(); + try + { + var second = executor.submit(retry); + assertThat(second.get(10, java.util.concurrent.TimeUnit.SECONDS)).isZero(); + } + finally + { + storage.allowDelete.countDown(); + } + assertThat(first.get(20, java.util.concurrent.TimeUnit.SECONDS)).isEqualTo(1); + } + assertThat(rowCount()).isZero(); + assertThat(pendingCount()).isZero(); + assertThat(storage.objects).isEmpty(); + } + + + /******************************************************************************* + ** Recovery locations are admin-only and cannot be forged through table writes. + *******************************************************************************/ + @Test + void cleanupQueueIsServiceManaged() throws Exception + { + var query = new com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput( + com.kingsrook.qbits.collaboration.model.CollaborationTables.ATTACHMENT_CLEANUP); + assertThatThrownBy(() -> new com.kingsrook.qqq.backend.core.actions.tables.QueryAction().execute(query)) + .isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + admin(); + assertThat(new com.kingsrook.qqq.backend.core.actions.tables.QueryAction().execute(query).getRecords()).isEmpty(); + var insert = new com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput( + com.kingsrook.qbits.collaboration.model.CollaborationTables.ATTACHMENT_CLEANUP) + .withRecords(List.of(new com.kingsrook.qqq.backend.core.model.data.QRecord())); + assertThatThrownBy(() -> new com.kingsrook.qqq.backend.core.actions.tables.InsertAction().execute(insert)) + .isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException.class); + } + + + /******************************************************************************* + ** Use the actual service and its caller's record permissions. + *******************************************************************************/ + private AttachmentEntry upload() throws Exception + { + return (service.upload("document", 1, "hello.txt", "text/plain", new ByteArrayInputStream(new byte[] { 1, 2, 3 }), null)); + } + + + /******************************************************************************* + ** Independent JDBC observation, outside the service transaction. + *******************************************************************************/ + private int rowCount() throws Exception + { + try(Connection connection = connect(); var sql = connection.createStatement(); + var rows = sql.executeQuery("SELECT COUNT(*) FROM collaboration_attachment")) + { + rows.next(); + return (rows.getInt(1)); + } + } + + + /******************************************************************************* + ** Connect only to this disposable fixture's mapped port. + *******************************************************************************/ + private static Connection connect() throws Exception + { + return (DriverManager.getConnection("jdbc:postgresql://" + postgres.getHost() + ":" + + postgres.getMappedPort(5432) + "/postgres", "postgres", "attachment-fixture-only")); + } + + + /******************************************************************************* + ** Preserve bytes and inject provider failure independently of PostgreSQL. + *******************************************************************************/ + private static class RecordingStorage extends AttachmentStorage + { + private final Map objects = new java.util.concurrent.ConcurrentHashMap<>(); + private boolean failDelete; + private boolean failPutAfterWrite; + private boolean deferPutUntilAfterFailure; + private Runnable deferredPut; + private Runnable beforePut; + private Runnable afterPut; + private final java.util.Set failedKeys = new java.util.HashSet<>(); + private final java.util.concurrent.atomic.AtomicInteger deleteAttempts = new java.util.concurrent.atomic.AtomicInteger(); + private java.util.concurrent.CountDownLatch concurrentDeletes; + private java.util.concurrent.CountDownLatch allowDelete; + + /*************************************************************************** + ** Keep the production service's upload bound active. + ***************************************************************************/ + @Override + public long maxBytes(String name) + { + return (1024L); + } + + /*************************************************************************** + ** Consume the same upload stream a real provider receives. + ***************************************************************************/ + @Override + public void put(String name, PutRequest request) + { + try + { + if(beforePut != null) + { + beforePut.run(); + } + byte[] bytes = request.getStream().readAllBytes(); + if(deferPutUntilAfterFailure) + { + deferredPut = () -> + { + objects.put(name + "/" + request.getKey(), bytes); + deferPutUntilAfterFailure = false; + }; + throw (new IllegalStateException("injected upload timeout")); + } + objects.put(name + "/" + request.getKey(), bytes); + if(afterPut != null) + { + afterPut.run(); + } + if(failPutAfterWrite) + { + throw (new IllegalStateException("injected upload acknowledgement failure")); + } + } + catch(IOException e) + { + throw (new IllegalStateException(e)); + } + } + + /*************************************************************************** + ** This synchronous fixture knows whether its deferred PUT has completed. + ***************************************************************************/ + @Override + public boolean cleanupUpload(String name, String key) + { + delete(name, key); + return (!deferPutUntilAfterFailure); + } + + /*************************************************************************** + ** Simulate a provider refusing deletion without losing stored bytes. + ***************************************************************************/ + @Override + public void delete(String name, String key) + { + deleteAttempts.incrementAndGet(); + if(concurrentDeletes != null) + { + concurrentDeletes.countDown(); + try + { + if(!allowDelete.await(20, java.util.concurrent.TimeUnit.SECONDS)) + { + throw (new IllegalStateException("Timed out waiting to release the cleanup owner")); + } + } + catch(InterruptedException e) + { + Thread.currentThread().interrupt(); + throw (new IllegalStateException(e)); + } + } + if(failDelete || failedKeys.contains(key)) + { + throw (new IllegalStateException("injected storage delete failure")); + } + objects.remove(name + "/" + key); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentsServiceTest.java b/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentsServiceTest.java index 1b0d7f1..d8b4dea 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentsServiceTest.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/attachments/AttachmentsServiceTest.java @@ -383,7 +383,7 @@ public class AttachmentsServiceTest extends PersonalServiceFixture } /******************************************************************************* - ** Storage failures leave rows for retries; trusted cleanup works after parent deletion. + ** Storage failures leave durable cleanup; trusted cleanup works after parent deletion. *******************************************************************************/ @Test void failuresAndCleanup() throws Exception @@ -391,8 +391,13 @@ public class AttachmentsServiceTest extends PersonalServiceFixture AttachmentEntry row = upload("file.txt", "x"); storage.failDelete = true; assertThatThrownBy(() -> service.delete(List.of(row.id()))).hasMessage("Storage unavailable"); - assertThat(service.list("document", 1)).hasSize(1); + assertThat(service.list("document", 1)).isEmpty(); + assertThat(storage.objects).hasSize(1); storage.failDelete = false; + admin(); + assertThat(new AttachmentCleanupQueue(storage).retryPending()).isEqualTo(1); + login("alice"); + upload("parent-cleanup.txt", "parent bytes"); new DeleteAction().execute(new DeleteInput("document").withPrimaryKeys(List.of(1))); assertThatThrownBy(() -> service.list("document", 1)).hasMessage("Not found"); QContext.getQSession().getPermissions().clear(); @@ -400,6 +405,8 @@ public class AttachmentsServiceTest extends PersonalServiceFixture cleanup.onParentsDeleted("document", List.of()); cleanup.onParentsDeleted("document", List.of(1)); cleanup.onParentsDeleted("document", List.of(1)); + admin(); + new AttachmentCleanupQueue(storage).retryPending(); assertThat(storage.objects).isEmpty(); assertThat(AttachmentRecords.query(new QQueryFilter())).isEmpty(); } diff --git a/src/test/java/com/kingsrook/qbits/collaboration/devserver/CapacityPresenceDevServer.java b/src/test/java/com/kingsrook/qbits/collaboration/devserver/CapacityPresenceDevServer.java new file mode 100644 index 0000000..1d2752d --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/collaboration/devserver/CapacityPresenceDevServer.java @@ -0,0 +1,174 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + + +package com.kingsrook.qbits.collaboration.devserver; + +import com.kingsrook.qbits.collaboration.CollaborationQBitConfig; +import com.kingsrook.qbits.collaboration.presence.PresenceRegistry; +import com.kingsrook.qbits.collaboration.presence.PresenceScopeProvider; +import com.kingsrook.qbits.collaboration.presence.RedisPresenceConfig; +import com.kingsrook.qbits.redis.RedisConnectionConfig; +import com.kingsrook.qbits.redis.RedisQBitConfig; +import com.kingsrook.qbits.redis.RedisQBitProducer; +import com.kingsrook.qbits.redis.RedisRealtimeRuntimeService; +import com.kingsrook.qbits.redis.RedisRuntimeService; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.realtime.RealtimeBus; + +/******************************************************************************* + ** Loopback-only browser acceptance node. Run separate JVMs against one disposable + ** Redis endpoint. Record data is independently seeded, not a shared database. + *******************************************************************************/ +public class CapacityPresenceDevServer +{ + /*************************************************************************** + ** Arguments are HTTP port and disposable local Redis port. Never deploy this + ** fixture: it uses the demo's mock authentication and plaintext loopback Redis. + ***************************************************************************/ + public static void main(String[] args) throws Exception + { + if(args.length != 2) + { + throw (new IllegalArgumentException("Expected HTTP port and local Redis port")); + } + int port = Integer.parseInt(args[0]); + int redisPort = Integer.parseInt(args[1]); + var instance = CollaborationDevServer.buildInstance(port); + CollaborationDevServer.seed(instance); + instance.withInstanceDefaultAuthentication(new com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData() + .withName("mock").withType(com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType.MOCK) + .withCustomizer(new QCodeReference(AcceptanceSessions.class))); + PresenceRegistry.close(instance); + RealtimeBus.close(instance); + new RedisQBitProducer().withQBitConfig(new RedisQBitConfig().withConnection("presence", + new RedisConnectionConfig().withHost(System.getProperty("collab.capacity.redisHost", "127.0.0.1")).withPort(redisPort).withTls(false).withMaximumConnections(8).withMaximumWaitMillis(500)) + .withRealtime("presence", "acceptance", "collaboration")).produce(instance); + instance.getQBits().values().stream().map(bit -> bit.getConfig()) + .filter(CollaborationQBitConfig.class::isInstance).map(CollaborationQBitConfig.class::cast) + .findFirst().orElseThrow().withRedisPresence(new RedisPresenceConfig().withConnectionName("presence") + .withDeployment("acceptance").withApplication("collaboration") + .withScopeProvider(new QCodeReference(FixtureTenant.class))); + var redis = new RedisRuntimeService(); + var transport = new RedisRealtimeRuntimeService(); + new com.kingsrook.qqq.backend.core.health.HealthMetaDataProvider().defineAll(instance, "memory", null); + var health = new com.kingsrook.qqq.backend.core.health.HealthRunner(instance, "acceptance-" + port, java.time.Clock.systemUTC()); + var server = CollaborationDevServer.server(instance, port, false); + var sampler = java.util.concurrent.Executors.newSingleThreadScheduledExecutor(); + try + { + redis.start(instance); + transport.start(instance); + health.start(); + server.start(); + sampler.scheduleAtFixedRate(() -> + { + var pool = RedisRuntimeService.connection(instance, "presence").metrics(); + var heap = java.lang.management.ManagementFactory.getMemoryMXBean().getHeapMemoryUsage(); + var values = new java.util.LinkedHashMap(); + values.put("uptimeMillis", java.lang.management.ManagementFactory.getRuntimeMXBean().getUptime()); + values.put("cpuNanos", ProcessHandle.current().info().totalCpuDuration().map(java.time.Duration::toNanos).orElse(-1L)); + values.put("heapUsedBytes", heap.getUsed()); + values.put("heapCommittedBytes", heap.getCommitted()); + values.put("residentBytes", residentBytes()); + values.put("poolActive", pool.active()); + values.put("poolIdle", pool.idle()); + values.put("poolWaiting", pool.waiting()); + values.put("poolMaximum", pool.maximum()); + System.out.println("CAPACITY_METRIC " + com.kingsrook.qqq.backend.core.utils.JsonUtils.toJson(values)); + }, 0, 100, java.util.concurrent.TimeUnit.MILLISECONDS); + System.out.println("CAPACITY_PRESENCE_READY " + port); + Thread.currentThread().join(); + } + finally + { + sampler.shutdownNow(); + server.stop(); + health.stop(); + transport.stop(); + PresenceRegistry.close(instance); + RealtimeBus.close(instance); + redis.stop(); + redis.afterApplicationStop(); + } + } + + /*************************************************************************** + ** Read Linux process resident memory; unsupported platforms report -1. + ***************************************************************************/ + private static long residentBytes() + { + try + { + for(String line : java.nio.file.Files.readAllLines(java.nio.file.Path.of("/proc/self/status"))) + { + if(line.startsWith("VmRSS:")) + { + return (Long.parseLong(line.trim().split("\\s+")[1]) * 1024L); + } + } + } + catch(java.io.IOException | NumberFormatException e) + { + return (-1L); + } + return (-1L); + } + + /*************************************************************************** + ** Tenant comes from the configured mock identity, never a request tenant parameter. + ***************************************************************************/ + public static class FixtureTenant implements PresenceScopeProvider + { + /************************************************************************ + ** Identical record IDs are isolated by the authenticated fixture identity. + ************************************************************************/ + @Override + public String tenantScope() + { + String identity = com.kingsrook.qqq.backend.core.context.QContext.getQSession().getUser().getIdReference(); + if(!identity.matches("load-t[0-7]-u[0-9]{1,2}")) + { + throw (new IllegalStateException("Capacity presence requires a configured load identity")); + } + return (identity.substring(0, identity.indexOf("-u"))); + } + } + + /*************************************************************************** + ** Grant Redis dashboard access only to the fixture's existing administrator. + ***************************************************************************/ + public static class AcceptanceSessions extends CollaborationDevServer.DemoSessions + { + /************************************************************************ + ** Keep normal demo identity/grants and add the service-view permission. + ************************************************************************/ + @Override + public void customizeSession(com.kingsrook.qqq.backend.core.model.metadata.QInstance instance, + com.kingsrook.qqq.backend.core.model.session.QSession target, java.util.Map context) + { + super.customizeSession(instance, target, context); + if("admin".equals(target.getUser().getIdReference())) + { + target.withPermissions("redisView.hasAccess", "health.read"); + } + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/collaboration/devserver/CollaborationDevServer.java b/src/test/java/com/kingsrook/qbits/collaboration/devserver/CollaborationDevServer.java index 752c826..16bed25 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/devserver/CollaborationDevServer.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/devserver/CollaborationDevServer.java @@ -123,11 +123,35 @@ import io.javalin.config.JavalinConfig; public class CollaborationDevServer extends AbstractQQQApplication { private final QInstance instance; - static final List USERS = List.of("alice", "bob", "admin"); + static final List USERS = fixtureUsers(); static final String USER_COOKIE = "collab_dev_user"; private static final ThreadLocal REQUEST_USER = new ThreadLocal<>(); + /******************************************************************************* + ** Opt-in bounded identities for the disposable capacity fixture only. + *******************************************************************************/ + private static List fixtureUsers() + { + int tenants = Integer.getInteger("collab.capacity.tenants", 0); + int users = Integer.getInteger("collab.capacity.users", 0); + if(tenants < 0 || tenants > 8 || users < 0 || users > 100) + { + throw (new IllegalArgumentException("Capacity fixture permits at most 8 tenants and 100 users each")); + } + var identities = new java.util.ArrayList<>(List.of("alice", "bob", "admin")); + for(int tenant = 0; tenant < tenants; tenant++) + { + for(int user = 0; user < users; user++) + { + identities.add("load-t" + tenant + "-u" + user); + } + } + return (List.copyOf(identities)); + } + + + /******************************************************************************* ** Keep the already seeded instance when Javalin initializes the application. @@ -186,7 +210,7 @@ public class CollaborationDevServer extends AbstractQQQApplication .withServeLegacyUnversionedMiddlewareAPI(false).withStrictUiApiRoutes(true) .withJavalinConfigCustomizer(config -> { - config.jetty.host = "127.0.0.1"; + config.jetty.host = System.getProperty("collab.dev.bindAddress", "127.0.0.1"); userSelection(config); MemoryStorage.routes(config); })); @@ -332,7 +356,7 @@ public class CollaborationDevServer extends AbstractQQQApplication { case "alice" -> "Alice Morgan"; case "bob" -> "Bob Chen"; - default -> "Demo Administrator"; + default -> user.equals("admin") ? "Demo Administrator" : user; }; QSession session = new QSession().withUser(new QUser().withIdReference(user).withFullName(name)) .withPermissions("person.read", "person.comment") diff --git a/src/test/java/com/kingsrook/qbits/collaboration/devserver/SharedPresenceDevServer.java b/src/test/java/com/kingsrook/qbits/collaboration/devserver/SharedPresenceDevServer.java index 2f618b0..8530b32 100644 --- a/src/test/java/com/kingsrook/qbits/collaboration/devserver/SharedPresenceDevServer.java +++ b/src/test/java/com/kingsrook/qbits/collaboration/devserver/SharedPresenceDevServer.java @@ -65,12 +65,13 @@ public class SharedPresenceDevServer .filter(CollaborationQBitConfig.class::isInstance).map(CollaborationQBitConfig.class::cast) .findFirst().orElseThrow().withRedisPresence(new RedisPresenceConfig().withConnectionName("presence") .withDeployment("acceptance").withApplication("collaboration") + .withRecoveryGeneration(System.getProperty("collab.dev.recoveryGeneration", "initial")) .withScopeProvider(new QCodeReference(SingleTenant.class))); var redis = new RedisRuntimeService(); var transport = new RedisRealtimeRuntimeService(); new com.kingsrook.qqq.backend.core.health.HealthMetaDataProvider().defineAll(instance, "memory", null); var health = new com.kingsrook.qqq.backend.core.health.HealthRunner(instance, "acceptance-" + port, java.time.Clock.systemUTC()); - var server = CollaborationDevServer.server(instance, port, false); + var server = CollaborationDevServer.server(instance, port, Boolean.getBoolean("collab.dev.frontend")); try { redis.start(instance);