Register object-storage caches with ClusterCaches (multi-node) #27

2026-10-15T17:20:13Z

KofTwentyTwo commented

2026-10-15 17:20:13 +00:00

(Migrated from github.com)

What

Makes qbit-object-storage correct on multi-node deployments (QQQ cluster HA spec, item 4 and owner decision 1).

Cache	Decision	Why
`ConfigSync.ROW_LIMITS` (bucket-row presign limits, 15 s)	shared `qbit-object-storage.bucket-row-limits`	`maxPresignGetSeconds` / `maxPresignPutSeconds` are edited on the bucket row in the admin UI.
`DefaultObjectHeaderPolicy` enabled rules (15 s)	shared `qbit-object-storage.header-rules`	Header rules are admin-editable rows. Moved from two volatile fields to an `AtomicReference`.
`CapabilityStore.CACHE` (+ `LOADED`)	shared `qbit-object-storage.capabilities`, key-level	A probe on one node rewrote the row but other nodes kept the old status until restart (no TTL). `record()` now writes the row, invalidates that key everywhere (others reload it from the row), then caches it locally.
`ConfigSync.BUCKET_IDS`, `CONNECTION_IDS`	node-local	Alias/name to row id from the config sync; bucket and connection rows cannot be inserted or deleted through the UI/API, so ids match on every node.
`StorageBucket.PRESIGN_COUNTS`	node-local	Per-pod presign rate limiter; per-node limits accepted for now (owner decision 4).
`ObjectStorageRuntime.CONNECTIONS`, `PROCESS_PERMISSIONS`	node-local	This node's S3 clients / permissions built from code config.
`ProviderDefaults.TABLE`, `ObjectIndex.RUNNING`	node-local	Code constants / this node's in-flight scan guard.

New ConfigRowChangeCustomizer (post insert/update on object_storage_bucket; post insert/update/delete on object_storage_header_rule) publishes the invalidations. TTLs are unchanged and remain the safety net.

Tests

New ObjectStorageClusterCachesTest: ClusterCacheAudit.unregisteredStaticCaches("com.kingsrook.qbits.objectstorage") is empty; a remote capability invalidation (node-b over a test wire transport) clears the key and the next read reloads the row; recording a probe result, editing a bucket row and inserting a header rule each publish the right event from node-a; a bucket-row edit takes effect without waiting for the 15 s TTL.
./mvnw -o verify (with -Denforcer.skip=true and StandInGuardrailTest excluded, see below): 163 tests pass, coverage checks met.

Release notes

QQQ dependency. ClusterCaches is only in QQQ 4.1.0-SNAPSHOT today (qqq#1228; not in 4.1.0-RC.12). This PR sets the one-line property qqq.snapshot.version to 4.1.0-SNAPSHOT. The G7 guardrail (no SNAPSHOT QQQ dependencies) fails CI until it points at the QQQ RC that ships ClusterCaches. Before merging or releasing, change qqq.snapshot.version to that RC.
Expired stand-ins (pre-existing). StandInGuardrailTest (G4) fails on any QQQ newer than 4.1.0-RC.8: ConnectionRuntime.SessionCredentialsClient and SecretRefs carry STAND-IN(core=QRun-IO/qqq#1165, removeBy=4.1.0-RC.9), and qqq#1165 is still open. Moving to the new RC needs either qqq#1165 landed (and the stand-ins removed) or an owner-approved removeBy extension. This PR does not change those markers.

🤖 Generated with Claude Code

## What Makes qbit-object-storage correct on multi-node deployments (QQQ cluster HA spec, item 4 and owner decision 1). | Cache | Decision | Why | |---|---|---| | `ConfigSync.ROW_LIMITS` (bucket-row presign limits, 15 s) | **shared** `qbit-object-storage.bucket-row-limits` | `maxPresignGetSeconds` / `maxPresignPutSeconds` are edited on the bucket row in the admin UI. | | `DefaultObjectHeaderPolicy` enabled rules (15 s) | **shared** `qbit-object-storage.header-rules` | Header rules are admin-editable rows. Moved from two volatile fields to an `AtomicReference`. | | `CapabilityStore.CACHE` (+ `LOADED`) | **shared** `qbit-object-storage.capabilities`, key-level | A probe on one node rewrote the row but other nodes kept the old status **until restart** (no TTL). `record()` now writes the row, invalidates that key everywhere (others reload it from the row), then caches it locally. | | `ConfigSync.BUCKET_IDS`, `CONNECTION_IDS` | node-local | Alias/name to row id from the config sync; bucket and connection rows cannot be inserted or deleted through the UI/API, so ids match on every node. | | `StorageBucket.PRESIGN_COUNTS` | node-local | Per-pod presign rate limiter; per-node limits accepted for now (owner decision 4). | | `ObjectStorageRuntime.CONNECTIONS`, `PROCESS_PERMISSIONS` | node-local | This node's S3 clients / permissions built from code config. | | `ProviderDefaults.TABLE`, `ObjectIndex.RUNNING` | node-local | Code constants / this node's in-flight scan guard. | New `ConfigRowChangeCustomizer` (post insert/update on `object_storage_bucket`; post insert/update/delete on `object_storage_header_rule`) publishes the invalidations. TTLs are unchanged and remain the safety net. ## Tests - New `ObjectStorageClusterCachesTest`: `ClusterCacheAudit.unregisteredStaticCaches("com.kingsrook.qbits.objectstorage")` is empty; a remote capability invalidation (node-b over a test wire transport) clears the key and the next read reloads the row; recording a probe result, editing a bucket row and inserting a header rule each publish the right event from node-a; a bucket-row edit takes effect without waiting for the 15 s TTL. - `./mvnw -o verify` (with `-Denforcer.skip=true` and `StandInGuardrailTest` excluded, see below): 163 tests pass, coverage checks met. ## Release notes 1. **QQQ dependency.** `ClusterCaches` is only in QQQ `4.1.0-SNAPSHOT` today (qqq#1228; not in 4.1.0-RC.12). This PR sets the one-line property `qqq.snapshot.version` to `4.1.0-SNAPSHOT`. The G7 guardrail (no SNAPSHOT QQQ dependencies) fails CI until it points at the QQQ RC that ships `ClusterCaches`. **Before merging or releasing, change `qqq.snapshot.version` to that RC.** 2. **Expired stand-ins (pre-existing).** `StandInGuardrailTest` (G4) fails on any QQQ newer than 4.1.0-RC.8: `ConnectionRuntime.SessionCredentialsClient` and `SecretRefs` carry `STAND-IN(core=QRun-IO/qqq#1165, removeBy=4.1.0-RC.9)`, and qqq#1165 is still open. Moving to the new RC needs either qqq#1165 landed (and the stand-ins removed) or an owner-approved `removeBy` extension. This PR does not change those markers. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-object-storage#27