WIP: feat(realtime): print on several application nodes (shared wakeups, durable connection record, claim gate on any node) #36

2026-11-10T22:51:32Z

james.maes commented

2026-11-10 22:51:32 +00:00

Draft. Do not merge, tag or release from this pull request without the owner's review.

Why

An application that runs as several nodes (pods) over one database could not print reliably. A device holds one realtime connection, on one node, and three things about it lived only in that JVM:

Held in one JVM at 1.1.2	Where	With two nodes
Wakeup listeners	`PrintingRealtimeBus.java:32` (`listeners`), `publish` `:77-91`	A job accepted on the node without the device's socket is never announced. It stays QUEUED until something else wakes that device
Connection ownership	`PrintingRealtimeBus.java:33` (`owners`), `PrintingRealtimeRoutes.java:66` (`channels`)	A device that reconnects through another node leaves its old connection unfenced until that socket times out
Claim gate	`PrintingRealtimeRoutes.java:181-188` (`requireReady` reads `channels`), per-connection `appliedState` / `appliedSettings` `:326-329`	A claim that reaches the other node answers 409. The Windows client keeps no load balancer cookie, so about half its claims do

Already shared and unchanged: job rows, events and the claim itself (PrintingService.claimForJob, device row FOR UPDATE; events are idempotent by event id), device control and settings revisions.

What changes

Wakeups cross nodes on QQQ's own realtime bus. PrintingRealtimeBus.join(instance) subscribes to RealtimeBus.get(instance): in memory on one node, the host's shared bus (Redis through qbit-redis) on several. publish delivers to this JVM directly, then hands the wakeup to the shared bus through a bounded background worker, so the committing request never waits for the transport. The route provider joins when it starts; a host adds no wiring. Channel qbit-printing is not one a browser stream can subscribe to, and the message carries a tenant and a device identifier only.
The connection is recorded on the device row (PrintingRealtimeConnections; four additive columns). Every statement is one guarded update: the newest connection takes the row, and only the connection named on the row can renew it, mark it ready or clear it.
The claim gate can be answered by any node. The node holding the socket answers from the connection, as before. Any other node accepts the claim when the row shows a live, ready connection and the device has applied the current desired state (control_applied_revision = control_revision) and settings (realtime_applied_revision = the settings revision); otherwise 409, as before.
The bus is best effort and printing does not depend on it. On every client heartbeat (PING, every heartbeatIntervalSeconds: 20 by default, 5 at the least) the connection renews its lease and asks the database whether anything was committed that it has not been told (newer desired state or settings, a pending command, a job a claim would hand out now). If so it runs the ordinary wakeup. A wakeup the bus lost is late by at most one heartbeat interval. An idle heartbeat costs a few indexed reads and sends no frame. When the transport reports a gap (resync), every connection reconciles at once.
One connection per device across nodes. Connecting through another node takes the row and publishes; the old node closes its socket on that message, or on the old socket's next frame when the bus is down.
The QUEUED to CLAIMED update carries a status guard as well as the device row lock.
The socket upgrade answers 503 when the database could not answer during authorization (it answered 403, which the Windows client treats as "ask a person to sign in").

Delivery guarantee

At least once to the device, idempotent acknowledgement, never two spooler submissions for one job:

A job is handed out by exactly one QUEUED to CLAIMED transition; every later claim on any node replays the same assignment until RECEIVED commits, and the next job is not handed out before that.
An event is recorded once per event id on any node; a replay is accepted, a different event under the same id is refused, and a job cannot move backwards.
A received job is never handed out again by any node.
(Client, unchanged) a job its journal already holds in a final state is not submitted again; an interrupted submission is reported OUTCOME_UNKNOWN, not reprinted.

Proof: PrintingClaimPostgresTest on PostgreSQL 17 (eight claims racing through two service instances, twelve rounds: one transition, one job, the next job untouched; six concurrent deliveries of one event: one row) and PrintingMultiNodeTest (claim and events over HTTP on the node without the socket).

Tests

PrintingMultiNodeTest: two services, each with its own bus and route provider (its own sockets), one H2 database, one application realtime bus the test can switch to "drop" or "throw". A job accepted on A wakes the device connected to B; the claim is made over HTTP on A (409 before this change); events through either node; lease expiry and renewal; settings and pause changes close the gate on both nodes until applied; bus down, then found on the next heartbeat; a transport gap; replacement through another node with and without the bus; upgrade status mapping.
PrintingRealtimeBusTest: exactly-once delivery across joined buses including the transport's echo, foreign and malformed messages ignored, join counting, a blocked or failing transport never delays or fails the publisher.
PrintingClaimPostgresTest (Testcontainers postgres:17-alpine; the suite now needs Docker, which the shared container-build runner provides): racing claims, concurrent acknowledgements, the connection record's guards, the reconciliation predicate, and initialize() itself on PostgreSQL.

Gates (seen)

Local, commit 4aa63ae, bash ./mvnw -B -Pui-demo verify (no skip flags; QRun artifacts from the Gitea registry): 215 tests, 0 failures, 0 errors, 1 skipped (the opt-in 10,000-job workload), Checkstyle 0 violations, Error Prone / NullAway -Werror clean, guardrails pass.
Hosted, commit 4aa63ae: Gitea run 7392 (maven / build (push)), success.
Head 42dff1f adds documentation only; its hosted run is recorded in a comment below.

Release and adoption

Releases are cut from main by tag (docs/RELEASES.md; shared ci-workflows Maven workflow). main already carries tag v1.2.0-RC.1 with the same source as 1.1.1 / 1.1.2. After merge, tag v1.2.0-RC.2 on the qualified main commit: the workflow publishes com.kingsrook.qbits:qbit-printing:1.2.0-RC.2 and commits nothing back. The POM version is not changed here.
Hosts that own their schema add four columns before running it (in docs/REALTIME.md; additive, the previous release keeps working on the new schema). Hosts that call initialize() get them from it.
No wire or contract change: extension 2.4.0, wire v1, subprotocol unchanged. Windows connector 1.2.x works unchanged; its cookie fix (qqq-print-agent) is independent and complementary.

Not verified

Two real processes behind a load balancer; a node dying mid-claim or mid-download; Redis as the transport (the test transport stands in for it; RedisRealtimeBus is exercised only by its own repository's tests); clock skew between nodes beyond ordinary synchronization; a physical printer.

🤖 Generated with Claude Code

Draft. Do not merge, tag or release from this pull request without the owner's review. ## Why An application that runs as several nodes (pods) over one database could not print reliably. A device holds one realtime connection, on one node, and three things about it lived only in that JVM: | Held in one JVM at 1.1.2 | Where | With two nodes | |---|---|---| | Wakeup listeners | `PrintingRealtimeBus.java:32` (`listeners`), `publish` `:77-91` | A job accepted on the node without the device's socket is never announced. It stays QUEUED until something else wakes that device | | Connection ownership | `PrintingRealtimeBus.java:33` (`owners`), `PrintingRealtimeRoutes.java:66` (`channels`) | A device that reconnects through another node leaves its old connection unfenced until that socket times out | | Claim gate | `PrintingRealtimeRoutes.java:181-188` (`requireReady` reads `channels`), per-connection `appliedState` / `appliedSettings` `:326-329` | A claim that reaches the other node answers 409. The Windows client keeps no load balancer cookie, so about half its claims do | Already shared and unchanged: job rows, events and the claim itself (`PrintingService.claimForJob`, device row `FOR UPDATE`; events are idempotent by event id), device control and settings revisions. ## What changes - **Wakeups cross nodes on QQQ's own realtime bus.** `PrintingRealtimeBus.join(instance)` subscribes to `RealtimeBus.get(instance)`: in memory on one node, the host's shared bus (Redis through qbit-redis) on several. `publish` delivers to this JVM directly, then hands the wakeup to the shared bus through a bounded background worker, so the committing request never waits for the transport. The route provider joins when it starts; a host adds no wiring. Channel `qbit-printing` is not one a browser stream can subscribe to, and the message carries a tenant and a device identifier only. - **The connection is recorded on the device row** (`PrintingRealtimeConnections`; four additive columns). Every statement is one guarded update: the newest connection takes the row, and only the connection named on the row can renew it, mark it ready or clear it. - **The claim gate can be answered by any node.** The node holding the socket answers from the connection, as before. Any other node accepts the claim when the row shows a live, ready connection and the device has applied the current desired state (`control_applied_revision = control_revision`) and settings (`realtime_applied_revision` = the settings revision); otherwise 409, as before. - **The bus is best effort and printing does not depend on it.** On every client heartbeat (`PING`, every `heartbeatIntervalSeconds`: 20 by default, 5 at the least) the connection renews its lease and asks the database whether anything was committed that it has not been told (newer desired state or settings, a pending command, a job a claim would hand out now). If so it runs the ordinary wakeup. A wakeup the bus lost is late by at most one heartbeat interval. An idle heartbeat costs a few indexed reads and sends no frame. When the transport reports a gap (`resync`), every connection reconciles at once. - **One connection per device across nodes.** Connecting through another node takes the row and publishes; the old node closes its socket on that message, or on the old socket's next frame when the bus is down. - The QUEUED to CLAIMED update carries a status guard as well as the device row lock. - The socket upgrade answers **503** when the database could not answer during authorization (it answered 403, which the Windows client treats as "ask a person to sign in"). ## Delivery guarantee At least once to the device, idempotent acknowledgement, never two spooler submissions for one job: 1. A job is handed out by exactly one QUEUED to CLAIMED transition; every later claim on any node replays the same assignment until RECEIVED commits, and the next job is not handed out before that. 2. An event is recorded once per event id on any node; a replay is accepted, a different event under the same id is refused, and a job cannot move backwards. 3. A received job is never handed out again by any node. 4. (Client, unchanged) a job its journal already holds in a final state is not submitted again; an interrupted submission is reported `OUTCOME_UNKNOWN`, not reprinted. Proof: `PrintingClaimPostgresTest` on PostgreSQL 17 (eight claims racing through two service instances, twelve rounds: one transition, one job, the next job untouched; six concurrent deliveries of one event: one row) and `PrintingMultiNodeTest` (claim and events over HTTP on the node without the socket). ## Tests - `PrintingMultiNodeTest`: two services, each with its own bus and route provider (its own sockets), one H2 database, one application realtime bus the test can switch to "drop" or "throw". A job accepted on A wakes the device connected to B; the claim is made over HTTP on A (409 before this change); events through either node; lease expiry and renewal; settings and pause changes close the gate on both nodes until applied; bus down, then found on the next heartbeat; a transport gap; replacement through another node with and without the bus; upgrade status mapping. - `PrintingRealtimeBusTest`: exactly-once delivery across joined buses including the transport's echo, foreign and malformed messages ignored, join counting, a blocked or failing transport never delays or fails the publisher. - `PrintingClaimPostgresTest` (Testcontainers `postgres:17-alpine`; the suite now needs Docker, which the shared `container-build` runner provides): racing claims, concurrent acknowledgements, the connection record's guards, the reconciliation predicate, and `initialize()` itself on PostgreSQL. ## Gates (seen) - Local, commit `4aa63ae`, `bash ./mvnw -B -Pui-demo verify` (no skip flags; QRun artifacts from the Gitea registry): **215 tests, 0 failures, 0 errors, 1 skipped** (the opt-in 10,000-job workload), Checkstyle 0 violations, Error Prone / NullAway `-Werror` clean, guardrails pass. - Hosted, commit `4aa63ae`: Gitea run 7392 (`maven / build (push)`), success. - Head `42dff1f` adds documentation only; its hosted run is recorded in a comment below. ## Release and adoption - Releases are cut from `main` by tag (`docs/RELEASES.md`; shared `ci-workflows` Maven workflow). `main` already carries tag `v1.2.0-RC.1` with the same source as 1.1.1 / 1.1.2. After merge, tag **`v1.2.0-RC.2`** on the qualified `main` commit: the workflow publishes `com.kingsrook.qbits:qbit-printing:1.2.0-RC.2` and commits nothing back. The POM version is not changed here. - Hosts that own their schema add four columns before running it (in `docs/REALTIME.md`; additive, the previous release keeps working on the new schema). Hosts that call `initialize()` get them from it. - No wire or contract change: extension 2.4.0, wire v1, subprotocol unchanged. Windows connector 1.2.x works unchanged; its cookie fix (qqq-print-agent) is independent and complementary. ## Not verified Two real processes behind a load balancer; a node dying mid-claim or mid-download; Redis as the transport (the test transport stands in for it; `RedisRealtimeBus` is exercised only by its own repository's tests); clock skew between nodes beyond ordinary synchronization; a physical printer. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

james.maes added 2 commits 2026-11-10 22:51:32 +00:00

feat(realtime): print on several nodes: shared wakeups, a durable connection record and a claim gate any node can answer

maven / build (push) Successful in 3m37s

Details

build / maven (push) Successful in 3m37s

Details

4aa63ae1d1

A device holds one realtime connection, on one node. Until now the wakeup
listeners, the connection ownership and the claim gate lived in that JVM, so a
job accepted on another node never reached the device, and a claim that
reached another node was refused with 409.

- PrintingRealtimeBus joins the application's QQQ realtime bus (in memory on
  one node, the configured shared bus such as Redis on several). A wakeup is
  delivered to this JVM directly and handed to the shared bus by a bounded
  background worker, so the committing caller never waits for the transport.
  The route provider joins at start; hosts need no new wiring.
- The connection is recorded on the device row (connection id, node, ready
  flag, lease). Every statement is a guarded update: the newest connection
  takes the row, only the named connection can renew, mark or clear it. The
  claim gate accepts a claim on a node without the socket when the row shows
  a live, ready connection and the device has applied the current desired
  state and settings.
- On every heartbeat (PING) the connection renews its lease and asks the
  database whether anything was committed that it has not been told; a
  wakeup the bus lost costs at most one heartbeat interval. A connection
  replaced through another node closes on the bus message, or on its next
  frame when the bus is down.
- The QUEUED to CLAIMED update carries a status guard in addition to the
  device row lock.
- A database failure during the socket upgrade answers 503 (retryable), not
  403, which a client treats as "ask a person to sign in".

Tests: two services with separate buses and route providers over one database
and one application bus (wake, claim over HTTP on the other node, events on
either node, bus down, replacement), the bus on its own, and the claim,
acknowledgement and connection record on PostgreSQL 17 (Testcontainers).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

docs(realtime): describe printing on several application nodes

maven / build (push) Successful in 4m20s

Details

build / maven (push) Successful in 4m20s

Details

42dff1ff39

REALTIME.md replaces the single-node boundary with what is shared and where
(jobs and the claim, the connection record, the claim gate, wakeups), what
happens when the shared bus loses a wakeup, the delivery guarantee, the four
columns a host that owns its schema must add, and what is and is not verified.
README, OPERATIONS and the changelog follow.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

james.maes commented

2026-11-10 22:57:01 +00:00

Gates seen on head 42dff1f (documentation added on top of 4aa63ae; no source or test change):

Local: bash ./mvnw -B -Pui-demo verify, no skip flags, QRun artifacts resolved from the QRun-IO Gitea registry: 215 tests, 0 failures, 0 errors, 1 skipped (the opt-in 10,000-job workload), Checkstyle 0 violations, BUILD SUCCESS. The same result was seen on 4aa63ae.
Hosted: Gitea run 7394 (maven / build (push), -Pui-demo verify on the container-build runner, PostgreSQL 17 through Testcontainers): 215 tests, 0 failures, 0 errors, 1 skipped, success in 4m20s. Run 7392 on 4aa63ae: success in 3m37s.
The local build needs a Maven settings file that names the QRun-IO Gitea registry (this POM declares no repository); the shared workflow supplies its own.

🤖 Generated with Claude Code

Gates seen on head `42dff1f` (documentation added on top of `4aa63ae`; no source or test change): - Local: `bash ./mvnw -B -Pui-demo verify`, no skip flags, QRun artifacts resolved from the QRun-IO Gitea registry: **215 tests, 0 failures, 0 errors, 1 skipped** (the opt-in 10,000-job workload), Checkstyle 0 violations, BUILD SUCCESS. The same result was seen on `4aa63ae`. - Hosted: Gitea run 7394 (`maven / build (push)`, `-Pui-demo verify` on the `container-build` runner, PostgreSQL 17 through Testcontainers): 215 tests, 0 failures, 0 errors, 1 skipped, success in 4m20s. Run 7392 on `4aa63ae`: success in 3m37s. - The local build needs a Maven settings file that names the QRun-IO Gitea registry (this POM declares no repository); the shared workflow supplies its own. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

james.maes referenced this pull request

2026-11-10 23:55:52 +00:00

WIP: feat(station): a print station authenticates as its own identity (printing_station, station enrollment code) #37

All checks were successful

maven / build (push) Successful in 4m20s

Required

Details

build / maven (push) Successful in 4m20s

Details

This pull request has changes conflicting with the target branch.

CHANGELOG.md

This pull request is marked as a work in progress.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feat/multi-node-realtime-20261110:feat/multi-node-realtime-20261110

git checkout feat/multi-node-realtime-20261110

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-printing#36