WIP: fix(session): keep the load balancer's affinity cookies so requests and the realtime socket reach one server node #32

2026-11-10T22:53:54Z

james.maes commented

2026-11-10 22:53:54 +00:00

Draft. Do not merge, tag or release from this pull request without the owner's review. A release from it must be signed through the existing release workflow; nothing here was signed or packaged.

Why

A QQQ server that runs as several nodes sits behind a load balancer with cookie stickiness. The session client attaches its cookies by hand (QqqSessionClient.BuildRequest, ConfigureSocket) and dropped everything but the session, so each request could reach a different node: the realtime socket on one, claims and downloads on another. A server whose claim gate needs the socket on the node that receives the claim (QQQ Printing 1.1.x) then answers 409 to about half the claims. Roaming also exposes the client to a token-refresh race between nodes that a browser, which keeps the cookie, does not see.

What changes (Core only)

QqqSessionClient.Affinity.cs (new): keeps the Application Load Balancer's cookies (names beginning AWSALB: AWSALB, AWSALBCORS, AWSALBTG, AWSALBTGCORS, AWSALBAPP-n) from any answer of the session origin, including the answer to the WebSocket upgrade whether accepted or refused. Sends them on every later request to that origin and on the upgrade. Replaces or drops them when the load balancer rewrites or expires them, which is how it moves a client off a node that stopped.
Bounds: scoped to the origin's host and root path, plain value of at most 4096 characters, eight at most, a Secure cookie only over HTTPS. In memory only; never saved with the session; never sent to the identity provider.
ReadSessionCookie: cookies under names other than sessionUUID / sessionId no longer decide whether a session answer is valid. A load balancer adds its own cookies to every answer, and one the strict parser rejects must not fail the request. The session cookie is checked exactly as before.
Discovery runs immediately before each upgrade (unchanged), so the socket opens on the node the HTTP requests already reach, and claims then go to the node that holds the socket.

No change to the Windows runtime, the pipe, pairing, the journal or the printing protocol. Nothing changes against a single-node server.

What the client does when its node stops (confirmed by reading the runtime, and by the tests named)

Seen	Behaviour	Person needed
Socket closes or misses heartbeats	`RealtimeReconnectAsync`: not ready, claim wave cancelled, reconnect after backoff (1 to 30 s). The load balancer picks a healthy node and rewrites the cookie; after WELCOME and applied state the server announces work again; claimed jobs replay, recorded jobs are not submitted twice (`ReconnectReplaysLostSpoolReceiptAndDrainsBufferedJobsWithoutPrintingAgain`)	No
5xx or 409 on a claim, download or event	Answer handed back, session kept (`AServerErrorKeepsTheSessionAndFollowsTheLoadBalancer`). The lane logs `Print lane stopped`; the job stays on the server until the next job notification	No
Connection dropped with no answer	Transport failure, session kept (`ADroppedConnectionKeepsTheSession`)	No
5xx on the upgrade	Failed attempt, retried after backoff; the refusal's cookie is adopted (`ARefusedUpgradeIsRetryableUnlessAuthorizationWasWithdrawn`)	No
401 on a request; 401 or 403 on the upgrade	Authorization withdrawn: session cleared on 401, reconnecting blocked until sign-in	Yes

Known gap, not changed here: after a failed claim the lane does not wake itself; it waits for the next JOB_AVAILABLE. QQQ Printing's several-node change (qbit-printing pull request 36) re-announces outstanding work on the next heartbeat, which bounds that wait to one heartbeat interval. qbit-printing 1.1.x also answered 403 to an upgrade when its database could not answer, which blocks this client until a person signs in; pull request 36 changes that to 503.

Tests

LoadBalancerAffinityTests (Core.Tests, 34 cases): cookies in the load balancer's real format travel on every later request beside the session and on sign-out; which cookies are kept (20 cases); bounds; rotation and expiry; foreign cookies never invalidate a session answer while a mis-scoped session cookie still does; 5xx / 409 keep the session and follow the rewritten cookie, 401 ends it; a dropped connection; the upgrade carries the cookie and adopts the one on its answer (real loopback WebSocket handshake); a refused upgrade (502, 503, 401). LoopbackWebSocketPeer gained optional answer headers and a refusal.

Gates (seen)

Recorded in a comment below for the head commit: ./build/Verify.ps1 (locked restore, Release x64 solution build with -warnaserror, Core.Tests and the Windows/CLI suite), check-source-headers.py. Gitea runs no checks for this repository yet (pull request 17), and the GitHub shadow was not pushed to.

Release

Base main (the 1.2.x line). It ships in the next 1.2.x tag after merge: 1.2.2 if merged before pull request 19 is tagged, otherwise 1.2.3 (VersionPrefix is not changed here). The stable tag runs the signing workflow on the GitHub bridge; a release must be signed and was not attempted.
release/2.0 needs the same change ported by hand: Core/Authentication there was refactored today (pull request 29: QqqSocketRequest, QqqTransportPolicy), so this does not cherry-pick.

Not verified

A real Application Load Balancer: whether it sets AWSALB on the 101 upgrade answer (the client does not depend on it, because discovery precedes the upgrade), and its behaviour while a target is draining. Load balancers other than AWS's are not recognized. No installer was built and nothing was printed.

🤖 Generated with Claude Code

Draft. Do not merge, tag or release from this pull request without the owner's review. A release from it must be signed through the existing release workflow; nothing here was signed or packaged. ## Why A QQQ server that runs as several nodes sits behind a load balancer with cookie stickiness. The session client attaches its cookies by hand (`QqqSessionClient.BuildRequest`, `ConfigureSocket`) and dropped everything but the session, so each request could reach a different node: the realtime socket on one, claims and downloads on another. A server whose claim gate needs the socket on the node that receives the claim (QQQ Printing 1.1.x) then answers 409 to about half the claims. Roaming also exposes the client to a token-refresh race between nodes that a browser, which keeps the cookie, does not see. ## What changes (Core only) - `QqqSessionClient.Affinity.cs` (new): keeps the Application Load Balancer's cookies (names beginning `AWSALB`: `AWSALB`, `AWSALBCORS`, `AWSALBTG`, `AWSALBTGCORS`, `AWSALBAPP-n`) from any answer of the session origin, including the answer to the WebSocket upgrade whether accepted or refused. Sends them on every later request to that origin and on the upgrade. Replaces or drops them when the load balancer rewrites or expires them, which is how it moves a client off a node that stopped. - Bounds: scoped to the origin's host and root path, plain value of at most 4096 characters, eight at most, a `Secure` cookie only over HTTPS. In memory only; never saved with the session; never sent to the identity provider. - `ReadSessionCookie`: cookies under names other than `sessionUUID` / `sessionId` no longer decide whether a session answer is valid. A load balancer adds its own cookies to every answer, and one the strict parser rejects must not fail the request. The session cookie is checked exactly as before. - Discovery runs immediately before each upgrade (unchanged), so the socket opens on the node the HTTP requests already reach, and claims then go to the node that holds the socket. No change to the Windows runtime, the pipe, pairing, the journal or the printing protocol. Nothing changes against a single-node server. ## What the client does when its node stops (confirmed by reading the runtime, and by the tests named) | Seen | Behaviour | Person needed | |---|---|---| | Socket closes or misses heartbeats | `RealtimeReconnectAsync`: not ready, claim wave cancelled, reconnect after backoff (1 to 30 s). The load balancer picks a healthy node and rewrites the cookie; after WELCOME and applied state the server announces work again; claimed jobs replay, recorded jobs are not submitted twice (`ReconnectReplaysLostSpoolReceiptAndDrainsBufferedJobsWithoutPrintingAgain`) | No | | 5xx or 409 on a claim, download or event | Answer handed back, session kept (`AServerErrorKeepsTheSessionAndFollowsTheLoadBalancer`). The lane logs `Print lane stopped`; the job stays on the server until the next job notification | No | | Connection dropped with no answer | Transport failure, session kept (`ADroppedConnectionKeepsTheSession`) | No | | 5xx on the upgrade | Failed attempt, retried after backoff; the refusal's cookie is adopted (`ARefusedUpgradeIsRetryableUnlessAuthorizationWasWithdrawn`) | No | | 401 on a request; 401 or 403 on the upgrade | Authorization withdrawn: session cleared on 401, reconnecting blocked until sign-in | Yes | Known gap, not changed here: after a failed claim the lane does not wake itself; it waits for the next `JOB_AVAILABLE`. QQQ Printing's several-node change (qbit-printing pull request 36) re-announces outstanding work on the next heartbeat, which bounds that wait to one heartbeat interval. qbit-printing 1.1.x also answered 403 to an upgrade when its database could not answer, which blocks this client until a person signs in; pull request 36 changes that to 503. ## Tests `LoadBalancerAffinityTests` (Core.Tests, 34 cases): cookies in the load balancer's real format travel on every later request beside the session and on sign-out; which cookies are kept (20 cases); bounds; rotation and expiry; foreign cookies never invalidate a session answer while a mis-scoped session cookie still does; 5xx / 409 keep the session and follow the rewritten cookie, 401 ends it; a dropped connection; the upgrade carries the cookie and adopts the one on its answer (real loopback WebSocket handshake); a refused upgrade (502, 503, 401). `LoopbackWebSocketPeer` gained optional answer headers and a refusal. ## Gates (seen) Recorded in a comment below for the head commit: `./build/Verify.ps1` (locked restore, Release x64 solution build with `-warnaserror`, Core.Tests and the Windows/CLI suite), `check-source-headers.py`. Gitea runs no checks for this repository yet (pull request 17), and the GitHub shadow was not pushed to. ## Release - Base `main` (the 1.2.x line). It ships in the next 1.2.x tag after merge: 1.2.2 if merged before pull request 19 is tagged, otherwise 1.2.3 (`VersionPrefix` is not changed here). The stable tag runs the signing workflow on the GitHub bridge; a release must be signed and was not attempted. - `release/2.0` needs the same change ported by hand: `Core/Authentication` there was refactored today (pull request 29: `QqqSocketRequest`, `QqqTransportPolicy`), so this does not cherry-pick. ## Not verified A real Application Load Balancer: whether it sets `AWSALB` on the 101 upgrade answer (the client does not depend on it, because discovery precedes the upgrade), and its behaviour while a target is draining. Load balancers other than AWS's are not recognized. No installer was built and nothing was printed. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

james.maes added 2 commits 2026-11-10 22:53:54 +00:00

fix(session): keep the load balancer's affinity cookies so requests and the realtime socket reach one node 530e2e302b

A QQQ server that runs as several nodes sits behind a load balancer that names
the node it chose in a cookie. The session client attaches its cookies by hand
and dropped everything but the session, so the realtime socket could open on
one node while claims and downloads went to another; a server whose claim gate
needs the socket on the same node (qbit-printing 1.1.x) then answers 409.

- Keep the Application Load Balancer's cookies (names beginning AWSALB) from
  any answer of the session origin, the WebSocket upgrade answer included,
  whether it was accepted or refused. Send them on every later request to that
  origin and on the upgrade. Replace or drop them when the load balancer
  rewrites or expires them. In memory only; never saved with the session.
- Only cookies scoped to this host and the root path with a plain, bounded
  value are kept; at most eight; a Secure cookie only over HTTPS.
- Cookies under names other than the session's no longer decide whether a
  session answer is valid: a load balancer adds its own to every answer.

Tests cover the cookie rules, rotation and expiry, the upgrade carrying and
adopting the cookie, and what a refused request means when a node stops: a 5xx
or a dropped connection keeps the session, only a 401 ends it.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: James Maes <james@kof22.com>

docs: load balancer affinity, and what the client does when a server node stops 87e3f18b83

Changelog (unreleased), REALTIME.md (a server that runs as several nodes, with
the 401 / 5xx / dropped-connection table) and the threat model (the only
cookies besides the session that the client ever sends, and their bounds).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: James Maes <james@kof22.com>

james.maes commented

2026-11-10 22:54:04 +00:00

Gates seen locally on head 87e3f18 (maintainer workstation; the GitHub shadow was not pushed to and Gitea runs no checks for this repository yet):

./build/Verify.ps1 -ProtocolFixtures <fixtures exported from qqq-print-protocol ac835f8, the commit pinned in .github/workflows/ci.yml>: locked restore; Release x64 solution build with -warnaserror, 0 warnings, 0 errors; Core.Tests 231 passed, 0 failed (197 before, 34 new); Windows/CLI suite 377 passed, 0 failed.
An earlier run of the same script on the same commit had 3 failures in the Windows suite (AgentRuntimeQueuedJobsTests, AgentRuntimeHeartbeatTests, AgentControlConcurrencyTests): each timed out on its own 10 second budget while other builds and test runs were competing for the machine. None of them uses the session client or the loopback peer. The rerun above passed all 377.
python .github/scripts/check-source-headers.py: 321 tracked files verified.
Not run: the installer build, UI and layout suites, CodeQL and the script-quality jobs (no script changed). Nothing was signed, packaged, installed or printed.

🤖 Generated with Claude Code

Gates seen locally on head `87e3f18` (maintainer workstation; the GitHub shadow was not pushed to and Gitea runs no checks for this repository yet): - `./build/Verify.ps1 -ProtocolFixtures <fixtures exported from qqq-print-protocol ac835f8, the commit pinned in .github/workflows/ci.yml>`: locked restore; Release x64 solution build with `-warnaserror`, 0 warnings, 0 errors; **Core.Tests 231 passed, 0 failed** (197 before, 34 new); **Windows/CLI suite 377 passed, 0 failed**. - An earlier run of the same script on the same commit had 3 failures in the Windows suite (`AgentRuntimeQueuedJobsTests`, `AgentRuntimeHeartbeatTests`, `AgentControlConcurrencyTests`): each timed out on its own 10 second budget while other builds and test runs were competing for the machine. None of them uses the session client or the loopback peer. The rerun above passed all 377. - `python .github/scripts/check-source-headers.py`: 321 tracked files verified. - Not run: the installer build, UI and layout suites, CodeQL and the script-quality jobs (no script changed). Nothing was signed, packaged, installed or printed. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

You are not authorized to merge this pull request.

This pull request can be merged automatically.

This pull request is marked as a work in progress.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin fix/load-balancer-affinity-20261110:fix/load-balancer-affinity-20261110

git checkout fix/load-balancer-affinity-20261110

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qqq-print-agent#32