fix(session): keep the load balancer's affinity cookies so requests and the realtime socket reach one server node #48

2026-11-11T03:01:22Z

james.maes commented

2026-11-11 03:01:22 +00:00

Port to release/2.0 of the load-balancer affinity fix that is open against main as pull request 32 (work in progress there).

What was wrong

Behind an AWS load balancer with sticky sessions, the agent did not keep the balancer's affinity cookies, so its HTTP requests and its realtime socket could land on different server nodes. A job offered over the socket of one node was then claimed through another, which refused it.

What changes

The session keeps the balancer's affinity cookies (names beginning AWSALB) from answers on the server's own origin and sends them on later requests to that origin and on the realtime upgrade, so both reach one node.
Cookies are never sent to another origin (the sign-in provider included), never logged (a count only), and never stored with the pairing.
A cookie value may not hold a control character (carriage return, line feed and tab included), a space, a quote, a comma, a semicolon, a backslash, DEL or any non-ASCII character, and a Set-Cookie header over 8,192 characters is ignored, so a cookie cannot inject a header or shadow the session's own cookies.

What it does not do (recorded for later work)

Nothing reconnects the socket when the balancer moves the HTTP side to another node while the socket stays up: after a health-check flap, claims can be refused until the socket drops for another reason. The fix belongs in the runtime. This is the most important open item of the port.
The total Cookie header can reach about 33 KB if a server keeps issuing distinct names; a cap on the total is cheap and not in this port.
Domain is compared with the typed host; the lock is taken per header, not per answer.

Verification

Head dd931d0 on release/2.0 at 7b99a29.
build/Verify.ps1 passed on that head: 1,166 Core, 1,568 Windows runtime and command line, 360 presentation tests, none failed; Core 100% of 2,303 lines; coverage gate passed.

Review (independent reviewer, three passes)

c452d3e: the port is faithful to the original; one should-fix (three bounds had no test that failed without them). The reviewer found no way to send, smuggle, log or store a cookie where it should not go.
8d8833b: fixed; an optional nit (rows for carriage return and line feed).
95688e9: Verdict on 95688e9: 0 blockers, 0 should-fix, 1 nit.
The head merged is 95688e9 replayed onto 7b99a29 after pull request 47 merged; git range-diff ca408e4..95688e9 7b99a29..dd931d0 reports all five patches identical.

🤖 Generated with Claude Code

Port to `release/2.0` of the load-balancer affinity fix that is open against `main` as pull request 32 (work in progress there). **What was wrong** - Behind an AWS load balancer with sticky sessions, the agent did not keep the balancer's affinity cookies, so its HTTP requests and its realtime socket could land on different server nodes. A job offered over the socket of one node was then claimed through another, which refused it. **What changes** - The session keeps the balancer's affinity cookies (names beginning `AWSALB`) from answers on the server's own origin and sends them on later requests to that origin and on the realtime upgrade, so both reach one node. - Cookies are never sent to another origin (the sign-in provider included), never logged (a count only), and never stored with the pairing. - A cookie value may not hold a control character (carriage return, line feed and tab included), a space, a quote, a comma, a semicolon, a backslash, DEL or any non-ASCII character, and a `Set-Cookie` header over 8,192 characters is ignored, so a cookie cannot inject a header or shadow the session's own cookies. **What it does not do (recorded for later work)** - Nothing reconnects the socket when the balancer moves the HTTP side to another node while the socket stays up: after a health-check flap, claims can be refused until the socket drops for another reason. The fix belongs in the runtime. This is the most important open item of the port. - The total Cookie header can reach about 33 KB if a server keeps issuing distinct names; a cap on the total is cheap and not in this port. - `Domain` is compared with the typed host; the lock is taken per header, not per answer. **Verification** - Head `dd931d0` on `release/2.0` at `7b99a29`. - `build/Verify.ps1` passed on that head: 1,166 Core, 1,568 Windows runtime and command line, 360 presentation tests, none failed; Core 100% of 2,303 lines; coverage gate passed. **Review (independent reviewer, three passes)** - `c452d3e`: the port is faithful to the original; one should-fix (three bounds had no test that failed without them). The reviewer found no way to send, smuggle, log or store a cookie where it should not go. - `8d8833b`: fixed; an optional nit (rows for carriage return and line feed). - `95688e9`: `Verdict on 95688e9: 0 blockers, 0 should-fix, 1 nit.` - The head merged is `95688e9` replayed onto `7b99a29` after pull request 47 merged; `git range-diff ca408e4..95688e9 7b99a29..dd931d0` reports all five patches identical. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

james.maes added 5 commits 2026-11-11 03:01:22 +00:00

fix(session): keep the load balancer's affinity cookies so requests and the realtime socket reach one node 5806f84f45

Port to 2.0 of pull request 32 on main (branch
fix/load-balancer-affinity-20261110), source commit 530e2e3 (code and tests).
Compare against that commit, and against 87e3f18 for the documents, when the
pull request changes and this needs porting again.

A QQQ server may run as several nodes behind an AWS Application Load
Balancer, which names the node it chose in a cookie (names beginning AWSALB)
and expects it back. The session client attaches its cookies by hand and
sent only the session, so its HTTP requests and its realtime socket could
land on different nodes.

Same behavior, bounds and rules as the source:
- the cookies are kept in memory only, for the life of the client object,
  never saved with the session, and kept across sign-out
- they are sent to the session origin only, with or without a session, and
  never to the sign-in provider, whose answers are not read for them either
- every answer of the session origin replaces or removes them, an error
  answer and a refused socket upgrade included
- at most eight, names of at most 64 characters, values of at most 4096,
  headers of at most 8192; a cookie for another host or path, or marked
  Secure over plain HTTP, is ignored, and a malformed header is skipped
- the session-cookie parser ignores cookies under any other name

Adapted to how Core is built on 2.0:
- the source read ClientWebSocket.HttpResponseHeaders in a finally around
  socket.ConnectAsync. Here the upgrade is IQqqSocketConnector.ConnectAsync,
  so the connector gains TakeUpgradeHeaders, which hands over the answer's
  headers and lets go of them. The platform connector only reads and clears
  the property; the session client picks Set-Cookie out, whatever its
  letter case, and applies the rule, in a finally around the connect, so it
  runs on the accepted, the 401/403 and the other failure paths.
- the upgrade's Cookie header is built in PrepareSocketAsync, after the
  discovery made just before the upgrade, so it carries the newest cookies
- the log counts the cookies and never shows a name or a value

Tests: every case of the source's LoadBalancerAffinityTests, on the
scripted server and the fake transport, and what those make possible: a
refused upgrade (502, 503, 401, 403) that carries a new cookie, an upgrade
with no answer, an answer that expires a cookie, an expiry judged on the
client's clock, both provider requests (sign-in and sign-out) carrying
none and setting none, the bounds on count, name and value, the cookies
kept across sign-out, absent from the stored session and from every log
line. Two tests send a real upgrade over loopback through the platform
connector, accepted and refused, because that read is the one step a fake
cannot show.

Signed-off-by: James Maes <james@kof22.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

docs: load balancer affinity, and what the client does when a server node stops af77e8907f

Port to 2.0 of the documents of pull request 32 on main, source commit
87e3f18: the REALTIME section on a server that runs as several nodes, the
threat-model sentence on the only other cookies the session client sends,
and a changelog entry under Unreleased.

Reworded where 2.0 differs. Job events and receipts travel on the socket
only, so the table no longer lists an HTTP failure of an event; a failed
claim ends the wave with one log line; a failed download is its own row;
and a refused upgrade is said to block printing in the service, where the
source said the session is cleared. The threat model also says that a
request without a credential, the compatibility check's included, may carry
these cookies, and that the log only counts them.

Signed-off-by: James Maes <james@kof22.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

test(session): pin the affinity cookie's character and header bounds 47156d06db

Three bounds on a load balancer cookie were covered but not pinned: each
could be removed and every test still passed.

- A value with a control character other than a space was refused only
  in the code. Rows for a tab, U+0001 and U+001F now fail without it.
- So was a value with DEL or a character outside ASCII. Rows for DEL,
  a Latin-1 letter and a line separator now fail without it, and a row
  with every allowed punctuation mark holds the other side of the line.
- The 8,192-character bound on a header was shown only by a header whose
  value was over its own bound as well. A header one character over the
  bound with a short value is now refused, and one of exactly the bound
  is read.

These are the rules that keep a second cookie pair and a header break out
of the Cookie header the client writes. Tests only; the code is unchanged.

Signed-off-by: James Maes <james@kof22.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

docs: say when requests and the socket part ways, and for how long d79a4bc15d

REALTIME.md said the load balancer's cookie keeps a client's requests and
its socket on one node "except while a node is being drained". That is
one case. The two part whenever the load balancer chooses another node
for the requests while the socket is still open, after a failed health
check for instance, and they stay apart until the socket closes for any
reason: the client does not yet close it on that account. Against a
QQQ Printing 1.1.x server every claim is refused for as long as that
lasts.

Documentation only.

Signed-off-by: James Maes <james@kof22.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

test(session): refuse a cookie value with a carriage return or a line feed dd931d000f

The rows for control characters used a tab and two others, and left out
the two characters that actually end a header line. They were left out
on the belief that the platform's header collection drops such a value
before the client sees it. It does not: a value added without validation
is handed back unchanged, so the client's own rule is what refuses it.

Rows for a carriage return, a line feed, and a second Set-Cookie header
forged behind a line break now stand beside the tab row, and the summary
says why they test the client and not the platform. A rule that refused
the tab and the other two control characters but let these through fails
exactly the three new rows.

Tests only.

Signed-off-by: James Maes <james@kof22.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

james.maes merged commit 65a99d3049 into release/2.0

2026-11-11 03:01:37 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qqq-print-agent#48