fix: durable assistant turns - lease, heartbeat, cross-pod Stop, never refuse a question #49

2026-10-14T21:37:18Z

KofTwentyTwo commented

2026-10-14 21:37:18 +00:00

(Migrated from github.com)

Why

A production incident on kof22 (RC.11, two backend replicas), 2026-10-14. During a rolling deploy, a turn started on the old pod while that pod was shutting down. The pod died mid-turn and nothing finalised the assistant_turn row, so it stayed RUNNING. The panel's resumed stream on the new pod kept polling a turn that would never end. After a refresh, every question got 429 "You already have an assistant answer in progress". Stop could not help either: on a replica that does not own the turn it only set CANCELLING, which still counts as running. The orphan reaper only fired after maxTurnSeconds + 60 (6 minutes).

The owner's requirements: never block asking, make every run durable and cancellable with a heartbeat, lease and hard max, make Stop always work, re-attach after a refresh, and show runs in the Jobs tray.

What

Lease and heartbeat. A per-turn watchdog runs with the user's own session. Every 5 s it renews heartbeat_at/phase, picks up a Stop requested on another replica (cancel_requested_at), and polls the kill switch. It cancels the turn at maxTurnSeconds and force-finalises a wedged engine after a 20 s grace.
Orphans end within the lease (45 s), not after 6 minutes. A reap happens on replay, on turns/active and on the next question. It is conditional, and a partial unique index guarantees a single message.end per turn.
One idempotent finaliser. Every terminal step (message, turn row, conversation totals, usage, events, tracker) is isolated, so a failure in post-processing can no longer strand a run as RUNNING or leave it registered. If the row is already terminal (another replica or the reaper finished it), the finaliser never overwrites it.
Graceful shutdown. On a JVM shutdown hook the replica stops accepting turns (503 SHUTTING_DOWN, which is safe to retry), cancels its turns, waits 8 s, and finalises any that remain as ERROR/INTERRUPTED ("The server restarted while answering. Ask again.").
Never refuse a question. The 429 "already in progress" refusal and the in-memory concurrent counter are removed. A new question supersedes the running one in the same conversation (the default, recorded as cancelReason=SUPERSEDED with a notice). With concurrency: "concurrent" both run. The per-user cap (now 3) supersedes the oldest run instead of refusing. Per-minute limits and budgets still apply.
Stop works everywhere. A turn on this replica is cancelled at once; one on another replica gets a cancel request that its watchdog applies within a heartbeat; a dead run is reaped as CANCELLED. Cancel is idempotent and returns 200 with the status (202 stays documented for mixed rollouts).
GET /qqq/v1/assistant/turns/active lets the UI find and re-attach to running answers after a refresh. turn.phase events (thinking / tool / writing) and message.start.startedAt drive the progress and elapsed-time display.
Jobs tray. A new AssistantRunTracker SPI (no-op by default) is a STAND-IN(core=QRun-IO/qqq#1210, removeBy=RC.9); it maps 1:1 onto core JobTracking (processName assistant.turn). The label is "Assistant answer", never the title or prompt.
UI API extension assistant 1.2.0. All changes are additive, and 429 remains documented.
Schema. A new block in docs/schema-postgres.sql; see UPGRADE.md. The kof22 Liquibase changeset ships in the Website-Backend pin PR.

Tests

./mvnw verify -pl qbit-assistant -am passes: 275 tests, with the JaCoCo gate met. The new DurableTurnsTest plus AssistantRouteProviderTest cover:

completion
an error after streaming (accounting throws, the turn still finalises)
client disconnect mid-stream and replay
refresh mid-answer (turns/active, then replay that follows live)
concurrent questions
supersede and the soft cap
a stuck provider (timeout and force-finalise)
Stop of a wedged provider
a heartbeat failure that does not kill the watchdog
a pod killed mid-answer (orphan reaped within the lease, a fresh lease left alone, and simultaneous reapers ending the turn once)
a rolling deploy (shutdown finalises the turn, and a start during shutdown returns 503 with nothing inserted)
cross-pod Stop
the tracker, including a tracker that throws.

Implementation was by Codex from a written design. Claude reviewed it, trimmed out-of-scope changes (a checkstyle/pom rewrite and import reorders), and fixed the tracker label so no prompt text leaks.

Companion: QRun-IO/qqq-frontend-next feature/assistant-reattach.

🤖 Generated with Claude Code

## Why A production incident on kof22 (RC.11, two backend replicas), 2026-10-14. During a rolling deploy, a turn started on the old pod while that pod was shutting down. The pod died mid-turn and nothing finalised the `assistant_turn` row, so it stayed `RUNNING`. The panel's resumed stream on the new pod kept polling a turn that would never end. After a refresh, every question got 429 "You already have an assistant answer in progress". Stop could not help either: on a replica that does not own the turn it only set `CANCELLING`, which still counts as running. The orphan reaper only fired after `maxTurnSeconds + 60` (6 minutes). The owner's requirements: never block asking, make every run durable and cancellable with a heartbeat, lease and hard max, make Stop always work, re-attach after a refresh, and show runs in the Jobs tray. ## What - **Lease and heartbeat.** A per-turn watchdog runs with the user's own session. Every 5 s it renews `heartbeat_at`/`phase`, picks up a Stop requested on another replica (`cancel_requested_at`), and polls the kill switch. It cancels the turn at `maxTurnSeconds` and force-finalises a wedged engine after a 20 s grace. - **Orphans end within the lease (45 s), not after 6 minutes.** A reap happens on replay, on `turns/active` and on the next question. It is conditional, and a partial unique index guarantees a single `message.end` per turn. - **One idempotent finaliser.** Every terminal step (message, turn row, conversation totals, usage, events, tracker) is isolated, so a failure in post-processing can no longer strand a run as `RUNNING` or leave it registered. If the row is already terminal (another replica or the reaper finished it), the finaliser never overwrites it. - **Graceful shutdown.** On a JVM shutdown hook the replica stops accepting turns (`503 SHUTTING_DOWN`, which is safe to retry), cancels its turns, waits 8 s, and finalises any that remain as `ERROR/INTERRUPTED` ("The server restarted while answering. Ask again."). - **Never refuse a question.** The 429 "already in progress" refusal and the in-memory concurrent counter are removed. A new question supersedes the running one in the same conversation (the default, recorded as `cancelReason=SUPERSEDED` with a notice). With `concurrency: "concurrent"` both run. The per-user cap (now 3) supersedes the oldest run instead of refusing. Per-minute limits and budgets still apply. - **Stop works everywhere.** A turn on this replica is cancelled at once; one on another replica gets a cancel request that its watchdog applies within a heartbeat; a dead run is reaped as `CANCELLED`. Cancel is idempotent and returns 200 with the status (202 stays documented for mixed rollouts). - **`GET /qqq/v1/assistant/turns/active`** lets the UI find and re-attach to running answers after a refresh. `turn.phase` events (thinking / tool / writing) and `message.start.startedAt` drive the progress and elapsed-time display. - **Jobs tray.** A new `AssistantRunTracker` SPI (no-op by default) is a `STAND-IN(core=QRun-IO/qqq#1210, removeBy=RC.9)`; it maps 1:1 onto core `JobTracking` (processName `assistant.turn`). The label is "Assistant answer", never the title or prompt. - **UI API extension `assistant` 1.2.0.** All changes are additive, and 429 remains documented. - **Schema.** A new block in `docs/schema-postgres.sql`; see UPGRADE.md. The kof22 Liquibase changeset ships in the Website-Backend pin PR. ## Tests `./mvnw verify -pl qbit-assistant -am` passes: 275 tests, with the JaCoCo gate met. The new `DurableTurnsTest` plus `AssistantRouteProviderTest` cover: - completion - an error after streaming (accounting throws, the turn still finalises) - client disconnect mid-stream and replay - refresh mid-answer (`turns/active`, then replay that follows live) - concurrent questions - supersede and the soft cap - a stuck provider (timeout and force-finalise) - Stop of a wedged provider - a heartbeat failure that does not kill the watchdog - a pod killed mid-answer (orphan reaped within the lease, a fresh lease left alone, and simultaneous reapers ending the turn once) - a rolling deploy (shutdown finalises the turn, and a start during shutdown returns 503 with nothing inserted) - cross-pod Stop - the tracker, including a tracker that throws. Implementation was by Codex from a written design. Claude reviewed it, trimmed out-of-scope changes (a checkstyle/pom rewrite and import reorders), and fixed the tracker label so no prompt text leaks. Companion: QRun-IO/qqq-frontend-next feature/assistant-reattach. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-assistant#49