WIP: Recover presence leases after Redis state loss #47

2026-10-26T11:43:26Z

KofTwentyTwo commented

2026-10-26 11:43:26 +00:00

(Migrated from github.com)

Redis data loss can make an empty presence snapshot look like a room with no viewers. This candidate preserves a per-table recovery guard, returns 503 during recovery, and rejects invalidated leases so clients rejoin. A host-configured generation separates restored Redis state from current authority; nodes sharing a namespace must agree on lease lifetime.

The v2 key prefix requires coordinated rollout. Cold namespaces enter recovery; ordinary idle expiry retains a known-empty guard. Controlled restore requires stopped old nodes and a never-reused generation outside Redis. Host tooling to enforce this remains incomplete; this does not claim arbitrary rollback or split-brain recovery.

Validation on runtime commit 4b0d6a3: full wrapper verify passed 184 tests with no failures/errors/skips, packaging and coverage passed, and CircleCI job160 passed. Tests cover total loss, retained-guard projection loss, guard loss across untouched records, restored bytes under a fresh generation, HTTP 503/409/rejoin and preserved access checks.

The final browser matrix passed Chromium, Firefox, WebKit, Pixel 7 emulation and iPad emulation against two independent JVMs. Both pages clear stale people during Redis loss and restore viewers without reload. Source/compiled Lua hashes were checked around every profile. This uses fixture authentication and an existing local frontend export, not a newly published frontend artifact, physical devices or deployed Kof22.

The controlled-restore harness saves a real Redis RDB image, verifies both old processes and listeners have stopped, clears and reloads the image, then starts two nodes with a fresh external generation. It requires old-lease rejection on both nodes before normal expiry, recovery 503s, and fresh cross-node rejoin. Redis DEBUG RELOAD is enabled only on the owned disposable loopback fixture.

Local cost diagnostics at 1,000 and 10,000 leases are documented in docs/acceptance/RECOVERY-COST.md. Removing repeated table scans reduced 1,000-lease recovery-renewal p95 from 4.20 ms to 0.82 ms in these individual runs. Single-connection loopback measurements without TLS/HTTP do not establish production capacity or an SLA.

Remaining before adoption: enforced host maintenance/reset and generation wiring, compatible artifact integration, deployed-host qualification, sustained capacity qualification. No recovery snapshot has been published or deployed.

Tracks QRun-IO/qqq#1300; does not close it.

Additional focused verification covers shared cold readiness, persistent known-empty state after normal expiry, and loss of each state/expiry half of record and table projections using server-issued leases. The surviving projection must remain unchanged on rejected renewal.

Redis data loss can make an empty presence snapshot look like a room with no viewers. This candidate preserves a per-table recovery guard, returns 503 during recovery, and rejects invalidated leases so clients rejoin. A host-configured generation separates restored Redis state from current authority; nodes sharing a namespace must agree on lease lifetime. The v2 key prefix requires coordinated rollout. Cold namespaces enter recovery; ordinary idle expiry retains a known-empty guard. Controlled restore requires stopped old nodes and a never-reused generation outside Redis. Host tooling to enforce this remains incomplete; this does not claim arbitrary rollback or split-brain recovery. Validation on runtime commit 4b0d6a3: full wrapper verify passed 184 tests with no failures/errors/skips, packaging and coverage passed, and CircleCI job160 passed. Tests cover total loss, retained-guard projection loss, guard loss across untouched records, restored bytes under a fresh generation, HTTP 503/409/rejoin and preserved access checks. The final browser matrix passed Chromium, Firefox, WebKit, Pixel 7 emulation and iPad emulation against two independent JVMs. Both pages clear stale people during Redis loss and restore viewers without reload. Source/compiled Lua hashes were checked around every profile. This uses fixture authentication and an existing local frontend export, not a newly published frontend artifact, physical devices or deployed Kof22. The controlled-restore harness saves a real Redis RDB image, verifies both old processes and listeners have stopped, clears and reloads the image, then starts two nodes with a fresh external generation. It requires old-lease rejection on both nodes before normal expiry, recovery 503s, and fresh cross-node rejoin. Redis DEBUG RELOAD is enabled only on the owned disposable loopback fixture. Local cost diagnostics at 1,000 and 10,000 leases are documented in docs/acceptance/RECOVERY-COST.md. Removing repeated table scans reduced 1,000-lease recovery-renewal p95 from 4.20 ms to 0.82 ms in these individual runs. Single-connection loopback measurements without TLS/HTTP do not establish production capacity or an SLA. Remaining before adoption: enforced host maintenance/reset and generation wiring, compatible artifact integration, deployed-host qualification, sustained capacity qualification. No recovery snapshot has been published or deployed. Tracks QRun-IO/qqq#1300; does not close it. Additional focused verification covers shared cold readiness, persistent known-empty state after normal expiry, and loss of each state/expiry half of record and table projections using server-issued leases. The surviving projection must remain unchanged on rejected renewal.

You are not authorized to merge this pull request.

This pull request can be merged automatically.

This pull request is marked as a work in progress.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feature/presence-recovery-1300:feature/presence-recovery-1300

git checkout feature/presence-recovery-1300

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qbit-collaboration#47