perf: avoid repeated table scans during presence recovery

Clean the table when its recovery generation is created, retaining lazy cleanup for record rooms. Add guard-loss regression and diagnostic measurements at 1,000 and 10,000 leases; expose browser profiles and await dashboard redirect before navigation.

Co-Authored-By: GPT-6 (Codex) <noreply@openai.com>

docs/PRESENCE-RECOVERY.md

@@ -51,12 +51,12 @@ version rolling deployment. Existing published v1 snapshots are unchanged.
 
 - Before the patch, a two-client disposable Redis test failed because total loss
   returned an empty snapshot.
-- Five focused tests now pass: full loss, loss of volatile projections while the
+- Six focused tests now pass: full loss, loss of volatile projections while the
   guard survives, restored guard/lease bytes followed by a host generation
   change, bounded generation validation, and rejection of inconsistent lease
-  lifetimes across nodes. These use independent clients in
+  lifetimes across nodes, plus guard loss across previously unvisited records. These use independent clients in
   one JVM, not a deployed HA cluster.
-- Full wrapper verification passed on October 26: 183 tests, no failures,
+- Full wrapper verification passed on October 26: 184 tests, no failures,
   errors or skips; packaging and coverage gates passed. This includes the
   original seven store tests and two-node six-viewer/two-editor scenario.
   The admin dashboard reads every table, so the fixture initializes every

docs/acceptance/RECOVERY-COST.md

@@ -0,0 +1,56 @@
+# Recovery cost diagnostic — October 26, 2026
+
+One Java 21 process, one Redis connection, disposable loopback Redis 7.4-alpine,
+1,000 issued leases, ten viewers per record, no TLS or HTTP. The probe keeps
+all leases alive through the real 45-second recovery window and verifies that
+the final table snapshot contains all 1,000 viewers. Each phase samples 200
+renewals with increasing sequence numbers. This is not a capacity or SLA claim.
+
+| Renewal latency (ms) | During recovery | Ready |
+|---|---:|---:|
+| p50 | 3.6779 | 0.5134 |
+| p95 | 4.1956 | 0.7226 |
+| p99 | 4.5447 | 0.9122 |
+| Maximum | 6.8271 | 1.4886 |
+
+Population took 4,640.799 ms. The run passed all correctness checks. This first
+implementation scanned touched projections on every request during recovery.
+
+After changing table cleanup to occur atomically only when creating its recovery
+generation (individual record cleanup remains lazy), the same diagnostic passed:
+
+| Renewal latency (ms) | Optimized recovery | Ready |
+|---|---:|---:|
+| p50 | 0.6375 | 0.5029 |
+| p95 | 0.8150 | 0.5762 |
+| p99 | 0.9406 | 0.7500 |
+| Maximum | 4.2280 | 1.2251 |
+
+Optimized population took 1,634.486 ms. All 1,000 leases survived. The focused
+recovery/store suite also passed 13 tests, including guard loss with previously
+unvisited record rooms. Full optimized verification passed 184 tests with no failures/errors/skips and coverage gates passed. These are
+individual local runs and need further qualification before claiming supported
+production capacity. They do not isolate every source
+of latency, compare against an earlier release, or measure sustained concurrent
+load, saturation, TLS, HTTP or browser propagation.
+
+At 10,000 entries the optimized diagnostic also passed, retaining all leases.
+Population took 10,687.2105 ms; the same 200-sample phases measured:
+
+| Renewal latency (ms) | Optimized recovery, 10,000 entries | Ready |
+|---|---:|---:|
+| p50 | 0.4978 | 0.5341 |
+| p95 | 0.5914 | 0.7108 |
+| p99 | 0.7255 | 1.2964 |
+| Maximum | 0.7544 | 1.8799 |
+
+Do not infer that larger tables are faster from these separate local runs.
+JIT warmup, scheduling and concurrent machine activity were not controlled.
+The useful result is that the optimization avoids a repeated whole-table scan
+during recovery and both populated-table runs preserved lease correctness.
+
+The explicit `RedisPresenceRecoveryCostProbe` test-source main class is not
+automatically run by the unit suite and imposes no machine-dependent CI gate.
+After test compilation, run it on the test classpath with argument `1000`.
+It fails if the real recovery window ends before the cold measurement, rather
+than silently reporting ready-state latency as recovery latency.

docs/acceptance/presence_recovery_browser.mjs

@@ -3,19 +3,30 @@ import { createRequire } from 'node:module'
 import { execFileSync } from 'node:child_process'
 import path from 'node:path'
 
-const [frontend, first, second, container, output] = process.argv.slice(2)
+const [frontend, first, second, container, output, profile = 'chromium'] = process.argv.slice(2)
 const require = createRequire(path.join(frontend, 'package.json'))
-const { chromium, expect } = require('@playwright/test')
-const browser = await chromium.launch({ headless: true })
+const { chromium, firefox, webkit, devices, expect } = require('@playwright/test')
+const profiles = {
+  chromium: [chromium, devices['Desktop Chrome']],
+  firefox: [firefox, devices['Desktop Firefox']],
+  webkit: [webkit, devices['Desktop Safari']],
+  mobile: [chromium, devices['Pixel 7']],
+  tablet: [webkit, devices['iPad (gen 7)']],
+}
+if (!profiles[profile]) throw new Error(`Unknown browser profile: ${profile}`)
+const [engine, device] = profiles[profile]
+const browser = await engine.launch({ headless: true })
 const contexts = []
 try {
   const pages = []
   for (const [port, user] of [[first, 'alice'], [second, 'bob']]) {
-    const context = await browser.newContext()
+    const context = await browser.newContext(device)
     contexts.push(context)
     await context.tracing.start({ screenshots: true, snapshots: true })
     const page = await context.newPage()
     await page.goto(`http://127.0.0.1:${port}/?createMockSession=true&user=${user}`)
+    // The root export redirects after hydration; finish it before record navigation.
+    await page.waitForURL(url => /^\/app\/?$/.test(url.pathname), { waitUntil: 'domcontentloaded' })
     await page.goto(`http://127.0.0.1:${port}/app/person/1`)
     pages.push(page)
   }
@@ -32,7 +43,7 @@ try {
     await expect(unavailable(page)).toHaveCount(0)
     await page.screenshot({ path: path.join(output, `recovered-${index}.png`) })
   }
-  console.log('PASS: Chromium shows recovery unavailable, clears stale people, and rejoins both JVMs without reload.')
+  console.log(`PASS: ${profile} shows recovery unavailable, clears stale people, and rejoins both JVMs without reload.`)
 } finally {
   for (const [index, context] of contexts.entries()) {
     await context.tracing.stop({ path: path.join(output, `browser-${index}.zip`) })

docs/acceptance/presence_recovery_processes.py

@@ -76,6 +76,7 @@ def main():
     parser.add_argument("--java", required=True)
     parser.add_argument("--repo", type=Path, required=True)
     parser.add_argument("--frontend-repo", type=Path)
+    parser.add_argument("--browser-profile", choices=("chromium", "firefox", "webkit", "mobile", "tablet"), default="chromium")
     args = parser.parse_args()
     repo = args.repo.resolve()
     output = repo / "target" / ("recovery-processes-" + uuid.uuid4().hex[:10])
@@ -110,7 +111,7 @@ def main():
                 time.sleep(0.2)
         if args.frontend_repo:
             subprocess.run(["node", str(repo / "docs/acceptance/presence_recovery_browser.mjs"),
-                            str(args.frontend_repo.resolve()), str(ports[0]), str(ports[1]), container, str(output)], check=True)
+                            str(args.frontend_repo.resolve()), str(ports[0]), str(ports[1]), container, str(output), args.browser_profile], check=True)
             return
         def join():
             """Acquire each person's new lease on a different node."""

src/main/resources/presence/redis-recovery.lua

@@ -3,6 +3,7 @@
 local function presenceRecovery(keys, ttl, candidate, now)
    local encoded = redis.call('GET', keys[#keys])
    local guard
+   local created = false
    if encoded then
       guard = cjson.decode(encoded)
       if type(guard.generation) ~= 'string' or type(guard.readyAt) ~= 'number' or type(guard.liveUntil) ~= 'number' or type(guard.ttl) ~= 'number' then
@@ -17,18 +18,23 @@ local function presenceRecovery(keys, ttl, candidate, now)
       end
    end
    if not guard then
+      created = true
       guard = { generation = candidate, readyAt = now + ttl, liveUntil = 0, ttl = ttl }
       redis.call('SET', keys[#keys], cjson.encode(guard))
    end
    -- During recovery, discard only projections from an invalidated generation.
-   -- This also frees capacity for new reservations without waiting for old TTLs.
+   -- The table is cleaned atomically when its generation is created. Subsequent
+   -- requests only need to clean the touched record; all new table writes carry
+   -- the current generation. Avoid rescanning the whole table on every renewal.
    if now < guard.readyAt then
       for i = 1, #keys - 1, 2 do
-         for _, member in ipairs(redis.call('HKEYS', keys[i + 1])) do
-            local data = cjson.decode(redis.call('HGET', keys[i + 1], member))
-            if data.generation ~= guard.generation then
-               redis.call('HDEL', keys[i + 1], member)
-               redis.call('ZREM', keys[i], member)
+         if created or i < #keys - 2 then
+            for _, member in ipairs(redis.call('HKEYS', keys[i + 1])) do
+               local data = cjson.decode(redis.call('HGET', keys[i + 1], member))
+               if data.generation ~= guard.generation then
+                  redis.call('HDEL', keys[i + 1], member)
+                  redis.call('ZREM', keys[i], member)
+               end
             end
          end
       end

src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceRecoveryCostProbe.java

@@ -0,0 +1,150 @@
+/*
+ * QQQ - Low-code Application Framework for Engineers.
+ * Copyright (C) 2021-2025.  Kingsrook, LLC
+ * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
+ * contact@kingsrook.com
+ * https://github.com/Kingsrook/
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+package com.kingsrook.qbits.collaboration;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException;
+import com.kingsrook.qbits.collaboration.presence.RedisPresenceStore;
+import com.kingsrook.qbits.redis.RedisConnection;
+import com.kingsrook.qbits.redis.api.RedisNamespace;
+import com.kingsrook.qqq.backend.core.utils.JsonUtils;
+import org.testcontainers.containers.GenericContainer;
+import redis.clients.jedis.DefaultJedisClientConfig;
+import redis.clients.jedis.HostAndPort;
+
+/*******************************************************************************
+ ** Explicit local diagnostic, not a CI latency gate or deployment capacity claim.
+ *******************************************************************************/
+public class RedisPresenceRecoveryCostProbe
+{
+   /***************************************************************************
+    ** Compare issued-lease renewal cost during real recovery and after readiness.
+    ***************************************************************************/
+   public static void main(String[] args) throws Exception
+   {
+      int count = args.length == 0 ? 1000 : Integer.parseInt(args[0]);
+      if(count < 10 || count > 10000 || count % 10 != 0)
+      {
+         throw (new IllegalArgumentException("Use a multiple of ten between 10 and 10000"));
+      }
+      try(var redis = new GenericContainer<>("redis:7.4-alpine").withExposedPorts(6379))
+      {
+         redis.start();
+         String host = "localhost".equals(redis.getHost()) ? "127.0.0.1" : redis.getHost();
+         try(var connection = new RedisConnection(new HostAndPort(host, redis.getMappedPort(6379)),
+            DefaultJedisClientConfig.builder().socketTimeoutMillis(5000).build(), 1, Duration.ofSeconds(1)))
+         {
+            var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("cost", "local", "tenant", "presence"), Duration.ofSeconds(45), count);
+            String[] leases = new String[count];
+            long[] sequences = new long[count];
+            long start = System.nanoTime();
+            for(int i = 0; i < count; i++)
+            {
+               leases[i] = store.openLease("user" + i, "Person " + i, "document", Integer.toString(i / 10));
+               renew(store, leases, sequences, i);
+            }
+            double populationMillis = (System.nanoTime() - start) / 1_000_000.0;
+            requireRecovering(store);
+            Map<String, Object> cold = sample(store, leases, sequences);
+            requireRecovering(store);
+            long deadline = System.nanoTime() + Duration.ofSeconds(60).toNanos();
+            while(true)
+            {
+               for(int i = 0; i < count; i++)
+               {
+                  renew(store, leases, sequences, i);
+               }
+               try
+               {
+                  if(store.snapshot("document", null).size() != count)
+                  {
+                     throw (new AssertionError("Population did not survive recovery"));
+                  }
+                  break;
+               }
+               catch(PresenceRecoveringException expected)
+               {
+                  if(System.nanoTime() >= deadline)
+                  {
+                     throw (new AssertionError("Recovery exceeded deadline"));
+                  }
+                  Thread.sleep(5000);
+               }
+            }
+            var result = new LinkedHashMap<String, Object>();
+            result.put("entries", count);
+            result.put("samplesPerPhase", 200);
+            result.put("populationMillis", populationMillis);
+            result.put("recoveryRenewMillis", cold);
+            result.put("readyRenewMillis", sample(store, leases, sequences));
+            result.put("scope", "One JVM, one connection, loopback Redis, no TLS/HTTP; diagnostic only");
+            System.out.println(JsonUtils.toJson(result));
+         }
+      }
+   }
+
+   /***************************************************************************
+    ** Sample the same issued leases with increasing sequence numbers.
+    ***************************************************************************/
+   private static Map<String, Object> sample(RedisPresenceStore store, String[] leases, long[] sequences)
+   {
+      double[] times = new double[200];
+      for(int i = 0; i < times.length; i++)
+      {
+         long before = System.nanoTime();
+         renew(store, leases, sequences, i % leases.length);
+         times[i] = (System.nanoTime() - before) / 1_000_000.0;
+      }
+      Arrays.sort(times);
+      return (Map.of("p50", times[99], "p95", times[189], "p99", times[197], "max", times[199]));
+   }
+
+   /***************************************************************************
+    ** Renewal must not silently fail while measuring its elapsed time.
+    ***************************************************************************/
+   private static void renew(RedisPresenceStore store, String[] leases, long[] sequences, int i)
+   {
+      if(!store.renewLease("user" + i, "Person " + i, leases[i], "document", Integer.toString(i / 10), "VIEWING", null, ++sequences[i]))
+      {
+         throw (new AssertionError("Renewal refused"));
+      }
+   }
+
+   /***************************************************************************
+    ** Do not label a measurement cold if the actual recovery window has ended.
+    ***************************************************************************/
+   private static void requireRecovering(RedisPresenceStore store)
+   {
+      try
+      {
+         store.snapshot("document", "0");
+      }
+      catch(PresenceRecoveringException expected)
+      {
+         return;
+      }
+      throw (new AssertionError("Recovery finished before cold measurement"));
+   }
+}

src/test/java/com/kingsrook/qbits/collaboration/RedisPresenceRecoveryTest.java

@@ -24,6 +24,7 @@ package com.kingsrook.qbits.collaboration;
 
 import java.time.Duration;
 import com.kingsrook.qbits.collaboration.presence.PresenceLeaseExpiredException;
+import com.kingsrook.qbits.collaboration.presence.PresenceRecoveringException;
 import com.kingsrook.qbits.collaboration.presence.RedisPresenceStore;
 import com.kingsrook.qbits.collaboration.presence.RedisPresenceAudience;
 import com.kingsrook.qbits.collaboration.presence.RedisPresenceConfig;
@@ -116,6 +117,37 @@ class RedisPresenceRecoveryTest extends PersonalServiceFixture
    }
 
 
+   /***************************************************************************
+    ** Guard loss cleans the table once and untouched record rooms on demand.
+    ***************************************************************************/
+   @Test
+   void missingGuardRetiresUnvisitedRecordsWithoutRemovingNewLeases() throws Exception
+   {
+      try(var redis = new GenericContainer<>("redis:7.4-alpine").withExposedPorts(6379))
+      {
+         redis.start();
+         try(var connection = new RedisConnection(new HostAndPort(redis.getHost(), redis.getMappedPort(6379)),
+            DefaultJedisClientConfig.builder().build(), 2, Duration.ofMillis(500)))
+         {
+            var store = new RedisPresenceStore(connection.scripts(), new RedisNamespace("test", "guard-loss", "tenant", "presence"), Duration.ofSeconds(2), 2);
+            String oldAlice = store.openLease("alice", "Alice", "document", "1");
+            String oldBob = store.openLease("bob", "Bob", "document", "2");
+            connection.client().del(connection.client().keys("*:recovery").toArray(String[]::new));
+            assertThatThrownBy(() -> store.snapshot("document", null)).isInstanceOf(PresenceRecoveringException.class);
+            String alice = store.openLease("alice", "Alice", "document", "1");
+            String bob = store.openLease("bob", "Bob", "document", "2");
+            assertThatThrownBy(() -> store.renewLease("alice", "Alice", oldAlice, "document", "1", "VIEWING", null, 99L))
+               .isInstanceOf(PresenceLeaseExpiredException.class);
+            assertThatThrownBy(() -> store.renewLease("bob", "Bob", oldBob, "document", "2", "VIEWING", null, 99L))
+               .isInstanceOf(PresenceLeaseExpiredException.class);
+            assertThat(store.renewLease("bob", "Bob", bob, "document", "2", "VIEWING", null, 1L)).isTrue();
+            awaitVisible(store, alice);
+            assertThat(store.snapshot("document", "1")).extracting("userId").containsExactly("alice");
+         }
+      }
+   }
+
+
    /***************************************************************************
     ** Restore must happen in maintenance, followed by a fresh host-owned generation.
     ** Replaying both old guard and lease bytes cannot authorize the new deployment.