test: measure presence resources and node loss

Co-Authored-By: GPT-6 (Codex) <noreply@openai.com>
2026-10-26 11:07:11 -05:00
1 parent 5b7c947480
commit ea06d6fb4e
@@ -26,6 +26,16 @@ nearest-rank HTTP p50/p95/p99/max. Renewal throughput includes snapshot checks
 and is not maximum server throughput. All owned JVMs and the Redis container are
 removed in `finally`; the receipt reports cleanup separately from test success.

+Each fixture JVM emits resource samples every 100 ms using Java management APIs
+and the Redis QBit's existing pool metrics. The receipt summarizes sampled heap
+used/committed bytes, active/idle/waiting pool counts and the configured maximum.
+CPU core equivalents are process CPU time divided by observed wall time; they
+can exceed one for a multithreaded JVM. Observation begins after server startup
+and includes lease setup and workload execution. Unsupported CPU counters are
+reported as null. Heap is not total resident memory. Sampled peaks may miss
+short bursts and must not be represented as exact maxima. Raw samples remain in
+node logs; missing node samples or a sampled pool-bound violation fail the run.
+
 Initial smoke run: three JVMs, two tenants, six users each, four tabs per user,
 three rounds: 144 renewals and 36 snapshots passed with tenant isolation,
 duplicate-tab deduplication and two editors retained. Cleanup passed. This short
@@ -38,8 +48,77 @@ tenant membership, deduplication and two editors on both records at every node.
 Cleanup passed. These are measurements from one local Windows/Docker run;
 they do not establish deployment limits or browser propagation latency.

-Remaining qualification includes larger sustained workloads, CPU/memory and pool
-utilization sampling, browser/SSE propagation percentiles, saturation, churn,
+## Sustained-run limitation under investigation
+
+The 1,000-round workload has intermittently failed with HTTP 503 on lease
+renewal. Four runs failed; one completed 96,000 renewals and 12,000 snapshots.
+The passing run does not resolve the failures. Temporary local diagnostics on
+the third failure identified `JedisConnectionException` caused by
+`SocketTimeoutException` while reading the `EVALSHA` response in
+`RedisPresenceStore.mutate`. The configured socket timeout remained two seconds.
+The temporary production-file diagnostics were removed after identifying this
+path. No timeout increase, hidden request retry or production fix is included.
+
+Resource samples did not show pool waiters, but cannot rule out short bursts.
+Redis slow-command timings and latency events are now captured before fixture
+cleanup, alongside Redis CPU, memory and command counters, to distinguish server
+execution cost from transport or scheduling delays. Slow-command summaries omit
+command arguments. These diagnostics belong only to the disposable fixture.
+Until the cause is established and qualified, this workload establishes no
+supported sustained throughput or production alert thresholds.
+
+The timing-instrumented Windows-JVM run failed at round 480, with no commands
+at or above the configured one-millisecond Redis slow-log threshold and no
+latency-monitor events. Redis reported about 93 microseconds per `EVALSHA` on
+average. This weakens a slow-Lua explanation but does not identify the delayed
+component or exclude time spent outside command execution.
+
+## Linux comparison
+
+`--node-image eclipse-temurin:21-jre` optionally runs each JVM in its own Linux
+container on an owned Docker network shared with Redis. The HTTP driver remains
+on the host. Only loopback HTTP ports are published. Compiled fixture classes
+and copied dependency JARs are mounted read-only; Redis remains disposable.
+The existing loopback defaults remain unchanged when this option is absent.
+Container mode records Docker client PIDs separately from container identities.
+It removes the Windows-JVM-to-published-Redis-port path, but also changes the JVM
+platform and scheduling environment; it is a diagnostic comparison, not proof
+of a specific operating-system or Docker defect. Cleanup includes every owned
+node container and the owned network.
+
+```powershell
+python docs/performance/presence_nodes.py --java java --node-image eclipse-temurin:21-jre --nodes 3 --tenants 2 --users 12 --tabs 4 --rounds 1000
+```
+
+The first Linux comparison completed 1,000 rounds in 250.811 seconds: 96,000
+renewals and 12,000 snapshots, with exact membership/editor assertions and
+cleanup passing. Renewal p95/p99 were 15.331/17.760 ms. The sampled active pool
+peak was five of eight, with no sampled waiters. This one run is separate from
+the unresolved Windows failures and does not establish a production SLA.
+
+`--crash-round 50` abruptly kills the first node before that round, then sends
+all existing leases through surviving nodes. It does not acquire replacement
+leases or hide failed requests with retries. Exact tenant membership, duplicate
+tabs and editor counts must remain correct on every surviving node. This tests
+HTTP lease continuity after process loss, not browser/SSE reconnect, ingress
+failover, database durability or a Redis outage.
+
+The Linux crash run passed 100 rounds (9,600 renewals and 996 snapshots) in
+25.148 seconds. The first JVM exited with code 137 at round 50; the remaining
+two retained all issued leases and expected membership. Cleanup passed.
+
+Recorded receipts retain both successful and failing evidence:
+
+- [Windows timeout with Redis diagnostics](results/windows-timeout.json)
+- [Linux 1,000-round run](results/linux-1000-rounds.json)
+- [Linux node crash](results/linux-node-crash.json)
+
+Only the Java executable path is normalized in these receipts. The Linux image
+used for these runs was
+`eclipse-temurin@sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99`.
+
+Remaining qualification includes larger sustained workloads, total resident
+memory and Redis-server resource measurements, browser/SSE propagation percentiles, saturation, churn,
 missed messages and process crashes. Loopback HTTP latency must not be reported
 as browser propagation or a production SLA. No deployed capacity threshold is
 established by this fixture.
@@ -11,9 +11,12 @@ import json
 import math
 import os
 from pathlib import Path
+import shutil
 import socket
 import subprocess
+import sys
 import time
+import urllib.error
 import urllib.request
 import uuid

@@ -33,7 +36,13 @@ def request(node, user, path, body=None, tab=0):
        data=None if body is None else json.dumps(body).encode(),
        headers={"Content-Type": "application/json",
                 "Cookie": f"sessionUUID={user}-capacity-{tab}"})
-    with urllib.request.urlopen(req, timeout=10) as response:
+    try:
+        response = urllib.request.urlopen(req, timeout=10)
+    except urllib.error.HTTPError as error:
+        with error:
+            detail = error.read(4096).decode(errors="replace")
+        raise RuntimeError(f"HTTP {error.code} node={node} user={user} path={path}: {detail}") from error
+    with response:
        if response.status != 200:
            raise AssertionError(f"Unexpected status {response.status}")
        return json.load(response)
@@ -51,17 +60,21 @@ def main():
    """Own child JVMs/container and retain results even when a check fails."""
    parser = argparse.ArgumentParser()
    parser.add_argument("--java", required=True)
+    parser.add_argument("--node-image", help="Optional Linux Java21 image; isolate Redis traffic on an owned Docker network")
    parser.add_argument("--nodes", type=int, default=3)
    parser.add_argument("--tenants", type=int, default=2)
    parser.add_argument("--users", type=int, default=6)
    parser.add_argument("--tabs", type=int, default=4)
    parser.add_argument("--rounds", type=int, default=10)
    parser.add_argument("--workers", type=int, default=8)
+    parser.add_argument("--crash-round", type=int, default=0, help="Kill the first node before this round; zero disables")
    args = parser.parse_args()
    for name, lower, upper in (("nodes", 2, 8), ("tenants", 2, 8), ("users", 2, 100),
                               ("tabs", 2, 8), ("rounds", 1, 1000), ("workers", 1, 64)):
        if not lower <= getattr(args, name) <= upper:
            parser.error(f"{name} must be {lower}..{upper}")
+    if args.crash_round and not 2 <= args.crash_round <= args.rounds:
+        parser.error("crash-round must be zero or 2..rounds")
    repo = Path(__file__).resolve().parents[2]
    output = repo / "target" / ("capacity-nodes-" + uuid.uuid4().hex[:10])
    output.mkdir(parents=True)
@@ -69,28 +82,58 @@ def main():
                                (repo / "target/capacity-classpath.txt").read_text().strip()])
    processes, logs = [], []
    container = "qqq-capacity-" + uuid.uuid4().hex[:12]
+    network = container + "-net"
+    node_containers = []
+    network_created = False
    result = {"status": "FAIL", "parameters": vars(args), "scope": "loopback HTTP, mock identities, memory records"}
    try:
+        network_args = []
+        if args.node_image:
+            result["node_image_id"] = subprocess.check_output(
+                ["docker", "image", "inspect", args.node_image, "--format", "{{.Id}}"], text=True).strip()
+            subprocess.run(["docker", "network", "create", network], check=True, capture_output=True)
+            network_created = True
+            network_args = ["--network", network]
+            dependencies = output / "dependencies"
+            dependencies.mkdir()
+            linux_classpath = ["/fixture/target/test-classes", "/fixture/target/classes"]
+            for index, dependency in enumerate((repo / "target/capacity-classpath.txt").read_text().strip().split(os.pathsep)):
+                destination = dependencies / f"{index}.jar"
+                shutil.copyfile(dependency, destination)
+                linux_classpath.append("/fixture/" + destination.relative_to(repo).as_posix())
+            classpath = ":".join(linux_classpath)
        subprocess.run(["docker", "run", "-d", "--name", container, "-p", "127.0.0.1::6379",
-                        "redis:7.4-alpine"], check=True, capture_output=True)
+                        *network_args,
+                        "redis:7.4-alpine", "redis-server", "--slowlog-log-slower-than", "1000",
+                        "--slowlog-max-len", "128", "--latency-monitor-threshold", "1"], check=True, capture_output=True)
        redis_port = int(subprocess.check_output(["docker", "port", container, "6379"], text=True).strip().rsplit(":", 1)[1])
        nodes = [port() for _ in range(args.nodes)]
        if len(set(nodes)) != len(nodes):
            raise RuntimeError("Port allocation collision; no nodes started")
        for node in nodes:
            argfile = output / f"node-{node}.args"
+            connection_properties = (f"-Dcollab.capacity.redisHost={container}\n-Dcollab.dev.bindAddress=0.0.0.0\n"
+                                     if args.node_image else "")
            argfile.write_text(f'-Xmx384m\n-Dcollab.capacity.tenants={args.tenants}\n-Dcollab.capacity.users={args.users}\n'
+                               + connection_properties +
                               '-cp\n"' + classpath.replace("\\", "/") + '"\n'
                               'com.kingsrook.qbits.collaboration.devserver.CapacityPresenceDevServer\n'
-                               f'{node}\n{redis_port}\n', encoding="utf-8")
+                               + (f'8080\n6379\n' if args.node_image else f'{node}\n{redis_port}\n'), encoding="utf-8")
            log = (output / f"node-{node}.log").open("w")
            logs.append(log)
-            processes.append(subprocess.Popen([args.java, "@" + str(argfile)], cwd=repo,
+            command = [args.java, "@" + str(argfile)]
+            if args.node_image:
+                node_container = f"{container}-node-{node}"
+                node_containers.append(node_container)
+                command = ["docker", "run", "--rm", "--name", node_container, "--network", network,
+                           "-p", f"127.0.0.1:{node}:8080", "--mount", f"type=bind,source={repo},target=/fixture,readonly",
+                           args.node_image, "java", "@/fixture/" + argfile.relative_to(repo).as_posix()]
+            processes.append(subprocess.Popen(command, cwd=repo,
                                              stdout=log, stderr=subprocess.STDOUT,
                                              creationflags=getattr(subprocess, "CREATE_NO_WINDOW", 0)))
        deadline = time.monotonic() + 120
        for node, process in zip(nodes, processes):
-            while f"CAPACITY_PRESENCE_READY {node}" not in (output / f"node-{node}.log").read_text(errors="replace"):
+            while "CAPACITY_PRESENCE_READY " not in (output / f"node-{node}.log").read_text(errors="replace"):
                if process.poll() is not None or time.monotonic() > deadline:
                    raise RuntimeError(f"Node {node} failed startup; inspect {output}")
                time.sleep(.2)
@@ -117,6 +160,16 @@ def main():
        started = time.perf_counter()
        with ThreadPoolExecutor(max_workers=args.workers) as workers:
            for sequence in range(1, args.rounds + 1):
+                result["current_round"] = sequence
+                if sequence == args.crash_round:
+                    crashed = nodes.pop(0)
+                    if args.node_image:
+                        subprocess.run(["docker", "kill", node_containers[0]], check=True, capture_output=True)
+                    else:
+                        processes[0].kill()
+                    processes[0].wait(timeout=10)
+                    result["crash"] = {"round": sequence, "http_port": crashed,
+                                       "surviving_nodes": len(nodes), "process_exit": processes[0].returncode}
                renewals.extend(workers.map(lambda item: renew(item, sequence), enumerate(actors)))
                for tenant in range(args.tenants):
                    expected = {f"load-t{tenant}-u{user}" for user in range(args.users)}
@@ -130,14 +183,35 @@ def main():
                                raise AssertionError(f"Tenant isolation/deduplication mismatch: {actual} != {expected}")
                            if sum(viewer["mode"] == "EDITING" for viewer in viewers) != 2:
                                raise AssertionError("Expected exactly two editors")
+                result["completed_rounds"] = sequence
        elapsed = time.perf_counter() - started
        result.update(status="PASS", seconds=elapsed, renewals=distribution(renewals),
                      snapshots=distribution(snapshots), renewal_throughput=len(renewals) / elapsed,
-                      node_pids=[process.pid for process in processes])
+                      node_pids=[process.pid for process in processes],
+                      pid_kind="docker-client" if args.node_image else "JVM", node_containers=node_containers)
    except Exception as error:
        result["error"] = str(error)
        raise
    finally:
+        failure_active = sys.exc_info()[0] is not None
+        try:
+            slow = json.loads(subprocess.check_output(
+                ["docker", "exec", container, "redis-cli", "--json", "SLOWLOG", "GET", "128"], text=True, timeout=10))
+            result["redis_slow_commands"] = [{"unix_time": entry[1], "duration_us": entry[2], "command": entry[3][0]}
+                                             for entry in slow]
+            result["redis_latency_events"] = json.loads(subprocess.check_output(
+                ["docker", "exec", container, "redis-cli", "--json", "LATENCY", "LATEST"], text=True, timeout=10))
+            for section in ("cpu", "memory", "commandstats", "stats"):
+                value = subprocess.check_output(["docker", "exec", container, "redis-cli", "INFO", section], text=True, timeout=10)
+                (output / f"redis-{section}.txt").write_text(value)
+        except (subprocess.SubprocessError, ValueError) as error:
+            result["redis_diagnostic_error"] = type(error).__name__
+        node_cleanup = True
+        for node_container in node_containers:
+            removed = subprocess.run(["docker", "rm", "-f", node_container], capture_output=True)
+            if removed.returncode != 0:
+                remaining = subprocess.run(["docker", "ps", "-aq", "--filter", f"name=^{node_container}$"], capture_output=True, text=True)
+                node_cleanup = node_cleanup and remaining.returncode == 0 and not remaining.stdout.strip()
        for process in processes:
            if process.poll() is None:
                process.terminate()
@@ -149,13 +223,38 @@ def main():
                process.wait(timeout=10)
        for log in logs:
            log.close()
+        resources = []
+        for node_log in sorted(output.glob("node-*.log")):
+            samples = [json.loads(line.split("CAPACITY_METRIC ", 1)[1])
+                       for line in node_log.read_text(errors="replace").splitlines()
+                       if line.startswith("CAPACITY_METRIC ")]
+            if samples:
+                first, last = samples[0], samples[-1]
+                wall_ms = last["uptimeMillis"] - first["uptimeMillis"]
+                cpu_ns = last["cpuNanos"] - first["cpuNanos"]
+                resources.append({"node_log": node_log.name, "samples": len(samples),
+                                  "observed_wall_ms": wall_ms,
+                                  "cpu_seconds": cpu_ns / 1e9 if first["cpuNanos"] >= 0 else None,
+                                  "cpu_core_equivalents": cpu_ns / (wall_ms * 1e6) if wall_ms > 0 and first["cpuNanos"] >= 0 else None,
+                                  **{key + "_sampled_max": max(sample[key] for sample in samples)
+                                     for key in ("heapUsedBytes", "heapCommittedBytes", "poolActive", "poolIdle", "poolWaiting", "poolMaximum")}})
+                if any(sample["poolActive"] > sample["poolMaximum"] for sample in samples):
+                    result.update(status="FAIL", error="Sampled pool active count exceeded configured maximum")
+        result["resources"] = resources
+        if result["status"] == "PASS" and len(resources) != args.nodes:
+            result.update(status="FAIL", error="Missing resource samples for a node")
        cleanup = subprocess.run(["docker", "rm", "-f", container], capture_output=True)
-        result["cleanup"] = cleanup.returncode == 0 and all(process.poll() is not None for process in processes)
+        network_cleanup = True
+        if network_created:
+            network_cleanup = subprocess.run(["docker", "network", "rm", network], capture_output=True).returncode == 0
+        result["cleanup"] = cleanup.returncode == 0 and node_cleanup and network_cleanup and all(process.poll() is not None for process in processes)
        (output / "result.json").write_text(json.dumps(result, indent=2))
        print(json.dumps(result, indent=2))
        print(output)
        if result["status"] == "PASS" and not result["cleanup"]:
            raise RuntimeError("Workload passed but owned resource cleanup failed")
+        if result["status"] == "FAIL" and not failure_active:
+            raise RuntimeError(result.get("error", "Capacity resource validation failed"))


 if __name__ == "__main__":
@@ -0,0 +1,101 @@
+{
+  "status": "PASS",
+  "parameters": {
+    "java": "java (Java 21)",
+    "node_image": "eclipse-temurin:21-jre",
+    "nodes": 3,
+    "tenants": 2,
+    "users": 12,
+    "tabs": 4,
+    "rounds": 1000,
+    "workers": 8
+  },
+  "scope": "loopback HTTP, mock identities, memory records",
+  "current_round": 1000,
+  "completed_rounds": 1000,
+  "seconds": 250.81072249996942,
+  "renewals": {
+    "samples": 96000,
+    "p50_ms": 6.353199947625399,
+    "p95_ms": 15.330899972468615,
+    "p99_ms": 17.760399961844087,
+    "max_ms": 326.8311999272555
+  },
+  "snapshots": {
+    "samples": 12000,
+    "p50_ms": 13.105699908919632,
+    "p95_ms": 14.261199976317585,
+    "p99_ms": 16.519399941898882,
+    "max_ms": 915.3468000004068
+  },
+  "renewal_throughput": 382.7587554595546,
+  "node_pids": [
+    64000,
+    40240,
+    115400
+  ],
+  "pid_kind": "docker-client",
+  "node_containers": [
+    "qqq-capacity-0a8b7956499d-node-64951",
+    "qqq-capacity-0a8b7956499d-node-64952",
+    "qqq-capacity-0a8b7956499d-node-64953"
+  ],
+  "redis_slow_commands": [
+    {
+      "unix_time": 1793030517,
+      "duration_us": 1117,
+      "command": "EVALSHA"
+    }
+  ],
+  "redis_latency_events": [
+    [
+      "command",
+      1793030517,
+      1,
+      1
+    ]
+  ],
+  "resources": [
+    {
+      "node_log": "node-64951.log",
+      "samples": 2580,
+      "observed_wall_ms": 257899,
+      "cpu_seconds": 165.55,
+      "cpu_core_equivalents": 0.6419179601316795,
+      "heapUsedBytes_sampled_max": 166667368,
+      "heapCommittedBytes_sampled_max": 188743680,
+      "poolActive_sampled_max": 4,
+      "poolIdle_sampled_max": 5,
+      "poolWaiting_sampled_max": 0,
+      "poolMaximum_sampled_max": 8
+    },
+    {
+      "node_log": "node-64952.log",
+      "samples": 2569,
+      "observed_wall_ms": 256799,
+      "cpu_seconds": 165.85,
+      "cpu_core_equivalents": 0.6458358482704372,
+      "heapUsedBytes_sampled_max": 169216944,
+      "heapCommittedBytes_sampled_max": 192937984,
+      "poolActive_sampled_max": 5,
+      "poolIdle_sampled_max": 4,
+      "poolWaiting_sampled_max": 0,
+      "poolMaximum_sampled_max": 8
+    },
+    {
+      "node_log": "node-64953.log",
+      "samples": 2608,
+      "observed_wall_ms": 260700,
+      "cpu_seconds": 164.72,
+      "cpu_core_equivalents": 0.631837360951285,
+      "heapUsedBytes_sampled_max": 166319008,
+      "heapCommittedBytes_sampled_max": 188743680,
+      "poolActive_sampled_max": 4,
+      "poolIdle_sampled_max": 5,
+      "poolWaiting_sampled_max": 0,
+      "poolMaximum_sampled_max": 8
+    }
+  ],
+  "cleanup": true,
+  "receipt_note": "Java executable path normalized; measurements unchanged. Disposable fixture identities only."
+}
@@ -0,0 +1,96 @@
+{
+  "status": "PASS",
+  "parameters": {
+    "java": "java (Java 21)",
+    "node_image": "eclipse-temurin@sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99",
+    "nodes": 3,
+    "tenants": 2,
+    "users": 12,
+    "tabs": 4,
+    "rounds": 100,
+    "workers": 8,
+    "crash_round": 50
+  },
+  "scope": "loopback HTTP, mock identities, memory records",
+  "node_image_id": "sha256:227118764cccd3c3294617061aaa93a08309c63a395eefa83e6b35032992ea99",
+  "current_round": 100,
+  "completed_rounds": 100,
+  "crash": {
+    "round": 50,
+    "http_port": 54039,
+    "surviving_nodes": 2,
+    "process_exit": 137
+  },
+  "seconds": 25.147547999978997,
+  "renewals": {
+    "samples": 9600,
+    "p50_ms": 7.125200005248189,
+    "p95_ms": 17.407900071702898,
+    "p99_ms": 27.217599912546575,
+    "max_ms": 74.53640003222972
+  },
+  "snapshots": {
+    "samples": 996,
+    "p50_ms": 13.538200058974326,
+    "p95_ms": 15.59239998459816,
+    "p99_ms": 20.58999997097999,
+    "max_ms": 315.0679999962449
+  },
+  "renewal_throughput": 381.7469599822622,
+  "node_pids": [
+    85832,
+    90280,
+    67764
+  ],
+  "pid_kind": "docker-client",
+  "node_containers": [
+    "qqq-capacity-e599dca8e163-node-54039",
+    "qqq-capacity-e599dca8e163-node-54040",
+    "qqq-capacity-e599dca8e163-node-54041"
+  ],
+  "redis_slow_commands": [],
+  "redis_latency_events": [],
+  "resources": [
+    {
+      "node_log": "node-54039.log",
+      "samples": 198,
+      "observed_wall_ms": 19699,
+      "cpu_seconds": 28.58,
+      "cpu_core_equivalents": 1.4508350677699375,
+      "heapUsedBytes_sampled_max": 163478640,
+      "heapCommittedBytes_sampled_max": 188743680,
+      "poolActive_sampled_max": 3,
+      "poolIdle_sampled_max": 4,
+      "poolWaiting_sampled_max": 0,
+      "poolMaximum_sampled_max": 8
+    },
+    {
+      "node_log": "node-54040.log",
+      "samples": 300,
+      "observed_wall_ms": 29900,
+      "cpu_seconds": 41.85,
+      "cpu_core_equivalents": 1.399665551839465,
+      "heapUsedBytes_sampled_max": 165177008,
+      "heapCommittedBytes_sampled_max": 188743680,
+      "poolActive_sampled_max": 4,
+      "poolIdle_sampled_max": 5,
+      "poolWaiting_sampled_max": 0,
+      "poolMaximum_sampled_max": 8
+    },
+    {
+      "node_log": "node-54041.log",
+      "samples": 331,
+      "observed_wall_ms": 33000,
+      "cpu_seconds": 41.52,
+      "cpu_core_equivalents": 1.2581818181818183,
+      "heapUsedBytes_sampled_max": 168546936,
+      "heapCommittedBytes_sampled_max": 192937984,
+      "poolActive_sampled_max": 4,
+      "poolIdle_sampled_max": 5,
+      "poolWaiting_sampled_max": 0,
+      "poolMaximum_sampled_max": 8
+    }
+  ],
+  "cleanup": true,
+  "receipt_note": "Java executable path normalized; measurements unchanged. Disposable fixture identities only."
+}
@@ -0,0 +1,61 @@
+{
+  "status": "FAIL",
+  "parameters": {
+    "java": "java (Java 21)",
+    "nodes": 3,
+    "tenants": 2,
+    "users": 12,
+    "tabs": 4,
+    "rounds": 1000,
+    "workers": 8
+  },
+  "scope": "loopback HTTP, mock identities, memory records",
+  "current_round": 480,
+  "completed_rounds": 479,
+  "error": "HTTP 503 node=56028 user=load-t0-u5 path=/presence/leases/renew: {\"error\":\"Presence is temporarily unavailable. Please retry.\",\"code\":\"ERROR\",\"message\":\"Presence is temporarily unavailable. Please retry.\",\"traceId\":\"cea30b82-e82b-49ec-af8a-64d8b6bd5af6\"}",
+  "redis_slow_commands": [],
+  "redis_latency_events": [],
+  "resources": [
+    {
+      "node_log": "node-56027.log",
+      "samples": 1067,
+      "observed_wall_ms": 106600,
+      "cpu_seconds": 94.328125,
+      "cpu_core_equivalents": 0.8848792213883677,
+      "heapUsedBytes_sampled_max": 160559696,
+      "heapCommittedBytes_sampled_max": 178257920,
+      "poolActive_sampled_max": 5,
+      "poolIdle_sampled_max": 5,
+      "poolWaiting_sampled_max": 0,
+      "poolMaximum_sampled_max": 8
+    },
+    {
+      "node_log": "node-56028.log",
+      "samples": 1066,
+      "observed_wall_ms": 106501,
+      "cpu_seconds": 95.8125,
+      "cpu_core_equivalents": 0.8996394400052582,
+      "heapUsedBytes_sampled_max": 160251104,
+      "heapCommittedBytes_sampled_max": 178257920,
+      "poolActive_sampled_max": 5,
+      "poolIdle_sampled_max": 4,
+      "poolWaiting_sampled_max": 0,
+      "poolMaximum_sampled_max": 8
+    },
+    {
+      "node_log": "node-56029.log",
+      "samples": 1066,
+      "observed_wall_ms": 106500,
+      "cpu_seconds": 95.25,
+      "cpu_core_equivalents": 0.8943661971830986,
+      "heapUsedBytes_sampled_max": 157995216,
+      "heapCommittedBytes_sampled_max": 175112192,
+      "poolActive_sampled_max": 5,
+      "poolIdle_sampled_max": 4,
+      "poolWaiting_sampled_max": 0,
+      "poolMaximum_sampled_max": 8
+    }
+  ],
+  "cleanup": true,
+  "receipt_note": "Java executable path normalized; measurements unchanged. Disposable fixture identities only."
+}
@@ -59,7 +59,7 @@ public class CapacityPresenceDevServer
      PresenceRegistry.close(instance);
      RealtimeBus.close(instance);
      new RedisQBitProducer().withQBitConfig(new RedisQBitConfig().withConnection("presence",
-         new RedisConnectionConfig().withHost("127.0.0.1").withPort(redisPort).withTls(false).withMaximumConnections(8).withMaximumWaitMillis(500))
+         new RedisConnectionConfig().withHost(System.getProperty("collab.capacity.redisHost", "127.0.0.1")).withPort(redisPort).withTls(false).withMaximumConnections(8).withMaximumWaitMillis(500))
         .withRealtime("presence", "acceptance", "collaboration")).produce(instance);
      instance.getQBits().values().stream().map(bit -> bit.getConfig())
         .filter(CollaborationQBitConfig.class::isInstance).map(CollaborationQBitConfig.class::cast)
@@ -71,17 +71,34 @@ public class CapacityPresenceDevServer
      new com.kingsrook.qqq.backend.core.health.HealthMetaDataProvider().defineAll(instance, "memory", null);
      var health = new com.kingsrook.qqq.backend.core.health.HealthRunner(instance, "acceptance-" + port, java.time.Clock.systemUTC());
      var server = CollaborationDevServer.server(instance, port, false);
+      var sampler = java.util.concurrent.Executors.newSingleThreadScheduledExecutor();
      try
      {
         redis.start(instance);
         transport.start(instance);
         health.start();
         server.start();
+         sampler.scheduleAtFixedRate(() ->
+         {
+            var pool = RedisRuntimeService.connection(instance, "presence").metrics();
+            var heap = java.lang.management.ManagementFactory.getMemoryMXBean().getHeapMemoryUsage();
+            var values = new java.util.LinkedHashMap<String, Object>();
+            values.put("uptimeMillis", java.lang.management.ManagementFactory.getRuntimeMXBean().getUptime());
+            values.put("cpuNanos", ProcessHandle.current().info().totalCpuDuration().map(java.time.Duration::toNanos).orElse(-1L));
+            values.put("heapUsedBytes", heap.getUsed());
+            values.put("heapCommittedBytes", heap.getCommitted());
+            values.put("poolActive", pool.active());
+            values.put("poolIdle", pool.idle());
+            values.put("poolWaiting", pool.waiting());
+            values.put("poolMaximum", pool.maximum());
+            System.out.println("CAPACITY_METRIC " + com.kingsrook.qqq.backend.core.utils.JsonUtils.toJson(values));
+         }, 0, 100, java.util.concurrent.TimeUnit.MILLISECONDS);
         System.out.println("CAPACITY_PRESENCE_READY " + port);
         Thread.currentThread().join();
      }
      finally
      {
+         sampler.shutdownNow();
         server.stop();
         health.stop();
         transport.stop();
@@ -210,7 +210,7 @@ public class CollaborationDevServer extends AbstractQQQApplication
         .withServeLegacyUnversionedMiddlewareAPI(false).withStrictUiApiRoutes(true)
         .withJavalinConfigCustomizer(config ->
         {
-            config.jetty.host = "127.0.0.1";
+            config.jetty.host = System.getProperty("collab.dev.bindAddress", "127.0.0.1");
            userSelection(config);
            MemoryStorage.routes(config);
         }));