From fee73fc69982eb92af53ef13b2fa48b18cd4beab Mon Sep 17 00:00:00 2001 From: James Maes Date: Wed, 13 May 2026 17:22:47 -0500 Subject: [PATCH 1/2] docs(superpowers): brainstorm/spec for multi-runner expansion on Grogu Brainstorm-phase design for scaling Grogu from 1 to 3 self-hosted CI runners. Covers components, precondition (PR #189 must land first), sequential registration sequence, acceptance validation, failure modes + mitigations, rollback, and explicit out-of-scope items. Acceptance bar (minimum): green develop CI with >=3 SPM jobs in flight simultaneously. Warm cache + runner-health monitoring are explicit follow-ups. Path follows the superpowers:brainstorming skill convention: docs/superpowers/specs/YYYY-MM-DD--design.md. Refs #180 #182 #186 #188 --- .../2026-05-13-multi-runner-grogu-design.md | 133 ++++++++++++++++++ 1 file changed, 133 insertions(+) create mode 100644 docs/superpowers/specs/2026-05-13-multi-runner-grogu-design.md diff --git a/docs/superpowers/specs/2026-05-13-multi-runner-grogu-design.md b/docs/superpowers/specs/2026-05-13-multi-runner-grogu-design.md new file mode 100644 index 0000000..41247a6 --- /dev/null +++ b/docs/superpowers/specs/2026-05-13-multi-runner-grogu-design.md @@ -0,0 +1,133 @@ +# Multi-runner expansion on Grogu — design + +**Date:** 2026-05-13 +**Status:** approved (brainstorm phase, awaiting implementation plan) +**Workstream:** A — local CI capacity scaling +**Related issues:** #180 (single-runner setup), #182 (zombie cleanup), #186 (auto-release), #188 (age-filter) + +## Problem + +Develop CI wall-clock is bottlenecked on a single self-hosted runner (`Grogu-jarvis`) serializing the 13-package SwiftPM matrix. Each cold matrix job takes 1–2 minutes, so a full CI run is 13–15 minutes of serial work plus app-build on github-hosted in parallel. Grogu's hardware (24 physical cores, 128 GB RAM, 1.3 TB free on HD-2) is heavily under-utilized — single-runner load is ~5% of capacity. + +Adding sibling runners would parallelize the matrix and roughly cut CI wall-clock by the runner count. Three runners is the recommended target: comfortably within hardware capacity (~24 cores used at peak when all three are mid-SwiftPM-build) without contending with interactive developer work. + +## Goal + +Three self-hosted runners on Grogu (`Grogu-jarvis`, `Grogu-jarvis-2`, `Grogu-jarvis-3`), each with its own `_work/` directory on HD-2, each registered as a LaunchAgent under james.maes's user session, all sharing labels `[self-hosted, macOS, ARM64, jarvis]` so GitHub Actions dispatches matrix jobs to whichever runner is free. + +**Acceptance bar (minimum):** a green develop CI run after registration where `gh run view --json jobs` shows ≥3 SPM matrix jobs in `in_progress` state simultaneously. Warm-cache persistence and runner-health monitoring are explicit non-goals for this phase — they fall to follow-up tickets. + +## Components + +| Runner | Agent dir | `_work` symlink target | LaunchAgent plist | GitHub name | +|---|---|---|---|---| +| 1 (existing) | `~/actions-runner-jarvis/` | `/Volumes/HD-2/jarvis-runner-work/` | `actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis.plist` | `Grogu-jarvis` | +| 2 (new) | `~/actions-runner-jarvis-2/` | `/Volumes/HD-2/jarvis-runner-work-2/` | `actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-2.plist` | `Grogu-jarvis-2` | +| 3 (new) | `~/actions-runner-jarvis-3/` | `/Volumes/HD-2/jarvis-runner-work-3/` | `actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-3.plist` | `Grogu-jarvis-3` | + +All three runners share the same label set; jobs targeting `runs-on: [self-hosted, macOS, jarvis]` will dispatch to any free runner. + +Agent binaries are v2.334.0 (osx-arm64) — already extracted into the new runner dirs as a pre-staging step; only `config.sh` + `svc.sh install/start` remain. + +## Precondition + +PR #189 (age-filter for the orphan-process cleanup step) **must be on `develop`** before either new runner starts accepting jobs. The current cleanup step kills ANY `xctest`/`swift-test`/`swift-driver` process at job start; with concurrent runners, sibling cleanup steps would mutually destroy each other's in-flight tests. PR #189 narrows the kill set to processes ≥1 hour old — concurrent CI jobs (typically <10 min) are safe. + +The gating sequence is enforced operationally, not via code: don't run `./svc.sh start` on runner 2 or 3 until `git log origin/develop` shows the age-filter merge. + +## Sequence — register one runner + +For each `N` in `{2, 3}`: + +1. Obtain a fresh registration token (1-hour expiry): + ``` + gh api -X POST /repos/KofTwentyTwo/Jarvis/actions/runners/registration-token + ``` +2. Configure: + ``` + cd ~/actions-runner-jarvis-N + ./config.sh \ + --url https://github.com/KofTwentyTwo/Jarvis \ + --token "$TOKEN" \ + --name "Grogu-jarvis-N" \ + --labels "jarvis" \ + --work "_work" \ + --unattended \ + --replace + ``` +3. Install as LaunchAgent + start: + ``` + ./svc.sh install + ./svc.sh start + ``` +4. Verify online via GitHub API: + ``` + gh api /repos/KofTwentyTwo/Jarvis/actions/runners -q '.runners[] | select(.name=="Grogu-jarvis-N") | {status, busy, labels}' + ``` + Expected: `status=online, busy=false, labels=[self-hosted, macOS, ARM64, jarvis]`. + +Sequential ordering (runner 2 fully online before configuring runner 3) makes any misconfiguration easy to bisect. + +## Acceptance validation + +After both runners are online: + +1. Retrigger a stuck or cancelled CI run via `gh run rerun ` (PR #185 already has a failed AgentCore job that can serve as the test bed). +2. While the run is in progress, run: + ``` + gh run view --json jobs -q '.jobs[] | select(.status == "in_progress") | .name' + ``` + Expected: at least 3 SPM matrix job names listed simultaneously, e.g., `SPM Bus`, `SPM Memory`, `SPM Voice`. +3. Cross-verify on Grogu: + ``` + gh api /repos/KofTwentyTwo/Jarvis/actions/runners -q '.runners[] | {name, busy}' + ``` + Expected: at least 3 runners with `busy=true` during peak matrix execution. +4. Wait for the run to terminate. Confirm `conclusion = success` (no cross-runner interference caused failures). + +## Failure modes & mitigations + +| Mode | Probability | Mitigation | +|---|---|---| +| Mutual xctest kill from un-filtered cleanup | High (without #189) | PR #189 precondition — age-filter at ≥1h. Don't start new runners until merged. | +| HD-2 disk pressure | Negligible | 1.3 TB free; 3 `_work` dirs × ~10 GB peak = 30 GB. | +| CPU/RAM saturation | Low | 3 concurrent SwiftPM builds peak at ~24 cores + ~24 GB. Within 24/128 budget. Headroom for interactive work. | +| Keychain races (`unlock-keychain`) | Negligible | `security unlock-keychain` is idempotent; default-keychain set persistently for the user. | +| Shared SwiftPM global cache races | Low | `~/Library/Caches/org.swift.swiftpm` is mostly read-after-populate. If contention surfaces, the per-package `actions/checkout clean: true` wipes per-job state cleanly. | +| Test port conflicts | Low | Jarvis tests use process-local Mach IPC (WKWebView) and per-process temp DBs (SQLite). No fixed-port servers found in audit. | +| One runner goes offline silently | Medium | Surfaces only at the next CI run (other runners pick up jobs). No alerting — explicit follow-up scope. | +| Runner agent auto-update breaks during long-running job | Low | Unlikely on a single-job timescale; runner updates happen between jobs. | + +## Rollback + +Each runner is independently revertible. + +- **Pause a runner** (keeps registration): `cd ~/actions-runner-jarvis-N && ./svc.sh stop` +- **Deregister a runner** (free the slot): + - On Grogu: `./svc.sh uninstall` + - Via GitHub: `gh api -X DELETE /repos/KofTwentyTwo/Jarvis/actions/runners/` (or `./config.sh remove --token $(gh api -X POST /repos/.../actions/runners/remove-token)`) +- **Revert to single runner**: stop + uninstall + deregister runners 2 and 3. Workflow unchanged (the `runs-on` selector targets the shared label set, not specific runners). + +No code or workflow changes are required to roll back, which is the strongest possible safety property for an infrastructure change. + +## Out of scope + +- Warm-cache persistence (`actions/checkout clean: false` or custom DerivedData paths). Saves ~3 min per CI run but adds workflow complexity. Follow-up issue. +- Runner-health monitoring / alerting. Currently relies on `gh api .../actions/runners` snapshots and manual log inspection on Grogu. Follow-up issue. +- Per-runner-distinct labels (e.g., `runner-2`, `runner-3`). Default identical labels maximize matrix parallelism; per-runner labels would only matter for targeting a specific machine, which we don't need. +- Scaling beyond 3 runners. Hardware can support more, but matrix size (13 jobs) caps useful parallelism at ~4 runners. Cross that bridge only if matrix grows. + +## Definition of done + +- `gh api /repos/KofTwentyTwo/Jarvis/actions/runners` shows three runners, all `status=online`. +- A retriggered CI run shows ≥3 SPM jobs `in_progress` simultaneously at peak. +- The retriggered run terminates `success` (no failures attributable to multi-runner setup). +- `CLAUDE.md` updated with the new runner inventory and operational notes. +- Spec document committed and merged. + +## References + +- #180 — original single-runner setup +- #182 — zombie cleanup +- #188 — age-filter for multi-runner safety (precondition) +- `~/.claude/plugins/cache/claude-plugins-official/superpowers/5.1.0/skills/brainstorming` — process this design followed -- 2.54.0 From 711a4b6cbb2ff5bf6d6c664b86e3d8960c6abd3e Mon Sep 17 00:00:00 2001 From: James Maes Date: Wed, 13 May 2026 17:28:36 -0500 Subject: [PATCH 2/2] docs(superpowers): implementation plan for multi-runner Grogu expansion MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Bite-sized task breakdown for the multi-runner spec at docs/superpowers/specs/2026-05-13-multi-runner-grogu-design.md. Five tasks, each task several 2-5 minute steps with concrete commands and expected output: 1. Verify PR #189 (age-filter) is on develop — precondition gate 2. Register Grogu-jarvis-2 (verify pre-stage, token, config, install service, verify online) 3. Register Grogu-jarvis-3 (same sequence) 4. Acceptance — rerun PR #185 CI, observe >=3 SPM jobs concurrent 5. Update CLAUDE.md runner inventory + open PR Self-review pass: every spec component covered, no placeholders, type-name consistency verified across tasks. Refs #180 #188 --- .../plans/2026-05-13-multi-runner-grogu.md | 374 ++++++++++++++++++ 1 file changed, 374 insertions(+) create mode 100644 docs/superpowers/plans/2026-05-13-multi-runner-grogu.md diff --git a/docs/superpowers/plans/2026-05-13-multi-runner-grogu.md b/docs/superpowers/plans/2026-05-13-multi-runner-grogu.md new file mode 100644 index 0000000..4d7f534 --- /dev/null +++ b/docs/superpowers/plans/2026-05-13-multi-runner-grogu.md @@ -0,0 +1,374 @@ +# Multi-Runner Expansion on Grogu — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Scale Grogu from 1 to 3 self-hosted GitHub Actions runners so the 13-package SwiftPM matrix parallelizes, cutting develop CI wall-clock from ~13 min serial to ~5 min. + +**Architecture:** Two new LaunchAgent runners (`Grogu-jarvis-2`, `Grogu-jarvis-3`), each in its own `~/actions-runner-jarvis-{2,3}/` directory with `_work/` symlinked to `/Volumes/HD-2/jarvis-runner-work-{2,3}/`. All three runners share labels `[self-hosted, macOS, ARM64, jarvis]` so matrix jobs dispatch to whichever is free. + +**Tech Stack:** GitHub Actions self-hosted runner v2.334.0 (osx-arm64), `gh` CLI for token + verification, macOS LaunchAgents for per-user service management. + +**Spec:** `docs/superpowers/specs/2026-05-13-multi-runner-grogu-design.md` + +**Related issues:** #180 (single-runner setup), #182 (zombie cleanup), #186 (auto-release), #188 (age-filter — precondition). + +--- + +## File Structure + +This plan is primarily operational; the only repo file modified is `CLAUDE.md` (runner inventory documentation). + +| Change | Path | Responsibility | +|---|---|---| +| **System (new)** | `~/actions-runner-jarvis-2/` | Runner-2 agent binary tree (already pre-staged) | +| **System (new)** | `~/actions-runner-jarvis-2/_work` → `/Volumes/HD-2/jarvis-runner-work-2/` | Symlink for build artifacts (already pre-staged) | +| **System (new)** | `~/Library/LaunchAgents/actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-2.plist` | LaunchAgent service plist (created by `svc.sh install`) | +| **System (new)** | `~/actions-runner-jarvis-3/` | Runner-3 agent binary tree (already pre-staged) | +| **System (new)** | `~/actions-runner-jarvis-3/_work` → `/Volumes/HD-2/jarvis-runner-work-3/` | Symlink (already pre-staged) | +| **System (new)** | `~/Library/LaunchAgents/actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-3.plist` | LaunchAgent plist | +| **Repo (modify)** | `CLAUDE.md` | Runner inventory section updated to list all 3 runners | + +The pre-staging step (download + extract agent + create `_work` symlinks for runner 2 and 3) was performed earlier in the session. Task 2 step 1 verifies that pre-stage rather than redoing it. + +--- + +## Task 1: Verify precondition — PR #189 merged to develop + +**Files:** +- Read-only check: `origin/develop` git log + +PR #189 (age-filter zombie reaper) MUST be on `develop` before either new runner starts accepting jobs. The current cleanup step kills any `xctest`/`swift-test`/`swift-driver` regardless of age; concurrent runners' cleanup steps would mutually kill each other's in-flight tests. + +- [ ] **Step 1: Fetch latest develop** + +```bash +git fetch origin develop +``` + +Expected: no output, or "Updating ..". + +- [ ] **Step 2: Verify the age-filter commit is on develop** + +```bash +git log origin/develop --grep="age-filter zombie reaper" --oneline | head -3 +``` + +Expected: at least one line containing `676c54d ci(actions): age-filter zombie reaper (>= 1h) for multi-runner safety` (the exact SHA prefix may differ post-squash-merge). + +If empty: PR #189 is not yet merged. Stop the plan here and re-run from Task 1 once it is. Do NOT proceed to Task 2 — starting concurrent runners against an un-filtered cleanup step will trigger mutual xctest kill. + +--- + +## Task 2: Register Grogu-jarvis-2 + +**Files:** +- System: `~/actions-runner-jarvis-2/` (pre-staged binaries — verify only) +- System: `~/actions-runner-jarvis-2/_work` symlink (pre-staged — verify only) +- System (new): `~/actions-runner-jarvis-2/.runner` (created by `config.sh`) +- System (new): `~/Library/LaunchAgents/actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-2.plist` (created by `svc.sh install`) + +- [ ] **Step 1: Verify pre-stage** + +```bash +test -x ~/actions-runner-jarvis-2/config.sh && \ + readlink ~/actions-runner-jarvis-2/_work | grep -q '/Volumes/HD-2/jarvis-runner-work-2$' && \ + echo "OK: pre-stage intact" || echo "FAIL" +``` + +Expected: `OK: pre-stage intact`. + +If FAIL: re-run the pre-stage block from earlier in the session — `mkdir -p` the dirs on HD-2, download `actions-runner-osx-arm64-2.334.0.tar.gz`, extract into `~/actions-runner-jarvis-2/`, `ln -s /Volumes/HD-2/jarvis-runner-work-2 ~/actions-runner-jarvis-2/_work`. + +- [ ] **Step 2: Get a fresh registration token** + +```bash +TOKEN=$(gh api -X POST /repos/KofTwentyTwo/Jarvis/actions/runners/registration-token -q '.token') +test -n "$TOKEN" && echo "Token acquired (len=${#TOKEN})" || echo "FAIL" +``` + +Expected: `Token acquired (len=33)` or similar non-zero length. Token expires in 1 hour; do not pause between this step and step 4. + +If FAIL: check `gh auth status` — the active account must have `workflow` scope on the KofTwentyTwo/Jarvis repo. + +- [ ] **Step 3: Configure runner-2 (unattended)** + +```bash +cd ~/actions-runner-jarvis-2 +./config.sh \ + --url https://github.com/KofTwentyTwo/Jarvis \ + --token "$TOKEN" \ + --name "Grogu-jarvis-2" \ + --labels "jarvis" \ + --work "_work" \ + --unattended \ + --replace 2>&1 | grep -v -i 'token' | tail -10 +``` + +Expected last lines include `√ Connected to GitHub`, `√ Runner successfully added`, `√ Settings Saved.`. The `grep -v token` strips any line containing the literal token string from the logged output as a defense-in-depth measure (the token itself is short-lived but should not be left in scrollback). + +- [ ] **Step 4: Install + start as LaunchAgent** + +```bash +cd ~/actions-runner-jarvis-2 +./svc.sh install +./svc.sh start +./svc.sh status +``` + +Expected last block: +``` +status actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-2: +/Users/james.maes/Library/LaunchAgents/actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-2.plist + +Started: + 0 actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-2 +``` + +(PID will vary.) + +- [ ] **Step 5: Verify runner-2 is online via GitHub API** + +```bash +gh api /repos/KofTwentyTwo/Jarvis/actions/runners \ + -q '.runners[] | select(.name=="Grogu-jarvis-2") | {name, status, busy, labels: [.labels[].name]}' +``` + +Expected: +```json +{"busy":false,"labels":["self-hosted","macOS","ARM64","jarvis"],"name":"Grogu-jarvis-2","status":"online"} +``` + +If `status` is anything other than `online`, check `~/Library/Logs/actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-2/stdout.log` for the runner-side error and stop the plan. + +--- + +## Task 3: Register Grogu-jarvis-3 + +Identical sequence to Task 2 with `-2` replaced by `-3` everywhere. Per the spec's sequential-validation principle, do NOT start runner-3 until runner-2 is confirmed `status=online, busy=false`. + +**Files:** +- System: `~/actions-runner-jarvis-3/` (pre-staged — verify only) +- System: `~/actions-runner-jarvis-3/_work` symlink (pre-staged — verify only) +- System (new): `~/actions-runner-jarvis-3/.runner` (created by `config.sh`) +- System (new): `~/Library/LaunchAgents/actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-3.plist` + +- [ ] **Step 1: Verify pre-stage** + +```bash +test -x ~/actions-runner-jarvis-3/config.sh && \ + readlink ~/actions-runner-jarvis-3/_work | grep -q '/Volumes/HD-2/jarvis-runner-work-3$' && \ + echo "OK: pre-stage intact" || echo "FAIL" +``` + +Expected: `OK: pre-stage intact`. + +- [ ] **Step 2: Get a fresh registration token** + +```bash +TOKEN=$(gh api -X POST /repos/KofTwentyTwo/Jarvis/actions/runners/registration-token -q '.token') +test -n "$TOKEN" && echo "Token acquired (len=${#TOKEN})" || echo "FAIL" +``` + +Expected: `Token acquired (len=33)` or similar. + +- [ ] **Step 3: Configure runner-3 (unattended)** + +```bash +cd ~/actions-runner-jarvis-3 +./config.sh \ + --url https://github.com/KofTwentyTwo/Jarvis \ + --token "$TOKEN" \ + --name "Grogu-jarvis-3" \ + --labels "jarvis" \ + --work "_work" \ + --unattended \ + --replace 2>&1 | grep -v -i 'token' | tail -10 +``` + +Expected last lines: `√ Connected to GitHub`, `√ Runner successfully added`, `√ Settings Saved.` + +- [ ] **Step 4: Install + start as LaunchAgent** + +```bash +cd ~/actions-runner-jarvis-3 +./svc.sh install +./svc.sh start +./svc.sh status +``` + +Expected: `Started: 0 actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-3`. + +- [ ] **Step 5: Verify runner-3 is online** + +```bash +gh api /repos/KofTwentyTwo/Jarvis/actions/runners \ + -q '.runners[] | select(.name=="Grogu-jarvis-3") | {name, status, busy, labels: [.labels[].name]}' +``` + +Expected: +```json +{"busy":false,"labels":["self-hosted","macOS","ARM64","jarvis"],"name":"Grogu-jarvis-3","status":"online"} +``` + +- [ ] **Step 6: Confirm all 3 runners visible** + +```bash +gh api /repos/KofTwentyTwo/Jarvis/actions/runners \ + -q '.runners | length, [.runners[].name] | sort' +``` + +Expected (exactly): +``` +3 +["Grogu-jarvis","Grogu-jarvis-2","Grogu-jarvis-3"] +``` + +If the count is anything other than 3, stop and investigate which runner failed to register. + +--- + +## Task 4: Acceptance — observe matrix parallelism + +**Files:** none. + +**Premise:** PR #185 (`fix/agent-cache-text-blocks-23-24`) was cancelled earlier in this session with one previously-failed SPM job. Rerun it to get a CI run with the new runner topology in effect. Watch for ≥3 SPM jobs in `in_progress` simultaneously. + +- [ ] **Step 1: Identify the PR #185 run id** + +```bash +RUN_ID=$(gh run list --branch fix/agent-cache-text-blocks-23-24 --workflow=CI --limit 1 --json databaseId -q '.[0].databaseId') +echo "PR #185 latest run: $RUN_ID" +``` + +Expected: a numeric run id, e.g. `25827802988`. + +- [ ] **Step 2: Trigger a full rerun** + +```bash +gh run rerun "$RUN_ID" --repo KofTwentyTwo/Jarvis +sleep 5 +gh run view "$RUN_ID" --json status,conclusion -q '"\(.status) / \(.conclusion // "(none)")"' +``` + +Expected: `queued / (none)` or `in_progress / (none)`. (Using `gh run rerun` without `--failed` reruns the whole workflow so we observe a fresh matrix dispatch across all runners.) + +- [ ] **Step 3: Wait for matrix dispatch, then sample concurrent jobs** + +Wait ~60-90 seconds for the webview/boundary jobs to complete and the SPM matrix to start dispatching. + +```bash +sleep 90 +gh run view "$RUN_ID" --json jobs -q '[.jobs[] | select(.name | startswith("SPM ")) | select(.status == "in_progress") | .name] | "in_progress count: \(length)\nnames: \(.)"' +``` + +Expected: `in_progress count: 3` (or higher if the matrix has dispatched more quickly than expected) followed by three SPM job names. + +If `in_progress count: 1`: GitHub may have only just started dispatching. Wait another 60 seconds and re-sample. If still 1 after 5 minutes, investigate — possibly one or both new runners are offline. Cross-check with `gh api /repos/.../actions/runners -q '.runners[].busy'` (should show ≥2 `true`). + +- [ ] **Step 4: Cross-verify on Grogu** + +```bash +gh api /repos/KofTwentyTwo/Jarvis/actions/runners \ + -q '[.runners[] | select(.busy == true) | .name] | "busy runners: \(length)\nnames: \(.)"' +``` + +Expected: `busy runners: 3` and the three runner names. (If 2, the matrix is partially dispatched; if 1, only the original runner is being used — investigate.) + +- [ ] **Step 5: Wait for the run to terminate** + +```bash +gh run watch "$RUN_ID" --repo KofTwentyTwo/Jarvis --exit-status +echo "Exit: $?" +``` + +Expected: process exits with 0 and a `✓ run completed with 'success'` message. Wall-clock from trigger to completion should be approximately 5–7 minutes (vs the prior ~13 min single-runner baseline). If the run conclusion is `failure`, inspect failed jobs via `gh run view "$RUN_ID" --log-failed` to determine whether the failure is multi-runner-related (e.g., test-suite race) or pre-existing flakiness (e.g., the AgentCore deadlock seen earlier). + +--- + +## Task 5: Update CLAUDE.md with the new runner inventory + +**Files:** +- Modify: `CLAUDE.md` (the "Self-hosted CI runner (Grogu)" section) + +- [ ] **Step 1: Create a branch** + +```bash +git fetch origin develop +git checkout -b docs/multi-runner-inventory origin/develop +``` + +Expected: `Switched to a new branch 'docs/multi-runner-inventory'`. + +- [ ] **Step 2: Locate the runner section in CLAUDE.md** + +```bash +grep -n "Where it lives\|Service:\|jarvis-runner.keychain" CLAUDE.md | head -3 +``` + +Expected: three line numbers pointing into the `### Self-hosted CI runner (Grogu)` section. + +- [ ] **Step 3: Replace the "Where it lives" + "Service" bullets with multi-runner equivalents** + +Edit the two bullets via the `Edit` tool. The OLD text: + +``` +- **Where it lives:** `~/actions-runner-jarvis/` on Grogu. `_work/` symlinked to `/Volumes/HD-2/jarvis-runner-work/` (1.3 TB) to keep boot-disk pressure off. +- **Service:** launchd LaunchAgent `actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis` (`~/Library/LaunchAgents/`). Manage with `cd ~/actions-runner-jarvis && ./svc.sh {status,start,stop}`. Only runs while user is logged in. +``` + +The NEW text: + +``` +- **Where they live:** three runners on Grogu — `~/actions-runner-jarvis/` (Grogu-jarvis), `~/actions-runner-jarvis-2/` (Grogu-jarvis-2), `~/actions-runner-jarvis-3/` (Grogu-jarvis-3). Each `_work/` symlinked to `/Volumes/HD-2/jarvis-runner-work{,-2,-3}/` (1.3 TB available, ~10 GB peak per runner) to keep boot-disk pressure off. +- **Service:** one launchd LaunchAgent per runner — `actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis{,-2,-3}.plist` in `~/Library/LaunchAgents/`. Manage individually with `cd ~/actions-runner-jarvis-N && ./svc.sh {status,start,stop}` (omit `-N` for runner 1). All three only run while user is logged in. +- **Why three:** the 13-package SPM matrix parallelizes across runners. Single-runner serial = ~13 min CI; three-runner parallel = ~5 min. Grogu has 24 cores / 128 GB RAM; 3 concurrent SwiftPM cold builds peak at ~24 cores + ~24 GB, well within budget. See `docs/superpowers/specs/2026-05-13-multi-runner-grogu-design.md`. +``` + +- [ ] **Step 4: Stage + commit** + +```bash +git add CLAUDE.md +git diff --cached --stat +git commit -m "docs(claude-md): runner inventory updated for 3-runner Grogu setup + +Replaces single-runner 'Where it lives' / 'Service' bullets with the +3-runner inventory (Grogu-jarvis, -2, -3). Adds a brief 'Why three' +bullet documenting the parallelism win and links to the spec. + +Refs #180" +``` + +Expected output: `[docs/multi-runner-inventory ] docs(claude-md): runner inventory updated...` and `1 file changed`. + +- [ ] **Step 5: Push + open PR** + +```bash +git push -u origin docs/multi-runner-inventory +gh pr create --base develop --head docs/multi-runner-inventory \ + --title "docs(claude-md): runner inventory updated for 3-runner Grogu setup" \ + --body "Updates CLAUDE.md's \`Self-hosted CI runner\` section to reflect the new 3-runner Grogu topology (Grogu-jarvis, -2, -3) post-multi-runner-expansion. No behavior change — documentation only. + +Spec: \`docs/superpowers/specs/2026-05-13-multi-runner-grogu-design.md\` +Plan: \`docs/superpowers/plans/2026-05-13-multi-runner-grogu.md\` + +Refs #180" +``` + +Expected: the new PR URL is printed. Note the PR number; the runner-side CI for this PR is itself an acceptance test (it will exercise the new 3-runner topology). + +--- + +## Definition of done + +- All steps above are checked off. +- `gh api /repos/KofTwentyTwo/Jarvis/actions/runners` returns 3 runners, all `status=online`. +- The retriggered PR #185 CI run terminates with `conclusion=success` and showed ≥3 SPM jobs `in_progress` simultaneously during execution. +- `docs/multi-runner-inventory` PR is open (or merged) updating CLAUDE.md. + +## Self-review (run by author before handoff) + +- ✅ **Spec coverage:** every component in the spec is covered by a task — runner registration (Tasks 2+3), acceptance validation (Task 4), doc update (Task 5), precondition (Task 1). +- ✅ **No placeholders:** every step has concrete commands and expected output. +- ✅ **Type consistency:** runner names (`Grogu-jarvis-N`), labels (`jarvis`), plist paths, and `_work` symlink targets are spelled identically across tasks. +- ✅ **Spec out-of-scope items not present:** no warm-cache work, no monitoring/alerting, no per-runner labels. +- ✅ **Rollback path documented in spec, referenced as needed.** -- 2.54.0