Compare commits

...
Author SHA1 Message Date
james.maes 45142840a7 Merge remote-tracking branch 'origin/develop' into docs/multi-runner-inventory 2026-05-14 06:39:45 -05:00
james.maes 774e60a020 ci(actions): create GitHub Release directly in publish-tag job (#187)
The default GITHUB_TOKEN cannot trigger downstream workflows (GitHub
prevents the chain to avoid infinite loops). publish-tag's `git push
origin "$TAG"` therefore never fires release.yml's `push: tags:`
listener — observed on v0.1.0-dev.50, where the tag landed but the
Release object was missing until manually created.

Fold the Release creation into the publish-tag job using `gh release
create` with GH_TOKEN=secrets.GITHUB_TOKEN. Direct API calls work fine
with the default token; only the push→workflow chain is blocked.

- Idempotent: skips if release already exists.
- Notes: auto-generated from `git log $prev..$TAG` plus a Verification
  block linking back to the CI run.
- Prerelease flag respects steps.version.outputs.prerelease (true for
  dev/rc tags, false for vX.Y.Z final tags on main).

release.yml stays untouched as a fallback for manually-pushed tags
(humans pushing tags directly CAN trigger workflows).

Closes #186
2026-05-13 18:28:07 -05:00
james.maes a26f49fdbd docs(superpowers): multi-runner Grogu spec + implementation plan (#190)
* docs(superpowers): brainstorm/spec for multi-runner expansion on Grogu

Brainstorm-phase design for scaling Grogu from 1 to 3 self-hosted CI
runners. Covers components, precondition (PR #189 must land first),
sequential registration sequence, acceptance validation, failure
modes + mitigations, rollback, and explicit out-of-scope items.

Acceptance bar (minimum): green develop CI with >=3 SPM jobs in
flight simultaneously. Warm cache + runner-health monitoring are
explicit follow-ups.

Path follows the superpowers:brainstorming skill convention:
docs/superpowers/specs/YYYY-MM-DD-<topic>-design.md.

Refs #180 #182 #186 #188

* docs(superpowers): implementation plan for multi-runner Grogu expansion

Bite-sized task breakdown for the multi-runner spec at
docs/superpowers/specs/2026-05-13-multi-runner-grogu-design.md.
Five tasks, each task several 2-5 minute steps with concrete commands
and expected output:

  1. Verify PR #189 (age-filter) is on develop — precondition gate
  2. Register Grogu-jarvis-2 (verify pre-stage, token, config,
     install service, verify online)
  3. Register Grogu-jarvis-3 (same sequence)
  4. Acceptance — rerun PR #185 CI, observe >=3 SPM jobs concurrent
  5. Update CLAUDE.md runner inventory + open PR

Self-review pass: every spec component covered, no placeholders,
type-name consistency verified across tasks.

Refs #180 #188
2026-05-13 17:58:29 -05:00
james.maes 7308946c71 docs(claude-md): runner inventory updated for 3-runner Grogu setup
Replaces single-runner "Where it lives" / "Service" bullets with the
3-runner inventory (Grogu-jarvis, -2, -3). Adds a "Why three" bullet
documenting the parallelism win (~13 min serial → ~5 min parallel) and
the Grogu hardware envelope, with a link to the spec.

Refs #180
2026-05-13 17:37:37 -05:00
james.maes 4cdb3560e9 ci(actions): age-filter zombie reaper (>= 1h) for multi-runner safety (#189)
Grogu will host multiple self-hosted runners simultaneously (planned: 3
runners). Each runner's job-start cleanup step previously killed ANY
xctest/swift-test/swift-driver — meaning sibling runners would nuke
each other's in-flight tests.

Real CI jobs complete in <10 min. Zombies from interactive Xcode use
are typically >hours old. A 1-hour age cutoff cleanly separates the
two populations.

awk filter: ps -o etime formats are `MM:SS` (<1h), `HH:MM:SS` (>=1h),
or `DD-HH:MM:SS` (days). Match the latter two by testing for `-` (days
prefix) or three `:`-separated groups.

Single-runner behavior unchanged. Validated against synthetic ps
output covering all three etime format cases.

Closes #188
2026-05-13 17:33:03 -05:00
james.maes 81ce64c3f7 ci(actions): reap orphan xctest/swift-test on self-hosted before each job (#183)
Grogu is a shared workstation, not an ephemeral VM. Interactive Xcode
test runs, aborted swift test sessions, or debugger crashes can leave
orphan xctest/swift-test/swift-driver processes around for hours,
holding Mach services + sockets + ports that subsequent CI jobs block
on when mounting their own test infrastructure.

Concrete incident: develop run 25824607025 hung in SPM AgentCore for
34 min (0% CPU, sleeping state) while 6 unrelated xctest zombies aged
22-24h were alive from prior interactive use. The runner only executes
one job at a time, so any such process at job start belongs to no live
job and is safe to kill.

Add a first step that runs SIGTERM then SIGKILL on any
xctest/swift-test/swift-driver process. Gated to runner.environment ==
'self-hosted' since github-hosted VMs start clean.

Closes #182
2026-05-13 16:37:27 -05:00
james.maes 5fc2d3fabf docs: reconcile Ollama model baseline to actual production names (#184)
The codebase moved to qwen3.6:latest (memory extractor) and
qwen2.5-coder:32b-instruct-q8_0 (agent fallback) in commit 8d0a959, but
CLAUDE.md and the MemoryRegressionCorpusTests doc-comments still cited
the vanilla qwen2.5-coder:32b string. Both updated to match what
ModelID.swift, MemoryExtractor.swift, and BootHealth.swift actually
use today.

No behavior change — documentation alignment only.

Closes #58
2026-05-13 16:25:00 -05:00
5 changed files with 593 additions and 8 deletions

No files matched your search

+76
View File
@@ -154,6 +154,34 @@ jobs:
- Vision
- Voice
steps:
# Grogu is a shared workstation, not an ephemeral VM. Interactive
# Xcode test sessions, debugging, or aborted `swift test` runs can
# leave orphan xctest/swift-test/swift-driver processes around for
# hours, holding Mach services + sockets + ports that a new
# WKWebView mount or test-target launch then blocks on (observed:
# AgentCore hung 34min on develop CI 25824607025 with 6 zombie
# xctests aged 22-24h from prior interactive use).
#
# Age filter (>= 1 hour): Grogu now has SIBLING runners
# (Grogu-jarvis, -2, -3) so concurrent CI jobs can be in flight
# simultaneously. CI jobs complete in <10 min, so a 1-hour cutoff
# safely reaps interactive-use orphans without killing concurrent
# CI work on sibling runners. `ps -o etime` formats: `MM:SS`
# (<1h, skip), `HH:MM:SS` (>=1h, kill), `DD-HH:MM:SS` (days, kill).
# github-hosted VMs start clean, so this step is gated to self-hosted.
- name: Reap orphan xctest/swift-test processes older than 1h (self-hosted only)
if: runner.environment == 'self-hosted'
run: |
age_filter='/(xctest|swift-test|swift-driver)$/ && !/awk/ && (match($2, /-/) || match($2, /[0-9]+:[0-9]+:[0-9]+/)) {print $1}'
/bin/ps -A -o pid,etime,comm \
| awk "$age_filter" \
| xargs -r -I{} kill -TERM {} 2>/dev/null || true
sleep 2
/bin/ps -A -o pid,etime,comm \
| awk "$age_filter" \
| xargs -r -I{} kill -9 {} 2>/dev/null || true
echo "Surviving xctest/swift-test processes (any age — includes concurrent CI work on sibling runners):"
/bin/ps -A -o pid,etime,comm | grep -E '(xctest|swift-test|swift-driver)$' || echo " (none)"
- uses: actions/checkout@v4
# Project targets macOS 26 Tahoe + Swift 6.2/6.3. Xcode 16 (Swift 6.0)
# cannot parse `mlx-audio-swift` v0.1.2's `swift-tools-version:6.2`,
@@ -403,3 +431,51 @@ jobs:
fi
git tag -a "$TAG" -m "Automated tag from CI for $GH_REF"
git push origin "$TAG"
# GitHub Actions GITHUB_TOKEN cannot trigger downstream workflows. The
# separate `release.yml` workflow listens for `push: tags:` events but
# never sees a CI-pushed tag — observed on v0.1.0-dev.50 (tag created,
# Release object missing, had to be manually created). Create the
# Release object directly here in the same job so the Releases page
# reflects CI-pushed tags without a manual step. `release.yml` stays
# as a fallback for manually-pushed tags (which CAN trigger workflows).
- name: Create GitHub Release
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
TAG: ${{ steps.version.outputs.tag }}
PRERELEASE: ${{ steps.version.outputs.prerelease }}
RUN_URL: ${{ format('https://github.com/{0}/actions/runs/{1}', github.repository, github.run_id) }}
run: |
set -euo pipefail
if ! git rev-parse "refs/tags/$TAG" >/dev/null 2>&1; then
echo "Tag $TAG not present locally; nothing to release." >&2
exit 0
fi
if gh release view "$TAG" >/dev/null 2>&1; then
echo "Release $TAG already exists — skipping create (idempotent)."
exit 0
fi
prev=$(git tag --list 'v*' --sort=-v:refname | grep -v "^${TAG}$" | head -n 1 || true)
notes_file=$(mktemp)
{
echo "## Changes"
echo
if [ -n "$prev" ]; then
git log --pretty='format:- %s (%h)' "${prev}..${TAG}"
else
git log --pretty='format:- %s (%h)' -20
fi
echo
echo
echo "## Verification"
echo
echo "- CI build: $RUN_URL"
echo "- Boundary gates, webview, app-build, 13/13 SPM packages: all green on this commit"
} > "$notes_file"
flag=()
if [ "$PRERELEASE" = "true" ]; then
flag+=("--prerelease")
fi
gh release create "$TAG" \
--title "$TAG" \
--notes-file "$notes_file" \
"${flag[@]}"
+8 -6
View File
@@ -64,7 +64,7 @@ The top-level shape is fixed and agreed. Push back with specific reasoning only
- New tokenizer produces ~35% more tokens than Opus 3.x for the same text. Instrument token budgets accordingly; cap `tool_result` content at 8 KB.
- Cache TTL default is 5 minutes ephemeral. For 1-hour TTL, pass `ttl: "1h"` explicitly on `cache_control` blocks **AND** include the request header `anthropic-beta: extended-cache-ttl-2025-04-11`. Without the beta header, 1h is silently ignored. Verify via DevOverlay watching `cache_creation_input_tokens` vs `cache_read_input_tokens` per turn.
- Tool-use and extended thinking both stream; the SSE parser must handle `content_block_start` with `input_json_delta` for tool args. Close on `message_stop`, not `message_delta`; swallow `ping`; route `thinking_delta` to its own case; treat `stop_reason: "refusal"` as first-class; emit `partial_tool_use_at_disconnect` on mid-delta termination.
- **Ollama caveat:** as of April 2026, Qwen 3 / 3.5 / Gemma 4 tool-calling is broken in Ollama — confirmed via live issue tracker (ollama/ollama#14493 Qwen 3.5 27B non-functional; #14601 Qwen3 malformed tool defs via `/api/chat`; #14745 qwen3.5:9b print-not-execute; #15315 Gemma 4 tool parser). Root cause: Ollama's renderer/parser maps Qwen 3.5 through a Hermes-style JSON pipeline, but the model was trained on Qwen3-Coder XML format; unclosed `<think>` tags corrupt multi-turn. **Known-good local tool-calling baseline: `qwen2.5-coder:32b`.** Qwen3 stays disabled as opt-in until upstream closes these issues. Llama 4 with the `llama4_pythonic` parser is worth testing but not yet trusted. Document the local model in use alongside every eval run.
- **Ollama caveat:** as of April 2026, Qwen 3 / 3.5 / Gemma 4 tool-calling is broken in Ollama — confirmed via live issue tracker (ollama/ollama#14493 Qwen 3.5 27B non-functional; #14601 Qwen3 malformed tool defs via `/api/chat`; #14745 qwen3.5:9b print-not-execute; #15315 Gemma 4 tool parser). Root cause: Ollama's renderer/parser maps Qwen 3.5 through a Hermes-style JSON pipeline, but the model was trained on Qwen3-Coder XML format; unclosed `<think>` tags corrupt multi-turn. **Known-good local tool-calling baseline: `qwen2.5-coder:32b-instruct-q8_0`** (the variant pinned by `ModelID.qwen25coder32b`; vanilla `qwen2.5-coder:32b` is not used). Qwen3 stays disabled as opt-in until upstream closes these issues. Llama 4 with the `llama4_pythonic` parser is worth testing but not yet trusted. Document the local model in use alongside every eval run.
- **Ollama transport gotcha:** `/api/chat` (native NDJSON) emits `tool_calls` on the chunk **preceding** the `done: true` terminator, not with it. Decoder must read `tool_calls` whenever seen and never gate on `done`. Separate decoders for `/api/chat` vs `/v1/chat/completions` (OpenAI-compat SSE with atomic `tool_calls`).
- **Tool-choice discipline:** `LLMProvider.stream(..., toolChoice:)` is mandatory. Cap-recovery "one more call" sets `.none` (Anthropic: `{type: "none"}`; Ollama: drop `tools` array entirely). Eval asserts zero `.toolUseRequested` on recovery call.
@@ -87,7 +87,7 @@ The top-level shape is fixed and agreed. Push back with specific reasoning only
**Memory stack:**
- **Store:** SQLite with **FTS5** (keyword search) + **sqlite-vec** extension (vector search), single file under `~/Library/Application Support/Jarvis/`.
- **Embeddings:** `nomic-embed-text` via Ollama (local, free, 768-dim).
- **Extraction:** mem0's `ADD / UPDATE / NOOP` pattern — after each turn, a local extractor (Qwen 2.5-Coder 32B via Ollama) decides what facts to commit/update/ignore. **Fully local; no user data ever leaves the machine for memory.**
- **Extraction:** mem0's `ADD / UPDATE / NOOP` pattern — after each turn, a local extractor (`qwen3.6:latest` via Ollama, per commit `8d0a959`) decides what facts to commit/update/ignore. **Fully local; no user data ever leaves the machine for memory.**
- **Temporal validity:** record `valid_from` / `valid_to` on facts (Zep/Graphiti style) so "Sarah works at Acme" can be superseded without deleting history.
- **Scope tiers:** `UserDefaults` for trivial prefs, JSON for tool configs/feature flags, SQLite for everything conversational.
- Secrets (API keys) → macOS **Keychain**, never plaintext, never checked in.
@@ -227,7 +227,7 @@ Codesign + entitlement verification: `bash scripts/verify-entitlements.sh --pre-
```bash
JARVIS_REAL_MODELS=1 swift test --package-path packages/Memory --filter MemoryRegressionCorpusTests
# requires: local Ollama with nomic-embed-text + qwen2.5-coder:32b + loadable vec0.dylib
# requires: local Ollama with nomic-embed-text + qwen3.6:latest (extractor) + qwen2.5-coder:32b-instruct-q8_0 (agent fallback) + loadable vec0.dylib
JARVIS_REAL_CAMERA=1 swift test --package-path packages/Vision --filter CameraCaptureRealHardwareTests
# requires: real AVCaptureDevice + .authorized TCC
@@ -236,20 +236,22 @@ JARVIS_REAL_MODELS=1 swift test --package-path packages/Voice --filter OrpheusTT
# requires: ~6 GB Orpheus weights downloaded; produces empirical TTFA measurement
```
Eval harness (Anthropic + local-model eval matrix; corpus in `.planning/evals/`) — pinned to `qwen2.5-coder:32b` for the local-model lane.
Eval harness (Anthropic + local-model eval matrix; corpus in `.planning/evals/`) — pinned to `qwen2.5-coder:32b-instruct-q8_0` for the local-model lane (matches `ModelID.qwen25coder32b`).
### Self-hosted CI runner (Grogu)
`swift-packages` matrix in `.github/workflows/ci.yml` runs on a self-hosted macOS runner (`Grogu-jarvis`, labels `[self-hosted, macOS, ARM64, jarvis]`) rather than the github-hosted `macos-15` pool — the public pool was the v0.1 rate-limiter (1–2h queues for 13-job matrix). See #180.
- **Where it lives:** `~/actions-runner-jarvis/` on Grogu. `_work/` symlinked to `/Volumes/HD-2/jarvis-runner-work/` (1.3 TB) to keep boot-disk pressure off.
- **Service:** launchd LaunchAgent `actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis` (`~/Library/LaunchAgents/`). Manage with `cd ~/actions-runner-jarvis && ./svc.sh {status,start,stop}`. Only runs while user is logged in.
- **Where they live:** three runners on Grogu — `~/actions-runner-jarvis/` (Grogu-jarvis), `~/actions-runner-jarvis-2/` (Grogu-jarvis-2), `~/actions-runner-jarvis-3/` (Grogu-jarvis-3). Each `_work/` symlinked to `/Volumes/HD-2/jarvis-runner-work{,-2,-3}/` (1.3 TB available, ~10 GB peak per runner) to keep boot-disk pressure off.
- **Service:** one launchd LaunchAgent per runner — `actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis{,-2,-3}.plist` in `~/Library/LaunchAgents/`. Manage individually with `cd ~/actions-runner-jarvis-N && ./svc.sh {status,start,stop}` (omit `-N` for runner 1). All three only run while user is logged in.
- **Why three:** the 13-package SPM matrix parallelizes across runners. Single-runner serial ≈ ~13 min CI; three-runner parallel ≈ ~5 min. Grogu has 24 cores / 128 GB RAM; 3 concurrent SwiftPM cold builds peak at ~24 cores + ~24 GB, well within budget. See `docs/superpowers/specs/2026-05-13-multi-runner-grogu-design.md`.
- **Security gate:** the job's `if:` clause blocks fork PRs from triggering it — push, workflow_dispatch, and same-repo PRs only. Belt-and-suspenders: enable "Require approval for all outside collaborators" under Settings → Actions → General.
- **Still on github-hosted `macos-15`:** `app-build` (the xcodebuild app target) and the `real-hardware` opt-in lane. Move opportunistically once the swift-packages path is proven.
- **Xcode selection:** uses `DEVELOPER_DIR` env (per-process), tries `Xcode_26.3.app` → `Xcode_26.app` → `Xcode.app` in order. Grogu currently has Xcode 26.5 at the bare path.
- **Cache:** the `actions/cache@v4` step is gated to `runner.environment == 'github-hosted'`. Self-hosted relies on the persistent `_work/` dir on HD-2 (and SwiftPM's natural `.build/` reuse) instead. Github-hosted caches bake absolute `/Users/runner/...` paths into `.build/artifacts/onnxruntime.xcframework` and other XCFrameworks, which Grogu cannot resolve — restricting the cache action keeps these caches strictly per-environment.
- **Runner keychain** (`~/Library/Keychains/jarvis-runner.keychain-db`): created once on Grogu with password `grogu-jarvis-ci` (also stored as repo secret `JARVIS_RUNNER_KEYCHAIN_PASSWORD`). The LaunchAgent has no UI session, so `SecItemAdd` against a locked login keychain returns `errSecInteractionNotAllowed (-25308)`. CI unlocks `jarvis-runner.keychain-db`, prepends it to the user search list, AND sets it as the user's default keychain before `swift test`. The default-keychain step is necessary because on the legacy file-based macOS keychain, `SecItemAdd` writes to the user's *default* keychain (not the first writable in the search list). **Side effects on interactive use:** (1) `jarvis-runner.keychain-db` sits first in your user search list, and (2) it is also your default keychain — so `SecItemAdd` calls *without* an explicit keychain target write there instead of `login.keychain-db`. Apps that target login keychain explicitly (most do) are unaffected. To undo: `security default-keychain -d user -s ~/Library/Keychains/login.keychain-db && security list-keychains -d user -s ~/Library/Keychains/login.keychain-db`.
- **When Grogu reboots or sleeps**, the LaunchAgent restarts on login but jobs queued during downtime will sit waiting. For long offline windows, push to a branch other than develop or temporarily revert `runs-on:` to `macos-15`.
- **Process hygiene:** the workflow's first step on self-hosted is a `Reap orphan xctest/swift-test processes older than 1h` cleanup. Grogu is a shared workstation — interactive Xcode test runs, aborted `swift test`, or debugger sessions can leak `xctest` / `swift-test` / `swift-driver` processes that hold Mach services + sockets and cause subsequent CI jobs to deadlock (observed: AgentCore hung 34 min on a develop run when 6 such zombies aged 22–24h were present). The age filter (≥1h) keeps concurrent CI work on sibling runners safe (jobs complete in <10 min).
# CLAUDE.md
@@ -0,0 +1,374 @@
# Multi-Runner Expansion on Grogu — Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Scale Grogu from 1 to 3 self-hosted GitHub Actions runners so the 13-package SwiftPM matrix parallelizes, cutting develop CI wall-clock from ~13 min serial to ~5 min.
**Architecture:** Two new LaunchAgent runners (`Grogu-jarvis-2`, `Grogu-jarvis-3`), each in its own `~/actions-runner-jarvis-{2,3}/` directory with `_work/` symlinked to `/Volumes/HD-2/jarvis-runner-work-{2,3}/`. All three runners share labels `[self-hosted, macOS, ARM64, jarvis]` so matrix jobs dispatch to whichever is free.
**Tech Stack:** GitHub Actions self-hosted runner v2.334.0 (osx-arm64), `gh` CLI for token + verification, macOS LaunchAgents for per-user service management.
**Spec:** `docs/superpowers/specs/2026-05-13-multi-runner-grogu-design.md`
**Related issues:** #180 (single-runner setup), #182 (zombie cleanup), #186 (auto-release), #188 (age-filter — precondition).
---
## File Structure
This plan is primarily operational; the only repo file modified is `CLAUDE.md` (runner inventory documentation).
| Change | Path | Responsibility |
|---|---|---|
| **System (new)** | `~/actions-runner-jarvis-2/` | Runner-2 agent binary tree (already pre-staged) |
| **System (new)** | `~/actions-runner-jarvis-2/_work` → `/Volumes/HD-2/jarvis-runner-work-2/` | Symlink for build artifacts (already pre-staged) |
| **System (new)** | `~/Library/LaunchAgents/actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-2.plist` | LaunchAgent service plist (created by `svc.sh install`) |
| **System (new)** | `~/actions-runner-jarvis-3/` | Runner-3 agent binary tree (already pre-staged) |
| **System (new)** | `~/actions-runner-jarvis-3/_work` → `/Volumes/HD-2/jarvis-runner-work-3/` | Symlink (already pre-staged) |
| **System (new)** | `~/Library/LaunchAgents/actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-3.plist` | LaunchAgent plist |
| **Repo (modify)** | `CLAUDE.md` | Runner inventory section updated to list all 3 runners |
The pre-staging step (download + extract agent + create `_work` symlinks for runner 2 and 3) was performed earlier in the session. Task 2 step 1 verifies that pre-stage rather than redoing it.
---
## Task 1: Verify precondition — PR #189 merged to develop
**Files:**
- Read-only check: `origin/develop` git log
PR #189 (age-filter zombie reaper) MUST be on `develop` before either new runner starts accepting jobs. The current cleanup step kills any `xctest`/`swift-test`/`swift-driver` regardless of age; concurrent runners' cleanup steps would mutually kill each other's in-flight tests.
- [ ] **Step 1: Fetch latest develop**
```bash
git fetch origin develop
```
Expected: no output, or "Updating <hash>..<hash>".
- [ ] **Step 2: Verify the age-filter commit is on develop**
```bash
git log origin/develop --grep="age-filter zombie reaper" --oneline | head -3
```
Expected: at least one line containing `676c54d ci(actions): age-filter zombie reaper (>= 1h) for multi-runner safety` (the exact SHA prefix may differ post-squash-merge).
If empty: PR #189 is not yet merged. Stop the plan here and re-run from Task 1 once it is. Do NOT proceed to Task 2 — starting concurrent runners against an un-filtered cleanup step will trigger mutual xctest kill.
---
## Task 2: Register Grogu-jarvis-2
**Files:**
- System: `~/actions-runner-jarvis-2/` (pre-staged binaries — verify only)
- System: `~/actions-runner-jarvis-2/_work` symlink (pre-staged — verify only)
- System (new): `~/actions-runner-jarvis-2/.runner` (created by `config.sh`)
- System (new): `~/Library/LaunchAgents/actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-2.plist` (created by `svc.sh install`)
- [ ] **Step 1: Verify pre-stage**
```bash
test -x ~/actions-runner-jarvis-2/config.sh && \
readlink ~/actions-runner-jarvis-2/_work | grep -q '/Volumes/HD-2/jarvis-runner-work-2$' && \
echo "OK: pre-stage intact" || echo "FAIL"
```
Expected: `OK: pre-stage intact`.
If FAIL: re-run the pre-stage block from earlier in the session — `mkdir -p` the dirs on HD-2, download `actions-runner-osx-arm64-2.334.0.tar.gz`, extract into `~/actions-runner-jarvis-2/`, `ln -s /Volumes/HD-2/jarvis-runner-work-2 ~/actions-runner-jarvis-2/_work`.
- [ ] **Step 2: Get a fresh registration token**
```bash
TOKEN=$(gh api -X POST /repos/KofTwentyTwo/Jarvis/actions/runners/registration-token -q '.token')
test -n "$TOKEN" && echo "Token acquired (len=${#TOKEN})" || echo "FAIL"
```
Expected: `Token acquired (len=33)` or similar non-zero length. Token expires in 1 hour; do not pause between this step and step 4.
If FAIL: check `gh auth status` — the active account must have `workflow` scope on the KofTwentyTwo/Jarvis repo.
- [ ] **Step 3: Configure runner-2 (unattended)**
```bash
cd ~/actions-runner-jarvis-2
./config.sh \
--url https://github.com/KofTwentyTwo/Jarvis \
--token "$TOKEN" \
--name "Grogu-jarvis-2" \
--labels "jarvis" \
--work "_work" \
--unattended \
--replace 2>&1 | grep -v -i 'token' | tail -10
```
Expected last lines include `√ Connected to GitHub`, `√ Runner successfully added`, `√ Settings Saved.`. The `grep -v token` strips any line containing the literal token string from the logged output as a defense-in-depth measure (the token itself is short-lived but should not be left in scrollback).
- [ ] **Step 4: Install + start as LaunchAgent**
```bash
cd ~/actions-runner-jarvis-2
./svc.sh install
./svc.sh start
./svc.sh status
```
Expected last block:
```
status actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-2:
/Users/james.maes/Library/LaunchAgents/actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-2.plist
Started:
<PID> 0 actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-2
```
(PID will vary.)
- [ ] **Step 5: Verify runner-2 is online via GitHub API**
```bash
gh api /repos/KofTwentyTwo/Jarvis/actions/runners \
-q '.runners[] | select(.name=="Grogu-jarvis-2") | {name, status, busy, labels: [.labels[].name]}'
```
Expected:
```json
{"busy":false,"labels":["self-hosted","macOS","ARM64","jarvis"],"name":"Grogu-jarvis-2","status":"online"}
```
If `status` is anything other than `online`, check `~/Library/Logs/actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-2/stdout.log` for the runner-side error and stop the plan.
---
## Task 3: Register Grogu-jarvis-3
Identical sequence to Task 2 with `-2` replaced by `-3` everywhere. Per the spec's sequential-validation principle, do NOT start runner-3 until runner-2 is confirmed `status=online, busy=false`.
**Files:**
- System: `~/actions-runner-jarvis-3/` (pre-staged — verify only)
- System: `~/actions-runner-jarvis-3/_work` symlink (pre-staged — verify only)
- System (new): `~/actions-runner-jarvis-3/.runner` (created by `config.sh`)
- System (new): `~/Library/LaunchAgents/actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-3.plist`
- [ ] **Step 1: Verify pre-stage**
```bash
test -x ~/actions-runner-jarvis-3/config.sh && \
readlink ~/actions-runner-jarvis-3/_work | grep -q '/Volumes/HD-2/jarvis-runner-work-3$' && \
echo "OK: pre-stage intact" || echo "FAIL"
```
Expected: `OK: pre-stage intact`.
- [ ] **Step 2: Get a fresh registration token**
```bash
TOKEN=$(gh api -X POST /repos/KofTwentyTwo/Jarvis/actions/runners/registration-token -q '.token')
test -n "$TOKEN" && echo "Token acquired (len=${#TOKEN})" || echo "FAIL"
```
Expected: `Token acquired (len=33)` or similar.
- [ ] **Step 3: Configure runner-3 (unattended)**
```bash
cd ~/actions-runner-jarvis-3
./config.sh \
--url https://github.com/KofTwentyTwo/Jarvis \
--token "$TOKEN" \
--name "Grogu-jarvis-3" \
--labels "jarvis" \
--work "_work" \
--unattended \
--replace 2>&1 | grep -v -i 'token' | tail -10
```
Expected last lines: `√ Connected to GitHub`, `√ Runner successfully added`, `√ Settings Saved.`
- [ ] **Step 4: Install + start as LaunchAgent**
```bash
cd ~/actions-runner-jarvis-3
./svc.sh install
./svc.sh start
./svc.sh status
```
Expected: `Started: <PID> 0 actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-3`.
- [ ] **Step 5: Verify runner-3 is online**
```bash
gh api /repos/KofTwentyTwo/Jarvis/actions/runners \
-q '.runners[] | select(.name=="Grogu-jarvis-3") | {name, status, busy, labels: [.labels[].name]}'
```
Expected:
```json
{"busy":false,"labels":["self-hosted","macOS","ARM64","jarvis"],"name":"Grogu-jarvis-3","status":"online"}
```
- [ ] **Step 6: Confirm all 3 runners visible**
```bash
gh api /repos/KofTwentyTwo/Jarvis/actions/runners \
-q '.runners | length, [.runners[].name] | sort'
```
Expected (exactly):
```
3
["Grogu-jarvis","Grogu-jarvis-2","Grogu-jarvis-3"]
```
If the count is anything other than 3, stop and investigate which runner failed to register.
---
## Task 4: Acceptance — observe matrix parallelism
**Files:** none.
**Premise:** PR #185 (`fix/agent-cache-text-blocks-23-24`) was cancelled earlier in this session with one previously-failed SPM job. Rerun it to get a CI run with the new runner topology in effect. Watch for ≥3 SPM jobs in `in_progress` simultaneously.
- [ ] **Step 1: Identify the PR #185 run id**
```bash
RUN_ID=$(gh run list --branch fix/agent-cache-text-blocks-23-24 --workflow=CI --limit 1 --json databaseId -q '.[0].databaseId')
echo "PR #185 latest run: $RUN_ID"
```
Expected: a numeric run id, e.g. `25827802988`.
- [ ] **Step 2: Trigger a full rerun**
```bash
gh run rerun "$RUN_ID" --repo KofTwentyTwo/Jarvis
sleep 5
gh run view "$RUN_ID" --json status,conclusion -q '"\(.status) / \(.conclusion // "(none)")"'
```
Expected: `queued / (none)` or `in_progress / (none)`. (Using `gh run rerun` without `--failed` reruns the whole workflow so we observe a fresh matrix dispatch across all runners.)
- [ ] **Step 3: Wait for matrix dispatch, then sample concurrent jobs**
Wait ~60-90 seconds for the webview/boundary jobs to complete and the SPM matrix to start dispatching.
```bash
sleep 90
gh run view "$RUN_ID" --json jobs -q '[.jobs[] | select(.name | startswith("SPM ")) | select(.status == "in_progress") | .name] | "in_progress count: \(length)\nnames: \(.)"'
```
Expected: `in_progress count: 3` (or higher if the matrix has dispatched more quickly than expected) followed by three SPM job names.
If `in_progress count: 1`: GitHub may have only just started dispatching. Wait another 60 seconds and re-sample. If still 1 after 5 minutes, investigate — possibly one or both new runners are offline. Cross-check with `gh api /repos/.../actions/runners -q '.runners[].busy'` (should show ≥2 `true`).
- [ ] **Step 4: Cross-verify on Grogu**
```bash
gh api /repos/KofTwentyTwo/Jarvis/actions/runners \
-q '[.runners[] | select(.busy == true) | .name] | "busy runners: \(length)\nnames: \(.)"'
```
Expected: `busy runners: 3` and the three runner names. (If 2, the matrix is partially dispatched; if 1, only the original runner is being used — investigate.)
- [ ] **Step 5: Wait for the run to terminate**
```bash
gh run watch "$RUN_ID" --repo KofTwentyTwo/Jarvis --exit-status
echo "Exit: $?"
```
Expected: process exits with 0 and a `✓ run completed with 'success'` message. Wall-clock from trigger to completion should be approximately 5–7 minutes (vs the prior ~13 min single-runner baseline). If the run conclusion is `failure`, inspect failed jobs via `gh run view "$RUN_ID" --log-failed` to determine whether the failure is multi-runner-related (e.g., test-suite race) or pre-existing flakiness (e.g., the AgentCore deadlock seen earlier).
---
## Task 5: Update CLAUDE.md with the new runner inventory
**Files:**
- Modify: `CLAUDE.md` (the "Self-hosted CI runner (Grogu)" section)
- [ ] **Step 1: Create a branch**
```bash
git fetch origin develop
git checkout -b docs/multi-runner-inventory origin/develop
```
Expected: `Switched to a new branch 'docs/multi-runner-inventory'`.
- [ ] **Step 2: Locate the runner section in CLAUDE.md**
```bash
grep -n "Where it lives\|Service:\|jarvis-runner.keychain" CLAUDE.md | head -3
```
Expected: three line numbers pointing into the `### Self-hosted CI runner (Grogu)` section.
- [ ] **Step 3: Replace the "Where it lives" + "Service" bullets with multi-runner equivalents**
Edit the two bullets via the `Edit` tool. The OLD text:
```
- **Where it lives:** `~/actions-runner-jarvis/` on Grogu. `_work/` symlinked to `/Volumes/HD-2/jarvis-runner-work/` (1.3 TB) to keep boot-disk pressure off.
- **Service:** launchd LaunchAgent `actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis` (`~/Library/LaunchAgents/`). Manage with `cd ~/actions-runner-jarvis && ./svc.sh {status,start,stop}`. Only runs while user is logged in.
```
The NEW text:
```
- **Where they live:** three runners on Grogu — `~/actions-runner-jarvis/` (Grogu-jarvis), `~/actions-runner-jarvis-2/` (Grogu-jarvis-2), `~/actions-runner-jarvis-3/` (Grogu-jarvis-3). Each `_work/` symlinked to `/Volumes/HD-2/jarvis-runner-work{,-2,-3}/` (1.3 TB available, ~10 GB peak per runner) to keep boot-disk pressure off.
- **Service:** one launchd LaunchAgent per runner — `actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis{,-2,-3}.plist` in `~/Library/LaunchAgents/`. Manage individually with `cd ~/actions-runner-jarvis-N && ./svc.sh {status,start,stop}` (omit `-N` for runner 1). All three only run while user is logged in.
- **Why three:** the 13-package SPM matrix parallelizes across runners. Single-runner serial = ~13 min CI; three-runner parallel = ~5 min. Grogu has 24 cores / 128 GB RAM; 3 concurrent SwiftPM cold builds peak at ~24 cores + ~24 GB, well within budget. See `docs/superpowers/specs/2026-05-13-multi-runner-grogu-design.md`.
```
- [ ] **Step 4: Stage + commit**
```bash
git add CLAUDE.md
git diff --cached --stat
git commit -m "docs(claude-md): runner inventory updated for 3-runner Grogu setup
Replaces single-runner 'Where it lives' / 'Service' bullets with the
3-runner inventory (Grogu-jarvis, -2, -3). Adds a brief 'Why three'
bullet documenting the parallelism win and links to the spec.
Refs #180"
```
Expected output: `[docs/multi-runner-inventory <sha>] docs(claude-md): runner inventory updated...` and `1 file changed`.
- [ ] **Step 5: Push + open PR**
```bash
git push -u origin docs/multi-runner-inventory
gh pr create --base develop --head docs/multi-runner-inventory \
--title "docs(claude-md): runner inventory updated for 3-runner Grogu setup" \
--body "Updates CLAUDE.md's \`Self-hosted CI runner\` section to reflect the new 3-runner Grogu topology (Grogu-jarvis, -2, -3) post-multi-runner-expansion. No behavior change — documentation only.
Spec: \`docs/superpowers/specs/2026-05-13-multi-runner-grogu-design.md\`
Plan: \`docs/superpowers/plans/2026-05-13-multi-runner-grogu.md\`
Refs #180"
```
Expected: the new PR URL is printed. Note the PR number; the runner-side CI for this PR is itself an acceptance test (it will exercise the new 3-runner topology).
---
## Definition of done
- All steps above are checked off.
- `gh api /repos/KofTwentyTwo/Jarvis/actions/runners` returns 3 runners, all `status=online`.
- The retriggered PR #185 CI run terminates with `conclusion=success` and showed ≥3 SPM jobs `in_progress` simultaneously during execution.
- `docs/multi-runner-inventory` PR is open (or merged) updating CLAUDE.md.
## Self-review (run by author before handoff)
- ✅ **Spec coverage:** every component in the spec is covered by a task — runner registration (Tasks 2+3), acceptance validation (Task 4), doc update (Task 5), precondition (Task 1).
- ✅ **No placeholders:** every step has concrete commands and expected output.
- ✅ **Type consistency:** runner names (`Grogu-jarvis-N`), labels (`jarvis`), plist paths, and `_work` symlink targets are spelled identically across tasks.
- ✅ **Spec out-of-scope items not present:** no warm-cache work, no monitoring/alerting, no per-runner labels.
- ✅ **Rollback path documented in spec, referenced as needed.**
@@ -0,0 +1,133 @@
# Multi-runner expansion on Grogu — design
**Date:** 2026-05-13
**Status:** approved (brainstorm phase, awaiting implementation plan)
**Workstream:** A — local CI capacity scaling
**Related issues:** #180 (single-runner setup), #182 (zombie cleanup), #186 (auto-release), #188 (age-filter)
## Problem
Develop CI wall-clock is bottlenecked on a single self-hosted runner (`Grogu-jarvis`) serializing the 13-package SwiftPM matrix. Each cold matrix job takes 1–2 minutes, so a full CI run is 13–15 minutes of serial work plus app-build on github-hosted in parallel. Grogu's hardware (24 physical cores, 128 GB RAM, 1.3 TB free on HD-2) is heavily under-utilized — single-runner load is ~5% of capacity.
Adding sibling runners would parallelize the matrix and roughly cut CI wall-clock by the runner count. Three runners is the recommended target: comfortably within hardware capacity (~24 cores used at peak when all three are mid-SwiftPM-build) without contending with interactive developer work.
## Goal
Three self-hosted runners on Grogu (`Grogu-jarvis`, `Grogu-jarvis-2`, `Grogu-jarvis-3`), each with its own `_work/` directory on HD-2, each registered as a LaunchAgent under james.maes's user session, all sharing labels `[self-hosted, macOS, ARM64, jarvis]` so GitHub Actions dispatches matrix jobs to whichever runner is free.
**Acceptance bar (minimum):** a green develop CI run after registration where `gh run view --json jobs` shows ≥3 SPM matrix jobs in `in_progress` state simultaneously. Warm-cache persistence and runner-health monitoring are explicit non-goals for this phase — they fall to follow-up tickets.
## Components
| Runner | Agent dir | `_work` symlink target | LaunchAgent plist | GitHub name |
|---|---|---|---|---|
| 1 (existing) | `~/actions-runner-jarvis/` | `/Volumes/HD-2/jarvis-runner-work/` | `actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis.plist` | `Grogu-jarvis` |
| 2 (new) | `~/actions-runner-jarvis-2/` | `/Volumes/HD-2/jarvis-runner-work-2/` | `actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-2.plist` | `Grogu-jarvis-2` |
| 3 (new) | `~/actions-runner-jarvis-3/` | `/Volumes/HD-2/jarvis-runner-work-3/` | `actions.runner.KofTwentyTwo-Jarvis.Grogu-jarvis-3.plist` | `Grogu-jarvis-3` |
All three runners share the same label set; jobs targeting `runs-on: [self-hosted, macOS, jarvis]` will dispatch to any free runner.
Agent binaries are v2.334.0 (osx-arm64) — already extracted into the new runner dirs as a pre-staging step; only `config.sh` + `svc.sh install/start` remain.
## Precondition
PR #189 (age-filter for the orphan-process cleanup step) **must be on `develop`** before either new runner starts accepting jobs. The current cleanup step kills ANY `xctest`/`swift-test`/`swift-driver` process at job start; with concurrent runners, sibling cleanup steps would mutually destroy each other's in-flight tests. PR #189 narrows the kill set to processes ≥1 hour old — concurrent CI jobs (typically <10 min) are safe.
The gating sequence is enforced operationally, not via code: don't run `./svc.sh start` on runner 2 or 3 until `git log origin/develop` shows the age-filter merge.
## Sequence — register one runner
For each `N` in `{2, 3}`:
1. Obtain a fresh registration token (1-hour expiry):
```
gh api -X POST /repos/KofTwentyTwo/Jarvis/actions/runners/registration-token
```
2. Configure:
```
cd ~/actions-runner-jarvis-N
./config.sh \
--url https://github.com/KofTwentyTwo/Jarvis \
--token "$TOKEN" \
--name "Grogu-jarvis-N" \
--labels "jarvis" \
--work "_work" \
--unattended \
--replace
```
3. Install as LaunchAgent + start:
```
./svc.sh install
./svc.sh start
```
4. Verify online via GitHub API:
```
gh api /repos/KofTwentyTwo/Jarvis/actions/runners -q '.runners[] | select(.name=="Grogu-jarvis-N") | {status, busy, labels}'
```
Expected: `status=online, busy=false, labels=[self-hosted, macOS, ARM64, jarvis]`.
Sequential ordering (runner 2 fully online before configuring runner 3) makes any misconfiguration easy to bisect.
## Acceptance validation
After both runners are online:
1. Retrigger a stuck or cancelled CI run via `gh run rerun <id>` (PR #185 already has a failed AgentCore job that can serve as the test bed).
2. While the run is in progress, run:
```
gh run view <id> --json jobs -q '.jobs[] | select(.status == "in_progress") | .name'
```
Expected: at least 3 SPM matrix job names listed simultaneously, e.g., `SPM Bus`, `SPM Memory`, `SPM Voice`.
3. Cross-verify on Grogu:
```
gh api /repos/KofTwentyTwo/Jarvis/actions/runners -q '.runners[] | {name, busy}'
```
Expected: at least 3 runners with `busy=true` during peak matrix execution.
4. Wait for the run to terminate. Confirm `conclusion = success` (no cross-runner interference caused failures).
## Failure modes & mitigations
| Mode | Probability | Mitigation |
|---|---|---|
| Mutual xctest kill from un-filtered cleanup | High (without #189) | PR #189 precondition — age-filter at ≥1h. Don't start new runners until merged. |
| HD-2 disk pressure | Negligible | 1.3 TB free; 3 `_work` dirs × ~10 GB peak = 30 GB. |
| CPU/RAM saturation | Low | 3 concurrent SwiftPM builds peak at ~24 cores + ~24 GB. Within 24/128 budget. Headroom for interactive work. |
| Keychain races (`unlock-keychain`) | Negligible | `security unlock-keychain` is idempotent; default-keychain set persistently for the user. |
| Shared SwiftPM global cache races | Low | `~/Library/Caches/org.swift.swiftpm` is mostly read-after-populate. If contention surfaces, the per-package `actions/checkout clean: true` wipes per-job state cleanly. |
| Test port conflicts | Low | Jarvis tests use process-local Mach IPC (WKWebView) and per-process temp DBs (SQLite). No fixed-port servers found in audit. |
| One runner goes offline silently | Medium | Surfaces only at the next CI run (other runners pick up jobs). No alerting — explicit follow-up scope. |
| Runner agent auto-update breaks during long-running job | Low | Unlikely on a single-job timescale; runner updates happen between jobs. |
## Rollback
Each runner is independently revertible.
- **Pause a runner** (keeps registration): `cd ~/actions-runner-jarvis-N && ./svc.sh stop`
- **Deregister a runner** (free the slot):
- On Grogu: `./svc.sh uninstall`
- Via GitHub: `gh api -X DELETE /repos/KofTwentyTwo/Jarvis/actions/runners/<id>` (or `./config.sh remove --token $(gh api -X POST /repos/.../actions/runners/remove-token)`)
- **Revert to single runner**: stop + uninstall + deregister runners 2 and 3. Workflow unchanged (the `runs-on` selector targets the shared label set, not specific runners).
No code or workflow changes are required to roll back, which is the strongest possible safety property for an infrastructure change.
## Out of scope
- Warm-cache persistence (`actions/checkout clean: false` or custom DerivedData paths). Saves ~3 min per CI run but adds workflow complexity. Follow-up issue.
- Runner-health monitoring / alerting. Currently relies on `gh api .../actions/runners` snapshots and manual log inspection on Grogu. Follow-up issue.
- Per-runner-distinct labels (e.g., `runner-2`, `runner-3`). Default identical labels maximize matrix parallelism; per-runner labels would only matter for targeting a specific machine, which we don't need.
- Scaling beyond 3 runners. Hardware can support more, but matrix size (13 jobs) caps useful parallelism at ~4 runners. Cross that bridge only if matrix grows.
## Definition of done
- `gh api /repos/KofTwentyTwo/Jarvis/actions/runners` shows three runners, all `status=online`.
- A retriggered CI run shows ≥3 SPM jobs `in_progress` simultaneously at peak.
- The retriggered run terminates `success` (no failures attributable to multi-runner setup).
- `CLAUDE.md` updated with the new runner inventory and operational notes.
- Spec document committed and merged.
## References
- #180 — original single-runner setup
- #182 — zombie cleanup
- #188 — age-filter for multi-runner safety (precondition)
- `~/.claude/plugins/cache/claude-plugins-official/superpowers/5.1.0/skills/brainstorming` — process this design followed
@@ -12,7 +12,7 @@ import OllamaProvider
//
// Gated by JARVIS_REAL_MODELS=1 — skipped in CI. Runs interactively against
// a real local Ollama with both `nomic-embed-text` (embedding) and
// `qwen2.5-coder:32b` (extractor) pulled.
// `qwen3.6:latest` (extractor, per commit 8d0a959) pulled.
//
// Run interactively on the developer's Apple Silicon Mac:
// JARVIS_REAL_MODELS=1 swift test --package-path packages/Memory --filter MemoryRegressionCorpusTests
@@ -37,7 +37,7 @@ final class MemoryRegressionCorpusTests: XCTestCase {
private func skipIfNotRealModels() throws {
try XCTSkipUnless(
ProcessInfo.processInfo.environment["JARVIS_REAL_MODELS"] == "1",
"Set JARVIS_REAL_MODELS=1 to run memory regression corpus (requires local Ollama with nomic-embed-text + qwen2.5-coder:32b, plus a loadable vec0.dylib)"
"Set JARVIS_REAL_MODELS=1 to run memory regression corpus (requires local Ollama with nomic-embed-text + qwen3.6:latest, plus a loadable vec0.dylib)"
)
}