From 676c54da55ddb0fe51f56615df03eb286b078bf4 Mon Sep 17 00:00:00 2001 From: James Maes Date: Wed, 13 May 2026 17:00:48 -0500 Subject: [PATCH] ci(actions): age-filter zombie reaper (>= 1h) for multi-runner safety MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Grogu will host multiple self-hosted runners simultaneously (planned: 3 runners). Each runner's job-start cleanup step previously killed ANY xctest/swift-test/swift-driver — meaning sibling runners would nuke each other's in-flight tests. Real CI jobs complete in <10 min. Zombies from interactive Xcode use are typically >hours old. A 1-hour age cutoff cleanly separates the two populations. awk filter: ps -o etime formats are `MM:SS` (<1h), `HH:MM:SS` (>=1h), or `DD-HH:MM:SS` (days). Match the latter two by testing for `-` (days prefix) or three `:`-separated groups. Single-runner behavior unchanged. Validated against synthetic ps output covering all three etime format cases. Closes #188 --- .github/workflows/ci.yml | 22 ++++++++++++++-------- CLAUDE.md | 2 +- 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9c10fea..58fca49 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -160,21 +160,27 @@ jobs: # hours, holding Mach services + sockets + ports that a new # WKWebView mount or test-target launch then blocks on (observed: # AgentCore hung 34min on develop CI 25824607025 with 6 zombie - # xctests aged 22-24h from prior interactive use). The runner only - # executes one job at a time, so any such process at job-start - # belongs to no live job and is safe to kill. github-hosted VMs - # start clean, so this step is gated to self-hosted. - - name: Reap orphan xctest/swift-test processes (self-hosted only) + # xctests aged 22-24h from prior interactive use). + # + # Age filter (>= 1 hour): Grogu now has SIBLING runners + # (Grogu-jarvis, -2, -3) so concurrent CI jobs can be in flight + # simultaneously. CI jobs complete in <10 min, so a 1-hour cutoff + # safely reaps interactive-use orphans without killing concurrent + # CI work on sibling runners. `ps -o etime` formats: `MM:SS` + # (<1h, skip), `HH:MM:SS` (>=1h, kill), `DD-HH:MM:SS` (days, kill). + # github-hosted VMs start clean, so this step is gated to self-hosted. + - name: Reap orphan xctest/swift-test processes older than 1h (self-hosted only) if: runner.environment == 'self-hosted' run: | + age_filter='/(xctest|swift-test|swift-driver)$/ && !/awk/ && (match($2, /-/) || match($2, /[0-9]+:[0-9]+:[0-9]+/)) {print $1}' /bin/ps -A -o pid,etime,comm \ - | awk '/(xctest|swift-test|swift-driver)$/ && !/awk/ {print $1}' \ + | awk "$age_filter" \ | xargs -r -I{} kill -TERM {} 2>/dev/null || true sleep 2 /bin/ps -A -o pid,etime,comm \ - | awk '/(xctest|swift-test|swift-driver)$/ && !/awk/ {print $1}' \ + | awk "$age_filter" \ | xargs -r -I{} kill -9 {} 2>/dev/null || true - echo "Surviving xctest/swift-test processes after cleanup:" + echo "Surviving xctest/swift-test processes (any age — includes concurrent CI work on sibling runners):" /bin/ps -A -o pid,etime,comm | grep -E '(xctest|swift-test|swift-driver)$' || echo " (none)" - uses: actions/checkout@v4 # Project targets macOS 26 Tahoe + Swift 6.2/6.3. Xcode 16 (Swift 6.0) diff --git a/CLAUDE.md b/CLAUDE.md index cd6e634..489b8fe 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -250,7 +250,7 @@ Eval harness (Anthropic + local-model eval matrix; corpus in `.planning/evals/`) - **Cache:** the `actions/cache@v4` step is gated to `runner.environment == 'github-hosted'`. Self-hosted relies on the persistent `_work/` dir on HD-2 (and SwiftPM's natural `.build/` reuse) instead. Github-hosted caches bake absolute `/Users/runner/...` paths into `.build/artifacts/onnxruntime.xcframework` and other XCFrameworks, which Grogu cannot resolve — restricting the cache action keeps these caches strictly per-environment. - **Runner keychain** (`~/Library/Keychains/jarvis-runner.keychain-db`): created once on Grogu with password `grogu-jarvis-ci` (also stored as repo secret `JARVIS_RUNNER_KEYCHAIN_PASSWORD`). The LaunchAgent has no UI session, so `SecItemAdd` against a locked login keychain returns `errSecInteractionNotAllowed (-25308)`. CI unlocks `jarvis-runner.keychain-db`, prepends it to the user search list, AND sets it as the user's default keychain before `swift test`. The default-keychain step is necessary because on the legacy file-based macOS keychain, `SecItemAdd` writes to the user's *default* keychain (not the first writable in the search list). **Side effects on interactive use:** (1) `jarvis-runner.keychain-db` sits first in your user search list, and (2) it is also your default keychain — so `SecItemAdd` calls *without* an explicit keychain target write there instead of `login.keychain-db`. Apps that target login keychain explicitly (most do) are unaffected. To undo: `security default-keychain -d user -s ~/Library/Keychains/login.keychain-db && security list-keychains -d user -s ~/Library/Keychains/login.keychain-db`. - **When Grogu reboots or sleeps**, the LaunchAgent restarts on login but jobs queued during downtime will sit waiting. For long offline windows, push to a branch other than develop or temporarily revert `runs-on:` to `macos-15`. -- **Process hygiene:** the workflow's first step on self-hosted is a `Reap orphan xctest/swift-test processes` cleanup. Grogu is a shared workstation — interactive Xcode test runs, aborted `swift test`, or debugger sessions can leak `xctest` / `swift-test` / `swift-driver` processes that hold Mach services + sockets and cause subsequent CI jobs to deadlock (observed: AgentCore hung 34 min on a develop run when 6 such zombies aged 22–24h were present). The cleanup is harmless on a clean machine and decisive when not. +- **Process hygiene:** the workflow's first step on self-hosted is a `Reap orphan xctest/swift-test processes older than 1h` cleanup. Grogu is a shared workstation — interactive Xcode test runs, aborted `swift test`, or debugger sessions can leak `xctest` / `swift-test` / `swift-driver` processes that hold Mach services + sockets and cause subsequent CI jobs to deadlock (observed: AgentCore hung 34 min on a develop run when 6 such zombies aged 22–24h were present). The age filter (≥1h) keeps concurrent CI work on sibling runners safe (jobs complete in <10 min). # CLAUDE.md -- 2.54.0