From 7ac877b2cf518d25864247e430dd2272ee202c3b Mon Sep 17 00:00:00 2001 From: James Maes Date: Wed, 13 May 2026 16:09:15 -0500 Subject: [PATCH] ci(actions): reap orphan xctest/swift-test on self-hosted before each job Grogu is a shared workstation, not an ephemeral VM. Interactive Xcode test runs, aborted swift test sessions, or debugger crashes can leave orphan xctest/swift-test/swift-driver processes around for hours, holding Mach services + sockets + ports that subsequent CI jobs block on when mounting their own test infrastructure. Concrete incident: develop run 25824607025 hung in SPM AgentCore for 34 min (0% CPU, sleeping state) while 6 unrelated xctest zombies aged 22-24h were alive from prior interactive use. The runner only executes one job at a time, so any such process at job start belongs to no live job and is safe to kill. Add a first step that runs SIGTERM then SIGKILL on any xctest/swift-test/swift-driver process. Gated to runner.environment == 'self-hosted' since github-hosted VMs start clean. Closes #182 --- .github/workflows/ci.yml | 22 ++++++++++++++++++++++ CLAUDE.md | 1 + 2 files changed, 23 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 13815a5..9c10fea 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -154,6 +154,28 @@ jobs: - Vision - Voice steps: + # Grogu is a shared workstation, not an ephemeral VM. Interactive + # Xcode test sessions, debugging, or aborted `swift test` runs can + # leave orphan xctest/swift-test/swift-driver processes around for + # hours, holding Mach services + sockets + ports that a new + # WKWebView mount or test-target launch then blocks on (observed: + # AgentCore hung 34min on develop CI 25824607025 with 6 zombie + # xctests aged 22-24h from prior interactive use). The runner only + # executes one job at a time, so any such process at job-start + # belongs to no live job and is safe to kill. github-hosted VMs + # start clean, so this step is gated to self-hosted. + - name: Reap orphan xctest/swift-test processes (self-hosted only) + if: runner.environment == 'self-hosted' + run: | + /bin/ps -A -o pid,etime,comm \ + | awk '/(xctest|swift-test|swift-driver)$/ && !/awk/ {print $1}' \ + | xargs -r -I{} kill -TERM {} 2>/dev/null || true + sleep 2 + /bin/ps -A -o pid,etime,comm \ + | awk '/(xctest|swift-test|swift-driver)$/ && !/awk/ {print $1}' \ + | xargs -r -I{} kill -9 {} 2>/dev/null || true + echo "Surviving xctest/swift-test processes after cleanup:" + /bin/ps -A -o pid,etime,comm | grep -E '(xctest|swift-test|swift-driver)$' || echo " (none)" - uses: actions/checkout@v4 # Project targets macOS 26 Tahoe + Swift 6.2/6.3. Xcode 16 (Swift 6.0) # cannot parse `mlx-audio-swift` v0.1.2's `swift-tools-version:6.2`, diff --git a/CLAUDE.md b/CLAUDE.md index 48a19cb..f408ac8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -250,6 +250,7 @@ Eval harness (Anthropic + local-model eval matrix; corpus in `.planning/evals/`) - **Cache:** the `actions/cache@v4` step is gated to `runner.environment == 'github-hosted'`. Self-hosted relies on the persistent `_work/` dir on HD-2 (and SwiftPM's natural `.build/` reuse) instead. Github-hosted caches bake absolute `/Users/runner/...` paths into `.build/artifacts/onnxruntime.xcframework` and other XCFrameworks, which Grogu cannot resolve — restricting the cache action keeps these caches strictly per-environment. - **Runner keychain** (`~/Library/Keychains/jarvis-runner.keychain-db`): created once on Grogu with password `grogu-jarvis-ci` (also stored as repo secret `JARVIS_RUNNER_KEYCHAIN_PASSWORD`). The LaunchAgent has no UI session, so `SecItemAdd` against a locked login keychain returns `errSecInteractionNotAllowed (-25308)`. CI unlocks `jarvis-runner.keychain-db`, prepends it to the user search list, AND sets it as the user's default keychain before `swift test`. The default-keychain step is necessary because on the legacy file-based macOS keychain, `SecItemAdd` writes to the user's *default* keychain (not the first writable in the search list). **Side effects on interactive use:** (1) `jarvis-runner.keychain-db` sits first in your user search list, and (2) it is also your default keychain — so `SecItemAdd` calls *without* an explicit keychain target write there instead of `login.keychain-db`. Apps that target login keychain explicitly (most do) are unaffected. To undo: `security default-keychain -d user -s ~/Library/Keychains/login.keychain-db && security list-keychains -d user -s ~/Library/Keychains/login.keychain-db`. - **When Grogu reboots or sleeps**, the LaunchAgent restarts on login but jobs queued during downtime will sit waiting. For long offline windows, push to a branch other than develop or temporarily revert `runs-on:` to `macos-15`. +- **Process hygiene:** the workflow's first step on self-hosted is a `Reap orphan xctest/swift-test processes` cleanup. Grogu is a shared workstation — interactive Xcode test runs, aborted `swift test`, or debugger sessions can leak `xctest` / `swift-test` / `swift-driver` processes that hold Mach services + sockets and cause subsequent CI jobs to deadlock (observed: AgentCore hung 34 min on a develop run when 6 such zombies aged 22–24h were present). The cleanup is harmless on a clean machine and decisive when not. # CLAUDE.md -- 2.54.0