diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 9c10fea..58fca49 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -160,21 +160,27 @@ jobs: # hours, holding Mach services + sockets + ports that a new # WKWebView mount or test-target launch then blocks on (observed: # AgentCore hung 34min on develop CI 25824607025 with 6 zombie - # xctests aged 22-24h from prior interactive use). The runner only - # executes one job at a time, so any such process at job-start - # belongs to no live job and is safe to kill. github-hosted VMs - # start clean, so this step is gated to self-hosted. - - name: Reap orphan xctest/swift-test processes (self-hosted only) + # xctests aged 22-24h from prior interactive use). + # + # Age filter (>= 1 hour): Grogu now has SIBLING runners + # (Grogu-jarvis, -2, -3) so concurrent CI jobs can be in flight + # simultaneously. CI jobs complete in <10 min, so a 1-hour cutoff + # safely reaps interactive-use orphans without killing concurrent + # CI work on sibling runners. `ps -o etime` formats: `MM:SS` + # (<1h, skip), `HH:MM:SS` (>=1h, kill), `DD-HH:MM:SS` (days, kill). + # github-hosted VMs start clean, so this step is gated to self-hosted. + - name: Reap orphan xctest/swift-test processes older than 1h (self-hosted only) if: runner.environment == 'self-hosted' run: | + age_filter='/(xctest|swift-test|swift-driver)$/ && !/awk/ && (match($2, /-/) || match($2, /[0-9]+:[0-9]+:[0-9]+/)) {print $1}' /bin/ps -A -o pid,etime,comm \ - | awk '/(xctest|swift-test|swift-driver)$/ && !/awk/ {print $1}' \ + | awk "$age_filter" \ | xargs -r -I{} kill -TERM {} 2>/dev/null || true sleep 2 /bin/ps -A -o pid,etime,comm \ - | awk '/(xctest|swift-test|swift-driver)$/ && !/awk/ {print $1}' \ + | awk "$age_filter" \ | xargs -r -I{} kill -9 {} 2>/dev/null || true - echo "Surviving xctest/swift-test processes after cleanup:" + echo "Surviving xctest/swift-test processes (any age — includes concurrent CI work on sibling runners):" /bin/ps -A -o pid,etime,comm | grep -E '(xctest|swift-test|swift-driver)$' || echo " (none)" - uses: actions/checkout@v4 # Project targets macOS 26 Tahoe + Swift 6.2/6.3. Xcode 16 (Swift 6.0) diff --git a/CLAUDE.md b/CLAUDE.md index cd6e634..489b8fe 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -250,7 +250,7 @@ Eval harness (Anthropic + local-model eval matrix; corpus in `.planning/evals/`) - **Cache:** the `actions/cache@v4` step is gated to `runner.environment == 'github-hosted'`. Self-hosted relies on the persistent `_work/` dir on HD-2 (and SwiftPM's natural `.build/` reuse) instead. Github-hosted caches bake absolute `/Users/runner/...` paths into `.build/artifacts/onnxruntime.xcframework` and other XCFrameworks, which Grogu cannot resolve — restricting the cache action keeps these caches strictly per-environment. - **Runner keychain** (`~/Library/Keychains/jarvis-runner.keychain-db`): created once on Grogu with password `grogu-jarvis-ci` (also stored as repo secret `JARVIS_RUNNER_KEYCHAIN_PASSWORD`). The LaunchAgent has no UI session, so `SecItemAdd` against a locked login keychain returns `errSecInteractionNotAllowed (-25308)`. CI unlocks `jarvis-runner.keychain-db`, prepends it to the user search list, AND sets it as the user's default keychain before `swift test`. The default-keychain step is necessary because on the legacy file-based macOS keychain, `SecItemAdd` writes to the user's *default* keychain (not the first writable in the search list). **Side effects on interactive use:** (1) `jarvis-runner.keychain-db` sits first in your user search list, and (2) it is also your default keychain — so `SecItemAdd` calls *without* an explicit keychain target write there instead of `login.keychain-db`. Apps that target login keychain explicitly (most do) are unaffected. To undo: `security default-keychain -d user -s ~/Library/Keychains/login.keychain-db && security list-keychains -d user -s ~/Library/Keychains/login.keychain-db`. - **When Grogu reboots or sleeps**, the LaunchAgent restarts on login but jobs queued during downtime will sit waiting. For long offline windows, push to a branch other than develop or temporarily revert `runs-on:` to `macos-15`. -- **Process hygiene:** the workflow's first step on self-hosted is a `Reap orphan xctest/swift-test processes` cleanup. Grogu is a shared workstation — interactive Xcode test runs, aborted `swift test`, or debugger sessions can leak `xctest` / `swift-test` / `swift-driver` processes that hold Mach services + sockets and cause subsequent CI jobs to deadlock (observed: AgentCore hung 34 min on a develop run when 6 such zombies aged 22–24h were present). The cleanup is harmless on a clean machine and decisive when not. +- **Process hygiene:** the workflow's first step on self-hosted is a `Reap orphan xctest/swift-test processes older than 1h` cleanup. Grogu is a shared workstation — interactive Xcode test runs, aborted `swift test`, or debugger sessions can leak `xctest` / `swift-test` / `swift-driver` processes that hold Mach services + sockets and cause subsequent CI jobs to deadlock (observed: AgentCore hung 34 min on a develop run when 6 such zombies aged 22–24h were present). The age filter (≥1h) keeps concurrent CI work on sibling runners safe (jobs complete in <10 min). # CLAUDE.md