ci: age-filter zombie reaper (>=1h) for multi-runner safety #189

Merged
KofTwentyTwo merged 1 commits from ci/age-filter-multi-runner into develop 2026-05-13 22:33:03 +00:00
2 changed files with 15 additions and 9 deletions

No files matched your search

+14 -8
View File
@@ -160,21 +160,27 @@ jobs:
# hours, holding Mach services + sockets + ports that a new
# WKWebView mount or test-target launch then blocks on (observed:
# AgentCore hung 34min on develop CI 25824607025 with 6 zombie
# xctests aged 22-24h from prior interactive use). The runner only
# executes one job at a time, so any such process at job-start
# belongs to no live job and is safe to kill. github-hosted VMs
# start clean, so this step is gated to self-hosted.
- name: Reap orphan xctest/swift-test processes (self-hosted only)
# xctests aged 22-24h from prior interactive use).
#
# Age filter (>= 1 hour): Grogu now has SIBLING runners
# (Grogu-jarvis, -2, -3) so concurrent CI jobs can be in flight
# simultaneously. CI jobs complete in <10 min, so a 1-hour cutoff
# safely reaps interactive-use orphans without killing concurrent
# CI work on sibling runners. `ps -o etime` formats: `MM:SS`
# (<1h, skip), `HH:MM:SS` (>=1h, kill), `DD-HH:MM:SS` (days, kill).
# github-hosted VMs start clean, so this step is gated to self-hosted.
- name: Reap orphan xctest/swift-test processes older than 1h (self-hosted only)
if: runner.environment == 'self-hosted'
run: |
age_filter='/(xctest|swift-test|swift-driver)$/ && !/awk/ && (match($2, /-/) || match($2, /[0-9]+:[0-9]+:[0-9]+/)) {print $1}'
/bin/ps -A -o pid,etime,comm \
| awk '/(xctest|swift-test|swift-driver)$/ && !/awk/ {print $1}' \
| awk "$age_filter" \
| xargs -r -I{} kill -TERM {} 2>/dev/null || true
sleep 2
/bin/ps -A -o pid,etime,comm \
| awk '/(xctest|swift-test|swift-driver)$/ && !/awk/ {print $1}' \
| awk "$age_filter" \
| xargs -r -I{} kill -9 {} 2>/dev/null || true
echo "Surviving xctest/swift-test processes after cleanup:"
echo "Surviving xctest/swift-test processes (any age — includes concurrent CI work on sibling runners):"
/bin/ps -A -o pid,etime,comm | grep -E '(xctest|swift-test|swift-driver)$' || echo " (none)"
- uses: actions/checkout@v4
# Project targets macOS 26 Tahoe + Swift 6.2/6.3. Xcode 16 (Swift 6.0)
+1 -1
View File
@@ -250,7 +250,7 @@ Eval harness (Anthropic + local-model eval matrix; corpus in `.planning/evals/`)
- **Cache:** the `actions/cache@v4` step is gated to `runner.environment == 'github-hosted'`. Self-hosted relies on the persistent `_work/` dir on HD-2 (and SwiftPM's natural `.build/` reuse) instead. Github-hosted caches bake absolute `/Users/runner/...` paths into `.build/artifacts/onnxruntime.xcframework` and other XCFrameworks, which Grogu cannot resolve — restricting the cache action keeps these caches strictly per-environment.
- **Runner keychain** (`~/Library/Keychains/jarvis-runner.keychain-db`): created once on Grogu with password `grogu-jarvis-ci` (also stored as repo secret `JARVIS_RUNNER_KEYCHAIN_PASSWORD`). The LaunchAgent has no UI session, so `SecItemAdd` against a locked login keychain returns `errSecInteractionNotAllowed (-25308)`. CI unlocks `jarvis-runner.keychain-db`, prepends it to the user search list, AND sets it as the user's default keychain before `swift test`. The default-keychain step is necessary because on the legacy file-based macOS keychain, `SecItemAdd` writes to the user's *default* keychain (not the first writable in the search list). **Side effects on interactive use:** (1) `jarvis-runner.keychain-db` sits first in your user search list, and (2) it is also your default keychain — so `SecItemAdd` calls *without* an explicit keychain target write there instead of `login.keychain-db`. Apps that target login keychain explicitly (most do) are unaffected. To undo: `security default-keychain -d user -s ~/Library/Keychains/login.keychain-db && security list-keychains -d user -s ~/Library/Keychains/login.keychain-db`.
- **When Grogu reboots or sleeps**, the LaunchAgent restarts on login but jobs queued during downtime will sit waiting. For long offline windows, push to a branch other than develop or temporarily revert `runs-on:` to `macos-15`.
- **Process hygiene:** the workflow's first step on self-hosted is a `Reap orphan xctest/swift-test processes` cleanup. Grogu is a shared workstation — interactive Xcode test runs, aborted `swift test`, or debugger sessions can leak `xctest` / `swift-test` / `swift-driver` processes that hold Mach services + sockets and cause subsequent CI jobs to deadlock (observed: AgentCore hung 34 min on a develop run when 6 such zombies aged 22–24h were present). The cleanup is harmless on a clean machine and decisive when not.
- **Process hygiene:** the workflow's first step on self-hosted is a `Reap orphan xctest/swift-test processes older than 1h` cleanup. Grogu is a shared workstation — interactive Xcode test runs, aborted `swift test`, or debugger sessions can leak `xctest` / `swift-test` / `swift-driver` processes that hold Mach services + sockets and cause subsequent CI jobs to deadlock (observed: AgentCore hung 34 min on a develop run when 6 such zombies aged 22–24h were present). The age filter (≥1h) keeps concurrent CI work on sibling runners safe (jobs complete in <10 min).
# CLAUDE.md