ci: reap orphan xctest/swift-test on self-hosted runner before each job #182

New Issue

2026-05-13T21:09:05Z

KofTwentyTwo commented

2026-05-13 21:09:05 +00:00

(Migrated from github.com)

What

Add a 'Reap orphan xctest/swift-test processes' step at the start of the swift-packages job, gated to runner.environment == 'self-hosted'.

Why

Grogu (the self-hosted runner) is a shared workstation, not an ephemeral VM. Interactive Xcode test sessions, aborted swift test runs, or debugger crashes can leave orphan xctest / swift-test / swift-driver processes around for hours. They hold Mach services, sockets, and ports that subsequent CI jobs may block on when they try to mount their own test infrastructure.

Concrete incident: the develop CI run on eef17252 hung in SPM AgentCore for 34 min (0% CPU, sleeping state) while 6 zombie xctest processes aged 22–24h were alive. They were from prior interactive Xcode use, not CI. Killing them did not unstick the wedged AgentCore (it was already blocked), but a fresh rerun with zombies absent should pass.

Approach

Single run: step at the top of swift-packages (after the matrix dispatches the job, before checkout/build/test). On self-hosted only — github-hosted VMs start clean.

/bin/ps -A -o pid,etime,comm \
  | awk '/(xctest|swift-test|swift-driver)$/ && !/awk/ {print $1}' \
  | xargs -r -I{} kill -TERM {} 2>/dev/null || true
sleep 2
# SIGKILL holdouts

Single-runner serialization invariant guarantees that any matching process at job start belongs to no live job and is safe to kill.

Acceptance

First step of swift-packages logs zero zombies when clean.
When zombies exist, they get killed and the job proceeds.
No regression on github-hosted (step is gated).

## What Add a 'Reap orphan xctest/swift-test processes' step at the start of the swift-packages job, gated to `runner.environment == 'self-hosted'`. ## Why Grogu (the self-hosted runner) is a shared workstation, not an ephemeral VM. Interactive Xcode test sessions, aborted `swift test` runs, or debugger crashes can leave orphan `xctest` / `swift-test` / `swift-driver` processes around for hours. They hold Mach services, sockets, and ports that subsequent CI jobs may block on when they try to mount their own test infrastructure. **Concrete incident:** the develop CI run on `eef17252` hung in `SPM AgentCore` for 34 min (0% CPU, sleeping state) while 6 zombie xctest processes aged 22–24h were alive. They were from prior interactive Xcode use, not CI. Killing them did not unstick the wedged AgentCore (it was already blocked), but a fresh rerun with zombies absent should pass. ## Approach Single `run:` step at the top of `swift-packages` (after the matrix dispatches the job, before checkout/build/test). On self-hosted only — github-hosted VMs start clean. ```bash /bin/ps -A -o pid,etime,comm \ | awk '/(xctest|swift-test|swift-driver)$/ && !/awk/ {print $1}' \ | xargs -r -I{} kill -TERM {} 2>/dev/null || true sleep 2 # SIGKILL holdouts ``` Single-runner serialization invariant guarantees that any matching process at job start belongs to no live job and is safe to kill. ## Acceptance - [ ] First step of swift-packages logs zero zombies when clean. - [ ] When zombies exist, they get killed and the job proceeds. - [ ] No regression on github-hosted (step is gated).

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/Jarvis#182