fix(voice): v0.1 CRIT cluster — wake / STT / TTS / boot probes #158

2026-05-12T23:08:54Z

KofTwentyTwo commented

2026-05-12 23:08:54 +00:00

(Migrated from github.com)

v0.1 voice CRITICAL/HIGH audit follow-ups from .planning/audit-2026-05-12/voice-audit.md. Eight tactical patches that take voice from "silently dead today" to "wake, talk, hear" working. Some of these will be re-fixed during M-6 Voice API migration; the patches land in v0.1 because the user needs voice working now.

Issues closed

#	Severity	Summary
#28	CRIT	wake-word dies on every audio-graph rebuild — `cancelInFlight` now calls `stopFeed()` (not `cancel()`); rebuild consumer re-arms the DAG against the new ring
#29	CRIT	STT factory reads `PerTurnSnapshot.stt.whisperKitFallback`; `get_self_state` returns the canonical snake_case backend strings — round-trip through `STTBackendSelector.make` no longer falls into the unknown-backend default
#30	CRIT	`TTSEngineActor.synthesize` emits `.ttsStopped` after `.finished` on natural completion; barge-in still emits at `InterruptSequence` step 5 — the "two-emit-site" model unblocks the future ducking consumer
#31	CRIT	`VoiceTTSAdapter.tierResolver` is wired from `configStore` so `tts.tier = "tier2"` will flip the tier the moment Orpheus weights ship
#32	HIGH	rolling-window trim on `OpenWakeWordSession.melFrameBuffer` (keep last 84) + `embeddingBuffer` (keep last 16); `nextEmbStart` slides left when frames are trimmed
#33	HIGH	three new BootHealth watchdogs — `WakeWordBootHealthProbe`, `STTBootHealthProbe`, `TTSBootHealthProbe`; `VoiceBootHealthProbe` no longer reports `ok` when the DAG is dead / SpeechAnalyzer TCC is denied / TTS engine is dormant
#34	HIGH	`VoiceController.startSTTSession` calls `vad.reset()` before each session — clears LSTM hidden state + `wasSpeech` so session N+1's first soft-spoken syllable isn't dropped
#36	HIGH	new `OrchestratorEventBroadcaster.Priority.agentHud` (protects `.stateChange` / `.turnEnd` / `.error`); `installAgent` adds an `agentHudSubscriberTask` that pumps `AgentHudIntent` values into `dormantAgentContinuation` — HUD ring now reflects thinking/speaking

Tests added

#28: WD-3 — DAG survives stopFeed() + start(ring: newRing); consumer Task stays alive AND new ring's marker drives the session.
#29: B5 — canonical STTBackendSelector.backend* constants round-trip through make without emitting an unknown-backend warning; camelCase strings still produce the warning (locks against silent drift).
#30: TET1-5 — real AVSpeechSynthesizer tier-1 synthesize; collector drains until .ttsStopped; asserts .finished precedes .ttsStopped.
#31: tierResolver invocation count — adapter invokes its resolver once per synthesize call.
#32: B1/B2 — 10 000 simulated frames keep both buffers bounded at the documented limits; short streams don't trigger premature trim.
#33: W1/W2/W3 + T1/T2 + S1 — each watchdog's .ok / .degraded / .failed / .unknown shape, including the canonical post-P0-1 "DAG dead" and pre-Track-B-3 "TTS dormant" regressions.
#34: VAD-6 — single SileroVAD reused across two VoiceController sessions (production-shape factory pattern); session 2 fed only silence does NOT auto-submit (pre-fix the carried-over wasSpeech=true would have armed a stale finalize).
#36: extends testIsProtectedForPriorityMatrix to cover the new .agentHud priority's protection rules.

Pre-merge checks

swift test --package-path packages/Voice — 90 tests, all green (3 skipped — real-hardware gates)
swift test --package-path packages/AgentCore — 242 tests, all green
bash scripts/check-app-builds.sh — green
All 18 boundary gates green (for s in scripts/check-*.sh; do bash "$s"; done)
User UAT: plug headphones in mid-session → wake-word still triggers (#28 live test)
User UAT: flip stt.whisperKitFallback = true → next session uses WhisperKit (#29 live test)

Architecture notes

VoiceController.sttFactory signature changed from () -> any STTProvider to () async -> any STTProvider (#29). All existing test sites using { stt } syntax still compile because async closures with no awaits are implicit.
New OrchestratorEventBroadcaster.Priority.agentHud (#36) — fully exhaustive switch arm in isProtectedForPriority + the matrix test. No other broadcaster behavior changes.
Two new stored fields on AppDelegate for #33: wakeWordDAG and voiceTTSAdapter. Both nil-safe — probes return .unknown when the install path short-circuited.

What's NOT in this PR (out of scope)

Real Orpheus tier-2 — gated behind the ~6 GB weight download (#31 wires the resolver; tier-2 still degrades to tier-1 inside TTSEngineActor when no Orpheus is wired).
Richer agent state machine that distinguishes .thinking vs .speaking based on TTS playback — M-6 work; today both come through as .thinking via OrchestratorEvent.stateChange(.thinking) until terminal.
WhisperKit asset download flow — #29 makes the fallback reachable from config but doesn't ship the weights.

🤖 Generated with Claude Code

v0.1 voice CRITICAL/HIGH audit follow-ups from `.planning/audit-2026-05-12/voice-audit.md`. Eight tactical patches that take voice from "silently dead today" to "wake, talk, hear" working. Some of these will be re-fixed during M-6 Voice API migration; the patches land in v0.1 because the user needs voice working now. ## Issues closed | # | Severity | Summary | |---|----------|---------| | #28 | CRIT | wake-word dies on every audio-graph rebuild — `cancelInFlight` now calls `stopFeed()` (not `cancel()`); rebuild consumer re-arms the DAG against the new ring | | #29 | CRIT | STT factory reads `PerTurnSnapshot.stt.whisperKitFallback`; `get_self_state` returns the canonical snake_case backend strings — round-trip through `STTBackendSelector.make` no longer falls into the unknown-backend default | | #30 | CRIT | `TTSEngineActor.synthesize` emits `.ttsStopped` after `.finished` on natural completion; barge-in still emits at `InterruptSequence` step 5 — the "two-emit-site" model unblocks the future ducking consumer | | #31 | CRIT | `VoiceTTSAdapter.tierResolver` is wired from `configStore` so `tts.tier = "tier2"` will flip the tier the moment Orpheus weights ship | | #32 | HIGH | rolling-window trim on `OpenWakeWordSession.melFrameBuffer` (keep last 84) + `embeddingBuffer` (keep last 16); `nextEmbStart` slides left when frames are trimmed | | #33 | HIGH | three new BootHealth watchdogs — `WakeWordBootHealthProbe`, `STTBootHealthProbe`, `TTSBootHealthProbe`; `VoiceBootHealthProbe` no longer reports `ok` when the DAG is dead / SpeechAnalyzer TCC is denied / TTS engine is dormant | | #34 | HIGH | `VoiceController.startSTTSession` calls `vad.reset()` before each session — clears LSTM hidden state + `wasSpeech` so session N+1's first soft-spoken syllable isn't dropped | | #36 | HIGH | new `OrchestratorEventBroadcaster.Priority.agentHud` (protects `.stateChange` / `.turnEnd` / `.error`); `installAgent` adds an `agentHudSubscriberTask` that pumps `AgentHudIntent` values into `dormantAgentContinuation` — HUD ring now reflects thinking/speaking | ## Tests added - **#28**: WD-3 — DAG survives `stopFeed()` + `start(ring: newRing)`; consumer Task stays alive AND new ring's marker drives the session. - **#29**: B5 — canonical `STTBackendSelector.backend*` constants round-trip through `make` without emitting an unknown-backend warning; camelCase strings still produce the warning (locks against silent drift). - **#30**: TET1-5 — real AVSpeechSynthesizer tier-1 synthesize; collector drains until `.ttsStopped`; asserts `.finished` precedes `.ttsStopped`. - **#31**: tierResolver invocation count — adapter invokes its resolver once per `synthesize` call. - **#32**: B1/B2 — 10 000 simulated frames keep both buffers bounded at the documented limits; short streams don't trigger premature trim. - **#33**: W1/W2/W3 + T1/T2 + S1 — each watchdog's `.ok` / `.degraded` / `.failed` / `.unknown` shape, including the canonical post-P0-1 "DAG dead" and pre-Track-B-3 "TTS dormant" regressions. - **#34**: VAD-6 — single `SileroVAD` reused across two `VoiceController` sessions (production-shape factory pattern); session 2 fed only silence does NOT auto-submit (pre-fix the carried-over `wasSpeech=true` would have armed a stale finalize). - **#36**: extends `testIsProtectedForPriorityMatrix` to cover the new `.agentHud` priority's protection rules. ## Pre-merge checks - [x] `swift test --package-path packages/Voice` — 90 tests, all green (3 skipped — real-hardware gates) - [x] `swift test --package-path packages/AgentCore` — 242 tests, all green - [x] `bash scripts/check-app-builds.sh` — green - [x] All 18 boundary gates green (`for s in scripts/check-*.sh; do bash "$s"; done`) - [ ] User UAT: plug headphones in mid-session → wake-word still triggers (#28 live test) - [ ] User UAT: flip `stt.whisperKitFallback = true` → next session uses WhisperKit (#29 live test) ## Architecture notes - `VoiceController.sttFactory` signature changed from `() -> any STTProvider` to `() async -> any STTProvider` (#29). All existing test sites using `{ stt }` syntax still compile because async closures with no awaits are implicit. - New `OrchestratorEventBroadcaster.Priority.agentHud` (#36) — fully exhaustive switch arm in `isProtectedForPriority` + the matrix test. No other broadcaster behavior changes. - Two new stored fields on `AppDelegate` for #33: `wakeWordDAG` and `voiceTTSAdapter`. Both nil-safe — probes return `.unknown` when the install path short-circuited. ## What's NOT in this PR (out of scope) - Real Orpheus tier-2 — gated behind the ~6 GB weight download (#31 wires the resolver; tier-2 still degrades to tier-1 inside `TTSEngineActor` when no Orpheus is wired). - Richer agent state machine that distinguishes `.thinking` vs `.speaking` based on TTS playback — M-6 work; today both come through as `.thinking` via `OrchestratorEvent.stateChange(.thinking)` until terminal. - WhisperKit asset download flow — #29 makes the fallback reachable from config but doesn't ship the weights. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/Jarvis#158