0 Fork 0

Files

T

History

james.maesandClaude Opus 4.7 a121d79174 fix: STT backend constant casing + audioLevel NaN guard (#37 , #47 ) (#196 )

* docs(voice): pin STT backend constant identity contract (#37)

The runtime mismatch flagged by #37 was already fixed at the call sites
in `App/AppDelegate.swift:1631-1642` and `:2562-2575` — both routes now
reference `STTBackendSelector.backendSpeechAnalyzer` /
`backendWhisperKit` (snake_case) instead of the pre-2026-05-12 camelCase
string literals. The regression test
`STTBackendSwitchTests.testB5_canonicalConstantsRoundTripThroughMake`
already locks both halves of the contract:

  1. The snake_case constants round-trip through `make(backend:)` without
     hitting the unknown-backend default branch.
  2. The pre-fix camelCase shape (`"whisperKit"`) DOES land in the
     default branch and DOES emit an unknown-backend warning.

So the #37 close action is just to make that contract self-documenting
at the declaration site rather than relying on readers finding the test.
Add a doc-comment to the two `public static let backend*` constants that
explains the wire shape, names the regression test, references Issues
#29 + #37, and warns against renaming without updating `PerTurnSnapshot`
and the test in lockstep.

No code change; `swift test --package-path packages/Voice` continues to
pass (26 tests).

Closes #37

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* fix(bus): clamp non-finite audioLevel rms to 0.0 at encode (#47)

`BusOutbound.audioLevel` carries a `Float rms` that `AudioLevelEmitter`
produces per audio frame (~100 Hz once the consumer is wired up).
Foundation's `JSONEncoder.NonConformingFloatEncodingStrategy` defaults
to `.throw`, so a single NaN — easy to land via a CoreAudio
over-/underflow or an all-zero buffer fed through certain RMS shapes —
would throw inside `WebviewBridge.sendRaw` and tear down the per-frame
bus path. The HUD-side consumer in `webview/packages/hud/src/bus/
client.ts:135` is currently a no-op (the ring shader is still on the
synthetic-sine animation from Plan 03-03), so the bug is LATENT, not
actively firing — but the clamp is cheap, defensive, and pins the
contract before the consumer activates.

Inline clamp at the encode site (`BusOutbound.encode(to:)` switch arm
for `.audioLevel`): replace non-finite floats with `0.0` via
`rms.isFinite ? rms : 0.0` immediately before `container.encode(_:)`.
No refactor of `AudioLevelEmitter` — the per-spec call here is
defensive at the serialization boundary, where every encoder-level
overflow path converges.

Two new unit tests in `CodableRoundTripTests`:
- `test_audioLevel_nanIsClampedToZero` — encodes `.audioLevel(rms: .nan)`,
  decodes, asserts `rms == 0.0` and `rms.isFinite`.
- `test_audioLevel_infinityIsClampedToZero` — companion for ±∞ so a
  future refactor cannot narrow the guard to NaN-only.

`swift test --package-path packages/Bus --filter CodableRoundTripTests`
passes (30 tests, +2 from this commit).

Closes #47

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>

2026-05-14 09:50:50 -05:00

Sources/Voice

fix: STT backend constant casing + audioLevel NaN guard (#37 , #47 ) (#196 )

2026-05-14 09:50:50 -05:00

Tests/VoiceTests

fix(voice): widen CI timing budgets for I3 + V4 (round 2) (#171 )

2026-05-13 09:36:15 -05:00

Package.swift

fix(voice p1-4): RingBuffer index atomics via swift-atomics for explicit acquire/release semantics

2026-05-04 16:01:14 -05:00

README.md

docs(packages): per-package README files

2026-05-04 17:13:33 -05:00

README.md

Voice

The voice loop: audio graph + wake word + VAD + STT + TTS + the state machine actor that stitches them. Local-only by user constraint — no cloud STT/TTS.

Key public types

Type	Purpose
`VoiceController` (actor)	Top-level state machine — `idle / listening / thinking / speaking`
`VoiceState`	State enum
`AudioGraphOwner` (actor)	`AVAudioEngine` lifecycle + canonical six-step teardown (VOICE-10)
`BufferBroadcaster`	Multi-consumer fan-out from the Core Audio tap (Track B-7)
`RingBuffer`	SPSC ring (one consumer per ring; never read directly outside Voice)
`WakeWordDAG`	mel ring → embedding ring → classifier; openWakeWord `hey_jarvis`; 4-frame hysteresis
`OpenWakeWordSession`	ONNX Runtime session wrapper
`SileroVAD`	VAD on 512-sample / 32 ms / 16 kHz windows
`SpeechAnalyzerSTT`	macOS 26 Tahoe primary (gated by `@available(macOS 26, *)`)
`WhisperKitSTT`	Argmax `large-v3-v20240930_626MB` fallback (feature-flagged)
`STTProvider` (protocol)	Common contract
`TTSEngineActor`	Tier-1 / tier-2 dispatcher with serial executor (prevents Metal cmd-buffer deadlock)
`AVSpeechSynth`	Tier-1 `AVSpeechSynthesizer` (instant, mediocre)
`OrpheusTTS`	Tier-2 (deferred — weights download outstanding)
`TTSKitFallback`	Tier-2 fallback via Argmax `TTSKit`
`AudioLevelEmitter`	RMS → `BusOutbound.audioLevel` for HUD pulse (production wiring open)
`PushToTalk` / `MuteWakeWord`	Per-call control surface

Depends on

AgentCore, Bus, Logging, Config. External: onnxruntime-swift-package-manager, argmax-oss-swift (WhisperKit + TTSKit), mlx-audio-swift (Orpheus), swift-atomics, swift-log.

Used by

App/AppDelegate, App/Voice/ (production adapters — VoiceOrchestratorAdapter, VoiceTTSAdapter, VoiceBusEmitterAdapter).

Key invariants / contracts

VOICE-14 single cancelAndSubmit call site. Only VoiceController.bargeIn(). Grep-gated.
T-06-05-03: never log voice transcript text. Every Voice logger.debug call site reviewed.
T-06-05-02: AEC banner via native AppKit, never webview modal. scripts/check-no-modal-presentation.sh.
T-06-05-04: 200 ms barge-in debounce. BargeInTests B4.
VOICE-12: PTT works when wake-word is muted. pttDown() / pttUp() bypass the mute gate.
VOICE-09: AEC-off as distinct variant. If setVoiceProcessingEnabled(true) throws, emit .aecUnavailable THEN rebuild with aec: false.
VOICE-10: six-step teardown. All four rebuild triggers call the single teardown() private method on AudioGraphOwner.
Multi-consumer audio: BufferBroadcaster.subscribe(), never raw RingBuffer outside the Voice package. RingBuffer is documented SPSC.
BufferBroadcaster.publish is real-time-thread-safe. Lock-free / wait-free OSAllocatedUnfairLock snapshot — no actor hops, no Foundation locks.
VAD-gated session end. STT session ends on .speechEnd + 5-chunk hangover (~160 ms). Mid-utterance speech resumption cancels the pending finalize.
TTSEngineActor serial executor. Prevents Metal command-buffer serialization deadlock on rapid-fire syntheses.
Test seam visibility. _forceState, _testFireSpeechEnd, _testCurrentSttSessionId are internal (P2-14); tests use @testable import.

Tests

87 XCTest + 23 swift-testing (3 skipped). Includes BufferBroadcasterTests (BB-1..5 incl. concurrent stress), VoiceLoopE2ETests (E1..E3 — preset chunks → orchestrator.submit), VADGatedSessionTests (VAD-1..5), BargeInTests (B1..B4). Pre-existing 10s flake on TTSInterruptTests.testI3 (fixed in audit-2026-05-04 batch — propagates consumer cancellation in ScriptedSpeechModel).

Notable files

Sources/Voice/VoiceController.swift — top-level state machine
Sources/Voice/AudioGraph/AudioGraphOwner.swift — engine lifecycle
Sources/Voice/AudioGraph/BufferBroadcaster.swift — multi-consumer fan-out
Sources/Voice/STT/SpeechAnalyzerSTT.swift — macOS 26 STT
Sources/Voice/TTS/TTSEngineActor.swift — tier-1 / tier-2 dispatcher