M-6: Voice surface (audio-graph fix + synthesizeTurn; B-04/B-05 closure) #10

New Issue

2026-05-12T19:02:42Z

KofTwentyTwo commented

2026-05-12 19:02:42 +00:00

(Migrated from github.com)

Goal

Land Voice surface. Substrate audio-graph fix + synthesizeTurn command (UQ-3 (A) explicit-command contract). Closes B-04 (voice input dead) + B-05 (TTS silent). Highest-risk migration per REVIEW-MIGRATION-RISK. Long-lived feature branch (4–5 days); rebase on develop daily.

Scope — surfaces in this phase

M-6a — surface mapping (no audio-graph change):

JarvisAPI.Voice with getVoiceState query (read-only)
Events: voiceStateChanged, audioLevelChanged, wakeWordDetected, sttTranscriptPartial/Final, ttsStarted, ttsEnded, voiceDegraded
Runtime-control verbs: pttDown, pttUp, cancelTTS (Settings owns config; Turn owns bargeIn)
Harness commands (UQ-1 gated): _injectAudioFrame, _injectWakeWord, _injectSTT, _observeTTSAudio

M-6b — substrate fix (B-04, B-05):

startVoice() returns audioGraphFailed if ringBuffer nil after open() (B-04) — no silent log-and-degrade
Voice.synthesizeTurn(turnId:text:tier:) -> Result<Void, TTSError> (B-05)
Orchestrator wires synthesizeTurn on voice-source turnEnded(.completed) — UQ-3 (A) explicit-command contract
bargeIn-driven cancelTTS internal invariant (R-018 substrate; surface in M-7)

Files (created/modified)

packages/JarvisAPI/Sources/JarvisAPI/Surfaces/Voice+Adapter.swift (new)
packages/Voice/Sources/Voice/VoiceController.swift — startVoice error surfacing
packages/Voice/Sources/Voice/TTS/ — synthesizeTurn wiring
packages/AgentCore/Sources/AgentOrchestrator/AgentOrchestrator.swift — issue synthesizeTurn on voice-source .completed
packages/Harness/Tests/HarnessTests/VoiceScenarioTests.swift (new)

Test contract

Scenarios S-VOICE-001..S-VOICE-012. Both transport modes mandatory.

B-04 gate (R-007): startVoice() success ⇒ ≥10 audioLevelChanged within 500 ms
B-05 gate (UQ-3): voice-source turn ending in .completed ⇒ synthesizeTurn invoked exactly once AND ttsStarted event observed
S-VOICE-bargeIn-cancels-TTS — bargeIn during ttsStarted ⇒ ttsEnded(reason: .cancelled) BEFORE turnStarted of new turn
TTS cancel on cancelTTS command
audioLevelChanged cadence
Wake word detection + STT transcript partial/final
Real-hardware gates: JARVIS_REAL_MODELS=1 swift test --filter OrpheusTTFATests (deferred if Orpheus weights absent — see plan §6)

Acceptance criteria

bash scripts/check-no-null-voice-adapters.sh green
bash scripts/check-presence-bus-no-tts-orchestrator.sh green
All Voice scenarios green via .inProcessActor runner
All Voice scenarios green via .jsonRoundTripWebView runner
B-04 gate (R-007): startVoice() success ⇒ ≥10 audioLevelChanged within 500 ms
B-05 gate (UQ-3): synthesizeTurn invoked exactly once per voice-source .completed turn; ttsStarted observed
bash scripts/verify-entitlements.sh --pre-codesign green (R-012)
bash scripts/verify-codesign-settings.sh green (R-013)
JARVIS_REAL_MODELS=1 ... OrpheusTTFATests green if Orpheus weights present (DEFERRED otherwise)
PR merged to develop

Carry-forward closures

Closes #49 (B-04: voice input dead)
Closes #50 (B-05: TTS silent)

Dependencies

Blocked by: #9 (M-5)
Blocks: #11 (M-7 Turn)

Related issues

May dissolve: voice-audit findings P0-1..P0-4

Sub-issues

#126 — M-6a: define JarvisAPI.Voice + getVoiceState query (no audio-graph change)
#127 — M-6a: Voice events bridged through API surface (no audio-graph change)
#128 — M-6a: Voice runtime verbs — pttDown/pttUp/cancelTTS (UQ-4 slim Voice)
#129 — M-6a: Voice harness commands — _injectAudioFrame/_injectWakeWord/_injectSTT/_observeTTSAudio (UQ-1, G-001)
#130 — M-6b: startVoice() surfaces audioGraphFailed if ringBuffer nil (B-04, R-007)
#131 — M-6b: Voice.synthesizeTurn explicit-command contract (B-05, UQ-3 (A))
#132 — M-6b: orchestrator issues synthesizeTurn on voice-source turnEnded(.completed) (B-05)
#133 — M-6b: bargeIn-driven cancelTTS internal invariant (R-018 substrate)
#134 — M-6: harness scenarios S-VOICE-001..012 in both transport modes (B-04 + B-05 BLOCKING gates)

Estimate

4.0–5.5 engineer-days

Source

.planning/architecture/JARVIS-API-MIGRATION-PLAN.md §2.7
.planning/architecture/JARVIS-API-DESIGN-v0.1.md — Voice surface
.planning/architecture/JARVIS-API-DESIGN-v0.2.md §5 — UQ-3 (A) explicit-command, UQ-4 slim Voice
.planning/architecture/REVIEW-MIGRATION-RISK.md R-007, R-012, R-013, R-018
Parent epic: v1.0 API Migration — UI/Backend separation (M-0..M-7) (#76)

## Goal Land Voice surface. Substrate audio-graph fix + `synthesizeTurn` command (UQ-3 (A) explicit-command contract). Closes B-04 (voice input dead) + B-05 (TTS silent). **Highest-risk migration** per `REVIEW-MIGRATION-RISK`. Long-lived feature branch (4–5 days); rebase on develop daily. ## Scope — surfaces in this phase **M-6a — surface mapping (no audio-graph change):** - **`JarvisAPI.Voice`** with `getVoiceState` query (read-only) - Events: `voiceStateChanged`, `audioLevelChanged`, `wakeWordDetected`, `sttTranscriptPartial/Final`, `ttsStarted`, `ttsEnded`, `voiceDegraded` - Runtime-control verbs: `pttDown`, `pttUp`, `cancelTTS` (Settings owns config; Turn owns `bargeIn`) - Harness commands (UQ-1 gated): `_injectAudioFrame`, `_injectWakeWord`, `_injectSTT`, `_observeTTSAudio` **M-6b — substrate fix (B-04, B-05):** - `startVoice()` returns `audioGraphFailed` if ringBuffer nil after `open()` (B-04) — **no silent log-and-degrade** - `Voice.synthesizeTurn(turnId:text:tier:) -> Result<Void, TTSError>` (B-05) - Orchestrator wires `synthesizeTurn` on voice-source `turnEnded(.completed)` — UQ-3 (A) explicit-command contract - bargeIn-driven cancelTTS internal invariant (R-018 substrate; surface in M-7) ## Files (created/modified) - `packages/JarvisAPI/Sources/JarvisAPI/Surfaces/Voice+Adapter.swift` (new) - `packages/Voice/Sources/Voice/VoiceController.swift` — startVoice error surfacing - `packages/Voice/Sources/Voice/TTS/` — `synthesizeTurn` wiring - `packages/AgentCore/Sources/AgentOrchestrator/AgentOrchestrator.swift` — issue `synthesizeTurn` on voice-source `.completed` - `packages/Harness/Tests/HarnessTests/VoiceScenarioTests.swift` (new) ## Test contract Scenarios `S-VOICE-001..S-VOICE-012`. Both transport modes mandatory. - **B-04 gate (R-007):** `startVoice()` success ⇒ ≥10 `audioLevelChanged` within 500 ms - **B-05 gate (UQ-3):** voice-source turn ending in `.completed` ⇒ `synthesizeTurn` invoked exactly once AND `ttsStarted` event observed - `S-VOICE-bargeIn-cancels-TTS` — bargeIn during ttsStarted ⇒ `ttsEnded(reason: .cancelled)` BEFORE `turnStarted` of new turn - TTS cancel on cancelTTS command - audioLevelChanged cadence - Wake word detection + STT transcript partial/final - Real-hardware gates: `JARVIS_REAL_MODELS=1 swift test --filter OrpheusTTFATests` (deferred if Orpheus weights absent — see plan §6) ## Acceptance criteria - [ ] `bash scripts/check-no-null-voice-adapters.sh` green - [ ] `bash scripts/check-presence-bus-no-tts-orchestrator.sh` green - [ ] All Voice scenarios green via `.inProcessActor` runner - [ ] All Voice scenarios green via `.jsonRoundTripWebView` runner - [ ] **B-04 gate (R-007):** `startVoice()` success ⇒ ≥10 `audioLevelChanged` within 500 ms - [ ] **B-05 gate (UQ-3):** `synthesizeTurn` invoked exactly once per voice-source `.completed` turn; `ttsStarted` observed - [ ] `bash scripts/verify-entitlements.sh --pre-codesign` green (R-012) - [ ] `bash scripts/verify-codesign-settings.sh` green (R-013) - [ ] `JARVIS_REAL_MODELS=1 ... OrpheusTTFATests` green if Orpheus weights present (DEFERRED otherwise) - [ ] PR merged to develop ## Carry-forward closures - Closes #49 (B-04: voice input dead) - Closes #50 (B-05: TTS silent) ## Dependencies - Blocked by: #9 (M-5) - Blocks: #11 (M-7 Turn) ## Related issues - May dissolve: voice-audit findings P0-1..P0-4 ## Sub-issues - #126 — M-6a: define JarvisAPI.Voice + getVoiceState query (no audio-graph change) - #127 — M-6a: Voice events bridged through API surface (no audio-graph change) - #128 — M-6a: Voice runtime verbs — pttDown/pttUp/cancelTTS (UQ-4 slim Voice) - #129 — M-6a: Voice harness commands — _injectAudioFrame/_injectWakeWord/_injectSTT/_observeTTSAudio (UQ-1, G-001) - #130 — M-6b: startVoice() surfaces audioGraphFailed if ringBuffer nil (B-04, R-007) - #131 — M-6b: Voice.synthesizeTurn explicit-command contract (B-05, UQ-3 (A)) - #132 — M-6b: orchestrator issues synthesizeTurn on voice-source turnEnded(.completed) (B-05) - #133 — M-6b: bargeIn-driven cancelTTS internal invariant (R-018 substrate) - #134 — M-6: harness scenarios S-VOICE-001..012 in both transport modes (B-04 + B-05 BLOCKING gates) ## Estimate 4.0–5.5 engineer-days ## Source - `.planning/architecture/JARVIS-API-MIGRATION-PLAN.md` §2.7 - `.planning/architecture/JARVIS-API-DESIGN-v0.1.md` — `Voice` surface - `.planning/architecture/JARVIS-API-DESIGN-v0.2.md` §5 — UQ-3 (A) explicit-command, UQ-4 slim Voice - `.planning/architecture/REVIEW-MIGRATION-RISK.md` R-007, R-012, R-013, R-018 - Parent epic: #76

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/Jarvis#10