From 61e06fb2dca080e22df006dff0cffcfa020e90fd Mon Sep 17 00:00:00 2001 From: James Maes Date: Tue, 12 May 2026 14:52:25 -0500 Subject: [PATCH] docs(roadmap): user-outcome-focused milestone roadmap MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All 142 open issues triaged into 6 GitHub milestones organized by user-facing capability rather than technical area: - v0.1 — It actually works (28 issues): voice, listening, seeing, facts, debugging dialogs, basic UI. Tactical patches for CRITICAL audit findings so user has working Jarvis NOW. - v0.2 — Observable + stable (23): MED/LOW audit punch-list + PTT/TCC. - v0.5 — API M-0..M-3 (31): Self/Settings/Diagnostics surfaces; B-08. - v0.8 — API M-4..M-7 (36): Memory/Vision/Voice/Turn; B-03/B-04/B-05/B-06. - v1.0 — Shipping (13): commenting, doc reconciliation. - v1.1 — Backlog (11): Orpheus, WhisperKit, Voice Log, presence. ROADMAP-2026-05-12.md supersedes the build-phase ROADMAP.md from 2026-04-22 (kept for history; describes shipped phases). CLAUDE.md gains a Roadmap section pointing to the new doc + milestones URL. Also lands the 5 audit-2026-05-12 reports referenced by the roadmap (bus-hud, mcp-agent, memory, vision, voice). --- .planning/ROADMAP-2026-05-12.md | 189 ++++++++ .planning/audit-2026-05-12/bus-hud-audit.md | 279 ++++++++++++ .planning/audit-2026-05-12/mcp-agent-audit.md | 375 ++++++++++++++++ .planning/audit-2026-05-12/memory-audit.md | 98 +++++ .planning/audit-2026-05-12/vision-audit.md | 126 ++++++ .planning/audit-2026-05-12/voice-audit.md | 407 ++++++++++++++++++ CLAUDE.md | 12 + 7 files changed, 1486 insertions(+) create mode 100644 .planning/ROADMAP-2026-05-12.md create mode 100644 .planning/audit-2026-05-12/bus-hud-audit.md create mode 100644 .planning/audit-2026-05-12/mcp-agent-audit.md create mode 100644 .planning/audit-2026-05-12/memory-audit.md create mode 100644 .planning/audit-2026-05-12/vision-audit.md create mode 100644 .planning/audit-2026-05-12/voice-audit.md diff --git a/.planning/ROADMAP-2026-05-12.md b/.planning/ROADMAP-2026-05-12.md new file mode 100644 index 0000000..db637f5 --- /dev/null +++ b/.planning/ROADMAP-2026-05-12.md @@ -0,0 +1,189 @@ +# Jarvis Roadmap — 2026-05-12 + +**Supersedes** `.planning/ROADMAP.md` (2026-04-22), which describes the original 9-phase GSD build sequence. That sequence shipped — v0.12.0 is in tree. This document re-frames remaining work around **user-facing capability** rather than build phase. + +## What we're building + +A personal, always-on macOS AI assistant styled after the Iron Man "Jarvis" HUD. Native Swift/SwiftUI host with an embedded React + R3F web layer, an in-process agent loop talking to Claude Opus 4.7 (Ollama as first-class alternate), MCP tools for OS integration, and a fully-local voice pipeline (openWakeWord → SpeechAnalyzer → AVSpeechSynthesizer/Orpheus). Single user, single machine, no commercial shipping concerns. + +The core value the user is buying is **ambient presence**: "it's always there, I can talk to it, it gets things done on my Mac." Voice that actually fires. Listening that produces transcripts. Seeing through the camera. Facts that persist and get recalled. Debugging dialogs that show real state. UI that doesn't have basic correctness bugs. + +The architecture is settled and is not relitigated here. The story this roadmap tells is sequencing: **a working system first, a typed API second, polish third.** + +## Where we are today + +142 open issues. Five fresh audits (2026-05-12) covering Memory, MCP/Agent, Voice, Vision, and Bus/HUD — collectively producing ~36 findings ranging from `severity:critical` (system-breaking) to `severity:low` (cosmetic drift). Today's work landed the schema fix that unblocks memory writes, brought the Dev Overlay from wired-but-dead to fully populated, and surfaced a comprehensive picture of what is broken right now. The Round 4 strict-mode milestone is on tree; v0.12.0 audit-and-stabilize is the active workstream. + +The Jarvis API Migration Plan (`.planning/architecture/JARVIS-API-MIGRATION-PLAN.md`) is locked: 8 sequenced phases (M-0 → M-7) that move the entire client-facing surface from ad-hoc `BusInbound`/`BusOutbound` cases to typed `JarvisAPI` surfaces (`Self`, `Settings`, `Diagnostics`, `Memory`, `Vision`, `Voice`, `Turn`). That migration dissolves the carry-forward bugs B-02 through B-08 as a natural consequence — but it is 4–5.5 calendar weeks of work, and the user does not have a working system to use while it is in flight. + +This roadmap therefore splits work into two lanes: **v0.1/v0.2 = tactical fixes so today's Jarvis works end-to-end**, and **v0.5/v0.8 = the typed-API migration**. Some CRITICAL audit findings will be re-fixed during the migration; v0.1 applies the tactical patch anyway, because the user needs Jarvis working *now*. + +## Milestones + +### v0.1 — It actually works (basic capabilities) — 28 issues + +**User outcome:** I open Jarvis. Wake word fires. I say "Hey Jarvis, what time is it?" The ring pulses. STT transcribes me. The agent thinks. The HUD animates through state. TTS speaks the answer. I open the camera button — a real JPEG frame attaches to the next turn. I tell Jarvis "I prefer dark mode." Later I ask "what did I say I prefer?" and the fact comes back. I toggle the Dev Overlay and see real agent state, real tool calls, real latency. The Status panel shows real subsystem health. The chat panel auto-scrolls as tokens stream. Nothing in the basic flow is silently dead. + +**Scope:** Critical audit findings that break basic capabilities, plus the chat-scroll and startup-surface UX gaps. + +**Issues (28):** +- **Voice CRITICAL/HIGH (8):** #28 wake-word dies on audio-graph rebuild, #29 STT backend hard-coded, #30 .ttsStopped only on barge-in, #31 VoiceTTSAdapter.tierResolver dead, #32 OpenWakeWordSession buffers grow unbounded, #33 BootHealthProbe doesn't probe voice DAG, #34 SileroVAD hidden state persists, #36 AgentHudIntent stream dormant +- **Memory CRITICAL/HIGH (5):** #12 facts_vec never written, #13 replay log poisoned by synthetic memory-trigger turn_ids, #14 search_memory missing from preamble, #15 source_turn_id stores FNV hash, #16 hybridSearchSQL passes raw user text to FTS5 MATCH +- **MCP/Agent CRITICAL (2):** #20 ReplayingToolResultObserver.turnIDResolver permanently nil, #21 tool-result bytes double-capped at 8KB +- **Bus/HUD CRITICAL (3):** #40 BridgeNavigationDelegate no didFail arm, #41 MenuBarIconController.transition dead, #42 DevOverlay hollow shell +- **Vision (3):** #1 T2 misadvertised when MissingT2Provider is wired, #2 FrameAttachController pendingFrame leak, #3 mid-session camera TCC revoke not observed +- **UX essentials (2):** #51 (B-06) chat panel auto-scroll, #89 Open Status panel + Dev Overlay on app start +- **Env setup unblocking facts (3):** #56 D-5 build libsqlite3 with extensions, #57 D-6 bundle vec0.dylib (likely already done via static link), #58 D-7 reconcile Ollama model baseline +- **In-flight today (1):** #65 P0 AppDelegate commenting style rewrite +- **Loose-end triage (1):** #63 stash@{0} resurface-or-drop + +**Estimated effort:** 6–9 engineer-days. Most of these are tactical and independent. Voice CRITs (#28–#33) cluster together; Memory CRITs (#12–#16) are a tight sequence after today's schema fix. The DevOverlay (#42) is mostly already done; #41 and #36 are HUD-signal wiring. + +**Key risks:** Voice fixes may surface deeper audio-graph install-order issues (re-emerge as B-04 in M-6); if so, the tactical patch in v0.1 stops short of full fix and that work moves to v0.8/M-6. The facts pipeline depends on user-side env work (#56, #58) — if the user hasn't run `ollama pull qwen2.5-coder:32b` and built `libsqlite3.dylib` with extensions, the memory CRITs cannot be fully verified. + +**Cross-references:** several v0.1 fixes will be supersededd during the API migration — for example, #41 (MenuBarIconController) overlaps with HUD-state moving to Diagnostics in M-3 (v0.5). The tactical fix ships in v0.1 anyway; the migration replaces the implementation but preserves the user-visible behavior. + +### v0.2 — Observable + stable (audit punch-list) — 23 issues + +**User outcome:** Jarvis is not just working — it is observable. Every audit finding from 2026-05-12 that wasn't a v0.1 blocker is addressed. Token caching actually triggers (#23). Tool-result bytes are accurate in replay (#21 was v0.1, #18 LOW here). Banners surface correctly. Confirmation panels render in fullscreen and on multiple monitors (#26). PTT works (#87). Input Monitoring denial surfaces a HUD banner with a deep link to System Settings (#88). + +**Scope:** Remaining MED/LOW audit findings and the two non-CRIT carry-forward UX gaps (#87 PTT, #88 Input Monitoring TCC). + +**Issues (23):** +- **Memory MED/LOW (3):** #17, #18, #19 +- **MCP/Agent MED/HIGH (6):** #22, #23, #24, #25, #26, #27 +- **Voice MED/LOW (6):** #35, #37, #38, #39, #54, #55 +- **Bus MED/LOW (4):** #43, #44, #45, #47 +- **Build LOW (1):** #46 +- **Carry-forward UX (2):** #87 PTT, #88 Input Monitoring +- **Live verify (1):** #62 + +**Estimated effort:** 3–5 engineer-days. Most issues are <2 hours individually; the long pole is #26 (confirmation panel multi-monitor) which needs real-hardware testing. + +**Key risks:** None significant. v0.2 is mostly mop-up. + +### v0.5 — API M-0..M-3 (Self / Settings / Diagnostics surfaces) — 31 issues + +**User outcome:** The webview talks to typed Swift surfaces, not ad-hoc bus cases. From the user's lens this is mostly invisible — but B-08 (Claude paraphrases the model name) dissolves, HUD state has exactly one writer (no more dual-emit races), the Dev Overlay snapshot is a typed query rather than a bus dead-letter, and the Status panel surfaces typed `Diagnostics.systemBannerEnqueued` events. The substrate for v0.8 lands here. + +**Scope:** Migration phases M-0 (substrate), M-1 (Self), M-2 (Settings), M-3 (Diagnostics) per `JARVIS-API-MIGRATION-PLAN.md`. + +**Issues (31):** +- **Epics (4):** #4 M-0, #5 M-1, #6 M-2, #7 M-3 — plus #76 (parent migration epic) +- **M-0 substrate (10):** #90–#99 +- **M-1 Self surface (5):** #100–#104, plus #53 (B-08 dissolves here) +- **M-2 Settings (4):** #105–#108 +- **M-3 Diagnostics (6):** #109–#114 + +**Estimated effort:** 6–9 engineer-days (M-0 2.5–3.5 + M-1 1.0–1.5 + M-2 1.5–2.0 + M-3 1.0–1.5 per migration plan). + +**Key risks:** M-0.6 (HUD single-writer behavioral gate) and M-3.2 (HUD state moves to Diagnostics, old emit deleted) is the highest-risk pair — if the old emit isn't deleted, dual-write races appear in production. Failure mode #1 in the migration plan covers this; behavioral gate is the safety net. + +### v0.8 — API M-4..M-7 (Memory / Vision / Voice / Turn surfaces) — 36 issues + +**User outcome:** Every carry-forward UX bug closes. Camera button (B-03) works through real WKWebView round-trip. Voice input (B-04) and TTS (B-05) are wired through typed `synthesizeTurn`/`getVoiceState`. The chat panel hydrates on connect via `Turn.listTurns` instead of a frozen `sessionHistory` push (B-06 partial). The `OutboundBatcher` is partitioned per-`turnId` so submit-cancel-submit races stop leaking tokens across turns. After N≥3 days of soak, `BUS_PROTOCOL_VERSION` bumps to `2.4.0` strict and the dual-channel fallback retires. + +**Scope:** Migration phases M-4 (Memory), M-5 (Vision), M-6 (Voice), M-7 (Turn) per migration plan. + +**Issues (36):** +- **Epics (4):** #8 M-4, #9 M-5, #10 M-6, #11 M-7 +- **M-4 Memory (6):** #115–#120 +- **M-5 Vision (5):** #121–#125 + #48 (B-03) +- **M-6 Voice (9):** #126–#134 + #49 (B-04) + #50 (B-05) +- **M-7 Turn (9):** #135–#143 + +**Estimated effort:** 12.5–17.5 engineer-days (M-4 2.0–3.0 + M-5 1.5–2.0 + M-6 4.0–5.5 + M-7 5.0–7.0 per migration plan). + +**Key risks:** M-6 is the highest-risk migration in plan — audio-graph install-order risk plus entitlement re-verify. Failure mode #4 (substrate not actually fixed; surface lies green while audio is dead) is mitigated by the `audioLevelChanged ≥10 within 500ms` behavioral gate. M-7 has the worst rollback in plan; mitigation is dual-channel handshake stays active across N≥3-day soak before M-7.8 lands. + +### v1.0 — Shipping (commenting, behavioral gates, first-launch polish) — 13 issues + +**User outcome:** Code-commenting style applied to load-bearing files so the next contributor (or compaction-recovered Claude session) can read the codebase without rebuilding context from audit docs. Stale plan-id comments grep-gated out. Doc reconciliation against actually-shipped state. First-launch experience polished. + +**Scope:** Commenting style guide rollout (#64 epic + #66–#72 + #74 grep gate + #75 CLAUDE.md TL;DR), doc reconciliation (#59, #60, #61). + +**Issues (13):** #59, #60, #61, #64, #66, #67, #68, #69, #70, #71, #72, #74, #75 + +**Estimated effort:** 4–6 engineer-days, mostly cosmetic. + +**Key risks:** None. This is polish. + +### v1.1 — Backlog (Orpheus, WhisperKit, Voice Log, presence, advanced) — 11 issues + +**User outcome:** TTS tier 2 (Orpheus, in-process MLX) actually lights up — quality TTS within 150–250 ms TTFA. WhisperKit STT fallback activated behind a feature flag. Voice Log menu item (B-07) resurfaces, finally giving a live wake-word/STT/TTS event stream visible in the UI. Vision T2 sidecar comes online (v1.2). Presence pipeline (`atDesk`/`presenceConfidence`) reactivates. + +**Scope:** All v1.1+ deferrals from the migration plan §10. + +**Issues (11):** #52 B-07, #73 leaf-package commenting, #77 Orpheus, #78 WhisperKit, #79 F-Q-E query cancellation, #80 F-15 rename polish, #81 OQ-T4 origin tags, #82 OQ-T2 fixture spec, #84 Vision T2 sidecar, #85 presence pipeline, #86 multi-client subscription + +**Estimated effort:** open-ended; activation work per capability. Orpheus requires ~6 GB weights download + empirical TTFA verification. + +## Dependency map + +``` +v0.1 (basic capabilities) ─────────────────────┐ + │ │ + │ same audit corpus │ user sees working Jarvis NOW + ▼ │ +v0.2 (audit punch-list) │ + │ │ + │ stability + observability │ + ▼ ▼ +v0.5 (API M-0..M-3) ──────► v0.8 (API M-4..M-7) ──────► v1.1 (deferred caps) + │ + │ strict topo-sort: M-0 → M-1 → M-2 → M-3 → M-4 → M-5 → M-6 → M-7 + │ + └──► v1.0 (polish — orthogonal, any time after v0.1) +``` + +- **v0.1 is independent** — no migration dependency, tactical patches. +- **v0.2 depends on v0.1** for stability (don't pile observability on top of broken capabilities). +- **v0.5 depends on v0.1+v0.2** for stable substrate; M-0..M-3 sequence is strict. +- **v0.8 depends on v0.5** strictly per migration plan. +- **v1.0 is orthogonal** — commenting + docs work can run any time after v0.1. +- **v1.1 depends on v0.8** for the carry-forward bugs to be closed first. + +## How to read this roadmap + +Each milestone is **a coherent user outcome**, not a date. The issues in each milestone are sequenced for atomic-commit work; within v0.1, most fixes are independent (Voice CRITs cluster, Memory CRITs cluster, UX is its own lane). Within v0.5/v0.8 the migration plan is the authoritative sequence: M-0 blocks M-1 blocks M-2 etc., strict topo-sort. + +Carry-forward bugs B-02 through B-08 close as follows: +- B-02 (history threading) — already closed by the tactical patch on `develop` (commit `48ad8d6`-ish vicinity); regression-gated again in v0.8/M-4. +- B-03 (camera button) — closes in v0.8/M-5. +- B-04 (voice input dead) — closes in v0.8/M-6 substrate fix. +- B-05 (TTS silent) — closes in v0.8/M-6 via `synthesizeTurn` command. +- B-06 (chat scroll) — **tactical fix in v0.1** (#51); migration partial-close in v0.8/M-7 via `turnTextComplete` event. +- B-07 (Voice Log menu) — DEFERRED to v1.1. +- B-08 (model paraphrase) — closes in v0.5/M-1 via typed `modelDisplayName`. + +## What this roadmap deliberately does NOT include + +- **Multi-tenant / auth / commercial shipping** — out of scope per `PROJECT.md`. Single-user personal project. +- **Screen capture / ScreenCaptureKit** — deferred to v2. +- **Multi-panel holographic HUD / ambient data tiles** — v2. +- **Broader MCP tool catalog beyond the starter three** — additive once the loop works; v2. +- **Cloud TTS/STT** — hard rule. +- **Sonnet as primary model** — superseded by Opus 4.7. +- **Multi-machine sync (iCloud / backend)** — local-first until single-machine proves out. +- **Fixing all 22 HIGH + 22 MEDIUM AUDIT-R4 findings as a discrete gate** — absorbed into build phases per `PROJECT.md`. + +## Top 10 v0.1 sequencing recommendation + +For the user picking up v0.1 work next, the natural execution order is: + +1. **#65** — AppDelegate P0 commenting rewrite (in flight today) +2. **#42** — DevOverlay wire-up (mostly done today; verify and close) +3. **#41** — MenuBarIconController.transition wired +4. **#40** — BridgeNavigationDelegate didFail arm +5. **#28** — wake-word survives audio-graph rebuild +6. **#29** — STT backend selection wired +7. **#30, #31, #32, #33, #34, #36** — remaining voice CRITs as a cluster +8. **#12, #13, #14, #15, #16** — memory CRITs as a cluster +9. **#20, #21** — MCP turn-id observer + double-cap +10. **#1, #2, #3** — vision CRITs/HIGHs + +#51 (chat auto-scroll) and #89 (startup surfaces) drop in anywhere — they're independent webview work. + +--- + +*Authoritative state lives in `.planning/STATE.md` (current GSD phase position) and the GitHub milestones at https://github.com/KofTwentyTwo/Jarvis/milestones. This document is the narrative; those are the system of record.* diff --git a/.planning/audit-2026-05-12/bus-hud-audit.md b/.planning/audit-2026-05-12/bus-hud-audit.md new file mode 100644 index 0000000..4ff1ed2 --- /dev/null +++ b/.planning/audit-2026-05-12/bus-hud-audit.md @@ -0,0 +1,279 @@ +# Bus + HUD + WKWebView Bridge — End-to-End Audit + +**Date:** 2026-05-12 +**HEAD:** `a5999e2` +**Scope:** `packages/Bus/`, `webview/packages/bus/`, `webview/packages/hud/`, `App/HUD/`, `App/MenuBar/`, `App/AppDelegate.swift` bridge / handshake / inbound-outbound surfaces. +**Mode:** Read-only on production; written to this file only. +**Calibration:** Look for shape-of-today's-bug — probe says ok, reality is wired-but-dead. Surface "wants to write HudState but goes through a non-coordinator path", outbound cases with no producer, inbound cases with `// TODO` arms, dead-letter views. + +--- + +## TL;DR — top-3 by severity + +1. **`BridgeNavigationDelegate` has no failure arm.** Only `didFinish` is implemented. If `index.html`, the JS bundle, or any sub-resource fails to load (bad reference, 404 on `file://`, JS parse error before mount), the navigation never finishes, `startHandshake()` is never invoked, the 2-second handshake timeout **is never even armed**, and the user sees a silent black HUD with no banner, no hard-block. The "missing index.html" hard-block on AppDelegate.swift:2553 is the only failure mode that surfaces. +2. **`MenuBarIconController.transition(to:)` is production-dead.** The full CABasicAnimation surface (breath / rotate / shimmer / glow per state) ships and works in unit tests, but **no production code ever calls `transition(to:)`** — `HudStateCoordinator`'s emit closure fans HudState to the webview bridge only. The menu-bar icon never animates regardless of agent state. Only `applyHealth(_:)` (Round 4) is wired. +3. **`DevOverlay` is a hollow shell.** `DevOverlayWindow` opens with a default `DevOverlayViewModel` holding `DevSnapshot.initial`. The `.devOverlay` broadcaster subscription on AppDelegate.swift:1808-1813 drains events into `_` with the comment "reserved for the DevOverlay emitter wiring in a follow-on plan." `DevOverlayBridge` is never instantiated. Opening the overlay shows static initial values (state=idle, 0 tokens, empty turn id) forever. + +--- + +## Severity-ordered findings + +### S1 — `BridgeNavigationDelegate` only implements `didFinish` (silent black HUD on bundle failure) + +**Files:** `App/AppDelegate.swift:2715-2723`, `packages/Bus/Sources/Bus/WebviewBridge.swift:269-288`, `packages/Bus/Sources/Bus/Handshake.swift:24-26`. + +`BridgeNavigationDelegate` is a 9-line class with one method: + +```swift +nonisolated func webView(_ webView: WKWebView, didFinish navigation: WKNavigation!) { + MainActor.assumeIsolated { onDidFinish() } +} +``` + +It has no `webView(_:didFail:withError:)` and no `webView(_:didFailProvisionalNavigation:withError:)`. `WebviewBridge.startHandshake()` schedules the 2-second timeout (`HandshakeTiming.timeout = .seconds(2)`) **inside** `startHandshake`, which is only invoked from `didFinish`. So the failure mode chain is: + +- `loadFileURL` succeeds (file exists on disk). +- `index.html` body loads OK, but `