feat: local-first LLM routing with reactive Opus escalation #198

2026-05-16T21:21:14Z

KofTwentyTwo commented

2026-05-16 21:21:14 +00:00

(Migrated from github.com)

Summary

Flips the default chat-turn provider from Anthropic to local Ollama (qwen2.5-coder:32b-instruct-q8_0). If Ollama fails in any of six detectable ways, the orchestrator escalates the same turn to Claude Opus 4.7 and the HUD renders a small badge on the affected message.

Implements the design spec in docs/superpowers/specs/2026-05-14-local-first-llm-routing-design.md (PR #197) per the plan at docs/superpowers/plans/2026-05-14-local-first-llm-routing.md. Executed via the Superpowers subagent-driven-development skill, 9 implementation tasks + 1 verification task, fresh subagent per task with two-stage review (spec compliance + code quality) after each.

Behavior changes

default-config.json now ships provider: "ollama" + escalationEnabled: true. Old configs auto-opt-in via decodeIfPresent ?? true.
New Settings → Provider 3-mode picker: Local-first (recommended) / Anthropic only / Ollama only.
Per-turn escalation budget is 1. A second failure on the same turn surfaces a terminal error (no infinite loop).
HUD inline badge ("Escalated to Opus — ") for the escalated message; subsequent messages render clean.

Detection — closed enum `OllamaFailureKind`

Trigger	Detection site
`streamTruncated`	Existing AGENT-09 retry path (unchanged)
`malformedToolCall`	NDJSONDecoder parses `tool_calls.arguments`; emits `providerError(.malformedToolCall(reason:))` on parse failure
`unknownTool`	Orchestrator-side (registry check)
`refusal`	Existing `StopReason.refusal` event
`connectionFailure`	URLSession error before any tokens received
`emptyResponse`	NDJSONDecoder tracks `sawAnyTextDelta`/`sawAnyToolUseRequested`; emits `.emptyResponse` if both false at `done:true`

Architecture notes

Bus → AgentCore dependency NOT inverted. BusOutbound.escalated carries a Bus-local EscalationDecision mirror (with BusProviderSelection, BusOllamaFailureKind), following the existing TurnRow precedent. Translation happens in AppBusForwarderSink.
escalatedThisTurn is a local var inside runTurnLoop — resets naturally per turn, no cross-turn sticky state.
The existing streamTruncated retry branch stays parallel to the new escalation path. Different LLMEvent shapes (transport-mid-stream EOF vs. parsed-failure), independent budgets — they can co-occur and should not share a counter.
Memory extraction untouched. Pipeline was already Ollama-only via MemoryExtractor; this PR doesn't modify it.
Bus protocol version bumped 2.3.0 → 2.4.0 (additive MINOR per HUD-05). TS decoder validates per-field with specific error messages for unknown providers/reasons.

Test plan

OllamaFailureKindTests: 6-case allCases round-trip + full struct equality (packages/AgentCore)
PerTurnSnapshotEscalationTests: 3 tests — field round-trip, decodeIfPresent defaulting, encode-decode false survives (packages/Config)
test_default_config_resource_is_local_first — asserts shipped JSON is local-first (packages/Config)
NDJSONDecoderTests N8/N9 + 2 negative-suppression tests: malformed + empty detection (packages/AgentCore)
LocalFirstEscalationTests A/B/C: reactive fire-once, budget honored, disabled passthrough (packages/AgentCore)
CodableRoundTripTests.test_roundTrip_escalated + cross-side JSON fixture parity (packages/Bus + webview/packages/bus)
BusForwarderTests: 2 escalation translation tests (packages/AgentCore)
EscalationBadge.test.tsx: friendly text, testid, all 6 reasons render (webview/packages/hud)
bus-dispatch.test.ts: in-flight stamp + pre-stream parked-decision attach (webview/packages/hud)
ProviderModeTests: 3 mapping tests, including anthropic-true normalization (App/Tests)
All 18 boundary gates PASS (incl. bus parity + protocol version)
App target xcodebuild Debug — BUILD SUCCEEDED
Webview: Bus 38/38, HUD 106/106
AgentCore 259/259, Bus 65/65, Config 20/20, Memory 98/98 (verified during per-task subagent loops)

Non-goals

No predictive routing or router-model classifier
No "stream Anthropic continuing Ollama's partial output" — clean restart only
No automatic re-Ollama after Anthropic finishes — once escalated, the rest of the turn stays on Anthropic
No memory-extraction changes (already Ollama-only)
No non-chat surfaces (TTS, MCP, voice loop)

Stacked on

Spec PR #197 (docs/local-first-llm-routing-spec). This PR can land independently; the spec PR adds the design doc + plan doc that explain this PR.

🤖 Generated with Claude Code

## Summary Flips the default chat-turn provider from Anthropic to local Ollama (`qwen2.5-coder:32b-instruct-q8_0`). If Ollama fails in any of six detectable ways, the orchestrator escalates the same turn to Claude Opus 4.7 and the HUD renders a small badge on the affected message. Implements the design spec in `docs/superpowers/specs/2026-05-14-local-first-llm-routing-design.md` (PR #197) per the plan at `docs/superpowers/plans/2026-05-14-local-first-llm-routing.md`. Executed via the Superpowers subagent-driven-development skill, 9 implementation tasks + 1 verification task, fresh subagent per task with two-stage review (spec compliance + code quality) after each. ## Behavior changes - `default-config.json` now ships `provider: "ollama"` + `escalationEnabled: true`. Old configs auto-opt-in via `decodeIfPresent ?? true`. - New Settings → Provider 3-mode picker: `Local-first (recommended)` / `Anthropic only` / `Ollama only`. - Per-turn escalation budget is 1. A second failure on the same turn surfaces a terminal error (no infinite loop). - HUD inline badge ("Escalated to Opus — <reason>") for the escalated message; subsequent messages render clean. ## Detection — closed enum `OllamaFailureKind` | Trigger | Detection site | |---|---| | `streamTruncated` | Existing AGENT-09 retry path (unchanged) | | `malformedToolCall` | NDJSONDecoder parses `tool_calls.arguments`; emits `providerError(.malformedToolCall(reason:))` on parse failure | | `unknownTool` | Orchestrator-side (registry check) | | `refusal` | Existing `StopReason.refusal` event | | `connectionFailure` | URLSession error before any tokens received | | `emptyResponse` | NDJSONDecoder tracks `sawAnyTextDelta`/`sawAnyToolUseRequested`; emits `.emptyResponse` if both false at `done:true` | ## Architecture notes - **Bus → AgentCore dependency NOT inverted.** `BusOutbound.escalated` carries a Bus-local `EscalationDecision` mirror (with `BusProviderSelection`, `BusOllamaFailureKind`), following the existing `TurnRow` precedent. Translation happens in `AppBusForwarderSink`. - **`escalatedThisTurn` is a local var inside `runTurnLoop`** — resets naturally per turn, no cross-turn sticky state. - **The existing `streamTruncated` retry branch stays parallel** to the new escalation path. Different `LLMEvent` shapes (transport-mid-stream EOF vs. parsed-failure), independent budgets — they can co-occur and should not share a counter. - **Memory extraction untouched.** Pipeline was already Ollama-only via `MemoryExtractor`; this PR doesn't modify it. - **Bus protocol version bumped** 2.3.0 → 2.4.0 (additive MINOR per HUD-05). TS decoder validates per-field with specific error messages for unknown providers/reasons. ## Test plan - [x] `OllamaFailureKindTests`: 6-case `allCases` round-trip + full struct equality (`packages/AgentCore`) - [x] `PerTurnSnapshotEscalationTests`: 3 tests — field round-trip, `decodeIfPresent` defaulting, encode-decode `false` survives (`packages/Config`) - [x] `test_default_config_resource_is_local_first` — asserts shipped JSON is local-first (`packages/Config`) - [x] `NDJSONDecoderTests` N8/N9 + 2 negative-suppression tests: malformed + empty detection (`packages/AgentCore`) - [x] `LocalFirstEscalationTests` A/B/C: reactive fire-once, budget honored, disabled passthrough (`packages/AgentCore`) - [x] `CodableRoundTripTests.test_roundTrip_escalated` + cross-side JSON fixture parity (`packages/Bus` + `webview/packages/bus`) - [x] `BusForwarderTests`: 2 escalation translation tests (`packages/AgentCore`) - [x] `EscalationBadge.test.tsx`: friendly text, testid, all 6 reasons render (`webview/packages/hud`) - [x] `bus-dispatch.test.ts`: in-flight stamp + pre-stream parked-decision attach (`webview/packages/hud`) - [x] `ProviderModeTests`: 3 mapping tests, including anthropic-true normalization (`App/Tests`) - [x] All 18 boundary gates PASS (incl. bus parity + protocol version) - [x] App target xcodebuild Debug — BUILD SUCCEEDED - [x] Webview: Bus 38/38, HUD 106/106 - [x] AgentCore 259/259, Bus 65/65, Config 20/20, Memory 98/98 (verified during per-task subagent loops) ## Non-goals - No predictive routing or router-model classifier - No "stream Anthropic continuing Ollama's partial output" — clean restart only - No automatic re-Ollama after Anthropic finishes — once escalated, the rest of the turn stays on Anthropic - No memory-extraction changes (already Ollama-only) - No non-chat surfaces (TTS, MCP, voice loop) ## Stacked on - Spec PR #197 (`docs/local-first-llm-routing-spec`). This PR can land independently; the spec PR adds the design doc + plan doc that explain this PR. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

You are not authorized to merge this pull request.

This pull request can be merged automatically.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feat/local-first-llm-routing:feat/local-first-llm-routing

git checkout feat/local-first-llm-routing

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/Jarvis#198