Files
james.maesandClaude Opus 4.7 1f8e03edeb fix(track-a): demo-blocker class + audit reports + chat-turn e2e tests
User feedback: 50+ hours of agent work, nothing demonstrably working.
Fanned out 6 deep auditors (voice, HUD, vision, memory, LLM, tests);
all reports landed at .planning/audit-2026-05-03/ with brutal-honesty
mandates. Synthesis (SYNTHESIS.md) confirmed: every modality has at
least one fatal break between code-that-passes-tests and
code-that-runs. Track A is the smallest credible path to one working
modality (text turn + animated rings).

Closed under TDD discipline:

**Item 4 — HUD idle ring static rendering (F-A1-01).**
- webview/packages/hud/src/hud/stateUniforms.ts:30 — was
  `pulse: 0.0, rotate: 0.0` for idle, the all-static configuration
  that rendered STANDBY as dots instead of animated rings. Now
  pulse: 0.4, rotate: 0.05 — gentle ambient distinct from booting +
  thinking.
- webview/packages/hud/tests/stateUniforms.test.ts — table updated;
  new regression-guard `STATE_PARAMS[*] is not all-static` (7 cases,
  one per HudState) prevents anyone from re-zeroing a state.

**Item 1 — CacheHints eligibility gate (likely streamTruncated cause).**
- packages/AgentCore/Sources/AgentCore/CacheHints.swift —
  + static func eligibleForSystemPrompt(_:ttl:). Anthropic's prompt
  cache requires ≥1024 tokens at the breakpoint; below that, the
  server returns 200 OK + immediate-EOF for `extended-cache-ttl`
  hints. The orchestrator's default system prompt is ~10 tokens.
  Helper returns nil for sub-threshold prompts so no cache_control
  marker is emitted.
- packages/AgentCore/Sources/AgentOrchestrator/AgentOrchestrator.swift
  — call sites now pass `CacheHints.eligibleForSystemPrompt(finalSystem)`
  through runTurnLoop instead of unconditional `.extended1h`.
- Tests: CacheHintsEligibilityTests (6 cases), RequestBodyTests R5a/R5b
  (2 cases verifying small-prompt → no cache_control in encoded JSON,
  large-prompt → cache_control present).

**Item 2 — StreamOutcome diagnostic.**
- packages/AgentCore/Sources/AnthropicProvider/StreamOutcome.swift —
  new pure classifier that distinguishes:
    - non-2xx HTTP status
    - 200 OK + 0 bytes (cache_control / auth-header rejection class)
    - 200 OK + bytes + 0 frames (malformed body)
    - 200 OK + frames + EOF before message_stop (mid-stream drop)
    - 200 OK complete (healthy)
  AnthropicProvider's read loop counts frames + bytes and logs the
  summary once per stream. Production logs now distinguish failure
  modes that previously all surfaced as `streamTruncatedFinal`.
- Tests: StreamOutcomeTests (7 cases, all branches).

**Item 3 — KeychainError surfacing.**
- packages/AgentCore/Sources/AnthropicProvider/AnthropicAPIKeyProvider.swift
  — new builder `make(keychain:item:)` returns the
  `@Sendable () async throws -> String` closure with errors
  propagating verbatim. Closes the audit's hidden-401 finding:
  `(try? keychainStoreLocal.get(.anthropic)) ?? ""` swallowed the
  real KeychainError into an empty x-api-key, producing
  `401 x-api-key header is required` indistinguishable from a
  stale-key 401. Now the actual error reaches the chat panel via
  AnthropicProvider → orchestrator.error → BusForwarder →
  submitRejected.
- App/AppDelegate.swift — both Anthropic provider construction sites
  (installAgent, installVision T3 wiring) now use the propagating
  builder.
- Tests: AnthropicAPIKeyProviderTests (6 cases — itemNotFound,
  unexpectedStatus, duplicateItem propagation; happy path; explicit
  item; default item).

**Item 5 — F1 #2 chat-turn round-trip e2e tests.**
- packages/Bus/Tests/BusTests/RealWKWebViewIntegrationTests.swift —
  + 2 new tests using a real WKWebView:
    - test_chatTurnRoundTrip_busOutboundReachesPageHandler — Swift
      `bridge.send(.turnStarted/tokenDelta/turnEnded)` reaches the
      page's onOutbound handler. This is the test the v0.12.0 audit
      called out as catching BLOCKER-INT-1, INT-2, INT-3,
      F-E-RACE-1, F-E-FK-1, F-E-WIRE-1 in one shot.
    - test_chatTurnRoundTrip_busInboundFromJSReachesSwiftHandler —
      JS-side `window.jarvisBus.send({type:'chatSubmit',...})` —
      exactly what ChatInput.tsx does — reaches the bridge's
      onInbound handler with the right body.

**Verification:**
- AgentCore: 215/215 (was 178; +37 across BusForwarder, CacheHints,
  StreamOutcome, AnthropicAPIKeyProvider, RequestBody R5).
- Replay: 33/33.
- Bus: 57/58 (1 pre-existing failure: F-A2-01 stale `JarvisBusWorld`
  assertion documented in AUDIT-FINDINGS, separate scope).
- webview/packages/hud: 94/94 (was 87; +7 stateUniforms regression-
  guards + idle table updates).
- All 15 boundary grep gates: PASS.
- scripts/check-app-builds.sh: PASS.

**Audit reports retained at .planning/audit-2026-05-03/:**
- SYNTHESIS.md (consolidated, with triage plan)
- hud-audit.md, llm-audit.md, memory-audit.md, tests-audit.md,
  vision-audit.md, voice-audit.md, voice.md (per-modality detail)

**Carry-forward (NOT closed this commit):**
- F-A2-01 stale WebviewBridgeOutboundTests assertion (pre-existing).
- Voice (F): models never bundled, audio graph never instantiated,
  STT bridge body is `_ = chunk`, TTS engine nil. ~1 day.
- Vision (F): AVCapturePhotoOutput allocated but capturePhoto never
  called; HUD camera button doesn't exist. ~1 day.
- Memory (F): Apple's stripped libsqlite3 prevents sqlite-vec from
  loading; "future ops plan" has zero commits. ~3 days.
- LLM (D+): NoopBusGateway still drops tool-call events
  (BLOCKER-INT-1).
- Tests (C-): 5 XCTAssertTrue(true) tautologies + multiple
  rubber-stamp tests defending stubs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-03 17:23:13 -05:00

121 lines
4.7 KiB
Swift

import XCTest
import Keychain
@testable import AnthropicProvider
/// Coverage for `AnthropicAPIKeyProvider.make(keychain:item:)` — the
/// builder that propagates `KeychainError` instead of swallowing it into
/// the empty-string fallback that produced the 2026-05-03 audit's
/// silent-401 symptom.
final class AnthropicAPIKeyProviderTests: XCTestCase {
// MARK: - Test doubles
private struct ThrowingKeychain: KeychainStore {
let error: KeychainError
func set(_ value: String, for item: KeychainItem) throws { throw error }
func get(_ item: KeychainItem) throws -> String { throw error }
func delete(_ item: KeychainItem) throws { throw error }
}
private struct StaticKeychain: KeychainStore {
let value: String
func set(_ value: String, for item: KeychainItem) throws {}
func get(_ item: KeychainItem) throws -> String { value }
func delete(_ item: KeychainItem) throws {}
}
private struct CapturingKeychain: KeychainStore, @unchecked Sendable {
// @unchecked Sendable: the captured array is mutated only inside
// a serial test method; no cross-thread access in tests.
final class Box: @unchecked Sendable {
var requested: [KeychainItem] = []
}
let box = Box()
let value: String
func set(_ value: String, for item: KeychainItem) throws {}
func get(_ item: KeychainItem) throws -> String {
box.requested.append(item)
return value
}
func delete(_ item: KeychainItem) throws {}
}
// MARK: - Error propagation (KP-1..KP-3)
/// KP-1: itemNotFound throws verbatim — does NOT collapse to empty
/// string. This is the literal regression the 2026-05-03 audit named.
func test_propagates_itemNotFound() async {
let kc = ThrowingKeychain(error: .itemNotFound)
let provider = AnthropicAPIKeyProvider.make(keychain: kc, item: .anthropic)
do {
_ = try await provider()
XCTFail("provider must throw, not silently substitute empty string")
} catch let err as KeychainError {
XCTAssertEqual(err, .itemNotFound)
} catch {
XCTFail("expected KeychainError, got \(error)")
}
}
/// KP-2: unexpectedStatus also propagates verbatim with its OSStatus.
func test_propagates_unexpectedStatus() async {
let kc = ThrowingKeychain(error: .unexpectedStatus(-25300))
let provider = AnthropicAPIKeyProvider.make(keychain: kc, item: .anthropic)
do {
_ = try await provider()
XCTFail("provider must throw")
} catch let err as KeychainError {
XCTAssertEqual(err, .unexpectedStatus(-25300))
} catch {
XCTFail("expected KeychainError, got \(error)")
}
}
/// KP-3: duplicateItem also propagates (less likely on read, but the
/// contract is total — every KeychainError variant flows through).
func test_propagates_duplicateItem() async {
let kc = ThrowingKeychain(error: .duplicateItem)
let provider = AnthropicAPIKeyProvider.make(keychain: kc, item: .anthropic)
do {
_ = try await provider()
XCTFail("provider must throw")
} catch let err as KeychainError {
XCTAssertEqual(err, .duplicateItem)
} catch {
XCTFail("expected KeychainError, got \(error)")
}
}
// MARK: - Happy path (KP-4..KP-5)
/// KP-4: when the keychain returns a value, the provider relays it.
func test_returns_value_on_success() async throws {
let kc = StaticKeychain(value: "sk-ant-test-12345")
let provider = AnthropicAPIKeyProvider.make(keychain: kc, item: .anthropic)
let key = try await provider()
XCTAssertEqual(key, "sk-ant-test-12345")
}
/// KP-5: provider passes the requested item through unchanged. Locks
/// in the contract that callers can target a different item without
/// the builder substituting `.anthropic` silently.
func test_uses_specified_item() async throws {
let kc = CapturingKeychain(value: "_")
let other = KeychainItem(service: "com.example.test", account: "test-account")
let provider = AnthropicAPIKeyProvider.make(keychain: kc, item: other)
_ = try await provider()
XCTAssertEqual(kc.box.requested, [other])
}
/// KP-6: default `item` parameter is `.anthropic`.
func test_default_item_is_anthropic() async throws {
let kc = CapturingKeychain(value: "_")
let provider = AnthropicAPIKeyProvider.make(keychain: kc)
_ = try await provider()
XCTAssertEqual(kc.box.requested, [.anthropic])
}
}