0 Fork 0

Files

T

History

james.maes 05411f82ff fix(vision): v0.1 cluster — T2 honesty + pendingFrame leak + TCC revoke observation (#1 , #2 , #3 , #152 )

#1 (HIGH): new hasRealT2Provider init param on AgentOrchestrator; t2AvailableForThisTurn now requires both decision.tier != .t3Cloud AND hasRealT2Provider. Production passes false (MissingT2Provider only). Pre-existing escalation test still passes true with mock T2.

#2 (MED): new FrameAttachController.releaseAfterRejectedSubmit() routes through the sole discardFrame() emission site (D-15 single-emission grep gate still green). Both handleChatSubmit and handleChatCancelAndSubmit call it on .rejected.

#3 (MED): AVCaptureSessionWasInterruptedNotification observer yields .midSessionRevoked. macOS limitation surfaced: AVCaptureSessionInterruptionReasonKey is API_UNAVAILABLE(macos), so we cannot discriminate by reason — any interruption surfaces the banner. NSApplication.didBecomeActive observer calls becameAuthorized() on non-authorized → .authorized flip.

Vision 75 + AgentCore 243 tests pass; all 18 boundary gates green.

2026-05-13 06:30:09 -05:00

Sources/Vision

fix(vision): v0.1 cluster — T2 honesty + pendingFrame leak + TCC revoke observation (#1 , #2 , #3 , #152 )

2026-05-13 06:30:09 -05:00

Tests/VisionTests

fix(vision): v0.1 cluster — T2 honesty + pendingFrame leak + TCC revoke observation (#1 , #2 , #3 , #152 )

2026-05-13 06:30:09 -05:00

Package.swift

test(07-05): add failing VllmMlxSidecar spawn invariant tests + Vision deps (Task 4 RED)

2026-04-29 17:53:28 -05:00

README.md

docs(packages): per-package README files

2026-05-04 17:13:33 -05:00

README.md

Vision

Library product JarvisVision. Camera capture, presence detection, frame-attach to a turn, and tier routing for vision-capable LLMs.

Key public types

Type	Purpose
`CameraCapture` (actor)	`AVCaptureSession` lifecycle. `captureFrame()` for one-shot stills via `AVCapturePhotoOutput`; `frameStream(forPresence:)` for continuous fan-out
`FrameAttachController` (actor)	Lifecycle owner of the pending-frame slot. `requestAttach(reason:)` → `confirmSend(userText:)` / `cancel()` / `onAssistantTurnComplete()`
`PresenceMonitor`	At-desk detection from frame stream
`PresenceEvent` / `PresenceStateSnapshot`	Presence stream types
`VisionRouter`	Picks T1 (Ollama) / T2 (vllm-mlx) / T3 (Anthropic cloud-escape)
`VisionTier`	T1 / T2 / T3 enum
`MissingT2Provider`	Explicit-missing T2 — surfaces unwired sidecar (no silent T1 fallback)
`VllmMlxProvider` / `VllmMlxSidecar`	Skeleton for the T2 sidecar (not yet wired)
`EscalationPhraseDetector` / `VisionEscalationHeuristic`	Heuristics for tier escalation
`ContextBuilder`	Phrase trigger detection (`take a look at this` → `requestAttach`)
`CapturedFrame`	JPEG bytes + dimensions
`VisionError`	Error union

Depends on

AgentCore, MCP (JarvisChildSpawn for sidecar spawning), Logging. External: swift-log, OllamaProvider, AnthropicProvider (T1/T3).

Used by

App/AppDelegate, App/Vision/, packages/AgentCore (orchestrator imports JarvisVision for the multimodal-stream overload's image type bridging).

Key invariants / contracts

captureFrame() returns ONE still frame on demand. Continuous video is not exposed outside the package — only frameStream(forPresence:), and PresenceMonitor is the only in-tree consumer.
FrameAttachController.discardFrame() is the SOLE emission site for clearing the pending-frame slot (D-15). Every release path (cancel, timeout, assistant-turn-complete) routes through that single function. FrameAttachDiscardSiteGrepTests enforces structurally.
Single ingest path. requestAttach(reason:) is reachable from BOTH (a) ContextBuilder.matchesFrameAttachPhrase AND (b) the HUD camera-icon Bus message. Both producers reach the same method (D-13 dual-trigger contract).
MissingT2Provider is intentional. Replaces the silent t2Provider: t1 AppDelegate hardcode (Track C-4); makes the missing wiring debuggable instead of silent.
Vision isolation. scripts/check-vision-isolation.sh + check-presence-vision-isolation.sh enforce the package boundary (no upward leaks).
Presence bus doesn't reach TTS or orchestrator directly. scripts/check-presence-bus-no-tts-orchestrator.sh.

Tests

72 XCTest. CameraCaptureRealHardwareTests is gated by JARVIS_REAL_CAMERA=1 + .authorized TCC — runs interactively only.

Notable files

Sources/Vision/CameraCapture.swift — actor + AVCaptureSession lifecycle
Sources/Vision/FrameAttachController.swift — pending-frame lifecycle
Sources/Vision/PresenceMonitor.swift — at-desk detection
Sources/Vision/VisionRouter.swift — T1/T2/T3 routing
Sources/Vision/MissingT2Provider.swift — explicit-missing T2