Three asks landing together. ## History view — now identical controls to Log Stream - Oldest-first order with auto-scroll-to-bottom on append (was newest- first, no auto-scroll). ScrollViewReader + onChange pattern lifted from LogStreamView for behavioral parity. - Toolbar now matches the Log Stream layout exactly: Category filter, Auto-scroll toggle, Pause toggle, filtered/total count, Clear button. - Poll rate bumped 1 Hz → 0.10 s so new events appear instantly. Pause toggle freezes refresh without dropping the running event collection. ## Proper line shader (much better hand + body rendering) - New `line_vertex` / `line_fragment` in Boxes.metal alongside the existing box shader. `LineUniforms` carries endpoints + thickness + soft-edge falloff + a dot-mode flag. - Line mode: emits a rotated thin quad oriented along (end - start) with width = thickness. Soft-edge antialiasing via smoothstep on the signed perpendicular UV makes diagonal skeleton segments read as actual strokes instead of axis-aligned bounding boxes. - Dot mode: emits a square around `start` and the fragment discards outside the unit-circle — produces filled disks for joint markers with sub-pixel-clean soft edges. - Pipelines gains a `lines` blended pipeline. - Renderer.drawPoses() rewritten to use the line pipeline — one draw per segment, one draw per joint, all visually consistent regardless of segment angle. Theme.Palette.bodyPose (mint) for body, handPose (amber) for hand. ## Status panel (⌘I) - New `CurrentStateStore` singleton subscribes to the event bus and retains the most recent value per concept: expression (smile / frown / neutral_face), activity (standing / sitting / etc.), per-chirality finger counts, plus a small ring of recent non-finger non-expression gestures. - New StatusView (SwiftUI) reads three sources at 4 Hz: the live ObjectDetector for FPS + DET timing + objects + faces, the CurrentStateStore for hands/expression/activity/recent-gestures, and the HistoryStore for an 8-entry tail of recent events. - New StatusWindowController hosts the SwiftUI view as a dark-aqua floating window with frame persistence (VPOC.StatusWindowFrame). - AppDelegate `openStatus(_:)` builds a DetectorSnapshot from the live windowController + detector each tick, partitioning detections into face entries (anything labeled FACE or matching an enrolled name) vs object entries. Object entries carry the same per-track color the JARVIS pane uses (via ColorHash.colorFor) so the panel's dots match the on-screen box colors. - Renderer exposes `smoothedFPSForReadout` so the panel can show the same FPS the footer does without reaching into private state. - FingerCounter now records its count per chirality via CurrentStateStore.recordFingers (in addition to the existing bus emit), using VNHumanHandPoseObservation.chirality. The panel displays "Left: 3 fingers / Right: 5 fingers" correctly. Menu structure now: - App menu: Settings (⌘,) · Log Stream (⌘L) · Status (⌘I) · Save Snapshot (⌘S) · Start/Stop Recording (⇧⌘R) · Quit - Faces menu: Enroll (⌘E) · Forget (⇧⌘E) · History (⌘H) - Camera menu: Devices ▶ · Refresh · Privacy Mode (⇧⌘P) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
VisionPOC
A proof-of-concept real-time camera vision pipeline for macOS Apple Silicon. Captures live video and renders four synchronized views in a 2×2 grid: raw passthrough, a Jarvis HUD stylization (cyan tint, scanlines, hex grid, scanning beam), an edge map (MPS Sobel), and an object-detection overlay (Vision saliency + face / human / animal recognizers running on the Apple Neural Engine).
Sibling to MetalPOC (Jarvis HUD rendering spine) and VoicePOC (voice stack). This POC proves the camera + ANE + Metal pipeline so future Jarvis "sight" features can build on a known-good foundation.
Highlights
- Zero-copy capture —
AVCaptureSession→CVPixelBuffer→MTLTextureviaCVMetalTextureCache(no CPU roundtrip) - 601-class object detection — YOLOv8x trained on Open Images V7, exported to a Vision-friendly Core ML pipeline. Recognizes everyday vocabulary like coffee cup, mug, wine glass, mobile phone, laptop, computer monitor, sunglasses, plus body parts (human face / hand / eye) and a broad household / vehicle / animal vocabulary
- Object tracking —
VNTrackObjectRequestupdates each track's box on every frame between YOLO refreshes, so detections drift smoothly with motion instead of teleporting - Face recognition — enroll a face via the menu bar (⌘E), it shows up labeled by name in the JARVIS pane on subsequent frames. Powered by Vision's image FeaturePrint on padded face crops; templates persist to
~/Library/Application Support/VisionPOC/ - Apple Neural Engine — Vision requests configured with
MLComputeUnits.all; runs on the ANE wherever possible - Single render pass, four viewports — one
MTKView, one render pass, four fragment pipelines, one HUD overlay - MPS Sobel for edge detection — GPU-native, sub-millisecond at 1080p
- Jarvis stylization shader — fragment-only Metal shader matching MetalPOC's HUD palette and fonts
- Detector decoupled from render — runs at ~20 Hz on a serial background queue; never blocks the 60 fps render thread
- Centralized tunables in
Sources/Theme.swift— palette, layout, fonts, capture preset, detector cadence, performance profile
Requirements
- macOS 26 Tahoe (Apple Silicon)
- Xcode 16+ with Metal toolchain
- xcodegen (
brew install xcodegen) - A built-in or attached camera
Build & Run
Prerequisite: Git LFS is required because the bundled YOLOv8x model (~131 MB, Open Images V7, 601 classes) is tracked via LFS. Install once on your machine: brew install git-lfs && git lfs install.
git clone https://github.com/KofTwentyTwo/VisionPOC.git
cd VisionPOC
git lfs pull # materialize the bundled model
xcodegen generate
xcodebuild -project VisionPOC.xcodeproj -scheme VisionPOC -configuration Debug build
open build/Debug/VisionPOC.app
If LFS isn't installed, scripts/fetch-model.sh regenerates the model from Ultralytics' weights via coremltools (requires python 3.12 or 3.13 and a few minutes — installs into a /tmp venv).
On first launch, macOS will prompt for camera access. Grant it and the four-pane grid will activate.
Architecture
See docs/superpowers/specs/2026-05-15-visionpoc-design.md for the full design.
AVCaptureSession (1080p30 BGRA)
│
▼ CVMetalTextureCache (zero-copy)
source MTLTexture
│
├── live ───► passthrough fragment
├── jarvis ──► stylization fragment (cyan, scanlines, hex, beam)
├── edges ───► MPS Sobel + threshold
└── detect ─► Vision (saliency + faces + humans + animals) on ANE
│
▼
bounding-box overlay on live texture
│
▼ single render pass, 4 viewports + HUD chrome
MTKView in resizable NSWindow
Project Structure
Sources/
├── App/ NSApplication, window, menu bar
├── Capture/ CameraCapture (AVFoundation + CVMetalTextureCache)
├── Process/ ObjectDetector (Vision), EdgePass (MPS), JarvisStylePass (Metal)
├── Render/ Renderer, Pipelines, FrameContext
├── Shaders/ .metal files — Common, Live, Jarvis, Edges, Boxes
├── Text/ TextRasterizer (CoreText → MTLTexture)
├── Resources/Fonts/ Bundled Share Tech Mono + Orbitron
└── Theme.swift All tunable parameters
project.yml xcodegen project definition
The Xcode project is regenerated from project.yml, so VisionPOC.xcodeproj/ is gitignored.
Smoke Test
- macOS shows camera permission prompt → grant.
- Window opens with a 2×2 grid.
- LIVE pane shows raw feed.
- JARVIS pane shows the same feed with cyan tint, scanlines, hex-grid overlay, and a scanning beam.
- EDGES pane shows white-on-black edges that respond to motion.
- DETECT pane shows the live feed with cyan bounding boxes around objects, faces, humans, and animals in view.
- Footer FPS counter stays ≥ 55.
Tech Stack
Swift 6 · AppKit · Metal · MetalKit · MetalPerformanceShaders · AVFoundation · CoreVideo · Vision · CoreML · CoreText · xcodegen
License
MIT © James Maes
Acknowledgments
- Fonts bundled under the SIL Open Font License:
- Share Tech Mono by Carrois Apostrophe
- Orbitron by Matt McInerney
- Stylization inspired by the Jarvis / Iron Man cinematic HUD aesthetic.
- Shares the rendering pattern established in MetalPOC.