james.maesandClaude Opus 4.7
79ecd0184a
Coordinated three agents working in parallel on non-overlapping files,
with a shared `ProcessedFrame.swift` contract locking the API surface
between detector and renderer. All build clean on first integrate.
## Detector (ObjectDetector.swift)
- `VNRecognizeTextRequest` runs alongside YOLO at ~5 Hz; recognized
strings + boxes published via `currentTextDetections()`
- `VNDetectHumanBodyPoseRequest` produces a 14-segment skeleton (nose-
neck-shoulders-arms, neck-root-hips-legs); confidence-gated at 0.3
- `VNDetectHumanHandPoseRequest` produces 5 finger chains + palm ring
per hand, up to 2 hands; published as kind=.hand poses
- Body + hand published as `[PoseDetection]` via `currentPoses()`
- New `lastDetectMode: DetectMode { .yolo | .track }` so the footer can
distinguish the two latency regimes
- YOLO warmup runs one dummy inference at startup on the detector
queue — first user-visible detection no longer pays the ~200ms
model-compile latency
## Renderer (Renderer.swift + Render/ColorHash.swift)
- Box labels include 10%-bucketed confidence: "CUP 80%", "MOBILE PHONE
60%". Bucket keeps the label texture cache from exploding.
- Per-track stable color via UUID-byte → golden-ratio hash → HSV(h,
0.8, 1.0). Face labels stay cyan (calmer when JARVIS pane has
multiple labeled people in frame).
- Detection labels clamp to pane edges so boxes near the right margin
don't overflow into the next pane.
- Footer now reads "DET 28.4ms (yolo)" or "DET 2.1ms (track)" — the
flicker between the two values is informative, not a bug.
- LRU bound on `labelTextureCache` (128 entries, parallel access-order
array, drop-head-on-insert when at capacity).
- OCR text rendered as magenta boxes with truncated text labels (32
chars).
- Body pose drawn in mint, hand pose in amber; axis-aligned segment
rects with a 0.003 minimum thickness so horizontal/vertical lines
don't vanish at small viewport sizes. Joint dots at each unique
point.
- New `snapshotPNG() -> Data?` public method. Render loop piggybacks
one extra blit pass + waitUntilCompleted + CGImage encoding. <50ms
in practice, 1-second semaphore timeout if rendering stalls.
## App / Window / Capture
- Camera menu with a dynamic Devices submenu (rebuilt via
NSMenuDelegate.menuNeedsUpdate). Active device gets a checkmark,
Refresh List forces re-enumeration.
- CameraCapture.switchToDevice(_:) stops the session, swaps the input,
restarts. activeDeviceUniqueID exposed so the menu can match.
- Save Snapshot (⌘S) writes the 4-pane composite to
~/Desktop/VisionPOC-YYYY-MM-DD-HHMMSS.png
- Window-frame persistence for all three windows via UserDefaults
keys VPOC.MainWindowFrame / VPOC.SettingsWindowFrame /
VPOC.LogWindowFrame
- AVCaptureDevice.wasDisconnectedNotification → session stop + flag
set; permissionDenied returns true so the existing
"WAITING FOR CAMERA…" status path triggers without renderer changes
## Shared contract (new file)
- Sources/Process/ProcessedFrame.swift — TextDetection, PoseDetection,
PoseDetection.Segment, PoseKind (.body | .hand), DetectMode
(.yolo | .track). Sendable, value types, normalized [0..1] Vision
coords throughout.
## Theme
- Palette gains textStroke (magenta), bodyPose (mint), handPose (amber)
for the new overlays.
Note: Orbitron font and app icon (also Tier 2 items) deferred — no
clean source available for either without manual work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
VisionPOC
A proof-of-concept real-time camera vision pipeline for macOS Apple Silicon. Captures live video and renders four synchronized views in a 2×2 grid: raw passthrough, a Jarvis HUD stylization (cyan tint, scanlines, hex grid, scanning beam), an edge map (MPS Sobel), and an object-detection overlay (Vision saliency + face / human / animal recognizers running on the Apple Neural Engine).
Sibling to MetalPOC (Jarvis HUD rendering spine) and VoicePOC (voice stack). This POC proves the camera + ANE + Metal pipeline so future Jarvis "sight" features can build on a known-good foundation.
Highlights
- Zero-copy capture —
AVCaptureSession→CVPixelBuffer→MTLTextureviaCVMetalTextureCache(no CPU roundtrip) - 601-class object detection — YOLOv8x trained on Open Images V7, exported to a Vision-friendly Core ML pipeline. Recognizes everyday vocabulary like coffee cup, mug, wine glass, mobile phone, laptop, computer monitor, sunglasses, plus body parts (human face / hand / eye) and a broad household / vehicle / animal vocabulary
- Object tracking —
VNTrackObjectRequestupdates each track's box on every frame between YOLO refreshes, so detections drift smoothly with motion instead of teleporting - Face recognition — enroll a face via the menu bar (⌘E), it shows up labeled by name in the JARVIS pane on subsequent frames. Powered by Vision's image FeaturePrint on padded face crops; templates persist to
~/Library/Application Support/VisionPOC/ - Apple Neural Engine — Vision requests configured with
MLComputeUnits.all; runs on the ANE wherever possible - Single render pass, four viewports — one
MTKView, one render pass, four fragment pipelines, one HUD overlay - MPS Sobel for edge detection — GPU-native, sub-millisecond at 1080p
- Jarvis stylization shader — fragment-only Metal shader matching MetalPOC's HUD palette and fonts
- Detector decoupled from render — runs at ~20 Hz on a serial background queue; never blocks the 60 fps render thread
- Centralized tunables in
Sources/Theme.swift— palette, layout, fonts, capture preset, detector cadence, performance profile
Requirements
- macOS 26 Tahoe (Apple Silicon)
- Xcode 16+ with Metal toolchain
- xcodegen (
brew install xcodegen) - A built-in or attached camera
Build & Run
Prerequisite: Git LFS is required because the bundled YOLOv8x model (~131 MB, Open Images V7, 601 classes) is tracked via LFS. Install once on your machine: brew install git-lfs && git lfs install.
git clone https://github.com/KofTwentyTwo/VisionPOC.git
cd VisionPOC
git lfs pull # materialize the bundled model
xcodegen generate
xcodebuild -project VisionPOC.xcodeproj -scheme VisionPOC -configuration Debug build
open build/Debug/VisionPOC.app
If LFS isn't installed, scripts/fetch-model.sh regenerates the model from Ultralytics' weights via coremltools (requires python 3.12 or 3.13 and a few minutes — installs into a /tmp venv).
On first launch, macOS will prompt for camera access. Grant it and the four-pane grid will activate.
Architecture
See docs/superpowers/specs/2026-05-15-visionpoc-design.md for the full design.
AVCaptureSession (1080p30 BGRA)
│
▼ CVMetalTextureCache (zero-copy)
source MTLTexture
│
├── live ───► passthrough fragment
├── jarvis ──► stylization fragment (cyan, scanlines, hex, beam)
├── edges ───► MPS Sobel + threshold
└── detect ─► Vision (saliency + faces + humans + animals) on ANE
│
▼
bounding-box overlay on live texture
│
▼ single render pass, 4 viewports + HUD chrome
MTKView in resizable NSWindow
Project Structure
Sources/
├── App/ NSApplication, window, menu bar
├── Capture/ CameraCapture (AVFoundation + CVMetalTextureCache)
├── Process/ ObjectDetector (Vision), EdgePass (MPS), JarvisStylePass (Metal)
├── Render/ Renderer, Pipelines, FrameContext
├── Shaders/ .metal files — Common, Live, Jarvis, Edges, Boxes
├── Text/ TextRasterizer (CoreText → MTLTexture)
├── Resources/Fonts/ Bundled Share Tech Mono + Orbitron
└── Theme.swift All tunable parameters
project.yml xcodegen project definition
The Xcode project is regenerated from project.yml, so VisionPOC.xcodeproj/ is gitignored.
Smoke Test
- macOS shows camera permission prompt → grant.
- Window opens with a 2×2 grid.
- LIVE pane shows raw feed.
- JARVIS pane shows the same feed with cyan tint, scanlines, hex-grid overlay, and a scanning beam.
- EDGES pane shows white-on-black edges that respond to motion.
- DETECT pane shows the live feed with cyan bounding boxes around objects, faces, humans, and animals in view.
- Footer FPS counter stays ≥ 55.
Tech Stack
Swift 6 · AppKit · Metal · MetalKit · MetalPerformanceShaders · AVFoundation · CoreVideo · Vision · CoreML · CoreText · xcodegen
License
MIT © James Maes
Acknowledgments
- Fonts bundled under the SIL Open Font License:
- Share Tech Mono by Carrois Apostrophe
- Orbitron by Matt McInerney
- Stylization inspired by the Jarvis / Iron Man cinematic HUD aesthetic.
- Shares the rendering pattern established in MetalPOC.