james.maesandClaude Opus 4.7 3cd94535c8 History view uniform with Log Stream + proper line shader + Status panel (⌘I)
Three asks landing together.

## History view — now identical controls to Log Stream
- Oldest-first order with auto-scroll-to-bottom on append (was newest-
  first, no auto-scroll). ScrollViewReader + onChange pattern lifted
  from LogStreamView for behavioral parity.
- Toolbar now matches the Log Stream layout exactly: Category filter,
  Auto-scroll toggle, Pause toggle, filtered/total count, Clear button.
- Poll rate bumped 1 Hz → 0.10 s so new events appear instantly. Pause
  toggle freezes refresh without dropping the running event collection.

## Proper line shader (much better hand + body rendering)
- New `line_vertex` / `line_fragment` in Boxes.metal alongside the
  existing box shader. `LineUniforms` carries endpoints + thickness +
  soft-edge falloff + a dot-mode flag.
- Line mode: emits a rotated thin quad oriented along (end - start)
  with width = thickness. Soft-edge antialiasing via smoothstep on the
  signed perpendicular UV makes diagonal skeleton segments read as
  actual strokes instead of axis-aligned bounding boxes.
- Dot mode: emits a square around `start` and the fragment discards
  outside the unit-circle — produces filled disks for joint markers
  with sub-pixel-clean soft edges.
- Pipelines gains a `lines` blended pipeline.
- Renderer.drawPoses() rewritten to use the line pipeline — one draw
  per segment, one draw per joint, all visually consistent regardless
  of segment angle. Theme.Palette.bodyPose (mint) for body, handPose
  (amber) for hand.

## Status panel (⌘I)
- New `CurrentStateStore` singleton subscribes to the event bus and
  retains the most recent value per concept: expression (smile / frown /
  neutral_face), activity (standing / sitting / etc.), per-chirality
  finger counts, plus a small ring of recent non-finger non-expression
  gestures.
- New StatusView (SwiftUI) reads three sources at 4 Hz: the live
  ObjectDetector for FPS + DET timing + objects + faces, the
  CurrentStateStore for hands/expression/activity/recent-gestures, and
  the HistoryStore for an 8-entry tail of recent events.
- New StatusWindowController hosts the SwiftUI view as a dark-aqua
  floating window with frame persistence (VPOC.StatusWindowFrame).
- AppDelegate `openStatus(_:)` builds a DetectorSnapshot from the live
  windowController + detector each tick, partitioning detections into
  face entries (anything labeled FACE or matching an enrolled name)
  vs object entries. Object entries carry the same per-track color the
  JARVIS pane uses (via ColorHash.colorFor) so the panel's dots match
  the on-screen box colors.
- Renderer exposes `smoothedFPSForReadout` so the panel can show the
  same FPS the footer does without reaching into private state.
- FingerCounter now records its count per chirality via
  CurrentStateStore.recordFingers (in addition to the existing bus
  emit), using VNHumanHandPoseObservation.chirality. The panel
  displays "Left: 3 fingers / Right: 5 fingers" correctly.

Menu structure now:
- App menu: Settings (⌘,) · Log Stream (⌘L) · Status (⌘I) · Save
  Snapshot (⌘S) · Start/Stop Recording (⇧⌘R) · Quit
- Faces menu: Enroll (⌘E) · Forget (⇧⌘E) · History (⌘H)
- Camera menu: Devices ▶ · Refresh · Privacy Mode (⇧⌘P)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-16 17:46:27 -05:00

VisionPOC

A proof-of-concept real-time camera vision pipeline for macOS Apple Silicon. Captures live video and renders four synchronized views in a 2×2 grid: raw passthrough, a Jarvis HUD stylization (cyan tint, scanlines, hex grid, scanning beam), an edge map (MPS Sobel), and an object-detection overlay (Vision saliency + face / human / animal recognizers running on the Apple Neural Engine).

Sibling to MetalPOC (Jarvis HUD rendering spine) and VoicePOC (voice stack). This POC proves the camera + ANE + Metal pipeline so future Jarvis "sight" features can build on a known-good foundation.

Highlights

  • Zero-copy capture — AVCaptureSession → CVPixelBuffer → MTLTexture via CVMetalTextureCache (no CPU roundtrip)
  • 601-class object detection — YOLOv8x trained on Open Images V7, exported to a Vision-friendly Core ML pipeline. Recognizes everyday vocabulary like coffee cup, mug, wine glass, mobile phone, laptop, computer monitor, sunglasses, plus body parts (human face / hand / eye) and a broad household / vehicle / animal vocabulary
  • Object tracking — VNTrackObjectRequest updates each track's box on every frame between YOLO refreshes, so detections drift smoothly with motion instead of teleporting
  • Face recognition — enroll a face via the menu bar (⌘E), it shows up labeled by name in the JARVIS pane on subsequent frames. Powered by Vision's image FeaturePrint on padded face crops; templates persist to ~/Library/Application Support/VisionPOC/
  • Apple Neural Engine — Vision requests configured with MLComputeUnits.all; runs on the ANE wherever possible
  • Single render pass, four viewports — one MTKView, one render pass, four fragment pipelines, one HUD overlay
  • MPS Sobel for edge detection — GPU-native, sub-millisecond at 1080p
  • Jarvis stylization shader — fragment-only Metal shader matching MetalPOC's HUD palette and fonts
  • Detector decoupled from render — runs at ~20 Hz on a serial background queue; never blocks the 60 fps render thread
  • Centralized tunables in Sources/Theme.swift — palette, layout, fonts, capture preset, detector cadence, performance profile

Requirements

  • macOS 26 Tahoe (Apple Silicon)
  • Xcode 16+ with Metal toolchain
  • xcodegen (brew install xcodegen)
  • A built-in or attached camera

Build & Run

Prerequisite: Git LFS is required because the bundled YOLOv8x model (~131 MB, Open Images V7, 601 classes) is tracked via LFS. Install once on your machine: brew install git-lfs && git lfs install.

git clone https://github.com/KofTwentyTwo/VisionPOC.git
cd VisionPOC
git lfs pull                                  # materialize the bundled model
xcodegen generate
xcodebuild -project VisionPOC.xcodeproj -scheme VisionPOC -configuration Debug build
open build/Debug/VisionPOC.app

If LFS isn't installed, scripts/fetch-model.sh regenerates the model from Ultralytics' weights via coremltools (requires python 3.12 or 3.13 and a few minutes — installs into a /tmp venv).

On first launch, macOS will prompt for camera access. Grant it and the four-pane grid will activate.

Architecture

See docs/superpowers/specs/2026-05-15-visionpoc-design.md for the full design.

AVCaptureSession (1080p30 BGRA)
        │
        ▼  CVMetalTextureCache (zero-copy)
   source MTLTexture
        │
        ├── live  ───► passthrough fragment
        ├── jarvis ──► stylization fragment (cyan, scanlines, hex, beam)
        ├── edges ───► MPS Sobel + threshold
        └── detect ─► Vision (saliency + faces + humans + animals) on ANE
                        │
                        ▼
                  bounding-box overlay on live texture
        │
        ▼  single render pass, 4 viewports + HUD chrome
   MTKView in resizable NSWindow

Project Structure

Sources/
├── App/                  NSApplication, window, menu bar
├── Capture/              CameraCapture (AVFoundation + CVMetalTextureCache)
├── Process/              ObjectDetector (Vision), EdgePass (MPS), JarvisStylePass (Metal)
├── Render/               Renderer, Pipelines, FrameContext
├── Shaders/              .metal files — Common, Live, Jarvis, Edges, Boxes
├── Text/                 TextRasterizer (CoreText → MTLTexture)
├── Resources/Fonts/      Bundled Share Tech Mono + Orbitron
└── Theme.swift           All tunable parameters
project.yml               xcodegen project definition

The Xcode project is regenerated from project.yml, so VisionPOC.xcodeproj/ is gitignored.

Smoke Test

  1. macOS shows camera permission prompt → grant.
  2. Window opens with a 2×2 grid.
  3. LIVE pane shows raw feed.
  4. JARVIS pane shows the same feed with cyan tint, scanlines, hex-grid overlay, and a scanning beam.
  5. EDGES pane shows white-on-black edges that respond to motion.
  6. DETECT pane shows the live feed with cyan bounding boxes around objects, faces, humans, and animals in view.
  7. Footer FPS counter stays ≥ 55.

Tech Stack

Swift 6 · AppKit · Metal · MetalKit · MetalPerformanceShaders · AVFoundation · CoreVideo · Vision · CoreML · CoreText · xcodegen

License

MIT © James Maes

Acknowledgments

S
Description
Real-time camera vision POC: live + Jarvis HUD + edge map + object detection (Apple Silicon, Metal, Vision/ANE)
Readme MIT
369 MiB
0 Stars 2 Watchers 0 Forks
Languages
Swift 92.5%
Metal 6.5%
Shell 0.6%
Python 0.4%