79ecd0184ada161d262d113a95751fca1b499d2a
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  james.maesandClaude Opus 4.7
|
79ecd0184a |
Tier 1 + Tier 2 polish: 3 parallel agents — overlays, snapshots, menus
Coordinated three agents working in parallel on non-overlapping files,
with a shared `ProcessedFrame.swift` contract locking the API surface
between detector and renderer. All build clean on first integrate.
## Detector (ObjectDetector.swift)
- `VNRecognizeTextRequest` runs alongside YOLO at ~5 Hz; recognized
strings + boxes published via `currentTextDetections()`
- `VNDetectHumanBodyPoseRequest` produces a 14-segment skeleton (nose-
neck-shoulders-arms, neck-root-hips-legs); confidence-gated at 0.3
- `VNDetectHumanHandPoseRequest` produces 5 finger chains + palm ring
per hand, up to 2 hands; published as kind=.hand poses
- Body + hand published as `[PoseDetection]` via `currentPoses()`
- New `lastDetectMode: DetectMode { .yolo | .track }` so the footer can
distinguish the two latency regimes
- YOLO warmup runs one dummy inference at startup on the detector
queue — first user-visible detection no longer pays the ~200ms
model-compile latency
## Renderer (Renderer.swift + Render/ColorHash.swift)
- Box labels include 10%-bucketed confidence: "CUP 80%", "MOBILE PHONE
60%". Bucket keeps the label texture cache from exploding.
- Per-track stable color via UUID-byte → golden-ratio hash → HSV(h,
0.8, 1.0). Face labels stay cyan (calmer when JARVIS pane has
multiple labeled people in frame).
- Detection labels clamp to pane edges so boxes near the right margin
don't overflow into the next pane.
- Footer now reads "DET 28.4ms (yolo)" or "DET 2.1ms (track)" — the
flicker between the two values is informative, not a bug.
- LRU bound on `labelTextureCache` (128 entries, parallel access-order
array, drop-head-on-insert when at capacity).
- OCR text rendered as magenta boxes with truncated text labels (32
chars).
- Body pose drawn in mint, hand pose in amber; axis-aligned segment
rects with a 0.003 minimum thickness so horizontal/vertical lines
don't vanish at small viewport sizes. Joint dots at each unique
point.
- New `snapshotPNG() -> Data?` public method. Render loop piggybacks
one extra blit pass + waitUntilCompleted + CGImage encoding. <50ms
in practice, 1-second semaphore timeout if rendering stalls.
## App / Window / Capture
- Camera menu with a dynamic Devices submenu (rebuilt via
NSMenuDelegate.menuNeedsUpdate). Active device gets a checkmark,
Refresh List forces re-enumeration.
- CameraCapture.switchToDevice(_:) stops the session, swaps the input,
restarts. activeDeviceUniqueID exposed so the menu can match.
- Save Snapshot (⌘S) writes the 4-pane composite to
~/Desktop/VisionPOC-YYYY-MM-DD-HHMMSS.png
- Window-frame persistence for all three windows via UserDefaults
keys VPOC.MainWindowFrame / VPOC.SettingsWindowFrame /
VPOC.LogWindowFrame
- AVCaptureDevice.wasDisconnectedNotification → session stop + flag
set; permissionDenied returns true so the existing
"WAITING FOR CAMERA…" status path triggers without renderer changes
## Shared contract (new file)
- Sources/Process/ProcessedFrame.swift — TextDetection, PoseDetection,
PoseDetection.Segment, PoseKind (.body | .hand), DetectMode
(.yolo | .track). Sendable, value types, normalized [0..1] Vision
coords throughout.
## Theme
- Palette gains textStroke (magenta), bodyPose (mint), handPose (amber)
for the new overlays.
Note: Orbitron font and app icon (also Tier 2 items) deferred — no
clean source available for either without manual work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
james.maesandClaude Opus 4.7
|
0d23d7ede9 |
Add a live Log Stream window (⌘L) and emit events from every subsystem
Built a Jarvis-style scrolling log viewer that surfaces what each subsystem is doing in real time. The buffer is a thread-safe ring of the last 1,000 entries; the SwiftUI viewer polls at 10 Hz, only re-renders when the buffer version bumps, and supports level / source filters, pause, auto-scroll, and clear. Components - Sources/App/LogStream.swift — singleton with Level (DBG/INF/WRN/ERR) and Source (APP / CAM / DETECT / TRACK / FACE / RENDER / VPOC) taxonomies, an os_unfair_lock-guarded buffer, and stdout mirroring so `log stream --process VisionPOC` shows the same feed in Terminal. - Sources/App/LogStreamView.swift — SwiftUI viewer with toolbar (level menu, source menu, auto-scroll, pause, clear, filtered/total counts) and a monospaced row-per-entry list that auto-scrolls to the bottom when new entries arrive. - Sources/App/LogStreamWindowController.swift — separate dark-aqua floating window so the log doesn't compete with the main 4-pane HUD for real estate. - AppDelegate gains a ⌘L "Log Stream…" menu item and a held reference to the controller (lazy-instantiated on first open). Wiring - AppDelegate: launch + ready - CameraCapture: device selection, session start, permission denied, enumeration failures - ObjectDetector: model load (success or fallback), 1 Hz detector stats (yolo vs track mode, ms, track count + label list), per-track add and per-track lose - ObjectDetector face match: when a face is recognized via FaceRegistry the distance is logged at DBG level - FaceRegistry: load on init, enroll (+prints, total), forget Implementation notes - The viewer is not @Observable-bound to LogStream so background- thread log emit calls don't pay a MainActor hop. Instead the view polls .version every 100ms and reloads on bump — bursts up to ~100 entries per tick render fine and the buffer absorbs writes from any rate the os_unfair_lock can sustain (~10 M/s on M-series). - LogStream.shared.log() can be called from any thread/queue. Stdout is also written from the same call so terminal tail works. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
ba1ecac72a |
Add a live Settings panel for runtime tuning
⌘, or VisionPOC → Settings… opens a SwiftUI panel with sliders for every value that's worth tuning at runtime: YOLO frequency, detection min confidence, tracker age/confidence/IoU, face match threshold, edge Sobel threshold, ASCII column count. Reset-to-defaults button at the bottom. Plumbing - Theme.Performance's runtime-tunable values converted from `static let` to `nonisolated(unsafe) static var`. Acknowledges the threading model: cooperative single-writer (MainActor UI) and many readers (detector queue, render queue). All values are word-sized primitives, atomic on ARM at the hardware level — no data races in practice. - New @Observable @MainActor `TunableSettings` mirrors those statics for SwiftUI binding; each property's didSet writes back through so detector and renderer code paths see the change on the next frame. - New `SettingsWindowController` hosts the panel as a separate floating window with a dark-aqua appearance to match the HUD aesthetic. - FaceRegistry.matchThreshold becomes a thin getter/setter passthrough to Theme.Performance.faceMatchThreshold so the face slider drives it directly. UI niceties - Sections grouped by concern (Detection, Tracking, Faces, Edges, ASCII) - Each slider shows its current value to two decimal places (or the natural format for Hz / seconds / integer counts) - Face section includes an in-context caption explaining the typical distance range so the user knows where to aim the slider Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
878fc9aece |
Add face recognition with enrollment via the menu bar
Vision's public API doesn't expose a face-specific embedding model, so we use VNGenerateImageFeaturePrintRequest with each face's bounding box as the regionOfInterest. The general-purpose image FeaturePrint applied to a padded face crop produces distance metrics that separate "same person" from "different person" reliably for the small number of enrolled identities a POC needs. Components - Sources/Process/FaceRegistry.swift — thread-safe enrollment store. Persists per-person VNFeaturePrintObservation arrays to ~/Library/Application Support/VisionPOC/faces.json via NSKeyedArchiver. bestMatch(for:) walks every stored template and returns the closest name + distance under matchThreshold. - ObjectDetector now generates a feature print per detected face on each YOLO refresh, matches against the registry, and labels the detection with the matched name (uppercased) instead of "FACE" when it recognizes someone. Enrollment flow - New Faces menu in the menu bar. - ⌘E "Enroll Face…" prompts for a name, then captures feature prints from the next 5 frames that contain the largest face in view, saves them, and shows a confirmation alert. - ⌘⇧E "Forget Face…" lists enrolled names in a popup and deletes the selected entry from disk. - ObjectDetector.captureFeaturePrints(count:completion:) drives the capture: the next N detect-pass frames that see a face have the largest face's print collected, then the completion fires once N prints are in hand. Threading - The detector completion runs on the detector queue and feeds the prints straight into FaceRegistry.enroll (which is thread-safe). Only the count is sent back to MainActor for the confirmation dialog, sidestepping the Sendable issues with passing VNFeaturePrintObservation across actor boundaries. Threshold note - FaceRegistry.matchThreshold defaults to 18.0 — tuned for the image- level FeaturePrint distances we see on face crops. Apple's published ~0.6 threshold targets face-specific embedding models that aren't exposed publicly. The threshold is a `var` so it's tunable at runtime if real-world testing wants it tighter or looser. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
0c6911bcea |
Add VNTrackObjectRequest-based tracking between YOLO refreshes
ObjectDetector now interleaves real detection with cheap tracking so bounding boxes follow objects smoothly rather than teleporting whenever a new YOLO inference completes. Pipeline - YOLO + face + animal detectors run at ~5 Hz (Theme.yoloDetectionHz), produce ground-truth detections. - VNTrackObjectRequest runs on every submitted frame between those detections, updating each active track's bounding box from the latest pixel buffer. Tracker cost is ~1ms per object. - New YOLO detections IoU-match against existing tracks (threshold 0.4, same-label bias of +0.05). Matches refresh the track; unmatched detections bootstrap fresh tracks. - Tracks expire after Theme.trackMaxAgeSeconds (0.6s) without a YOLO refresh, which covers brief occlusions but drops ghosts when objects leave the scene. - Tracker confidence below Theme.trackerMinConfidence (0.3) leaves the previous observation in place rather than degrading the box, since Vision's tracker can momentarily lose objects through motion blur or hand occlusion. Detection - Detection gains a stable UUID `trackId` so downstream consumers (a future Jarvis brain, for instance) can reason about object identity across frames without their own re-association logic. - Renderer doesn't yet use trackId — it would unlock per-object stable colors, motion trails, and a "seen recently" event ticker if surfaced. Theme knobs (Performance namespace) - yoloDetectionHz: 5.0 - trackMaxAgeSeconds: 0.6 - trackerMinConfidence: 0.3 - trackMatchIoU: 0.4 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
3c5a171d34 |
Add YOLOv3 object detection (80 COCO classes) via Git LFS
Drops Apple's full YOLOv3 (~248 MB) into Sources/Resources/Models/ as the primary object detector. Xcode auto-compiles the .mlmodel into a .mlmodelc inside the app bundle; the model is a Vision-friendly pipeline so VNCoreMLRequest returns VNRecognizedObjectObservation directly — no manual NMS or output decoding required. Detection pipeline - YOLOv3 is the primary detector, returning bounding boxes labeled with COCO classes: person, bicycle, car, ..., bottle, wine glass, cup, fork, knife, spoon, bowl, ... laptop, mouse, remote, keyboard, cell phone, ... book, clock, vase. 80 classes total — covers the "is that a coffee mug" question and many others. - VNDetectFaceRectanglesRequest stays as a supplementary FACE labeler (more precise than YOLO's "person" for the face region). - VNRecognizeAnimalsRequest stays for finer-grained cat/dog labels. - Saliency and human-rectangle requests removed — YOLO supersedes them. - Per-label NMS with IoU>0.5 inside filterAndMerge drops same-label overlaps. Cross-label overlaps (e.g. person + face) are kept since they carry different information. - Min-confidence filter (Theme.Performance.detectionMinConfidence, 0.30) prunes weak guesses. Distribution - Model is tracked via Git LFS (.gitattributes) — it exceeds GitHub's 100 MB per-file limit. - README documents the `git lfs install && git lfs pull` prerequisite. - scripts/fetch-model.sh remains as a no-LFS fallback that downloads the same model from Apple's Core ML catalog. Theme knobs - Performance.detectionMinConfidence (default 0.30) - Performance.detectorModelName (default "YOLOv3") Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
d4c358d943 |
Fix HUD text rotation, fix edge threshold, merge boxes into JARVIS, add ASCII pane
Bug fixes - text_fragment now v-flips its sample. TextRasterizer writes glyphs via a y-flipped CGContext (CoreText y-up → image-y-down), so memory row 0 holds the visual top of the text. quad_vertex already flips v for camera frames, so HUD text needed its own compensating flip in the fragment. - EdgePass was creating the threshold compute pipeline but never binding the threshold value to buffer(0). The kernel read garbage memory which usually evaluated as ~0, letting almost every fragment pass and producing the inverted look in the EDGES pane. Now binds Theme.Performance.edgeThreshold (default 0.18) explicitly. - Footer was anchored to y=2 (top of drawable). Moved to drawableSize.height minus footer height so it sits along the bottom edge where it belongs. Pane changes (per user request) - Pane 1 (JARVIS) now combines the stylization with the detection overlay: jarvis fragment → bounding-box strokes → per-box text labels. Labels are cached by string in labelTextureCache so a typical scene only rasterizes half a dozen unique strings ever. - Pane 3 was DETECT; now ASCII. Renders the live source through a new ascii_fragment that quantizes the image into a grid of monospaced glyphs picked from a pre-rasterized dark→bright atlas (` .:-=+*#%@`). New components - Sources/Shaders/Ascii.metal — ascii_fragment with AsciiUniforms (gridSize, atlasCharCount, tintColor). - Sources/Process/AsciiPass.swift — builds the glyph atlas at startup via CTLineDraw into a CGContext, uploads to a BGRA MTLTexture, and produces uniforms sized to keep glyph cells roughly square at any pane aspect. - Pipelines gains an ascii render state (quad_vertex + ascii_fragment). - Theme: paneLabels now ["LIVE", "JARVIS", "EDGES", "ASCII"], plus Performance.edgeThreshold and Performance.asciiColumns knobs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
f6601fa593 |
Implement camera capture, processing, rendering, and Jarvis HUD
End-to-end VisionPOC pipeline: Capture (Sources/Capture/CameraCapture.swift) - AVCaptureSession on a dedicated queue, BGRA @ 1080p - Zero-copy CVPixelBuffer -> MTLTexture via CVMetalTextureCache - os_unfair_lock around the latest-frame snapshot - Permission flow with denied state surfaced to the renderer Processing - ObjectDetector: VNGenerateObjectnessBasedSaliencyImageRequest is the primary all-objects path; face, human, and animal detectors layer in as labels. IOU-merge keeps labels and drops duplicate generic boxes. Throttled to Theme.Performance.detectorTargetHz on a serial queue so the renderer never blocks. Runs on the ANE via Vision's default routing. - EdgePass: MPSImageSobel into an intermediate texture, then an edge_threshold compute kernel binarizes to crisp white-on-black. - JarvisStylePass: builds the per-frame uniform struct (time, beam phase, scanline density, hex grid scale, tint). Render - Single MTKView render pass, four viewports, one HUD overlay - Live passthrough, Jarvis stylization, edge map, and live-plus-boxes - Corner brackets and pane labels composited with the boxes/text pipelines - Footer reports FPS and detector latency, rebuilt every 10 frames Shaders - quad_vertex emits a v-flipped fullscreen quad (origin-top-left camera textures display upright) - jarvis_fragment: luma -> cyan tint, scanlines, hex grid, scanning beam, vignette, light chromatic edge ghost - edges_fragment: cyan-tinted edge display with nearest sampling - box_fragment: dual-mode (hollow stroke for detections, solid fill for HUD chrome) selected by BoxUniforms.params.x - text_fragment: glyph texture multiplied by tint at fragment buffer 0 App - @main AppDelegate, regular activation policy, ⌘Q menu - Bundled-font registration via CTFontManagerRegisterFontsForURL - MainWindowController owns the MTKView and wires the pipeline Bumped deployment target to macOS 26 to match the installed SDK. Build verified: xcodegen generate && xcodebuild ... && open the .app — launches, render loop runs, camera permission prompt fires on first frame. Known cosmetic issue: Orbitron-Bold.ttf inherited from MetalPOC is a GitHub HTML page rather than a real font; Theme.Font falls back to Helvetica Neue. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |