main
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  james.maesandClaude Opus 4.7
|
7e7e3e8f75 |
Settings: configurable snapshot + recording output locations
New `OutputLocations` helper stores per-app folder choices as security- scoped bookmarks in UserDefaults under VPOC.SnapshotDirectoryBookmark and VPOC.RecordingDirectoryBookmark. Both default to ~/Desktop so existing behavior is unchanged until the user picks a different folder. Settings panel gains an "Output Locations" section with two rows — Snapshots and Recordings — each showing the current path (home-relative, e.g. "~/Movies/VisionPOC") and a Choose… button that runs NSOpenPanel for picking a directory. Selecting a folder stores the bookmark and refreshes the displayed path. Bookmarks are tried with .withSecurityScope first (forward-compatible with a future sandboxed build) and fall back to a plain bookmark when the security-scoped form isn't available — so this works in both the current unsigned dev build and any future signed/sandboxed build. Both Recorder.makeOutputURL and MainWindowController.saveSnapshot now read their destination from OutputLocations instead of hardcoding .desktopDirectory. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
79ecd0184a |
Tier 1 + Tier 2 polish: 3 parallel agents — overlays, snapshots, menus
Coordinated three agents working in parallel on non-overlapping files,
with a shared `ProcessedFrame.swift` contract locking the API surface
between detector and renderer. All build clean on first integrate.
## Detector (ObjectDetector.swift)
- `VNRecognizeTextRequest` runs alongside YOLO at ~5 Hz; recognized
strings + boxes published via `currentTextDetections()`
- `VNDetectHumanBodyPoseRequest` produces a 14-segment skeleton (nose-
neck-shoulders-arms, neck-root-hips-legs); confidence-gated at 0.3
- `VNDetectHumanHandPoseRequest` produces 5 finger chains + palm ring
per hand, up to 2 hands; published as kind=.hand poses
- Body + hand published as `[PoseDetection]` via `currentPoses()`
- New `lastDetectMode: DetectMode { .yolo | .track }` so the footer can
distinguish the two latency regimes
- YOLO warmup runs one dummy inference at startup on the detector
queue — first user-visible detection no longer pays the ~200ms
model-compile latency
## Renderer (Renderer.swift + Render/ColorHash.swift)
- Box labels include 10%-bucketed confidence: "CUP 80%", "MOBILE PHONE
60%". Bucket keeps the label texture cache from exploding.
- Per-track stable color via UUID-byte → golden-ratio hash → HSV(h,
0.8, 1.0). Face labels stay cyan (calmer when JARVIS pane has
multiple labeled people in frame).
- Detection labels clamp to pane edges so boxes near the right margin
don't overflow into the next pane.
- Footer now reads "DET 28.4ms (yolo)" or "DET 2.1ms (track)" — the
flicker between the two values is informative, not a bug.
- LRU bound on `labelTextureCache` (128 entries, parallel access-order
array, drop-head-on-insert when at capacity).
- OCR text rendered as magenta boxes with truncated text labels (32
chars).
- Body pose drawn in mint, hand pose in amber; axis-aligned segment
rects with a 0.003 minimum thickness so horizontal/vertical lines
don't vanish at small viewport sizes. Joint dots at each unique
point.
- New `snapshotPNG() -> Data?` public method. Render loop piggybacks
one extra blit pass + waitUntilCompleted + CGImage encoding. <50ms
in practice, 1-second semaphore timeout if rendering stalls.
## App / Window / Capture
- Camera menu with a dynamic Devices submenu (rebuilt via
NSMenuDelegate.menuNeedsUpdate). Active device gets a checkmark,
Refresh List forces re-enumeration.
- CameraCapture.switchToDevice(_:) stops the session, swaps the input,
restarts. activeDeviceUniqueID exposed so the menu can match.
- Save Snapshot (⌘S) writes the 4-pane composite to
~/Desktop/VisionPOC-YYYY-MM-DD-HHMMSS.png
- Window-frame persistence for all three windows via UserDefaults
keys VPOC.MainWindowFrame / VPOC.SettingsWindowFrame /
VPOC.LogWindowFrame
- AVCaptureDevice.wasDisconnectedNotification → session stop + flag
set; permissionDenied returns true so the existing
"WAITING FOR CAMERA…" status path triggers without renderer changes
## Shared contract (new file)
- Sources/Process/ProcessedFrame.swift — TextDetection, PoseDetection,
PoseDetection.Segment, PoseKind (.body | .hand), DetectMode
(.yolo | .track). Sendable, value types, normalized [0..1] Vision
coords throughout.
## Theme
- Palette gains textStroke (magenta), bodyPose (mint), handPose (amber)
for the new overlays.
Note: Orbitron font and app icon (also Tier 2 items) deferred — no
clean source available for either without manual work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
james.maesandClaude Opus 4.7
|
878fc9aece |
Add face recognition with enrollment via the menu bar
Vision's public API doesn't expose a face-specific embedding model, so we use VNGenerateImageFeaturePrintRequest with each face's bounding box as the regionOfInterest. The general-purpose image FeaturePrint applied to a padded face crop produces distance metrics that separate "same person" from "different person" reliably for the small number of enrolled identities a POC needs. Components - Sources/Process/FaceRegistry.swift — thread-safe enrollment store. Persists per-person VNFeaturePrintObservation arrays to ~/Library/Application Support/VisionPOC/faces.json via NSKeyedArchiver. bestMatch(for:) walks every stored template and returns the closest name + distance under matchThreshold. - ObjectDetector now generates a feature print per detected face on each YOLO refresh, matches against the registry, and labels the detection with the matched name (uppercased) instead of "FACE" when it recognizes someone. Enrollment flow - New Faces menu in the menu bar. - ⌘E "Enroll Face…" prompts for a name, then captures feature prints from the next 5 frames that contain the largest face in view, saves them, and shows a confirmation alert. - ⌘⇧E "Forget Face…" lists enrolled names in a popup and deletes the selected entry from disk. - ObjectDetector.captureFeaturePrints(count:completion:) drives the capture: the next N detect-pass frames that see a face have the largest face's print collected, then the completion fires once N prints are in hand. Threading - The detector completion runs on the detector queue and feeds the prints straight into FaceRegistry.enroll (which is thread-safe). Only the count is sent back to MainActor for the confirmation dialog, sidestepping the Sendable issues with passing VNFeaturePrintObservation across actor boundaries. Threshold note - FaceRegistry.matchThreshold defaults to 18.0 — tuned for the image- level FeaturePrint distances we see on face crops. Apple's published ~0.6 threshold targets face-specific embedding models that aren't exposed publicly. The threshold is a `var` so it's tunable at runtime if real-world testing wants it tighter or looser. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
d4c358d943 |
Fix HUD text rotation, fix edge threshold, merge boxes into JARVIS, add ASCII pane
Bug fixes - text_fragment now v-flips its sample. TextRasterizer writes glyphs via a y-flipped CGContext (CoreText y-up → image-y-down), so memory row 0 holds the visual top of the text. quad_vertex already flips v for camera frames, so HUD text needed its own compensating flip in the fragment. - EdgePass was creating the threshold compute pipeline but never binding the threshold value to buffer(0). The kernel read garbage memory which usually evaluated as ~0, letting almost every fragment pass and producing the inverted look in the EDGES pane. Now binds Theme.Performance.edgeThreshold (default 0.18) explicitly. - Footer was anchored to y=2 (top of drawable). Moved to drawableSize.height minus footer height so it sits along the bottom edge where it belongs. Pane changes (per user request) - Pane 1 (JARVIS) now combines the stylization with the detection overlay: jarvis fragment → bounding-box strokes → per-box text labels. Labels are cached by string in labelTextureCache so a typical scene only rasterizes half a dozen unique strings ever. - Pane 3 was DETECT; now ASCII. Renders the live source through a new ascii_fragment that quantizes the image into a grid of monospaced glyphs picked from a pre-rasterized dark→bright atlas (` .:-=+*#%@`). New components - Sources/Shaders/Ascii.metal — ascii_fragment with AsciiUniforms (gridSize, atlasCharCount, tintColor). - Sources/Process/AsciiPass.swift — builds the glyph atlas at startup via CTLineDraw into a CGContext, uploads to a BGRA MTLTexture, and produces uniforms sized to keep glyph cells roughly square at any pane aspect. - Pipelines gains an ascii render state (quad_vertex + ascii_fragment). - Theme: paneLabels now ["LIVE", "JARVIS", "EDGES", "ASCII"], plus Performance.edgeThreshold and Performance.asciiColumns knobs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
f6601fa593 |
Implement camera capture, processing, rendering, and Jarvis HUD
End-to-end VisionPOC pipeline: Capture (Sources/Capture/CameraCapture.swift) - AVCaptureSession on a dedicated queue, BGRA @ 1080p - Zero-copy CVPixelBuffer -> MTLTexture via CVMetalTextureCache - os_unfair_lock around the latest-frame snapshot - Permission flow with denied state surfaced to the renderer Processing - ObjectDetector: VNGenerateObjectnessBasedSaliencyImageRequest is the primary all-objects path; face, human, and animal detectors layer in as labels. IOU-merge keeps labels and drops duplicate generic boxes. Throttled to Theme.Performance.detectorTargetHz on a serial queue so the renderer never blocks. Runs on the ANE via Vision's default routing. - EdgePass: MPSImageSobel into an intermediate texture, then an edge_threshold compute kernel binarizes to crisp white-on-black. - JarvisStylePass: builds the per-frame uniform struct (time, beam phase, scanline density, hex grid scale, tint). Render - Single MTKView render pass, four viewports, one HUD overlay - Live passthrough, Jarvis stylization, edge map, and live-plus-boxes - Corner brackets and pane labels composited with the boxes/text pipelines - Footer reports FPS and detector latency, rebuilt every 10 frames Shaders - quad_vertex emits a v-flipped fullscreen quad (origin-top-left camera textures display upright) - jarvis_fragment: luma -> cyan tint, scanlines, hex grid, scanning beam, vignette, light chromatic edge ghost - edges_fragment: cyan-tinted edge display with nearest sampling - box_fragment: dual-mode (hollow stroke for detections, solid fill for HUD chrome) selected by BoxUniforms.params.x - text_fragment: glyph texture multiplied by tint at fragment buffer 0 App - @main AppDelegate, regular activation policy, ⌘Q menu - Bundled-font registration via CTFontManagerRegisterFontsForURL - MainWindowController owns the MTKView and wires the pipeline Bumped deployment target to macOS 26 to match the installed SDK. Build verified: xcodegen generate && xcodebuild ... && open the .app — launches, render loop runs, camera permission prompt fires on first frame. Known cosmetic issue: Orbitron-Bold.ttf inherited from MetalPOC is a GitHub HTML page rather than a real font; Theme.Font falls back to Helvetica Neue. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |