main
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  james.maesandClaude Opus 4.7
|
7e7e3e8f75 |
Settings: configurable snapshot + recording output locations
New `OutputLocations` helper stores per-app folder choices as security- scoped bookmarks in UserDefaults under VPOC.SnapshotDirectoryBookmark and VPOC.RecordingDirectoryBookmark. Both default to ~/Desktop so existing behavior is unchanged until the user picks a different folder. Settings panel gains an "Output Locations" section with two rows — Snapshots and Recordings — each showing the current path (home-relative, e.g. "~/Movies/VisionPOC") and a Choose… button that runs NSOpenPanel for picking a directory. Selecting a folder stores the bookmark and refreshes the displayed path. Bookmarks are tried with .withSecurityScope first (forward-compatible with a future sandboxed build) and fall back to a plain bookmark when the security-scoped form isn't available — so this works in both the current unsigned dev build and any future signed/sandboxed build. Both Recorder.makeOutputURL and MainWindowController.saveSnapshot now read their destination from OutputLocations instead of hardcoding .desktopDirectory. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
3cd94535c8 |
History view uniform with Log Stream + proper line shader + Status panel (⌘I)
Three asks landing together. ## History view — now identical controls to Log Stream - Oldest-first order with auto-scroll-to-bottom on append (was newest- first, no auto-scroll). ScrollViewReader + onChange pattern lifted from LogStreamView for behavioral parity. - Toolbar now matches the Log Stream layout exactly: Category filter, Auto-scroll toggle, Pause toggle, filtered/total count, Clear button. - Poll rate bumped 1 Hz → 0.10 s so new events appear instantly. Pause toggle freezes refresh without dropping the running event collection. ## Proper line shader (much better hand + body rendering) - New `line_vertex` / `line_fragment` in Boxes.metal alongside the existing box shader. `LineUniforms` carries endpoints + thickness + soft-edge falloff + a dot-mode flag. - Line mode: emits a rotated thin quad oriented along (end - start) with width = thickness. Soft-edge antialiasing via smoothstep on the signed perpendicular UV makes diagonal skeleton segments read as actual strokes instead of axis-aligned bounding boxes. - Dot mode: emits a square around `start` and the fragment discards outside the unit-circle — produces filled disks for joint markers with sub-pixel-clean soft edges. - Pipelines gains a `lines` blended pipeline. - Renderer.drawPoses() rewritten to use the line pipeline — one draw per segment, one draw per joint, all visually consistent regardless of segment angle. Theme.Palette.bodyPose (mint) for body, handPose (amber) for hand. ## Status panel (⌘I) - New `CurrentStateStore` singleton subscribes to the event bus and retains the most recent value per concept: expression (smile / frown / neutral_face), activity (standing / sitting / etc.), per-chirality finger counts, plus a small ring of recent non-finger non-expression gestures. - New StatusView (SwiftUI) reads three sources at 4 Hz: the live ObjectDetector for FPS + DET timing + objects + faces, the CurrentStateStore for hands/expression/activity/recent-gestures, and the HistoryStore for an 8-entry tail of recent events. - New StatusWindowController hosts the SwiftUI view as a dark-aqua floating window with frame persistence (VPOC.StatusWindowFrame). - AppDelegate `openStatus(_:)` builds a DetectorSnapshot from the live windowController + detector each tick, partitioning detections into face entries (anything labeled FACE or matching an enrolled name) vs object entries. Object entries carry the same per-track color the JARVIS pane uses (via ColorHash.colorFor) so the panel's dots match the on-screen box colors. - Renderer exposes `smoothedFPSForReadout` so the panel can show the same FPS the footer does without reaching into private state. - FingerCounter now records its count per chirality via CurrentStateStore.recordFingers (in addition to the existing bus emit), using VNHumanHandPoseObservation.chirality. The panel displays "Left: 3 fingers / Right: 5 fingers" correctly. Menu structure now: - App menu: Settings (⌘,) · Log Stream (⌘L) · Status (⌘I) · Save Snapshot (⌘S) · Start/Stop Recording (⇧⌘R) · Quit - Faces menu: Enroll (⌘E) · Forget (⇧⌘E) · History (⌘H) - Camera menu: Devices ▶ · Refresh · Privacy Mode (⇧⌘P) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
958da3bb95 |
Event bus + greeter + gestures + activity + spatial + history + recording + tests + CI + CLAUDE.md
Four parallel agents shipped Tier 1+2+3 of the punch list, coordinated through a fixed DetectionEventBus contract created upfront. ## Detection event bus (shared contract) - Sources/Process/DetectionEvents.swift — `DetectionEvent.Kind` enum covering objectAppeared/Refreshed/Disappeared, faceRecognized, faceSeenUnknown, textRecognized, gestureDetected, activityDetected, spatialRelation. - Sources/Process/DetectionEventBus.swift — thread-safe pub/sub, Token pattern with auto-unsubscribe on deinit, synchronous emit on caller's queue. Subscribers hop to their own queue as needed. ## ObjectDetector additions (Agent A) - Publishes events for every track-add, track-loss, face-match, unmatched-face, recognized-text — with per-event throttles (text 2s per string, face name 5s, etc.) so the bus doesn't drown in repeats. - New `DetectorStageTiming` struct + `lastStageTiming` accessor exposes the visionBundleMs / featurePrintMs / trackerMs / totalMs breakdown. Logged once a second at DEBUG level for visibility. - Track UUID persistence via Sources/Process/TrackStore.swift — label→UUID at ~/Library/Application Support/VisionPOC/tracks.json, MRU-pruned to 50 entries, debounced 500ms saves. Same object across launches gets the same trackId → consistent per-track color. - YOLO warmup at init (dispatched on detector queue): one dummy inference against a 416×416 mid-gray image so the first user-visible detection doesn't pay model-compile latency. ## Event consumers (Agent A) - Sources/Process/Greeter.swift — AVSpeechSynthesizer-driven "Hello, <name>" on faceRecognized events. Honors Theme.greeterMuted and the Privacy Mode toggle. Throttle: 30s per name on top of the bus-level 5s throttle. - Sources/Process/GestureRecognizer.swift — five gesture heuristics (thumbs_up, open_hand, fist, pointing, peace) over hand-pose PoseDetections. Throttled 2s per gesture name. - Sources/Process/ActivityRecognizer.swift — standing/sitting/ crouching/hand_raised over body-pose PoseDetections. Throttled 5s per activity. - Sources/Process/SpatialReasoner.swift — near/above/inside/holding relations between Detection pairs. Throttled 10s per triple. ## Video recording (Agent B) - Sources/Render/Recorder.swift — @MainActor AVAssetWriter pipeline writing H.264 / 12 Mbps / 60fps to ~/Desktop/VisionPOC-...mov. Captures the rendered MTKView output via a new frame-tap callback on Renderer. Synchronous blit→staging-texture→CVPixelBuffer→ AVAssetWriterInputPixelBufferAdaptor pattern; ~5-15ms per frame at 1080p, drops dynamically when active. Format-guarded for bgra8Unorm drawables and disabled when not recording (early return). - Renderer.swift gained `var frameTap` callback; invoked inside draw(in:) just before present, when set. ## History window (Agent C) - Sources/App/HistoryEntry.swift, HistoryStore.swift, HistoryView.swift, HistoryWindowController.swift — singleton ring buffer of 500 recent events translated into human-readable summaries with category tags (object/face-known/face-unknown/ocr/gesture/activity/spatial). - Open via ⌘H from the Faces menu. Polls the store at 1Hz. - Window position persists under VPOC.HistoryWindowFrame. ## Privacy mode (Agent C) - Theme.Performance.faceRecognitionDisabled — new live tunable. - Camera menu toggle (⇧⌘P) flips both faceRecognitionDisabled and greeterMuted. The History viewer also filters out face entries when the flag is set. - Documented limitation: bounding-box labels may still show enrolled names if the detector doesn't consult faceRecognitionDisabled (this round it doesn't; flag is for History + TTS only). ## Recording menu wiring (Agent C) - App menu gains "Start Recording" / "Stop Recording" (⇧⌘R). Title flips based on the Recorder's isRecording state. - Start: instantiates the recorder, calls startRecording at the current drawable size, hooks renderer.frameTap. - Stop: clears frameTap first (no more frames queue) then awaits the asset writer's finalize. On completion (MainActor hop), shows an NSAlert with the saved-path or failure. ## Settings panel additions (Agent C) - New "Faces" subsection: "Mute greeter (TTS)" toggle bound to TunableSettings.greeterMuted (writes through to Theme). - New "Diagnostics" section: per-stage detector timing readout polled at 2Hz. Decoupled from ObjectDetector via a closure-based timing provider so the App layer doesn't depend on the Process module's DetectorStageTiming type directly. ## Hygiene (Agent D) - Tests/IoUTests.swift, AspectFitTests.swift, LogStreamTests.swift, DetectionEventBusTests.swift, FaceRegistryDistanceTests.swift — 24 tests covering pure-logic algorithms. Private methods are mirrored verbatim into the test files with a "keep in sync" comment. - project.yml gains a `VisionPOCTests` target of type `bundle.unit-test`, depending on the VisionPOC application target. - .github/workflows/build.yml — macos-15 runner, brew install xcodegen, generate, build, test. LFS skipped to save bandwidth (the build tolerates a missing model since ObjectDetector falls back to Vision built-ins). - CLAUDE.md — onboarding doc covering project layout, theme tunables, pipelines, event vocabulary, observability, build idiosyncrasies, and "how to add a new detector / overlay" recipes. ## Integration fix - Recorder.swift's `stopRecording` completion path crossed actor isolation boundaries — captured a `(URL?) -> Void` closure and the AVAssetWriter instance into a @Sendable closure. Fixed by extracting `writer.status` / `writer.error?.localizedDescription` on the writer's queue before hopping, and wrapping the completion in a `nonisolated(unsafe)` shuttle (it's called from MainActor only, so the unsafe annotation is honest). ## Build status - `xcodebuild build` — clean - `xcodebuild test` — 24 / 24 passing Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
79ecd0184a |
Tier 1 + Tier 2 polish: 3 parallel agents — overlays, snapshots, menus
Coordinated three agents working in parallel on non-overlapping files,
with a shared `ProcessedFrame.swift` contract locking the API surface
between detector and renderer. All build clean on first integrate.
## Detector (ObjectDetector.swift)
- `VNRecognizeTextRequest` runs alongside YOLO at ~5 Hz; recognized
strings + boxes published via `currentTextDetections()`
- `VNDetectHumanBodyPoseRequest` produces a 14-segment skeleton (nose-
neck-shoulders-arms, neck-root-hips-legs); confidence-gated at 0.3
- `VNDetectHumanHandPoseRequest` produces 5 finger chains + palm ring
per hand, up to 2 hands; published as kind=.hand poses
- Body + hand published as `[PoseDetection]` via `currentPoses()`
- New `lastDetectMode: DetectMode { .yolo | .track }` so the footer can
distinguish the two latency regimes
- YOLO warmup runs one dummy inference at startup on the detector
queue — first user-visible detection no longer pays the ~200ms
model-compile latency
## Renderer (Renderer.swift + Render/ColorHash.swift)
- Box labels include 10%-bucketed confidence: "CUP 80%", "MOBILE PHONE
60%". Bucket keeps the label texture cache from exploding.
- Per-track stable color via UUID-byte → golden-ratio hash → HSV(h,
0.8, 1.0). Face labels stay cyan (calmer when JARVIS pane has
multiple labeled people in frame).
- Detection labels clamp to pane edges so boxes near the right margin
don't overflow into the next pane.
- Footer now reads "DET 28.4ms (yolo)" or "DET 2.1ms (track)" — the
flicker between the two values is informative, not a bug.
- LRU bound on `labelTextureCache` (128 entries, parallel access-order
array, drop-head-on-insert when at capacity).
- OCR text rendered as magenta boxes with truncated text labels (32
chars).
- Body pose drawn in mint, hand pose in amber; axis-aligned segment
rects with a 0.003 minimum thickness so horizontal/vertical lines
don't vanish at small viewport sizes. Joint dots at each unique
point.
- New `snapshotPNG() -> Data?` public method. Render loop piggybacks
one extra blit pass + waitUntilCompleted + CGImage encoding. <50ms
in practice, 1-second semaphore timeout if rendering stalls.
## App / Window / Capture
- Camera menu with a dynamic Devices submenu (rebuilt via
NSMenuDelegate.menuNeedsUpdate). Active device gets a checkmark,
Refresh List forces re-enumeration.
- CameraCapture.switchToDevice(_:) stops the session, swaps the input,
restarts. activeDeviceUniqueID exposed so the menu can match.
- Save Snapshot (⌘S) writes the 4-pane composite to
~/Desktop/VisionPOC-YYYY-MM-DD-HHMMSS.png
- Window-frame persistence for all three windows via UserDefaults
keys VPOC.MainWindowFrame / VPOC.SettingsWindowFrame /
VPOC.LogWindowFrame
- AVCaptureDevice.wasDisconnectedNotification → session stop + flag
set; permissionDenied returns true so the existing
"WAITING FOR CAMERA…" status path triggers without renderer changes
## Shared contract (new file)
- Sources/Process/ProcessedFrame.swift — TextDetection, PoseDetection,
PoseDetection.Segment, PoseKind (.body | .hand), DetectMode
(.yolo | .track). Sendable, value types, normalized [0..1] Vision
coords throughout.
## Theme
- Palette gains textStroke (magenta), bodyPose (mint), handPose (amber)
for the new overlays.
Note: Orbitron font and app icon (also Tier 2 items) deferred — no
clean source available for either without manual work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
james.maesandClaude Opus 4.7
|
ecc2ca7c06 |
Letterbox panes to preserve camera aspect ratio
Each pane now centers an aspect-fit inner rect sized to the source camera's native aspect (default 16:9 at 1080p). The image, the JARVIS overlay, detection boxes, edges, and the ASCII grid all share that inner rect; corner brackets hug the video frame instead of the outer cell so the HUD visually wraps the picture, not the letterbox bar. Pane labels stay anchored to the top-left of the inner rect so they sit inside the visible video. The capture-source dimensions come from CameraCapture's exposed width/height (1920×1080 in the default quality profile, 1280×720 in balanced). Renderer plumbing - New aspectFitRect(into:sourceAspect:) helper centers the largest source-aspect rect that fits inside a given pane. - innerRects: [CGRect] computed once per frame and threaded through drawCornerBrackets and drawPaneLabels. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
d4c358d943 |
Fix HUD text rotation, fix edge threshold, merge boxes into JARVIS, add ASCII pane
Bug fixes - text_fragment now v-flips its sample. TextRasterizer writes glyphs via a y-flipped CGContext (CoreText y-up → image-y-down), so memory row 0 holds the visual top of the text. quad_vertex already flips v for camera frames, so HUD text needed its own compensating flip in the fragment. - EdgePass was creating the threshold compute pipeline but never binding the threshold value to buffer(0). The kernel read garbage memory which usually evaluated as ~0, letting almost every fragment pass and producing the inverted look in the EDGES pane. Now binds Theme.Performance.edgeThreshold (default 0.18) explicitly. - Footer was anchored to y=2 (top of drawable). Moved to drawableSize.height minus footer height so it sits along the bottom edge where it belongs. Pane changes (per user request) - Pane 1 (JARVIS) now combines the stylization with the detection overlay: jarvis fragment → bounding-box strokes → per-box text labels. Labels are cached by string in labelTextureCache so a typical scene only rasterizes half a dozen unique strings ever. - Pane 3 was DETECT; now ASCII. Renders the live source through a new ascii_fragment that quantizes the image into a grid of monospaced glyphs picked from a pre-rasterized dark→bright atlas (` .:-=+*#%@`). New components - Sources/Shaders/Ascii.metal — ascii_fragment with AsciiUniforms (gridSize, atlasCharCount, tintColor). - Sources/Process/AsciiPass.swift — builds the glyph atlas at startup via CTLineDraw into a CGContext, uploads to a BGRA MTLTexture, and produces uniforms sized to keep glyph cells roughly square at any pane aspect. - Pipelines gains an ascii render state (quad_vertex + ascii_fragment). - Theme: paneLabels now ["LIVE", "JARVIS", "EDGES", "ASCII"], plus Performance.edgeThreshold and Performance.asciiColumns knobs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
f6601fa593 |
Implement camera capture, processing, rendering, and Jarvis HUD
End-to-end VisionPOC pipeline: Capture (Sources/Capture/CameraCapture.swift) - AVCaptureSession on a dedicated queue, BGRA @ 1080p - Zero-copy CVPixelBuffer -> MTLTexture via CVMetalTextureCache - os_unfair_lock around the latest-frame snapshot - Permission flow with denied state surfaced to the renderer Processing - ObjectDetector: VNGenerateObjectnessBasedSaliencyImageRequest is the primary all-objects path; face, human, and animal detectors layer in as labels. IOU-merge keeps labels and drops duplicate generic boxes. Throttled to Theme.Performance.detectorTargetHz on a serial queue so the renderer never blocks. Runs on the ANE via Vision's default routing. - EdgePass: MPSImageSobel into an intermediate texture, then an edge_threshold compute kernel binarizes to crisp white-on-black. - JarvisStylePass: builds the per-frame uniform struct (time, beam phase, scanline density, hex grid scale, tint). Render - Single MTKView render pass, four viewports, one HUD overlay - Live passthrough, Jarvis stylization, edge map, and live-plus-boxes - Corner brackets and pane labels composited with the boxes/text pipelines - Footer reports FPS and detector latency, rebuilt every 10 frames Shaders - quad_vertex emits a v-flipped fullscreen quad (origin-top-left camera textures display upright) - jarvis_fragment: luma -> cyan tint, scanlines, hex grid, scanning beam, vignette, light chromatic edge ghost - edges_fragment: cyan-tinted edge display with nearest sampling - box_fragment: dual-mode (hollow stroke for detections, solid fill for HUD chrome) selected by BoxUniforms.params.x - text_fragment: glyph texture multiplied by tint at fragment buffer 0 App - @main AppDelegate, regular activation policy, ⌘Q menu - Bundled-font registration via CTFontManagerRegisterFontsForURL - MainWindowController owns the MTKView and wires the pipeline Bumped deployment target to macOS 26 to match the installed SDK. Build verified: xcodegen generate && xcodebuild ... && open the .app — launches, render loop runs, camera permission prompt fires on first frame. Known cosmetic issue: Orbitron-Bold.ttf inherited from MetalPOC is a GitHub HTML page rather than a real font; Theme.Font falls back to Helvetica Neue. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |