main
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  james.maesandClaude Opus 4.7
|
813c6cc750 |
Add finger counting and facial expression (smile/frown/neutral) detection
Both build on Vision's existing requests — no new model files, no
Python conversion. The FingerCounter reads named hand-joint positions
from the existing VNDetectHumanHandPoseRequest output; the
FacialExpressionAnalyzer reads outer-lip landmarks from a swapped-in
VNDetectFaceLandmarksRequest (a superset of the existing rectangles
request — same bbox, plus the landmarks).
## FingerCounter
For each detected hand:
- Index / middle / ring / pinky are "extended" when their tip is
farther from the wrist than the MCP base joint by >1.25× (tolerates
partial curl while rejecting fully-tucked fingers)
- Thumb is special — measures perpendicular distance from the wrist→
middleMCP axis; if the thumb tip is >0.045 normalized units lateral
to that line, it's extended
- Emits `gestureDetected("fingers_<N>", confidence)` on the event bus
where confidence is the average joint confidence
- Throttled to 1.5s per N — so showing 3 fingers, opening to 5,
closing to 3 will emit cleanly but holding 3 fingers up steady won't
flood the bus
## FacialExpressionAnalyzer
For each face with landmarks:
- Read `landmarks.outerLips.normalizedPoints` — closed contour around
the mouth
- Mouth corners = leftmost + rightmost x in the contour
- Centroid y = average of all contour y values
- Compare corner_avg_y vs centroid_y with a dead-band scaled to the
face's bounding-box height (default 1%):
- Corners above centroid → smile
- Corners below centroid → frown
- Within dead-band → neutral_face
- Emits `gestureDetected("smile" | "frown" | "neutral_face", confidence)`
- Throttled to 4s per expression name
## ObjectDetector wiring
- `VNDetectFaceRectanglesRequest` → `VNDetectFaceLandmarksRequest`
(still returns VNFaceObservation with .boundingBox for tracking +
face recognition, but now also populates .landmarks for expression)
- Two new private members: `fingerCounter` + `facialExpressionAnalyzer`
- After the face-result loop in `detectAndBootstrap`, both analyzers
receive the raw Vision observations (PoseDetection.points are
unnamed, so the analyzers go straight to the source)
History and Log Stream both display the new events automatically since
they reuse the existing `gestureDetected` case in DetectionEvent.Kind.
No event-vocabulary changes, no UI changes needed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
james.maesandClaude Opus 4.7
|
958da3bb95 |
Event bus + greeter + gestures + activity + spatial + history + recording + tests + CI + CLAUDE.md
Four parallel agents shipped Tier 1+2+3 of the punch list, coordinated through a fixed DetectionEventBus contract created upfront. ## Detection event bus (shared contract) - Sources/Process/DetectionEvents.swift — `DetectionEvent.Kind` enum covering objectAppeared/Refreshed/Disappeared, faceRecognized, faceSeenUnknown, textRecognized, gestureDetected, activityDetected, spatialRelation. - Sources/Process/DetectionEventBus.swift — thread-safe pub/sub, Token pattern with auto-unsubscribe on deinit, synchronous emit on caller's queue. Subscribers hop to their own queue as needed. ## ObjectDetector additions (Agent A) - Publishes events for every track-add, track-loss, face-match, unmatched-face, recognized-text — with per-event throttles (text 2s per string, face name 5s, etc.) so the bus doesn't drown in repeats. - New `DetectorStageTiming` struct + `lastStageTiming` accessor exposes the visionBundleMs / featurePrintMs / trackerMs / totalMs breakdown. Logged once a second at DEBUG level for visibility. - Track UUID persistence via Sources/Process/TrackStore.swift — label→UUID at ~/Library/Application Support/VisionPOC/tracks.json, MRU-pruned to 50 entries, debounced 500ms saves. Same object across launches gets the same trackId → consistent per-track color. - YOLO warmup at init (dispatched on detector queue): one dummy inference against a 416×416 mid-gray image so the first user-visible detection doesn't pay model-compile latency. ## Event consumers (Agent A) - Sources/Process/Greeter.swift — AVSpeechSynthesizer-driven "Hello, <name>" on faceRecognized events. Honors Theme.greeterMuted and the Privacy Mode toggle. Throttle: 30s per name on top of the bus-level 5s throttle. - Sources/Process/GestureRecognizer.swift — five gesture heuristics (thumbs_up, open_hand, fist, pointing, peace) over hand-pose PoseDetections. Throttled 2s per gesture name. - Sources/Process/ActivityRecognizer.swift — standing/sitting/ crouching/hand_raised over body-pose PoseDetections. Throttled 5s per activity. - Sources/Process/SpatialReasoner.swift — near/above/inside/holding relations between Detection pairs. Throttled 10s per triple. ## Video recording (Agent B) - Sources/Render/Recorder.swift — @MainActor AVAssetWriter pipeline writing H.264 / 12 Mbps / 60fps to ~/Desktop/VisionPOC-...mov. Captures the rendered MTKView output via a new frame-tap callback on Renderer. Synchronous blit→staging-texture→CVPixelBuffer→ AVAssetWriterInputPixelBufferAdaptor pattern; ~5-15ms per frame at 1080p, drops dynamically when active. Format-guarded for bgra8Unorm drawables and disabled when not recording (early return). - Renderer.swift gained `var frameTap` callback; invoked inside draw(in:) just before present, when set. ## History window (Agent C) - Sources/App/HistoryEntry.swift, HistoryStore.swift, HistoryView.swift, HistoryWindowController.swift — singleton ring buffer of 500 recent events translated into human-readable summaries with category tags (object/face-known/face-unknown/ocr/gesture/activity/spatial). - Open via ⌘H from the Faces menu. Polls the store at 1Hz. - Window position persists under VPOC.HistoryWindowFrame. ## Privacy mode (Agent C) - Theme.Performance.faceRecognitionDisabled — new live tunable. - Camera menu toggle (⇧⌘P) flips both faceRecognitionDisabled and greeterMuted. The History viewer also filters out face entries when the flag is set. - Documented limitation: bounding-box labels may still show enrolled names if the detector doesn't consult faceRecognitionDisabled (this round it doesn't; flag is for History + TTS only). ## Recording menu wiring (Agent C) - App menu gains "Start Recording" / "Stop Recording" (⇧⌘R). Title flips based on the Recorder's isRecording state. - Start: instantiates the recorder, calls startRecording at the current drawable size, hooks renderer.frameTap. - Stop: clears frameTap first (no more frames queue) then awaits the asset writer's finalize. On completion (MainActor hop), shows an NSAlert with the saved-path or failure. ## Settings panel additions (Agent C) - New "Faces" subsection: "Mute greeter (TTS)" toggle bound to TunableSettings.greeterMuted (writes through to Theme). - New "Diagnostics" section: per-stage detector timing readout polled at 2Hz. Decoupled from ObjectDetector via a closure-based timing provider so the App layer doesn't depend on the Process module's DetectorStageTiming type directly. ## Hygiene (Agent D) - Tests/IoUTests.swift, AspectFitTests.swift, LogStreamTests.swift, DetectionEventBusTests.swift, FaceRegistryDistanceTests.swift — 24 tests covering pure-logic algorithms. Private methods are mirrored verbatim into the test files with a "keep in sync" comment. - project.yml gains a `VisionPOCTests` target of type `bundle.unit-test`, depending on the VisionPOC application target. - .github/workflows/build.yml — macos-15 runner, brew install xcodegen, generate, build, test. LFS skipped to save bandwidth (the build tolerates a missing model since ObjectDetector falls back to Vision built-ins). - CLAUDE.md — onboarding doc covering project layout, theme tunables, pipelines, event vocabulary, observability, build idiosyncrasies, and "how to add a new detector / overlay" recipes. ## Integration fix - Recorder.swift's `stopRecording` completion path crossed actor isolation boundaries — captured a `(URL?) -> Void` closure and the AVAssetWriter instance into a @Sendable closure. Fixed by extracting `writer.status` / `writer.error?.localizedDescription` on the writer's queue before hopping, and wrapping the completion in a `nonisolated(unsafe)` shuttle (it's called from MainActor only, so the unsafe annotation is honest). ## Build status - `xcodebuild build` — clean - `xcodebuild test` — 24 / 24 passing Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
79ecd0184a |
Tier 1 + Tier 2 polish: 3 parallel agents — overlays, snapshots, menus
Coordinated three agents working in parallel on non-overlapping files,
with a shared `ProcessedFrame.swift` contract locking the API surface
between detector and renderer. All build clean on first integrate.
## Detector (ObjectDetector.swift)
- `VNRecognizeTextRequest` runs alongside YOLO at ~5 Hz; recognized
strings + boxes published via `currentTextDetections()`
- `VNDetectHumanBodyPoseRequest` produces a 14-segment skeleton (nose-
neck-shoulders-arms, neck-root-hips-legs); confidence-gated at 0.3
- `VNDetectHumanHandPoseRequest` produces 5 finger chains + palm ring
per hand, up to 2 hands; published as kind=.hand poses
- Body + hand published as `[PoseDetection]` via `currentPoses()`
- New `lastDetectMode: DetectMode { .yolo | .track }` so the footer can
distinguish the two latency regimes
- YOLO warmup runs one dummy inference at startup on the detector
queue — first user-visible detection no longer pays the ~200ms
model-compile latency
## Renderer (Renderer.swift + Render/ColorHash.swift)
- Box labels include 10%-bucketed confidence: "CUP 80%", "MOBILE PHONE
60%". Bucket keeps the label texture cache from exploding.
- Per-track stable color via UUID-byte → golden-ratio hash → HSV(h,
0.8, 1.0). Face labels stay cyan (calmer when JARVIS pane has
multiple labeled people in frame).
- Detection labels clamp to pane edges so boxes near the right margin
don't overflow into the next pane.
- Footer now reads "DET 28.4ms (yolo)" or "DET 2.1ms (track)" — the
flicker between the two values is informative, not a bug.
- LRU bound on `labelTextureCache` (128 entries, parallel access-order
array, drop-head-on-insert when at capacity).
- OCR text rendered as magenta boxes with truncated text labels (32
chars).
- Body pose drawn in mint, hand pose in amber; axis-aligned segment
rects with a 0.003 minimum thickness so horizontal/vertical lines
don't vanish at small viewport sizes. Joint dots at each unique
point.
- New `snapshotPNG() -> Data?` public method. Render loop piggybacks
one extra blit pass + waitUntilCompleted + CGImage encoding. <50ms
in practice, 1-second semaphore timeout if rendering stalls.
## App / Window / Capture
- Camera menu with a dynamic Devices submenu (rebuilt via
NSMenuDelegate.menuNeedsUpdate). Active device gets a checkmark,
Refresh List forces re-enumeration.
- CameraCapture.switchToDevice(_:) stops the session, swaps the input,
restarts. activeDeviceUniqueID exposed so the menu can match.
- Save Snapshot (⌘S) writes the 4-pane composite to
~/Desktop/VisionPOC-YYYY-MM-DD-HHMMSS.png
- Window-frame persistence for all three windows via UserDefaults
keys VPOC.MainWindowFrame / VPOC.SettingsWindowFrame /
VPOC.LogWindowFrame
- AVCaptureDevice.wasDisconnectedNotification → session stop + flag
set; permissionDenied returns true so the existing
"WAITING FOR CAMERA…" status path triggers without renderer changes
## Shared contract (new file)
- Sources/Process/ProcessedFrame.swift — TextDetection, PoseDetection,
PoseDetection.Segment, PoseKind (.body | .hand), DetectMode
(.yolo | .track). Sendable, value types, normalized [0..1] Vision
coords throughout.
## Theme
- Palette gains textStroke (magenta), bodyPose (mint), handPose (amber)
for the new overlays.
Note: Orbitron font and app icon (also Tier 2 items) deferred — no
clean source available for either without manual work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
james.maesandClaude Opus 4.7
|
0d23d7ede9 |
Add a live Log Stream window (⌘L) and emit events from every subsystem
Built a Jarvis-style scrolling log viewer that surfaces what each subsystem is doing in real time. The buffer is a thread-safe ring of the last 1,000 entries; the SwiftUI viewer polls at 10 Hz, only re-renders when the buffer version bumps, and supports level / source filters, pause, auto-scroll, and clear. Components - Sources/App/LogStream.swift — singleton with Level (DBG/INF/WRN/ERR) and Source (APP / CAM / DETECT / TRACK / FACE / RENDER / VPOC) taxonomies, an os_unfair_lock-guarded buffer, and stdout mirroring so `log stream --process VisionPOC` shows the same feed in Terminal. - Sources/App/LogStreamView.swift — SwiftUI viewer with toolbar (level menu, source menu, auto-scroll, pause, clear, filtered/total counts) and a monospaced row-per-entry list that auto-scrolls to the bottom when new entries arrive. - Sources/App/LogStreamWindowController.swift — separate dark-aqua floating window so the log doesn't compete with the main 4-pane HUD for real estate. - AppDelegate gains a ⌘L "Log Stream…" menu item and a held reference to the controller (lazy-instantiated on first open). Wiring - AppDelegate: launch + ready - CameraCapture: device selection, session start, permission denied, enumeration failures - ObjectDetector: model load (success or fallback), 1 Hz detector stats (yolo vs track mode, ms, track count + label list), per-track add and per-track lose - ObjectDetector face match: when a face is recognized via FaceRegistry the distance is logged at DBG level - FaceRegistry: load on init, enroll (+prints, total), forget Implementation notes - The viewer is not @Observable-bound to LogStream so background- thread log emit calls don't pay a MainActor hop. Instead the view polls .version every 100ms and reloads on bump — bursts up to ~100 entries per tick render fine and the buffer absorbs writes from any rate the os_unfair_lock can sustain (~10 M/s on M-series). - LogStream.shared.log() can be called from any thread/queue. Stdout is also written from the same call so terminal tail works. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
878fc9aece |
Add face recognition with enrollment via the menu bar
Vision's public API doesn't expose a face-specific embedding model, so we use VNGenerateImageFeaturePrintRequest with each face's bounding box as the regionOfInterest. The general-purpose image FeaturePrint applied to a padded face crop produces distance metrics that separate "same person" from "different person" reliably for the small number of enrolled identities a POC needs. Components - Sources/Process/FaceRegistry.swift — thread-safe enrollment store. Persists per-person VNFeaturePrintObservation arrays to ~/Library/Application Support/VisionPOC/faces.json via NSKeyedArchiver. bestMatch(for:) walks every stored template and returns the closest name + distance under matchThreshold. - ObjectDetector now generates a feature print per detected face on each YOLO refresh, matches against the registry, and labels the detection with the matched name (uppercased) instead of "FACE" when it recognizes someone. Enrollment flow - New Faces menu in the menu bar. - ⌘E "Enroll Face…" prompts for a name, then captures feature prints from the next 5 frames that contain the largest face in view, saves them, and shows a confirmation alert. - ⌘⇧E "Forget Face…" lists enrolled names in a popup and deletes the selected entry from disk. - ObjectDetector.captureFeaturePrints(count:completion:) drives the capture: the next N detect-pass frames that see a face have the largest face's print collected, then the completion fires once N prints are in hand. Threading - The detector completion runs on the detector queue and feeds the prints straight into FaceRegistry.enroll (which is thread-safe). Only the count is sent back to MainActor for the confirmation dialog, sidestepping the Sendable issues with passing VNFeaturePrintObservation across actor boundaries. Threshold note - FaceRegistry.matchThreshold defaults to 18.0 — tuned for the image- level FeaturePrint distances we see on face crops. Apple's published ~0.6 threshold targets face-specific embedding models that aren't exposed publicly. The threshold is a `var` so it's tunable at runtime if real-world testing wants it tighter or looser. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
0c6911bcea |
Add VNTrackObjectRequest-based tracking between YOLO refreshes
ObjectDetector now interleaves real detection with cheap tracking so bounding boxes follow objects smoothly rather than teleporting whenever a new YOLO inference completes. Pipeline - YOLO + face + animal detectors run at ~5 Hz (Theme.yoloDetectionHz), produce ground-truth detections. - VNTrackObjectRequest runs on every submitted frame between those detections, updating each active track's bounding box from the latest pixel buffer. Tracker cost is ~1ms per object. - New YOLO detections IoU-match against existing tracks (threshold 0.4, same-label bias of +0.05). Matches refresh the track; unmatched detections bootstrap fresh tracks. - Tracks expire after Theme.trackMaxAgeSeconds (0.6s) without a YOLO refresh, which covers brief occlusions but drops ghosts when objects leave the scene. - Tracker confidence below Theme.trackerMinConfidence (0.3) leaves the previous observation in place rather than degrading the box, since Vision's tracker can momentarily lose objects through motion blur or hand occlusion. Detection - Detection gains a stable UUID `trackId` so downstream consumers (a future Jarvis brain, for instance) can reason about object identity across frames without their own re-association logic. - Renderer doesn't yet use trackId — it would unlock per-object stable colors, motion trails, and a "seen recently" event ticker if surfaced. Theme knobs (Performance namespace) - yoloDetectionHz: 5.0 - trackMaxAgeSeconds: 0.6 - trackerMinConfidence: 0.3 - trackMatchIoU: 0.4 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
3c5a171d34 |
Add YOLOv3 object detection (80 COCO classes) via Git LFS
Drops Apple's full YOLOv3 (~248 MB) into Sources/Resources/Models/ as the primary object detector. Xcode auto-compiles the .mlmodel into a .mlmodelc inside the app bundle; the model is a Vision-friendly pipeline so VNCoreMLRequest returns VNRecognizedObjectObservation directly — no manual NMS or output decoding required. Detection pipeline - YOLOv3 is the primary detector, returning bounding boxes labeled with COCO classes: person, bicycle, car, ..., bottle, wine glass, cup, fork, knife, spoon, bowl, ... laptop, mouse, remote, keyboard, cell phone, ... book, clock, vase. 80 classes total — covers the "is that a coffee mug" question and many others. - VNDetectFaceRectanglesRequest stays as a supplementary FACE labeler (more precise than YOLO's "person" for the face region). - VNRecognizeAnimalsRequest stays for finer-grained cat/dog labels. - Saliency and human-rectangle requests removed — YOLO supersedes them. - Per-label NMS with IoU>0.5 inside filterAndMerge drops same-label overlaps. Cross-label overlaps (e.g. person + face) are kept since they carry different information. - Min-confidence filter (Theme.Performance.detectionMinConfidence, 0.30) prunes weak guesses. Distribution - Model is tracked via Git LFS (.gitattributes) — it exceeds GitHub's 100 MB per-file limit. - README documents the `git lfs install && git lfs pull` prerequisite. - scripts/fetch-model.sh remains as a no-LFS fallback that downloads the same model from Apple's Core ML catalog. Theme knobs - Performance.detectionMinConfidence (default 0.30) - Performance.detectorModelName (default "YOLOv3") Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
f6601fa593 |
Implement camera capture, processing, rendering, and Jarvis HUD
End-to-end VisionPOC pipeline: Capture (Sources/Capture/CameraCapture.swift) - AVCaptureSession on a dedicated queue, BGRA @ 1080p - Zero-copy CVPixelBuffer -> MTLTexture via CVMetalTextureCache - os_unfair_lock around the latest-frame snapshot - Permission flow with denied state surfaced to the renderer Processing - ObjectDetector: VNGenerateObjectnessBasedSaliencyImageRequest is the primary all-objects path; face, human, and animal detectors layer in as labels. IOU-merge keeps labels and drops duplicate generic boxes. Throttled to Theme.Performance.detectorTargetHz on a serial queue so the renderer never blocks. Runs on the ANE via Vision's default routing. - EdgePass: MPSImageSobel into an intermediate texture, then an edge_threshold compute kernel binarizes to crisp white-on-black. - JarvisStylePass: builds the per-frame uniform struct (time, beam phase, scanline density, hex grid scale, tint). Render - Single MTKView render pass, four viewports, one HUD overlay - Live passthrough, Jarvis stylization, edge map, and live-plus-boxes - Corner brackets and pane labels composited with the boxes/text pipelines - Footer reports FPS and detector latency, rebuilt every 10 frames Shaders - quad_vertex emits a v-flipped fullscreen quad (origin-top-left camera textures display upright) - jarvis_fragment: luma -> cyan tint, scanlines, hex grid, scanning beam, vignette, light chromatic edge ghost - edges_fragment: cyan-tinted edge display with nearest sampling - box_fragment: dual-mode (hollow stroke for detections, solid fill for HUD chrome) selected by BoxUniforms.params.x - text_fragment: glyph texture multiplied by tint at fragment buffer 0 App - @main AppDelegate, regular activation policy, ⌘Q menu - Bundled-font registration via CTFontManagerRegisterFontsForURL - MainWindowController owns the MTKView and wires the pipeline Bumped deployment target to macOS 26 to match the installed SDK. Build verified: xcodegen generate && xcodebuild ... && open the .app — launches, render loop runs, camera permission prompt fires on first frame. Known cosmetic issue: Orbitron-Bold.ttf inherited from MetalPOC is a GitHub HTML page rather than a real font; Theme.Font falls back to Helvetica Neue. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |