main
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  james.maesandClaude Opus 4.7
|
6995a1bdeb |
Fix duplicate "unknown" finger row + reorganize menus by concern
## Finger counting fix
Two writers were racing on CurrentStateStore.fingerCounts:
- FingerCounter.analyze writes by actual chirality (left/right/unknown)
from VNHumanHandPoseObservation.chirality
- The CurrentStateStore event-bus subscriber ALSO wrote, but always to
the key "unknown" regardless of which hand actually fired the event
Result: any frame with a hand showed both the correct "Left: 3 fingers"
row AND a duplicate "Unknown: 3 fingers" row in the Status panel.
Plus stale entries: a hand leaving the frame didn't clean up. The Left
entry stuck around forever.
Fix:
- CurrentStateStore is no longer a writer for finger counts from the
bus path (the subscriber comment now flags FingerCounter as the
authoritative writer).
- FingerCounter.analyze gets two new behaviors:
- When hands.isEmpty → CurrentStateStore.clearFingers() wipes all
stale per-chirality entries.
- When at least one hand is detected → it records every observed
chirality this cycle, then calls ageFingerEntries(currentSeen:at:)
to drop chirality entries that weren't seen this frame and are
older than the 1.2-second staleness window.
## Menu reorganization
Followed Mac UI conventions instead of the previous "App menu = dumping
ground" layout.
Before (everything in 3 menus):
- App: Settings, Log Stream, Status, Save Snapshot, Start Recording, Quit
- Faces: Enroll, Forget, Detection History
- Camera: Devices ▶, Refresh, Privacy Mode
After (5 menus by concern):
- VisionPOC: Settings (⌘,), Hide/Hide Others/Show All, Quit (⌘Q)
- File: Save Snapshot (⌘S), Start/Stop Recording (⇧⌘R)
- View: Status Panel (⌘I), Detection History (⇧⌘H), Log Stream (⌘L)
- Camera: <devices flat list>, Refresh List, Privacy Mode (⇧⌘P)
- Faces: Enroll Face (⌘E), Forget Face (⇧⌘E)
Notes:
- Camera devices are now a flat list at the Camera menu's root (no
nested submenu) for easier access. The menu uses NSMenuDelegate to
rebuild itself when opened, so devices are always fresh.
- Detection History keybinding moved from ⌘H to ⇧⌘H because ⌘H is the
standard macOS "Hide application" shortcut.
- A small `menuItem(_:action:key:mods:target:)` helper drains the
installMainMenu function from boilerplate so the 5 menus fit on one
screen of code.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
james.maesandClaude Opus 4.7
|
3cd94535c8 |
History view uniform with Log Stream + proper line shader + Status panel (⌘I)
Three asks landing together. ## History view — now identical controls to Log Stream - Oldest-first order with auto-scroll-to-bottom on append (was newest- first, no auto-scroll). ScrollViewReader + onChange pattern lifted from LogStreamView for behavioral parity. - Toolbar now matches the Log Stream layout exactly: Category filter, Auto-scroll toggle, Pause toggle, filtered/total count, Clear button. - Poll rate bumped 1 Hz → 0.10 s so new events appear instantly. Pause toggle freezes refresh without dropping the running event collection. ## Proper line shader (much better hand + body rendering) - New `line_vertex` / `line_fragment` in Boxes.metal alongside the existing box shader. `LineUniforms` carries endpoints + thickness + soft-edge falloff + a dot-mode flag. - Line mode: emits a rotated thin quad oriented along (end - start) with width = thickness. Soft-edge antialiasing via smoothstep on the signed perpendicular UV makes diagonal skeleton segments read as actual strokes instead of axis-aligned bounding boxes. - Dot mode: emits a square around `start` and the fragment discards outside the unit-circle — produces filled disks for joint markers with sub-pixel-clean soft edges. - Pipelines gains a `lines` blended pipeline. - Renderer.drawPoses() rewritten to use the line pipeline — one draw per segment, one draw per joint, all visually consistent regardless of segment angle. Theme.Palette.bodyPose (mint) for body, handPose (amber) for hand. ## Status panel (⌘I) - New `CurrentStateStore` singleton subscribes to the event bus and retains the most recent value per concept: expression (smile / frown / neutral_face), activity (standing / sitting / etc.), per-chirality finger counts, plus a small ring of recent non-finger non-expression gestures. - New StatusView (SwiftUI) reads three sources at 4 Hz: the live ObjectDetector for FPS + DET timing + objects + faces, the CurrentStateStore for hands/expression/activity/recent-gestures, and the HistoryStore for an 8-entry tail of recent events. - New StatusWindowController hosts the SwiftUI view as a dark-aqua floating window with frame persistence (VPOC.StatusWindowFrame). - AppDelegate `openStatus(_:)` builds a DetectorSnapshot from the live windowController + detector each tick, partitioning detections into face entries (anything labeled FACE or matching an enrolled name) vs object entries. Object entries carry the same per-track color the JARVIS pane uses (via ColorHash.colorFor) so the panel's dots match the on-screen box colors. - Renderer exposes `smoothedFPSForReadout` so the panel can show the same FPS the footer does without reaching into private state. - FingerCounter now records its count per chirality via CurrentStateStore.recordFingers (in addition to the existing bus emit), using VNHumanHandPoseObservation.chirality. The panel displays "Left: 3 fingers / Right: 5 fingers" correctly. Menu structure now: - App menu: Settings (⌘,) · Log Stream (⌘L) · Status (⌘I) · Save Snapshot (⌘S) · Start/Stop Recording (⇧⌘R) · Quit - Faces menu: Enroll (⌘E) · Forget (⇧⌘E) · History (⌘H) - Camera menu: Devices ▶ · Refresh · Privacy Mode (⇧⌘P) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
813c6cc750 |
Add finger counting and facial expression (smile/frown/neutral) detection
Both build on Vision's existing requests — no new model files, no
Python conversion. The FingerCounter reads named hand-joint positions
from the existing VNDetectHumanHandPoseRequest output; the
FacialExpressionAnalyzer reads outer-lip landmarks from a swapped-in
VNDetectFaceLandmarksRequest (a superset of the existing rectangles
request — same bbox, plus the landmarks).
## FingerCounter
For each detected hand:
- Index / middle / ring / pinky are "extended" when their tip is
farther from the wrist than the MCP base joint by >1.25× (tolerates
partial curl while rejecting fully-tucked fingers)
- Thumb is special — measures perpendicular distance from the wrist→
middleMCP axis; if the thumb tip is >0.045 normalized units lateral
to that line, it's extended
- Emits `gestureDetected("fingers_<N>", confidence)` on the event bus
where confidence is the average joint confidence
- Throttled to 1.5s per N — so showing 3 fingers, opening to 5,
closing to 3 will emit cleanly but holding 3 fingers up steady won't
flood the bus
## FacialExpressionAnalyzer
For each face with landmarks:
- Read `landmarks.outerLips.normalizedPoints` — closed contour around
the mouth
- Mouth corners = leftmost + rightmost x in the contour
- Centroid y = average of all contour y values
- Compare corner_avg_y vs centroid_y with a dead-band scaled to the
face's bounding-box height (default 1%):
- Corners above centroid → smile
- Corners below centroid → frown
- Within dead-band → neutral_face
- Emits `gestureDetected("smile" | "frown" | "neutral_face", confidence)`
- Throttled to 4s per expression name
## ObjectDetector wiring
- `VNDetectFaceRectanglesRequest` → `VNDetectFaceLandmarksRequest`
(still returns VNFaceObservation with .boundingBox for tracking +
face recognition, but now also populates .landmarks for expression)
- Two new private members: `fingerCounter` + `facialExpressionAnalyzer`
- After the face-result loop in `detectAndBootstrap`, both analyzers
receive the raw Vision observations (PoseDetection.points are
unnamed, so the analyzers go straight to the source)
History and Log Stream both display the new events automatically since
they reuse the existing `gestureDetected` case in DetectionEvent.Kind.
No event-vocabulary changes, no UI changes needed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
james.maesandClaude Opus 4.7
|
958da3bb95 |
Event bus + greeter + gestures + activity + spatial + history + recording + tests + CI + CLAUDE.md
Four parallel agents shipped Tier 1+2+3 of the punch list, coordinated through a fixed DetectionEventBus contract created upfront. ## Detection event bus (shared contract) - Sources/Process/DetectionEvents.swift — `DetectionEvent.Kind` enum covering objectAppeared/Refreshed/Disappeared, faceRecognized, faceSeenUnknown, textRecognized, gestureDetected, activityDetected, spatialRelation. - Sources/Process/DetectionEventBus.swift — thread-safe pub/sub, Token pattern with auto-unsubscribe on deinit, synchronous emit on caller's queue. Subscribers hop to their own queue as needed. ## ObjectDetector additions (Agent A) - Publishes events for every track-add, track-loss, face-match, unmatched-face, recognized-text — with per-event throttles (text 2s per string, face name 5s, etc.) so the bus doesn't drown in repeats. - New `DetectorStageTiming` struct + `lastStageTiming` accessor exposes the visionBundleMs / featurePrintMs / trackerMs / totalMs breakdown. Logged once a second at DEBUG level for visibility. - Track UUID persistence via Sources/Process/TrackStore.swift — label→UUID at ~/Library/Application Support/VisionPOC/tracks.json, MRU-pruned to 50 entries, debounced 500ms saves. Same object across launches gets the same trackId → consistent per-track color. - YOLO warmup at init (dispatched on detector queue): one dummy inference against a 416×416 mid-gray image so the first user-visible detection doesn't pay model-compile latency. ## Event consumers (Agent A) - Sources/Process/Greeter.swift — AVSpeechSynthesizer-driven "Hello, <name>" on faceRecognized events. Honors Theme.greeterMuted and the Privacy Mode toggle. Throttle: 30s per name on top of the bus-level 5s throttle. - Sources/Process/GestureRecognizer.swift — five gesture heuristics (thumbs_up, open_hand, fist, pointing, peace) over hand-pose PoseDetections. Throttled 2s per gesture name. - Sources/Process/ActivityRecognizer.swift — standing/sitting/ crouching/hand_raised over body-pose PoseDetections. Throttled 5s per activity. - Sources/Process/SpatialReasoner.swift — near/above/inside/holding relations between Detection pairs. Throttled 10s per triple. ## Video recording (Agent B) - Sources/Render/Recorder.swift — @MainActor AVAssetWriter pipeline writing H.264 / 12 Mbps / 60fps to ~/Desktop/VisionPOC-...mov. Captures the rendered MTKView output via a new frame-tap callback on Renderer. Synchronous blit→staging-texture→CVPixelBuffer→ AVAssetWriterInputPixelBufferAdaptor pattern; ~5-15ms per frame at 1080p, drops dynamically when active. Format-guarded for bgra8Unorm drawables and disabled when not recording (early return). - Renderer.swift gained `var frameTap` callback; invoked inside draw(in:) just before present, when set. ## History window (Agent C) - Sources/App/HistoryEntry.swift, HistoryStore.swift, HistoryView.swift, HistoryWindowController.swift — singleton ring buffer of 500 recent events translated into human-readable summaries with category tags (object/face-known/face-unknown/ocr/gesture/activity/spatial). - Open via ⌘H from the Faces menu. Polls the store at 1Hz. - Window position persists under VPOC.HistoryWindowFrame. ## Privacy mode (Agent C) - Theme.Performance.faceRecognitionDisabled — new live tunable. - Camera menu toggle (⇧⌘P) flips both faceRecognitionDisabled and greeterMuted. The History viewer also filters out face entries when the flag is set. - Documented limitation: bounding-box labels may still show enrolled names if the detector doesn't consult faceRecognitionDisabled (this round it doesn't; flag is for History + TTS only). ## Recording menu wiring (Agent C) - App menu gains "Start Recording" / "Stop Recording" (⇧⌘R). Title flips based on the Recorder's isRecording state. - Start: instantiates the recorder, calls startRecording at the current drawable size, hooks renderer.frameTap. - Stop: clears frameTap first (no more frames queue) then awaits the asset writer's finalize. On completion (MainActor hop), shows an NSAlert with the saved-path or failure. ## Settings panel additions (Agent C) - New "Faces" subsection: "Mute greeter (TTS)" toggle bound to TunableSettings.greeterMuted (writes through to Theme). - New "Diagnostics" section: per-stage detector timing readout polled at 2Hz. Decoupled from ObjectDetector via a closure-based timing provider so the App layer doesn't depend on the Process module's DetectorStageTiming type directly. ## Hygiene (Agent D) - Tests/IoUTests.swift, AspectFitTests.swift, LogStreamTests.swift, DetectionEventBusTests.swift, FaceRegistryDistanceTests.swift — 24 tests covering pure-logic algorithms. Private methods are mirrored verbatim into the test files with a "keep in sync" comment. - project.yml gains a `VisionPOCTests` target of type `bundle.unit-test`, depending on the VisionPOC application target. - .github/workflows/build.yml — macos-15 runner, brew install xcodegen, generate, build, test. LFS skipped to save bandwidth (the build tolerates a missing model since ObjectDetector falls back to Vision built-ins). - CLAUDE.md — onboarding doc covering project layout, theme tunables, pipelines, event vocabulary, observability, build idiosyncrasies, and "how to add a new detector / overlay" recipes. ## Integration fix - Recorder.swift's `stopRecording` completion path crossed actor isolation boundaries — captured a `(URL?) -> Void` closure and the AVAssetWriter instance into a @Sendable closure. Fixed by extracting `writer.status` / `writer.error?.localizedDescription` on the writer's queue before hopping, and wrapping the completion in a `nonisolated(unsafe)` shuttle (it's called from MainActor only, so the unsafe annotation is honest). ## Build status - `xcodebuild build` — clean - `xcodebuild test` — 24 / 24 passing Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
79ecd0184a |
Tier 1 + Tier 2 polish: 3 parallel agents — overlays, snapshots, menus
Coordinated three agents working in parallel on non-overlapping files,
with a shared `ProcessedFrame.swift` contract locking the API surface
between detector and renderer. All build clean on first integrate.
## Detector (ObjectDetector.swift)
- `VNRecognizeTextRequest` runs alongside YOLO at ~5 Hz; recognized
strings + boxes published via `currentTextDetections()`
- `VNDetectHumanBodyPoseRequest` produces a 14-segment skeleton (nose-
neck-shoulders-arms, neck-root-hips-legs); confidence-gated at 0.3
- `VNDetectHumanHandPoseRequest` produces 5 finger chains + palm ring
per hand, up to 2 hands; published as kind=.hand poses
- Body + hand published as `[PoseDetection]` via `currentPoses()`
- New `lastDetectMode: DetectMode { .yolo | .track }` so the footer can
distinguish the two latency regimes
- YOLO warmup runs one dummy inference at startup on the detector
queue — first user-visible detection no longer pays the ~200ms
model-compile latency
## Renderer (Renderer.swift + Render/ColorHash.swift)
- Box labels include 10%-bucketed confidence: "CUP 80%", "MOBILE PHONE
60%". Bucket keeps the label texture cache from exploding.
- Per-track stable color via UUID-byte → golden-ratio hash → HSV(h,
0.8, 1.0). Face labels stay cyan (calmer when JARVIS pane has
multiple labeled people in frame).
- Detection labels clamp to pane edges so boxes near the right margin
don't overflow into the next pane.
- Footer now reads "DET 28.4ms (yolo)" or "DET 2.1ms (track)" — the
flicker between the two values is informative, not a bug.
- LRU bound on `labelTextureCache` (128 entries, parallel access-order
array, drop-head-on-insert when at capacity).
- OCR text rendered as magenta boxes with truncated text labels (32
chars).
- Body pose drawn in mint, hand pose in amber; axis-aligned segment
rects with a 0.003 minimum thickness so horizontal/vertical lines
don't vanish at small viewport sizes. Joint dots at each unique
point.
- New `snapshotPNG() -> Data?` public method. Render loop piggybacks
one extra blit pass + waitUntilCompleted + CGImage encoding. <50ms
in practice, 1-second semaphore timeout if rendering stalls.
## App / Window / Capture
- Camera menu with a dynamic Devices submenu (rebuilt via
NSMenuDelegate.menuNeedsUpdate). Active device gets a checkmark,
Refresh List forces re-enumeration.
- CameraCapture.switchToDevice(_:) stops the session, swaps the input,
restarts. activeDeviceUniqueID exposed so the menu can match.
- Save Snapshot (⌘S) writes the 4-pane composite to
~/Desktop/VisionPOC-YYYY-MM-DD-HHMMSS.png
- Window-frame persistence for all three windows via UserDefaults
keys VPOC.MainWindowFrame / VPOC.SettingsWindowFrame /
VPOC.LogWindowFrame
- AVCaptureDevice.wasDisconnectedNotification → session stop + flag
set; permissionDenied returns true so the existing
"WAITING FOR CAMERA…" status path triggers without renderer changes
## Shared contract (new file)
- Sources/Process/ProcessedFrame.swift — TextDetection, PoseDetection,
PoseDetection.Segment, PoseKind (.body | .hand), DetectMode
(.yolo | .track). Sendable, value types, normalized [0..1] Vision
coords throughout.
## Theme
- Palette gains textStroke (magenta), bodyPose (mint), handPose (amber)
for the new overlays.
Note: Orbitron font and app icon (also Tier 2 items) deferred — no
clean source available for either without manual work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
james.maesandClaude Opus 4.7
|
0d23d7ede9 |
Add a live Log Stream window (⌘L) and emit events from every subsystem
Built a Jarvis-style scrolling log viewer that surfaces what each subsystem is doing in real time. The buffer is a thread-safe ring of the last 1,000 entries; the SwiftUI viewer polls at 10 Hz, only re-renders when the buffer version bumps, and supports level / source filters, pause, auto-scroll, and clear. Components - Sources/App/LogStream.swift — singleton with Level (DBG/INF/WRN/ERR) and Source (APP / CAM / DETECT / TRACK / FACE / RENDER / VPOC) taxonomies, an os_unfair_lock-guarded buffer, and stdout mirroring so `log stream --process VisionPOC` shows the same feed in Terminal. - Sources/App/LogStreamView.swift — SwiftUI viewer with toolbar (level menu, source menu, auto-scroll, pause, clear, filtered/total counts) and a monospaced row-per-entry list that auto-scrolls to the bottom when new entries arrive. - Sources/App/LogStreamWindowController.swift — separate dark-aqua floating window so the log doesn't compete with the main 4-pane HUD for real estate. - AppDelegate gains a ⌘L "Log Stream…" menu item and a held reference to the controller (lazy-instantiated on first open). Wiring - AppDelegate: launch + ready - CameraCapture: device selection, session start, permission denied, enumeration failures - ObjectDetector: model load (success or fallback), 1 Hz detector stats (yolo vs track mode, ms, track count + label list), per-track add and per-track lose - ObjectDetector face match: when a face is recognized via FaceRegistry the distance is logged at DBG level - FaceRegistry: load on init, enroll (+prints, total), forget Implementation notes - The viewer is not @Observable-bound to LogStream so background- thread log emit calls don't pay a MainActor hop. Instead the view polls .version every 100ms and reloads on bump — bursts up to ~100 entries per tick render fine and the buffer absorbs writes from any rate the os_unfair_lock can sustain (~10 M/s on M-series). - LogStream.shared.log() can be called from any thread/queue. Stdout is also written from the same call so terminal tail works. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
ba1ecac72a |
Add a live Settings panel for runtime tuning
⌘, or VisionPOC → Settings… opens a SwiftUI panel with sliders for every value that's worth tuning at runtime: YOLO frequency, detection min confidence, tracker age/confidence/IoU, face match threshold, edge Sobel threshold, ASCII column count. Reset-to-defaults button at the bottom. Plumbing - Theme.Performance's runtime-tunable values converted from `static let` to `nonisolated(unsafe) static var`. Acknowledges the threading model: cooperative single-writer (MainActor UI) and many readers (detector queue, render queue). All values are word-sized primitives, atomic on ARM at the hardware level — no data races in practice. - New @Observable @MainActor `TunableSettings` mirrors those statics for SwiftUI binding; each property's didSet writes back through so detector and renderer code paths see the change on the next frame. - New `SettingsWindowController` hosts the panel as a separate floating window with a dark-aqua appearance to match the HUD aesthetic. - FaceRegistry.matchThreshold becomes a thin getter/setter passthrough to Theme.Performance.faceMatchThreshold so the face slider drives it directly. UI niceties - Sections grouped by concern (Detection, Tracking, Faces, Edges, ASCII) - Each slider shows its current value to two decimal places (or the natural format for Hz / seconds / integer counts) - Face section includes an in-context caption explaining the typical distance range so the user knows where to aim the slider Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
878fc9aece |
Add face recognition with enrollment via the menu bar
Vision's public API doesn't expose a face-specific embedding model, so we use VNGenerateImageFeaturePrintRequest with each face's bounding box as the regionOfInterest. The general-purpose image FeaturePrint applied to a padded face crop produces distance metrics that separate "same person" from "different person" reliably for the small number of enrolled identities a POC needs. Components - Sources/Process/FaceRegistry.swift — thread-safe enrollment store. Persists per-person VNFeaturePrintObservation arrays to ~/Library/Application Support/VisionPOC/faces.json via NSKeyedArchiver. bestMatch(for:) walks every stored template and returns the closest name + distance under matchThreshold. - ObjectDetector now generates a feature print per detected face on each YOLO refresh, matches against the registry, and labels the detection with the matched name (uppercased) instead of "FACE" when it recognizes someone. Enrollment flow - New Faces menu in the menu bar. - ⌘E "Enroll Face…" prompts for a name, then captures feature prints from the next 5 frames that contain the largest face in view, saves them, and shows a confirmation alert. - ⌘⇧E "Forget Face…" lists enrolled names in a popup and deletes the selected entry from disk. - ObjectDetector.captureFeaturePrints(count:completion:) drives the capture: the next N detect-pass frames that see a face have the largest face's print collected, then the completion fires once N prints are in hand. Threading - The detector completion runs on the detector queue and feeds the prints straight into FaceRegistry.enroll (which is thread-safe). Only the count is sent back to MainActor for the confirmation dialog, sidestepping the Sendable issues with passing VNFeaturePrintObservation across actor boundaries. Threshold note - FaceRegistry.matchThreshold defaults to 18.0 — tuned for the image- level FeaturePrint distances we see on face crops. Apple's published ~0.6 threshold targets face-specific embedding models that aren't exposed publicly. The threshold is a `var` so it's tunable at runtime if real-world testing wants it tighter or looser. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
0c6911bcea |
Add VNTrackObjectRequest-based tracking between YOLO refreshes
ObjectDetector now interleaves real detection with cheap tracking so bounding boxes follow objects smoothly rather than teleporting whenever a new YOLO inference completes. Pipeline - YOLO + face + animal detectors run at ~5 Hz (Theme.yoloDetectionHz), produce ground-truth detections. - VNTrackObjectRequest runs on every submitted frame between those detections, updating each active track's bounding box from the latest pixel buffer. Tracker cost is ~1ms per object. - New YOLO detections IoU-match against existing tracks (threshold 0.4, same-label bias of +0.05). Matches refresh the track; unmatched detections bootstrap fresh tracks. - Tracks expire after Theme.trackMaxAgeSeconds (0.6s) without a YOLO refresh, which covers brief occlusions but drops ghosts when objects leave the scene. - Tracker confidence below Theme.trackerMinConfidence (0.3) leaves the previous observation in place rather than degrading the box, since Vision's tracker can momentarily lose objects through motion blur or hand occlusion. Detection - Detection gains a stable UUID `trackId` so downstream consumers (a future Jarvis brain, for instance) can reason about object identity across frames without their own re-association logic. - Renderer doesn't yet use trackId — it would unlock per-object stable colors, motion trails, and a "seen recently" event ticker if surfaced. Theme knobs (Performance namespace) - yoloDetectionHz: 5.0 - trackMaxAgeSeconds: 0.6 - trackerMinConfidence: 0.3 - trackMatchIoU: 0.4 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
3c5a171d34 |
Add YOLOv3 object detection (80 COCO classes) via Git LFS
Drops Apple's full YOLOv3 (~248 MB) into Sources/Resources/Models/ as the primary object detector. Xcode auto-compiles the .mlmodel into a .mlmodelc inside the app bundle; the model is a Vision-friendly pipeline so VNCoreMLRequest returns VNRecognizedObjectObservation directly — no manual NMS or output decoding required. Detection pipeline - YOLOv3 is the primary detector, returning bounding boxes labeled with COCO classes: person, bicycle, car, ..., bottle, wine glass, cup, fork, knife, spoon, bowl, ... laptop, mouse, remote, keyboard, cell phone, ... book, clock, vase. 80 classes total — covers the "is that a coffee mug" question and many others. - VNDetectFaceRectanglesRequest stays as a supplementary FACE labeler (more precise than YOLO's "person" for the face region). - VNRecognizeAnimalsRequest stays for finer-grained cat/dog labels. - Saliency and human-rectangle requests removed — YOLO supersedes them. - Per-label NMS with IoU>0.5 inside filterAndMerge drops same-label overlaps. Cross-label overlaps (e.g. person + face) are kept since they carry different information. - Min-confidence filter (Theme.Performance.detectionMinConfidence, 0.30) prunes weak guesses. Distribution - Model is tracked via Git LFS (.gitattributes) — it exceeds GitHub's 100 MB per-file limit. - README documents the `git lfs install && git lfs pull` prerequisite. - scripts/fetch-model.sh remains as a no-LFS fallback that downloads the same model from Apple's Core ML catalog. Theme knobs - Performance.detectionMinConfidence (default 0.30) - Performance.detectorModelName (default "YOLOv3") Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
d4c358d943 |
Fix HUD text rotation, fix edge threshold, merge boxes into JARVIS, add ASCII pane
Bug fixes - text_fragment now v-flips its sample. TextRasterizer writes glyphs via a y-flipped CGContext (CoreText y-up → image-y-down), so memory row 0 holds the visual top of the text. quad_vertex already flips v for camera frames, so HUD text needed its own compensating flip in the fragment. - EdgePass was creating the threshold compute pipeline but never binding the threshold value to buffer(0). The kernel read garbage memory which usually evaluated as ~0, letting almost every fragment pass and producing the inverted look in the EDGES pane. Now binds Theme.Performance.edgeThreshold (default 0.18) explicitly. - Footer was anchored to y=2 (top of drawable). Moved to drawableSize.height minus footer height so it sits along the bottom edge where it belongs. Pane changes (per user request) - Pane 1 (JARVIS) now combines the stylization with the detection overlay: jarvis fragment → bounding-box strokes → per-box text labels. Labels are cached by string in labelTextureCache so a typical scene only rasterizes half a dozen unique strings ever. - Pane 3 was DETECT; now ASCII. Renders the live source through a new ascii_fragment that quantizes the image into a grid of monospaced glyphs picked from a pre-rasterized dark→bright atlas (` .:-=+*#%@`). New components - Sources/Shaders/Ascii.metal — ascii_fragment with AsciiUniforms (gridSize, atlasCharCount, tintColor). - Sources/Process/AsciiPass.swift — builds the glyph atlas at startup via CTLineDraw into a CGContext, uploads to a BGRA MTLTexture, and produces uniforms sized to keep glyph cells roughly square at any pane aspect. - Pipelines gains an ascii render state (quad_vertex + ascii_fragment). - Theme: paneLabels now ["LIVE", "JARVIS", "EDGES", "ASCII"], plus Performance.edgeThreshold and Performance.asciiColumns knobs. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
f6601fa593 |
Implement camera capture, processing, rendering, and Jarvis HUD
End-to-end VisionPOC pipeline: Capture (Sources/Capture/CameraCapture.swift) - AVCaptureSession on a dedicated queue, BGRA @ 1080p - Zero-copy CVPixelBuffer -> MTLTexture via CVMetalTextureCache - os_unfair_lock around the latest-frame snapshot - Permission flow with denied state surfaced to the renderer Processing - ObjectDetector: VNGenerateObjectnessBasedSaliencyImageRequest is the primary all-objects path; face, human, and animal detectors layer in as labels. IOU-merge keeps labels and drops duplicate generic boxes. Throttled to Theme.Performance.detectorTargetHz on a serial queue so the renderer never blocks. Runs on the ANE via Vision's default routing. - EdgePass: MPSImageSobel into an intermediate texture, then an edge_threshold compute kernel binarizes to crisp white-on-black. - JarvisStylePass: builds the per-frame uniform struct (time, beam phase, scanline density, hex grid scale, tint). Render - Single MTKView render pass, four viewports, one HUD overlay - Live passthrough, Jarvis stylization, edge map, and live-plus-boxes - Corner brackets and pane labels composited with the boxes/text pipelines - Footer reports FPS and detector latency, rebuilt every 10 frames Shaders - quad_vertex emits a v-flipped fullscreen quad (origin-top-left camera textures display upright) - jarvis_fragment: luma -> cyan tint, scanlines, hex grid, scanning beam, vignette, light chromatic edge ghost - edges_fragment: cyan-tinted edge display with nearest sampling - box_fragment: dual-mode (hollow stroke for detections, solid fill for HUD chrome) selected by BoxUniforms.params.x - text_fragment: glyph texture multiplied by tint at fragment buffer 0 App - @main AppDelegate, regular activation policy, ⌘Q menu - Bundled-font registration via CTFontManagerRegisterFontsForURL - MainWindowController owns the MTKView and wires the pipeline Bumped deployment target to macOS 26 to match the installed SDK. Build verified: xcodegen generate && xcodebuild ... && open the .app — launches, render loop runs, camera permission prompt fires on first frame. Known cosmetic issue: Orbitron-Bold.ttf inherited from MetalPOC is a GitHub HTML page rather than a real font; Theme.Font falls back to Helvetica Neue. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |