main
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  james.maesandClaude Opus 4.7
|
6995a1bdeb |
Fix duplicate "unknown" finger row + reorganize menus by concern
## Finger counting fix
Two writers were racing on CurrentStateStore.fingerCounts:
- FingerCounter.analyze writes by actual chirality (left/right/unknown)
from VNHumanHandPoseObservation.chirality
- The CurrentStateStore event-bus subscriber ALSO wrote, but always to
the key "unknown" regardless of which hand actually fired the event
Result: any frame with a hand showed both the correct "Left: 3 fingers"
row AND a duplicate "Unknown: 3 fingers" row in the Status panel.
Plus stale entries: a hand leaving the frame didn't clean up. The Left
entry stuck around forever.
Fix:
- CurrentStateStore is no longer a writer for finger counts from the
bus path (the subscriber comment now flags FingerCounter as the
authoritative writer).
- FingerCounter.analyze gets two new behaviors:
- When hands.isEmpty → CurrentStateStore.clearFingers() wipes all
stale per-chirality entries.
- When at least one hand is detected → it records every observed
chirality this cycle, then calls ageFingerEntries(currentSeen:at:)
to drop chirality entries that weren't seen this frame and are
older than the 1.2-second staleness window.
## Menu reorganization
Followed Mac UI conventions instead of the previous "App menu = dumping
ground" layout.
Before (everything in 3 menus):
- App: Settings, Log Stream, Status, Save Snapshot, Start Recording, Quit
- Faces: Enroll, Forget, Detection History
- Camera: Devices ▶, Refresh, Privacy Mode
After (5 menus by concern):
- VisionPOC: Settings (⌘,), Hide/Hide Others/Show All, Quit (⌘Q)
- File: Save Snapshot (⌘S), Start/Stop Recording (⇧⌘R)
- View: Status Panel (⌘I), Detection History (⇧⌘H), Log Stream (⌘L)
- Camera: <devices flat list>, Refresh List, Privacy Mode (⇧⌘P)
- Faces: Enroll Face (⌘E), Forget Face (⇧⌘E)
Notes:
- Camera devices are now a flat list at the Camera menu's root (no
nested submenu) for easier access. The menu uses NSMenuDelegate to
rebuild itself when opened, so devices are always fresh.
- Detection History keybinding moved from ⌘H to ⇧⌘H because ⌘H is the
standard macOS "Hide application" shortcut.
- A small `menuItem(_:action:key:mods:target:)` helper drains the
installMainMenu function from boilerplate so the 5 menus fit on one
screen of code.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
james.maesandClaude Opus 4.7
|
3cd94535c8 |
History view uniform with Log Stream + proper line shader + Status panel (⌘I)
Three asks landing together. ## History view — now identical controls to Log Stream - Oldest-first order with auto-scroll-to-bottom on append (was newest- first, no auto-scroll). ScrollViewReader + onChange pattern lifted from LogStreamView for behavioral parity. - Toolbar now matches the Log Stream layout exactly: Category filter, Auto-scroll toggle, Pause toggle, filtered/total count, Clear button. - Poll rate bumped 1 Hz → 0.10 s so new events appear instantly. Pause toggle freezes refresh without dropping the running event collection. ## Proper line shader (much better hand + body rendering) - New `line_vertex` / `line_fragment` in Boxes.metal alongside the existing box shader. `LineUniforms` carries endpoints + thickness + soft-edge falloff + a dot-mode flag. - Line mode: emits a rotated thin quad oriented along (end - start) with width = thickness. Soft-edge antialiasing via smoothstep on the signed perpendicular UV makes diagonal skeleton segments read as actual strokes instead of axis-aligned bounding boxes. - Dot mode: emits a square around `start` and the fragment discards outside the unit-circle — produces filled disks for joint markers with sub-pixel-clean soft edges. - Pipelines gains a `lines` blended pipeline. - Renderer.drawPoses() rewritten to use the line pipeline — one draw per segment, one draw per joint, all visually consistent regardless of segment angle. Theme.Palette.bodyPose (mint) for body, handPose (amber) for hand. ## Status panel (⌘I) - New `CurrentStateStore` singleton subscribes to the event bus and retains the most recent value per concept: expression (smile / frown / neutral_face), activity (standing / sitting / etc.), per-chirality finger counts, plus a small ring of recent non-finger non-expression gestures. - New StatusView (SwiftUI) reads three sources at 4 Hz: the live ObjectDetector for FPS + DET timing + objects + faces, the CurrentStateStore for hands/expression/activity/recent-gestures, and the HistoryStore for an 8-entry tail of recent events. - New StatusWindowController hosts the SwiftUI view as a dark-aqua floating window with frame persistence (VPOC.StatusWindowFrame). - AppDelegate `openStatus(_:)` builds a DetectorSnapshot from the live windowController + detector each tick, partitioning detections into face entries (anything labeled FACE or matching an enrolled name) vs object entries. Object entries carry the same per-track color the JARVIS pane uses (via ColorHash.colorFor) so the panel's dots match the on-screen box colors. - Renderer exposes `smoothedFPSForReadout` so the panel can show the same FPS the footer does without reaching into private state. - FingerCounter now records its count per chirality via CurrentStateStore.recordFingers (in addition to the existing bus emit), using VNHumanHandPoseObservation.chirality. The panel displays "Left: 3 fingers / Right: 5 fingers" correctly. Menu structure now: - App menu: Settings (⌘,) · Log Stream (⌘L) · Status (⌘I) · Save Snapshot (⌘S) · Start/Stop Recording (⇧⌘R) · Quit - Faces menu: Enroll (⌘E) · Forget (⇧⌘E) · History (⌘H) - Camera menu: Devices ▶ · Refresh · Privacy Mode (⇧⌘P) Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
james.maesandClaude Opus 4.7
|
813c6cc750 |
Add finger counting and facial expression (smile/frown/neutral) detection
Both build on Vision's existing requests — no new model files, no
Python conversion. The FingerCounter reads named hand-joint positions
from the existing VNDetectHumanHandPoseRequest output; the
FacialExpressionAnalyzer reads outer-lip landmarks from a swapped-in
VNDetectFaceLandmarksRequest (a superset of the existing rectangles
request — same bbox, plus the landmarks).
## FingerCounter
For each detected hand:
- Index / middle / ring / pinky are "extended" when their tip is
farther from the wrist than the MCP base joint by >1.25× (tolerates
partial curl while rejecting fully-tucked fingers)
- Thumb is special — measures perpendicular distance from the wrist→
middleMCP axis; if the thumb tip is >0.045 normalized units lateral
to that line, it's extended
- Emits `gestureDetected("fingers_<N>", confidence)` on the event bus
where confidence is the average joint confidence
- Throttled to 1.5s per N — so showing 3 fingers, opening to 5,
closing to 3 will emit cleanly but holding 3 fingers up steady won't
flood the bus
## FacialExpressionAnalyzer
For each face with landmarks:
- Read `landmarks.outerLips.normalizedPoints` — closed contour around
the mouth
- Mouth corners = leftmost + rightmost x in the contour
- Centroid y = average of all contour y values
- Compare corner_avg_y vs centroid_y with a dead-band scaled to the
face's bounding-box height (default 1%):
- Corners above centroid → smile
- Corners below centroid → frown
- Within dead-band → neutral_face
- Emits `gestureDetected("smile" | "frown" | "neutral_face", confidence)`
- Throttled to 4s per expression name
## ObjectDetector wiring
- `VNDetectFaceRectanglesRequest` → `VNDetectFaceLandmarksRequest`
(still returns VNFaceObservation with .boundingBox for tracking +
face recognition, but now also populates .landmarks for expression)
- Two new private members: `fingerCounter` + `facialExpressionAnalyzer`
- After the face-result loop in `detectAndBootstrap`, both analyzers
receive the raw Vision observations (PoseDetection.points are
unnamed, so the analyzers go straight to the source)
History and Log Stream both display the new events automatically since
they reuse the existing `gestureDetected` case in DetectionEvent.Kind.
No event-vocabulary changes, no UI changes needed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|