james.maesandClaude Opus 4.7
79ecd0184a
Coordinated three agents working in parallel on non-overlapping files,
with a shared `ProcessedFrame.swift` contract locking the API surface
between detector and renderer. All build clean on first integrate.
## Detector (ObjectDetector.swift)
- `VNRecognizeTextRequest` runs alongside YOLO at ~5 Hz; recognized
strings + boxes published via `currentTextDetections()`
- `VNDetectHumanBodyPoseRequest` produces a 14-segment skeleton (nose-
neck-shoulders-arms, neck-root-hips-legs); confidence-gated at 0.3
- `VNDetectHumanHandPoseRequest` produces 5 finger chains + palm ring
per hand, up to 2 hands; published as kind=.hand poses
- Body + hand published as `[PoseDetection]` via `currentPoses()`
- New `lastDetectMode: DetectMode { .yolo | .track }` so the footer can
distinguish the two latency regimes
- YOLO warmup runs one dummy inference at startup on the detector
queue — first user-visible detection no longer pays the ~200ms
model-compile latency
## Renderer (Renderer.swift + Render/ColorHash.swift)
- Box labels include 10%-bucketed confidence: "CUP 80%", "MOBILE PHONE
60%". Bucket keeps the label texture cache from exploding.
- Per-track stable color via UUID-byte → golden-ratio hash → HSV(h,
0.8, 1.0). Face labels stay cyan (calmer when JARVIS pane has
multiple labeled people in frame).
- Detection labels clamp to pane edges so boxes near the right margin
don't overflow into the next pane.
- Footer now reads "DET 28.4ms (yolo)" or "DET 2.1ms (track)" — the
flicker between the two values is informative, not a bug.
- LRU bound on `labelTextureCache` (128 entries, parallel access-order
array, drop-head-on-insert when at capacity).
- OCR text rendered as magenta boxes with truncated text labels (32
chars).
- Body pose drawn in mint, hand pose in amber; axis-aligned segment
rects with a 0.003 minimum thickness so horizontal/vertical lines
don't vanish at small viewport sizes. Joint dots at each unique
point.
- New `snapshotPNG() -> Data?` public method. Render loop piggybacks
one extra blit pass + waitUntilCompleted + CGImage encoding. <50ms
in practice, 1-second semaphore timeout if rendering stalls.
## App / Window / Capture
- Camera menu with a dynamic Devices submenu (rebuilt via
NSMenuDelegate.menuNeedsUpdate). Active device gets a checkmark,
Refresh List forces re-enumeration.
- CameraCapture.switchToDevice(_:) stops the session, swaps the input,
restarts. activeDeviceUniqueID exposed so the menu can match.
- Save Snapshot (⌘S) writes the 4-pane composite to
~/Desktop/VisionPOC-YYYY-MM-DD-HHMMSS.png
- Window-frame persistence for all three windows via UserDefaults
keys VPOC.MainWindowFrame / VPOC.SettingsWindowFrame /
VPOC.LogWindowFrame
- AVCaptureDevice.wasDisconnectedNotification → session stop + flag
set; permissionDenied returns true so the existing
"WAITING FOR CAMERA…" status path triggers without renderer changes
## Shared contract (new file)
- Sources/Process/ProcessedFrame.swift — TextDetection, PoseDetection,
PoseDetection.Segment, PoseKind (.body | .hand), DetectMode
(.yolo | .track). Sendable, value types, normalized [0..1] Vision
coords throughout.
## Theme
- Palette gains textStroke (magenta), bodyPose (mint), handPose (amber)
for the new overlays.
Note: Orbitron font and app icon (also Tier 2 items) deferred — no
clean source available for either without manual work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
48 lines
1.7 KiB
Swift
48 lines
1.7 KiB
Swift
import Foundation
|
|
import CoreGraphics
|
|
|
|
/// Shared frame-annotation types produced by `ObjectDetector` and consumed by
|
|
/// `Renderer`. All coordinates use Vision's normalized [0..1] image space
|
|
/// (origin bottom-left), matching the existing `Detection.rect`. The renderer
|
|
/// converts to viewport NDC the same way it does for detections, so adding a
|
|
/// new overlay type is just an array of rects-or-points and a draw call.
|
|
|
|
/// Recognized text region — produced by `VNRecognizeTextRequest`.
|
|
struct TextDetection: Sendable {
|
|
var rect: CGRect // normalized, origin bottom-left
|
|
var text: String // the recognized string
|
|
var confidence: Float
|
|
}
|
|
|
|
enum PoseKind: Sendable {
|
|
case body
|
|
case hand
|
|
}
|
|
|
|
/// A pose detection — a set of normalized line segments (joint pairs) plus
|
|
/// the keypoint positions themselves. The renderer draws lines between the
|
|
/// segment endpoints and small dots at each unique point.
|
|
struct PoseDetection: Sendable {
|
|
/// Normalized line segments (each segment goes from start to end in
|
|
/// Vision's [0..1] image coords, origin bottom-left).
|
|
var segments: [Segment]
|
|
/// All unique joint points present in this pose, for dot rendering.
|
|
var points: [CGPoint]
|
|
/// Whether this pose is a body skeleton or a hand skeleton.
|
|
var kind: PoseKind
|
|
/// Overall confidence of the pose (max of constituent joint confidences).
|
|
var confidence: Float
|
|
|
|
struct Segment: Sendable {
|
|
var start: CGPoint
|
|
var end: CGPoint
|
|
}
|
|
}
|
|
|
|
/// Tells the renderer which detector mode produced the most recent
|
|
/// inference, so the footer can label `DET 28.4ms (yolo)` vs `DET 2.1ms (track)`.
|
|
enum DetectMode: String, Sendable {
|
|
case yolo = "yolo"
|
|
case track = "track"
|
|
}
|