Files
james.maesandClaude Opus 4.7 958da3bb95 Event bus + greeter + gestures + activity + spatial + history + recording + tests + CI + CLAUDE.md
Four parallel agents shipped Tier 1+2+3 of the punch list, coordinated
through a fixed DetectionEventBus contract created upfront.

## Detection event bus (shared contract)
- Sources/Process/DetectionEvents.swift — `DetectionEvent.Kind` enum
  covering objectAppeared/Refreshed/Disappeared, faceRecognized,
  faceSeenUnknown, textRecognized, gestureDetected, activityDetected,
  spatialRelation.
- Sources/Process/DetectionEventBus.swift — thread-safe pub/sub, Token
  pattern with auto-unsubscribe on deinit, synchronous emit on caller's
  queue. Subscribers hop to their own queue as needed.

## ObjectDetector additions (Agent A)
- Publishes events for every track-add, track-loss, face-match,
  unmatched-face, recognized-text — with per-event throttles (text 2s
  per string, face name 5s, etc.) so the bus doesn't drown in repeats.
- New `DetectorStageTiming` struct + `lastStageTiming` accessor exposes
  the visionBundleMs / featurePrintMs / trackerMs / totalMs breakdown.
  Logged once a second at DEBUG level for visibility.
- Track UUID persistence via Sources/Process/TrackStore.swift
  — label→UUID at ~/Library/Application Support/VisionPOC/tracks.json,
  MRU-pruned to 50 entries, debounced 500ms saves. Same object across
  launches gets the same trackId → consistent per-track color.
- YOLO warmup at init (dispatched on detector queue): one dummy
  inference against a 416×416 mid-gray image so the first user-visible
  detection doesn't pay model-compile latency.

## Event consumers (Agent A)
- Sources/Process/Greeter.swift — AVSpeechSynthesizer-driven "Hello,
  <name>" on faceRecognized events. Honors Theme.greeterMuted and the
  Privacy Mode toggle. Throttle: 30s per name on top of the bus-level
  5s throttle.
- Sources/Process/GestureRecognizer.swift — five gesture heuristics
  (thumbs_up, open_hand, fist, pointing, peace) over hand-pose
  PoseDetections. Throttled 2s per gesture name.
- Sources/Process/ActivityRecognizer.swift — standing/sitting/
  crouching/hand_raised over body-pose PoseDetections. Throttled 5s
  per activity.
- Sources/Process/SpatialReasoner.swift — near/above/inside/holding
  relations between Detection pairs. Throttled 10s per triple.

## Video recording (Agent B)
- Sources/Render/Recorder.swift — @MainActor AVAssetWriter pipeline
  writing H.264 / 12 Mbps / 60fps to ~/Desktop/VisionPOC-...mov.
  Captures the rendered MTKView output via a new frame-tap callback
  on Renderer. Synchronous blit→staging-texture→CVPixelBuffer→
  AVAssetWriterInputPixelBufferAdaptor pattern; ~5-15ms per frame at
  1080p, drops dynamically when active. Format-guarded for bgra8Unorm
  drawables and disabled when not recording (early return).
- Renderer.swift gained `var frameTap` callback; invoked inside
  draw(in:) just before present, when set.

## History window (Agent C)
- Sources/App/HistoryEntry.swift, HistoryStore.swift, HistoryView.swift,
  HistoryWindowController.swift — singleton ring buffer of 500 recent
  events translated into human-readable summaries with category tags
  (object/face-known/face-unknown/ocr/gesture/activity/spatial).
- Open via ⌘H from the Faces menu. Polls the store at 1Hz.
- Window position persists under VPOC.HistoryWindowFrame.

## Privacy mode (Agent C)
- Theme.Performance.faceRecognitionDisabled — new live tunable.
- Camera menu toggle (⇧⌘P) flips both faceRecognitionDisabled and
  greeterMuted. The History viewer also filters out face entries when
  the flag is set.
- Documented limitation: bounding-box labels may still show enrolled
  names if the detector doesn't consult faceRecognitionDisabled (this
  round it doesn't; flag is for History + TTS only).

## Recording menu wiring (Agent C)
- App menu gains "Start Recording" / "Stop Recording" (⇧⌘R). Title
  flips based on the Recorder's isRecording state.
- Start: instantiates the recorder, calls startRecording at the
  current drawable size, hooks renderer.frameTap.
- Stop: clears frameTap first (no more frames queue) then awaits the
  asset writer's finalize. On completion (MainActor hop), shows an
  NSAlert with the saved-path or failure.

## Settings panel additions (Agent C)
- New "Faces" subsection: "Mute greeter (TTS)" toggle bound to
  TunableSettings.greeterMuted (writes through to Theme).
- New "Diagnostics" section: per-stage detector timing readout polled
  at 2Hz. Decoupled from ObjectDetector via a closure-based timing
  provider so the App layer doesn't depend on the Process module's
  DetectorStageTiming type directly.

## Hygiene (Agent D)
- Tests/IoUTests.swift, AspectFitTests.swift, LogStreamTests.swift,
  DetectionEventBusTests.swift, FaceRegistryDistanceTests.swift
  — 24 tests covering pure-logic algorithms. Private methods are
  mirrored verbatim into the test files with a "keep in sync" comment.
- project.yml gains a `VisionPOCTests` target of type
  `bundle.unit-test`, depending on the VisionPOC application target.
- .github/workflows/build.yml — macos-15 runner, brew install xcodegen,
  generate, build, test. LFS skipped to save bandwidth (the build
  tolerates a missing model since ObjectDetector falls back to Vision
  built-ins).
- CLAUDE.md — onboarding doc covering project layout, theme tunables,
  pipelines, event vocabulary, observability, build idiosyncrasies,
  and "how to add a new detector / overlay" recipes.

## Integration fix
- Recorder.swift's `stopRecording` completion path crossed actor
  isolation boundaries — captured a `(URL?) -> Void` closure and the
  AVAssetWriter instance into a @Sendable closure. Fixed by extracting
  `writer.status` / `writer.error?.localizedDescription` on the
  writer's queue before hopping, and wrapping the completion in a
  `nonisolated(unsafe)` shuttle (it's called from MainActor only, so
  the unsafe annotation is honest).

## Build status
- `xcodebuild build` — clean
- `xcodebuild test` — 24 / 24 passing

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-16 17:20:54 -05:00

200 lines
8.8 KiB
Swift
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import AppKit
import CoreGraphics
import simd
/// All tunable parameters for VisionPOC.
/// Edit a value here, rebuild, see the effect everywhere it's used.
enum Theme {
// MARK: - Palette
enum Palette {
/// Primary cyan tint used everywhere (corner brackets, box strokes, scanline highlights).
static let cyan = SIMD4<Float>(0.20, 0.95, 1.00, 1.0)
/// Dimmer cyan for secondary chrome.
static let cyanDim = SIMD4<Float>(0.10, 0.55, 0.62, 1.0)
/// Background fill behind panes.
static let background = SIMD4<Float>(0.02, 0.03, 0.04, 1.0)
/// Pane label text.
static let label = SIMD4<Float>(0.85, 0.98, 1.00, 1.0)
/// Footer micro-readouts.
static let micro = SIMD4<Float>(0.55, 0.85, 0.92, 1.0)
/// Edge map foreground (white-on-black look).
static let edgeForeground = SIMD4<Float>(0.85, 1.00, 1.00, 1.0)
/// Bounding-box stroke.
static let boxStroke = SIMD4<Float>(0.20, 0.95, 1.00, 1.0)
/// Bounding-box label background.
static let boxLabelBg = SIMD4<Float>(0.0, 0.15, 0.18, 0.85)
/// Text-recognition (OCR) overlay color — distinct from object boxes.
static let textStroke = SIMD4<Float>(1.00, 0.45, 0.85, 1.0) // magenta
/// Body-pose skeleton color.
static let bodyPose = SIMD4<Float>(0.30, 1.00, 0.80, 1.0) // mint
/// Hand-pose skeleton color.
static let handPose = SIMD4<Float>(1.00, 0.75, 0.20, 1.0) // amber
}
// MARK: - Layout
enum Layout {
/// 2x2 grid pane normalized rects (origin bottom-left, 0..1).
/// [topLeft, topRight, bottomLeft, bottomRight]
static let panes: [CGRect] = [
CGRect(x: 0.02, y: 0.52, width: 0.47, height: 0.46), // top-left LIVE
CGRect(x: 0.51, y: 0.52, width: 0.47, height: 0.46), // top-right JARVIS
CGRect(x: 0.02, y: 0.04, width: 0.47, height: 0.46), // bottom-left EDGES
CGRect(x: 0.51, y: 0.04, width: 0.47, height: 0.46) // bottom-right DETECT
]
static let paneLabels: [String] = ["LIVE", "JARVIS", "EDGES", "ASCII"]
/// Window default size on first launch.
static let defaultWindowSize = CGSize(width: 1440, height: 900)
/// Window minimum size.
static let minWindowSize = CGSize(width: 800, height: 500)
}
// MARK: - Font
enum Font {
static let bodyName = "ShareTechMono-Regular"
static let titleName = "Orbitron-Bold"
static let bodyFallback = "Menlo"
static let titleFallback = "Helvetica Neue"
static let paneLabelSize: CGFloat = 18
static let footerSize: CGFloat = 12
static let boxLabelSize: CGFloat = 11
static func body(_ size: CGFloat) -> NSFont {
NSFont(name: bodyName, size: size) ?? NSFont(name: bodyFallback, size: size) ?? NSFont.systemFont(ofSize: size)
}
static func title(_ size: CGFloat) -> NSFont {
NSFont(name: titleName, size: size) ?? NSFont(name: titleFallback, size: size) ?? NSFont.boldSystemFont(ofSize: size)
}
}
// MARK: - Capture & Detection
enum Performance {
enum Profile {
case balanced // 720p, 15 Hz detector
case quality // 1080p, 20 Hz detector
}
static let profile: Profile = .quality
static var capturePreset: AVCapturePresetName {
switch profile {
case .balanced: return .preset1280x720
case .quality: return .preset1920x1080
}
}
static var detectorTargetHz: Double {
switch profile {
case .balanced: return 15
case .quality: return 20
}
}
/// Maximum number of boxes to keep from detector output (saliency can return many regions).
static let maxDetectionsPerFrame = 20
/// Minimum normalized area for a box to be drawn (filters out tiny noise regions).
static let minBoxArea: CGFloat = 0.001
/// Bundled object detector model. The .mlmodel / .mlpackage file must
/// live under `Sources/Resources/Models/<name>.{mlmodel|mlpackage}` so
/// Xcode compiles it into the app bundle as `<name>.mlmodelc`.
///
/// Default: yolov8x-oiv7 — Ultralytics' YOLOv8x trained on Open Images
/// V7, 601 classes including coffee cup, mug, wine glass, laptop,
/// computer monitor, mobile phone, human face / hand / eye / body, and
/// a broad household / vehicle / animal vocabulary.
static let detectorModelName: String = "yolov8x-oiv7"
// MARK: - Live tunables
//
// These values are mutated at runtime by the Settings panel. Detector
// and renderer code paths read them many times per second from various
// queues; primitives (Float/Double/Int) are word-sized on Apple
// Silicon so the explicit `nonisolated(unsafe)` is honest about the
// intent: cooperative single-writer (MainActor UI) and many readers.
/// Sobel magnitude threshold for the edges pane. Higher values produce
/// fewer, more confident edge pixels. Values are unit-normalized [0..1].
nonisolated(unsafe) static var edgeThreshold: Float = 0.18
/// ASCII art pane: number of character cells across the pane horizontally.
/// The vertical cell count is derived to match the source aspect ratio.
nonisolated(unsafe) static var asciiColumns: Int = 120
/// Minimum top-label confidence (0..1) for a YOLO detection to be drawn.
/// Lower values surface more objects but include weaker guesses.
nonisolated(unsafe) static var detectionMinConfidence: Float = 0.30
/// How often YOLO re-detects (Hz). Between detections, `VNTrackObjectRequest`
/// updates positions on every submitted frame, so boxes follow objects
/// smoothly. Lower the rate to spend less ANE budget on detection.
nonisolated(unsafe) static var yoloDetectionHz: Double = 5.0
/// Tracks expire if YOLO doesn't re-confirm them within this many seconds.
/// Cover brief occlusions but drop ghosts of objects that left the scene.
nonisolated(unsafe) static var trackMaxAgeSeconds: Double = 0.6
/// Minimum tracker confidence to keep a track alive between YOLO refreshes.
/// Vision's tracker reports 0..1; anything below this is treated as lost.
nonisolated(unsafe) static var trackerMinConfidence: Float = 0.3
/// IoU threshold for matching a new YOLO detection to an existing track.
/// Above this, the existing track is refreshed; below, a new track is
/// bootstrapped.
nonisolated(unsafe) static var trackMatchIoU: Float = 0.4
/// Face match threshold for FaceRegistry. Tuned for image-FeaturePrint
/// distances on padded face crops (those run ~10–25; same-person
/// matches usually <18, different people >25).
nonisolated(unsafe) static var faceMatchThreshold: Float = 18.0
/// When true, the Greeter subscribes to face-recognized events but
/// suppresses the AVSpeechSynthesizer call. Surfaced by the UI agent
/// as a menu toggle; bound from the Settings panel.
nonisolated(unsafe) static var greeterMuted: Bool = false
/// When true, the face-recognition pipeline should skip identification
/// (suppress names on bounding-box labels). Read by ObjectDetector on
/// the detection path. Set by the Privacy Mode toggle. (If the
/// detector doesn't honor this flag, Privacy Mode still mutes the
/// greeter and hides face entries from the history view.)
nonisolated(unsafe) static var faceRecognitionDisabled: Bool = false
}
// MARK: - Tick
enum Tick {
/// Target render FPS (display vsync usually caps this).
static let renderFPS: Double = 60
/// Scanning beam vertical sweep period in seconds.
static let scanningBeamPeriod: Double = 3.0
/// Scanline density (lines per normalized vertical unit, JARVIS pane).
static let scanlineDensity: Float = 220.0
/// Hex grid scale (JARVIS pane).
static let hexGridScale: Float = 28.0
}
// MARK: - HUD chrome
enum HUD {
static let cornerBracketLength: CGFloat = 16
static let cornerBracketThickness: CGFloat = 2
static let paneFrameInset: CGFloat = 6
static let footerHeight: CGFloat = 22
}
}
/// Type-safe wrapper around AVCaptureSession.Preset string values to keep Theme.swift framework-light.
struct AVCapturePresetName: RawRepresentable, Equatable {
let rawValue: String
static let preset1280x720 = AVCapturePresetName(rawValue: "AVCaptureSessionPreset1280x720")
static let preset1920x1080 = AVCapturePresetName(rawValue: "AVCaptureSessionPreset1920x1080")
}