Four parallel agents shipped Tier 1+2+3 of the punch list, coordinated through a fixed DetectionEventBus contract created upfront. ## Detection event bus (shared contract) - Sources/Process/DetectionEvents.swift — `DetectionEvent.Kind` enum covering objectAppeared/Refreshed/Disappeared, faceRecognized, faceSeenUnknown, textRecognized, gestureDetected, activityDetected, spatialRelation. - Sources/Process/DetectionEventBus.swift — thread-safe pub/sub, Token pattern with auto-unsubscribe on deinit, synchronous emit on caller's queue. Subscribers hop to their own queue as needed. ## ObjectDetector additions (Agent A) - Publishes events for every track-add, track-loss, face-match, unmatched-face, recognized-text — with per-event throttles (text 2s per string, face name 5s, etc.) so the bus doesn't drown in repeats. - New `DetectorStageTiming` struct + `lastStageTiming` accessor exposes the visionBundleMs / featurePrintMs / trackerMs / totalMs breakdown. Logged once a second at DEBUG level for visibility. - Track UUID persistence via Sources/Process/TrackStore.swift — label→UUID at ~/Library/Application Support/VisionPOC/tracks.json, MRU-pruned to 50 entries, debounced 500ms saves. Same object across launches gets the same trackId → consistent per-track color. - YOLO warmup at init (dispatched on detector queue): one dummy inference against a 416×416 mid-gray image so the first user-visible detection doesn't pay model-compile latency. ## Event consumers (Agent A) - Sources/Process/Greeter.swift — AVSpeechSynthesizer-driven "Hello, <name>" on faceRecognized events. Honors Theme.greeterMuted and the Privacy Mode toggle. Throttle: 30s per name on top of the bus-level 5s throttle. - Sources/Process/GestureRecognizer.swift — five gesture heuristics (thumbs_up, open_hand, fist, pointing, peace) over hand-pose PoseDetections. Throttled 2s per gesture name. - Sources/Process/ActivityRecognizer.swift — standing/sitting/ crouching/hand_raised over body-pose PoseDetections. Throttled 5s per activity. - Sources/Process/SpatialReasoner.swift — near/above/inside/holding relations between Detection pairs. Throttled 10s per triple. ## Video recording (Agent B) - Sources/Render/Recorder.swift — @MainActor AVAssetWriter pipeline writing H.264 / 12 Mbps / 60fps to ~/Desktop/VisionPOC-...mov. Captures the rendered MTKView output via a new frame-tap callback on Renderer. Synchronous blit→staging-texture→CVPixelBuffer→ AVAssetWriterInputPixelBufferAdaptor pattern; ~5-15ms per frame at 1080p, drops dynamically when active. Format-guarded for bgra8Unorm drawables and disabled when not recording (early return). - Renderer.swift gained `var frameTap` callback; invoked inside draw(in:) just before present, when set. ## History window (Agent C) - Sources/App/HistoryEntry.swift, HistoryStore.swift, HistoryView.swift, HistoryWindowController.swift — singleton ring buffer of 500 recent events translated into human-readable summaries with category tags (object/face-known/face-unknown/ocr/gesture/activity/spatial). - Open via ⌘H from the Faces menu. Polls the store at 1Hz. - Window position persists under VPOC.HistoryWindowFrame. ## Privacy mode (Agent C) - Theme.Performance.faceRecognitionDisabled — new live tunable. - Camera menu toggle (⇧⌘P) flips both faceRecognitionDisabled and greeterMuted. The History viewer also filters out face entries when the flag is set. - Documented limitation: bounding-box labels may still show enrolled names if the detector doesn't consult faceRecognitionDisabled (this round it doesn't; flag is for History + TTS only). ## Recording menu wiring (Agent C) - App menu gains "Start Recording" / "Stop Recording" (⇧⌘R). Title flips based on the Recorder's isRecording state. - Start: instantiates the recorder, calls startRecording at the current drawable size, hooks renderer.frameTap. - Stop: clears frameTap first (no more frames queue) then awaits the asset writer's finalize. On completion (MainActor hop), shows an NSAlert with the saved-path or failure. ## Settings panel additions (Agent C) - New "Faces" subsection: "Mute greeter (TTS)" toggle bound to TunableSettings.greeterMuted (writes through to Theme). - New "Diagnostics" section: per-stage detector timing readout polled at 2Hz. Decoupled from ObjectDetector via a closure-based timing provider so the App layer doesn't depend on the Process module's DetectorStageTiming type directly. ## Hygiene (Agent D) - Tests/IoUTests.swift, AspectFitTests.swift, LogStreamTests.swift, DetectionEventBusTests.swift, FaceRegistryDistanceTests.swift — 24 tests covering pure-logic algorithms. Private methods are mirrored verbatim into the test files with a "keep in sync" comment. - project.yml gains a `VisionPOCTests` target of type `bundle.unit-test`, depending on the VisionPOC application target. - .github/workflows/build.yml — macos-15 runner, brew install xcodegen, generate, build, test. LFS skipped to save bandwidth (the build tolerates a missing model since ObjectDetector falls back to Vision built-ins). - CLAUDE.md — onboarding doc covering project layout, theme tunables, pipelines, event vocabulary, observability, build idiosyncrasies, and "how to add a new detector / overlay" recipes. ## Integration fix - Recorder.swift's `stopRecording` completion path crossed actor isolation boundaries — captured a `(URL?) -> Void` closure and the AVAssetWriter instance into a @Sendable closure. Fixed by extracting `writer.status` / `writer.error?.localizedDescription` on the writer's queue before hopping, and wrapping the completion in a `nonisolated(unsafe)` shuttle (it's called from MainActor only, so the unsafe annotation is honest). ## Build status - `xcodebuild build` — clean - `xcodebuild test` — 24 / 24 passing Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
200 lines
8.8 KiB
Swift
200 lines
8.8 KiB
Swift
import AppKit
|
||
import CoreGraphics
|
||
import simd
|
||
|
||
/// All tunable parameters for VisionPOC.
|
||
/// Edit a value here, rebuild, see the effect everywhere it's used.
|
||
enum Theme {
|
||
|
||
// MARK: - Palette
|
||
|
||
enum Palette {
|
||
/// Primary cyan tint used everywhere (corner brackets, box strokes, scanline highlights).
|
||
static let cyan = SIMD4<Float>(0.20, 0.95, 1.00, 1.0)
|
||
/// Dimmer cyan for secondary chrome.
|
||
static let cyanDim = SIMD4<Float>(0.10, 0.55, 0.62, 1.0)
|
||
/// Background fill behind panes.
|
||
static let background = SIMD4<Float>(0.02, 0.03, 0.04, 1.0)
|
||
/// Pane label text.
|
||
static let label = SIMD4<Float>(0.85, 0.98, 1.00, 1.0)
|
||
/// Footer micro-readouts.
|
||
static let micro = SIMD4<Float>(0.55, 0.85, 0.92, 1.0)
|
||
/// Edge map foreground (white-on-black look).
|
||
static let edgeForeground = SIMD4<Float>(0.85, 1.00, 1.00, 1.0)
|
||
/// Bounding-box stroke.
|
||
static let boxStroke = SIMD4<Float>(0.20, 0.95, 1.00, 1.0)
|
||
/// Bounding-box label background.
|
||
static let boxLabelBg = SIMD4<Float>(0.0, 0.15, 0.18, 0.85)
|
||
|
||
/// Text-recognition (OCR) overlay color — distinct from object boxes.
|
||
static let textStroke = SIMD4<Float>(1.00, 0.45, 0.85, 1.0) // magenta
|
||
/// Body-pose skeleton color.
|
||
static let bodyPose = SIMD4<Float>(0.30, 1.00, 0.80, 1.0) // mint
|
||
/// Hand-pose skeleton color.
|
||
static let handPose = SIMD4<Float>(1.00, 0.75, 0.20, 1.0) // amber
|
||
}
|
||
|
||
// MARK: - Layout
|
||
|
||
enum Layout {
|
||
/// 2x2 grid pane normalized rects (origin bottom-left, 0..1).
|
||
/// [topLeft, topRight, bottomLeft, bottomRight]
|
||
static let panes: [CGRect] = [
|
||
CGRect(x: 0.02, y: 0.52, width: 0.47, height: 0.46), // top-left LIVE
|
||
CGRect(x: 0.51, y: 0.52, width: 0.47, height: 0.46), // top-right JARVIS
|
||
CGRect(x: 0.02, y: 0.04, width: 0.47, height: 0.46), // bottom-left EDGES
|
||
CGRect(x: 0.51, y: 0.04, width: 0.47, height: 0.46) // bottom-right DETECT
|
||
]
|
||
static let paneLabels: [String] = ["LIVE", "JARVIS", "EDGES", "ASCII"]
|
||
/// Window default size on first launch.
|
||
static let defaultWindowSize = CGSize(width: 1440, height: 900)
|
||
/// Window minimum size.
|
||
static let minWindowSize = CGSize(width: 800, height: 500)
|
||
}
|
||
|
||
// MARK: - Font
|
||
|
||
enum Font {
|
||
static let bodyName = "ShareTechMono-Regular"
|
||
static let titleName = "Orbitron-Bold"
|
||
static let bodyFallback = "Menlo"
|
||
static let titleFallback = "Helvetica Neue"
|
||
|
||
static let paneLabelSize: CGFloat = 18
|
||
static let footerSize: CGFloat = 12
|
||
static let boxLabelSize: CGFloat = 11
|
||
|
||
static func body(_ size: CGFloat) -> NSFont {
|
||
NSFont(name: bodyName, size: size) ?? NSFont(name: bodyFallback, size: size) ?? NSFont.systemFont(ofSize: size)
|
||
}
|
||
|
||
static func title(_ size: CGFloat) -> NSFont {
|
||
NSFont(name: titleName, size: size) ?? NSFont(name: titleFallback, size: size) ?? NSFont.boldSystemFont(ofSize: size)
|
||
}
|
||
}
|
||
|
||
// MARK: - Capture & Detection
|
||
|
||
enum Performance {
|
||
enum Profile {
|
||
case balanced // 720p, 15 Hz detector
|
||
case quality // 1080p, 20 Hz detector
|
||
}
|
||
static let profile: Profile = .quality
|
||
|
||
static var capturePreset: AVCapturePresetName {
|
||
switch profile {
|
||
case .balanced: return .preset1280x720
|
||
case .quality: return .preset1920x1080
|
||
}
|
||
}
|
||
|
||
static var detectorTargetHz: Double {
|
||
switch profile {
|
||
case .balanced: return 15
|
||
case .quality: return 20
|
||
}
|
||
}
|
||
|
||
/// Maximum number of boxes to keep from detector output (saliency can return many regions).
|
||
static let maxDetectionsPerFrame = 20
|
||
|
||
/// Minimum normalized area for a box to be drawn (filters out tiny noise regions).
|
||
static let minBoxArea: CGFloat = 0.001
|
||
|
||
/// Bundled object detector model. The .mlmodel / .mlpackage file must
|
||
/// live under `Sources/Resources/Models/<name>.{mlmodel|mlpackage}` so
|
||
/// Xcode compiles it into the app bundle as `<name>.mlmodelc`.
|
||
///
|
||
/// Default: yolov8x-oiv7 — Ultralytics' YOLOv8x trained on Open Images
|
||
/// V7, 601 classes including coffee cup, mug, wine glass, laptop,
|
||
/// computer monitor, mobile phone, human face / hand / eye / body, and
|
||
/// a broad household / vehicle / animal vocabulary.
|
||
static let detectorModelName: String = "yolov8x-oiv7"
|
||
|
||
// MARK: - Live tunables
|
||
//
|
||
// These values are mutated at runtime by the Settings panel. Detector
|
||
// and renderer code paths read them many times per second from various
|
||
// queues; primitives (Float/Double/Int) are word-sized on Apple
|
||
// Silicon so the explicit `nonisolated(unsafe)` is honest about the
|
||
// intent: cooperative single-writer (MainActor UI) and many readers.
|
||
|
||
/// Sobel magnitude threshold for the edges pane. Higher values produce
|
||
/// fewer, more confident edge pixels. Values are unit-normalized [0..1].
|
||
nonisolated(unsafe) static var edgeThreshold: Float = 0.18
|
||
|
||
/// ASCII art pane: number of character cells across the pane horizontally.
|
||
/// The vertical cell count is derived to match the source aspect ratio.
|
||
nonisolated(unsafe) static var asciiColumns: Int = 120
|
||
|
||
/// Minimum top-label confidence (0..1) for a YOLO detection to be drawn.
|
||
/// Lower values surface more objects but include weaker guesses.
|
||
nonisolated(unsafe) static var detectionMinConfidence: Float = 0.30
|
||
|
||
/// How often YOLO re-detects (Hz). Between detections, `VNTrackObjectRequest`
|
||
/// updates positions on every submitted frame, so boxes follow objects
|
||
/// smoothly. Lower the rate to spend less ANE budget on detection.
|
||
nonisolated(unsafe) static var yoloDetectionHz: Double = 5.0
|
||
|
||
/// Tracks expire if YOLO doesn't re-confirm them within this many seconds.
|
||
/// Cover brief occlusions but drop ghosts of objects that left the scene.
|
||
nonisolated(unsafe) static var trackMaxAgeSeconds: Double = 0.6
|
||
|
||
/// Minimum tracker confidence to keep a track alive between YOLO refreshes.
|
||
/// Vision's tracker reports 0..1; anything below this is treated as lost.
|
||
nonisolated(unsafe) static var trackerMinConfidence: Float = 0.3
|
||
|
||
/// IoU threshold for matching a new YOLO detection to an existing track.
|
||
/// Above this, the existing track is refreshed; below, a new track is
|
||
/// bootstrapped.
|
||
nonisolated(unsafe) static var trackMatchIoU: Float = 0.4
|
||
|
||
/// Face match threshold for FaceRegistry. Tuned for image-FeaturePrint
|
||
/// distances on padded face crops (those run ~10–25; same-person
|
||
/// matches usually <18, different people >25).
|
||
nonisolated(unsafe) static var faceMatchThreshold: Float = 18.0
|
||
|
||
/// When true, the Greeter subscribes to face-recognized events but
|
||
/// suppresses the AVSpeechSynthesizer call. Surfaced by the UI agent
|
||
/// as a menu toggle; bound from the Settings panel.
|
||
nonisolated(unsafe) static var greeterMuted: Bool = false
|
||
|
||
/// When true, the face-recognition pipeline should skip identification
|
||
/// (suppress names on bounding-box labels). Read by ObjectDetector on
|
||
/// the detection path. Set by the Privacy Mode toggle. (If the
|
||
/// detector doesn't honor this flag, Privacy Mode still mutes the
|
||
/// greeter and hides face entries from the history view.)
|
||
nonisolated(unsafe) static var faceRecognitionDisabled: Bool = false
|
||
}
|
||
|
||
// MARK: - Tick
|
||
|
||
enum Tick {
|
||
/// Target render FPS (display vsync usually caps this).
|
||
static let renderFPS: Double = 60
|
||
/// Scanning beam vertical sweep period in seconds.
|
||
static let scanningBeamPeriod: Double = 3.0
|
||
/// Scanline density (lines per normalized vertical unit, JARVIS pane).
|
||
static let scanlineDensity: Float = 220.0
|
||
/// Hex grid scale (JARVIS pane).
|
||
static let hexGridScale: Float = 28.0
|
||
}
|
||
|
||
// MARK: - HUD chrome
|
||
|
||
enum HUD {
|
||
static let cornerBracketLength: CGFloat = 16
|
||
static let cornerBracketThickness: CGFloat = 2
|
||
static let paneFrameInset: CGFloat = 6
|
||
static let footerHeight: CGFloat = 22
|
||
}
|
||
}
|
||
|
||
/// Type-safe wrapper around AVCaptureSession.Preset string values to keep Theme.swift framework-light.
|
||
struct AVCapturePresetName: RawRepresentable, Equatable {
|
||
let rawValue: String
|
||
static let preset1280x720 = AVCapturePresetName(rawValue: "AVCaptureSessionPreset1280x720")
|
||
static let preset1920x1080 = AVCapturePresetName(rawValue: "AVCaptureSessionPreset1920x1080")
|
||
}
|