Bug fixes
- text_fragment now v-flips its sample. TextRasterizer writes glyphs via a
y-flipped CGContext (CoreText y-up → image-y-down), so memory row 0 holds
the visual top of the text. quad_vertex already flips v for camera frames,
so HUD text needed its own compensating flip in the fragment.
- EdgePass was creating the threshold compute pipeline but never binding the
threshold value to buffer(0). The kernel read garbage memory which usually
evaluated as ~0, letting almost every fragment pass and producing the
inverted look in the EDGES pane. Now binds Theme.Performance.edgeThreshold
(default 0.18) explicitly.
- Footer was anchored to y=2 (top of drawable). Moved to drawableSize.height
minus footer height so it sits along the bottom edge where it belongs.
Pane changes (per user request)
- Pane 1 (JARVIS) now combines the stylization with the detection overlay:
jarvis fragment → bounding-box strokes → per-box text labels. Labels are
cached by string in labelTextureCache so a typical scene only rasterizes
half a dozen unique strings ever.
- Pane 3 was DETECT; now ASCII. Renders the live source through a new
ascii_fragment that quantizes the image into a grid of monospaced glyphs
picked from a pre-rasterized dark→bright atlas (` .:-=+*#%@`).
New components
- Sources/Shaders/Ascii.metal — ascii_fragment with AsciiUniforms (gridSize,
atlasCharCount, tintColor).
- Sources/Process/AsciiPass.swift — builds the glyph atlas at startup via
CTLineDraw into a CGContext, uploads to a BGRA MTLTexture, and produces
uniforms sized to keep glyph cells roughly square at any pane aspect.
- Pipelines gains an ascii render state (quad_vertex + ascii_fragment).
- Theme: paneLabels now ["LIVE", "JARVIS", "EDGES", "ASCII"], plus
Performance.edgeThreshold and Performance.asciiColumns knobs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
End-to-end VisionPOC pipeline:
Capture (Sources/Capture/CameraCapture.swift)
- AVCaptureSession on a dedicated queue, BGRA @ 1080p
- Zero-copy CVPixelBuffer -> MTLTexture via CVMetalTextureCache
- os_unfair_lock around the latest-frame snapshot
- Permission flow with denied state surfaced to the renderer
Processing
- ObjectDetector: VNGenerateObjectnessBasedSaliencyImageRequest is the
primary all-objects path; face, human, and animal detectors layer in as
labels. IOU-merge keeps labels and drops duplicate generic boxes.
Throttled to Theme.Performance.detectorTargetHz on a serial queue so
the renderer never blocks. Runs on the ANE via Vision's default routing.
- EdgePass: MPSImageSobel into an intermediate texture, then an
edge_threshold compute kernel binarizes to crisp white-on-black.
- JarvisStylePass: builds the per-frame uniform struct (time, beam phase,
scanline density, hex grid scale, tint).
Render
- Single MTKView render pass, four viewports, one HUD overlay
- Live passthrough, Jarvis stylization, edge map, and live-plus-boxes
- Corner brackets and pane labels composited with the boxes/text pipelines
- Footer reports FPS and detector latency, rebuilt every 10 frames
Shaders
- quad_vertex emits a v-flipped fullscreen quad (origin-top-left camera
textures display upright)
- jarvis_fragment: luma -> cyan tint, scanlines, hex grid, scanning beam,
vignette, light chromatic edge ghost
- edges_fragment: cyan-tinted edge display with nearest sampling
- box_fragment: dual-mode (hollow stroke for detections, solid fill for
HUD chrome) selected by BoxUniforms.params.x
- text_fragment: glyph texture multiplied by tint at fragment buffer 0
App
- @main AppDelegate, regular activation policy, ⌘Q menu
- Bundled-font registration via CTFontManagerRegisterFontsForURL
- MainWindowController owns the MTKView and wires the pipeline
Bumped deployment target to macOS 26 to match the installed SDK.
Build verified: xcodegen generate && xcodebuild ... && open the .app —
launches, render loop runs, camera permission prompt fires on first
frame. Known cosmetic issue: Orbitron-Bold.ttf inherited from MetalPOC
is a GitHub HTML page rather than a real font; Theme.Font falls back to
Helvetica Neue.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
james.maesandClaude Opus 4.7