Swap YOLOv3 (80 classes) for YOLOv8x-OIV7 (601 classes)
Replaces the bundled detector with Ultralytics' YOLOv8x trained on Open Images V7 — 601 object classes including coffee cup, mug, wine glass, mobile phone, computer monitor, sunglasses, laptop, plus body parts (human face / hand / eye / arm / leg). Substantially better "is that a coffee mug" detection than the 80-class COCO model. Conversion - Ultralytics' yolov8x-oiv7.pt → Core ML mlpackage via ultralytics.YOLO(...).export(format="coreml", nms=True). Output is a Vision-friendly pipeline (preprocessing + model + NMS), so VNCoreMLRequest continues to return [VNRecognizedObjectObservation] directly with no manual decoding. - scripts/convert-yolo-oiv7.py is the conversion entry point. - scripts/fetch-model.sh rewritten as a no-LFS fallback that bootstraps a Python venv, installs coremltools + ultralytics, runs the conversion, and drops the result into Sources/Resources/Models/. Bundling - yolov8x-oiv7.mlpackage is a directory (~131 MB weight.bin + small model.mlmodel descriptor + Manifest.json). - .gitattributes adds LFS tracking for **/weight.bin and **/model.mlmodel inside Sources/Resources/Models so the contents go to LFS without the surrounding directory metadata pretending to be binary. - Old YOLOv3.mlmodel removed from the repo (still in LFS history if needed). Theme - Theme.Performance.detectorModelName updated to "yolov8x-oiv7" (the detector falls back to mlpackage extension automatically via the existing Bundle.main.url lookup). README + scripts updated to reflect the new model and 601-class vocabulary. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
james.maesandClaude Opus 4.7 committed
1 parent
ba1ecac72a
commit
0d46971563
9 files changed
+125
-28
No files matched your search
@@ -1,2 +1,4 @@
|
||||
Sources/Resources/Models/*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
Sources/Resources/Models/*.mlpackage filter=lfs diff=lfs merge=lfs -text
|
||||
Sources/Resources/Models/**/weight.bin filter=lfs diff=lfs merge=lfs -text
|
||||
Sources/Resources/Models/**/model.mlmodel filter=lfs diff=lfs merge=lfs -text
|
||||
@@ -7,6 +7,9 @@ Sibling to [MetalPOC](https://github.com/KofTwentyTwo/MetalPOC) (Jarvis HUD rend
|
||||
## Highlights
|
||||
|
||||
- **Zero-copy capture** — `AVCaptureSession` → `CVPixelBuffer` → `MTLTexture` via `CVMetalTextureCache` (no CPU roundtrip)
|
||||
- **601-class object detection** — YOLOv8x trained on Open Images V7, exported to a Vision-friendly Core ML pipeline. Recognizes everyday vocabulary like coffee cup, mug, wine glass, mobile phone, laptop, computer monitor, sunglasses, plus body parts (human face / hand / eye) and a broad household / vehicle / animal vocabulary
|
||||
- **Object tracking** — `VNTrackObjectRequest` updates each track's box on every frame between YOLO refreshes, so detections drift smoothly with motion instead of teleporting
|
||||
- **Face recognition** — enroll a face via the menu bar (⌘E), it shows up labeled by name in the JARVIS pane on subsequent frames. Powered by Vision's image FeaturePrint on padded face crops; templates persist to `~/Library/Application Support/VisionPOC/`
|
||||
- **Apple Neural Engine** — Vision requests configured with `MLComputeUnits.all`; runs on the ANE wherever possible
|
||||
- **Single render pass, four viewports** — one `MTKView`, one render pass, four fragment pipelines, one HUD overlay
|
||||
- **MPS Sobel** for edge detection — GPU-native, sub-millisecond at 1080p
|
||||
@@ -23,18 +26,18 @@ Sibling to [MetalPOC](https://github.com/KofTwentyTwo/MetalPOC) (Jarvis HUD rend
|
||||
|
||||
## Build & Run
|
||||
|
||||
Prerequisite: [Git LFS](https://git-lfs.com) is required because the bundled YOLOv3 model (~248 MB) is tracked via LFS. Install once on your machine: `brew install git-lfs && git lfs install`.
|
||||
Prerequisite: [Git LFS](https://git-lfs.com) is required because the bundled YOLOv8x model (~131 MB, Open Images V7, 601 classes) is tracked via LFS. Install once on your machine: `brew install git-lfs && git lfs install`.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/KofTwentyTwo/VisionPOC.git
|
||||
cd VisionPOC
|
||||
git lfs pull # ensure the YOLOv3 model is materialized
|
||||
git lfs pull # materialize the bundled model
|
||||
xcodegen generate
|
||||
xcodebuild -project VisionPOC.xcodeproj -scheme VisionPOC -configuration Debug build
|
||||
open build/Debug/VisionPOC.app
|
||||
```
|
||||
|
||||
If you cloned without LFS installed, run `git lfs install && git lfs pull` and the model will be fetched into `Sources/Resources/Models/`. A fallback script (`scripts/fetch-model.sh`) is also provided for environments where LFS isn't available — it pulls the same model directly from Apple's Core ML catalog.
|
||||
If LFS isn't installed, `scripts/fetch-model.sh` regenerates the model from Ultralytics' weights via `coremltools` (requires python 3.12 or 3.13 and a few minutes — installs into a `/tmp` venv).
|
||||
|
||||
On first launch, macOS will prompt for camera access. Grant it and the four-pane grid will activate.
|
||||
|
||||
|
||||
Binary file not shown.
BIN
Binary file not shown.
BIN
Binary file not shown.
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"fileFormatVersion": "1.0.0",
|
||||
"itemInfoEntries": {
|
||||
"12443B55-FD03-4CDF-A6A4-0D1D724518CB": {
|
||||
"author": "com.apple.CoreML",
|
||||
"description": "CoreML Model Specification",
|
||||
"name": "model.mlmodel",
|
||||
"path": "com.apple.CoreML/model.mlmodel"
|
||||
},
|
||||
"39204B74-BF34-4D89-A44F-C4E7921E93E9": {
|
||||
"author": "com.apple.CoreML",
|
||||
"description": "CoreML Model Weights",
|
||||
"name": "weights",
|
||||
"path": "com.apple.CoreML/weights"
|
||||
}
|
||||
},
|
||||
"rootModelIdentifier": "12443B55-FD03-4CDF-A6A4-0D1D724518CB"
|
||||
}
|
||||
+9
-4
@@ -95,10 +95,15 @@ enum Theme {
|
||||
/// Minimum normalized area for a box to be drawn (filters out tiny noise regions).
|
||||
static let minBoxArea: CGFloat = 0.001
|
||||
|
||||
/// Bundled object detector model. The .mlmodel file must live under
|
||||
/// `Sources/Resources/Models/<name>.mlmodel` so Xcode compiles it into
|
||||
/// the app bundle as `<name>.mlmodelc`.
|
||||
static let detectorModelName: String = "YOLOv3"
|
||||
/// Bundled object detector model. The .mlmodel / .mlpackage file must
|
||||
/// live under `Sources/Resources/Models/<name>.{mlmodel|mlpackage}` so
|
||||
/// Xcode compiles it into the app bundle as `<name>.mlmodelc`.
|
||||
///
|
||||
/// Default: yolov8x-oiv7 — Ultralytics' YOLOv8x trained on Open Images
|
||||
/// V7, 601 classes including coffee cup, mug, wine glass, laptop,
|
||||
/// computer monitor, mobile phone, human face / hand / eye / body, and
|
||||
/// a broad household / vehicle / animal vocabulary.
|
||||
static let detectorModelName: String = "yolov8x-oiv7"
|
||||
|
||||
// MARK: - Live tunables
|
||||
//
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Convert Ultralytics' yolov8x-oiv7 (Open Images V7, 600 classes) to a
|
||||
Vision-friendly Core ML package and copy it into Sources/Resources/Models/.
|
||||
|
||||
Usage:
|
||||
/tmp/visionpoc-venv/bin/python scripts/convert-yolo-oiv7.py
|
||||
"""
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
MODELS_DIR = ROOT / "Sources" / "Resources" / "Models"
|
||||
MODELS_DIR.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
VARIANT = os.environ.get("VPOC_YOLO_VARIANT", "yolov8x-oiv7")
|
||||
WEIGHTS = f"{VARIANT}.pt"
|
||||
|
||||
from ultralytics import YOLO
|
||||
|
||||
print(f"Loading {WEIGHTS}…")
|
||||
model = YOLO(WEIGHTS)
|
||||
print(f"Class count: {len(model.names)}")
|
||||
sample_labels = ", ".join(list(model.names.values())[:25])
|
||||
print(f"Sample labels: {sample_labels}…")
|
||||
|
||||
print("Exporting to Core ML (Vision-friendly, NMS baked in)…")
|
||||
exported_path = model.export(
|
||||
format="coreml",
|
||||
nms=True,
|
||||
int8=False,
|
||||
half=False,
|
||||
imgsz=640,
|
||||
)
|
||||
print(f"Exported to: {exported_path}")
|
||||
|
||||
dest = MODELS_DIR / f"{VARIANT}.mlpackage"
|
||||
if dest.exists():
|
||||
shutil.rmtree(dest)
|
||||
shutil.copytree(exported_path, dest)
|
||||
print(f"Installed at: {dest}")
|
||||
print(f"DONE — set Theme.Performance.detectorModelName to \"{VARIANT}\"")
|
||||
+40
-18
@@ -1,35 +1,57 @@
|
||||
#!/usr/bin/env bash
|
||||
# Fetches the bundled object-detection model from Apple's Core ML model catalog.
|
||||
# The model is too large to commit to git (~248 MB), so we pull it on demand.
|
||||
# Fallback for environments without Git LFS. Re-runs the YOLO conversion
|
||||
# from Ultralytics' published Open Images V7 weights — produces a Core ML
|
||||
# mlpackage with 601 object classes in Sources/Resources/Models/.
|
||||
#
|
||||
# Run once after cloning, before the first `xcodegen generate && xcodebuild`:
|
||||
# Run once after cloning, if `git lfs pull` is unavailable or didn't fetch
|
||||
# the model:
|
||||
# ./scripts/fetch-model.sh
|
||||
#
|
||||
# Requirements:
|
||||
# - python3.12 or python3.13 (3.14 isn't supported by coremltools yet)
|
||||
# - ~3 GB free disk for the venv (PyTorch + ultralytics + coremltools)
|
||||
# - ~250 MB network for the pretrained weights
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
MODELS_DIR="$ROOT/Sources/Resources/Models"
|
||||
MODEL_NAME="YOLOv3"
|
||||
MODEL_URL="https://docs-assets.developer.apple.com/coreml/models/Image/ObjectDetection/YOLOv3/YOLOv3.mlmodel"
|
||||
DEST="$MODELS_DIR/$MODEL_NAME.mlmodel"
|
||||
MODEL_NAME="yolov8x-oiv7"
|
||||
DEST="$MODELS_DIR/$MODEL_NAME.mlpackage"
|
||||
VENV="${VPOC_VENV:-/tmp/visionpoc-venv}"
|
||||
|
||||
mkdir -p "$MODELS_DIR"
|
||||
|
||||
if [[ -f "$DEST" ]]; then
|
||||
size=$(stat -f%z "$DEST")
|
||||
if [[ "$size" -gt 100000000 ]]; then
|
||||
echo "$MODEL_NAME.mlmodel already present (${size} bytes). Skipping download."
|
||||
if [[ -d "$DEST" ]] && [[ -f "$DEST/Data/com.apple.CoreML/weights/weight.bin" ]]; then
|
||||
size=$(stat -f%z "$DEST/Data/com.apple.CoreML/weights/weight.bin")
|
||||
if [[ "$size" -gt 10000000 ]]; then
|
||||
echo "$MODEL_NAME.mlpackage already present and looks valid. Skipping rebuild."
|
||||
exit 0
|
||||
fi
|
||||
echo "Existing $MODEL_NAME.mlmodel looks truncated (${size} bytes); re-downloading."
|
||||
fi
|
||||
|
||||
echo "Downloading $MODEL_NAME from Apple's Core ML catalog…"
|
||||
curl -fL --progress-bar -o "$DEST" "$MODEL_URL"
|
||||
PY_BIN=""
|
||||
for cand in python3.13 python3.12; do
|
||||
if command -v "$cand" >/dev/null 2>&1; then
|
||||
PY_BIN="$cand"
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
actual=$(stat -f%z "$DEST")
|
||||
echo "Saved $DEST (${actual} bytes)."
|
||||
|
||||
if [[ "$actual" -lt 100000000 ]]; then
|
||||
echo "Download appears incomplete. Expected ~248 MB; got ${actual} bytes." >&2
|
||||
if [[ -z "$PY_BIN" ]]; then
|
||||
echo "Need python3.12 or python3.13. Install via: brew install python@3.13" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ ! -d "$VENV" ]]; then
|
||||
echo "Creating Python venv at $VENV…"
|
||||
"$PY_BIN" -m venv "$VENV"
|
||||
fi
|
||||
|
||||
echo "Installing coremltools + ultralytics into $VENV (this takes a couple of minutes)…"
|
||||
"$VENV/bin/pip" install --quiet --upgrade pip
|
||||
"$VENV/bin/pip" install --quiet coremltools ultralytics
|
||||
|
||||
echo "Running conversion…"
|
||||
"$VENV/bin/python" "$ROOT/scripts/convert-yolo-oiv7.py"
|
||||
|
||||
echo "Done. Set Theme.Performance.detectorModelName to \"$MODEL_NAME\" (default)."
|
||||
Reference in new issue
Block a user