0 Fork 0

Files

T

History

james.maes 928ce4bc61 fix(harness): move Corpora resource into Sources/Harness — fixes xcodebuild (#169 )

Moves Corpora dir from packages/Harness/Corpora into packages/Harness/Sources/Harness/Corpora and updates the resource declaration from .copy("../../Corpora") to .copy("Corpora"). xcodebuild's strict package resolution was flagging the package-root location as unhandled despite the swift-test pattern working. Bundle.module layout unchanged; all call sites unaffected. 63/63 Harness tests pass locally.

2026-05-13 09:26:54 -05:00

Sources

fix(harness): move Corpora resource into Sources/Harness — fixes xcodebuild (#169 )

2026-05-13 09:26:54 -05:00

Tests/HarnessTests

refactor(08-reviews): fold cross-AI suggestions into hardening substrate

2026-05-01 05:35:17 -05:00

Package.swift

fix(harness): move Corpora resource into Sources/Harness — fixes xcodebuild (#169 )

2026-05-13 09:26:54 -05:00

README.md

docs(packages): per-package README files

2026-05-04 17:13:33 -05:00

README.md

Harness

The eval harness + test fixtures + the jarvis-eval CLI executable. Not shipped in the user-facing app — it's the infrastructure for "run after significant changes" eval scenarios.

Key public types

Type	Purpose
`MockLLMProvider`	Records every prompt + tool-call exchange; scriptable response sequences
`ReplayMCPAdapter`	Replays a recorded MCP transcript against a fresh dispatcher
`MockHelperBuilder`	Generates fake MCP helper bundles for crash/restart tests
`CapturingURLProtocol`	Records HTTP traffic for SSE/NDJSON fixture capture
`FDLeakDetector`	Asserts no file descriptors leaked across a test boundary (SEC-08)
`*Corpus`	Fixture corpora (SSE, NDJSON, wake-hysteresis, injection, checklist)
`*Runner`	Per-corpus eval runners (`SSEFixtureRunner`, `WakeHysteresisRunner`, etc.)
`DriftClassifier` / `DriftReport`	Compare observed vs expected results across runs

Depends on

Logging, Config, AgentCore, Replay, MCP, Voice, Memory, DevOverlay. External: swift-argument-parser, swift-log, MCP SDK, Yams.

Used by

The jarvis-eval executable target; not imported by App/.

Key invariants / contracts

Corpora are versioned + checksummed. Fixture changes require deliberate updates (not silent drift).
Local-model eval pinned to qwen2.5-coder:32b — Qwen3 / Gemma 4 tool-calling broken in Ollama as of April 2026.
Live tests gated by env flags. JARVIS_REAL_MODELS=1, JARVIS_REAL_CAMERA=1. Default runs are offline + deterministic.
MockLLMProvider records calls (Mock taxonomy per CLAUDE.md §Test naming conventions).

Tests

The harness package itself has its own XCTests (smoke tests on the runners + corpus loaders). Most of the codebase's eval coverage is invoked through the harness, not of the harness.

Notable files

Sources/jarvis-eval/main.swift — CLI entry point
Sources/Harness/Adapters/MockLLMProvider.swift — the canonical mock LLM
Sources/Harness/Runners/SSEFixtureRunner.swift — Anthropic SSE replay runner
Sources/Harness/Runners/MCPCrashRunner.swift — helper-restart scenarios
Sources/Harness/Corpus/SSEFixtureCorpus.swift — recorded SSE traces