The model has never been connected: every conversation path is proven only against loopback stubs #8

New Issue

2026-10-01T22:50:36Z

KofTwentyTwo commented

2026-10-01 22:50:36 +00:00

(Migrated from github.com)

Summary

CarlConversation.infer calls the foundation AgentRuntime (src/main/java/com/kof22/carlai/domain/CarlConversation.java:688) and the docked chat, family API, report narration and plan conversations all go through it. The path is real code. It has never been exercised against the Anthropic API:

CarlConversationHttpTest runs a fake /v1/messages loopback server (:1500); AgentApplicationTest mocks AnthropicClient (:46).
Every browser and evaluation runner forces CARL_PREVIEW_LIVE_MODEL=false and deletes ANTHROPIC_API_KEY (scripts/e2e/docked-chat-browser.cjs:286-287, scripts/e2e/live-evaluation.cjs:33).
All six live evaluation cases LE-01 to LE-06 are BLOCKED with modelQualification: DISCONNECTED in docs/evidence/2026-10-01-controlled-model-evaluation.json.
The "22 evaluation assertions" cited in docs/TODO.md and docs/ACCEPTANCE.md are 22 assert calls in 12 node tests of the scoring helpers in scripts/tests/test_live_evaluation.cjs; no adapter, application or model is involved.
The docs attribute the hold to a 1Password token that could not be read.

Impact

Narration quality, tool selection, refusal/clarification behaviour and the UNKNOWN handling on real provider errors are unverified. The evaluation count in the docs reads as model evaluation and is not.

Suggested resolution

Run the existing live evaluation once with an authorized key in a disposable database and record the result.
Relabel the 22 assertions as scoring-script unit tests in docs/TODO.md, docs/ACCEPTANCE.md and docs/EVALUATION.md.
Unify the two environment names (KOF22_AGENT_ANTHROPIC_API_KEY in config/agent.properties, ANTHROPIC_API_KEY in the e2e harness).

Priority rationale

P1: the product's core capability is unproven against the real service.

Filed from the 2026-10-01 real-versus-fake audit of Carl (branch feature/cicd-liquibase-gitops, HEAD 5414d7a, main 1b334d1). Four read-only reviews covered domain and services, QQQ metadata and data model, tests/CI/evidence, and docs/integrations. A local offline build of a Carl copy against the installed foundation 0.5.0-SNAPSHOT is in progress; its result will be added as a comment. Existing issues #3, #4 and #5 (real Monarch accounts, Authentik with real data, CI/CD with Liquibase and ArgoCD) are referenced rather than duplicated.

## Summary `CarlConversation.infer` calls the foundation `AgentRuntime` (`src/main/java/com/kof22/carlai/domain/CarlConversation.java:688`) and the docked chat, family API, report narration and plan conversations all go through it. The path is real code. It has never been exercised against the Anthropic API: - `CarlConversationHttpTest` runs a fake `/v1/messages` loopback server (`:1500`); `AgentApplicationTest` mocks `AnthropicClient` (`:46`). - Every browser and evaluation runner forces `CARL_PREVIEW_LIVE_MODEL=false` and deletes `ANTHROPIC_API_KEY` (`scripts/e2e/docked-chat-browser.cjs:286-287`, `scripts/e2e/live-evaluation.cjs:33`). - All six live evaluation cases LE-01 to LE-06 are `BLOCKED` with `modelQualification: DISCONNECTED` in `docs/evidence/2026-10-01-controlled-model-evaluation.json`. - The "22 evaluation assertions" cited in `docs/TODO.md` and `docs/ACCEPTANCE.md` are 22 `assert` calls in 12 node tests of the scoring helpers in `scripts/tests/test_live_evaluation.cjs`; no adapter, application or model is involved. - The docs attribute the hold to a 1Password token that could not be read. ## Impact Narration quality, tool selection, refusal/clarification behaviour and the UNKNOWN handling on real provider errors are unverified. The evaluation count in the docs reads as model evaluation and is not. ## Suggested resolution 1. Run the existing live evaluation once with an authorized key in a disposable database and record the result. 2. Relabel the 22 assertions as scoring-script unit tests in `docs/TODO.md`, `docs/ACCEPTANCE.md` and `docs/EVALUATION.md`. 3. Unify the two environment names (`KOF22_AGENT_ANTHROPIC_API_KEY` in `config/agent.properties`, `ANTHROPIC_API_KEY` in the e2e harness). ## Priority rationale P1: the product's core capability is unproven against the real service. --- Filed from the 2026-10-01 real-versus-fake audit of Carl (branch `feature/cicd-liquibase-gitops`, HEAD `5414d7a`, main `1b334d1`). Four read-only reviews covered domain and services, QQQ metadata and data model, tests/CI/evidence, and docs/integrations. A local offline build of a Carl copy against the installed foundation 0.5.0-SNAPSHOT is in progress; its result will be added as a comment. Existing issues #3, #4 and #5 (real Monarch accounts, Authentik with real data, CI/CD with Liquibase and ArgoCD) are referenced rather than duplicated.

This repo is archived. You cannot comment on issues.

1 Participants

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: KofTwentyTwo/carl#8