feat(store): version the journal's schema and migrate it once, when the service starts #49

2026-11-11T04:16:11Z

james.maes commented

2026-11-11 04:16:11 +00:00

Plan step 5 of the multi-server build, and audit finding W4: the journal has a schema version, and it is brought up to date once, when the service starts.

What changes

The journal carries a schema version. One migration step (version 1) brings a journal left by any 1.x release, or by the prereleases before it, to one schema, in one transaction: either every pending change is made or none is.
Opening a store afterwards only opens it and checks the version. Nothing outside the migration creates or alters a table.
A journal written by a newer build is left exactly as it is. The service contacts no server and answers only status, log settings and update operations; every other operation is refused with one sentence telling the operator to install the newer version again.
A start that cannot migrate the journal fails as before, and now logs the state folder, the kind of failure and that the journal was not changed, never the journal's content.

What it does to an upgraded 1.2.x installation

On a 1.2.1 journal, version 1 changes nothing but the version stamp. Pairing, job history and settings are untouched.
Jobs caught mid-flight by the upgrade keep the reliability rule: a job left SUBMITTING becomes OUTCOME_UNKNOWN and is not printed again; a job left RECEIVED becomes FAILED, with Windows never asked.
Job events written before events recorded their pairing (only a journal last written by v0.1.0-rc.1 or earlier has any) are given the saved pairing, as every 1.x start has always done. ADR-0004 is corrected to say so.

Verification

Head b8d0e70 on release/2.0 at 65a99d3.
build/Verify.ps1 passed twice on that head: 1,166 Core, 1,596 Windows runtime and command line, 360 presentation tests, none failed; Windows 88.08% and 88.13%; minimum raised from 86 to 87.

Review (independent reviewer, two passes)

ec8b76a: Verdict on ec8b76a: 0 blockers, 3 should-fix, 9 nits. The reviewer found nothing that can lose or change a 1.x pairing, history or settings, checked the migration against the schema at all 15 release tags, and probed crash points, locked, truncated and full-disk cases.
b8d0e70, the head merged: Verdict on b8d0e70: 0 blockers, 0 should-fix, 2 nits. Fifteen single changes of the reviewer's own, each a full run of the Windows suite; fourteen caught by tests, one harmless by design.

Not proven here

The 1.2.1 journal used as evidence is hand-written to that release's schema and serialization (its 28 schema statements equal the tag's), not produced by a real installation. What a real upgrade leaves on disk, the protected files under LocalSystem and the installer's behaviour when the first start fails need the upgrade qualification on an installed 1.2.1.

Left open, to be recorded in the audit (nits)

A journal with a negative version is logged with advice that does not apply.
Each refused operation in the newer-version state writes the operation's name to the debug log.
One lock test misses a "take, release, then open" variant; two test-only paths open a store before the service loop runs.
AgentRealtimeBandwidthTests.ConcurrentDownloadsShareOneConfiguredBandwidthBudget timed out once in sixteen full runs under machine load and passed alone; not touched by this change.

🤖 Generated with Claude Code

Plan step 5 of the multi-server build, and audit finding W4: the journal has a schema version, and it is brought up to date once, when the service starts. **What changes** - The journal carries a schema version. One migration step (version 1) brings a journal left by any 1.x release, or by the prereleases before it, to one schema, in one transaction: either every pending change is made or none is. - Opening a store afterwards only opens it and checks the version. Nothing outside the migration creates or alters a table. - A journal written by a newer build is left exactly as it is. The service contacts no server and answers only status, log settings and update operations; every other operation is refused with one sentence telling the operator to install the newer version again. - A start that cannot migrate the journal fails as before, and now logs the state folder, the kind of failure and that the journal was not changed, never the journal's content. **What it does to an upgraded 1.2.x installation** - On a 1.2.1 journal, version 1 changes nothing but the version stamp. Pairing, job history and settings are untouched. - Jobs caught mid-flight by the upgrade keep the reliability rule: a job left SUBMITTING becomes OUTCOME_UNKNOWN and is not printed again; a job left RECEIVED becomes FAILED, with Windows never asked. - Job events written before events recorded their pairing (only a journal last written by v0.1.0-rc.1 or earlier has any) are given the saved pairing, as every 1.x start has always done. ADR-0004 is corrected to say so. **Verification** - Head `b8d0e70` on `release/2.0` at `65a99d3`. - `build/Verify.ps1` passed twice on that head: 1,166 Core, 1,596 Windows runtime and command line, 360 presentation tests, none failed; Windows 88.08% and 88.13%; minimum raised from 86 to 87. **Review (independent reviewer, two passes)** - `ec8b76a`: `Verdict on ec8b76a: 0 blockers, 3 should-fix, 9 nits.` The reviewer found nothing that can lose or change a 1.x pairing, history or settings, checked the migration against the schema at all 15 release tags, and probed crash points, locked, truncated and full-disk cases. - `b8d0e70`, the head merged: `Verdict on b8d0e70: 0 blockers, 0 should-fix, 2 nits.` Fifteen single changes of the reviewer's own, each a full run of the Windows suite; fourteen caught by tests, one harmless by design. **Not proven here** - The 1.2.1 journal used as evidence is hand-written to that release's schema and serialization (its 28 schema statements equal the tag's), not produced by a real installation. What a real upgrade leaves on disk, the protected files under LocalSystem and the installer's behaviour when the first start fails need the upgrade qualification on an installed 1.2.1. **Left open, to be recorded in the audit (nits)** - A journal with a negative version is logged with advice that does not apply. - Each refused operation in the newer-version state writes the operation's name to the debug log. - One lock test misses a "take, release, then open" variant; two test-only paths open a store before the service loop runs. - `AgentRealtimeBandwidthTests.ConcurrentDownloadsShareOneConfiguredBandwidthBudget` timed out once in sixteen full runs under machine load and passed alone; not touched by this change. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

james.maes added 17 commits 2026-11-11 04:16:11 +00:00

refactor(store): move the journal's connection string, opening and lock to JournalDatabase 4efdca2292

The store kept these as statics of its own: the connection string that also
names the journal's connection pool, the per-folder lock, and the rule that a
pooled connection is opened only while that lock is held. They move to
JournalDatabase unchanged, with an unpooled open beside them for the test
project's own connections, so that the migration that follows can use them
without going through a store. The SQLite provider is selected there too,
because every way of opening a journal now passes through that type.

No behavior changes: a store still opens its connection under the journal's
lock and still creates and migrates the schema as it opens. TestState forwards
to the new type, so no test changes.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: James Maes <james@kof22.com>

feat(store): version the journal's schema and migrate it once, when the service starts b26b41d9a1

The schema was created and altered every time a store was opened, under a
lock, with no version (audit finding W4). It now carries a version in PRAGMA
user_version, and the migration is the only code that creates or alters a
table.

- JournalSchema.Migrate reads the version through a read-only connection and
  runs the pending steps in one transaction, stamping the version in that
  transaction. A failing step rolls back all of them and the version. Callers
  in one process are serialized by the journal's lock. A version above the
  one this build knows is reported as StateStatus.NewerBuild and nothing is
  opened for writing; a negative one is refused.
- Step 1 is every schema change of the 1.x releases, each made only when it is
  missing: the two columns added to events, the job summaries built from the
  latest event, the creation time, the indexes, the control tables and the
  three tables that were created on first use. Any 1.x journal, and an empty
  one, ends in the same schema. Rows from before events carried a scope get
  the saved pairing's scope, as each 1.x open did.
- A store opens read-write without create and checks the version. On a journal
  that is missing or behind it fails with "The print journal has not been
  brought up to date"; on a newer one with the newer-build sentence.
  InitializeDatabase, InitializeControls, CreateJobIndexes,
  MigrateJobCreationTime, EnsureMetadataTable and the CREATE TABLE in front of
  the realtime-settings and trust statements are gone.
- The runtime migrates at the start of its loop. Until the next commit a
  journal from a newer build makes the service fail to start instead of
  serving its pipe.

Tests: TestState.OpenStore and SingleServerHarness migrate first, so test
files do not change, except four whose subject was that opening migrates.
LegacyStateBuilder builds the state an earlier release left from
Fixtures/journal-1.2.1.sql (transcribed from tag v1.2.1) and
journal-oldest.sql (the six-column events table the first commit's own ALTER
implies).

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: James Maes <james@kof22.com>

feat(runtime): on a journal from a newer build, start no connection and serve only the pipe e09b881545

An older build can be installed over a newer one between release candidates,
which share a numeric version. It then finds a journal at a schema version it
does not know. The service now leaves that state exactly as it found it and
keeps running so that the newer build can be installed again:

- RunAsync keeps what the migration reported. When the state cannot be used it
  logs why, starts no connection loop, and serves the management pipe alone.
- status and snapshot answer with one sentence, StateStatus.NewerBuildText:
  the snapshot is red, unpaired, with no printers and no jobs, and is built
  without a store. transport, get-log-settings, set-log-settings, clear-logs,
  prepare-update and resume-after-update work as usual; none needs a store.
- Every other operation is refused with that sentence before it is
  dispatched, and the runtime's OpenStore refuses too, so nothing reached
  another way can open a store on that state.

The runtime is otherwise unchanged: one field, a guard in RunAsync, OpenStore,
GetSnapshot and HandleCommandAsync, and the list of operations still answered.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: James Maes <james@kof22.com>

docs: describe the journal's schema version, its migration and the newer-build state 8dd94680f5

DEVELOPMENT.md gets a section on how the journal is versioned and migrated:
once at service start, in one transaction, never by opening a store; how a
step is added and why a shipped step is not edited; what step 1 covers; what
the service does with a journal from a newer build; how the version is read
without writing; and the fixtures tests build earlier releases' state from.
The paragraph on opening under the journal's lock names JournalDatabase.

The threat model's data-handling section says who changes the schema and what
a build does with newer state. The changelog records both changes, and the
audit marks W4 closed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: James Maes <james@kof22.com>

test(store): show that no store call recreates a missing table 230cf95418

Comparing sqlite_master before and after every store call cannot see a
CREATE TABLE IF NOT EXISTS for a table that is already there, which is
exactly what the store used to run in front of its realtime-settings, trust
and profile statements. A second test removes those tables, the control state
and an index from a migrated journal: a store still opens, each call that
needs a removed table fails with "no such table", and the schema stays as
damaged. With one of the old statements put back, this test fails and the
other does not.

Also: the test of a second migration wraps the product's own steps to count
their runs, so it stays valid when the step list grows; and one member
spacing is corrected.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: James Maes <james@kof22.com>

test(store): keep the store's type name out of LegacyStateBuilder 1195199876

Test files name the store only through TestState and its alias, so that a
rename of the product type changes one file. A comment in LegacyStateBuilder
named it; it now says "the store's SaveConfiguration".

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: James Maes <james@kof22.com>

docs(adr): record that schema version 1 adopts rows found under no scope, as 1.x did 1ffde935b5

ADR-0004 said that journal rows the migration finds under no scope belong
to no server. Version 1 of the journal's schema gives them the saved
pairing's scope, which is what every start from v0.1.6-rc.1 to v1.2.1 did.
The independent review of the change found the two in disagreement and
recommended correcting the record.

Decision taken on 2026-11-10 under the owner's standing instruction (take
the recommended option, record it, keep going): the code stays, and the
record says what it does and why. Only a journal last written by
v0.1.0-rc.1 or an earlier build has such rows, so no 1.x installation
reaches version 1 with any. After version 1 no journal has rows without a
scope, so the step that introduces several servers needs no rule for them.
The earlier sentence stays, marked as changed.

Signed-off-by: James Maes <james@kof22.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

test(store): show that the migration's steps and its version stamp are one transaction 1bbb185247

No test failed when the version stamp was moved out of the migration's
transaction, after the commit. A service that died between the two would
leave a journal with every step done and the old version, and the next
start would run the steps again; that is harmless for step 1 and not for
a later step that is not idempotent.

The new test migrates the oldest journal while a second connection keeps
the write-ahead log from being folded in, and counts the commit frames in
the log: one. With the stamp after the commit it counts two.
JournalProbe.CommitsInLog reads the log's frames.

Signed-off-by: James Maes <james@kof22.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

docs(audit): name the change that closes W4, not the branch it was built on 51ca0589f9

The row for W4 cited the branch name, which stops meaning anything once
the pull request is merged by squash and the branch is deleted. It now
says what the change is: the journal's schema version and its one-time
migration, plan step 5 of the 2.0 build. No row is renumbered or dropped.

Signed-off-by: James Maes <james@kof22.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

fix(store): escape each part of the path in the address of the read-only version read 8a81d1ead6

The immutable read of the journal's version is opened by a file URI, built
with Uri.AbsoluteUri. For a folder whose name holds a literal
percent-escape beside a letter outside ASCII ("c%20d é"), that leaves the
escape as it is, SQLite decodes it, looks for a folder named "c d é" and
fails with error 14 on the second start. The state folder under
ProgramData has no such name; a state folder elsewhere could.

The address is now the path with each part escaped by itself, so every
character reaches SQLite as the character it is. The test migrates and
reads back eight folder names with reserved characters; with the old
construction it fails on the third.

Signed-off-by: James Maes <james@kof22.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

fix(store): read the journal's version again inside the migration's transaction ab4b195bc4

The migration read the version through a read-only connection and applied
the pending steps afterwards, in a transaction that trusted that reading.
In one process the journal's lock covers both. Across processes only the
service's worker.lock does, so a second process could migrate between the
reading and the transaction, and its steps would be run a second time.
Step 1 is idempotent; a later step need not be.

The transaction now begins with SQLite's write lock (asked for by name,
not left to the library's default) and reads the version again under it.
That reading decides which steps run. A journal found at the current
version runs none; one found above it is reported as a newer build's and
nothing is written to it.

The test gives the migration a list of steps that acts when it is first
asked for its length, which is after the first reading and before the
transaction, and migrates the journal there through another connection.
With the version read only once, the newer build's journal in that test is
migrated down to version 1.

Signed-off-by: James Maes <james@kof22.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

test(store): open a store on a journal another program holds without sharing 9bf24faca1

The catch in JournalDatabase.Open, which disposes a connection that could
not be opened before the failure is passed on, was not covered by any test
and had been taken for unreachable. It is reached when the journal exists
and cannot be opened: here another handle holds journal.db with no
sharing, SQLite gives up after about a second with error 14, and the store
opens normally once the handle is released. Nothing in the repository
described the catch as unreachable, so no text changes.

Signed-off-by: James Maes <james@kof22.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

docs(store): a killed process leaves its write-ahead log behind, a stopped one does not 2a84a7106a

Three places said that a stopped service leaves its write-ahead log on
disk: the comment on the read-only connection string, the journal section
of DEVELOPMENT.md and a test comment. A process that exits in the ordinary
way is a different case: Microsoft.Data.Sqlite closes its pooled
connections on process exit, and SQLite folds the log into the journal
when the last connection closes. It is a killed process that leaves the
log with its latest changes. This is stated from the library's source and
SQLite's documented behavior; it was not observed on a stopped service.
No code changes.

Signed-off-by: James Maes <james@kof22.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

test(store): keep the review's upgrade evidence for 1.x installations as tests 093696acb5

The release depends on a 1.2.x installation upgrading in place. The
independent review showed three things by probe that no test in the
repository held; they are now tests in JournalUpgradeTests.

- A 1.2.1 journal left by a killed service, with every row only in the
  write-ahead log, comes out of the migration with the schema and the
  rows of a cleanly closed one. LegacyStateBuilder.WithJournalLeftByAKill
  builds that journal.
- Six shapes between the oldest journal and 1.2.1, among them 1.0.x
  without the certificate exception table and a 1.2.1 journal that never
  used a first-use table, end in the one schema and keep their rows.
- A 1.2.1 state folder under the service loop, with one job left in
  SUBMITTING and one in RECEIVED: the start migrates the journal, records
  the first as OUTCOME_UNKNOWN and the second as FAILED, prints nothing
  again, creates no session, and shows the pairing, the history, the
  shared printers, the names, the control state and the certificate
  exception as 1.2.1 left them.

No product code changes.

Signed-off-by: James Maes <james@kof22.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

feat(runtime): say which journal could not be brought up to date, and that it was not changed 4320224934

When the migration failed at start the log had SQLite's bare message: no
journal path and nothing about the state the journal was left in. That was
also the case before there was a migration.

The runtime now logs one line before the failure is passed on: the
journal's folder, the kind of failure in an operator's words with SQLite's
error number (the file could not be opened or written, another program has
it in use, it is damaged or not a journal, the disk is full), that the
journal was not changed, and what to check before starting the service
again. The failure's own message is never repeated in that line, so it
cannot carry a row, a credential or pairing data. The service still fails
to start, and the failure still reaches the host as it did.

Tests: a paired, signed-in computer whose journal is not a database (the
line, the unchanged bytes, no session), and the text for each kind of
failure with a message shaped like a credential and a row. With the bare
message logged instead, the first fails.

Signed-off-by: James Maes <james@kof22.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

fix(runtime): log an operation refused on a newer build's data as debug detail, not as an error 8adb2605b5

While the journal belongs to a newer build, every refused operation was
answered through the path for a failed command, which writes an ERROR
line. The reason is already logged once, as an error, when the service
starts. A companion that keeps asking added an error line for each
request, and the service's log keeps only the newest 100 entries for the
snapshot, so the one line that says why could be pushed out.

A refusal is now answered directly, with the same reply as before, and
logged at DEBUG: to the log file, and only while debug logging is on. The
operations answered and refused are unchanged, and so is every reply.

The newer-state test now requires that the reason is in the log exactly
once, that the snapshot's entries hold no error after all eighteen
refusals, and that each refusal is in the log file as debug detail. With
the refusals logged as errors it fails.

Signed-off-by: James Maes <james@kof22.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

build(coverage): raise the Windows assembly's minimum line coverage to 87% b8d0e70ad5

build/Verify.ps1 on this branch, replayed onto the release tip and with
the review fixes, measured the Windows assembly at 88.06% (3,963 of 4,500
lines). The minimum is the measured value rounded down, less one point:
87, up from 86. JournalDatabase.cs and JournalSchema.cs have no uncovered
line. Core, the presentation assembly and the command line stay at 100.

Signed-off-by: James Maes <james@kof22.com>
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>

james.maes merged commit ebea8613eb into release/2.0

2026-11-11 04:16:27 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: QRun-IO/qqq-print-agent#49