qbit-scripting/docs/WARM-LIFECYCLE.md

Per-node warm code lifecycle

Native warm lifecycle verification completed 2026-10-31 at11:40:55: parent handle45711 terminal0, full192 tests with zero failures/errors/skips, Checkstyle0 and unchanged coverage gates PASS. Focused42 (lifecycle14/TLS17/authoring11) passed under normal loopback access. Main SHA256 89B52743EC2EA4CB5670D69BF5927D632FC80A660E4A1093101AE2E3CC07506B; tests SHA256 D907AA9CC171A33A7ECC9359D16A4E2ECF980520E7E10073BCD497855F89221C. No install, shared-cache mutation or deployment occurred. Independent review accepted the five scoped corrections; runtime/supervisor deployment qualification remains separate. The immutable173 proof remains historical and untouched. Earlier handoff notes below are retained as history.

PersistentWorkerProvider implements the optional WarmWorkerMaintenance interface. When the producer sees that interface it registers native QRuntimeServiceInterface and readiness health metadata. Other providers retain their existing behavior. The cache provider is not ready without explicit WarmReconciliationConfig and a declared native service principal.

The host supplies a principal name, explicit target-table set, maximum active bindings (and every configured target table must exist), maximum unique code bytes, fixed delay, whole-pass budget and maximum confirmed age. The service owns one daemon scheduled future per QQQ instance, with coalesced non-overlap and cancellation on native stop. It closes its provider only at afterApplicationStop. No Jobs queue, new registry, record store, global timer or caller session is created.

Declare the principal through native ServiceIdentitiesMetaData; grant scripting.warm, scripting.view, current native access to the script projects and read access under any credential/channel ceilings. WARM permits bounded code-metadata maintenance only; it does not imply execution, publication, binding or record privileges. Scope the principal's project security keys narrowly. Each pass and readiness check obtains a fresh native service identity in a temporary context with no inherited transaction, authorized action scope or user backend variants. There is no system-session fallback, user delegation or reuse of binding creators' permissions. Ordinary invocation still requires current initiating-user and broker authority.

An inventory query reads enabled bindings only for the host's explicit target scope, with maxBindings + 1 limit. Overflow refuses rather than silently truncating readiness. Every selected binding must have current project visibility and an intact publication/build/artifact association verified by the existing HMAC coordinator. The service bounds aggregate unique code bytes, keeps immutable code only during the pass, and stores only bounded identity digests afterward. A changed binding, publication, signed artifact, service declaration, context or policy during preparation invalidates readiness. No source record values or capability tokens are sent to preparation.

Cold PING has exactly protocol/type/session/sequence/requestId. The authenticated peer replies with exact PONG then normal RETIRED(true), without guest allocation, code transfer, inventory or cache mutation. The same live connection/session must retain its ready-code epoch; a supervisor that loses its cache must close the session. Successful PREPARE/INVOKE retirement also refreshes confirmation age. Busy execution slots are skipped, never stolen by heartbeats. Missing code is transferred only on the cold maintenance lane. Ordinary execution never opens a connection, prepares or compiles on a cache miss.

Readiness independently re-reads current required scoped pins using a fresh metadata identity and verifies all exact local ready IDs/copy counts/profile under one cache lock and one timestamp, plus bounded acknowledgment age. A final metadata re-read follows maintenance callbacks before readiness is accepted. It does not prepare or execute code. An unavailable principal, hidden required project, metadata overflow, changed pin, expired acknowledgment, missing worker or stopped lifecycle reports a generic DOWN outcome with no code/identity details. Readiness concerns this configured scope, not arbitrary inactive publications or other QQQ instances.

Native database connect/socket/query timeouts and bounded authentication-provider/DNS behavior are host deployment prerequisites. Cooperative deadline checks and cancellation cannot interrupt an unbounded JDBC handshake, authentication call or operating-system DNS resolver. No additional resolver threads are introduced. Real supervisor session/cache-loss behavior, cold compiler availability, multiple-node startup/restart, workload capacity and hostile-runtime qualification remain separate deployment gates.

Lifecycle restart scope: a stopped lifecycle/provider is terminal. Application restart must construct a fresh ScriptingRuntime and provider; arbitrary stop/start reuse of the same objects is not supported or qualified. Worker connection/cache loss during a running lifecycle is handled by cold reconnection/reconciliation.

Independent corrected-source review accepted the five scoped corrections (native health timing, target existence, final pin check, aggregate readiness atomicity and volatile lifecycle publication). The new TLS aggregate regression checks partial retirement and recovery; it does not force a concurrent interleaving. Atomicity is established by the single existing slots lock and timestamp in source. Focused42 execution remains pending; no passing result is inferred from review.

Verification update: parent handle96235 completed with42 focused tests, zero failures/errors/skips and Checkstyle0 (lifecycle14, TLS17, authoring11). Full main handle45711 is running against the same frozen source/private cache with normal loopback access; no full-suite success or install is claimed yet. Prior failed style/compile/native-metadata gates remain preserved.

Cold-start scaling: steady-state reconciliation performs a fixed number of bounded full inventory/signature scans, linear in the selected bindings plus code bytes. Every newly cold artifact requires additional complete pin/current-project validation immediately before and after transfer. First-cold startup can therefore cost substantially more (up to quadratic metadata work within the configured count and cooperative pass budget). Partial prepared code is reusable on a later pass; no throughput/startup latency/SLA claim is made. Choose bounded scope, capacity and pass budgets using actual deployment measurements. Native JDBC connection/socket/query and authentication/DNS deadlines must be finite independently; thread cancellation is not a substitute.

Final verification: Native warm lifecycle verification completed 2026-10-31 at11:40:55: parent handle45711 terminal0, full192 tests with zero failures/errors/skips, Checkstyle0 and unchanged coverage gates PASS. Focused42 (lifecycle14/TLS17/authoring11) passed under normal loopback access. Main SHA256 89B52743EC2EA4CB5670D69BF5927D632FC80A660E4A1093101AE2E3CC07506B; tests SHA256 D907AA9CC171A33A7ECC9359D16A4E2ECF980520E7E10073BCD497855F89221C. No install, shared-cache mutation or deployment occurred. Independent review accepted the five scoped corrections; runtime/supervisor deployment qualification remains separate.

Immutable proof: warm-lifecycle-20261031-114806,116 sealed files/92 source inputs. Manifest SHA256 3B9ACBD28461EDD5FDA0DD17542FC7956FCB06AE523BB52E8C866579E7CA8757. The snapshot includes filtered summaries and hashes of all six preserved raw gate logs; prior173 evidence is unchanged.

Additional context publications (staged; verification pending)

A restart previously restored only active hook bindings. Optional ScriptingRuntime.withWarmPublicationSource(WarmPublicationSource) adds other required immutable CONTEXT publication pins to the same cold inventory. Its publications(Set<String> targetTables, int limit) runs under the freshly declared WARM + VIEW metadata identity, not a publication author or invoking user. The callback must authorize its own current owner metadata, return every required positive distinct publication/artifact/SHA-256 pin, and throw on uncertainty, denied metadata or overflow. An empty list means an authoritatively empty additional inventory; null is an error. The supplied limit is the remaining combined admission budget, possibly zero. No dependency on a consuming QBit is introduced.

The foundation independently checks current script-project access/isolation, exact publication/build/artifact identity, signed code and a successful TEST_CONTEXT on that exact build. It does not grant EXECUTE to the metadata identity. maxBindings retains its configuration name and now bounds active hook rows plus additional distinct publication IDs; identical code is counted/transferred once by hash/entry/toolchain within the shared byte budget. A publication used by both inventories consumes two pin slots but one code allocation. Removed consumer pins stop contributing to readiness; existing worker capacity/epoch eviction remains unchanged.

The callback object, native context and authority are checked before/after external work. Existing final snapshot rereads and atomic aggregate readiness cover both inventories. Health may read owner metadata but never warms; execute/save hooks never enumerate, connect, prepare or retry. Source changes do not affect immutable publications; consumer pin changes invalidate readiness until cold reconciliation. Owner queries, like native JDBC, need finite deadlines independently of the cooperative pass budget.

Eleven added protocol-fixture lifecycle tests cover unbound cold startup/reconnect, no hot preparation, changed/removed pins, malformed/uncertain inventories, context qualification/current project access, HMAC/missing artifact refusal, combined count/byte limits, code deduplication and callback authority/final-snapshot changes. They are staged for the root-owned gate; no new passing result or deployed-runtime qualification is claimed. The 192-test sealed baseline above is historical and unchanged.

Capacity reclamation and authoring use (2026-11-01, verified source)

Cold maintenance now retires idle sessions containing code outside the exact required inventory. A session is the existing cache epoch: removing one discards all its entries, including any still required entries. The same pass must reprepare and revalidate the complete inventory before UP. Busy invocation sessions are never stolen. No hot execute/readiness path connects, prepares, retries or performs reclamation. Changed inventory or maintenance authority aborts the pass.

WarmWorkerPreparation.prepareForUse(Artifact) returns an AutoCloseable local UseLease. Authoring test/context-test/publish/bind/enable holds it from cold preparation through dispatch or commit; every success/failure closes it. Persistent provider admission is immediate and bounded by Configuration.coldOperations (no wait queue). Code protection uses the existing provider's slot lock and bounded reference counts; there is no durable registry or timer. All sessions holding leased code are ineligible for cold retirement. A failed preparation releases its reservation; release is idempotent and performs no I/O. Existing non-reclaiming providers retain the compatible default preparation implementation. Actual transport/cache loss can still make execution unavailable.

Operators need capacity for required pins plus concurrent authoring code and finite native JDBC/ transport deadlines. A long synchronous authoring operation retains its bounded lease until it returns; maintenance may defer reclamation or remain DOWN meanwhile. No lease expires underneath an active operation. First-cold rebuilding can transfer the required set again; there is no claimed latency or throughput guarantee. Parent full Scripting227 and Generation focused67 gates passed (zero failures/errors/skips, Checkstyle0, scripting coverage PASS), including TLS and actual H2 migration/replay. The inherited MySQL migration case remains separate. The earlier sealed192-test result remains evidence only for that earlier snapshot. See Generation docs/warm-capacity-source-handoff.json for source/log hashes.

Bounded idle-heartbeat skip and local development capacity

WarmWorkerMaintenance.heartbeat(budget, confirmedWithin) lets the native lifecycle avoid a wire PING for an idle session whose last completed exchange already confirmed the same cache epoch. Only an open, idle slot with no waiting execution whose PREPARE, PING or INVOKE finished with RETIRED inside the window is skipped. Unconfirmed, stale, busy, closed or partial slots are never treated as ready, a late or incomplete PONG is still a failure, deadlines are unchanged and no invocation is retried. The default method performs the full heartbeat, so existing providers are unchanged. WarmReconciliationConfig.heartbeatSkipWindow() supplies the window: min(maxAge / 4, maxAge - interval - passBudget), never negative, so a skipped slot can still be refreshed before readiness age expires. Readiness maxAge checks are unchanged.

Local Docker Desktop finding (not a production claim): a trivial JSSE TLS 1.3 echo through a Docker-published loopback port showed roughly 10% of 1 Hz exchanges delayed by about 300 ms (some seconds), while the same exchange with plain TCP, or TLS directly on the host, showed none. Retransmit-counter correlation and controls are in the runtime lane status; no packet capture was possible. The supervisor slots=1 / idleMillis=60000 worker configuration also meant that one dropped host session held the only engine for up to a minute. The documented local development worker uses slots=2, connections=4, idleMillis=15000 (heartbeat cadence stays well below the idle limit). More slots do not replace the handoff/heartbeat code and are not a qualification.