qbit-mobile-scanners/docs/OPERATIONS.md

Single-instance deployment and recovery

Written 2026-11-05 from the static final review of source base a9eae04 (see review reconciliation). The procedure below is for the host/operator in a real incident. The only executed evidence is the automated isolated local-demo recovery acceptance, which covers the demo's stopped H2 database plus object snapshot and restore. No production storage, enterprise object-store or high-availability deployment is supported or qualified, and those are not tasks of this scanner release. The supported model is one server process per database.

Topology

• Run exactly one server JVM per database, with no overlap during an upgrade (stop the old process before starting the new one). Revocation and capture-commit exclusion use a per-JVM monitor, screen-mirror frames are held in memory only, and Echo job admission relies on a row lock in that one database.
• H2 must be used single-process. Do not point two servers at one database.
• The host owns HTTPS and the identity provider; the loopback HTTP demo is not a deployment (production readiness).

Backup and restore

• Take one consistent snapshot of the database and any file storage together (same moment, or with the server stopped). A database restored without its matching files, or the reverse, leaves dangling references.
• Treat bindings and backups as sensitive. mobileSessionBinding.coreSessionUuid is stored in plaintext at rest (reads are blocked by the application, not by the backup). Anyone holding a backup holds live session identifiers until they expire or end. Backups also contain barcode payloads and last-known device location.
• After any restore, before reopening to users:
  1. Stop all server processes, then restore the database and files from the same snapshot. Nothing can be done through the application while every server is stopped, so steps 2 and 3 need a running server.
  2. Restart in controlled, admin-only access (for example, reachable only by administrators, with scanner clients and other users kept off the network path) and keep it that way through steps 3 to 6. Do not reopen to users first: a device revoked after the snapshot may be approved again and a session ended after it may be active again.
  3. Re-audit device approvals. A restore reverts revocations, approvals and enrollments made after the snapshot, so a device revoked later may be approved again. Review every device and re-revoke as needed.
  4. End or reconcile active sessions. Tombstones written after the snapshot are lost. With the server running in that controlled state, apply the End Session process to the restored sessions so cleanup runs against the restored state, and check the cleanup status below.
  5. Devices enrolled after the snapshot no longer exist and must re-enroll.
  6. Echo: jobs accepted after the snapshot may be absent from the database although a print was already queued. Manually reconcile the physical or Printing queue against the restored jobs before authorizing any retry, and retry only what that comparison shows was not printed. This is a manual check; there is no exactly-once physical print guarantee, and a retry can still produce a duplicate or a missing print if the queue or printer state is unknown.
  7. Reopen to users only after the above are done.

Isolated local-demo recovery acceptance (automated, passed)

EchoDemoAcceptanceTest.restoredSnapshotKeepsScanAndRetainedFileAndRevertedApprovalStateIsReaudited (Echo composed demo) exercises this procedure on private temporary directories only.

Verification (2026-11-05): the combined five-module verify on the merged source (develop 9e8685f plus the storage fixes) passed with qualified private owner dependencies: 196 tests (root 53, demo 8, Record Files 22, Echo 74, Echo demo 39), 0 failures, 0 errors, 0 skips, and zero Checkstyle violations in every module. The qualified owner inputs are private, so this is not public-dependency qualification.

The case restores H2 scan rows, the exact leading-zero value and symbology, and the Record Files owner's own RecordFilesTables.FILE row. It asserts that row's exact table, record and attachment relationship and the SHA-256 of the object bytes. It shows that a revocation and session end made after the snapshot are reverted by the restore, then reapplies the normal Revoke and End Session processes and shows that a valid in-flight run and a new initialization are denied. No physical printer, service or demo data was changed.

Flow: it enrolls and approves a synthetic device, scans over real HTTP, and lets the real Record Files service retain the generated original in the demo local-file adapter. With the servers stopped and H2 shut down, it copies the database and object directory together, makes later changes (a second scan and file, a revocation and an ended session), restores only those two temporary directories and reads them without the application. It then restarts on a new ephemeral loopback port and applies the existing Revoke and End Session processes.

What this is not: it covers the local demo's H2 database and local files only, using private coherent fixture inputs. It is not published-cohort qualification (that is repeated when the final public dependencies are adopted), and enterprise object-store and high-availability recovery are unsupported and unqualified. In a real incident the full procedure above still requires an operator-owned audit of approvals, sessions and the print queue before a restored service returns to users.

Storage regressions in PrivateObjectFilesTest (part of the Echo demo's 39 tests): concurrentSameKeyWritersLeaveOneWinnersBytesAndMetadata (concurrent same-key writers leave one winner's bytes and metadata), dotBucketsAreRefusedAndValidBucketsStillWork (. and .. buckets refused), plus existingObjectIsNeverReplaced and keysCannotTraverseOrAlias.

Limits: the file store is the single-host demo adapter and the database is the demo's H2 file, so this does not qualify production storage, an object store, encryption or high availability. Other database backends are not supported or tested here. The renderer is controlled and the printer is a database fixture; no job is physically printed and no normal demo data is changed. The file check reads the Record Files rows and object bytes directly, not through a product service.

Session cleanup status

Ending a session (MobileSessionService) first persists the ended state with authCleanupStatus = PENDING. Cleanup then sets REQUESTED (with authCleanupAt) before calling the logout executor and removing the stored core session. If that call throws a QException, the status becomes RETRY_NEEDED. So REQUESTED means cleanup was attempted and no failure was reported; it is not independent proof that the provider session is gone. Cleanup is skipped when a binding does not exist, and for a single end inside an open transaction; a status left at PENDING also needs a retry. The bulk End Session process commits all tombstones first and runs cleanup afterwards.

To retry: find sessions with authCleanupStatus of RETRY_NEEDED or PENDING (admin view of the session table), then run End Session again for them. Ending an already ended session does not change its end time and reruns cleanup when a binding exists. Confirm the status reaches REQUESTED. There is no automatic monitor in this repository; the operator must check for these statuses.

Retention

Scans, sessions, bindings and Echo scans are kept indefinitely by default. Table deletes are blocked by permissions and foreign keys tie history to devices. Barcode payloads and last-known location are retained, so a different retention period is a host-owned policy, not a feature of this release: any purge is a host DBA procedure, taken with a snapshot first and respecting the foreign keys. Audit is current-state only; revoking overwrites the previous approver and time.

Database migration beyond H2

The supplied, tested scripts are H2: base/src/main/resources/db/mobile-scanners-h2.sql and the Echo echo-h2.sql. Other backends are unsupported and untested here; a host that uses one owns the equivalent migration; do not assume ADD COLUMN IF NOT EXISTS or the column types. Additive columns that older databases need before running current servers: on mobileScan, nullable aimIdentifier (VARCHAR(3)) and rawDataBase64; on Echo, nullable echo_scan.aimIdentifier, rendererValue and humanReadable, and echo_run.copies, which is INT NOT NULL DEFAULT 1 (existing runs keep one copy), so it is not nullable and the equivalent migration must supply the default. Echo also has nullable template columns (templateId, templateRevisionId) in echo-h2.sql. Test the migration on a copy, and snapshot first.

Demo-only file adapters

LocalFilesystemAttachmentStorage and LocalStagingChunkStorage (Echo demo) are single-host local-file adapters for the demo. They are not production storage; photo and record-file storage for a deployment is not qualified here. ECHO_RELEASE_GENERATED_FILES=true turns off generated-file quarantine and exists for the demo only; leave it off unless every producer permitted to keep generated output is trusted (Echo demo README).