4 Commits
Author SHA1 Message Date
james.maesandClaude Opus 5.5 415d0bdd0a feat(ddl): indexes for the job queue, dedupe, live sessions and retention
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 22:56:01 -05:00
james.maesandClaude Opus 5.5 89e468ca45 feat: endpoints, outbox, PGP, app home and retention (phase e)
Phase (e) of docs/DESIGN.md:
- endpoints (aliases) with path validation and the ENDPOINTS_ONLY chroot;
  BULK_LOAD, INBOX (consumer by name), VIEW, REPORT and OUTBOX kinds;
- the outbox provider (FileGatewayOutbox.put) in every deployment, outbox
  files, pickup tracking and onDownloaded;
- optional per-account PGP: decrypt uploads, encrypt downloads (keys in
  memory, BouncyCastle);
- the typed app home (KPIs, recent failures, top partners, active
  sessions, gateway servers via file_gateway_server heartbeats);
- Unlock/Disable account, Revoke key and End session record actions;
- the daily retention process; FileGatewayIntegrations (ESB, Quick
  Search, API);
- docs/k8s example manifests (SFTP port 22 only, LB IP 10.100.22.65);
- plain labels for fields, enum values and child tabs; a test-scope demo
  server for screenshots.

88 tests; JaCoCo 92% instructions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 22:52:41 -05:00
james.maesandClaude Opus 5.5 23896eb7f8 feat: upload to bulk load through object storage (phase b)
Phase (b) of docs/DESIGN.md:
- SFTP puts into upload/<profile>/ stream to qbit-object-storage with
  SHA-256 and size computed in flight, a bounded in-memory reorder window,
  size and daily quota enforcement, temporary-suffix and rename handling;
- upload jobs: dedupe, optional scanner (quarantine), a bounded worker pool
  with exclusive claims, stale requeue and the 1m..6h retry schedule;
- the consumer pipeline (FileGatewayConsumersMetaData) with the built-in
  bulk-load consumer, which runs <table>.bulkInsert as the user through
  /qqq/v1 with the saved profile (fails when value mappings are missing);
- sidecars (.processing/.done/.error), results CSVs and the archive, all
  in object storage, readable over SFTP; rm acknowledges a job;
- QAlerts.notify on failures (and success when asked);
- the "Retry upload" record action.

64 tests; JaCoCo 93% instructions, 97% classes; no-disk guard on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 21:55:43 -05:00
james.maesandClaude Opus 5.5 1cd1caa8c5 feat: SFTP server, virtual tree, SSH-key auth and listing (phase a)
Phase (a) of docs/DESIGN.md:
- the seven file_gateway_* tables (entities, PVSs, joins, child lists,
  DDL), contingent possible-value filters on endpoint profile/view/report,
  SSH key validation with server-side fingerprints, permissions
  file_gateway.view/admin/connect, the sftp client channel and walls;
- Apache MINA SSHD 2.16 SFTP server: publickey only (no password or
  keyboard-interactive), no shell/exec/forwarding, restricted algorithms,
  in-memory host key from an env var name;
- SshKeyAuthenticator: account and key checks, CIDR allowlists, expiry,
  DB-backed lockouts with an alert, per-IP rate limit, concurrent session
  cap, live revocation; sessions minted with createAutomatedSessionForUser
  and registered as UiApiInProcessSessions, requiring file_gateway.connect;
- a virtual NIO filesystem over /qqq/v1 (channel sftp): apps, tables,
  upload/<profile>/, views/, reports/, results/, archive/; writes and
  reads denied until phases (b) and (c);
- connection and transfer logs;
- the com.kingsrook.qbits.filegateway.api consumer/outbox contract types.

54 tests including a no-disk guard; JaCoCo 95% instructions, 98% classes.
Builds against QRun-IO/qqq#1170 (UiApiInProcessSessions).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 21:09:03 -05:00