diff --git a/AGENTS.md b/AGENTS.md index 2d799ac..a25cc23 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,18 +14,68 @@ code and QInstance. Design: `docs/DESIGN.md`. ## Build commands ```bash -./mvnw compile -./mvnw test -./mvnw verify # Checkstyle; JaCoCo: 70% instructions, 90% classes +./mvnw -o -Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT compile +./mvnw -o -Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT test +./mvnw -o -Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT verify # Checkstyle; JaCoCo: 70% instructions, 90% classes ``` -Use the Maven 3.9.11 wrapper (Java 21); Maven 3.10 cannot read the parent POM. On Windows use `./mvnw.cmd`. +Use the Maven 3.9.11 wrapper (Java 21); Maven 3.10 cannot read the parent POM. On Windows use `./mvnw.cmd` and quote the `-Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT` argument in PowerShell. +Keep the POM default at `4.1.0-SNAPSHOT` for CI; the private version prevents local artifact replacement. ## Architecture Package: `com.kingsrook.qbits.filegateway`. Entry point: `FileGatewayQBitProducer implements QBitMetaDataProducer`. +- `metadata/`: seven stable tables, enum/picker sources, joins, typed child widgets, key/account + validators and streamed Review/Result admin actions. Producer adds metadata to the QInstance. +- `model/`: QRecordEntity mappings; DDL in `docs/ddl/file_gateway.sql` uses UTC zone-less timestamps. +- `permissions/`: view/admin/connect constants and the custom permission checker; system-written + logs/jobs/outbox tables remain read-only to people. +- `FileGatewayRuntime`: host-owned startup after the UI API server, in-memory host-key parsing, + injectable environment and clock/test hooks. Production polling interval is 60 seconds. +- `sftp/`: public-key-only MINA listener, per-account transactional authentication, per-pod IP + windows, token refresh/revocation and system-session bookkeeping restricted to owned tables. +- `vfs/`: custom NIO filesystem; per-session UI API metadata/saved-object caches. Every API call + checks its operation spec and sends user credentials plus `sftp/` client detail. +- `api/`: isolated consumer/outbox contract. Never import gateway implementation packages here. + +Phase (b) adds `upload/` (bounded streaming channels, database quota reservations, worker claims, +scanner/consumer pipeline and UI API bulk load) and `storage/` (the object-storage API adapter +and ranged read channels). Production uses object storage; the memory implementation is tests-only. +The runtime starts a bounded worker pool with SFTP and releases owned claims on stop. Sidecars, +results and archive are readable; sidecars/results can be acknowledged. Retry upload is a +standard Review/Result record action. The upload-job schema now includes `results_size_bytes`. + +Claims compare expected status under an account-row transaction; core's optimistic version check +alone does not add a SQL WHERE guard. Fresh worker user sessions outlive SSH connections and are +revoked after delivery. Bulk process multipart field: `theFile`; mapping frontend step: `fileMapping` +(resumes `receiveFileMapping`). RC.2 object storage owns S3 part size at connection level. The +private invoker supports multipart process values without a file part, so the mapping request +sends only its values. See README adoption prerequisites and compatibility notes. + +Phase (c) adds `GeneratedDownloads` (user-scoped export/render/download API flow and durable +account-locked download quotas) and `GeneratedReadChannel` (fixed rolling window, no spool). +Saved-view JSON uses `queryFilter.orderBys` and `queryColumns.columns` visibility/order. Generated +stat size is zero; reads fill requests until real EOF to avoid MINA's short-read EOF inference. +Reports list only with a configured render-process storage table. FieldSecurityLock DENY tests +require `enforcePermissionsInCoreActions=true` and match direct same-user UI export bytes. +Core `RenderSavedReportExecuteStep` stages a rendered report in a JVM temp file and deletes it +in try-with-resources; that is the bounded temp area the owner rule allows (in-memory `/tmp`). +The core-valid alert type is `filegateway.lockout` (dotted names only). + +Phase (e) adds endpoints and chroot (`AliasCustomizer` validates paths), `outbox/` (the +`FileGatewayOutboxProvider`, registered in every deployment), `pgp/` (BouncyCastle stream +decrypt/encrypt, keys in memory), `maintenance/` (daily retention process and record-action +effects), `FileGatewayHome` (typed STATISTICS and TABLE widgets only; guardrail G5), +`FileGatewayIntegrations` (ESB, Quick Search, API lists) and `file_gateway_server` heartbeats. +The Preferences "File transfer" section waits for QRun-IO/qqq#1162. The k8s example is in +`docs/k8s/`. + +Tests run a strict UI API server, H2 from the adopter DDL and real MINA clients. NoDiskExtension +snapshots the working tree (excluding target) and isolated test home, and asserts the test temp +directory is empty. The host supplies RDBMS; `qqq-backend-module-rdbms` must remain test-scoped. + ## Invariants and naming - Data access only through `/qqq/v1` (`UiApiInvoker`, channel `sftp`) as the signed-in user. The diff --git a/CHANGELOG.md b/CHANGELOG.md index 89a967a..6339a1c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,4 +6,51 @@ All notable changes are documented here, following [Keep a Changelog](https://ke ### Added -- Initial scaffold +- Phase (c) generated saved-view and saved-report downloads through the user-scoped UI API, + CSV/XLSX/JSON account formats, a bounded rolling read window and HTTP stream cleanup. +- Generated-file and UTC daily-download quotas, persisted byte charges under account locks, + per-file VIEW/REPORT transfer auditing and actionable SFTP API/limit errors. +- Real MINA download tests for filters/column order, POI XLSX validation, JSON, FieldSecurityLock + DENY parity, report rendering, hidden tables/storage, large streaming exports and quotas. +- Phase (b) bounded streaming uploads through the object-storage system API, SHA-256, concurrent + byte quotas, temporary-name promotion, dedupe and transfer auditing. +- Bounded upload workers with guarded claims, fresh user sessions, scanner/quarantine hooks, + named consumers, retry scheduling, stale recovery and shutdown requeue. +- Built-in bulk-load consumer using only spec-defined UI API operations, saved mappings, + inserted/error summaries, per-row examples, notifications and a Retry upload record action. +- Readable sidecars, results CSV and ranged archive downloads with account isolation and + acknowledgement. Added `results_size_bytes` to the upload-job schema. +- Real MINA upload acceptance coverage, including permissions, auditing, limits, reordering, + consumer retries, quarantine, worker concurrency and the unchanged no-disk guard. + +- Phase (a) SSH-key-only SFTP runtime with in-memory host keys, explicit modern algorithms, + account/key expiry, numeric CIDR allowlists, transactional account lockout/session limits, + per-IP limits, user-session refresh and live revocation. +- Virtual listing/stat of permitted apps, tables, saved bulk-load profiles, views and reports + through the strict UI API invoker, with per-session caches and paged saved-object queries. +- Seven gateway tables and record entities, H2/PostgreSQL DDL, enum sources, dependent endpoint + pickers, joins, typed child lists, permissions and channel walls. +- SSH key/account validation and Review/Result actions to unlock/disable accounts, revoke keys + and end sessions; connection and operation auditing, including denied mutations. +- Isolated consumer/outbox API contract and complete configuration for subsequent phases. +- Real SFTP/strict UI API acceptance tests, no-disk guard, schema round trips and adoption docs. + +- Endpoints (aliases) with path validation and ENDPOINTS_ONLY chroot; BULK_LOAD, INBOX, VIEW, + REPORT and OUTBOX kinds. +- The outbox provider (`FileGatewayOutbox.put`), outbox files, pickup tracking and `onDownloaded`. +- Optional per-account PGP decryption of uploads and encryption of downloads. +- The typed app home, `file_gateway_server` heartbeats, Unlock/Disable/Revoke/End session record + actions, the daily retention process and `FileGatewayIntegrations`. +- Kubernetes example manifests in `docs/k8s/`. + +### Compatibility + +- Local invoker integration builds use `4.1.0-FG-SNAPSHOT`; CI retains the POM's + `4.1.0-SNAPSHOT` default. RDBMS remains a host-provided runtime dependency. +- Alert type `filegateway.lockout` uses the dotted syntax required by core validation. +- Object-storage RC.2 controls S3 part size through its connection configuration. +- The Preferences "File transfer" section waits for QRun-IO/qqq#1162. +- Private core #1170 sends multipart process values without re-uploading the original file. +- FieldSecurityLock requires core permission enforcement enabled. Saved-report rendering still + uses a temporary file inside core; fully disk-free report generation needs a core change. + The report folder is hidden without the render process's configured storage table. diff --git a/CLAUDE.md b/CLAUDE.md index bbff6f7..252b732 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,18 +14,68 @@ code and QInstance. Design: `docs/DESIGN.md`. ## Build commands ```bash -./mvnw compile -./mvnw test -./mvnw verify # Checkstyle; JaCoCo: 70% instructions, 90% classes +./mvnw -o -Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT compile +./mvnw -o -Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT test +./mvnw -o -Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT verify # Checkstyle; JaCoCo: 70% instructions, 90% classes ``` -Use the Maven 3.9.11 wrapper (Java 21); Maven 3.10 cannot read the parent POM. On Windows use `./mvnw.cmd`. +Use the Maven 3.9.11 wrapper (Java 21); Maven 3.10 cannot read the parent POM. On Windows use `./mvnw.cmd` and quote the `-Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT` argument in PowerShell. +Keep the POM default at `4.1.0-SNAPSHOT` for CI; the private version prevents local artifact replacement. ## Architecture Package: `com.kingsrook.qbits.filegateway`. Entry point: `FileGatewayQBitProducer implements QBitMetaDataProducer`. +- `metadata/`: seven stable tables, enum/picker sources, joins, typed child widgets, key/account + validators and streamed Review/Result admin actions. Producer adds metadata to the QInstance. +- `model/`: QRecordEntity mappings; DDL in `docs/ddl/file_gateway.sql` uses UTC zone-less timestamps. +- `permissions/`: view/admin/connect constants and the custom permission checker; system-written + logs/jobs/outbox tables remain read-only to people. +- `FileGatewayRuntime`: host-owned startup after the UI API server, in-memory host-key parsing, + injectable environment and clock/test hooks. Production polling interval is 60 seconds. +- `sftp/`: public-key-only MINA listener, per-account transactional authentication, per-pod IP + windows, token refresh/revocation and system-session bookkeeping restricted to owned tables. +- `vfs/`: custom NIO filesystem; per-session UI API metadata/saved-object caches. Every API call + checks its operation spec and sends user credentials plus `sftp/` client detail. +- `api/`: isolated consumer/outbox contract. Never import gateway implementation packages here. + +Phase (b) adds `upload/` (bounded streaming channels, database quota reservations, worker claims, +scanner/consumer pipeline and UI API bulk load) and `storage/` (the object-storage API adapter +and ranged read channels). Production uses object storage; the memory implementation is tests-only. +The runtime starts a bounded worker pool with SFTP and releases owned claims on stop. Sidecars, +results and archive are readable; sidecars/results can be acknowledged. Retry upload is a +standard Review/Result record action. The upload-job schema now includes `results_size_bytes`. + +Claims compare expected status under an account-row transaction; core's optimistic version check +alone does not add a SQL WHERE guard. Fresh worker user sessions outlive SSH connections and are +revoked after delivery. Bulk process multipart field: `theFile`; mapping frontend step: `fileMapping` +(resumes `receiveFileMapping`). RC.2 object storage owns S3 part size at connection level. The +private invoker supports multipart process values without a file part, so the mapping request +sends only its values. See README adoption prerequisites and compatibility notes. + +Phase (c) adds `GeneratedDownloads` (user-scoped export/render/download API flow and durable +account-locked download quotas) and `GeneratedReadChannel` (fixed rolling window, no spool). +Saved-view JSON uses `queryFilter.orderBys` and `queryColumns.columns` visibility/order. Generated +stat size is zero; reads fill requests until real EOF to avoid MINA's short-read EOF inference. +Reports list only with a configured render-process storage table. FieldSecurityLock DENY tests +require `enforcePermissionsInCoreActions=true` and match direct same-user UI export bytes. +Core `RenderSavedReportExecuteStep` stages a rendered report in a JVM temp file and deletes it +in try-with-resources; that is the bounded temp area the owner rule allows (in-memory `/tmp`). +The core-valid alert type is `filegateway.lockout` (dotted names only). + +Phase (e) adds endpoints and chroot (`AliasCustomizer` validates paths), `outbox/` (the +`FileGatewayOutboxProvider`, registered in every deployment), `pgp/` (BouncyCastle stream +decrypt/encrypt, keys in memory), `maintenance/` (daily retention process and record-action +effects), `FileGatewayHome` (typed STATISTICS and TABLE widgets only; guardrail G5), +`FileGatewayIntegrations` (ESB, Quick Search, API lists) and `file_gateway_server` heartbeats. +The Preferences "File transfer" section waits for QRun-IO/qqq#1162. The k8s example is in +`docs/k8s/`. + +Tests run a strict UI API server, H2 from the adopter DDL and real MINA clients. NoDiskExtension +snapshots the working tree (excluding target) and isolated test home, and asserts the test temp +directory is empty. The host supplies RDBMS; `qqq-backend-module-rdbms` must remain test-scoped. + ## Invariants and naming - Data access only through `/qqq/v1` (`UiApiInvoker`, channel `sftp`) as the signed-in user. The diff --git a/README.md b/README.md index 6aa3667..1c85923 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,24 @@ # QBit File Gateway -An SFTP server for QQQ applications. Users and service accounts sign in with their SSH keys and see -the application as a folder tree of apps and tables. Uploading a file into a table's -`upload//` folder runs a saved bulk-load profile as that user; getting a file from -`views/` or `reports/` exports a saved view or runs a saved report. The gateway reaches data only -through the `/qqq/v1` UI API with the signed-in user's own session (client channel `sftp`), so -permissions, masking and audits apply exactly as in the UI. Design: [docs/DESIGN.md](docs/DESIGN.md). +An SSH-key-only SFTP server that presents a QQQ application's permitted apps and tables as a +virtual folder tree. Application metadata and saved objects are read through the spec-defined +`/qqq/v1` UI API as the signed-in user, using `UiApiInvoker` on channel `sftp`. -This repository currently holds the scaffold; the gateway is built in phases described in the design. +Phases (a) through (c) implement authentication, listing/stat, account administration, uploads into +saved bulk-load profiles, background consumer delivery, readable sidecars/results/archive and +saved-view/report downloads. Endpoint routing/chroot, outbox delivery, preferences and the app +home follow in later phases. No native filesystem, host-key file, home directory or spool is +used by the gateway. See [the approved design](docs/DESIGN.md). ## Requirements -Java 21 and QQQ 4.1.0-SNAPSHOT (UI API invoker and client channels, until 4.1.0-RC.6). The Maven -wrapper pins Maven 3.9.11. +Java 21, Apache MINA SSHD 2.16.0, and the QQQ 4.1 invoker APIs. The host supplies the RDBMS +backend, qbit-object-storage 0.1.0-RC.2, and core saved bulk-load profile, saved view and saved +report metadata. The gateway's +RDBMS module dependency is test-scoped. Maven 3.9.11 is pinned by the wrapper. + +Until the invoker core PR merges, local builds use the private `4.1.0-FG-SNAPSHOT` artifacts. +The POM default remains `4.1.0-SNAPSHOT` for CI, to be replaced by 4.1.0-RC.6 when released. ## Install @@ -24,32 +30,320 @@ wrapper pins Maven 3.9.11. ``` +## Adopting + +Apply [docs/ddl/file_gateway.sql](docs/ddl/file_gateway.sql) to the host's database. It supports +H2 and PostgreSQL and stores UTC timestamps without a zone, matching QQQ's RDBMS binding. +The gateway does not create or migrate tables at runtime. + +Register metadata while building the QInstance: + +```java +FileGatewayQBitConfig config = new FileGatewayQBitConfig() + .withBackendName("rdbms") + .withSftpEnabled(Boolean.parseBoolean(System.getenv("FILE_GATEWAY_SFTP_ENABLED"))) + .withHostKeyEnvVarName("FILE_GATEWAY_SSH_HOST_KEY"); + +new FileGatewayQBitProducer().withQBitConfig(config).produce(qInstance); +``` + +After starting the host's `QApplicationJavalinServer`, start SFTP only in the gateway deployment: + +```java +FileGatewayRuntime gateway = null; +if(config.getSftpEnabled()) +{ + gateway = FileGatewayRuntime.start(qInstance, server); +} +// Call gateway.stop() during application shutdown when gateway is non-null. +``` + +Supply an unencrypted OpenSSH private host key as the value of the named environment variable. +It is parsed in memory; a missing key fails startup. `allowGeneratedHostKey` is for development +only and generates an ephemeral key with a warning. Each deployment uses the same host application +and QInstance; expose its SFTP listener separately from the UI HTTP port. + +Create an ACTIVE account with a QQQ `userId`, then add an Ed25519, ECDSA, or RSA public key of at +least 3072 bits. Public-key validation runs on insert and update and computes its unique SHA-256 +fingerprint, type and size. The host authentication module must assign the user's permissions +when `createAutomatedSessionForUser` is called, including `file_gateway.connect`. + ## Tables and permissions -Planned tables (none are produced yet), all prefixed `file_gateway_`: accounts, SSH keys, aliases, -upload jobs, the transfer log and the connection log. Permissions use the full `file_gateway` -prefix. +All seven tables are registered, with camelCase QQQ fields mapped to snake_case database columns. +`backendName` selects the host backend; physical table names match the QQQ table names. + +| Table | Purpose | +|---|---| +| `file_gateway_account` | User/service login, limits, CIDRs, expiry and lockout state | +| `file_gateway_ssh_key` | Account child keys and computed fingerprints | +| `file_gateway_alias` | Endpoint definitions; routing is a later phase | +| `file_gateway_upload_job` | Receiving, queued and completed uploads, retries, results and consumer references | +| `file_gateway_outbox_file` | Stable schema for later outbound delivery | +| `file_gateway_transfer_log` | Listings, uploads, object downloads and denied operations | +| `file_gateway_connection_log` | Accept, auth outcome, heartbeat and disconnect | + +One File Gateway app lists Accounts, Endpoints, Upload Jobs, Transfers and Connections. Keys, +outbox files and related jobs/logs appear in typed child lists. Endpoint saved-object pickers +are filtered by the selected table. There is no HTML widget or navigation process. + +| Permission | Grants | +|---|---| +| `file_gateway.connect` | SFTP login, checked again when the user session is refreshed | +| `file_gateway.view` | App and read access to gateway tables | +| `file_gateway.admin` | Read access and management of accounts, keys, endpoints and record actions | + +System-written job/outbox/audit tables are read-only to people, including admins. Record actions +use the standard Review/Result flow: Unlock account, Disable account, Revoke key, End session and +Retry upload. End session is enforced by the owning listener's next revocation poll. Retry upload +requeues FAILED/REJECTED jobs with a completed stored object and resets their attempts; it +rechecks eligibility under an account lock when applying the action. + +The `sftp` channel cannot access gateway, Authentik, audit, credential, API-key, secret, password, +permission, role, user-session or webhook administration surfaces. The `assistant` channel also +cannot access gateway tables. + +## Listing and authentication + +The root lists visible apps; apps contain child apps and readable tables. Table folders include: + +- `upload//` when the user has insert permission, a visible profile and a bulk-insert process; +- `views/.csv` and `.xlsx` (or account-enabled JSON); +- `reports/.` when the saved-report table, render process and its configured storage table are available; +- account-scoped `results/-.results.csv` and `archive//-`. + +Hidden paths return no-such-file. Slug collisions append record IDs. Saved-object queries are +paged. Metadata and saved objects are cached per connection for 60 and 30 seconds respectively. +Saved-view/report files report size zero (unknown); clients read until EOF. Stored artifacts +report their actual lengths and support ranged reads. + +Only public-key auth is advertised. Password, keyboard-interactive, GSSAPI, host-based auth, +shell, exec and forwarding are disabled. The algorithm lists exclude CBC, SHA-1 MAC/KEX and +`ssh-rsa` signatures. Idle timeout is 15 minutes; authentication timeout is 30 seconds. + +Account failure counters, operation-rate windows and session reservations are serialized with a database row lock across +listeners. Per-IP failure and connection windows are held in each pod. Unlocking an account also +clears an IP lock caused solely by that account on its next attempt; mixed-user IP locks expire +independently. User sessions are re-minted after five minutes on the next operation, expire after +ten minutes, and are revoked on close. The 60-second poll closes disabled/expired/revoked sessions +and heartbeats connection rows; three-minute-old heartbeats no longer count as active sessions. + +Lockouts raise core `QAlerts` signals. **Core compatibility:** the alert type is +`filegateway.lockout`, because the core validator rejects the design's underscore-only +`file_gateway_lockout`. Without an alert sink the core logging fallback is used. + +## Generated downloads + +Opening `views/.` queries the saved view again as the account user and starts +`POST /table/{table}/export` through `UiApiInvoker.invokeForStream`. The saved view's +`queryFilter` (including `orderBys`) and visible `queryColumns.columns` are preserved in order; +the UI selection column and pagination are excluded. Legacy explicit `fieldNames` are also +supported. CSV includes a header; CSV, XLSX and JSON are offered only when enabled in the +account's `formats` (default `csv,xlsx`). + +Opening `reports/.` selects that saved report through the `renderSavedReport` +record action, submits `reportFormat`, polls asynchronous status and streams the returned +`/download/{file}` storage reference. The process's `storageTableName` input default must name +an existing storage-capable table, backed by object storage in production. Without that table, +`reports/` is hidden. Reports requiring additional interactive values fail with the API's error. + +Each generated handle retains at most `downloadReadAheadMb` MiB (default 16). Sequential reads +and reordered reads within this rolling window work; a seek outside it fails with a message +to download sequentially. The HTTP stream closes with the SFTP handle. Read requests are filled +until EOF so MINA does not mistake a short HTTP packet for the end of an unknown-size file. + +`maxGeneratedFileMb` (default 1024) and `max_download_mb_per_day` are enforced as generated +bytes enter the window. The daily total queries DOWNLOAD transfer rows for this account since +UTC midnight, including failed and in-progress transfers; updates hold the account row lock so +concurrent generated downloads share the cap. A rejected chunk is not exposed. Re-reading cached +bytes does not charge them twice. One DOWNLOAD row per handle records the view/report ID, format, +bytes, duration and outcome: OK, LIMIT, ERROR or DENIED for an HTTP 403. API error text reaches +the SFTP client. + +**Field security:** configure `QInstance.enforcePermissionsInCoreActions=true` when using this +private core build's `FieldSecurityLock` enforcement. Acceptance tests use a DENY lock and compare +the generated CSV byte-for-byte with a direct UI export by the same user: default columns omit +the denied field. Explicitly selecting a denied field fails as it does through the UI API. + +**Core filesystem limitation:** the current private core's `RenderSavedReportExecuteStep` +creates and deletes a temporary file before publishing to storage. The gateway does not spool, +and the existing no-disk snapshot/cleanup guard passes, but that guard cannot detect transient +files. Fully disk-free saved-report generation remains blocked on replacing core's temporary +file with storage streaming. CSV/XLSX/JSON table exports use core's streaming exporters; the +50,000-row CSV acceptance test consumes the result with a fixed buffer. + +## Uploads and object storage + +Configure an object-storage bucket alias matching `config.fileGatewayBucketAlias` (default +`file-gateway`). Production always uses `ObjectStorage.asSystem("file-gateway", runId)` and only +the allow-listed `com.kingsrook.qbits.objectstorage.api` package. Tests inject a memory store. +Objects use `file-gateway/uploads/{accountId}/{yyyy}/{mm}/{dd}/{uuid}/{fileName}`; +results and quarantine use separate `file-gateway/results/` and `file-gateway/quarantine/` prefixes. +The original object backs the virtual archive; it is not copied after a successful import. + +`uploadPartSizeMb` bounds the gateway pipe and `uploadReorderWindowMb` bounds pending writes +and gaps. The storage API streams unknown-length content and owns multipart aborts. Its actual +S3 part size is configured on the **object-storage connection** (`multipartPartSizeMb`); set it +to 8 for the design's default part size. RC.2 does not expose a per-put part-size override through +its API. Account file-size and UTC daily-upload quotas are reserved in the database while writing, +including concurrent uploads on other pods. Failed transfers retain the received-byte charge. + +The host's `QJavalinMetaData.uploadedFileArchiveTableName` must name a storage-capable table +backed by object storage, not a filesystem backend. The bulk-insert UI API uses that table for +its process input. The test harness uses core memory storage. Configure the host HTTP server's +request limits for the permitted file sizes; these limits also apply to loopback UI API calls. + +A normal close queues the job. Names ending in `.filepart`, `.tmp`, `.part` or `.partial` stay +RECEIVING after close until renamed within the same profile folder. Each put is a new job. +Dedupe skips content that already SUCCEEDED for the same account, table, profile and consumer +within `dedupeWindow`; its `.done` text identifies the earlier job. + +The worker mints a fresh account-user session, requires `file_gateway.connect`, sends every +application request through the `sftp` UI API channel, refreshes credentials after five minutes, +and revokes them after delivery. It follows the bulk-insert process through upload, mapping, +review and result. Profiles must contain their value mappings; the worker fails instead of +interactively guessing a mapping. Results contain core summary lines and row examples where +core supplies them; a blank CSV field column means no field identifier was supplied by core. + +**Process steps:** the mapping step sends its values as form fields only; the invoker encodes +them as multipart because the step operation declares a multipart body (QRun-IO/qqq#1170). The +file is uploaded once, at process init. + +Workers are bounded by `uploadWorkerThreads`. Claims compare the expected status inside a +short account-row transaction (core's expected-version check alone is not a SQL conditional +update). Completion also checks the owning pod and claim timestamp. Retries back off by +1 minute, 5 minutes, 15 minutes and 1 hour, then fail on the fifth attempt. The configured +schedule also includes a 6-hour slot; the five-attempt cap takes precedence. Consumer-requested +delays can lengthen the scheduled wait. Stale RUNNING jobs requeue with a three-start cap; +shutdown releases this pod's claims. Delivery is at least once, so custom consumers must dedupe +on job ID and digest. + +Optional `FileGatewayScannerInterface` implementations return CLEAN, INFECTED or ERROR. +INFECTED objects are copied into quarantine and removed from the original key; ERROR retries. +Consumers receive an `InboundFile` with fresh object streams and run with the user's `QContext`. +Unknown consumer names are rejected. + +`.processing` appears during receiving/queueing/running. Finished jobs expose `.done` or +`.error` text. Sidecars and results disappear after acknowledgement (`rm` on either artifact) +or seven days. Archive files remain until later retention processing. Core notifications go to +the account user (service account: `ownerUserId`) on FAILED/PARTIAL/REJECTED and on SUCCEEDED +when `notifyOnSuccess` is enabled. Without an alerts sink, sending is a no-op. + +Existing phase-(a) databases need this additional column before adopting phase (b): + +```sql +ALTER TABLE file_gateway_upload_job ADD COLUMN results_size_bytes BIGINT; +``` ## Configuration -`FileGatewayQBitConfig` holds the optional `tableMetaDataCustomizer`, `sftpEnabled` (only the -gateway deployment runs the server) and `sftpPort` (default 2222). +Every option has a getter and fluent `withX(...)` setter. Configuration for later phases remains +present without enabling those features. -## Adopting in another app +| Options | Defaults | +|---|---| +| `backendName`, `sftpEnabled`, `sftpPort` | `rdbms`, `false`, `2222` | +| `hostKeyEnvVarName`, `allowGeneratedHostKey` | `FILE_GATEWAY_SSH_HOST_KEY`, `false` | +| `lockoutThreshold`, `lockoutWindow`, `lockoutDuration` | 5, 15 minutes, 30 minutes | +| `maxConnectionsPerIpPerMinute` | 30 | +| `defaultMaxConcurrentSessions`, `defaultMaxOperationsPerMinute` | 4, 600 | +| `defaultMaxFileSizeMb` | 100 | +| `defaultMaxUploadMbPerDay`, `defaultMaxDownloadMbPerDay` | null (unlimited) | +| `fileGatewayBucketAlias` | `file-gateway` | +| `uploadPartSizeMb`, `uploadReorderWindowMb`, `downloadReadAheadMb`, `maxGeneratedFileMb` | 8, 64, 16, 1024 | +| `uploadWorkerThreads`, `dedupeWindow`, `jobStaleAfter` | 4, 7 days, 30 minutes | +| `temporaryUploadSuffixes` | `.filepart`, `.tmp`, `.part`, `.partial` | +| `scanner`, `pgpPrivateKeyEnvVarName`, `pgpPassphraseEnvVarName` | null | +| `logRetention`, `archiveRetention`, `outboxRetention`, `resultsRetention`, `receivingStaleAfter` | 180 days, 30 days, 30 days, 90 days, 1 hour | +| `tableMetaDataCustomizer` | null | -```java -new FileGatewayQBitProducer() - .withQBitConfig(new FileGatewayQBitConfig()) - .produce(qInstance); -``` +## Consumer and outbox API -## Build +The isolated `com.kingsrook.qbits.filegateway.api` package contains `FileGatewayConsumer`, +`InboundFile` (including remote address and a fresh stream supplier), `FileAcceptance`, +`OutboxFileRef`, and the consumer metadata registry. `FileGatewayOutbox` delegates to a registered +`FileGatewayOutboxProvider`; without one it reports unavailable and throws +`FileGatewayUnavailableException`. INBOX endpoints bind a consumer by `consumer_name`; the built-in +`bulk-load` consumer serves `upload//` folders and BULK_LOAD endpoints. The outbox +provider is registered by the producer in every deployment (it needs only the database and object +storage), so other qbits can call `FileGatewayOutbox.put(...)` from web pods too. + +## Endpoints and chroot + +`file_gateway_alias` rows ("Endpoints") mount partner-friendly paths such as `/acme/orders-in`: + +| Kind | Behaviour | +|---|---| +| `BULK_LOAD` | upload folder that bulk-loads with the endpoint's saved profile | +| `INBOX` | upload folder whose files go to the `consumer_name` consumer | +| `VIEW`, `REPORT` | one generated file (`file_name`, or `.`) | +| `OUTBOX` | files put by `FileGatewayOutbox`; partners download and delete them | + +Paths are absolute, `[a-z0-9._/-]`, unique, and never collide with app names. An endpoint is +listed only when it is active, bound to the account (or shared), and its target is reachable by +the user through `/qqq/v1`; endpoints grant nothing. `root_mode` FULL_TREE (default for users) +shows the app tree plus endpoints; ENDPOINTS_ONLY (default for service accounts) is the chroot. + +## Outbox + +`put` streams to `file-gateway/outbox/{aliasId}/{uuid}/{fileName}` with a SHA-256 and a +`file_gateway_outbox_file` row (READY). The first complete download marks it DOWNLOADED and calls +`onDownloaded` on the endpoint's consumer (at-least-once); `rm` marks it REMOVED; retention marks +old files EXPIRED and deletes the object. + +## PGP (optional, per account) + +`pgp_decrypt_uploads`: uploads named `*.pgp`/`*.gpg` are decrypted stream to stream into a new +object before the consumer runs, with the key and passphrase from the env vars named by +`pgpPrivateKeyEnvVarName`/`pgpPassphraseEnvVarName` (in memory only); a bad file is REJECTED. +`pgp_encrypt_downloads`: downloads are encrypted on the fly to the account's `pgp_public_key` +(validated on save) and listed with `.pgp` appended. + +## App home, actions and retention + +The `file_gateway` app home uses typed widgets only: a KPI row (transfers today, failed jobs in +24 hours, active sessions, MB in and out today, locked accounts), and full-width tables for recent +failures, top partners (7 days), active sessions and gateway servers (host key fingerprint, port, +pod, last seen, from `file_gateway_server` heartbeats). Record actions (standard Review/Result, +never in the nav): Unlock account, Disable account, Revoke key, End session, Retry upload. A daily +scheduled retention process applies the `*Retention` settings and aborts stale RECEIVING uploads. + +## Integrations + +`FileGatewayIntegrations` lists what a host wires in without this qbit depending on those modules: +`esbTableNames()` (accounts, endpoints, upload jobs), `quickSearchTables()` (accounts by username +and label, endpoints by name and path, upload jobs by file name) and `apiTableNames()` (all tables, +with the admin's rules). + +## Kubernetes + +`docs/k8s/gateway-deployment.yaml` and `docs/k8s/gateway-service.yaml`: the same image as the web +Deployment with `FILE_GATEWAY_SFTP_ENABLED=true`, the host key from a Secret, a read-only root +filesystem with an in-memory `/tmp`, 2 replicas and a PDB, and a LoadBalancer Service exposing +port 22 only (no passive range: v1 is SFTP only) with `externalTrafficPolicy: Local`. + +## Build and acceptance tests ```bash -./mvnw verify # Checkstyle; JaCoCo 70% instructions, 90% classes +./mvnw -o -Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT verify ``` -On Windows, use `./mvnw.cmd`. +PowerShell: + +```powershell +./mvnw.cmd -o '-Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT' verify +``` + +The tests use H2 from the adopter DDL, a strict UI API server, generated in-memory SSH keys and +real MINA clients. They exercise permissions, auth failures, lockout, competing listeners, +revocation/refresh, pagination/caching, audit, key validation, contingent pickers, API isolation, +metadata registration and record serialization. Checkstyle and JaCoCo gates require 70% +instruction coverage and 90% class coverage. + +Surefire isolates `user.home` under `target/fg-home` and `java.io.tmpdir` under `target/fg-tmp`. +The disk guard creates those directories before tests, compares working-tree and home snapshots, +and requires an empty temp directory after each protocol test. Build output under `target/` +is excluded from the working-tree snapshot. ## License diff --git a/docs/DESIGN.md b/docs/DESIGN.md index a18292f..7be5a01 100644 --- a/docs/DESIGN.md +++ b/docs/DESIGN.md @@ -165,7 +165,7 @@ exists on any disk. | table | the table is in the app and in `metaData.tables` (read permission) | | `upload/` | the table has the bulk-insert process (`.bulkInsert`) in metaData, and at least one saved bulk-load profile is visible (`POST /table/savedBulkLoadProfile/query` filtered by `tableName`) | | `views/` | the user can query `savedView` | -| `reports/` | the user can query `savedReport` and has the `renderSavedReport` process | +| `reports/` | the user can query `savedReport`, has the `renderSavedReport` process, and that process's `storageTableName` input default names an existing storage-capable table | | `results/`, `archive/` | always (they hold only this account's jobs) | - **Caching.** Metadata responses are cached in memory per SFTP session for 60 seconds. Saved-object @@ -333,6 +333,11 @@ The gateway registers a "File transfer" section with the provider `FileTransferP ## 5. Download Everything is streamed. Nothing is written to disk. +The gateway itself streams without disk. Core's `RenderSavedReportExecuteStep` stages the +rendered report in a JVM temp file and deletes it in try-with-resources before the process step +returns. That is the bounded temp area the owner rule allows, so the `emptyDir` temp mount must be +sized for the largest report. The no-disk test asserts the temp area is empty after each transfer. + - **Saved view** (`views/.`): - `POST /table/savedView/query` by table name, as the user; - the filter, visible columns and order are parsed from `viewJson`; @@ -390,7 +395,7 @@ Everything is streamed. Nothing is written to disk. - **Lockout.** - 5 failed auths within 15 minutes, per username or per IP, lock that username or IP for 30 minutes. All three numbers are configurable. - - A lock sets `locked_until` and raises an alert (`file_gateway_lockout`, `QAlerts.raise`). + - A lock sets `locked_until` and raises an alert (`filegateway.lockout`, `QAlerts.raise`; core alert-type names use dots, not the table prefix). - The record action "Unlock account" clears it. - Lockout state lives in the database, so it holds across pods. - A per-IP limit on new connections (30 per minute) applies before auth. @@ -460,6 +465,7 @@ the table names. | `file_gateway_outbox_file` (child of endpoint) | files waiting for pickup | `alias_id`, `file_name`, `size_bytes`, `sha256`, `storage_key`, `status` (READY, DOWNLOADED, REMOVED, EXPIRED), `first_downloaded_at`, `download_count`, `removed_at`, `expires_at`, `created_by_consumer` | | `file_gateway_transfer_log` (main, "Transfers") | every operation | `account_id`, `connection_log_id`, `operation` (LIST, UPLOAD, DOWNLOAD, RENAME, REMOVE), `path`, `object_kind` (APP, TABLE, PROFILE, VIEW, REPORT, RESULT, ARCHIVE, ENDPOINT, OUTBOX), `object_id`, `format`, `bytes`, `sha256`, `duration_ms`, `outcome` (OK, DENIED, NOT_FOUND, LIMIT, ERROR), `message`, `client_channel`, `upload_job_id` | | `file_gateway_connection_log` (main, "Connections") | every connection | `account_id`, `username`, `remote_ip`, `client_version`, `key_fingerprint`, `outcome` (CONNECTED, AUTH_FAILED, LOCKED_OUT, IP_DENIED, EXPIRED, DISABLED, LIMIT), `session_id`, `pod_name`, `connected_at`, `last_seen_at`, `disconnected_at`, `disconnect_reason`, `bytes_in`, `bytes_out`, `operation_count`, `client_channel` | +| `file_gateway_server` (system, not in nav) | gateway pod heartbeats for the app home | `pod_name` (unique), `host_key_fingerprint`, `port`, `started_at`, `last_seen_at`; rows unseen for a day are deleted by retention | - **Child lists:** - an account shows its keys, upload jobs and connections; @@ -593,8 +599,8 @@ The gateway is a separate, stateless Deployment of the same image. An example is - `FILE_GATEWAY_SSH_HOST_KEY` from a Secret; in kof22, through ESO from Infisical. - 2 replicas, with a PodDisruptionBudget of `minAvailable: 1`. - `readOnlyRootFilesystem: true`. The only writable mount is a small in-memory `emptyDir` - (`medium: Memory`, `sizeLimit: 64Mi`) for the JVM temp directory. Nothing is persisted. - - Requests of 250m CPU and 768Mi memory; a 1.5Gi memory limit. The in-memory buffers are bounded + (`medium: Memory`, `sizeLimit: 512Mi`, counted in the memory limit) for the JVM temp directory, sized for the largest saved report core stages there. Nothing is persisted. + - Requests of 250m CPU and 768Mi memory; a 2Gi memory limit. The in-memory buffers are bounded by the config. - **Ports and probes.** - Container ports: 2222 (sftp) and the app's HTTP port, which serves loopback and probes only. @@ -602,7 +608,7 @@ The gateway is a separate, stateless Deployment of the same image. An example is - Liveness: HTTP health. - Startup: allows 2 minutes for the QInstance. - **Service.** `type: LoadBalancer`, with these settings: - - the annotation `lbipam.cilium.io/ips: 10.100.22.65` in kof22 (owner, 2026-10-13; .60, .61 and .64 are taken); + - the annotation `lbipam.cilium.io/ips` (or the older `io.cilium/lb-ipam-ips`, which k8s-galaxy uses): 10.100.22.65 in kof22 (owner, 2026-10-13; .60, .61 and .64 are taken); - **port 22 → 2222 only (no passive range in v1)**; - `externalTrafficPolicy: Local`, so the pod sees the real client IP for allowlists and lockouts. diff --git a/docs/ddl/file_gateway.sql b/docs/ddl/file_gateway.sql new file mode 100644 index 0000000..c99207f --- /dev/null +++ b/docs/ddl/file_gateway.sql @@ -0,0 +1,191 @@ +-- File Gateway schema. PostgreSQL and H2. Timestamps are UTC, matching QQQ RDBMS binding. + +CREATE TABLE file_gateway_account ( + id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + create_date TIMESTAMP, + modify_date TIMESTAMP, + username VARCHAR(1024) NOT NULL UNIQUE, + label VARCHAR(1024), + account_type VARCHAR(1024), + user_id VARCHAR(1024), + owner_user_id VARCHAR(1024), + status VARCHAR(1024), + root_mode VARCHAR(1024), + allowed_cidrs TEXT, + expires_at TIMESTAMP, + formats VARCHAR(1024), + max_concurrent_sessions INTEGER, + max_operations_per_minute INTEGER, + max_file_size_mb INTEGER, + max_upload_mb_per_day INTEGER, + max_download_mb_per_day INTEGER, + dedupe_uploads BOOLEAN, + notify_on_success BOOLEAN, + pgp_public_key TEXT, + pgp_decrypt_uploads BOOLEAN, + pgp_encrypt_downloads BOOLEAN, + operation_window_at TIMESTAMP, + operation_window_count INTEGER, + failed_login_count INTEGER, + failed_login_window_at TIMESTAMP, + locked_until TIMESTAMP, + last_login_at TIMESTAMP, + last_login_ip VARCHAR(1024) +); + +CREATE TABLE file_gateway_ssh_key ( + id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + create_date TIMESTAMP, + modify_date TIMESTAMP, + account_id INTEGER, + label VARCHAR(1024), + public_key TEXT, + fingerprint VARCHAR(1024) NOT NULL UNIQUE, + key_type VARCHAR(1024), + key_bits INTEGER, + expires_at TIMESTAMP, + revoked_at TIMESTAMP, + last_used_at TIMESTAMP, + last_used_ip VARCHAR(1024) +); + +CREATE TABLE file_gateway_alias ( + id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + create_date TIMESTAMP, + modify_date TIMESTAMP, + name VARCHAR(1024) NOT NULL UNIQUE, + path VARCHAR(1024) NOT NULL UNIQUE, + kind VARCHAR(1024), + account_id INTEGER, + table_name VARCHAR(1024), + saved_bulk_load_profile_id INTEGER, + saved_view_id INTEGER, + saved_report_id INTEGER, + format VARCHAR(1024), + file_name VARCHAR(1024), + consumer_name VARCHAR(1024), + is_active BOOLEAN, + description TEXT +); + +CREATE TABLE file_gateway_upload_job ( + id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + create_date TIMESTAMP, + modify_date TIMESTAMP, + account_id INTEGER, + alias_id INTEGER, + connection_log_id INTEGER, + remote_address VARCHAR(1024), + table_name VARCHAR(1024), + saved_bulk_load_profile_id INTEGER, + consumer_name VARCHAR(1024), + consumer_reference VARCHAR(1024), + path VARCHAR(1024), + file_name VARCHAR(1024), + size_bytes BIGINT, + sha256 VARCHAR(1024), + storage_key VARCHAR(1024), + status VARCHAR(1024), + duplicate_of_job_id INTEGER, + process_uuid VARCHAR(1024), + inserted_count INTEGER, + error_count INTEGER, + result_summary TEXT, + results_storage_key VARCHAR(1024), + results_size_bytes BIGINT, + plaintext_storage_key VARCHAR(1024), + plaintext_sha256 VARCHAR(1024), + plaintext_size_bytes BIGINT, + error_message TEXT, + attempts INTEGER, + next_attempt_at TIMESTAMP, + claimed_by VARCHAR(1024), + claimed_at TIMESTAMP, + queued_at TIMESTAMP, + started_at TIMESTAMP, + finished_at TIMESTAMP, + acknowledged_at TIMESTAMP +); + +CREATE TABLE file_gateway_outbox_file ( + id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + create_date TIMESTAMP, + modify_date TIMESTAMP, + alias_id INTEGER, + file_name VARCHAR(1024), + size_bytes BIGINT, + sha256 VARCHAR(1024), + storage_key VARCHAR(1024), + status VARCHAR(1024), + first_downloaded_at TIMESTAMP, + download_count INTEGER, + on_downloaded_pending BOOLEAN DEFAULT FALSE, + removed_at TIMESTAMP, + expires_at TIMESTAMP, + created_by_consumer VARCHAR(1024) +); + +CREATE TABLE file_gateway_transfer_log ( + id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + create_date TIMESTAMP, + modify_date TIMESTAMP, + account_id INTEGER, + connection_log_id INTEGER, + operation VARCHAR(1024), + path VARCHAR(1024), + object_kind VARCHAR(1024), + object_id VARCHAR(1024), + format VARCHAR(1024), + bytes BIGINT, + sha256 VARCHAR(1024), + duration_ms BIGINT, + outcome VARCHAR(1024), + message TEXT, + client_channel VARCHAR(1024), + upload_job_id INTEGER +); + +CREATE TABLE file_gateway_connection_log ( + id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + create_date TIMESTAMP, + modify_date TIMESTAMP, + account_id INTEGER, + username VARCHAR(1024), + remote_ip VARCHAR(1024), + client_version VARCHAR(1024), + key_fingerprint VARCHAR(1024), + outcome VARCHAR(1024), + session_id VARCHAR(1024), + pod_name VARCHAR(1024), + connected_at TIMESTAMP, + last_seen_at TIMESTAMP, + disconnected_at TIMESTAMP, + end_requested_at TIMESTAMP, + disconnect_reason TEXT, + bytes_in BIGINT, + bytes_out BIGINT, + operation_count BIGINT, + client_channel VARCHAR(1024) +); + +CREATE TABLE file_gateway_server ( + id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + create_date TIMESTAMP, + modify_date TIMESTAMP, + pod_name VARCHAR(1024) NOT NULL UNIQUE, + host_key_fingerprint VARCHAR(1024), + port INTEGER, + started_at TIMESTAMP, + last_seen_at TIMESTAMP +); + +-- Indexes for the job queue, dedupe, live sessions, home queries and retention. +CREATE INDEX file_gateway_transfer_log_account_date ON file_gateway_transfer_log (account_id, create_date); +CREATE INDEX file_gateway_transfer_log_date ON file_gateway_transfer_log (create_date); +CREATE INDEX file_gateway_connection_log_account_live ON file_gateway_connection_log (account_id, disconnected_at); +CREATE INDEX file_gateway_connection_log_date ON file_gateway_connection_log (create_date); +CREATE INDEX file_gateway_upload_job_status_next ON file_gateway_upload_job (status, next_attempt_at); +CREATE INDEX file_gateway_upload_job_dedupe ON file_gateway_upload_job (account_id, sha256); +CREATE INDEX file_gateway_upload_job_date ON file_gateway_upload_job (create_date); +CREATE INDEX file_gateway_outbox_file_alias_status ON file_gateway_outbox_file (alias_id, status); +CREATE INDEX file_gateway_ssh_key_account ON file_gateway_ssh_key (account_id); diff --git a/docs/k8s/gateway-deployment.yaml b/docs/k8s/gateway-deployment.yaml new file mode 100644 index 0000000..5b8734b --- /dev/null +++ b/docs/k8s/gateway-deployment.yaml @@ -0,0 +1,91 @@ +# qbit-file-gateway: the SFTP gateway Deployment (docs/DESIGN.md section 11). +# Same image and QInstance as the application's web Deployment; only FILE_GATEWAY_SFTP_ENABLED differs. +# Stateless: no volumes except an in-memory JVM temp area. State lives in the database and object storage. +# Replace every and copy the application's usual environment (database, object storage, auth). +apiVersion: apps/v1 +kind: Deployment +metadata: + name: -file-gateway + labels: + app.kubernetes.io/name: -file-gateway + app.kubernetes.io/component: sftp-gateway +spec: + replicas: 2 + selector: + matchLabels: + app.kubernetes.io/name: -file-gateway + template: + metadata: + labels: + app.kubernetes.io/name: -file-gateway + app.kubernetes.io/component: sftp-gateway + spec: + terminationGracePeriodSeconds: 75 # SIGTERM: stop accepting, drain sessions up to 60 s, requeue claimed jobs + securityContext: + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault + containers: + - name: app + image: # the same image as the web Deployment + env: + - name: FILE_GATEWAY_SFTP_ENABLED + value: "true" + - name: FILE_GATEWAY_SSH_HOST_KEY # OpenSSH private key text; parsed in memory, never written to disk + valueFrom: + secretKeyRef: + name: file-gateway-host-key + key: ssh_host_key + - name: JAVA_TOOL_OPTIONS + value: "-Djava.io.tmpdir=/tmp" + # ... the application's usual environment (database, object storage, authentication) ... + ports: + - name: sftp + containerPort: 2222 + - name: http # loopback /qqq/v1 calls and kubelet probes only; not in the Service + containerPort: 8000 + readinessProbe: + tcpSocket: + port: sftp + periodSeconds: 10 + livenessProbe: + httpGet: + path: + port: http + periodSeconds: 20 + failureThreshold: 3 + startupProbe: + httpGet: + path: + port: http + periodSeconds: 5 + failureThreshold: 24 # up to 2 minutes for the QInstance + resources: + requests: + cpu: 250m + memory: 768Mi + limits: + memory: 2Gi # includes the in-memory /tmp below and the bounded transfer buffers + securityContext: + readOnlyRootFilesystem: true + allowPrivilegeEscalation: false + capabilities: + drop: [ALL] + volumeMounts: + - name: tmp + mountPath: /tmp + volumes: + - name: tmp # JVM temp only (core stages a rendered report here and deletes it) + emptyDir: + medium: Memory + sizeLimit: 512Mi +--- +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: -file-gateway +spec: + minAvailable: 1 + selector: + matchLabels: + app.kubernetes.io/name: -file-gateway diff --git a/docs/k8s/gateway-service.yaml b/docs/k8s/gateway-service.yaml new file mode 100644 index 0000000..fbfefc9 --- /dev/null +++ b/docs/k8s/gateway-service.yaml @@ -0,0 +1,20 @@ +# qbit-file-gateway: the SFTP LoadBalancer Service (docs/DESIGN.md section 11). +# One port only: v1 is SFTP only, so there is no FTPS passive port range (see DESIGN.md section 15). +apiVersion: v1 +kind: Service +metadata: + name: -file-gateway + annotations: + # Cilium LB-IPAM: a fixed address (kof22 on k8s-galaxy: 10.100.22.65, owner decision 2026-10-13). + # Cilium 1.15+ reads lbipam.cilium.io/ips; older clusters (and k8s-galaxy manifests) use io.cilium/lb-ipam-ips. + lbipam.cilium.io/ips: "10.100.22.65" +spec: + type: LoadBalancer + externalTrafficPolicy: Local # keep the client IP: IP allowlists and lockouts need it + selector: + app.kubernetes.io/name: -file-gateway + ports: + - name: sftp + port: 22 + targetPort: sftp + protocol: TCP diff --git a/pom.xml b/pom.xml index c8c9840..20ca951 100644 --- a/pom.xml +++ b/pom.xml @@ -27,6 +27,7 @@ 21 0.8.15 2.2.224 + 2.16.0 @@ -72,6 +73,32 @@ + + org.bouncycastle + bcpg-jdk18on + 1.84 + + + com.kingsrook.qbits + qbit-object-storage + 0.1.0-RC.2 + + + + net.i2p.crypto + eddsa + 0.3.0 + + + org.apache.sshd + sshd-core + ${sshd.version} + + + org.apache.sshd + sshd-sftp + ${sshd.version} + com.kingsrook.qqq qqq-backend-core @@ -114,6 +141,16 @@ + + org.apache.maven.plugins + maven-surefire-plugin + + + ${project.build.directory}/fg-tmp + ${project.build.directory}/fg-home + + + org.apache.maven.plugins maven-checkstyle-plugin diff --git a/src/main/java/com/kingsrook/qbits/filegateway/FileGatewayIntegrations.java b/src/main/java/com/kingsrook/qbits/filegateway/FileGatewayIntegrations.java new file mode 100644 index 0000000..3b98f13 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/FileGatewayIntegrations.java @@ -0,0 +1,94 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + + +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; + + +/******************************************************************************* + ** What a host wires into its ESB, Quick Search and API, without this qbit + ** depending on those modules (the same opt-in hooks as qbit-sql-workbench). + ** + **
    + **
  • {@link #esbTableNames()}: publish these tables' inserts, updates and + ** deletes to an ESB topic (the main tables).
  • + **
  • {@link #quickSearchTables()}: index these in Quick Search, with field + ** weights.
  • + **
  • {@link #apiTableNames()}: expose these in the host's QQQ API. The API + ** gets exactly the admin's rules: the same permissions, customizers and + ** read-only tables.
  • + **
+ *******************************************************************************/ +public final class FileGatewayIntegrations +{ + /******************************************************************************* + ** A table for Quick Search: field weights, the timestamp for incremental + ** indexing, and the record label. + *******************************************************************************/ + public record SearchableTable(String tableName, Map fieldWeights, String basepullTimestamp, String recordLabelFormat, List recordLabelFields) + { + } + + + + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private FileGatewayIntegrations() + { + } + + + + /******************************************************************************* + ** Tables to publish on the ESB. + *******************************************************************************/ + public static List esbTableNames() + { + return (List.of(FileGatewayTables.ACCOUNT, FileGatewayTables.ALIAS, FileGatewayTables.UPLOAD_JOB)); + } + + + + /******************************************************************************* + ** Tables for the host's QQQ API. + *******************************************************************************/ + public static List apiTableNames() + { + return (FileGatewayTables.ALL); + } + + + + /******************************************************************************* + ** Tables for Quick Search. + *******************************************************************************/ + public static List quickSearchTables() + { + return (List.of( + new SearchableTable(FileGatewayTables.ACCOUNT, Map.of("username", 10, "label", 6), "modifyDate", "%s", List.of("username")), + new SearchableTable(FileGatewayTables.ALIAS, Map.of("name", 10, "path", 6), "modifyDate", "%s", List.of("name")), + new SearchableTable(FileGatewayTables.UPLOAD_JOB, Map.of("fileName", 10), "modifyDate", "%s", List.of("fileName")))); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/FileGatewayQBitConfig.java b/src/main/java/com/kingsrook/qbits/filegateway/FileGatewayQBitConfig.java index 25779d1..57b2f2c 100644 --- a/src/main/java/com/kingsrook/qbits/filegateway/FileGatewayQBitConfig.java +++ b/src/main/java/com/kingsrook/qbits/filegateway/FileGatewayQBitConfig.java @@ -21,46 +21,79 @@ package com.kingsrook.qbits.filegateway; +import java.time.Duration; +import java.util.List; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; import com.kingsrook.qqq.backend.core.model.metadata.producers.MetaDataCustomizerInterface; import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitConfig; import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; /******************************************************************************* - ** Configuration for the file-gateway qbit. + ** Configuration shared by the admin application and gateway deployment. *******************************************************************************/ public class FileGatewayQBitConfig implements QBitConfig { - private MetaDataCustomizerInterface tableMetaDataCustomizer; + private String schedulerName = null; + private Duration serverRetention = Duration.ofDays(1); + private String backendName = "rdbms"; private boolean sftpEnabled = false; - private int sftpPort = 2222; + private int sftpPort = 2222; + private String hostKeyEnvVarName = "FILE_GATEWAY_SSH_HOST_KEY"; + private boolean allowGeneratedHostKey = false; + private String fileGatewayBucketAlias = "file-gateway"; + private int uploadPartSizeMb = 8; + private int uploadReorderWindowMb = 64; + private int downloadReadAheadMb = 16; + private int maxGeneratedFileMb = 1024; + private int uploadWorkerThreads = 4; + private Duration dedupeWindow = Duration.ofDays(7); + private Duration jobStaleAfter = Duration.ofMinutes(30); + private List temporaryUploadSuffixes = List.of(".filepart", ".tmp", ".part", ".partial"); + private int lockoutThreshold = 5; + private Duration lockoutWindow = Duration.ofMinutes(15); + private Duration lockoutDuration = Duration.ofMinutes(30); + private int maxConnectionsPerIpPerMinute = 30; + private int defaultMaxConcurrentSessions = 4; + private int defaultMaxOperationsPerMinute = 600; + private int defaultMaxFileSizeMb = 100; + private Integer defaultMaxUploadMbPerDay = null; + private Integer defaultMaxDownloadMbPerDay = null; + private QCodeReference scanner = null; + private String pgpPrivateKeyEnvVarName = null; + private String pgpPassphraseEnvVarName = null; + private Duration logRetention = Duration.ofDays(180); + private Duration archiveRetention = Duration.ofDays(30); + private Duration outboxRetention = Duration.ofDays(30); + private Duration resultsRetention = Duration.ofDays(90); + private Duration receivingStaleAfter = Duration.ofHours(1); + private MetaDataCustomizerInterface tableMetaDataCustomizer = null; /******************************************************************************* - ** Getter for the optional table customizer. + ** Getter for backendName. *******************************************************************************/ - @Override - public MetaDataCustomizerInterface getTableMetaDataCustomizer() + public String getBackendName() { - return (tableMetaDataCustomizer); + return (backendName); } /******************************************************************************* - ** Fluent setter for the optional table customizer. + ** Fluent setter for backendName. *******************************************************************************/ - public FileGatewayQBitConfig withTableMetaDataCustomizer(MetaDataCustomizerInterface tableMetaDataCustomizer) + public FileGatewayQBitConfig withBackendName(String backendName) { - this.tableMetaDataCustomizer = tableMetaDataCustomizer; + this.backendName = backendName; return (this); } /******************************************************************************* - ** Whether this process runs the SFTP server (only the gateway deployment does). + ** Getter for sftpEnabled. *******************************************************************************/ public boolean getSftpEnabled() { @@ -70,7 +103,7 @@ public class FileGatewayQBitConfig implements QBitConfig /******************************************************************************* - ** Fluent setter for whether this process runs the SFTP server. + ** Fluent setter for sftpEnabled. *******************************************************************************/ public FileGatewayQBitConfig withSftpEnabled(boolean sftpEnabled) { @@ -81,7 +114,7 @@ public class FileGatewayQBitConfig implements QBitConfig /******************************************************************************* - ** Getter for the SFTP listen port (default 2222). + ** Getter for sftpPort. *******************************************************************************/ public int getSftpPort() { @@ -91,11 +124,665 @@ public class FileGatewayQBitConfig implements QBitConfig /******************************************************************************* - ** Fluent setter for the SFTP listen port. + ** Fluent setter for sftpPort. *******************************************************************************/ public FileGatewayQBitConfig withSftpPort(int sftpPort) { this.sftpPort = sftpPort; return (this); } + + + + /******************************************************************************* + ** Getter for hostKeyEnvVarName. + *******************************************************************************/ + public String getHostKeyEnvVarName() + { + return (hostKeyEnvVarName); + } + + + + /******************************************************************************* + ** Fluent setter for hostKeyEnvVarName. + *******************************************************************************/ + public FileGatewayQBitConfig withHostKeyEnvVarName(String hostKeyEnvVarName) + { + this.hostKeyEnvVarName = hostKeyEnvVarName; + return (this); + } + + + + /******************************************************************************* + ** Getter for allowGeneratedHostKey. + *******************************************************************************/ + public boolean getAllowGeneratedHostKey() + { + return (allowGeneratedHostKey); + } + + + + /******************************************************************************* + ** Fluent setter for allowGeneratedHostKey. + *******************************************************************************/ + public FileGatewayQBitConfig withAllowGeneratedHostKey(boolean allowGeneratedHostKey) + { + this.allowGeneratedHostKey = allowGeneratedHostKey; + return (this); + } + + + + /******************************************************************************* + ** Getter for fileGatewayBucketAlias. + *******************************************************************************/ + public String getFileGatewayBucketAlias() + { + return (fileGatewayBucketAlias); + } + + + + /******************************************************************************* + ** Fluent setter for fileGatewayBucketAlias. + *******************************************************************************/ + public FileGatewayQBitConfig withFileGatewayBucketAlias(String fileGatewayBucketAlias) + { + this.fileGatewayBucketAlias = fileGatewayBucketAlias; + return (this); + } + + + + /******************************************************************************* + ** Getter for uploadPartSizeMb. + *******************************************************************************/ + public int getUploadPartSizeMb() + { + return (uploadPartSizeMb); + } + + + + /******************************************************************************* + ** Fluent setter for uploadPartSizeMb. + *******************************************************************************/ + public FileGatewayQBitConfig withUploadPartSizeMb(int uploadPartSizeMb) + { + this.uploadPartSizeMb = uploadPartSizeMb; + return (this); + } + + + + /******************************************************************************* + ** Getter for uploadReorderWindowMb. + *******************************************************************************/ + public int getUploadReorderWindowMb() + { + return (uploadReorderWindowMb); + } + + + + /******************************************************************************* + ** Fluent setter for uploadReorderWindowMb. + *******************************************************************************/ + public FileGatewayQBitConfig withUploadReorderWindowMb(int uploadReorderWindowMb) + { + this.uploadReorderWindowMb = uploadReorderWindowMb; + return (this); + } + + + + /******************************************************************************* + ** Getter for downloadReadAheadMb. + *******************************************************************************/ + public int getDownloadReadAheadMb() + { + return (downloadReadAheadMb); + } + + + + /******************************************************************************* + ** Fluent setter for downloadReadAheadMb. + *******************************************************************************/ + public FileGatewayQBitConfig withDownloadReadAheadMb(int downloadReadAheadMb) + { + this.downloadReadAheadMb = downloadReadAheadMb; + return (this); + } + + + + /******************************************************************************* + ** Getter for maxGeneratedFileMb. + *******************************************************************************/ + public int getMaxGeneratedFileMb() + { + return (maxGeneratedFileMb); + } + + + + /******************************************************************************* + ** Fluent setter for maxGeneratedFileMb. + *******************************************************************************/ + public FileGatewayQBitConfig withMaxGeneratedFileMb(int maxGeneratedFileMb) + { + this.maxGeneratedFileMb = maxGeneratedFileMb; + return (this); + } + + + + /******************************************************************************* + ** Getter for uploadWorkerThreads. + *******************************************************************************/ + public int getUploadWorkerThreads() + { + return (uploadWorkerThreads); + } + + + + /******************************************************************************* + ** Fluent setter for uploadWorkerThreads. + *******************************************************************************/ + public FileGatewayQBitConfig withUploadWorkerThreads(int uploadWorkerThreads) + { + this.uploadWorkerThreads = uploadWorkerThreads; + return (this); + } + + + + /******************************************************************************* + ** Getter for dedupeWindow. + *******************************************************************************/ + public Duration getDedupeWindow() + { + return (dedupeWindow); + } + + + + /******************************************************************************* + ** Fluent setter for dedupeWindow. + *******************************************************************************/ + public FileGatewayQBitConfig withDedupeWindow(Duration dedupeWindow) + { + this.dedupeWindow = dedupeWindow; + return (this); + } + + + + /******************************************************************************* + ** Getter for jobStaleAfter. + *******************************************************************************/ + public Duration getJobStaleAfter() + { + return (jobStaleAfter); + } + + + + /******************************************************************************* + ** Fluent setter for jobStaleAfter. + *******************************************************************************/ + public FileGatewayQBitConfig withJobStaleAfter(Duration jobStaleAfter) + { + this.jobStaleAfter = jobStaleAfter; + return (this); + } + + + + /******************************************************************************* + ** Getter for temporaryUploadSuffixes. + *******************************************************************************/ + public List getTemporaryUploadSuffixes() + { + return (temporaryUploadSuffixes); + } + + + + /******************************************************************************* + ** Fluent setter for temporaryUploadSuffixes. + *******************************************************************************/ + public FileGatewayQBitConfig withTemporaryUploadSuffixes(List temporaryUploadSuffixes) + { + this.temporaryUploadSuffixes = temporaryUploadSuffixes; + return (this); + } + + + + /******************************************************************************* + ** Getter for lockoutThreshold. + *******************************************************************************/ + public int getLockoutThreshold() + { + return (lockoutThreshold); + } + + + + /******************************************************************************* + ** Fluent setter for lockoutThreshold. + *******************************************************************************/ + public FileGatewayQBitConfig withLockoutThreshold(int lockoutThreshold) + { + this.lockoutThreshold = lockoutThreshold; + return (this); + } + + + + /******************************************************************************* + ** Getter for lockoutWindow. + *******************************************************************************/ + public Duration getLockoutWindow() + { + return (lockoutWindow); + } + + + + /******************************************************************************* + ** Fluent setter for lockoutWindow. + *******************************************************************************/ + public FileGatewayQBitConfig withLockoutWindow(Duration lockoutWindow) + { + this.lockoutWindow = lockoutWindow; + return (this); + } + + + + /******************************************************************************* + ** Getter for lockoutDuration. + *******************************************************************************/ + public Duration getLockoutDuration() + { + return (lockoutDuration); + } + + + + /******************************************************************************* + ** Fluent setter for lockoutDuration. + *******************************************************************************/ + public FileGatewayQBitConfig withLockoutDuration(Duration lockoutDuration) + { + this.lockoutDuration = lockoutDuration; + return (this); + } + + + + /******************************************************************************* + ** Getter for maxConnectionsPerIpPerMinute. + *******************************************************************************/ + public int getMaxConnectionsPerIpPerMinute() + { + return (maxConnectionsPerIpPerMinute); + } + + + + /******************************************************************************* + ** Fluent setter for maxConnectionsPerIpPerMinute. + *******************************************************************************/ + public FileGatewayQBitConfig withMaxConnectionsPerIpPerMinute(int maxConnectionsPerIpPerMinute) + { + this.maxConnectionsPerIpPerMinute = maxConnectionsPerIpPerMinute; + return (this); + } + + + + /******************************************************************************* + ** Getter for defaultMaxConcurrentSessions. + *******************************************************************************/ + public int getDefaultMaxConcurrentSessions() + { + return (defaultMaxConcurrentSessions); + } + + + + /******************************************************************************* + ** Fluent setter for defaultMaxConcurrentSessions. + *******************************************************************************/ + public FileGatewayQBitConfig withDefaultMaxConcurrentSessions(int defaultMaxConcurrentSessions) + { + this.defaultMaxConcurrentSessions = defaultMaxConcurrentSessions; + return (this); + } + + + + /******************************************************************************* + ** Getter for defaultMaxOperationsPerMinute. + *******************************************************************************/ + public int getDefaultMaxOperationsPerMinute() + { + return (defaultMaxOperationsPerMinute); + } + + + + /******************************************************************************* + ** Fluent setter for defaultMaxOperationsPerMinute. + *******************************************************************************/ + public FileGatewayQBitConfig withDefaultMaxOperationsPerMinute(int defaultMaxOperationsPerMinute) + { + this.defaultMaxOperationsPerMinute = defaultMaxOperationsPerMinute; + return (this); + } + + + + /******************************************************************************* + ** Getter for defaultMaxFileSizeMb. + *******************************************************************************/ + public int getDefaultMaxFileSizeMb() + { + return (defaultMaxFileSizeMb); + } + + + + /******************************************************************************* + ** Fluent setter for defaultMaxFileSizeMb. + *******************************************************************************/ + public FileGatewayQBitConfig withDefaultMaxFileSizeMb(int defaultMaxFileSizeMb) + { + this.defaultMaxFileSizeMb = defaultMaxFileSizeMb; + return (this); + } + + + + /******************************************************************************* + ** Getter for defaultMaxUploadMbPerDay. + *******************************************************************************/ + public Integer getDefaultMaxUploadMbPerDay() + { + return (defaultMaxUploadMbPerDay); + } + + + + /******************************************************************************* + ** Fluent setter for defaultMaxUploadMbPerDay. + *******************************************************************************/ + public FileGatewayQBitConfig withDefaultMaxUploadMbPerDay(Integer defaultMaxUploadMbPerDay) + { + this.defaultMaxUploadMbPerDay = defaultMaxUploadMbPerDay; + return (this); + } + + + + /******************************************************************************* + ** Getter for defaultMaxDownloadMbPerDay. + *******************************************************************************/ + public Integer getDefaultMaxDownloadMbPerDay() + { + return (defaultMaxDownloadMbPerDay); + } + + + + /******************************************************************************* + ** Fluent setter for defaultMaxDownloadMbPerDay. + *******************************************************************************/ + public FileGatewayQBitConfig withDefaultMaxDownloadMbPerDay(Integer defaultMaxDownloadMbPerDay) + { + this.defaultMaxDownloadMbPerDay = defaultMaxDownloadMbPerDay; + return (this); + } + + + + /******************************************************************************* + ** Getter for scanner. + *******************************************************************************/ + public QCodeReference getScanner() + { + return (scanner); + } + + + + /******************************************************************************* + ** Fluent setter for scanner. + *******************************************************************************/ + public FileGatewayQBitConfig withScanner(QCodeReference scanner) + { + this.scanner = scanner; + return (this); + } + + + + /******************************************************************************* + ** Getter for pgpPrivateKeyEnvVarName. + *******************************************************************************/ + public String getPgpPrivateKeyEnvVarName() + { + return (pgpPrivateKeyEnvVarName); + } + + + + /******************************************************************************* + ** Fluent setter for pgpPrivateKeyEnvVarName. + *******************************************************************************/ + public FileGatewayQBitConfig withPgpPrivateKeyEnvVarName(String pgpPrivateKeyEnvVarName) + { + this.pgpPrivateKeyEnvVarName = pgpPrivateKeyEnvVarName; + return (this); + } + + + + /******************************************************************************* + ** Getter for pgpPassphraseEnvVarName. + *******************************************************************************/ + public String getPgpPassphraseEnvVarName() + { + return (pgpPassphraseEnvVarName); + } + + + + /******************************************************************************* + ** Fluent setter for pgpPassphraseEnvVarName. + *******************************************************************************/ + public FileGatewayQBitConfig withPgpPassphraseEnvVarName(String pgpPassphraseEnvVarName) + { + this.pgpPassphraseEnvVarName = pgpPassphraseEnvVarName; + return (this); + } + + + + /******************************************************************************* + ** Getter for logRetention. + *******************************************************************************/ + public Duration getLogRetention() + { + return (logRetention); + } + + + + /******************************************************************************* + ** Fluent setter for logRetention. + *******************************************************************************/ + public FileGatewayQBitConfig withLogRetention(Duration logRetention) + { + this.logRetention = logRetention; + return (this); + } + + + + /******************************************************************************* + ** Getter for archiveRetention. + *******************************************************************************/ + public Duration getArchiveRetention() + { + return (archiveRetention); + } + + + + /******************************************************************************* + ** Fluent setter for archiveRetention. + *******************************************************************************/ + public FileGatewayQBitConfig withArchiveRetention(Duration archiveRetention) + { + this.archiveRetention = archiveRetention; + return (this); + } + + + + /******************************************************************************* + ** Getter for outboxRetention. + *******************************************************************************/ + public Duration getOutboxRetention() + { + return (outboxRetention); + } + + + + /******************************************************************************* + ** Fluent setter for outboxRetention. + *******************************************************************************/ + public FileGatewayQBitConfig withOutboxRetention(Duration outboxRetention) + { + this.outboxRetention = outboxRetention; + return (this); + } + + + + /******************************************************************************* + ** Getter for resultsRetention. + *******************************************************************************/ + public Duration getResultsRetention() + { + return (resultsRetention); + } + + + + /******************************************************************************* + ** Fluent setter for resultsRetention. + *******************************************************************************/ + public FileGatewayQBitConfig withResultsRetention(Duration resultsRetention) + { + this.resultsRetention = resultsRetention; + return (this); + } + + + + /******************************************************************************* + ** Getter for receivingStaleAfter. + *******************************************************************************/ + public Duration getReceivingStaleAfter() + { + return (receivingStaleAfter); + } + + + + /******************************************************************************* + ** Fluent setter for receivingStaleAfter. + *******************************************************************************/ + public FileGatewayQBitConfig withReceivingStaleAfter(Duration receivingStaleAfter) + { + this.receivingStaleAfter = receivingStaleAfter; + return (this); + } + + + + /******************************************************************************* + ** Getter for tableMetaDataCustomizer. + *******************************************************************************/ + public MetaDataCustomizerInterface getTableMetaDataCustomizer() + { + return (tableMetaDataCustomizer); + } + + + + /******************************************************************************* + ** Fluent setter for tableMetaDataCustomizer. + *******************************************************************************/ + public FileGatewayQBitConfig withTableMetaDataCustomizer(MetaDataCustomizerInterface tableMetaDataCustomizer) + { + this.tableMetaDataCustomizer = tableMetaDataCustomizer; + return (this); + } + + + + /******************************************************************************* + ** Getter for schedulerName. + *******************************************************************************/ + public String getSchedulerName() + { + return (schedulerName); + } + + + + /******************************************************************************* + ** Fluent setter for schedulerName. + *******************************************************************************/ + public FileGatewayQBitConfig withSchedulerName(String schedulerName) + { + this.schedulerName = schedulerName; + return (this); + } + + + + /******************************************************************************* + ** Getter for serverRetention. + *******************************************************************************/ + public Duration getServerRetention() + { + return (serverRetention); + } + + + + /******************************************************************************* + ** Fluent setter for serverRetention. + *******************************************************************************/ + public FileGatewayQBitConfig withServerRetention(Duration serverRetention) + { + this.serverRetention = serverRetention; + return (this); + } + + + } diff --git a/src/main/java/com/kingsrook/qbits/filegateway/FileGatewayQBitProducer.java b/src/main/java/com/kingsrook/qbits/filegateway/FileGatewayQBitProducer.java index c6e5149..7fcf70b 100644 --- a/src/main/java/com/kingsrook/qbits/filegateway/FileGatewayQBitProducer.java +++ b/src/main/java/com/kingsrook/qbits/filegateway/FileGatewayQBitProducer.java @@ -21,17 +21,37 @@ package com.kingsrook.qbits.filegateway; +import java.util.List; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayProcesses; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qbits.filegateway.permissions.FileGatewayPermissions; +import com.kingsrook.qqq.backend.core.alerts.QAlertTypeMetaData; +import com.kingsrook.qqq.backend.core.exceptions.QException; import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput; import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppSection; +import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.ClientChannelsMetaData; import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData; import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaDataProducer; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; /******************************************************************************* - ** Entry point for the file-gateway qbit; no tables are produced yet. + ** Entry point for the file-gateway metadata; runtime startup belongs to the host. *******************************************************************************/ public class FileGatewayQBitProducer implements QBitMetaDataProducer { + /******************************************************************************* + ** The sftp wall owns its list (core's qbit prefixes are transitional and the + ** assistant may now read audits): partners never reach identity, audits, + ** webhooks or this qbit's own tables, read or write. Core's generic secret, + ** credential, permission and session patterns are added on top. + *******************************************************************************/ + public static final java.util.List SFTP_DENIED_PATTERNS = java.util.List.of( + "file_gateway_*", "audit*", "authentik_*", "webhooks_*", "worm_audit_*", "secured_data_*", "assistant_*"); + public static final String GROUP_ID = "com.kingsrook.qbits"; public static final String ARTIFACT_ID = "qbit-file-gateway"; public static final String VERSION = "0.1.0"; @@ -108,13 +128,62 @@ public class FileGatewayQBitProducer implements QBitMetaDataProducer widget.getName()).toList()); + output.add(app); + FileGatewayProcesses.processes().forEach(output::add); + output.add(com.kingsrook.qbits.filegateway.maintenance.GatewayMaintenance.metadata(qBitConfig)); + output.add(new QAlertTypeMetaData().withName("filegateway.lockout").withLabel("File Gateway Lockout") + .withSourceQBitName(qBit.getName()).withSubjectTableName(FileGatewayTables.ACCOUNT) + .withVisibilityPermission(FileGatewayPermissions.VIEW)); + output.addSelfToInstance(qInstance); + ClientChannelsMetaData channels = ClientChannelsMetaData.ofOrWithNew(qInstance) + .withChannel("sftp", "SFTP file gateway").withDeniedPattern("sftp", "file_gateway_*") + .withDeniedPattern("assistant", "file_gateway_*"); + for(String pattern : SFTP_DENIED_PATTERNS) + { + channels.withDeniedPattern("sftp", pattern); + } + for(String pattern : ClientChannelsMetaData.ASSISTANT_GENERIC_DENIED_PATTERNS) + { + channels.withDeniedPattern("sftp", pattern); + } + com.kingsrook.qbits.filegateway.api.FileGatewayOutbox.setProvider(new com.kingsrook.qbits.filegateway.outbox.OutboxService(qInstance, + new com.kingsrook.qbits.filegateway.storage.ObjectStorageGatewayStore(qInstance, qBitConfig.getFileGatewayBucketAlias()), qBitConfig, java.time.Clock.systemUTC())); + return (new MetaDataProducerMultiOutput()); } } diff --git a/src/main/java/com/kingsrook/qbits/filegateway/FileGatewayRuntime.java b/src/main/java/com/kingsrook/qbits/filegateway/FileGatewayRuntime.java new file mode 100644 index 0000000..c4858eb --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/FileGatewayRuntime.java @@ -0,0 +1,202 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.security.KeyPair; +import java.time.Clock; +import java.time.Duration; +import java.util.function.Function; +import com.kingsrook.qbits.filegateway.sftp.SftpProtocolServer; +import com.kingsrook.qbits.filegateway.storage.GatewayObjectStore; +import com.kingsrook.qbits.filegateway.storage.ObjectStorageGatewayStore; +import com.kingsrook.qbits.filegateway.upload.UploadJobWorker; +import com.kingsrook.qqq.backend.core.logging.QLogger; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.middleware.javalin.QApplicationJavalinServer; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInvoker; +import org.apache.sshd.common.NamedResource; +import org.apache.sshd.common.config.keys.KeyUtils; +import org.apache.sshd.common.util.security.SecurityUtils; + + +/******************************************************************************* + ** Host-started SFTP runtime. Host private keys are parsed in memory and never persisted. + *******************************************************************************/ +public final class FileGatewayRuntime implements AutoCloseable +{ + private static final QLogger LOG = QLogger.getLogger(FileGatewayRuntime.class); + private final UiApiInvoker invoker; + private final SftpProtocolServer protocol; + private final UploadJobWorker worker; + + + + /******************************************************************************* + ** Bind the host UI API server to the SFTP channel. + *******************************************************************************/ + private FileGatewayRuntime(QInstance instance, QApplicationJavalinServer server, FileGatewayQBitConfig config, KeyPair key, Clock clock, Duration interval, String podName, GatewayObjectStore store) + { + invoker = UiApiInvoker.forServer(server, "sftp"); + protocol = new SftpProtocolServer(instance, config, invoker, key, clock, interval, podName, store); + worker = new UploadJobWorker(instance, invoker, config, store, clock, podName); + } + + + + /******************************************************************************* + ** Start only when the registered qbit configuration enables SFTP. + *******************************************************************************/ + public static FileGatewayRuntime start(QInstance instance, QApplicationJavalinServer server) throws Exception + { + return (start(instance, server, System::getenv)); + } + + + + /******************************************************************************* + ** Start with an injectable environment reader. + *******************************************************************************/ + public static FileGatewayRuntime start(QInstance instance, QApplicationJavalinServer server, Function environment) throws Exception + { + var qbit = instance.getQBits().get(FileGatewayQBitProducer.GROUP_ID + ":" + FileGatewayQBitProducer.ARTIFACT_ID); + if(qbit == null || !(qbit.getConfig() instanceof FileGatewayQBitConfig config) || !config.getSftpEnabled()) + { + return (null); + } + FileGatewayRuntime runtime = new FileGatewayRuntime(instance, server, config, hostKey(config, environment), Clock.systemUTC(), Duration.ofSeconds(60), environment.apply("HOSTNAME"), new ObjectStorageGatewayStore(instance, config.getFileGatewayBucketAlias())); + runtime.worker.withEnvironment(environment); + runtime.protocol.start(); + runtime.worker.start(Duration.ofSeconds(60)); + return (runtime); + } + + + + /******************************************************************************* + ** Start with an explicit ephemeral port, in-memory host key and controllable clock. + *******************************************************************************/ + public static FileGatewayRuntime startForTest(QInstance instance, QApplicationJavalinServer server, FileGatewayQBitConfig config, KeyPair key, Clock clock, Duration pollInterval) throws IOException + { + return (startForTest(instance, server, config, key, clock, pollInterval, new ObjectStorageGatewayStore(instance, config.getFileGatewayBucketAlias()))); + } + + + + /******************************************************************************* + ** Transfer tests supply an in-memory store with no production fallback. + *******************************************************************************/ + public static FileGatewayRuntime startForTest(QInstance instance, QApplicationJavalinServer server, FileGatewayQBitConfig config, KeyPair key, Clock clock, Duration pollInterval, GatewayObjectStore store) throws IOException + { + FileGatewayRuntime runtime = new FileGatewayRuntime(instance, server, config, key, clock, pollInterval, "test", store); + runtime.protocol.start(); + return (runtime); + } + + + + /******************************************************************************* + ** Read an OpenSSH private key directly from memory. + *******************************************************************************/ + public static KeyPair hostKey(FileGatewayQBitConfig config, Function environment) throws Exception + { + String pem = environment.apply(config.getHostKeyEnvVarName()); + if(pem == null || pem.isBlank()) + { + if(!config.getAllowGeneratedHostKey()) + { + throw (new IllegalStateException("SFTP host key environment variable is missing: " + config.getHostKeyEnvVarName())); + } + LOG.warn("Using a generated in-memory SFTP host key; do not use in production"); + return (KeyUtils.generateKeyPair("ecdsa-sha2-nistp256", 256)); + } + try(var stream = new ByteArrayInputStream(pem.getBytes(StandardCharsets.UTF_8))) + { + var keys = SecurityUtils.loadKeyPairIdentities(null, NamedResource.ofName("environment-host-key"), stream, null).iterator(); + if(!keys.hasNext()) + { + throw (new IllegalArgumentException("No SFTP host key in environment variable")); + } + return (keys.next()); + } + } + + + + /******************************************************************************* + ** Stop the SFTP runtime. + *******************************************************************************/ + public void stop() throws IOException + { + protocol.stop(); + try + { + worker.close(); + } + catch(com.kingsrook.qqq.backend.core.exceptions.QException e) + { + throw (new IOException("Unable to release upload jobs", e)); + } + } + + + + /******************************************************************************* + ** Close this runtime. + *******************************************************************************/ + @Override + public void close() throws IOException + { + stop(); + } + + + + /******************************************************************************* + ** Access the protocol listener. + *******************************************************************************/ + public SftpProtocolServer getProtocol() + { + return (protocol); + } + + + + /******************************************************************************* + ** The fixed sftp-channel invoker. + *******************************************************************************/ + public UiApiInvoker getInvoker() + { + return (invoker); + } + + /******************************************************************************* + ** Deterministic worker hook for hosts and integration tests. + *******************************************************************************/ + public UploadJobWorker getWorker() + { + return (worker); + } + +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/api/FileAcceptance.java b/src/main/java/com/kingsrook/qbits/filegateway/api/FileAcceptance.java new file mode 100644 index 0000000..23375bb --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/api/FileAcceptance.java @@ -0,0 +1,127 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.api; + + +import java.time.Duration; + + +/******************************************************************************* + ** Consumer decision, including its reference or reason and retry delay. + *******************************************************************************/ +public final class FileAcceptance +{ + /******************************************************************************* + ** Delivery decisions. + *******************************************************************************/ + public enum Status + { + ACCEPTED, REJECTED, RETRY_LATER + } + + private final Status status; + private final String consumerReference; + private final String reason; + private final Duration retryAfter; + + + + /******************************************************************************* + ** Create an immutable decision. + *******************************************************************************/ + private FileAcceptance(Status status, String consumerReference, String reason, Duration retryAfter) + { + this.status = status; + this.consumerReference = consumerReference; + this.reason = reason; + this.retryAfter = retryAfter; + } + + + + /******************************************************************************* + ** Accept the delivery. + *******************************************************************************/ + public static FileAcceptance accepted(String consumerReference) + { + return (new FileAcceptance(Status.ACCEPTED, consumerReference, null, null)); + } + + + + /******************************************************************************* + ** Reject the delivery. + *******************************************************************************/ + public static FileAcceptance rejected(String reason) + { + return (new FileAcceptance(Status.REJECTED, null, reason, null)); + } + + + + /******************************************************************************* + ** Request a retry. + *******************************************************************************/ + public static FileAcceptance retryLater(String reason, Duration retryAfter) + { + return (new FileAcceptance(Status.RETRY_LATER, null, reason, retryAfter)); + } + + + + /******************************************************************************* + ** Getter for status. + *******************************************************************************/ + public Status getStatus() + { + return (status); + } + + + + /******************************************************************************* + ** Getter for consumerReference. + *******************************************************************************/ + public String getConsumerReference() + { + return (consumerReference); + } + + + + /******************************************************************************* + ** Getter for reason. + *******************************************************************************/ + public String getReason() + { + return (reason); + } + + + + /******************************************************************************* + ** Getter for retryAfter. + *******************************************************************************/ + public Duration getRetryAfter() + { + return (retryAfter); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayConsumer.java b/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayConsumer.java new file mode 100644 index 0000000..1608bd8 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayConsumer.java @@ -0,0 +1,49 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.api; + + + + +/******************************************************************************* + ** Consumer SPI for at-least-once inbound delivery and download notification. + *******************************************************************************/ +public interface FileGatewayConsumer +{ + /******************************************************************************* + ** Stable lowercase consumer name. + *******************************************************************************/ + String getName(); + + /******************************************************************************* + ** Accept, reject or request a later retry. + *******************************************************************************/ + FileAcceptance accept(InboundFile file) throws Exception; + + + + /******************************************************************************* + ** Called after the first complete download. + *******************************************************************************/ + default void onDownloaded(OutboxFileRef ref) throws Exception + { + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayConsumersMetaData.java b/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayConsumersMetaData.java new file mode 100644 index 0000000..393b0de --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayConsumersMetaData.java @@ -0,0 +1,79 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.api; + + +import java.util.ArrayList; +import java.util.List; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.metadata.QSupplementalInstanceMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; + + +/******************************************************************************* + ** Host-declared consumer implementations; contains no gateway runtime dependencies. + *******************************************************************************/ +public class FileGatewayConsumersMetaData implements QSupplementalInstanceMetaData +{ + public static final String NAME = "fileGatewayConsumers"; + private final List consumers = new ArrayList<>(); + + + + /******************************************************************************* + ** Metadata registry name. + *******************************************************************************/ + public String getName() + { + return (NAME); + } + + + + /******************************************************************************* + ** Get or install the registry. + *******************************************************************************/ + public static FileGatewayConsumersMetaData ofOrWithNew(QInstance instance) + { + return (QSupplementalInstanceMetaData.ofOrWithNew(instance, NAME, FileGatewayConsumersMetaData::new)); + } + + + + /******************************************************************************* + ** Register a consumer implementation. + *******************************************************************************/ + public FileGatewayConsumersMetaData withConsumer(QCodeReference consumer) + { + consumers.add(consumer); + return (this); + } + + + + /******************************************************************************* + ** Registered consumer implementations. + *******************************************************************************/ + public List getConsumers() + { + return (List.copyOf(consumers)); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayOutbox.java b/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayOutbox.java new file mode 100644 index 0000000..2f945ec --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayOutbox.java @@ -0,0 +1,78 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.api; + + +import java.io.InputStream; +import com.kingsrook.qqq.backend.core.exceptions.QException; + + +/******************************************************************************* + ** Optional outbox facade; the host runtime owns provider registration. + *******************************************************************************/ +public final class FileGatewayOutbox +{ + private static volatile FileGatewayOutboxProvider provider; + + + + /******************************************************************************* + ** Static facade. + *******************************************************************************/ + private FileGatewayOutbox() + { + } + + + + /******************************************************************************* + ** Install or clear the runtime provider. + *******************************************************************************/ + public static void setProvider(FileGatewayOutboxProvider provider) + { + FileGatewayOutbox.provider = provider; + } + + + + /******************************************************************************* + ** Whether the runtime supports outbox writes. + *******************************************************************************/ + public static boolean isAvailable() + { + return (provider != null); + } + + + + /******************************************************************************* + ** Delegate to the installed provider. + *******************************************************************************/ + public static OutboxFileRef put(String endpointName, String fileName, InputStream content, long sizeBytes) throws FileGatewayUnavailableException, QException + { + FileGatewayOutboxProvider current = provider; + if(current == null) + { + throw (new FileGatewayUnavailableException()); + } + return (current.put(endpointName, fileName, content, sizeBytes)); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayOutboxProvider.java b/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayOutboxProvider.java new file mode 100644 index 0000000..95d883e --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayOutboxProvider.java @@ -0,0 +1,37 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.api; + + +import java.io.InputStream; +import com.kingsrook.qqq.backend.core.exceptions.QException; + + +/******************************************************************************* + ** Runtime implementation registered with the outbox facade. + *******************************************************************************/ +public interface FileGatewayOutboxProvider +{ + /******************************************************************************* + ** Stream content into the endpoint outbox. + *******************************************************************************/ + OutboxFileRef put(String endpointName, String fileName, InputStream content, long sizeBytes) throws QException; +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayScannerInterface.java b/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayScannerInterface.java new file mode 100644 index 0000000..6b9699b --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayScannerInterface.java @@ -0,0 +1,41 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.api; + +/******************************************************************************* + ** Optional streaming scanner invoked before delivery to a consumer. + *******************************************************************************/ +@FunctionalInterface +public interface FileGatewayScannerInterface +{ + /******************************************************************************* + ** ERROR is transient; INFECTED quarantines the object without delivery. + *******************************************************************************/ + enum Result + { + CLEAN, INFECTED, ERROR + } + + /******************************************************************************* + ** Inspect the file with fresh streams; implementations must close opened streams. + *******************************************************************************/ + Result scan(InboundFile file) throws Exception; +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayUnavailableException.java b/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayUnavailableException.java new file mode 100644 index 0000000..43a0b40 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/api/FileGatewayUnavailableException.java @@ -0,0 +1,41 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.api; + + +import com.kingsrook.qqq.backend.core.exceptions.QException; + + +/******************************************************************************* + ** The outbox runtime has not been installed. + *******************************************************************************/ +public class FileGatewayUnavailableException extends QException +{ + + + /******************************************************************************* + ** Create the unavailable error. + *******************************************************************************/ + public FileGatewayUnavailableException() + { + super("File gateway outbox is unavailable"); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/api/InboundFile.java b/src/main/java/com/kingsrook/qbits/filegateway/api/InboundFile.java new file mode 100644 index 0000000..7272bdd --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/api/InboundFile.java @@ -0,0 +1,45 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.api; + + +import java.io.InputStream; +import java.time.Instant; +import java.util.function.Supplier; + + +/******************************************************************************* + ** An inbound object; each openStream call supplies a fresh stream. + *******************************************************************************/ +public record InboundFile(Integer uploadJobId, String endpointName, String username, String userId, + String remoteAddress, String path, String fileName, long sizeBytes, String sha256, + Instant receivedAt, String storageBucketAlias, String storageKey, Supplier streamSupplier) +{ + + + /******************************************************************************* + ** Open a fresh object-storage stream. + *******************************************************************************/ + public InputStream openStream() + { + return (streamSupplier.get()); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/api/OutboxFileRef.java b/src/main/java/com/kingsrook/qbits/filegateway/api/OutboxFileRef.java new file mode 100644 index 0000000..9181c66 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/api/OutboxFileRef.java @@ -0,0 +1,33 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.api; + + +import java.time.Instant; + + +/******************************************************************************* + ** Identity and metadata of a file waiting for partner pickup. + *******************************************************************************/ +public record OutboxFileRef(Integer outboxFileId, String endpointName, String path, String fileName, + long sizeBytes, String sha256, Instant createdAt) +{ +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/maintenance/GatewayMaintenance.java b/src/main/java/com/kingsrook/qbits/filegateway/maintenance/GatewayMaintenance.java new file mode 100644 index 0000000..c8d5286 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/maintenance/GatewayMaintenance.java @@ -0,0 +1,217 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.maintenance; + +import java.io.IOException; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.util.List; +import com.kingsrook.qbits.filegateway.FileGatewayQBitConfig; +import com.kingsrook.qbits.filegateway.FileGatewayQBitProducer; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qbits.filegateway.outbox.OutboxService; +import com.kingsrook.qbits.filegateway.sftp.AuditWriter; +import com.kingsrook.qbits.filegateway.storage.GatewayObjectStore; +import com.kingsrook.qbits.filegateway.storage.ObjectStorageGatewayStore; +import com.kingsrook.qbits.filegateway.upload.UploadJobs; +import com.kingsrook.qqq.backend.core.actions.processes.BackendStep; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QBackendStepMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.scheduleing.QScheduleMetaData; + +/******************************************************************************* + ** Idempotent daily cleanup and at-least-once outbox callback retries. + *******************************************************************************/ +public class GatewayMaintenance implements BackendStep +{ + public static final String NAME = "file_gateway_retention"; + + + + /******************************************************************************* + ** Hidden host-scheduled process; no navigation or user action is registered. + *******************************************************************************/ + public static QProcessMetaData metadata(FileGatewayQBitConfig config) + { + QProcessMetaData process = new QProcessMetaData().withName(NAME).withLabel("File gateway retention").withIsHidden(true) + .withPermissionRules(FileGatewayTables.permissionRules(NAME)) + .withStep(new QBackendStepMetaData().withName("run").withCode(new QCodeReference(GatewayMaintenance.class))); + if(config.getSchedulerName() != null) + { + process.withSchedule(new QScheduleMetaData().withSchedulerName(config.getSchedulerName()).withRepeatSeconds(86400).withInitialDelaySeconds(60)); + } + return (process); + } + + + + /******************************************************************************* + ** Resolve registered host configuration without starting the protocol runtime. + *******************************************************************************/ + @Override + public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + QInstance instance = QContext.getQInstance(); + FileGatewayQBitConfig config = (FileGatewayQBitConfig) instance.getQBits().get(FileGatewayQBitProducer.GROUP_ID + ":" + FileGatewayQBitProducer.ARTIFACT_ID).getConfig(); + maintain(instance, config, new ObjectStorageGatewayStore(instance, config.getFileGatewayBucketAlias()), Clock.systemUTC()); + } + + + + /******************************************************************************* + ** Cursor batches remain correct while rows are deleted, and bound memory use. + *******************************************************************************/ + public static void maintain(QInstance instance, FileGatewayQBitConfig config, GatewayObjectStore store, Clock clock) throws QException + { + AuditWriter audit = new AuditWriter(instance); + OutboxService outbox = new OutboxService(instance, store, config, clock); + for(String table : List.of(FileGatewayTables.UPLOAD_JOB, FileGatewayTables.OUTBOX_FILE, FileGatewayTables.TRANSFER_LOG, FileGatewayTables.CONNECTION_LOG, FileGatewayTables.SERVER)) + { + int last = 0; + while(true) + { + var rows = audit.query(table, new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.GREATER_THAN, last)) + .withOrderBy(new QFilterOrderBy("id", true)).withLimit(250)); + if(rows.isEmpty()) + { + break; + } + for(QRecord row : rows) + { + last = row.getValueInteger("id"); + try + { + if(FileGatewayTables.UPLOAD_JOB.equals(table)) + { + audit.withAccountLock("id", row.getValue("accountId"), () -> + { + try + { + cleanUpload(audit, store, audit.query(table, UploadJobs.equal("id", row.getValue("id"))).getFirst(), config, clock.instant()); + } + catch(IOException e) + { + throw (new QException("Upload retention storage failure", e)); + } + return (null); + }); + } + else if(FileGatewayTables.OUTBOX_FILE.equals(table)) + { + outbox.notifyDownloaded(row); + if("REMOVED".equals(row.getValueString("status")) || older(row, "createDate", config.getOutboxRetention(), clock.instant()) + || (row.getValueInstant("expiresAt") != null && !row.getValueInstant("expiresAt").isAfter(clock.instant()))) + { + deleteObject(store, row, "storageKey"); + audit.update(table, new QRecord().withValue("id", row.getValue("id")).withValue("status", "EXPIRED").withValue("storageKey", null)); + } + } + else if(FileGatewayTables.SERVER.equals(table)) + { + if(older(row, "lastSeenAt", config.getServerRetention(), clock.instant())) + { + audit.delete(table, row.getValue("id")); + } + } + else if(older(row, "createDate", config.getLogRetention(), clock.instant()) + && (!FileGatewayTables.CONNECTION_LOG.equals(table) || older(row, "lastSeenAt", Duration.ofMinutes(3), clock.instant()))) + { + audit.delete(table, row.getValue("id")); + } + } + catch(IOException e) + { + throw (new QException("File gateway retention storage failure", e)); + } + } + } + } + } + + + + /******************************************************************************* + ** Never delete an object owned by an active worker; retain rows for the audit trail. + *******************************************************************************/ + private static void cleanUpload(AuditWriter audit, GatewayObjectStore store, QRecord row, FileGatewayQBitConfig config, Instant now) throws IOException, QException + { + String status = row.getValueString("status"); + QRecord changes = new QRecord().withValue("id", row.getValue("id")); + if("RECEIVING".equals(status) && older(row, "createDate", config.getReceivingStaleAfter(), now)) + { + deleteObject(store, row, "storageKey"); + changes.withValue("status", "ABORTED").withValue("storageKey", null).withValue("finishedAt", now).withValue("errorMessage", "Incomplete upload expired"); + } + else if(!List.of("RECEIVING", "RUNNING", "QUEUED").contains(status)) + { + if(older(row, "createDate", config.getArchiveRetention(), now)) + { + deleteObject(store, row, "storageKey"); + deleteObject(store, row, "plaintextStorageKey"); + changes.withValue("storageKey", null).withValue("plaintextStorageKey", null); + } + if(older(row, "createDate", config.getResultsRetention(), now)) + { + deleteObject(store, row, "resultsStorageKey"); + changes.withValue("resultsStorageKey", null).withValue("resultsSizeBytes", null); + } + } + if(changes.getValues().size() > 1) + { + audit.update(FileGatewayTables.UPLOAD_JOB, changes); + } + } + + + + /******************************************************************************* + ** Storage deletes precede database clearing so transient errors can be retried. + *******************************************************************************/ + private static void deleteObject(GatewayObjectStore store, QRecord row, String field) throws IOException + { + if(row.getValueString(field) != null) + { + store.delete(row.getValueString(field)); + } + } + + + + /******************************************************************************* + ** Null timestamps are not evidence that a record is old enough to remove. + *******************************************************************************/ + private static boolean older(QRecord row, String field, Duration age, Instant now) + { + return (row.getValueInstant(field) != null && !row.getValueInstant(field).plus(age).isAfter(now)); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/metadata/AccountCustomizer.java b/src/main/java/com/kingsrook/qbits/filegateway/metadata/AccountCustomizer.java new file mode 100644 index 0000000..a412c08 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/metadata/AccountCustomizer.java @@ -0,0 +1,78 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.metadata; + +import java.util.List; +import java.util.Optional; +import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface; +import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.statusmessages.BadInputStatusMessage; + +/******************************************************************************* + ** Reject account names that cannot be used as stable SSH login names. + *******************************************************************************/ +public class AccountCustomizer implements TableCustomizerInterface +{ + /******************************************************************************* + ** Partial updates preserve the current username; supplied names must be valid. + *******************************************************************************/ + @Override + public List preInsertOrUpdate(AbstractActionInput input, List records, boolean preview, Optional> oldRecords) + { + for(QRecord record : records) + { + if(record.getValues().containsKey("username") && (record.getValueString("username") == null || !record.getValueString("username").matches("[a-z0-9._-]{3,64}"))) + { + record.addError(new BadInputStatusMessage("Username must be 3 to 64 lowercase letters, digits, dots, underscores or hyphens")); + } + if(oldRecords.isEmpty() && record.getValue("rootMode") == null) + { + record.setValue("rootMode", "SERVICE".equals(record.getValueString("accountType")) ? "ENDPOINTS_ONLY" : "FULL_TREE"); + } + String pgpKey = record.getValueString("pgpPublicKey"); + if(pgpKey != null && !pgpKey.isBlank()) + { + try + { + com.kingsrook.qbits.filegateway.pgp.PgpStreams.encryptionKey(pgpKey); + } + catch(Exception e) + { + record.addError(new BadInputStatusMessage("PGP public key must parse and contain a valid encryption subkey")); + } + } + if(Boolean.TRUE.equals(record.getValueBoolean("pgpEncryptDownloads")) && (pgpKey == null || pgpKey.isBlank()) + && oldRecords.orElse(List.of()).stream().noneMatch(old -> java.util.Objects.equals(old.getValue("id"), record.getValue("id")) && old.getValueString("pgpPublicKey") != null)) + { + record.addError(new BadInputStatusMessage("PGP download encryption requires a public key")); + } + for(String limit : List.of("maxConcurrentSessions", "maxOperationsPerMinute", "maxFileSizeMb", "maxUploadMbPerDay", "maxDownloadMbPerDay")) + { + if(record.getValueInteger(limit) != null && record.getValueInteger(limit) < 1) + { + record.addError(new BadInputStatusMessage(limit + " must be positive, or blank for an unlimited daily quota")); + } + } + } + return (records); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/metadata/AliasCustomizer.java b/src/main/java/com/kingsrook/qbits/filegateway/metadata/AliasCustomizer.java new file mode 100644 index 0000000..7459857 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/metadata/AliasCustomizer.java @@ -0,0 +1,104 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.metadata; + +import java.util.List; +import java.util.Optional; +import com.kingsrook.qbits.filegateway.sftp.AuditWriter; +import com.kingsrook.qbits.filegateway.upload.UploadJobs; +import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.statusmessages.BadInputStatusMessage; + +/******************************************************************************* + ** Endpoints are canonical mounts, never a way to shadow application paths. + *******************************************************************************/ +public class AliasCustomizer implements TableCustomizerInterface +{ + /******************************************************************************* + ** Validate the merged row so sparse updates cannot bypass mount validation. + *******************************************************************************/ + @Override + public List preInsertOrUpdate(AbstractActionInput input, List records, boolean preview, Optional> oldRecords) throws QException + { + AuditWriter audit = new AuditWriter(QContext.getQInstance()); + for(QRecord record : records) + { + QRecord merged = new QRecord(); + oldRecords.orElse(List.of()).stream().filter(old -> java.util.Objects.equals(old.getValue("id"), record.getValue("id"))) + .findFirst().ifPresent(old -> merged.getValues().putAll(old.getValues())); + merged.getValues().putAll(record.getValues()); + String path = merged.getValueString("path"); + String error = null; + if(path == null || !path.matches("/[a-z0-9._/-]+") || path.contains("..") || path.contains("//") || path.endsWith("/") || List.of(path.split("/")).contains(".")) + { + error = "Endpoint path must be absolute, use lowercase letters, digits, dots, underscores, hyphens and slashes, and contain no dot segments or empty segments"; + } + else + { + boolean fullTree = merged.getValue("accountId") == null; + if(!fullTree) + { + var accounts = audit.query(FileGatewayTables.ACCOUNT, UploadJobs.equal("id", merged.getValue("accountId"))); + fullTree = accounts.isEmpty() || !"ENDPOINTS_ONLY".equals(accounts.getFirst().getValueString("rootMode")); + } + if(fullTree && QContext.getQInstance().getApps().containsKey(path.substring(1).split("/")[0])) + { + error = "Endpoint path collides with an application name for a FULL_TREE account"; + } + for(QRecord other : audit.query(FileGatewayTables.ALIAS, new QQueryFilter())) + { + String mount = other.getValueString("path"); + if(!java.util.Objects.equals(other.getValue("id"), merged.getValue("id")) && mount != null + && (path.equals(mount) || path.startsWith(mount + "/") || mount.startsWith(path + "/"))) + { + error = "Endpoint paths must be unique and must not overlap another endpoint mount"; + } + } + } + String filename = merged.getValueString("fileName"); + if(filename != null && !validFileName(filename)) + { + error = "Endpoint file name must be a single safe filename"; + } + if(error != null) + { + record.addError(new BadInputStatusMessage(error)); + } + } + return (records); + } + + + + /******************************************************************************* + ** Object names cannot escape a mount or inject control characters. + *******************************************************************************/ + public static boolean validFileName(String name) + { + return (name != null && !name.isBlank() && !name.equals(".") && !name.equals("..") && !name.contains("/") + && !name.contains("\\") && !name.contains(":") && name.chars().noneMatch(Character::isISOControl)); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/metadata/FileGatewayEnums.java b/src/main/java/com/kingsrook/qbits/filegateway/metadata/FileGatewayEnums.java new file mode 100644 index 0000000..6ff2062 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/metadata/FileGatewayEnums.java @@ -0,0 +1,519 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.metadata; + + +import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PossibleValueEnum; + + +/******************************************************************************* + ** Stable persisted values for the gateway schema. + *******************************************************************************/ +public final class FileGatewayEnums +{ + + + /******************************************************************************* + ** Namespace for schema values. + *******************************************************************************/ + private FileGatewayEnums() + { + } + + + /******************************************************************************* + ** AccountType values. + *******************************************************************************/ + public enum AccountType implements PossibleValueEnum + { + USER("USER"), SERVICE("SERVICE"); + + private final String value; + + /******************************************************************************* + ** Construct a persisted value. + *******************************************************************************/ + AccountType(String value) + { + this.value = value; + } + + + /******************************************************************************* + ** Possible value id. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (value); + } + + + /******************************************************************************* + ** Possible value label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (label(value)); + } + } + + + /******************************************************************************* + ** AccountStatus values. + *******************************************************************************/ + public enum AccountStatus implements PossibleValueEnum + { + ACTIVE("ACTIVE"), DISABLED("DISABLED"), LOCKED("LOCKED"); + + private final String value; + + /******************************************************************************* + ** Construct a persisted value. + *******************************************************************************/ + AccountStatus(String value) + { + this.value = value; + } + + + /******************************************************************************* + ** Possible value id. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (value); + } + + + /******************************************************************************* + ** Possible value label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (label(value)); + } + } + + + /******************************************************************************* + ** RootMode values. + *******************************************************************************/ + public enum RootMode implements PossibleValueEnum + { + FULL_TREE("FULL_TREE"), ENDPOINTS_ONLY("ENDPOINTS_ONLY"); + + private final String value; + + /******************************************************************************* + ** Construct a persisted value. + *******************************************************************************/ + RootMode(String value) + { + this.value = value; + } + + + /******************************************************************************* + ** Possible value id. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (value); + } + + + /******************************************************************************* + ** Possible value label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (label(value)); + } + } + + + /******************************************************************************* + ** AliasKind values. + *******************************************************************************/ + public enum AliasKind implements PossibleValueEnum + { + BULK_LOAD("BULK_LOAD"), INBOX("INBOX"), VIEW("VIEW"), REPORT("REPORT"), OUTBOX("OUTBOX"); + + private final String value; + + /******************************************************************************* + ** Construct a persisted value. + *******************************************************************************/ + AliasKind(String value) + { + this.value = value; + } + + + /******************************************************************************* + ** Possible value id. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (value); + } + + + /******************************************************************************* + ** Possible value label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (label(value)); + } + } + + + /******************************************************************************* + ** UploadStatus values. + *******************************************************************************/ + public enum UploadStatus implements PossibleValueEnum + { + RECEIVING("RECEIVING"), QUEUED("QUEUED"), RUNNING("RUNNING"), SUCCEEDED("SUCCEEDED"), PARTIAL("PARTIAL"), FAILED("FAILED"), REJECTED("REJECTED"), DUPLICATE("DUPLICATE"), ABORTED("ABORTED"); + + private final String value; + + /******************************************************************************* + ** Construct a persisted value. + *******************************************************************************/ + UploadStatus(String value) + { + this.value = value; + } + + + /******************************************************************************* + ** Possible value id. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (value); + } + + + /******************************************************************************* + ** Possible value label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (label(value)); + } + } + + + /******************************************************************************* + ** OutboxStatus values. + *******************************************************************************/ + public enum OutboxStatus implements PossibleValueEnum + { + READY("READY"), DOWNLOADED("DOWNLOADED"), REMOVED("REMOVED"), EXPIRED("EXPIRED"); + + private final String value; + + /******************************************************************************* + ** Construct a persisted value. + *******************************************************************************/ + OutboxStatus(String value) + { + this.value = value; + } + + + /******************************************************************************* + ** Possible value id. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (value); + } + + + /******************************************************************************* + ** Possible value label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (label(value)); + } + } + + + /******************************************************************************* + ** Operation values. + *******************************************************************************/ + public enum Operation implements PossibleValueEnum + { + LIST("LIST"), UPLOAD("UPLOAD"), DOWNLOAD("DOWNLOAD"), RENAME("RENAME"), REMOVE("REMOVE"); + + private final String value; + + /******************************************************************************* + ** Construct a persisted value. + *******************************************************************************/ + Operation(String value) + { + this.value = value; + } + + + /******************************************************************************* + ** Possible value id. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (value); + } + + + /******************************************************************************* + ** Possible value label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (label(value)); + } + } + + + /******************************************************************************* + ** ObjectKind values. + *******************************************************************************/ + public enum ObjectKind implements PossibleValueEnum + { + APP("APP"), TABLE("TABLE"), PROFILE("PROFILE"), VIEW("VIEW"), REPORT("REPORT"), RESULT("RESULT"), ARCHIVE("ARCHIVE"), ENDPOINT("ENDPOINT"), OUTBOX("OUTBOX"); + + private final String value; + + /******************************************************************************* + ** Construct a persisted value. + *******************************************************************************/ + ObjectKind(String value) + { + this.value = value; + } + + + /******************************************************************************* + ** Possible value id. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (value); + } + + + /******************************************************************************* + ** Possible value label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (label(value)); + } + } + + + /******************************************************************************* + ** TransferOutcome values. + *******************************************************************************/ + public enum TransferOutcome implements PossibleValueEnum + { + OK("OK"), DENIED("DENIED"), NOT_FOUND("NOT_FOUND"), LIMIT("LIMIT"), ERROR("ERROR"); + + private final String value; + + /******************************************************************************* + ** Construct a persisted value. + *******************************************************************************/ + TransferOutcome(String value) + { + this.value = value; + } + + + /******************************************************************************* + ** Possible value id. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (value); + } + + + /******************************************************************************* + ** Possible value label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (label(value)); + } + } + + + /******************************************************************************* + ** ConnectionOutcome values. + *******************************************************************************/ + public enum ConnectionOutcome implements PossibleValueEnum + { + CONNECTED("CONNECTED"), AUTH_FAILED("AUTH_FAILED"), LOCKED_OUT("LOCKED_OUT"), IP_DENIED("IP_DENIED"), EXPIRED("EXPIRED"), DISABLED("DISABLED"), LIMIT("LIMIT"); + + private final String value; + + /******************************************************************************* + ** Construct a persisted value. + *******************************************************************************/ + ConnectionOutcome(String value) + { + this.value = value; + } + + + /******************************************************************************* + ** Possible value id. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (value); + } + + + /******************************************************************************* + ** Possible value label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (label(value)); + } + } + + + /******************************************************************************* + ** Format values. + *******************************************************************************/ + public enum Format implements PossibleValueEnum + { + CSV("csv"), XLSX("xlsx"), JSON("json"); + + private final String value; + + /******************************************************************************* + ** Construct a persisted value. + *******************************************************************************/ + Format(String value) + { + this.value = value; + } + + + /******************************************************************************* + ** Possible value id. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (value); + } + + + /******************************************************************************* + ** Possible value label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (label(value)); + } + } + + + /******************************************************************************* + ** KeyType values. + *******************************************************************************/ + public enum KeyType implements PossibleValueEnum + { + SSH_ED25519("ssh-ed25519"), ECDSA_SHA2_NISTP256("ecdsa-sha2-nistp256"), ECDSA_SHA2_NISTP384("ecdsa-sha2-nistp384"), ECDSA_SHA2_NISTP521("ecdsa-sha2-nistp521"), SSH_RSA("ssh-rsa"); + + private final String value; + + /******************************************************************************* + ** Construct a persisted value. + *******************************************************************************/ + KeyType(String value) + { + this.value = value; + } + + + /******************************************************************************* + ** Possible value id. + *******************************************************************************/ + @Override + public String getPossibleValueId() + { + return (value); + } + + + /******************************************************************************* + ** Possible value label. + *******************************************************************************/ + @Override + public String getPossibleValueLabel() + { + return (label(value)); + } + } + + + + /******************************************************************************* + ** Plain label for a persisted value: ENDPOINTS_ONLY becomes "Endpoints only". + *******************************************************************************/ + static String label(String value) + { + String words = value.replace('_', ' ').toLowerCase(); + return (Character.toUpperCase(words.charAt(0)) + words.substring(1)); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/metadata/FileGatewayHome.java b/src/main/java/com/kingsrook/qbits/filegateway/metadata/FileGatewayHome.java new file mode 100644 index 0000000..7157a78 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/metadata/FileGatewayHome.java @@ -0,0 +1,229 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.metadata; + +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.filegateway.permissions.FileGatewayPermissions; +import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.AbstractWidgetRenderer; +import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetInput; +import com.kingsrook.qqq.backend.core.model.actions.widgets.RenderWidgetOutput; +import com.kingsrook.qqq.backend.core.model.dashboard.widgets.QWidgetData; +import com.kingsrook.qqq.backend.core.model.dashboard.widgets.StatisticsData; +import com.kingsrook.qqq.backend.core.model.dashboard.widgets.TableData; +import com.kingsrook.qqq.backend.core.model.dashboard.widgets.WidgetType; +import com.kingsrook.qqq.backend.core.model.dashboard.widgets.blocks.text.TextBlockData; +import com.kingsrook.qqq.backend.core.model.dashboard.widgets.blocks.text.TextValues; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData; + +/******************************************************************************* + ** Typed home widgets over real gateway data, queried as the viewing user. + *******************************************************************************/ +public class FileGatewayHome extends AbstractWidgetRenderer +{ + /** Readable UTC timestamps in home tables (the widget title already names the table). */ + private static final java.time.format.DateTimeFormatter WHEN = java.time.format.DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm 'UTC'").withZone(ZoneOffset.UTC); + + public static final List KEYS = List.of("transfers", "failures", "sessions", "in", "out", "locked", "recent", "partners", "active", "servers"); + private static final List LABELS = List.of("Transfers today", "Failed jobs (24 hours)", "Active sessions", "MB in today", "MB out today", "Locked accounts", + "Recent failures", "Top partners (7 days)", "Active sessions", "Server"); + + + + /******************************************************************************* + ** Six small statistics followed by full-width responsive tables, without HTML. + *******************************************************************************/ + public static List widgets() + { + List widgets = new ArrayList<>(); + for(int i = 0; i < KEYS.size(); i++) + { + widgets.add(new QWidgetMetaData().withName("file_gateway_home_" + KEYS.get(i)).withLabel(LABELS.get(i)) + .withType((i < 6 ? WidgetType.STATISTICS : WidgetType.TABLE).getType()).withGridColumns(i < 6 ? 2 : 12) + .withIsCard(true).withShowReloadButton(true).withCodeReference(new QCodeReference(FileGatewayHome.class)) + .withPermissionRules(FileGatewayTables.permissionRules("file_gateway"))); + } + return (widgets); + } + + + + /******************************************************************************* + ** The same permission is required for direct renderer calls and widget routes. + *******************************************************************************/ + @Override + public RenderWidgetOutput render(RenderWidgetInput input) throws QException + { + return (new RenderWidgetOutput(data(input.getWidgetMetaData().getName().replace("file_gateway_home_", ""), Instant.now()))); + } + + + + /******************************************************************************* + ** Accept an explicit time for deterministic seeded-data assertions. + *******************************************************************************/ + public QWidgetData data(String key, Instant now) throws QException + { + if(QContext.getQSession() == null || !(QContext.getQSession().hasPermission(FileGatewayPermissions.VIEW) || QContext.getQSession().hasPermission(FileGatewayPermissions.ADMIN))) + { + throw (new QPermissionDeniedException("Missing file_gateway.view")); + } + Instant today = now.atZone(ZoneOffset.UTC).toLocalDate().atStartOfDay(ZoneOffset.UTC).toInstant(); + if(List.of("transfers", "in", "out").contains(key)) + { + var rows = query(FileGatewayTables.TRANSFER_LOG, since("createDate", today)); + long value = "transfers".equals(key) ? rows.size() : rows.stream() + .filter(row -> ("in".equals(key) ? "UPLOAD" : "DOWNLOAD").equals(row.getValueString("operation"))) + .mapToLong(row -> number(row, "bytes")).sum(); + return (new StatisticsData("transfers".equals(key) ? value : Math.round(value / 10485.76) / 100.0, null, null)); + } + if("locked".equals(key)) + { + long count = query(FileGatewayTables.ACCOUNT, new QQueryFilter()).stream().filter(row -> "LOCKED".equals(row.getValueString("status")) + || (row.getValueInstant("lockedUntil") != null && row.getValueInstant("lockedUntil").isAfter(now))).count(); + return (new StatisticsData(count, null, null)); + } + if(List.of("failures", "recent").contains(key)) + { + var rows = query(FileGatewayTables.UPLOAD_JOB, since("finishedAt", now.minusSeconds("failures".equals(key) ? 86400 : 7 * 86400)) + .withCriteria(new QFilterCriteria("status", QCriteriaOperator.IN, "FAILED", "PARTIAL", "REJECTED"))); + if("failures".equals(key)) + { + return (new StatisticsData(rows.size(), null, null)); + } + rows.sort(java.util.Comparator.comparing((QRecord row) -> row.getValueInstant("finishedAt")).reversed()); + return (table("Recent failures", FileGatewayTables.UPLOAD_JOB, rows.stream().limit(20).toList(), + List.of("fileName", "status", "errorMessage", "finishedAt"), List.of("File", "Status", "Message", "Finished"))); + } + if(List.of("sessions", "active").contains(key)) + { + var rows = query(FileGatewayTables.CONNECTION_LOG, since("lastSeenAt", now.minusSeconds(180)) + .withCriteria(new QFilterCriteria("disconnectedAt", QCriteriaOperator.IS_BLANK)) + .withCriteria(new QFilterCriteria("outcome", QCriteriaOperator.EQUALS, "CONNECTED"))); + return ("sessions".equals(key) ? new StatisticsData(rows.size(), null, null) : table("Active sessions", FileGatewayTables.CONNECTION_LOG, rows, + List.of("username", "remoteIp", "connectedAt", "operationCount", "podName"), List.of("Username", "IP", "Connected", "Operations", "Pod"))); + } + if("servers".equals(key)) + { + return (table("Server", FileGatewayTables.SERVER, query(FileGatewayTables.SERVER, since("lastSeenAt", now.minusSeconds(180))), + List.of("podName", "hostKeyFingerprint", "port", "lastSeenAt"), List.of("Pod", "Host key fingerprint", "Port", "Last seen"))); + } + Map partners = new LinkedHashMap<>(); + for(QRecord account : query(FileGatewayTables.ACCOUNT, new QQueryFilter())) + { + partners.put(account.getValueInteger("id"), account.withValue("transfers", 0L).withValue("mb", 0.0)); + } + for(QRecord transfer : query(FileGatewayTables.TRANSFER_LOG, since("createDate", now.minusSeconds(7 * 86400)))) + { + QRecord partner = partners.get(transfer.getValueInteger("accountId")); + if(partner != null) + { + partner.withValue("transfers", number(partner, "transfers") + 1) + .withValue("mb", ((Number) partner.getValue("mb")).doubleValue() + number(transfer, "bytes") / 1048576.0); + } + } + List rows = partners.values().stream().filter(row -> number(row, "transfers") > 0) + .sorted(java.util.Comparator.comparingLong((QRecord row) -> number(row, "transfers")).reversed()).limit(10).toList(); + return (table("Top partners (7 days)", FileGatewayTables.ACCOUNT, rows, List.of("username", "transfers", "mb"), List.of("Partner", "Transfers", "MB"))); + } + + + + /******************************************************************************* + ** Typed text blocks carry record links; cells never contain markup. + *******************************************************************************/ + private TableData table(String label, String table, List records, List fields, List labels) + { + List columns = new ArrayList<>(); + for(int i = 0; i < fields.size(); i++) + { + columns.add(new TableData.Column(i == 0 ? "block" : "default", labels.get(i), fields.get(i), "1fr", null)); + } + List> rows = new ArrayList<>(); + for(QRecord record : records) + { + Map row = new LinkedHashMap<>(); + for(String field : fields) + { + Object value = record.getValue(field); + row.put(field, value == null ? "" : value instanceof Instant instant ? WHEN.format(instant) : value); + } + row.put(fields.getFirst(), new TextBlockData().withValues(new TextValues(record.getValueString(fields.getFirst()))) + .withLink("/record/" + table + "/" + record.getValue("id"))); + rows.add(row); + } + return (new TableData(null, columns, rows).withRowsPerPage(10).withHidePaginationDropdown(true)); + } + + + + /******************************************************************************* + ** Queries preserve viewer security and use bounded database pages. + *******************************************************************************/ + private List query(String table, QQueryFilter filter) throws QException + { + List rows = new ArrayList<>(); + for(int offset = 0; ; offset += 250) + { + var page = new QueryAction().execute(new QueryInput(table).withFilter(filter.withSkip(offset).withLimit(250).withOrderBy(new QFilterOrderBy("id", true)))).getRecords(); + rows.addAll(page); + if(page.size() < 250) + { + return (rows); + } + } + } + + + + /******************************************************************************* + ** UTC time windows agree with quota and transfer bookkeeping. + *******************************************************************************/ + private QQueryFilter since(String field, Instant from) + { + return (new QQueryFilter(new QFilterCriteria(field, QCriteriaOperator.GREATER_THAN_OR_EQUALS, from))); + } + + + + /******************************************************************************* + ** Older audit rows may have no counters. + *******************************************************************************/ + private long number(QRecord row, String field) + { + return (row.getValueLong(field) == null ? 0 : row.getValueLong(field)); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/metadata/FileGatewayProcesses.java b/src/main/java/com/kingsrook/qbits/filegateway/metadata/FileGatewayProcesses.java new file mode 100644 index 0000000..7559cff --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/metadata/FileGatewayProcesses.java @@ -0,0 +1,241 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.metadata; + +import java.io.Serializable; +import java.time.Instant; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.filegateway.permissions.FileGatewayPermissions; +import com.kingsrook.qbits.filegateway.sftp.AuditWriter; +import com.kingsrook.qbits.filegateway.upload.UploadJobs; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine; +import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput; +import com.kingsrook.qqq.backend.core.model.actions.processes.Status; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractLoadStep; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractTransformStep; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ExtractViaQueryStep; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ProcessSummaryProviderInterface; +import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess; + +/******************************************************************************* + ** Administrative record actions use the standard streamed Review and Result flow. + *******************************************************************************/ +public final class FileGatewayProcesses +{ + public static final String RETRY_UPLOAD = "file_gateway_retry_upload"; + public static final String UNLOCK = "file_gateway_unlock_account"; + public static final String DISABLE = "file_gateway_disable_account"; + public static final String REVOKE = "file_gateway_revoke_key"; + public static final String END_SESSION = "file_gateway_end_session"; + + /******************************************************************************* + ** Metadata factory only. + *******************************************************************************/ + private FileGatewayProcesses() + { + } + + + + /******************************************************************************* + ** Administrative actions bound to their owning tables. + *******************************************************************************/ + public static List processes() + { + return (List.of(action(UNLOCK, "Unlock account", FileGatewayTables.ACCOUNT), + action(DISABLE, "Disable account", FileGatewayTables.ACCOUNT), + action(REVOKE, "Revoke key", FileGatewayTables.SSH_KEY), + action(END_SESSION, "End session", FileGatewayTables.CONNECTION_LOG), + action(RETRY_UPLOAD, "Retry upload", FileGatewayTables.UPLOAD_JOB))); + } + + + + /******************************************************************************* + ** A table-bound action, never a navigation entry. + *******************************************************************************/ + private static QProcessMetaData action(String name, String label, String table) + { + Map values = new HashMap<>(); + values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, table); + values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, table); + values.put(StreamedETLWithFrontendProcess.FIELD_SUPPORTS_FULL_VALIDATION, true); + values.put(StreamedETLWithFrontendProcess.FIELD_DO_FULL_VALIDATION, true); + values.put(StreamedETLWithFrontendProcess.FIELD_PREVIEW_MESSAGE, label + " for the selected records."); + return (StreamedETLWithFrontendProcess.defineProcessMetaData(ExtractViaQueryStep.class, Review.class, Apply.class, values) + .withName(name).withLabel(label).withTableName(table).withPermissionRules(FileGatewayTables.permissionRules(name))); + } + + + + /******************************************************************************* + ** Direct step invocation must require the same permission as the process route. + *******************************************************************************/ + private static void requireAdmin() throws QPermissionDeniedException + { + if(QContext.getQSession() == null || !QContext.getQSession().hasPermission(FileGatewayPermissions.ADMIN)) + { + throw (new QPermissionDeniedException("Missing file_gateway.admin")); + } + } + + + + /******************************************************************************* + ** Preview selections without changing records. + *******************************************************************************/ + public static class Review extends AbstractTransformStep implements ProcessSummaryProviderInterface + { + private final ProcessSummaryLine summary = new ProcessSummaryLine(Status.OK, 0, null); + private final ProcessSummaryLine skipped = new ProcessSummaryLine(Status.INFO, 0, "uploads are not eligible for retry"); + + /******************************************************************************* + ** Review uses the framework's future and past tense summary lines. + *******************************************************************************/ + public Review() + { + summary.setSingularFutureMessage("record will be updated"); + summary.setPluralFutureMessage("records will be updated"); + summary.setSingularPastMessage("record was updated"); + summary.setPluralPastMessage("records were updated"); + } + + + + /******************************************************************************* + ** Pass only authorized selections to the load phase. + *******************************************************************************/ + @Override + public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + requireAdmin(); + for(QRecord record : input.getRecords()) + { + if(RETRY_UPLOAD.equals(input.getProcessName()) && (!List.of("FAILED", "REJECTED").contains(record.getValueString("status")) + || record.getValueString("storageKey") == null || record.getValueString("sha256") == null)) + { + skipped.incrementCountAndAddPrimaryKey(record.getValue("id")); + continue; + } + summary.incrementCountAndAddPrimaryKey(record.getValue("id")); + output.getRecords().add(record); + } + } + + + + /******************************************************************************* + ** Standard Review and Result summary. + *******************************************************************************/ + @Override + public ArrayList getProcessSummary(RunBackendStepOutput output, boolean result) + { + ArrayList lines = new ArrayList<>(List.of(summary)); + skipped.addSelfToListIfAnyCount(lines); + return (lines); + } + } + + + + /******************************************************************************* + ** Apply the approved action to the gateway's own bookkeeping tables. + *******************************************************************************/ + public static class Apply extends AbstractLoadStep + { + /******************************************************************************* + ** Whitelist mutations, using sparse updates so concurrent account changes survive. + *******************************************************************************/ + @Override + public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException + { + requireAdmin(); + AuditWriter audit = new AuditWriter(QContext.getQInstance()); + for(QRecord record : input.getRecords()) + { + if(RETRY_UPLOAD.equals(input.getProcessName())) + { + audit.withAccountLock("id", record.getValue("accountId"), () -> + { + QRecord current = audit.query(FileGatewayTables.UPLOAD_JOB, UploadJobs.equal("id", record.getValue("id"))).getFirst(); + if(!List.of("FAILED", "REJECTED").contains(current.getValueString("status")) + || current.getValueString("storageKey") == null || current.getValueString("sha256") == null) + { + throw (new QException("Only failed or rejected stored uploads can be retried")); + } + QRecord retry = new QRecord().withValue("id", record.getValue("id")) + .withValue("status", "QUEUED").withValue("attempts", 0).withValue("nextAttemptAt", null) + .withValue("claimedBy", null).withValue("claimedAt", null).withValue("finishedAt", null) + .withValue("acknowledgedAt", null).withValue("errorMessage", null).withValue("resultSummary", null) + .withValue("resultsStorageKey", null).withValue("resultsSizeBytes", null).withValue("processUuid", null) + .withValue("insertedCount", null).withValue("errorCount", null).withValue("queuedAt", Instant.now()); + audit.update(FileGatewayTables.UPLOAD_JOB, retry); + output.getRecords().add(retry); + return (null); + }); + continue; + } + QRecord update = new QRecord().withValue("id", record.getValue("id")); + String table; + switch(input.getProcessName()) + { + case UNLOCK -> + { + table = FileGatewayTables.ACCOUNT; + update.withValue("lockedUntil", null).withValue("failedLoginCount", 0).withValue("failedLoginWindowAt", null); + if("LOCKED".equals(record.getValueString("status"))) + { + update.withValue("status", "ACTIVE"); + } + } + case DISABLE -> + { + table = FileGatewayTables.ACCOUNT; + update.withValue("status", "DISABLED"); + } + case REVOKE -> + { + table = FileGatewayTables.SSH_KEY; + update.withValue("revokedAt", Instant.now()); + } + case END_SESSION -> + { + table = FileGatewayTables.CONNECTION_LOG; + update.withValue("endRequestedAt", Instant.now()).withValue("disconnectReason", "Ended by administrator"); + } + default -> throw (new QException("Unsupported gateway action")); + } + audit.update(table, update); + output.getRecords().add(update); + } + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/metadata/FileGatewayTables.java b/src/main/java/com/kingsrook/qbits/filegateway/metadata/FileGatewayTables.java new file mode 100644 index 0000000..063ba60 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/metadata/FileGatewayTables.java @@ -0,0 +1,492 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.metadata; + + +import java.util.ArrayList; +import java.util.List; +import com.kingsrook.qbits.filegateway.FileGatewayQBitConfig; +import com.kingsrook.qbits.filegateway.permissions.FileGatewayPermissionChecker; +import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData; +import com.kingsrook.qqq.backend.core.actions.ai.DataClassification; +import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers; +import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.ChildRecordListRenderer; +import com.kingsrook.qqq.backend.core.instances.QInstanceEnricher; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn; +import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType; +import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules; +import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource; +import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.tables.SectionFactory; +import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey; + + +/******************************************************************************* + ** The seven gateway tables, their stable columns, pickers and child lists. + *******************************************************************************/ +public final class FileGatewayTables +{ + public static final String ACCOUNT = "file_gateway_account"; + public static final String SSH_KEY = "file_gateway_ssh_key"; + public static final String ALIAS = "file_gateway_alias"; + public static final String UPLOAD_JOB = "file_gateway_upload_job"; + public static final String OUTBOX_FILE = "file_gateway_outbox_file"; + public static final String TRANSFER_LOG = "file_gateway_transfer_log"; + public static final String CONNECTION_LOG = "file_gateway_connection_log"; + public static final String SERVER = "file_gateway_server"; + public static final List ALL = List.of(ACCOUNT, SSH_KEY, ALIAS, UPLOAD_JOB, OUTBOX_FILE, TRANSFER_LOG, CONNECTION_LOG, SERVER); + public static final List MAIN = List.of(ACCOUNT, ALIAS, UPLOAD_JOB, TRANSFER_LOG, CONNECTION_LOG); + public static final List SYSTEM_WRITTEN = List.of(UPLOAD_JOB, OUTBOX_FILE, TRANSFER_LOG, CONNECTION_LOG, SERVER); + + + + /******************************************************************************* + ** Metadata factories only. + *******************************************************************************/ + private FileGatewayTables() + { + } + + + + /******************************************************************************* + ** Permission names use the full table or app name. + *******************************************************************************/ + public static QPermissionRules permissionRules(String name) + { + return (new QPermissionRules().withLevel(PermissionLevel.READ_WRITE_PERMISSIONS).withPermissionBaseName(name) + .withCustomPermissionChecker(new QCodeReference(FileGatewayPermissionChecker.class))); + } + + + + /******************************************************************************* + ** Enums and gateway record pickers. + *******************************************************************************/ + public static List possibleValueSources() + { + List sources = new ArrayList<>(); + sources.add(QPossibleValueSource.newForEnum("file_gateway_account_type", FileGatewayEnums.AccountType.values())); + sources.add(QPossibleValueSource.newForEnum("file_gateway_account_status", FileGatewayEnums.AccountStatus.values())); + sources.add(QPossibleValueSource.newForEnum("file_gateway_root_mode", FileGatewayEnums.RootMode.values())); + sources.add(QPossibleValueSource.newForEnum("file_gateway_alias_kind", FileGatewayEnums.AliasKind.values())); + sources.add(QPossibleValueSource.newForEnum("file_gateway_upload_status", FileGatewayEnums.UploadStatus.values())); + sources.add(QPossibleValueSource.newForEnum("file_gateway_outbox_status", FileGatewayEnums.OutboxStatus.values())); + sources.add(QPossibleValueSource.newForEnum("file_gateway_operation", FileGatewayEnums.Operation.values())); + sources.add(QPossibleValueSource.newForEnum("file_gateway_object_kind", FileGatewayEnums.ObjectKind.values())); + sources.add(QPossibleValueSource.newForEnum("file_gateway_transfer_outcome", FileGatewayEnums.TransferOutcome.values())); + sources.add(QPossibleValueSource.newForEnum("file_gateway_connection_outcome", FileGatewayEnums.ConnectionOutcome.values())); + sources.add(QPossibleValueSource.newForEnum("file_gateway_format", FileGatewayEnums.Format.values())); + sources.add(QPossibleValueSource.newForEnum("file_gateway_key_type", FileGatewayEnums.KeyType.values())); + ALL.forEach(name -> sources.add(QPossibleValueSource.newForTable(name))); + return (sources); + } + + + + /** Plain labels for child lists and their record-screen tabs. */ + static final java.util.Map CHILD_LABELS = java.util.Map.of( + SSH_KEY, "SSH keys", UPLOAD_JOB, "Upload jobs", CONNECTION_LOG, "Connections", + TRANSFER_LOG, "Transfers", OUTBOX_FILE, "Outbox files"); + + /** Plain field labels where the name-derived label reads badly. */ + static final java.util.Map FIELD_LABELS = java.util.Map.ofEntries( + java.util.Map.entry("allowedCidrs", "Allowed CIDRs"), java.util.Map.entry("userId", "User"), + java.util.Map.entry("ownerUserId", "Owner"), java.util.Map.entry("maxFileSizeMb", "Max file size (MB)"), + java.util.Map.entry("maxUploadMbPerDay", "Max upload per day (MB)"), java.util.Map.entry("maxDownloadMbPerDay", "Max download per day (MB)"), + java.util.Map.entry("lastLoginIp", "Last login IP"), java.util.Map.entry("lastUsedIp", "Last used IP"), + java.util.Map.entry("pgpPublicKey", "PGP public key"), java.util.Map.entry("pgpDecryptUploads", "Decrypt PGP uploads"), + java.util.Map.entry("pgpEncryptDownloads", "Encrypt downloads with PGP"), java.util.Map.entry("remoteIp", "Remote IP"), + java.util.Map.entry("remoteAddress", "Remote IP"), java.util.Map.entry("sha256", "SHA-256"), + java.util.Map.entry("processUuid", "Process UUID"), java.util.Map.entry("savedBulkLoadProfileId", "Bulk load profile"), + java.util.Map.entry("savedViewId", "Saved view"), java.util.Map.entry("savedReportId", "Saved report"), + java.util.Map.entry("sizeBytes", "Size (bytes)"), java.util.Map.entry("accountId", "Account"), + java.util.Map.entry("aliasId", "Endpoint"), java.util.Map.entry("connectionLogId", "Connection"), + java.util.Map.entry("hostKeyFingerprint", "Host key fingerprint"), java.util.Map.entry("keyFingerprint", "Key fingerprint")); + + + + /******************************************************************************* + ** Build all seven tables. + *******************************************************************************/ + public static List tables(FileGatewayQBitConfig config) + { + List tables = List.of(account(config), ssh_key(config), alias(config), upload_job(config), outbox_file(config), transfer_log(config), connection_log(config), server(config)); + for(QTableMetaData table : tables) + { + for(QFieldMetaData field : table.getFields().values()) + { + if(FIELD_LABELS.containsKey(field.getName())) + { + field.setLabel(FIELD_LABELS.get(field.getName())); + } + } + } + return (tables); + } + + + + /******************************************************************************* + ** Common identity and timestamps. + *******************************************************************************/ + private static QTableMetaData base(FileGatewayQBitConfig config, String name, String label) + { + return (new QTableMetaData().withName(name).withLabel(label).withBackendName(config.getBackendName()) + .withPrimaryKeyField("id") + .withPermissionRules(permissionRules(name)) + .withField(field("id", QFieldType.INTEGER).withIsEditable(false)) + .withField(field("createDate", QFieldType.DATE_TIME).withIsEditable(false)) + .withField(field("modifyDate", QFieldType.DATE_TIME).withIsEditable(false))); + } + + + + /******************************************************************************* + ** Apply the host backend naming convention explicitly. + *******************************************************************************/ + private static QFieldMetaData field(String name, QFieldType type) + { + return (new QFieldMetaData(name, type).withBackendName(QInstanceEnricher.inferBackendName(name))); + } + + + + /******************************************************************************* + ** Metadata for accounts. + *******************************************************************************/ + private static QTableMetaData account(FileGatewayQBitConfig config) + { + QTableMetaData table = base(config, ACCOUNT, "Accounts") + .withRecordLabelFormatAndFields("%s", "username") + .withField(field("username", QFieldType.STRING).withIsRequired(true)) + .withField(field("label", QFieldType.STRING)) + .withField(field("accountType", QFieldType.STRING).withPossibleValueSourceName("file_gateway_account_type").withDefaultValue("USER").withIsRequired(true)) + .withField(field("userId", QFieldType.STRING).withIsRequired(true)) + .withField(field("ownerUserId", QFieldType.STRING)) + .withField(field("status", QFieldType.STRING).withPossibleValueSourceName("file_gateway_account_status").withDefaultValue("ACTIVE").withIsRequired(true)) + .withField(field("rootMode", QFieldType.STRING).withPossibleValueSourceName("file_gateway_root_mode")) + .withField(field("allowedCidrs", QFieldType.TEXT)) + .withField(field("expiresAt", QFieldType.DATE_TIME)) + .withField(field("formats", QFieldType.STRING).withDefaultValue("csv,xlsx")) + .withField(field("maxConcurrentSessions", QFieldType.INTEGER).withDefaultValue(config.getDefaultMaxConcurrentSessions())) + .withField(field("maxOperationsPerMinute", QFieldType.INTEGER).withDefaultValue(config.getDefaultMaxOperationsPerMinute())) + .withField(field("maxFileSizeMb", QFieldType.INTEGER).withDefaultValue(config.getDefaultMaxFileSizeMb())) + .withField(field("maxUploadMbPerDay", QFieldType.INTEGER).withDefaultValue(config.getDefaultMaxUploadMbPerDay())) + .withField(field("maxDownloadMbPerDay", QFieldType.INTEGER).withDefaultValue(config.getDefaultMaxDownloadMbPerDay())) + .withField(field("dedupeUploads", QFieldType.BOOLEAN).withDefaultValue(true)) + .withField(field("notifyOnSuccess", QFieldType.BOOLEAN).withDefaultValue(false)) + .withField(field("pgpPublicKey", QFieldType.TEXT)) + .withField(field("pgpDecryptUploads", QFieldType.BOOLEAN).withDefaultValue(false)) + .withField(field("pgpEncryptDownloads", QFieldType.BOOLEAN).withDefaultValue(false)) + .withField(field("operationWindowAt", QFieldType.DATE_TIME).withIsEditable(false)) + .withField(field("operationWindowCount", QFieldType.INTEGER).withDefaultValue(0).withIsEditable(false)) + .withField(field("failedLoginCount", QFieldType.INTEGER).withDefaultValue(0).withIsEditable(false)) + .withField(field("failedLoginWindowAt", QFieldType.DATE_TIME).withIsEditable(false)) + .withField(field("lockedUntil", QFieldType.DATE_TIME).withIsEditable(false)) + .withField(field("lastLoginAt", QFieldType.DATE_TIME).withIsEditable(false)) + .withField(field("lastLoginIp", QFieldType.STRING).withIsEditable(false)); + table.withUniqueKey(new UniqueKey("username")); + table.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(AccountCustomizer.class)); + table.withCustomizer(TableCustomizers.PRE_UPDATE_RECORD, new QCodeReference(AccountCustomizer.class)); + table.withSection(SectionFactory.defaultT1("id", "username", "label", "accountType", "userId", "ownerUserId")); + table.withSection(SectionFactory.defaultT2("status", "rootMode", "allowedCidrs", "expiresAt", "formats", "maxConcurrentSessions", "maxOperationsPerMinute", "maxFileSizeMb", "maxUploadMbPerDay", "maxDownloadMbPerDay", "dedupeUploads", "notifyOnSuccess", "pgpPublicKey", "pgpDecryptUploads", "pgpEncryptDownloads", "operationWindowAt", "operationWindowCount", "failedLoginCount", "failedLoginWindowAt", "lockedUntil", "lastLoginAt", "lastLoginIp")); + table.withSection(SectionFactory.defaultT3("createDate", "modifyDate")); + table.withSection(SectionFactory.customT2(SSH_KEY, new QIcon("list")).withLabel(CHILD_LABELS.get(SSH_KEY)).withWidgetName(ACCOUNT + "_children_" + SSH_KEY)); + table.withSection(SectionFactory.customT2(UPLOAD_JOB, new QIcon("list")).withLabel(CHILD_LABELS.get(UPLOAD_JOB)).withWidgetName(ACCOUNT + "_children_" + UPLOAD_JOB)); + table.withSection(SectionFactory.customT2(CONNECTION_LOG, new QIcon("list")).withLabel(CHILD_LABELS.get(CONNECTION_LOG)).withWidgetName(ACCOUNT + "_children_" + CONNECTION_LOG)); + return (table); + } + + + + /******************************************************************************* + ** Metadata for ssh keys. + *******************************************************************************/ + private static QTableMetaData ssh_key(FileGatewayQBitConfig config) + { + QTableMetaData table = base(config, SSH_KEY, "SSH Keys") + .withRecordLabelFormatAndFields("%s", "label") + .withField(field("accountId", QFieldType.INTEGER).withPossibleValueSourceName(ACCOUNT).withIsRequired(true)) + .withField(field("label", QFieldType.STRING)) + .withField(field("publicKey", QFieldType.TEXT).withIsRequired(true)) + .withField(field("fingerprint", QFieldType.STRING).withIsEditable(false)) + .withField(field("keyType", QFieldType.STRING).withPossibleValueSourceName("file_gateway_key_type").withIsEditable(false)) + .withField(field("keyBits", QFieldType.INTEGER).withIsEditable(false)) + .withField(field("expiresAt", QFieldType.DATE_TIME)) + .withField(field("revokedAt", QFieldType.DATE_TIME)) + .withField(field("lastUsedAt", QFieldType.DATE_TIME).withIsEditable(false)) + .withField(field("lastUsedIp", QFieldType.STRING).withIsEditable(false)); + table.withUniqueKey(new UniqueKey("fingerprint")); + table.withSection(SectionFactory.defaultT1("id", "accountId", "label", "publicKey", "fingerprint", "keyType")); + table.withSection(SectionFactory.defaultT2("keyBits", "expiresAt", "revokedAt", "lastUsedAt", "lastUsedIp")); + table.withSection(SectionFactory.defaultT3("createDate", "modifyDate")); + table.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(SshKeyCustomizer.class)); + table.withCustomizer(TableCustomizers.PRE_UPDATE_RECORD, new QCodeReference(SshKeyCustomizer.class)); + table.getField("publicKey").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.CONFIDENTIAL)); + return (table); + } + + + + /******************************************************************************* + ** Metadata for endpoints. + *******************************************************************************/ + private static QTableMetaData alias(FileGatewayQBitConfig config) + { + QTableMetaData table = base(config, ALIAS, "Endpoints") + .withRecordLabelFormatAndFields("%s", "name") + .withField(field("name", QFieldType.STRING)) + .withField(field("path", QFieldType.STRING)) + .withField(field("kind", QFieldType.STRING).withPossibleValueSourceName("file_gateway_alias_kind")) + .withField(field("accountId", QFieldType.INTEGER).withPossibleValueSourceName(ACCOUNT)) + .withField(field("tableName", QFieldType.STRING).withPossibleValueSourceName("tables")) + .withField(field("savedBulkLoadProfileId", QFieldType.INTEGER).withPossibleValueSourceName("savedBulkLoadProfile").withPossibleValueSourceFilter(new QQueryFilter(new QFilterCriteria("tableName", QCriteriaOperator.EQUALS, "${input.tableName}")))) + .withField(field("savedViewId", QFieldType.INTEGER).withPossibleValueSourceName("savedView").withPossibleValueSourceFilter(new QQueryFilter(new QFilterCriteria("tableName", QCriteriaOperator.EQUALS, "${input.tableName}")))) + .withField(field("savedReportId", QFieldType.INTEGER).withPossibleValueSourceName("savedReport").withPossibleValueSourceFilter(new QQueryFilter(new QFilterCriteria("tableName", QCriteriaOperator.EQUALS, "${input.tableName}")))) + .withField(field("format", QFieldType.STRING).withPossibleValueSourceName("file_gateway_format")) + .withField(field("fileName", QFieldType.STRING)) + .withField(field("consumerName", QFieldType.STRING)) + .withField(field("isActive", QFieldType.BOOLEAN).withDefaultValue(true)) + .withField(field("description", QFieldType.TEXT)); + table.withUniqueKey(new UniqueKey("name")); + table.withUniqueKey(new UniqueKey("path")); + table.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(AliasCustomizer.class)); + table.withCustomizer(TableCustomizers.PRE_UPDATE_RECORD, new QCodeReference(AliasCustomizer.class)); + table.withSection(SectionFactory.defaultT1("id", "name", "path", "kind", "accountId", "tableName")); + table.withSection(SectionFactory.defaultT2("savedBulkLoadProfileId", "savedViewId", "savedReportId", "format", "fileName", "consumerName", "isActive", "description")); + table.withSection(SectionFactory.defaultT3("createDate", "modifyDate")); + table.withSection(SectionFactory.customT2(UPLOAD_JOB, new QIcon("list")).withLabel(CHILD_LABELS.get(UPLOAD_JOB)).withWidgetName(ALIAS + "_children_" + UPLOAD_JOB)); + table.withSection(SectionFactory.customT2(OUTBOX_FILE, new QIcon("list")).withLabel(CHILD_LABELS.get(OUTBOX_FILE)).withWidgetName(ALIAS + "_children_" + OUTBOX_FILE)); + return (table); + } + + + + /******************************************************************************* + ** Metadata for upload jobs. + *******************************************************************************/ + private static QTableMetaData upload_job(FileGatewayQBitConfig config) + { + QTableMetaData table = base(config, UPLOAD_JOB, "Upload Jobs").withRecordVersionField("status") + .withRecordLabelFormatAndFields("%s", "id") + .withField(field("accountId", QFieldType.INTEGER).withPossibleValueSourceName(ACCOUNT)) + .withField(field("aliasId", QFieldType.INTEGER).withPossibleValueSourceName(ALIAS)) + .withField(field("connectionLogId", QFieldType.INTEGER).withPossibleValueSourceName(CONNECTION_LOG)) + .withField(field("remoteAddress", QFieldType.STRING)) + .withField(field("tableName", QFieldType.STRING).withPossibleValueSourceName("tables")) + .withField(field("savedBulkLoadProfileId", QFieldType.INTEGER).withPossibleValueSourceName("savedBulkLoadProfile")) + .withField(field("consumerName", QFieldType.STRING)) + .withField(field("consumerReference", QFieldType.STRING)) + .withField(field("path", QFieldType.STRING)) + .withField(field("fileName", QFieldType.STRING)) + .withField(field("sizeBytes", QFieldType.LONG)) + .withField(field("sha256", QFieldType.STRING)) + .withField(field("storageKey", QFieldType.STRING)) + .withField(field("status", QFieldType.STRING).withPossibleValueSourceName("file_gateway_upload_status")) + .withField(field("duplicateOfJobId", QFieldType.INTEGER).withPossibleValueSourceName(UPLOAD_JOB)) + .withField(field("processUuid", QFieldType.STRING)) + .withField(field("insertedCount", QFieldType.INTEGER)) + .withField(field("errorCount", QFieldType.INTEGER)) + .withField(field("resultSummary", QFieldType.TEXT)) + .withField(field("resultsStorageKey", QFieldType.STRING)) + .withField(field("resultsSizeBytes", QFieldType.LONG)) + .withField(field("plaintextStorageKey", QFieldType.STRING)) + .withField(field("plaintextSha256", QFieldType.STRING)) + .withField(field("plaintextSizeBytes", QFieldType.LONG)) + .withField(field("errorMessage", QFieldType.TEXT)) + .withField(field("attempts", QFieldType.INTEGER)) + .withField(field("nextAttemptAt", QFieldType.DATE_TIME)) + .withField(field("claimedBy", QFieldType.STRING)) + .withField(field("claimedAt", QFieldType.DATE_TIME)) + .withField(field("queuedAt", QFieldType.DATE_TIME)) + .withField(field("startedAt", QFieldType.DATE_TIME)) + .withField(field("finishedAt", QFieldType.DATE_TIME)) + .withField(field("acknowledgedAt", QFieldType.DATE_TIME)); + table.withSection(SectionFactory.defaultT1("id", "accountId", "aliasId", "connectionLogId", "remoteAddress", "tableName")); + table.withSection(SectionFactory.defaultT2("savedBulkLoadProfileId", "consumerName", "consumerReference", "path", "fileName", "sizeBytes", "sha256", "storageKey", "status", "duplicateOfJobId", "processUuid", "insertedCount", "errorCount", "resultSummary", "resultsStorageKey", "resultsSizeBytes", "plaintextStorageKey", "plaintextSha256", "plaintextSizeBytes", "errorMessage", "attempts", "nextAttemptAt", "claimedBy", "claimedAt", "queuedAt", "startedAt", "finishedAt", "acknowledgedAt")); + table.withSection(SectionFactory.defaultT3("createDate", "modifyDate")); + table.withSection(SectionFactory.customT2(TRANSFER_LOG, new QIcon("list")).withLabel(CHILD_LABELS.get(TRANSFER_LOG)).withWidgetName(UPLOAD_JOB + "_children_" + TRANSFER_LOG)); + table.withoutCapabilities(Capability.TABLE_INSERT, Capability.TABLE_UPDATE, Capability.TABLE_DELETE); + return (table); + } + + + + /******************************************************************************* + ** Metadata for outbox files. + *******************************************************************************/ + private static QTableMetaData outbox_file(FileGatewayQBitConfig config) + { + QTableMetaData table = base(config, OUTBOX_FILE, "Outbox Files") + .withRecordLabelFormatAndFields("%s", "id") + .withField(field("aliasId", QFieldType.INTEGER).withPossibleValueSourceName(ALIAS)) + .withField(field("fileName", QFieldType.STRING)) + .withField(field("sizeBytes", QFieldType.LONG)) + .withField(field("sha256", QFieldType.STRING)) + .withField(field("storageKey", QFieldType.STRING)) + .withField(field("status", QFieldType.STRING).withPossibleValueSourceName("file_gateway_outbox_status")) + .withField(field("firstDownloadedAt", QFieldType.DATE_TIME)) + .withField(field("downloadCount", QFieldType.INTEGER)) + .withField(field("onDownloadedPending", QFieldType.BOOLEAN).withDefaultValue(false)) + .withField(field("removedAt", QFieldType.DATE_TIME)) + .withField(field("expiresAt", QFieldType.DATE_TIME)) + .withField(field("createdByConsumer", QFieldType.STRING)); + table.withSection(SectionFactory.defaultT1("id", "aliasId", "fileName", "sizeBytes", "sha256", "storageKey")); + table.withSection(SectionFactory.defaultT2("status", "firstDownloadedAt", "downloadCount", "onDownloadedPending", "removedAt", "expiresAt", "createdByConsumer")); + table.withSection(SectionFactory.defaultT3("createDate", "modifyDate")); + table.withoutCapabilities(Capability.TABLE_INSERT, Capability.TABLE_UPDATE, Capability.TABLE_DELETE); + return (table); + } + + + + /******************************************************************************* + ** Metadata for transfers. + *******************************************************************************/ + private static QTableMetaData transfer_log(FileGatewayQBitConfig config) + { + QTableMetaData table = base(config, TRANSFER_LOG, "Transfers") + .withRecordLabelFormatAndFields("%s", "id") + .withField(field("accountId", QFieldType.INTEGER).withPossibleValueSourceName(ACCOUNT)) + .withField(field("connectionLogId", QFieldType.INTEGER).withPossibleValueSourceName(CONNECTION_LOG)) + .withField(field("operation", QFieldType.STRING).withPossibleValueSourceName("file_gateway_operation")) + .withField(field("path", QFieldType.STRING)) + .withField(field("objectKind", QFieldType.STRING).withPossibleValueSourceName("file_gateway_object_kind")) + .withField(field("objectId", QFieldType.STRING)) + .withField(field("format", QFieldType.STRING).withPossibleValueSourceName("file_gateway_format")) + .withField(field("bytes", QFieldType.LONG)) + .withField(field("sha256", QFieldType.STRING)) + .withField(field("durationMs", QFieldType.LONG)) + .withField(field("outcome", QFieldType.STRING).withPossibleValueSourceName("file_gateway_transfer_outcome")) + .withField(field("message", QFieldType.TEXT)) + .withField(field("clientChannel", QFieldType.STRING)) + .withField(field("uploadJobId", QFieldType.INTEGER).withPossibleValueSourceName(UPLOAD_JOB)); + table.withSection(SectionFactory.defaultT1("id", "accountId", "connectionLogId", "operation", "path", "objectKind")); + table.withSection(SectionFactory.defaultT2("objectId", "format", "bytes", "sha256", "durationMs", "outcome", "message", "clientChannel", "uploadJobId")); + table.withSection(SectionFactory.defaultT3("createDate", "modifyDate")); + table.withoutCapabilities(Capability.TABLE_INSERT, Capability.TABLE_UPDATE, Capability.TABLE_DELETE); + return (table); + } + + + + /******************************************************************************* + ** Metadata for connections. + *******************************************************************************/ + private static QTableMetaData connection_log(FileGatewayQBitConfig config) + { + QTableMetaData table = base(config, CONNECTION_LOG, "Connections") + .withRecordLabelFormatAndFields("%s", "id") + .withField(field("accountId", QFieldType.INTEGER).withPossibleValueSourceName(ACCOUNT)) + .withField(field("username", QFieldType.STRING)) + .withField(field("remoteIp", QFieldType.STRING)) + .withField(field("clientVersion", QFieldType.STRING)) + .withField(field("keyFingerprint", QFieldType.STRING)) + .withField(field("outcome", QFieldType.STRING).withPossibleValueSourceName("file_gateway_connection_outcome")) + .withField(field("sessionId", QFieldType.STRING)) + .withField(field("podName", QFieldType.STRING)) + .withField(field("connectedAt", QFieldType.DATE_TIME)) + .withField(field("lastSeenAt", QFieldType.DATE_TIME)) + .withField(field("disconnectedAt", QFieldType.DATE_TIME)) + .withField(field("endRequestedAt", QFieldType.DATE_TIME)) + .withField(field("disconnectReason", QFieldType.TEXT)) + .withField(field("bytesIn", QFieldType.LONG)) + .withField(field("bytesOut", QFieldType.LONG)) + .withField(field("operationCount", QFieldType.LONG)) + .withField(field("clientChannel", QFieldType.STRING)); + table.withSection(SectionFactory.defaultT1("id", "accountId", "username", "remoteIp", "clientVersion", "keyFingerprint")); + table.withSection(SectionFactory.defaultT2("outcome", "sessionId", "podName", "connectedAt", "lastSeenAt", "disconnectedAt", "endRequestedAt", "disconnectReason", "bytesIn", "bytesOut", "operationCount", "clientChannel")); + table.withSection(SectionFactory.defaultT3("createDate", "modifyDate")); + table.withoutCapabilities(Capability.TABLE_INSERT, Capability.TABLE_UPDATE, Capability.TABLE_DELETE); + return (table); + } + + + + /******************************************************************************* + ** Gateway heartbeat shared with the main web deployment, never a navigation table. + *******************************************************************************/ + private static QTableMetaData server(FileGatewayQBitConfig config) + { + return (base(config, SERVER, "Servers").withRecordLabelFormatAndFields("%s", "podName") + .withField(field("podName", QFieldType.STRING).withIsRequired(true)) + .withField(field("hostKeyFingerprint", QFieldType.STRING)) + .withField(field("port", QFieldType.INTEGER)) + .withField(field("startedAt", QFieldType.DATE_TIME)) + .withField(field("lastSeenAt", QFieldType.DATE_TIME)) + .withUniqueKey(new UniqueKey("podName")) + .withSection(SectionFactory.defaultT1("id", "podName", "hostKeyFingerprint", "port", "startedAt", "lastSeenAt")) + .withSection(SectionFactory.defaultT3("createDate", "modifyDate")) + .withoutCapabilities(Capability.TABLE_INSERT, Capability.TABLE_UPDATE, Capability.TABLE_DELETE)); + } + + + + /******************************************************************************* + ** Parent and child relationships. + *******************************************************************************/ + public static List joins() + { + return (List.of( + new QJoinMetaData().withName(ACCOUNT + "_join_" + SSH_KEY).withType(JoinType.ONE_TO_MANY) + .withLeftTable(ACCOUNT).withRightTable(SSH_KEY).withJoinOn(new JoinOn("id", "accountId")), + new QJoinMetaData().withName(ACCOUNT + "_join_" + UPLOAD_JOB).withType(JoinType.ONE_TO_MANY) + .withLeftTable(ACCOUNT).withRightTable(UPLOAD_JOB).withJoinOn(new JoinOn("id", "accountId")), + new QJoinMetaData().withName(ACCOUNT + "_join_" + CONNECTION_LOG).withType(JoinType.ONE_TO_MANY) + .withLeftTable(ACCOUNT).withRightTable(CONNECTION_LOG).withJoinOn(new JoinOn("id", "accountId")), + new QJoinMetaData().withName(ALIAS + "_join_" + UPLOAD_JOB).withType(JoinType.ONE_TO_MANY) + .withLeftTable(ALIAS).withRightTable(UPLOAD_JOB).withJoinOn(new JoinOn("id", "aliasId")), + new QJoinMetaData().withName(ALIAS + "_join_" + OUTBOX_FILE).withType(JoinType.ONE_TO_MANY) + .withLeftTable(ALIAS).withRightTable(OUTBOX_FILE).withJoinOn(new JoinOn("id", "aliasId")), + new QJoinMetaData().withName(UPLOAD_JOB + "_join_" + TRANSFER_LOG).withType(JoinType.ONE_TO_MANY) + .withLeftTable(UPLOAD_JOB).withRightTable(TRANSFER_LOG).withJoinOn(new JoinOn("id", "uploadJobId")))); + } + + + + /******************************************************************************* + ** Typed child-record widgets; no home or HTML widget. + *******************************************************************************/ + public static List childWidgets() + { + List widgets = new ArrayList<>(); + for(QJoinMetaData join : joins()) + { + widgets.add(ChildRecordListRenderer.widgetMetaDataBuilder(join) + .withName(join.getLeftTable() + "_children_" + join.getRightTable()).withLabel(CHILD_LABELS.getOrDefault(join.getRightTable(), join.getRightTable())) + .withCanAddChildRecord(SSH_KEY.equals(join.getRightTable())).withMaxRows(100) + .getWidgetMetaData().withGridColumns(12).withPermissionRules(permissionRules(join.getRightTable()))); + } + return (widgets); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/metadata/SshKeyCustomizer.java b/src/main/java/com/kingsrook/qbits/filegateway/metadata/SshKeyCustomizer.java new file mode 100644 index 0000000..01449e7 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/metadata/SshKeyCustomizer.java @@ -0,0 +1,76 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.metadata; + + +import java.security.PublicKey; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; +import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface; +import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.statusmessages.BadInputStatusMessage; +import org.apache.sshd.common.config.keys.KeyUtils; +import org.apache.sshd.common.config.keys.PublicKeyEntry; +import org.apache.sshd.common.digest.BuiltinDigests; + + +/******************************************************************************* + ** Validates stored public keys and computes authoritative key metadata. + *******************************************************************************/ +public class SshKeyCustomizer implements TableCustomizerInterface +{ + + + + /******************************************************************************* + ** Parse and validate inserts and partial updates without logging key material. + *******************************************************************************/ + @Override + public List preInsertOrUpdate(AbstractActionInput input, List records, boolean isPreview, Optional> oldRecords) + { + for(QRecord record : records) + { + QRecord old = oldRecords.orElse(List.of()).stream().filter(candidate -> Objects.equals(candidate.getValue("id"), record.getValue("id"))).findFirst().orElse(null); + String text = record.getValues().containsKey("publicKey") ? record.getValueString("publicKey") : old == null ? null : old.getValueString("publicKey"); + try + { + PublicKey key = PublicKeyEntry.parsePublicKeyEntry(text).resolvePublicKey(null, Map.of(), null); + String type = KeyUtils.getKeyType(key); + int bits = KeyUtils.getKeySize(key); + if(!("ssh-ed25519".equals(type) || type.startsWith("ecdsa-sha2-nistp") || ("ssh-rsa".equals(type) && bits >= 3072))) + { + throw (new IllegalArgumentException("Unsupported key type or size")); + } + record.setValue("fingerprint", KeyUtils.getFingerPrint(BuiltinDigests.sha256, key)); + record.setValue("keyType", type); + record.setValue("keyBits", bits); + } + catch(Exception e) + { + record.addError(new BadInputStatusMessage("Use an OpenSSH Ed25519, ECDSA, or RSA key of at least 3072 bits")); + } + } + return (records); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayAccount.java b/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayAccount.java new file mode 100644 index 0000000..dd0a421 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayAccount.java @@ -0,0 +1,1076 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.model; + + +import java.time.Instant; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.data.QField; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.data.QRecordEntity; + + +/******************************************************************************* + ** Record entity for accounts. + *******************************************************************************/ +public class FileGatewayAccount extends QRecordEntity +{ + public static final String TABLE_NAME = "file_gateway_account"; + + @QField + private Integer id; + + @QField + private Instant createDate; + + @QField + private Instant modifyDate; + + @QField + private String username; + + @QField + private String label; + + @QField + private String accountType; + + @QField + private String userId; + + @QField + private String ownerUserId; + + @QField + private String status; + + @QField + private String rootMode; + + @QField + private String allowedCidrs; + + @QField + private Instant expiresAt; + + @QField + private String formats; + + @QField + private Integer maxConcurrentSessions; + + @QField + private Integer maxOperationsPerMinute; + + @QField + private Integer maxFileSizeMb; + + @QField + private Integer maxUploadMbPerDay; + + @QField + private Integer maxDownloadMbPerDay; + + @QField + private Boolean dedupeUploads; + + @QField + private Boolean notifyOnSuccess; + + @QField + private String pgpPublicKey; + + @QField + private Boolean pgpDecryptUploads; + + @QField + private Boolean pgpEncryptDownloads; + + @QField + private Instant operationWindowAt; + + @QField + private Integer operationWindowCount; + + @QField + private Integer failedLoginCount; + + @QField + private Instant failedLoginWindowAt; + + @QField + private Instant lockedUntil; + + @QField + private Instant lastLoginAt; + + @QField + private String lastLoginIp; + + + + /******************************************************************************* + ** Create an empty entity. + *******************************************************************************/ + public FileGatewayAccount() + { + } + + + + /******************************************************************************* + ** Populate from a database record. + *******************************************************************************/ + public FileGatewayAccount(QRecord record) throws QException + { + populateFromQRecord(record); + } + + + + /******************************************************************************* + ** Getter for id. + *******************************************************************************/ + public Integer getId() + { + return (id); + } + + + + /******************************************************************************* + ** Fluent setter for id. + *******************************************************************************/ + public FileGatewayAccount withId(Integer id) + { + this.id = id; + return (this); + } + + + + /******************************************************************************* + ** Getter for createDate. + *******************************************************************************/ + public Instant getCreateDate() + { + return (createDate); + } + + + + /******************************************************************************* + ** Fluent setter for createDate. + *******************************************************************************/ + public FileGatewayAccount withCreateDate(Instant createDate) + { + this.createDate = createDate; + return (this); + } + + + + /******************************************************************************* + ** Getter for modifyDate. + *******************************************************************************/ + public Instant getModifyDate() + { + return (modifyDate); + } + + + + /******************************************************************************* + ** Fluent setter for modifyDate. + *******************************************************************************/ + public FileGatewayAccount withModifyDate(Instant modifyDate) + { + this.modifyDate = modifyDate; + return (this); + } + + + + /******************************************************************************* + ** Getter for username. + *******************************************************************************/ + public String getUsername() + { + return (username); + } + + + + /******************************************************************************* + ** Fluent setter for username. + *******************************************************************************/ + public FileGatewayAccount withUsername(String username) + { + this.username = username; + return (this); + } + + + + /******************************************************************************* + ** Getter for label. + *******************************************************************************/ + public String getLabel() + { + return (label); + } + + + + /******************************************************************************* + ** Fluent setter for label. + *******************************************************************************/ + public FileGatewayAccount withLabel(String label) + { + this.label = label; + return (this); + } + + + + /******************************************************************************* + ** Getter for accountType. + *******************************************************************************/ + public String getAccountType() + { + return (accountType); + } + + + + /******************************************************************************* + ** Fluent setter for accountType. + *******************************************************************************/ + public FileGatewayAccount withAccountType(String accountType) + { + this.accountType = accountType; + return (this); + } + + + + /******************************************************************************* + ** Getter for userId. + *******************************************************************************/ + public String getUserId() + { + return (userId); + } + + + + /******************************************************************************* + ** Fluent setter for userId. + *******************************************************************************/ + public FileGatewayAccount withUserId(String userId) + { + this.userId = userId; + return (this); + } + + + + /******************************************************************************* + ** Getter for ownerUserId. + *******************************************************************************/ + public String getOwnerUserId() + { + return (ownerUserId); + } + + + + /******************************************************************************* + ** Fluent setter for ownerUserId. + *******************************************************************************/ + public FileGatewayAccount withOwnerUserId(String ownerUserId) + { + this.ownerUserId = ownerUserId; + return (this); + } + + + + /******************************************************************************* + ** Getter for status. + *******************************************************************************/ + public String getStatus() + { + return (status); + } + + + + /******************************************************************************* + ** Fluent setter for status. + *******************************************************************************/ + public FileGatewayAccount withStatus(String status) + { + this.status = status; + return (this); + } + + + + /******************************************************************************* + ** Getter for rootMode. + *******************************************************************************/ + public String getRootMode() + { + return (rootMode); + } + + + + /******************************************************************************* + ** Fluent setter for rootMode. + *******************************************************************************/ + public FileGatewayAccount withRootMode(String rootMode) + { + this.rootMode = rootMode; + return (this); + } + + + + /******************************************************************************* + ** Getter for allowedCidrs. + *******************************************************************************/ + public String getAllowedCidrs() + { + return (allowedCidrs); + } + + + + /******************************************************************************* + ** Fluent setter for allowedCidrs. + *******************************************************************************/ + public FileGatewayAccount withAllowedCidrs(String allowedCidrs) + { + this.allowedCidrs = allowedCidrs; + return (this); + } + + + + /******************************************************************************* + ** Getter for expiresAt. + *******************************************************************************/ + public Instant getExpiresAt() + { + return (expiresAt); + } + + + + /******************************************************************************* + ** Fluent setter for expiresAt. + *******************************************************************************/ + public FileGatewayAccount withExpiresAt(Instant expiresAt) + { + this.expiresAt = expiresAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for formats. + *******************************************************************************/ + public String getFormats() + { + return (formats); + } + + + + /******************************************************************************* + ** Fluent setter for formats. + *******************************************************************************/ + public FileGatewayAccount withFormats(String formats) + { + this.formats = formats; + return (this); + } + + + + /******************************************************************************* + ** Getter for maxConcurrentSessions. + *******************************************************************************/ + public Integer getMaxConcurrentSessions() + { + return (maxConcurrentSessions); + } + + + + /******************************************************************************* + ** Fluent setter for maxConcurrentSessions. + *******************************************************************************/ + public FileGatewayAccount withMaxConcurrentSessions(Integer maxConcurrentSessions) + { + this.maxConcurrentSessions = maxConcurrentSessions; + return (this); + } + + + + /******************************************************************************* + ** Getter for maxOperationsPerMinute. + *******************************************************************************/ + public Integer getMaxOperationsPerMinute() + { + return (maxOperationsPerMinute); + } + + + + /******************************************************************************* + ** Fluent setter for maxOperationsPerMinute. + *******************************************************************************/ + public FileGatewayAccount withMaxOperationsPerMinute(Integer maxOperationsPerMinute) + { + this.maxOperationsPerMinute = maxOperationsPerMinute; + return (this); + } + + + + /******************************************************************************* + ** Getter for maxFileSizeMb. + *******************************************************************************/ + public Integer getMaxFileSizeMb() + { + return (maxFileSizeMb); + } + + + + /******************************************************************************* + ** Fluent setter for maxFileSizeMb. + *******************************************************************************/ + public FileGatewayAccount withMaxFileSizeMb(Integer maxFileSizeMb) + { + this.maxFileSizeMb = maxFileSizeMb; + return (this); + } + + + + /******************************************************************************* + ** Getter for maxUploadMbPerDay. + *******************************************************************************/ + public Integer getMaxUploadMbPerDay() + { + return (maxUploadMbPerDay); + } + + + + /******************************************************************************* + ** Fluent setter for maxUploadMbPerDay. + *******************************************************************************/ + public FileGatewayAccount withMaxUploadMbPerDay(Integer maxUploadMbPerDay) + { + this.maxUploadMbPerDay = maxUploadMbPerDay; + return (this); + } + + + + /******************************************************************************* + ** Getter for maxDownloadMbPerDay. + *******************************************************************************/ + public Integer getMaxDownloadMbPerDay() + { + return (maxDownloadMbPerDay); + } + + + + /******************************************************************************* + ** Fluent setter for maxDownloadMbPerDay. + *******************************************************************************/ + public FileGatewayAccount withMaxDownloadMbPerDay(Integer maxDownloadMbPerDay) + { + this.maxDownloadMbPerDay = maxDownloadMbPerDay; + return (this); + } + + + + /******************************************************************************* + ** Getter for dedupeUploads. + *******************************************************************************/ + public Boolean getDedupeUploads() + { + return (dedupeUploads); + } + + + + /******************************************************************************* + ** Fluent setter for dedupeUploads. + *******************************************************************************/ + public FileGatewayAccount withDedupeUploads(Boolean dedupeUploads) + { + this.dedupeUploads = dedupeUploads; + return (this); + } + + + + /******************************************************************************* + ** Getter for notifyOnSuccess. + *******************************************************************************/ + public Boolean getNotifyOnSuccess() + { + return (notifyOnSuccess); + } + + + + /******************************************************************************* + ** Fluent setter for notifyOnSuccess. + *******************************************************************************/ + public FileGatewayAccount withNotifyOnSuccess(Boolean notifyOnSuccess) + { + this.notifyOnSuccess = notifyOnSuccess; + return (this); + } + + + + /******************************************************************************* + ** Getter for pgpPublicKey. + *******************************************************************************/ + public String getPgpPublicKey() + { + return (pgpPublicKey); + } + + + + /******************************************************************************* + ** Fluent setter for pgpPublicKey. + *******************************************************************************/ + public FileGatewayAccount withPgpPublicKey(String pgpPublicKey) + { + this.pgpPublicKey = pgpPublicKey; + return (this); + } + + + + /******************************************************************************* + ** Getter for pgpDecryptUploads. + *******************************************************************************/ + public Boolean getPgpDecryptUploads() + { + return (pgpDecryptUploads); + } + + + + /******************************************************************************* + ** Fluent setter for pgpDecryptUploads. + *******************************************************************************/ + public FileGatewayAccount withPgpDecryptUploads(Boolean pgpDecryptUploads) + { + this.pgpDecryptUploads = pgpDecryptUploads; + return (this); + } + + + + /******************************************************************************* + ** Getter for pgpEncryptDownloads. + *******************************************************************************/ + public Boolean getPgpEncryptDownloads() + { + return (pgpEncryptDownloads); + } + + + + /******************************************************************************* + ** Fluent setter for pgpEncryptDownloads. + *******************************************************************************/ + public FileGatewayAccount withPgpEncryptDownloads(Boolean pgpEncryptDownloads) + { + this.pgpEncryptDownloads = pgpEncryptDownloads; + return (this); + } + + + + /******************************************************************************* + ** Getter for failedLoginCount. + *******************************************************************************/ + public Integer getFailedLoginCount() + { + return (failedLoginCount); + } + + + + /******************************************************************************* + ** Fluent setter for failedLoginCount. + *******************************************************************************/ + public FileGatewayAccount withFailedLoginCount(Integer failedLoginCount) + { + this.failedLoginCount = failedLoginCount; + return (this); + } + + + + /******************************************************************************* + ** Getter for failedLoginWindowAt. + *******************************************************************************/ + public Instant getFailedLoginWindowAt() + { + return (failedLoginWindowAt); + } + + + + /******************************************************************************* + ** Fluent setter for failedLoginWindowAt. + *******************************************************************************/ + public FileGatewayAccount withFailedLoginWindowAt(Instant failedLoginWindowAt) + { + this.failedLoginWindowAt = failedLoginWindowAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for lockedUntil. + *******************************************************************************/ + public Instant getLockedUntil() + { + return (lockedUntil); + } + + + + /******************************************************************************* + ** Fluent setter for lockedUntil. + *******************************************************************************/ + public FileGatewayAccount withLockedUntil(Instant lockedUntil) + { + this.lockedUntil = lockedUntil; + return (this); + } + + + + /******************************************************************************* + ** Getter for lastLoginAt. + *******************************************************************************/ + public Instant getLastLoginAt() + { + return (lastLoginAt); + } + + + + /******************************************************************************* + ** Fluent setter for lastLoginAt. + *******************************************************************************/ + public FileGatewayAccount withLastLoginAt(Instant lastLoginAt) + { + this.lastLoginAt = lastLoginAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for lastLoginIp. + *******************************************************************************/ + public String getLastLoginIp() + { + return (lastLoginIp); + } + + + + /******************************************************************************* + ** Fluent setter for lastLoginIp. + *******************************************************************************/ + public FileGatewayAccount withLastLoginIp(String lastLoginIp) + { + this.lastLoginIp = lastLoginIp; + return (this); + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setId(Integer id) + { + this.id = id; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setCreateDate(Instant createDate) + { + this.createDate = createDate; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setModifyDate(Instant modifyDate) + { + this.modifyDate = modifyDate; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setUsername(String username) + { + this.username = username; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setLabel(String label) + { + this.label = label; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setAccountType(String accountType) + { + this.accountType = accountType; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setUserId(String userId) + { + this.userId = userId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setOwnerUserId(String ownerUserId) + { + this.ownerUserId = ownerUserId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setStatus(String status) + { + this.status = status; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setRootMode(String rootMode) + { + this.rootMode = rootMode; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setAllowedCidrs(String allowedCidrs) + { + this.allowedCidrs = allowedCidrs; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setExpiresAt(Instant expiresAt) + { + this.expiresAt = expiresAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setFormats(String formats) + { + this.formats = formats; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setMaxConcurrentSessions(Integer maxConcurrentSessions) + { + this.maxConcurrentSessions = maxConcurrentSessions; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setMaxOperationsPerMinute(Integer maxOperationsPerMinute) + { + this.maxOperationsPerMinute = maxOperationsPerMinute; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setMaxFileSizeMb(Integer maxFileSizeMb) + { + this.maxFileSizeMb = maxFileSizeMb; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setMaxUploadMbPerDay(Integer maxUploadMbPerDay) + { + this.maxUploadMbPerDay = maxUploadMbPerDay; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setMaxDownloadMbPerDay(Integer maxDownloadMbPerDay) + { + this.maxDownloadMbPerDay = maxDownloadMbPerDay; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setDedupeUploads(Boolean dedupeUploads) + { + this.dedupeUploads = dedupeUploads; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setNotifyOnSuccess(Boolean notifyOnSuccess) + { + this.notifyOnSuccess = notifyOnSuccess; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setPgpPublicKey(String pgpPublicKey) + { + this.pgpPublicKey = pgpPublicKey; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setPgpDecryptUploads(Boolean pgpDecryptUploads) + { + this.pgpDecryptUploads = pgpDecryptUploads; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setPgpEncryptDownloads(Boolean pgpEncryptDownloads) + { + this.pgpEncryptDownloads = pgpEncryptDownloads; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setFailedLoginCount(Integer failedLoginCount) + { + this.failedLoginCount = failedLoginCount; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setFailedLoginWindowAt(Instant failedLoginWindowAt) + { + this.failedLoginWindowAt = failedLoginWindowAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setLockedUntil(Instant lockedUntil) + { + this.lockedUntil = lockedUntil; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setLastLoginAt(Instant lastLoginAt) + { + this.lastLoginAt = lastLoginAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setLastLoginIp(String lastLoginIp) + { + this.lastLoginIp = lastLoginIp; + } + + + + /******************************************************************************* + ** Getter for operationWindowAt. + *******************************************************************************/ + public Instant getOperationWindowAt() + { + return (operationWindowAt); + } + + + + /******************************************************************************* + ** Fluent setter for operationWindowAt. + *******************************************************************************/ + public FileGatewayAccount withOperationWindowAt(Instant operationWindowAt) + { + this.operationWindowAt = operationWindowAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for operationWindowCount. + *******************************************************************************/ + public Integer getOperationWindowCount() + { + return (operationWindowCount); + } + + + + /******************************************************************************* + ** Fluent setter for operationWindowCount. + *******************************************************************************/ + public FileGatewayAccount withOperationWindowCount(Integer operationWindowCount) + { + this.operationWindowCount = operationWindowCount; + return (this); + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setOperationWindowAt(Instant operationWindowAt) + { + this.operationWindowAt = operationWindowAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setOperationWindowCount(Integer operationWindowCount) + { + this.operationWindowCount = operationWindowCount; + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayAlias.java b/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayAlias.java new file mode 100644 index 0000000..a5d8f59 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayAlias.java @@ -0,0 +1,600 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.model; + + +import java.time.Instant; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.data.QField; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.data.QRecordEntity; + + +/******************************************************************************* + ** Record entity for endpoints. + *******************************************************************************/ +public class FileGatewayAlias extends QRecordEntity +{ + public static final String TABLE_NAME = "file_gateway_alias"; + + @QField + private Integer id; + + @QField + private Instant createDate; + + @QField + private Instant modifyDate; + + @QField + private String name; + + @QField + private String path; + + @QField + private String kind; + + @QField + private Integer accountId; + + @QField + private String tableName; + + @QField + private Integer savedBulkLoadProfileId; + + @QField + private Integer savedViewId; + + @QField + private Integer savedReportId; + + @QField + private String format; + + @QField + private String fileName; + + @QField + private String consumerName; + + @QField + private Boolean isActive; + + @QField + private String description; + + + + /******************************************************************************* + ** Create an empty entity. + *******************************************************************************/ + public FileGatewayAlias() + { + } + + + + /******************************************************************************* + ** Populate from a database record. + *******************************************************************************/ + public FileGatewayAlias(QRecord record) throws QException + { + populateFromQRecord(record); + } + + + + /******************************************************************************* + ** Getter for id. + *******************************************************************************/ + public Integer getId() + { + return (id); + } + + + + /******************************************************************************* + ** Fluent setter for id. + *******************************************************************************/ + public FileGatewayAlias withId(Integer id) + { + this.id = id; + return (this); + } + + + + /******************************************************************************* + ** Getter for createDate. + *******************************************************************************/ + public Instant getCreateDate() + { + return (createDate); + } + + + + /******************************************************************************* + ** Fluent setter for createDate. + *******************************************************************************/ + public FileGatewayAlias withCreateDate(Instant createDate) + { + this.createDate = createDate; + return (this); + } + + + + /******************************************************************************* + ** Getter for modifyDate. + *******************************************************************************/ + public Instant getModifyDate() + { + return (modifyDate); + } + + + + /******************************************************************************* + ** Fluent setter for modifyDate. + *******************************************************************************/ + public FileGatewayAlias withModifyDate(Instant modifyDate) + { + this.modifyDate = modifyDate; + return (this); + } + + + + /******************************************************************************* + ** Getter for name. + *******************************************************************************/ + public String getName() + { + return (name); + } + + + + /******************************************************************************* + ** Fluent setter for name. + *******************************************************************************/ + public FileGatewayAlias withName(String name) + { + this.name = name; + return (this); + } + + + + /******************************************************************************* + ** Getter for path. + *******************************************************************************/ + public String getPath() + { + return (path); + } + + + + /******************************************************************************* + ** Fluent setter for path. + *******************************************************************************/ + public FileGatewayAlias withPath(String path) + { + this.path = path; + return (this); + } + + + + /******************************************************************************* + ** Getter for kind. + *******************************************************************************/ + public String getKind() + { + return (kind); + } + + + + /******************************************************************************* + ** Fluent setter for kind. + *******************************************************************************/ + public FileGatewayAlias withKind(String kind) + { + this.kind = kind; + return (this); + } + + + + /******************************************************************************* + ** Getter for accountId. + *******************************************************************************/ + public Integer getAccountId() + { + return (accountId); + } + + + + /******************************************************************************* + ** Fluent setter for accountId. + *******************************************************************************/ + public FileGatewayAlias withAccountId(Integer accountId) + { + this.accountId = accountId; + return (this); + } + + + + /******************************************************************************* + ** Getter for tableName. + *******************************************************************************/ + public String getTableName() + { + return (tableName); + } + + + + /******************************************************************************* + ** Fluent setter for tableName. + *******************************************************************************/ + public FileGatewayAlias withTableName(String tableName) + { + this.tableName = tableName; + return (this); + } + + + + /******************************************************************************* + ** Getter for savedBulkLoadProfileId. + *******************************************************************************/ + public Integer getSavedBulkLoadProfileId() + { + return (savedBulkLoadProfileId); + } + + + + /******************************************************************************* + ** Fluent setter for savedBulkLoadProfileId. + *******************************************************************************/ + public FileGatewayAlias withSavedBulkLoadProfileId(Integer savedBulkLoadProfileId) + { + this.savedBulkLoadProfileId = savedBulkLoadProfileId; + return (this); + } + + + + /******************************************************************************* + ** Getter for savedViewId. + *******************************************************************************/ + public Integer getSavedViewId() + { + return (savedViewId); + } + + + + /******************************************************************************* + ** Fluent setter for savedViewId. + *******************************************************************************/ + public FileGatewayAlias withSavedViewId(Integer savedViewId) + { + this.savedViewId = savedViewId; + return (this); + } + + + + /******************************************************************************* + ** Getter for savedReportId. + *******************************************************************************/ + public Integer getSavedReportId() + { + return (savedReportId); + } + + + + /******************************************************************************* + ** Fluent setter for savedReportId. + *******************************************************************************/ + public FileGatewayAlias withSavedReportId(Integer savedReportId) + { + this.savedReportId = savedReportId; + return (this); + } + + + + /******************************************************************************* + ** Getter for format. + *******************************************************************************/ + public String getFormat() + { + return (format); + } + + + + /******************************************************************************* + ** Fluent setter for format. + *******************************************************************************/ + public FileGatewayAlias withFormat(String format) + { + this.format = format; + return (this); + } + + + + /******************************************************************************* + ** Getter for fileName. + *******************************************************************************/ + public String getFileName() + { + return (fileName); + } + + + + /******************************************************************************* + ** Fluent setter for fileName. + *******************************************************************************/ + public FileGatewayAlias withFileName(String fileName) + { + this.fileName = fileName; + return (this); + } + + + + /******************************************************************************* + ** Getter for consumerName. + *******************************************************************************/ + public String getConsumerName() + { + return (consumerName); + } + + + + /******************************************************************************* + ** Fluent setter for consumerName. + *******************************************************************************/ + public FileGatewayAlias withConsumerName(String consumerName) + { + this.consumerName = consumerName; + return (this); + } + + + + /******************************************************************************* + ** Getter for isActive. + *******************************************************************************/ + public Boolean getIsActive() + { + return (isActive); + } + + + + /******************************************************************************* + ** Fluent setter for isActive. + *******************************************************************************/ + public FileGatewayAlias withIsActive(Boolean isActive) + { + this.isActive = isActive; + return (this); + } + + + + /******************************************************************************* + ** Getter for description. + *******************************************************************************/ + public String getDescription() + { + return (description); + } + + + + /******************************************************************************* + ** Fluent setter for description. + *******************************************************************************/ + public FileGatewayAlias withDescription(String description) + { + this.description = description; + return (this); + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setId(Integer id) + { + this.id = id; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setCreateDate(Instant createDate) + { + this.createDate = createDate; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setModifyDate(Instant modifyDate) + { + this.modifyDate = modifyDate; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setName(String name) + { + this.name = name; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setPath(String path) + { + this.path = path; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setKind(String kind) + { + this.kind = kind; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setAccountId(Integer accountId) + { + this.accountId = accountId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setTableName(String tableName) + { + this.tableName = tableName; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setSavedBulkLoadProfileId(Integer savedBulkLoadProfileId) + { + this.savedBulkLoadProfileId = savedBulkLoadProfileId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setSavedViewId(Integer savedViewId) + { + this.savedViewId = savedViewId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setSavedReportId(Integer savedReportId) + { + this.savedReportId = savedReportId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setFormat(String format) + { + this.format = format; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setFileName(String fileName) + { + this.fileName = fileName; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setConsumerName(String consumerName) + { + this.consumerName = consumerName; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setIsActive(Boolean isActive) + { + this.isActive = isActive; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setDescription(String description) + { + this.description = description; + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayConnectionLog.java b/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayConnectionLog.java new file mode 100644 index 0000000..2460779 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayConnectionLog.java @@ -0,0 +1,737 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.model; + + +import java.time.Instant; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.data.QField; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.data.QRecordEntity; + + +/******************************************************************************* + ** Record entity for connections. + *******************************************************************************/ +public class FileGatewayConnectionLog extends QRecordEntity +{ + public static final String TABLE_NAME = "file_gateway_connection_log"; + + @QField + private Integer id; + + @QField + private Instant createDate; + + @QField + private Instant modifyDate; + + @QField + private Integer accountId; + + @QField + private String username; + + @QField + private String remoteIp; + + @QField + private String clientVersion; + + @QField + private String keyFingerprint; + + @QField + private String outcome; + + @QField + private String sessionId; + + @QField + private String podName; + + @QField + private Instant connectedAt; + + @QField + private Instant lastSeenAt; + + @QField + private Instant disconnectedAt; + + @QField + private String disconnectReason; + + @QField + private Long bytesIn; + + @QField + private Long bytesOut; + + @QField + private Long operationCount; + + @QField + private String clientChannel; + + + + @QField + private Instant endRequestedAt; + + + /******************************************************************************* + ** Create an empty entity. + *******************************************************************************/ + public FileGatewayConnectionLog() + { + } + + + + /******************************************************************************* + ** Populate from a database record. + *******************************************************************************/ + public FileGatewayConnectionLog(QRecord record) throws QException + { + populateFromQRecord(record); + } + + + + /******************************************************************************* + ** Getter for id. + *******************************************************************************/ + public Integer getId() + { + return (id); + } + + + + /******************************************************************************* + ** Fluent setter for id. + *******************************************************************************/ + public FileGatewayConnectionLog withId(Integer id) + { + this.id = id; + return (this); + } + + + + /******************************************************************************* + ** Getter for createDate. + *******************************************************************************/ + public Instant getCreateDate() + { + return (createDate); + } + + + + /******************************************************************************* + ** Fluent setter for createDate. + *******************************************************************************/ + public FileGatewayConnectionLog withCreateDate(Instant createDate) + { + this.createDate = createDate; + return (this); + } + + + + /******************************************************************************* + ** Getter for modifyDate. + *******************************************************************************/ + public Instant getModifyDate() + { + return (modifyDate); + } + + + + /******************************************************************************* + ** Fluent setter for modifyDate. + *******************************************************************************/ + public FileGatewayConnectionLog withModifyDate(Instant modifyDate) + { + this.modifyDate = modifyDate; + return (this); + } + + + + /******************************************************************************* + ** Getter for accountId. + *******************************************************************************/ + public Integer getAccountId() + { + return (accountId); + } + + + + /******************************************************************************* + ** Fluent setter for accountId. + *******************************************************************************/ + public FileGatewayConnectionLog withAccountId(Integer accountId) + { + this.accountId = accountId; + return (this); + } + + + + /******************************************************************************* + ** Getter for username. + *******************************************************************************/ + public String getUsername() + { + return (username); + } + + + + /******************************************************************************* + ** Fluent setter for username. + *******************************************************************************/ + public FileGatewayConnectionLog withUsername(String username) + { + this.username = username; + return (this); + } + + + + /******************************************************************************* + ** Getter for remoteIp. + *******************************************************************************/ + public String getRemoteIp() + { + return (remoteIp); + } + + + + /******************************************************************************* + ** Fluent setter for remoteIp. + *******************************************************************************/ + public FileGatewayConnectionLog withRemoteIp(String remoteIp) + { + this.remoteIp = remoteIp; + return (this); + } + + + + /******************************************************************************* + ** Getter for clientVersion. + *******************************************************************************/ + public String getClientVersion() + { + return (clientVersion); + } + + + + /******************************************************************************* + ** Fluent setter for clientVersion. + *******************************************************************************/ + public FileGatewayConnectionLog withClientVersion(String clientVersion) + { + this.clientVersion = clientVersion; + return (this); + } + + + + /******************************************************************************* + ** Getter for keyFingerprint. + *******************************************************************************/ + public String getKeyFingerprint() + { + return (keyFingerprint); + } + + + + /******************************************************************************* + ** Fluent setter for keyFingerprint. + *******************************************************************************/ + public FileGatewayConnectionLog withKeyFingerprint(String keyFingerprint) + { + this.keyFingerprint = keyFingerprint; + return (this); + } + + + + /******************************************************************************* + ** Getter for outcome. + *******************************************************************************/ + public String getOutcome() + { + return (outcome); + } + + + + /******************************************************************************* + ** Fluent setter for outcome. + *******************************************************************************/ + public FileGatewayConnectionLog withOutcome(String outcome) + { + this.outcome = outcome; + return (this); + } + + + + /******************************************************************************* + ** Getter for sessionId. + *******************************************************************************/ + public String getSessionId() + { + return (sessionId); + } + + + + /******************************************************************************* + ** Fluent setter for sessionId. + *******************************************************************************/ + public FileGatewayConnectionLog withSessionId(String sessionId) + { + this.sessionId = sessionId; + return (this); + } + + + + /******************************************************************************* + ** Getter for podName. + *******************************************************************************/ + public String getPodName() + { + return (podName); + } + + + + /******************************************************************************* + ** Fluent setter for podName. + *******************************************************************************/ + public FileGatewayConnectionLog withPodName(String podName) + { + this.podName = podName; + return (this); + } + + + + /******************************************************************************* + ** Getter for connectedAt. + *******************************************************************************/ + public Instant getConnectedAt() + { + return (connectedAt); + } + + + + /******************************************************************************* + ** Fluent setter for connectedAt. + *******************************************************************************/ + public FileGatewayConnectionLog withConnectedAt(Instant connectedAt) + { + this.connectedAt = connectedAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for lastSeenAt. + *******************************************************************************/ + public Instant getLastSeenAt() + { + return (lastSeenAt); + } + + + + /******************************************************************************* + ** Fluent setter for lastSeenAt. + *******************************************************************************/ + public FileGatewayConnectionLog withLastSeenAt(Instant lastSeenAt) + { + this.lastSeenAt = lastSeenAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for disconnectedAt. + *******************************************************************************/ + public Instant getDisconnectedAt() + { + return (disconnectedAt); + } + + + + /******************************************************************************* + ** Fluent setter for disconnectedAt. + *******************************************************************************/ + public FileGatewayConnectionLog withDisconnectedAt(Instant disconnectedAt) + { + this.disconnectedAt = disconnectedAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for disconnectReason. + *******************************************************************************/ + public String getDisconnectReason() + { + return (disconnectReason); + } + + + + /******************************************************************************* + ** Fluent setter for disconnectReason. + *******************************************************************************/ + public FileGatewayConnectionLog withDisconnectReason(String disconnectReason) + { + this.disconnectReason = disconnectReason; + return (this); + } + + + + /******************************************************************************* + ** Getter for bytesIn. + *******************************************************************************/ + public Long getBytesIn() + { + return (bytesIn); + } + + + + /******************************************************************************* + ** Fluent setter for bytesIn. + *******************************************************************************/ + public FileGatewayConnectionLog withBytesIn(Long bytesIn) + { + this.bytesIn = bytesIn; + return (this); + } + + + + /******************************************************************************* + ** Getter for bytesOut. + *******************************************************************************/ + public Long getBytesOut() + { + return (bytesOut); + } + + + + /******************************************************************************* + ** Fluent setter for bytesOut. + *******************************************************************************/ + public FileGatewayConnectionLog withBytesOut(Long bytesOut) + { + this.bytesOut = bytesOut; + return (this); + } + + + + /******************************************************************************* + ** Getter for operationCount. + *******************************************************************************/ + public Long getOperationCount() + { + return (operationCount); + } + + + + /******************************************************************************* + ** Fluent setter for operationCount. + *******************************************************************************/ + public FileGatewayConnectionLog withOperationCount(Long operationCount) + { + this.operationCount = operationCount; + return (this); + } + + + + /******************************************************************************* + ** Getter for clientChannel. + *******************************************************************************/ + public String getClientChannel() + { + return (clientChannel); + } + + + + /******************************************************************************* + ** Fluent setter for clientChannel. + *******************************************************************************/ + public FileGatewayConnectionLog withClientChannel(String clientChannel) + { + this.clientChannel = clientChannel; + return (this); + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setId(Integer id) + { + this.id = id; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setCreateDate(Instant createDate) + { + this.createDate = createDate; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setModifyDate(Instant modifyDate) + { + this.modifyDate = modifyDate; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setAccountId(Integer accountId) + { + this.accountId = accountId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setUsername(String username) + { + this.username = username; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setRemoteIp(String remoteIp) + { + this.remoteIp = remoteIp; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setClientVersion(String clientVersion) + { + this.clientVersion = clientVersion; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setKeyFingerprint(String keyFingerprint) + { + this.keyFingerprint = keyFingerprint; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setOutcome(String outcome) + { + this.outcome = outcome; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setSessionId(String sessionId) + { + this.sessionId = sessionId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setPodName(String podName) + { + this.podName = podName; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setConnectedAt(Instant connectedAt) + { + this.connectedAt = connectedAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setLastSeenAt(Instant lastSeenAt) + { + this.lastSeenAt = lastSeenAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setDisconnectedAt(Instant disconnectedAt) + { + this.disconnectedAt = disconnectedAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setDisconnectReason(String disconnectReason) + { + this.disconnectReason = disconnectReason; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setBytesIn(Long bytesIn) + { + this.bytesIn = bytesIn; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setBytesOut(Long bytesOut) + { + this.bytesOut = bytesOut; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setOperationCount(Long operationCount) + { + this.operationCount = operationCount; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setClientChannel(String clientChannel) + { + this.clientChannel = clientChannel; + } + + + + /******************************************************************************* + ** Getter for endRequestedAt. + *******************************************************************************/ + public Instant getEndRequestedAt() + { + return (endRequestedAt); + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setEndRequestedAt(Instant endRequestedAt) + { + this.endRequestedAt = endRequestedAt; + } + + + + /******************************************************************************* + ** Fluent setter for endRequestedAt. + *******************************************************************************/ + public FileGatewayConnectionLog withEndRequestedAt(Instant endRequestedAt) + { + this.endRequestedAt = endRequestedAt; + return (this); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayOutboxFile.java b/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayOutboxFile.java new file mode 100644 index 0000000..b1a2863 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayOutboxFile.java @@ -0,0 +1,567 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.model; + + +import java.time.Instant; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.data.QField; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.data.QRecordEntity; + + +/******************************************************************************* + ** Record entity for outbox files. + *******************************************************************************/ +public class FileGatewayOutboxFile extends QRecordEntity +{ + public static final String TABLE_NAME = "file_gateway_outbox_file"; + + @QField + private Integer id; + + @QField + private Instant createDate; + + @QField + private Instant modifyDate; + + @QField + private Integer aliasId; + + @QField + private String fileName; + + @QField + private Long sizeBytes; + + @QField + private String sha256; + + @QField + private String storageKey; + + @QField + private String status; + + @QField + private Instant firstDownloadedAt; + + @QField + private Integer downloadCount; + + @QField + private Instant removedAt; + + @QField + private Instant expiresAt; + + @QField + private String createdByConsumer; + + + + @QField + private Boolean onDownloadedPending; + + + /******************************************************************************* + ** Create an empty entity. + *******************************************************************************/ + public FileGatewayOutboxFile() + { + } + + + + /******************************************************************************* + ** Populate from a database record. + *******************************************************************************/ + public FileGatewayOutboxFile(QRecord record) throws QException + { + populateFromQRecord(record); + } + + + + /******************************************************************************* + ** Getter for id. + *******************************************************************************/ + public Integer getId() + { + return (id); + } + + + + /******************************************************************************* + ** Fluent setter for id. + *******************************************************************************/ + public FileGatewayOutboxFile withId(Integer id) + { + this.id = id; + return (this); + } + + + + /******************************************************************************* + ** Getter for createDate. + *******************************************************************************/ + public Instant getCreateDate() + { + return (createDate); + } + + + + /******************************************************************************* + ** Fluent setter for createDate. + *******************************************************************************/ + public FileGatewayOutboxFile withCreateDate(Instant createDate) + { + this.createDate = createDate; + return (this); + } + + + + /******************************************************************************* + ** Getter for modifyDate. + *******************************************************************************/ + public Instant getModifyDate() + { + return (modifyDate); + } + + + + /******************************************************************************* + ** Fluent setter for modifyDate. + *******************************************************************************/ + public FileGatewayOutboxFile withModifyDate(Instant modifyDate) + { + this.modifyDate = modifyDate; + return (this); + } + + + + /******************************************************************************* + ** Getter for aliasId. + *******************************************************************************/ + public Integer getAliasId() + { + return (aliasId); + } + + + + /******************************************************************************* + ** Fluent setter for aliasId. + *******************************************************************************/ + public FileGatewayOutboxFile withAliasId(Integer aliasId) + { + this.aliasId = aliasId; + return (this); + } + + + + /******************************************************************************* + ** Getter for fileName. + *******************************************************************************/ + public String getFileName() + { + return (fileName); + } + + + + /******************************************************************************* + ** Fluent setter for fileName. + *******************************************************************************/ + public FileGatewayOutboxFile withFileName(String fileName) + { + this.fileName = fileName; + return (this); + } + + + + /******************************************************************************* + ** Getter for sizeBytes. + *******************************************************************************/ + public Long getSizeBytes() + { + return (sizeBytes); + } + + + + /******************************************************************************* + ** Fluent setter for sizeBytes. + *******************************************************************************/ + public FileGatewayOutboxFile withSizeBytes(Long sizeBytes) + { + this.sizeBytes = sizeBytes; + return (this); + } + + + + /******************************************************************************* + ** Getter for sha256. + *******************************************************************************/ + public String getSha256() + { + return (sha256); + } + + + + /******************************************************************************* + ** Fluent setter for sha256. + *******************************************************************************/ + public FileGatewayOutboxFile withSha256(String sha256) + { + this.sha256 = sha256; + return (this); + } + + + + /******************************************************************************* + ** Getter for storageKey. + *******************************************************************************/ + public String getStorageKey() + { + return (storageKey); + } + + + + /******************************************************************************* + ** Fluent setter for storageKey. + *******************************************************************************/ + public FileGatewayOutboxFile withStorageKey(String storageKey) + { + this.storageKey = storageKey; + return (this); + } + + + + /******************************************************************************* + ** Getter for status. + *******************************************************************************/ + public String getStatus() + { + return (status); + } + + + + /******************************************************************************* + ** Fluent setter for status. + *******************************************************************************/ + public FileGatewayOutboxFile withStatus(String status) + { + this.status = status; + return (this); + } + + + + /******************************************************************************* + ** Getter for firstDownloadedAt. + *******************************************************************************/ + public Instant getFirstDownloadedAt() + { + return (firstDownloadedAt); + } + + + + /******************************************************************************* + ** Fluent setter for firstDownloadedAt. + *******************************************************************************/ + public FileGatewayOutboxFile withFirstDownloadedAt(Instant firstDownloadedAt) + { + this.firstDownloadedAt = firstDownloadedAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for downloadCount. + *******************************************************************************/ + public Integer getDownloadCount() + { + return (downloadCount); + } + + + + /******************************************************************************* + ** Fluent setter for downloadCount. + *******************************************************************************/ + public FileGatewayOutboxFile withDownloadCount(Integer downloadCount) + { + this.downloadCount = downloadCount; + return (this); + } + + + + /******************************************************************************* + ** Getter for removedAt. + *******************************************************************************/ + public Instant getRemovedAt() + { + return (removedAt); + } + + + + /******************************************************************************* + ** Fluent setter for removedAt. + *******************************************************************************/ + public FileGatewayOutboxFile withRemovedAt(Instant removedAt) + { + this.removedAt = removedAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for expiresAt. + *******************************************************************************/ + public Instant getExpiresAt() + { + return (expiresAt); + } + + + + /******************************************************************************* + ** Fluent setter for expiresAt. + *******************************************************************************/ + public FileGatewayOutboxFile withExpiresAt(Instant expiresAt) + { + this.expiresAt = expiresAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for createdByConsumer. + *******************************************************************************/ + public String getCreatedByConsumer() + { + return (createdByConsumer); + } + + + + /******************************************************************************* + ** Fluent setter for createdByConsumer. + *******************************************************************************/ + public FileGatewayOutboxFile withCreatedByConsumer(String createdByConsumer) + { + this.createdByConsumer = createdByConsumer; + return (this); + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setId(Integer id) + { + this.id = id; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setCreateDate(Instant createDate) + { + this.createDate = createDate; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setModifyDate(Instant modifyDate) + { + this.modifyDate = modifyDate; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setAliasId(Integer aliasId) + { + this.aliasId = aliasId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setFileName(String fileName) + { + this.fileName = fileName; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setSizeBytes(Long sizeBytes) + { + this.sizeBytes = sizeBytes; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setSha256(String sha256) + { + this.sha256 = sha256; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setStorageKey(String storageKey) + { + this.storageKey = storageKey; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setStatus(String status) + { + this.status = status; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setFirstDownloadedAt(Instant firstDownloadedAt) + { + this.firstDownloadedAt = firstDownloadedAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setDownloadCount(Integer downloadCount) + { + this.downloadCount = downloadCount; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setRemovedAt(Instant removedAt) + { + this.removedAt = removedAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setExpiresAt(Instant expiresAt) + { + this.expiresAt = expiresAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setCreatedByConsumer(String createdByConsumer) + { + this.createdByConsumer = createdByConsumer; + } + + + + /******************************************************************************* + ** Getter for onDownloadedPending. + *******************************************************************************/ + public Boolean getOnDownloadedPending() + { + return (onDownloadedPending); + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setOnDownloadedPending(Boolean onDownloadedPending) + { + this.onDownloadedPending = onDownloadedPending; + } + + + + /******************************************************************************* + ** Fluent setter for onDownloadedPending. + *******************************************************************************/ + public FileGatewayOutboxFile withOnDownloadedPending(Boolean onDownloadedPending) + { + this.onDownloadedPending = onDownloadedPending; + return (this); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayServer.java b/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayServer.java new file mode 100644 index 0000000..6bae6c0 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayServer.java @@ -0,0 +1,324 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.model; + +import java.time.Instant; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.data.QField; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.data.QRecordEntity; + +/******************************************************************************* + ** Persisted gateway heartbeat, visible across deployments. + *******************************************************************************/ +public class FileGatewayServer extends QRecordEntity +{ + public static final String TABLE_NAME = "file_gateway_server"; + + @QField + private Integer id; + + @QField + private Instant createDate; + + @QField + private Instant modifyDate; + + @QField + private String podName; + + @QField + private String hostKeyFingerprint; + + @QField + private Integer port; + + @QField + private Instant startedAt; + + @QField + private Instant lastSeenAt; + + /******************************************************************************* + ** Empty record. + *******************************************************************************/ + public FileGatewayServer() + { + } + + + + /******************************************************************************* + ** Populate from gateway persistence. + *******************************************************************************/ + public FileGatewayServer(QRecord record) throws QException + { + populateFromQRecord(record); + } + + + + /******************************************************************************* + ** Access id. + *******************************************************************************/ + public Integer getId() + { + return (id); + } + + + + /******************************************************************************* + ** Access id. + *******************************************************************************/ + public void setId(Integer id) + { + this.id = id; + } + + + + /******************************************************************************* + ** Access id. + *******************************************************************************/ + public FileGatewayServer withId(Integer id) + { + this.id = id; + return (this); + } + + + + /******************************************************************************* + ** Access createDate. + *******************************************************************************/ + public Instant getCreateDate() + { + return (createDate); + } + + + + /******************************************************************************* + ** Access createDate. + *******************************************************************************/ + public void setCreateDate(Instant createDate) + { + this.createDate = createDate; + } + + + + /******************************************************************************* + ** Access createDate. + *******************************************************************************/ + public FileGatewayServer withCreateDate(Instant createDate) + { + this.createDate = createDate; + return (this); + } + + + + /******************************************************************************* + ** Access modifyDate. + *******************************************************************************/ + public Instant getModifyDate() + { + return (modifyDate); + } + + + + /******************************************************************************* + ** Access modifyDate. + *******************************************************************************/ + public void setModifyDate(Instant modifyDate) + { + this.modifyDate = modifyDate; + } + + + + /******************************************************************************* + ** Access modifyDate. + *******************************************************************************/ + public FileGatewayServer withModifyDate(Instant modifyDate) + { + this.modifyDate = modifyDate; + return (this); + } + + + + /******************************************************************************* + ** Access podName. + *******************************************************************************/ + public String getPodName() + { + return (podName); + } + + + + /******************************************************************************* + ** Access podName. + *******************************************************************************/ + public void setPodName(String podName) + { + this.podName = podName; + } + + + + /******************************************************************************* + ** Access podName. + *******************************************************************************/ + public FileGatewayServer withPodName(String podName) + { + this.podName = podName; + return (this); + } + + + + /******************************************************************************* + ** Access hostKeyFingerprint. + *******************************************************************************/ + public String getHostKeyFingerprint() + { + return (hostKeyFingerprint); + } + + + + /******************************************************************************* + ** Access hostKeyFingerprint. + *******************************************************************************/ + public void setHostKeyFingerprint(String hostKeyFingerprint) + { + this.hostKeyFingerprint = hostKeyFingerprint; + } + + + + /******************************************************************************* + ** Access hostKeyFingerprint. + *******************************************************************************/ + public FileGatewayServer withHostKeyFingerprint(String hostKeyFingerprint) + { + this.hostKeyFingerprint = hostKeyFingerprint; + return (this); + } + + + + /******************************************************************************* + ** Access port. + *******************************************************************************/ + public Integer getPort() + { + return (port); + } + + + + /******************************************************************************* + ** Access port. + *******************************************************************************/ + public void setPort(Integer port) + { + this.port = port; + } + + + + /******************************************************************************* + ** Access port. + *******************************************************************************/ + public FileGatewayServer withPort(Integer port) + { + this.port = port; + return (this); + } + + + + /******************************************************************************* + ** Access startedAt. + *******************************************************************************/ + public Instant getStartedAt() + { + return (startedAt); + } + + + + /******************************************************************************* + ** Access startedAt. + *******************************************************************************/ + public void setStartedAt(Instant startedAt) + { + this.startedAt = startedAt; + } + + + + /******************************************************************************* + ** Access startedAt. + *******************************************************************************/ + public FileGatewayServer withStartedAt(Instant startedAt) + { + this.startedAt = startedAt; + return (this); + } + + + + /******************************************************************************* + ** Access lastSeenAt. + *******************************************************************************/ + public Instant getLastSeenAt() + { + return (lastSeenAt); + } + + + + /******************************************************************************* + ** Access lastSeenAt. + *******************************************************************************/ + public void setLastSeenAt(Instant lastSeenAt) + { + this.lastSeenAt = lastSeenAt; + } + + + + /******************************************************************************* + ** Access lastSeenAt. + *******************************************************************************/ + public FileGatewayServer withLastSeenAt(Instant lastSeenAt) + { + this.lastSeenAt = lastSeenAt; + return (this); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewaySshKey.java b/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewaySshKey.java new file mode 100644 index 0000000..eb4951c --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewaySshKey.java @@ -0,0 +1,498 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.model; + + +import java.time.Instant; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.data.QField; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.data.QRecordEntity; + + +/******************************************************************************* + ** Record entity for ssh keys. + *******************************************************************************/ +public class FileGatewaySshKey extends QRecordEntity +{ + public static final String TABLE_NAME = "file_gateway_ssh_key"; + + @QField + private Integer id; + + @QField + private Instant createDate; + + @QField + private Instant modifyDate; + + @QField + private Integer accountId; + + @QField + private String label; + + @QField + private String publicKey; + + @QField + private String fingerprint; + + @QField + private String keyType; + + @QField + private Integer keyBits; + + @QField + private Instant expiresAt; + + @QField + private Instant revokedAt; + + @QField + private Instant lastUsedAt; + + @QField + private String lastUsedIp; + + + + /******************************************************************************* + ** Create an empty entity. + *******************************************************************************/ + public FileGatewaySshKey() + { + } + + + + /******************************************************************************* + ** Populate from a database record. + *******************************************************************************/ + public FileGatewaySshKey(QRecord record) throws QException + { + populateFromQRecord(record); + } + + + + /******************************************************************************* + ** Getter for id. + *******************************************************************************/ + public Integer getId() + { + return (id); + } + + + + /******************************************************************************* + ** Fluent setter for id. + *******************************************************************************/ + public FileGatewaySshKey withId(Integer id) + { + this.id = id; + return (this); + } + + + + /******************************************************************************* + ** Getter for createDate. + *******************************************************************************/ + public Instant getCreateDate() + { + return (createDate); + } + + + + /******************************************************************************* + ** Fluent setter for createDate. + *******************************************************************************/ + public FileGatewaySshKey withCreateDate(Instant createDate) + { + this.createDate = createDate; + return (this); + } + + + + /******************************************************************************* + ** Getter for modifyDate. + *******************************************************************************/ + public Instant getModifyDate() + { + return (modifyDate); + } + + + + /******************************************************************************* + ** Fluent setter for modifyDate. + *******************************************************************************/ + public FileGatewaySshKey withModifyDate(Instant modifyDate) + { + this.modifyDate = modifyDate; + return (this); + } + + + + /******************************************************************************* + ** Getter for accountId. + *******************************************************************************/ + public Integer getAccountId() + { + return (accountId); + } + + + + /******************************************************************************* + ** Fluent setter for accountId. + *******************************************************************************/ + public FileGatewaySshKey withAccountId(Integer accountId) + { + this.accountId = accountId; + return (this); + } + + + + /******************************************************************************* + ** Getter for label. + *******************************************************************************/ + public String getLabel() + { + return (label); + } + + + + /******************************************************************************* + ** Fluent setter for label. + *******************************************************************************/ + public FileGatewaySshKey withLabel(String label) + { + this.label = label; + return (this); + } + + + + /******************************************************************************* + ** Getter for publicKey. + *******************************************************************************/ + public String getPublicKey() + { + return (publicKey); + } + + + + /******************************************************************************* + ** Fluent setter for publicKey. + *******************************************************************************/ + public FileGatewaySshKey withPublicKey(String publicKey) + { + this.publicKey = publicKey; + return (this); + } + + + + /******************************************************************************* + ** Getter for fingerprint. + *******************************************************************************/ + public String getFingerprint() + { + return (fingerprint); + } + + + + /******************************************************************************* + ** Fluent setter for fingerprint. + *******************************************************************************/ + public FileGatewaySshKey withFingerprint(String fingerprint) + { + this.fingerprint = fingerprint; + return (this); + } + + + + /******************************************************************************* + ** Getter for keyType. + *******************************************************************************/ + public String getKeyType() + { + return (keyType); + } + + + + /******************************************************************************* + ** Fluent setter for keyType. + *******************************************************************************/ + public FileGatewaySshKey withKeyType(String keyType) + { + this.keyType = keyType; + return (this); + } + + + + /******************************************************************************* + ** Getter for keyBits. + *******************************************************************************/ + public Integer getKeyBits() + { + return (keyBits); + } + + + + /******************************************************************************* + ** Fluent setter for keyBits. + *******************************************************************************/ + public FileGatewaySshKey withKeyBits(Integer keyBits) + { + this.keyBits = keyBits; + return (this); + } + + + + /******************************************************************************* + ** Getter for expiresAt. + *******************************************************************************/ + public Instant getExpiresAt() + { + return (expiresAt); + } + + + + /******************************************************************************* + ** Fluent setter for expiresAt. + *******************************************************************************/ + public FileGatewaySshKey withExpiresAt(Instant expiresAt) + { + this.expiresAt = expiresAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for revokedAt. + *******************************************************************************/ + public Instant getRevokedAt() + { + return (revokedAt); + } + + + + /******************************************************************************* + ** Fluent setter for revokedAt. + *******************************************************************************/ + public FileGatewaySshKey withRevokedAt(Instant revokedAt) + { + this.revokedAt = revokedAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for lastUsedAt. + *******************************************************************************/ + public Instant getLastUsedAt() + { + return (lastUsedAt); + } + + + + /******************************************************************************* + ** Fluent setter for lastUsedAt. + *******************************************************************************/ + public FileGatewaySshKey withLastUsedAt(Instant lastUsedAt) + { + this.lastUsedAt = lastUsedAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for lastUsedIp. + *******************************************************************************/ + public String getLastUsedIp() + { + return (lastUsedIp); + } + + + + /******************************************************************************* + ** Fluent setter for lastUsedIp. + *******************************************************************************/ + public FileGatewaySshKey withLastUsedIp(String lastUsedIp) + { + this.lastUsedIp = lastUsedIp; + return (this); + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setId(Integer id) + { + this.id = id; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setCreateDate(Instant createDate) + { + this.createDate = createDate; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setModifyDate(Instant modifyDate) + { + this.modifyDate = modifyDate; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setAccountId(Integer accountId) + { + this.accountId = accountId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setLabel(String label) + { + this.label = label; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setPublicKey(String publicKey) + { + this.publicKey = publicKey; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setFingerprint(String fingerprint) + { + this.fingerprint = fingerprint; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setKeyType(String keyType) + { + this.keyType = keyType; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setKeyBits(Integer keyBits) + { + this.keyBits = keyBits; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setExpiresAt(Instant expiresAt) + { + this.expiresAt = expiresAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setRevokedAt(Instant revokedAt) + { + this.revokedAt = revokedAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setLastUsedAt(Instant lastUsedAt) + { + this.lastUsedAt = lastUsedAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setLastUsedIp(String lastUsedIp) + { + this.lastUsedIp = lastUsedIp; + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayTransferLog.java b/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayTransferLog.java new file mode 100644 index 0000000..2934850 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayTransferLog.java @@ -0,0 +1,634 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.model; + + +import java.time.Instant; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.data.QField; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.data.QRecordEntity; + + +/******************************************************************************* + ** Record entity for transfers. + *******************************************************************************/ +public class FileGatewayTransferLog extends QRecordEntity +{ + public static final String TABLE_NAME = "file_gateway_transfer_log"; + + @QField + private Integer id; + + @QField + private Instant createDate; + + @QField + private Instant modifyDate; + + @QField + private Integer accountId; + + @QField + private Integer connectionLogId; + + @QField + private String operation; + + @QField + private String path; + + @QField + private String objectKind; + + @QField + private String objectId; + + @QField + private String format; + + @QField + private Long bytes; + + @QField + private String sha256; + + @QField + private Long durationMs; + + @QField + private String outcome; + + @QField + private String message; + + @QField + private String clientChannel; + + @QField + private Integer uploadJobId; + + + + /******************************************************************************* + ** Create an empty entity. + *******************************************************************************/ + public FileGatewayTransferLog() + { + } + + + + /******************************************************************************* + ** Populate from a database record. + *******************************************************************************/ + public FileGatewayTransferLog(QRecord record) throws QException + { + populateFromQRecord(record); + } + + + + /******************************************************************************* + ** Getter for id. + *******************************************************************************/ + public Integer getId() + { + return (id); + } + + + + /******************************************************************************* + ** Fluent setter for id. + *******************************************************************************/ + public FileGatewayTransferLog withId(Integer id) + { + this.id = id; + return (this); + } + + + + /******************************************************************************* + ** Getter for createDate. + *******************************************************************************/ + public Instant getCreateDate() + { + return (createDate); + } + + + + /******************************************************************************* + ** Fluent setter for createDate. + *******************************************************************************/ + public FileGatewayTransferLog withCreateDate(Instant createDate) + { + this.createDate = createDate; + return (this); + } + + + + /******************************************************************************* + ** Getter for modifyDate. + *******************************************************************************/ + public Instant getModifyDate() + { + return (modifyDate); + } + + + + /******************************************************************************* + ** Fluent setter for modifyDate. + *******************************************************************************/ + public FileGatewayTransferLog withModifyDate(Instant modifyDate) + { + this.modifyDate = modifyDate; + return (this); + } + + + + /******************************************************************************* + ** Getter for accountId. + *******************************************************************************/ + public Integer getAccountId() + { + return (accountId); + } + + + + /******************************************************************************* + ** Fluent setter for accountId. + *******************************************************************************/ + public FileGatewayTransferLog withAccountId(Integer accountId) + { + this.accountId = accountId; + return (this); + } + + + + /******************************************************************************* + ** Getter for connectionLogId. + *******************************************************************************/ + public Integer getConnectionLogId() + { + return (connectionLogId); + } + + + + /******************************************************************************* + ** Fluent setter for connectionLogId. + *******************************************************************************/ + public FileGatewayTransferLog withConnectionLogId(Integer connectionLogId) + { + this.connectionLogId = connectionLogId; + return (this); + } + + + + /******************************************************************************* + ** Getter for operation. + *******************************************************************************/ + public String getOperation() + { + return (operation); + } + + + + /******************************************************************************* + ** Fluent setter for operation. + *******************************************************************************/ + public FileGatewayTransferLog withOperation(String operation) + { + this.operation = operation; + return (this); + } + + + + /******************************************************************************* + ** Getter for path. + *******************************************************************************/ + public String getPath() + { + return (path); + } + + + + /******************************************************************************* + ** Fluent setter for path. + *******************************************************************************/ + public FileGatewayTransferLog withPath(String path) + { + this.path = path; + return (this); + } + + + + /******************************************************************************* + ** Getter for objectKind. + *******************************************************************************/ + public String getObjectKind() + { + return (objectKind); + } + + + + /******************************************************************************* + ** Fluent setter for objectKind. + *******************************************************************************/ + public FileGatewayTransferLog withObjectKind(String objectKind) + { + this.objectKind = objectKind; + return (this); + } + + + + /******************************************************************************* + ** Getter for objectId. + *******************************************************************************/ + public String getObjectId() + { + return (objectId); + } + + + + /******************************************************************************* + ** Fluent setter for objectId. + *******************************************************************************/ + public FileGatewayTransferLog withObjectId(String objectId) + { + this.objectId = objectId; + return (this); + } + + + + /******************************************************************************* + ** Getter for format. + *******************************************************************************/ + public String getFormat() + { + return (format); + } + + + + /******************************************************************************* + ** Fluent setter for format. + *******************************************************************************/ + public FileGatewayTransferLog withFormat(String format) + { + this.format = format; + return (this); + } + + + + /******************************************************************************* + ** Getter for bytes. + *******************************************************************************/ + public Long getBytes() + { + return (bytes); + } + + + + /******************************************************************************* + ** Fluent setter for bytes. + *******************************************************************************/ + public FileGatewayTransferLog withBytes(Long bytes) + { + this.bytes = bytes; + return (this); + } + + + + /******************************************************************************* + ** Getter for sha256. + *******************************************************************************/ + public String getSha256() + { + return (sha256); + } + + + + /******************************************************************************* + ** Fluent setter for sha256. + *******************************************************************************/ + public FileGatewayTransferLog withSha256(String sha256) + { + this.sha256 = sha256; + return (this); + } + + + + /******************************************************************************* + ** Getter for durationMs. + *******************************************************************************/ + public Long getDurationMs() + { + return (durationMs); + } + + + + /******************************************************************************* + ** Fluent setter for durationMs. + *******************************************************************************/ + public FileGatewayTransferLog withDurationMs(Long durationMs) + { + this.durationMs = durationMs; + return (this); + } + + + + /******************************************************************************* + ** Getter for outcome. + *******************************************************************************/ + public String getOutcome() + { + return (outcome); + } + + + + /******************************************************************************* + ** Fluent setter for outcome. + *******************************************************************************/ + public FileGatewayTransferLog withOutcome(String outcome) + { + this.outcome = outcome; + return (this); + } + + + + /******************************************************************************* + ** Getter for message. + *******************************************************************************/ + public String getMessage() + { + return (message); + } + + + + /******************************************************************************* + ** Fluent setter for message. + *******************************************************************************/ + public FileGatewayTransferLog withMessage(String message) + { + this.message = message; + return (this); + } + + + + /******************************************************************************* + ** Getter for clientChannel. + *******************************************************************************/ + public String getClientChannel() + { + return (clientChannel); + } + + + + /******************************************************************************* + ** Fluent setter for clientChannel. + *******************************************************************************/ + public FileGatewayTransferLog withClientChannel(String clientChannel) + { + this.clientChannel = clientChannel; + return (this); + } + + + + /******************************************************************************* + ** Getter for uploadJobId. + *******************************************************************************/ + public Integer getUploadJobId() + { + return (uploadJobId); + } + + + + /******************************************************************************* + ** Fluent setter for uploadJobId. + *******************************************************************************/ + public FileGatewayTransferLog withUploadJobId(Integer uploadJobId) + { + this.uploadJobId = uploadJobId; + return (this); + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setId(Integer id) + { + this.id = id; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setCreateDate(Instant createDate) + { + this.createDate = createDate; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setModifyDate(Instant modifyDate) + { + this.modifyDate = modifyDate; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setAccountId(Integer accountId) + { + this.accountId = accountId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setConnectionLogId(Integer connectionLogId) + { + this.connectionLogId = connectionLogId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setOperation(String operation) + { + this.operation = operation; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setPath(String path) + { + this.path = path; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setObjectKind(String objectKind) + { + this.objectKind = objectKind; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setObjectId(String objectId) + { + this.objectId = objectId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setFormat(String format) + { + this.format = format; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setBytes(Long bytes) + { + this.bytes = bytes; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setSha256(String sha256) + { + this.sha256 = sha256; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setDurationMs(Long durationMs) + { + this.durationMs = durationMs; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setOutcome(String outcome) + { + this.outcome = outcome; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setMessage(String message) + { + this.message = message; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setClientChannel(String clientChannel) + { + this.clientChannel = clientChannel; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setUploadJobId(Integer uploadJobId) + { + this.uploadJobId = uploadJobId; + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayUploadJob.java b/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayUploadJob.java new file mode 100644 index 0000000..b196d89 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/model/FileGatewayUploadJob.java @@ -0,0 +1,1276 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.model; + + +import java.time.Instant; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.data.QField; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.data.QRecordEntity; + + +/******************************************************************************* + ** Record entity for upload jobs. + *******************************************************************************/ +public class FileGatewayUploadJob extends QRecordEntity +{ + public static final String TABLE_NAME = "file_gateway_upload_job"; + + @QField + private Integer id; + + @QField + private Instant createDate; + + @QField + private Instant modifyDate; + + @QField + private Integer accountId; + + @QField + private Integer aliasId; + + @QField + private Integer connectionLogId; + + @QField + private String remoteAddress; + + @QField + private String tableName; + + @QField + private Integer savedBulkLoadProfileId; + + @QField + private String consumerName; + + @QField + private String consumerReference; + + @QField + private String path; + + @QField + private String fileName; + + @QField + private Long sizeBytes; + + @QField + private String sha256; + + @QField + private String storageKey; + + @QField + private String status; + + @QField + private Integer duplicateOfJobId; + + @QField + private String processUuid; + + @QField + private Integer insertedCount; + + @QField + private Integer errorCount; + + @QField + private String resultSummary; + + @QField + private String resultsStorageKey; + + @QField + private Long resultsSizeBytes; + + @QField + private String errorMessage; + + @QField + private Integer attempts; + + @QField + private Instant nextAttemptAt; + + @QField + private String claimedBy; + + @QField + private Instant claimedAt; + + @QField + private Instant queuedAt; + + @QField + private Instant startedAt; + + @QField + private Instant finishedAt; + + @QField + private Instant acknowledgedAt; + + + + @QField + private String plaintextStorageKey; + + @QField + private String plaintextSha256; + + @QField + private Long plaintextSizeBytes; + + + /******************************************************************************* + ** Create an empty entity. + *******************************************************************************/ + public FileGatewayUploadJob() + { + } + + + + /******************************************************************************* + ** Populate from a database record. + *******************************************************************************/ + public FileGatewayUploadJob(QRecord record) throws QException + { + populateFromQRecord(record); + } + + + + /******************************************************************************* + ** Getter for id. + *******************************************************************************/ + public Integer getId() + { + return (id); + } + + + + /******************************************************************************* + ** Fluent setter for id. + *******************************************************************************/ + public FileGatewayUploadJob withId(Integer id) + { + this.id = id; + return (this); + } + + + + /******************************************************************************* + ** Getter for createDate. + *******************************************************************************/ + public Instant getCreateDate() + { + return (createDate); + } + + + + /******************************************************************************* + ** Fluent setter for createDate. + *******************************************************************************/ + public FileGatewayUploadJob withCreateDate(Instant createDate) + { + this.createDate = createDate; + return (this); + } + + + + /******************************************************************************* + ** Getter for modifyDate. + *******************************************************************************/ + public Instant getModifyDate() + { + return (modifyDate); + } + + + + /******************************************************************************* + ** Fluent setter for modifyDate. + *******************************************************************************/ + public FileGatewayUploadJob withModifyDate(Instant modifyDate) + { + this.modifyDate = modifyDate; + return (this); + } + + + + /******************************************************************************* + ** Getter for accountId. + *******************************************************************************/ + public Integer getAccountId() + { + return (accountId); + } + + + + /******************************************************************************* + ** Fluent setter for accountId. + *******************************************************************************/ + public FileGatewayUploadJob withAccountId(Integer accountId) + { + this.accountId = accountId; + return (this); + } + + + + /******************************************************************************* + ** Getter for aliasId. + *******************************************************************************/ + public Integer getAliasId() + { + return (aliasId); + } + + + + /******************************************************************************* + ** Fluent setter for aliasId. + *******************************************************************************/ + public FileGatewayUploadJob withAliasId(Integer aliasId) + { + this.aliasId = aliasId; + return (this); + } + + + + /******************************************************************************* + ** Getter for connectionLogId. + *******************************************************************************/ + public Integer getConnectionLogId() + { + return (connectionLogId); + } + + + + /******************************************************************************* + ** Fluent setter for connectionLogId. + *******************************************************************************/ + public FileGatewayUploadJob withConnectionLogId(Integer connectionLogId) + { + this.connectionLogId = connectionLogId; + return (this); + } + + + + /******************************************************************************* + ** Getter for remoteAddress. + *******************************************************************************/ + public String getRemoteAddress() + { + return (remoteAddress); + } + + + + /******************************************************************************* + ** Fluent setter for remoteAddress. + *******************************************************************************/ + public FileGatewayUploadJob withRemoteAddress(String remoteAddress) + { + this.remoteAddress = remoteAddress; + return (this); + } + + + + /******************************************************************************* + ** Getter for tableName. + *******************************************************************************/ + public String getTableName() + { + return (tableName); + } + + + + /******************************************************************************* + ** Fluent setter for tableName. + *******************************************************************************/ + public FileGatewayUploadJob withTableName(String tableName) + { + this.tableName = tableName; + return (this); + } + + + + /******************************************************************************* + ** Getter for savedBulkLoadProfileId. + *******************************************************************************/ + public Integer getSavedBulkLoadProfileId() + { + return (savedBulkLoadProfileId); + } + + + + /******************************************************************************* + ** Fluent setter for savedBulkLoadProfileId. + *******************************************************************************/ + public FileGatewayUploadJob withSavedBulkLoadProfileId(Integer savedBulkLoadProfileId) + { + this.savedBulkLoadProfileId = savedBulkLoadProfileId; + return (this); + } + + + + /******************************************************************************* + ** Getter for consumerName. + *******************************************************************************/ + public String getConsumerName() + { + return (consumerName); + } + + + + /******************************************************************************* + ** Fluent setter for consumerName. + *******************************************************************************/ + public FileGatewayUploadJob withConsumerName(String consumerName) + { + this.consumerName = consumerName; + return (this); + } + + + + /******************************************************************************* + ** Getter for consumerReference. + *******************************************************************************/ + public String getConsumerReference() + { + return (consumerReference); + } + + + + /******************************************************************************* + ** Fluent setter for consumerReference. + *******************************************************************************/ + public FileGatewayUploadJob withConsumerReference(String consumerReference) + { + this.consumerReference = consumerReference; + return (this); + } + + + + /******************************************************************************* + ** Getter for path. + *******************************************************************************/ + public String getPath() + { + return (path); + } + + + + /******************************************************************************* + ** Fluent setter for path. + *******************************************************************************/ + public FileGatewayUploadJob withPath(String path) + { + this.path = path; + return (this); + } + + + + /******************************************************************************* + ** Getter for fileName. + *******************************************************************************/ + public String getFileName() + { + return (fileName); + } + + + + /******************************************************************************* + ** Fluent setter for fileName. + *******************************************************************************/ + public FileGatewayUploadJob withFileName(String fileName) + { + this.fileName = fileName; + return (this); + } + + + + /******************************************************************************* + ** Getter for sizeBytes. + *******************************************************************************/ + public Long getSizeBytes() + { + return (sizeBytes); + } + + + + /******************************************************************************* + ** Fluent setter for sizeBytes. + *******************************************************************************/ + public FileGatewayUploadJob withSizeBytes(Long sizeBytes) + { + this.sizeBytes = sizeBytes; + return (this); + } + + + + /******************************************************************************* + ** Getter for sha256. + *******************************************************************************/ + public String getSha256() + { + return (sha256); + } + + + + /******************************************************************************* + ** Fluent setter for sha256. + *******************************************************************************/ + public FileGatewayUploadJob withSha256(String sha256) + { + this.sha256 = sha256; + return (this); + } + + + + /******************************************************************************* + ** Getter for storageKey. + *******************************************************************************/ + public String getStorageKey() + { + return (storageKey); + } + + + + /******************************************************************************* + ** Fluent setter for storageKey. + *******************************************************************************/ + public FileGatewayUploadJob withStorageKey(String storageKey) + { + this.storageKey = storageKey; + return (this); + } + + + + /******************************************************************************* + ** Getter for status. + *******************************************************************************/ + public String getStatus() + { + return (status); + } + + + + /******************************************************************************* + ** Fluent setter for status. + *******************************************************************************/ + public FileGatewayUploadJob withStatus(String status) + { + this.status = status; + return (this); + } + + + + /******************************************************************************* + ** Getter for duplicateOfJobId. + *******************************************************************************/ + public Integer getDuplicateOfJobId() + { + return (duplicateOfJobId); + } + + + + /******************************************************************************* + ** Fluent setter for duplicateOfJobId. + *******************************************************************************/ + public FileGatewayUploadJob withDuplicateOfJobId(Integer duplicateOfJobId) + { + this.duplicateOfJobId = duplicateOfJobId; + return (this); + } + + + + /******************************************************************************* + ** Getter for processUuid. + *******************************************************************************/ + public String getProcessUuid() + { + return (processUuid); + } + + + + /******************************************************************************* + ** Fluent setter for processUuid. + *******************************************************************************/ + public FileGatewayUploadJob withProcessUuid(String processUuid) + { + this.processUuid = processUuid; + return (this); + } + + + + /******************************************************************************* + ** Getter for insertedCount. + *******************************************************************************/ + public Integer getInsertedCount() + { + return (insertedCount); + } + + + + /******************************************************************************* + ** Fluent setter for insertedCount. + *******************************************************************************/ + public FileGatewayUploadJob withInsertedCount(Integer insertedCount) + { + this.insertedCount = insertedCount; + return (this); + } + + + + /******************************************************************************* + ** Getter for errorCount. + *******************************************************************************/ + public Integer getErrorCount() + { + return (errorCount); + } + + + + /******************************************************************************* + ** Fluent setter for errorCount. + *******************************************************************************/ + public FileGatewayUploadJob withErrorCount(Integer errorCount) + { + this.errorCount = errorCount; + return (this); + } + + + + /******************************************************************************* + ** Getter for resultSummary. + *******************************************************************************/ + public String getResultSummary() + { + return (resultSummary); + } + + + + /******************************************************************************* + ** Fluent setter for resultSummary. + *******************************************************************************/ + public FileGatewayUploadJob withResultSummary(String resultSummary) + { + this.resultSummary = resultSummary; + return (this); + } + + + + /******************************************************************************* + ** Getter for resultsStorageKey. + *******************************************************************************/ + public String getResultsStorageKey() + { + return (resultsStorageKey); + } + + + + /******************************************************************************* + ** Fluent setter for resultsStorageKey. + *******************************************************************************/ + public FileGatewayUploadJob withResultsStorageKey(String resultsStorageKey) + { + this.resultsStorageKey = resultsStorageKey; + return (this); + } + + + + /******************************************************************************* + ** Getter for errorMessage. + *******************************************************************************/ + public String getErrorMessage() + { + return (errorMessage); + } + + + + /******************************************************************************* + ** Fluent setter for errorMessage. + *******************************************************************************/ + public FileGatewayUploadJob withErrorMessage(String errorMessage) + { + this.errorMessage = errorMessage; + return (this); + } + + + + /******************************************************************************* + ** Getter for attempts. + *******************************************************************************/ + public Integer getAttempts() + { + return (attempts); + } + + + + /******************************************************************************* + ** Fluent setter for attempts. + *******************************************************************************/ + public FileGatewayUploadJob withAttempts(Integer attempts) + { + this.attempts = attempts; + return (this); + } + + + + /******************************************************************************* + ** Getter for nextAttemptAt. + *******************************************************************************/ + public Instant getNextAttemptAt() + { + return (nextAttemptAt); + } + + + + /******************************************************************************* + ** Fluent setter for nextAttemptAt. + *******************************************************************************/ + public FileGatewayUploadJob withNextAttemptAt(Instant nextAttemptAt) + { + this.nextAttemptAt = nextAttemptAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for claimedBy. + *******************************************************************************/ + public String getClaimedBy() + { + return (claimedBy); + } + + + + /******************************************************************************* + ** Fluent setter for claimedBy. + *******************************************************************************/ + public FileGatewayUploadJob withClaimedBy(String claimedBy) + { + this.claimedBy = claimedBy; + return (this); + } + + + + /******************************************************************************* + ** Getter for claimedAt. + *******************************************************************************/ + public Instant getClaimedAt() + { + return (claimedAt); + } + + + + /******************************************************************************* + ** Fluent setter for claimedAt. + *******************************************************************************/ + public FileGatewayUploadJob withClaimedAt(Instant claimedAt) + { + this.claimedAt = claimedAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for queuedAt. + *******************************************************************************/ + public Instant getQueuedAt() + { + return (queuedAt); + } + + + + /******************************************************************************* + ** Fluent setter for queuedAt. + *******************************************************************************/ + public FileGatewayUploadJob withQueuedAt(Instant queuedAt) + { + this.queuedAt = queuedAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for startedAt. + *******************************************************************************/ + public Instant getStartedAt() + { + return (startedAt); + } + + + + /******************************************************************************* + ** Fluent setter for startedAt. + *******************************************************************************/ + public FileGatewayUploadJob withStartedAt(Instant startedAt) + { + this.startedAt = startedAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for finishedAt. + *******************************************************************************/ + public Instant getFinishedAt() + { + return (finishedAt); + } + + + + /******************************************************************************* + ** Fluent setter for finishedAt. + *******************************************************************************/ + public FileGatewayUploadJob withFinishedAt(Instant finishedAt) + { + this.finishedAt = finishedAt; + return (this); + } + + + + /******************************************************************************* + ** Getter for acknowledgedAt. + *******************************************************************************/ + public Instant getAcknowledgedAt() + { + return (acknowledgedAt); + } + + + + /******************************************************************************* + ** Fluent setter for acknowledgedAt. + *******************************************************************************/ + public FileGatewayUploadJob withAcknowledgedAt(Instant acknowledgedAt) + { + this.acknowledgedAt = acknowledgedAt; + return (this); + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setId(Integer id) + { + this.id = id; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setCreateDate(Instant createDate) + { + this.createDate = createDate; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setModifyDate(Instant modifyDate) + { + this.modifyDate = modifyDate; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setAccountId(Integer accountId) + { + this.accountId = accountId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setAliasId(Integer aliasId) + { + this.aliasId = aliasId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setConnectionLogId(Integer connectionLogId) + { + this.connectionLogId = connectionLogId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setRemoteAddress(String remoteAddress) + { + this.remoteAddress = remoteAddress; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setTableName(String tableName) + { + this.tableName = tableName; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setSavedBulkLoadProfileId(Integer savedBulkLoadProfileId) + { + this.savedBulkLoadProfileId = savedBulkLoadProfileId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setConsumerName(String consumerName) + { + this.consumerName = consumerName; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setConsumerReference(String consumerReference) + { + this.consumerReference = consumerReference; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setPath(String path) + { + this.path = path; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setFileName(String fileName) + { + this.fileName = fileName; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setSizeBytes(Long sizeBytes) + { + this.sizeBytes = sizeBytes; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setSha256(String sha256) + { + this.sha256 = sha256; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setStorageKey(String storageKey) + { + this.storageKey = storageKey; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setStatus(String status) + { + this.status = status; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setDuplicateOfJobId(Integer duplicateOfJobId) + { + this.duplicateOfJobId = duplicateOfJobId; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setProcessUuid(String processUuid) + { + this.processUuid = processUuid; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setInsertedCount(Integer insertedCount) + { + this.insertedCount = insertedCount; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setErrorCount(Integer errorCount) + { + this.errorCount = errorCount; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setResultSummary(String resultSummary) + { + this.resultSummary = resultSummary; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setResultsStorageKey(String resultsStorageKey) + { + this.resultsStorageKey = resultsStorageKey; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setErrorMessage(String errorMessage) + { + this.errorMessage = errorMessage; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setAttempts(Integer attempts) + { + this.attempts = attempts; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setNextAttemptAt(Instant nextAttemptAt) + { + this.nextAttemptAt = nextAttemptAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setClaimedBy(String claimedBy) + { + this.claimedBy = claimedBy; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setClaimedAt(Instant claimedAt) + { + this.claimedAt = claimedAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setQueuedAt(Instant queuedAt) + { + this.queuedAt = queuedAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setStartedAt(Instant startedAt) + { + this.startedAt = startedAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setFinishedAt(Instant finishedAt) + { + this.finishedAt = finishedAt; + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setAcknowledgedAt(Instant acknowledgedAt) + { + this.acknowledgedAt = acknowledgedAt; + } + + /******************************************************************************* + ** Stored result artifact length. + *******************************************************************************/ + public Long getResultsSizeBytes() + { + return (resultsSizeBytes); + } + + /******************************************************************************* + ** Populate result artifact length. + *******************************************************************************/ + public void setResultsSizeBytes(Long value) + { + resultsSizeBytes = value; + } + + /******************************************************************************* + ** Fluent result artifact length. + *******************************************************************************/ + public FileGatewayUploadJob withResultsSizeBytes(Long value) + { + resultsSizeBytes = value; + return (this); + } + + + + + /******************************************************************************* + ** Getter for plaintextStorageKey. + *******************************************************************************/ + public String getPlaintextStorageKey() + { + return (plaintextStorageKey); + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setPlaintextStorageKey(String plaintextStorageKey) + { + this.plaintextStorageKey = plaintextStorageKey; + } + + + + /******************************************************************************* + ** Fluent setter for plaintextStorageKey. + *******************************************************************************/ + public FileGatewayUploadJob withPlaintextStorageKey(String plaintextStorageKey) + { + this.plaintextStorageKey = plaintextStorageKey; + return (this); + } + + + + /******************************************************************************* + ** Getter for plaintextSha256. + *******************************************************************************/ + public String getPlaintextSha256() + { + return (plaintextSha256); + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setPlaintextSha256(String plaintextSha256) + { + this.plaintextSha256 = plaintextSha256; + } + + + + /******************************************************************************* + ** Fluent setter for plaintextSha256. + *******************************************************************************/ + public FileGatewayUploadJob withPlaintextSha256(String plaintextSha256) + { + this.plaintextSha256 = plaintextSha256; + return (this); + } + + + + /******************************************************************************* + ** Getter for plaintextSizeBytes. + *******************************************************************************/ + public Long getPlaintextSizeBytes() + { + return (plaintextSizeBytes); + } + + + + /******************************************************************************* + ** Setter used by QRecordEntity population. + *******************************************************************************/ + public void setPlaintextSizeBytes(Long plaintextSizeBytes) + { + this.plaintextSizeBytes = plaintextSizeBytes; + } + + + + /******************************************************************************* + ** Fluent setter for plaintextSizeBytes. + *******************************************************************************/ + public FileGatewayUploadJob withPlaintextSizeBytes(Long plaintextSizeBytes) + { + this.plaintextSizeBytes = plaintextSizeBytes; + return (this); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/outbox/OutboxService.java b/src/main/java/com/kingsrook/qbits/filegateway/outbox/OutboxService.java new file mode 100644 index 0000000..7274bfc --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/outbox/OutboxService.java @@ -0,0 +1,244 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.outbox; + +import java.io.IOException; +import java.io.InputStream; +import java.time.Clock; +import java.util.List; +import java.util.UUID; +import com.kingsrook.qbits.filegateway.FileGatewayQBitConfig; +import com.kingsrook.qbits.filegateway.api.FileGatewayConsumer; +import com.kingsrook.qbits.filegateway.api.FileGatewayConsumersMetaData; +import com.kingsrook.qbits.filegateway.api.FileGatewayOutboxProvider; +import com.kingsrook.qbits.filegateway.api.OutboxFileRef; +import com.kingsrook.qbits.filegateway.metadata.AliasCustomizer; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qbits.filegateway.sftp.AuditWriter; +import com.kingsrook.qbits.filegateway.storage.GatewayObjectStore; +import com.kingsrook.qbits.filegateway.upload.UploadJobs; +import com.kingsrook.qqq.backend.core.actions.customizers.QCodeLoader; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.logging.QLogger; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; + +/******************************************************************************* + ** Database and object-storage outbox usable in web pods without an SFTP listener. + *******************************************************************************/ +public class OutboxService implements FileGatewayOutboxProvider +{ + private static final QLogger LOG = QLogger.getLogger(OutboxService.class); + private final QInstance instance; + private final AuditWriter audit; + private final GatewayObjectStore store; + private final FileGatewayQBitConfig config; + private final Clock clock; + + + + /******************************************************************************* + ** No runtime, threads or protocol credentials are required. + *******************************************************************************/ + public OutboxService(QInstance instance, GatewayObjectStore store, FileGatewayQBitConfig config, Clock clock) + { + this.instance = instance; + this.audit = new AuditWriter(instance); + this.store = store; + this.config = config; + this.clock = clock; + } + + + + /******************************************************************************* + ** Publish only after the object is complete and its caller-supplied size matches. + *******************************************************************************/ + @Override + public OutboxFileRef put(String endpointName, String fileName, InputStream content, long sizeBytes) throws QException + { + if(!AliasCustomizer.validFileName(fileName) || sizeBytes < 0) + { + throw (new QException("Invalid outbox filename or size")); + } + QRecord alias = endpoint(endpointName); + String key = "file-gateway/outbox/" + alias.getValue("id") + "/" + UUID.randomUUID() + "/" + fileName; + try + { + GatewayObjectStore.Receipt receipt = store.put(key, content); + if(receipt.size() != sizeBytes) + { + throw (new QException("Outbox stream size does not match sizeBytes")); + } + return (audit.withRowLock(FileGatewayTables.ALIAS, "id", alias.getValue("id"), () -> + { + QRecord current = endpoint(endpointName); + for(QRecord existing : list(current.getValueInteger("id"))) + { + if(fileName.equals(existing.getValueString("fileName"))) + { + throw (new QException("An active outbox file already has this filename")); + } + } + QRecord row = audit.insert(FileGatewayTables.OUTBOX_FILE, new QRecord().withValue("aliasId", current.getValue("id")) + .withValue("fileName", fileName).withValue("storageKey", key).withValue("sizeBytes", receipt.size()).withValue("sha256", receipt.sha256()) + .withValue("status", "READY").withValue("downloadCount", 0).withValue("onDownloadedPending", false) + .withValue("createdByConsumer", current.getValue("consumerName")).withValue("createDate", clock.instant()) + .withValue("expiresAt", clock.instant().plus(config.getOutboxRetention()))); + return (reference(row, current)); + })); + } + catch(Exception e) + { + try + { + store.delete(key); + } + catch(IOException cleanup) + { + e.addSuppressed(cleanup); + } + throw (new QException("Unable to publish outbox file: " + (e instanceof QException ? e.getMessage() : "storage failed"), e)); + } + } + + + + /******************************************************************************* + ** A missing, inactive or incorrectly typed endpoint cannot accept output. + *******************************************************************************/ + private QRecord endpoint(String name) throws QException + { + var rows = audit.query(FileGatewayTables.ALIAS, UploadJobs.equal("name", name)); + if(rows.isEmpty() || !Boolean.TRUE.equals(rows.getFirst().getValueBoolean("isActive")) || !"OUTBOX".equals(rows.getFirst().getValueString("kind"))) + { + throw (new QException("Outbox endpoint is missing, inactive or not OUTBOX")); + } + return (rows.getFirst()); + } + + + + /******************************************************************************* + ** Removed and expired files disappear immediately from the virtual folder. + *******************************************************************************/ + public List list(Integer aliasId) throws QException + { + return (audit.query(FileGatewayTables.OUTBOX_FILE, UploadJobs.equal("aliasId", aliasId)).stream() + .filter(row -> List.of("READY", "DOWNLOADED").contains(row.getValueString("status"))).toList()); + } + + + + /******************************************************************************* + ** Only a complete read schedules the durable, at-least-once notification. + *******************************************************************************/ + public void downloaded(QRecord row) throws QException + { + audit.withRowLock(FileGatewayTables.OUTBOX_FILE, "id", row.getValue("id"), () -> + { + QRecord current = audit.query(FileGatewayTables.OUTBOX_FILE, UploadJobs.equal("id", row.getValue("id"))).getFirst(); + if(List.of("READY", "DOWNLOADED").contains(current.getValueString("status"))) + { + QRecord changes = new QRecord().withValue("id", current.getValue("id")).withValue("status", "DOWNLOADED") + .withValue("downloadCount", current.getValueInteger("downloadCount") + 1); + if(current.getValue("firstDownloadedAt") == null) + { + changes.withValue("firstDownloadedAt", clock.instant()).withValue("onDownloadedPending", true); + } + audit.update(FileGatewayTables.OUTBOX_FILE, changes); + } + return (null); + }); + notifyDownloaded(row); + } + + + + /******************************************************************************* + ** Holding the row lock prevents duplicate concurrent delivery; crash retries remain possible. + *******************************************************************************/ + public void notifyDownloaded(QRecord row) throws QException + { + audit.withRowLock(FileGatewayTables.OUTBOX_FILE, "id", row.getValue("id"), () -> + { + QRecord current = audit.query(FileGatewayTables.OUTBOX_FILE, UploadJobs.equal("id", row.getValue("id"))).getFirst(); + if(!Boolean.TRUE.equals(current.getValueBoolean("onDownloadedPending"))) + { + return (null); + } + try + { + QRecord alias = audit.query(FileGatewayTables.ALIAS, UploadJobs.equal("id", current.getValue("aliasId"))).getFirst(); + String name = alias.getValueString("consumerName"); + if(name != null && !name.isBlank()) + { + FileGatewayConsumer selected = null; + for(var code : FileGatewayConsumersMetaData.ofOrWithNew(instance).getConsumers()) + { + FileGatewayConsumer consumer = QCodeLoader.getAdHoc(FileGatewayConsumer.class, code); + if(name.equals(consumer.getName())) + { + selected = consumer; + } + } + if(selected == null) + { + throw (new QException("Outbox consumer is not registered")); + } + selected.onDownloaded(reference(current, alias)); + } + audit.update(FileGatewayTables.OUTBOX_FILE, new QRecord().withValue("id", row.getValue("id")).withValue("onDownloadedPending", false)); + } + catch(Exception e) + { + LOG.warn("Outbox download notification failed; maintenance will retry"); + } + return (null); + }); + } + + + + /******************************************************************************* + ** Acknowledgment retains audit state until object retention runs. + *******************************************************************************/ + public void remove(QRecord row) throws QException + { + audit.withRowLock(FileGatewayTables.OUTBOX_FILE, "id", row.getValue("id"), () -> + { + audit.update(FileGatewayTables.OUTBOX_FILE, new QRecord().withValue("id", row.getValue("id")) + .withValue("status", "REMOVED").withValue("removedAt", clock.instant())); + return (null); + }); + } + + + + /******************************************************************************* + ** Consumer contract contains no implementation types or storage credentials. + *******************************************************************************/ + private OutboxFileRef reference(QRecord row, QRecord alias) + { + return (new OutboxFileRef(row.getValueInteger("id"), alias.getValueString("name"), alias.getValueString("path") + "/" + row.getValueString("fileName"), + row.getValueString("fileName"), row.getValueLong("sizeBytes"), row.getValueString("sha256"), row.getValueInstant("createDate"))); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/permissions/FileGatewayPermissionChecker.java b/src/main/java/com/kingsrook/qbits/filegateway/permissions/FileGatewayPermissionChecker.java new file mode 100644 index 0000000..b28729e --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/permissions/FileGatewayPermissionChecker.java @@ -0,0 +1,63 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.permissions; + + +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException; +import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules; +import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.core.model.session.QSession; + + +/******************************************************************************* + ** Read with view or admin; only administrators write configuration tables. + *******************************************************************************/ +public class FileGatewayPermissionChecker implements CustomPermissionChecker +{ + + + + /******************************************************************************* + ** Check the current user against the gateway permission model. + *******************************************************************************/ + @Override + public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException + { + boolean write = metadata instanceof QProcessMetaData || input instanceof InsertInput || input instanceof UpdateInput || input instanceof DeleteInput; + if(write && metadata instanceof QTableMetaData table && FileGatewayTables.SYSTEM_WRITTEN.contains(table.getName())) + { + throw (new QPermissionDeniedException("This table is written only by the file gateway")); + } + QSession session = QContext.getQSession(); + if(session == null || !(session.hasPermission(FileGatewayPermissions.ADMIN) || (!write && session.hasPermission(FileGatewayPermissions.VIEW)))) + { + throw (new QPermissionDeniedException("Missing file gateway permission")); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/permissions/FileGatewayPermissions.java b/src/main/java/com/kingsrook/qbits/filegateway/permissions/FileGatewayPermissions.java new file mode 100644 index 0000000..2e6dacf --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/permissions/FileGatewayPermissions.java @@ -0,0 +1,45 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.permissions; + + +import java.util.List; + + +/******************************************************************************* + ** Permissions granted by the host application. + *******************************************************************************/ +public final class FileGatewayPermissions +{ + public static final String VIEW = "file_gateway.view"; + public static final String ADMIN = "file_gateway.admin"; + public static final String CONNECT = "file_gateway.connect"; + public static final List ALL = List.of(VIEW, ADMIN, CONNECT); + + + + /******************************************************************************* + ** Constants only. + *******************************************************************************/ + private FileGatewayPermissions() + { + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/pgp/PgpStreams.java b/src/main/java/com/kingsrook/qbits/filegateway/pgp/PgpStreams.java new file mode 100644 index 0000000..e3b759a --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/pgp/PgpStreams.java @@ -0,0 +1,284 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.pgp; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.io.FilterInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.nio.charset.StandardCharsets; +import java.security.SecureRandom; +import java.util.Date; +import org.bouncycastle.bcpg.SymmetricKeyAlgorithmTags; +import org.bouncycastle.openpgp.PGPCompressedData; +import org.bouncycastle.openpgp.PGPEncryptedDataGenerator; +import org.bouncycastle.openpgp.PGPEncryptedDataList; +import org.bouncycastle.openpgp.PGPException; +import org.bouncycastle.openpgp.PGPLiteralData; +import org.bouncycastle.openpgp.PGPLiteralDataGenerator; +import org.bouncycastle.openpgp.PGPObjectFactory; +import org.bouncycastle.openpgp.PGPPublicKey; +import org.bouncycastle.openpgp.PGPPublicKeyEncryptedData; +import org.bouncycastle.openpgp.PGPPublicKeyRingCollection; +import org.bouncycastle.openpgp.PGPSecretKeyRingCollection; +import org.bouncycastle.openpgp.PGPUtil; +import org.bouncycastle.openpgp.operator.bc.BcKeyFingerprintCalculator; +import org.bouncycastle.openpgp.operator.bc.BcPBESecretKeyDecryptorBuilder; +import org.bouncycastle.openpgp.operator.bc.BcPGPDataEncryptorBuilder; +import org.bouncycastle.openpgp.operator.bc.BcPGPDigestCalculatorProvider; +import org.bouncycastle.openpgp.operator.bc.BcPublicKeyDataDecryptorFactory; +import org.bouncycastle.openpgp.operator.bc.BcPublicKeyKeyEncryptionMethodGenerator; + +/******************************************************************************* + ** Bounded, in-memory OpenPGP streams; no key files, providers or background threads. + *******************************************************************************/ +public final class PgpStreams +{ + /******************************************************************************* + ** Static stream factories only. + *******************************************************************************/ + private PgpStreams() + { + } + + + + /******************************************************************************* + ** Require a usable encryption subkey, rather than accepting signing-only keys. + *******************************************************************************/ + public static PGPPublicKey encryptionKey(String text) throws IOException, PGPException + { + if(text == null || text.isBlank()) + { + throw (new PGPException("PGP public key is required")); + } + var rings = new PGPPublicKeyRingCollection(PGPUtil.getDecoderStream(new ByteArrayInputStream(text.getBytes(StandardCharsets.UTF_8))), new BcKeyFingerprintCalculator()); + for(var ring : rings) + { + for(var key : ring) + { + if(!key.isMasterKey() && key.isEncryptionKey() && !key.hasRevocation() + && (key.getValidSeconds() == 0 || key.getCreationTime().toInstant().plusSeconds(key.getValidSeconds()).isAfter(java.time.Instant.now()))) + { + return (key); + } + } + } + throw (new PGPException("PGP public key must contain an unrevoked, unexpired encryption subkey")); + } + + + + /******************************************************************************* + ** AES-256 binary packets with integrity protection and bounded partial packets. + *******************************************************************************/ + public static InputStream encrypt(InputStream source, String keyText, String filename) throws IOException + { + try + { + ByteArrayOutputStream pending = new ByteArrayOutputStream(131072); + var generator = new PGPEncryptedDataGenerator(new BcPGPDataEncryptorBuilder(SymmetricKeyAlgorithmTags.AES_256) + .setWithIntegrityPacket(true).setSecureRandom(new SecureRandom())); + generator.addMethod(new BcPublicKeyKeyEncryptionMethodGenerator(encryptionKey(keyText))); + OutputStream encrypted = generator.open(pending, new byte[32768]); + var literalGenerator = new PGPLiteralDataGenerator(); + OutputStream literal = literalGenerator.open(encrypted, PGPLiteralData.BINARY, filename, new Date(0), new byte[32768]); + return (new InputStream() + { + private InputStream available = InputStream.nullInputStream(); + private boolean finished; + private boolean closed; + + @Override + public int read() throws IOException + { + byte[] one = new byte[1]; + return (read(one, 0, 1) < 0 ? -1 : one[0] & 255); + } + + @Override + public int read(byte[] bytes, int offset, int length) throws IOException + { + java.util.Objects.checkFromIndexSize(offset, length, bytes.length); + if(closed) + { + throw (new IOException("PGP stream closed")); + } + if(length == 0) + { + return (0); + } + while(true) + { + int count = available.read(bytes, offset, length); + if(count >= 0) + { + return (count); + } + if(pending.size() > 0) + { + available = new ByteArrayInputStream(pending.toByteArray()); + pending.reset(); + } + else if(finished) + { + return (-1); + } + else + { + byte[] chunk = source.readNBytes(32768); + if(chunk.length == 0) + { + literal.close(); + encrypted.close(); + finished = true; + } + else + { + literal.write(chunk); + } + } + } + } + + @Override + public void close() throws IOException + { + closed = true; + source.close(); + } + }); + } + catch(PGPException e) + { + throw (new IOException("Unable to encrypt PGP download", e)); + } + } + + + + /******************************************************************************* + ** Verify integrity at EOF before object storage can complete its plaintext upload. + *******************************************************************************/ + public static InputStream decrypt(InputStream source, String secret, char[] passphrase, long maximum) throws IOException + { + try + { + if(secret == null) + { + throw (new PGPException("PGP private key environment variable is missing")); + } + var keys = new PGPSecretKeyRingCollection(PGPUtil.getDecoderStream(new ByteArrayInputStream(secret.getBytes(StandardCharsets.UTF_8))), new BcKeyFingerprintCalculator()); + var factory = new PGPObjectFactory(PGPUtil.getDecoderStream(source), new BcKeyFingerprintCalculator()); + Object packet = factory.nextObject(); + if(!(packet instanceof PGPEncryptedDataList)) + { + packet = factory.nextObject(); + } + if(!(packet instanceof PGPEncryptedDataList encrypted)) + { + throw (new PGPException("Expected PGP encrypted data")); + } + for(var item : encrypted) + { + if(!(item instanceof PGPPublicKeyEncryptedData data)) + { + continue; + } + var key = keys.getSecretKey(data.getKeyID()); + if(key == null) + { + continue; + } + var privateKey = key.extractPrivateKey(new BcPBESecretKeyDecryptorBuilder(new BcPGPDigestCalculatorProvider()).build(passphrase)); + InputStream clear = data.getDataStream(new BcPublicKeyDataDecryptorFactory(privateKey)); + Object message = new PGPObjectFactory(clear, new BcKeyFingerprintCalculator()).nextObject(); + for(int depth = 0; message instanceof PGPCompressedData compressed && depth < 4; depth++) + { + message = new PGPObjectFactory(compressed.getDataStream(), new BcKeyFingerprintCalculator()).nextObject(); + } + if(!(message instanceof PGPLiteralData literal) || !data.isIntegrityProtected()) + { + throw (new PGPException("Expected integrity-protected literal PGP content")); + } + return (new FilterInputStream(literal.getInputStream()) + { + private long consumed; + private boolean verified; + + @Override + public int read() throws IOException + { + byte[] one = new byte[1]; + return (read(one, 0, 1) < 0 ? -1 : one[0] & 255); + } + + @Override + public int read(byte[] bytes, int offset, int length) throws IOException + { + int count = in.read(bytes, offset, length); + if(count > 0 && (consumed += count) > maximum) + { + throw (new IOException("Decrypted upload exceeds file size limit")); + } + if(count < 0 && !verified) + { + try + { + clear.transferTo(OutputStream.nullOutputStream()); + if(!data.verify()) + { + throw (new IOException("PGP integrity check failed")); + } + verified = true; + } + catch(PGPException e) + { + throw (new IOException("PGP integrity check failed", e)); + } + } + return (count); + } + + @Override + public void close() throws IOException + { + try + { + super.close(); + } + finally + { + source.close(); + } + } + }); + } + throw (new PGPException("No matching PGP private key")); + } + catch(PGPException | RuntimeException e) + { + throw (new IOException("Unable to decrypt PGP upload", e)); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/sftp/AuditWriter.java b/src/main/java/com/kingsrook/qbits/filegateway/sftp/AuditWriter.java new file mode 100644 index 0000000..2bb59a0 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/sftp/AuditWriter.java @@ -0,0 +1,274 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.sftp; + + +import java.io.Serializable; +import java.time.Instant; +import java.util.List; +import java.util.Map; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qqq.backend.core.actions.QBackendTransaction; +import com.kingsrook.qqq.backend.core.actions.permissions.CorePermissionEnforcer; +import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; +import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; +import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession; + + +/******************************************************************************* + ** System-session access limited to the gateway own tables; application data uses the UI API. + *******************************************************************************/ +public class AuditWriter +{ + private final QInstance instance; + private final ThreadLocal transaction = new ThreadLocal<>(); + + + + /******************************************************************************* + ** Bind the gateway database instance. + *******************************************************************************/ + public AuditWriter(QInstance instance) + { + this.instance = instance; + } + + + /******************************************************************************* + ** Work within a short system-session scope. + *******************************************************************************/ + @FunctionalInterface + public interface Work + { + /******************************************************************************* + ** Execute the database operation. + *******************************************************************************/ + T run() throws QException; + } + + + + /******************************************************************************* + ** Preserve caller context while writing gateway-owned state. + *******************************************************************************/ + public T system(Work work) throws QException + { + var previous = QContext.capture(); + try + { + QContext.init(instance, new QSystemUserSession(instance)); + try(var ignored = CorePermissionEnforcer.openAuthorizedScope()) + { + return (work.run()); + } + } + finally + { + QContext.init(previous); + } + } + + + + /******************************************************************************* + ** Serialize account authentication across pods with a transactional row update. + ** Re-reading after acquiring the write lock sees committed counters and sessions. + *******************************************************************************/ + public T withAccountLock(String username, Work work) throws QException + { + return (withAccountLock("username", username, work)); + } + + + + /******************************************************************************* + ** Lock one account row without changing any user-managed fields. + *******************************************************************************/ + public T withAccountLock(String field, Serializable value, Work work) throws QException + { + return (withRowLock(FileGatewayTables.ACCOUNT, field, value, work)); + } + + + + /******************************************************************************* + ** Serialize transitions on one gateway-owned row across gateway and web pods. + *******************************************************************************/ + public T withRowLock(String table, String field, Serializable value, Work work) throws QException + { + checkTable(table); + return (system(() -> + { + try(var tx = QBackendTransaction.openFor(new QueryInput(table))) + { + transaction.set(tx); + try + { + var rows = query(table, new QQueryFilter(new QFilterCriteria(field, QCriteriaOperator.EQUALS, value))); + if(!rows.isEmpty()) + { + update(table, new QRecord().withValue("id", rows.getFirst().getValue("id")).withValue("modifyDate", Instant.now())); + } + T result = work.run(); + tx.commit(); + return (result); + } + catch(Exception e) + { + tx.rollback(); + throw (new QException("Gateway account transaction failed", e)); + } + finally + { + transaction.remove(); + } + } + })); + } + + + + /******************************************************************************* + ** Charge an account-wide minute window atomically, including other gateway pods. + *******************************************************************************/ + public boolean operationAllowed(Integer accountId, Instant now, int maximum) throws QException + { + return (withAccountLock("id", accountId, () -> + { + var rows = query(FileGatewayTables.ACCOUNT, new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.EQUALS, accountId))); + if(rows.isEmpty()) + { + throw (new QException("Gateway account no longer exists")); + } + QRecord account = rows.getFirst(); + Instant window = account.getValueInstant("operationWindowAt"); + Integer previous = account.getValueInteger("operationWindowCount"); + boolean reset = window == null || !window.plusSeconds(60).isAfter(now); + int count = reset ? 1 : (previous == null ? 0 : previous) + 1; + update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", accountId) + .withValue("operationWindowAt", reset ? now : window).withValue("operationWindowCount", count)); + return (count <= maximum); + })); + } + + + + /******************************************************************************* + ** Reject accidental use on application data. + *******************************************************************************/ + private void checkTable(String table) + { + if(!FileGatewayTables.ALL.contains(table)) + { + throw (new IllegalArgumentException("System access is limited to gateway tables")); + } + } + + + + /******************************************************************************* + ** Read gateway state. + *******************************************************************************/ + public List query(String table, QQueryFilter filter) throws QException + { + checkTable(table); + return (system(() -> new QueryAction().execute(new QueryInput(table).withFilter(filter).withTransaction(transaction.get())).getRecords())); + } + + + + /******************************************************************************* + ** Insert and require successful persistence. + *******************************************************************************/ + public QRecord insert(String table, QRecord record) throws QException + { + checkTable(table); + QRecord result = system(() -> new InsertAction().execute(new InsertInput(table).withRecord(record).withTransaction(transaction.get())).getRecords().get(0)); + checkErrors(result); + return (result); + } + + + + /******************************************************************************* + ** Update gateway state. + *******************************************************************************/ + public void update(String table, QRecord record) throws QException + { + checkTable(table); + QRecord result = system(() -> new UpdateAction().execute(new UpdateInput(table).withRecords(List.of(record)).withTransaction(transaction.get())).getRecords().get(0)); + checkErrors(result); + } + + + + /******************************************************************************* + ** Retention deletes only gateway-owned rows under the bookkeeping session. + *******************************************************************************/ + public void delete(String table, Serializable id) throws QException + { + checkTable(table); + system(() -> new com.kingsrook.qqq.backend.core.actions.tables.DeleteAction().execute( + new com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput(table).withPrimaryKey(id).withTransaction(transaction.get()))); + } + + + + /******************************************************************************* + ** Optimistic status transition, called under withAccountLock because core checks + ** expected versions before SQL execution rather than in the SQL WHERE clause. + *******************************************************************************/ + public boolean transitionUpload(QRecord changes, String expectedStatus) throws QException + { + QRecord result = system(() -> new UpdateAction().execute(new UpdateInput(FileGatewayTables.UPLOAD_JOB) + .withRecords(List.of(changes)).withTransaction(transaction.get()) + .withExpectedVersions(Map.of(changes.getValue("id"), expectedStatus))).getRecords().getFirst()); + if(result.getErrors() != null && !result.getErrors().isEmpty()) + { + return (false); + } + return (true); + } + + + + /******************************************************************************* + ** Fail closed on bookkeeping errors without printing records. + *******************************************************************************/ + private void checkErrors(QRecord record) throws QException + { + if(record.getErrors() != null && !record.getErrors().isEmpty()) + { + throw (new QException("Unable to persist gateway state")); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/sftp/CidrMatcher.java b/src/main/java/com/kingsrook/qbits/filegateway/sftp/CidrMatcher.java new file mode 100644 index 0000000..19803ab --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/sftp/CidrMatcher.java @@ -0,0 +1,105 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.sftp; + + +import java.net.InetAddress; + + +/******************************************************************************* + ** Numeric IPv4 and IPv6 CIDR matching; hostnames never trigger DNS resolution. + *******************************************************************************/ +public final class CidrMatcher +{ + + + + /******************************************************************************* + ** Utility only. + *******************************************************************************/ + private CidrMatcher() + { + } + + + + /******************************************************************************* + ** Match an address against comma or whitespace separated networks. + *******************************************************************************/ + public static boolean matches(String address, String cidrs) + { + if(cidrs == null || cidrs.isBlank()) + { + return (true); + } + try + { + byte[] candidate = numeric(address); + for(String cidr : cidrs.split("[,\\s]+")) + { + String[] parts = cidr.split("/", -1); + byte[] network = numeric(parts[0]); + int bits = parts.length == 1 ? network.length * 8 : Integer.parseInt(parts[1]); + if(parts.length > 2 || bits < 0 || bits > network.length * 8) + { + return (false); + } + if(candidate.length != network.length) + { + continue; + } + boolean match = true; + for(int i = 0; i < bits; i++) + { + int mask = 1 << (7 - i % 8); + if((candidate[i / 8] & mask) != (network[i / 8] & mask)) + { + match = false; + break; + } + } + if(match) + { + return (true); + } + } + } + catch(Exception e) + { + return (false); + } + return (false); + } + + + + /******************************************************************************* + ** Parse numeric addresses only. + *******************************************************************************/ + private static byte[] numeric(String address) throws Exception + { + if(address == null || !address.matches("[0-9a-fA-F:.]+") || (!address.contains(":") && !address.matches("[0-9]{1,3}(\\.[0-9]{1,3}){3}"))) + { + throw (new IllegalArgumentException("Expected a numeric IP address")); + } + return (InetAddress.getByName(address).getAddress()); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/sftp/FileGatewayAuthenticatorInterface.java b/src/main/java/com/kingsrook/qbits/filegateway/sftp/FileGatewayAuthenticatorInterface.java new file mode 100644 index 0000000..f548bce --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/sftp/FileGatewayAuthenticatorInterface.java @@ -0,0 +1,37 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.sftp; + + +import java.security.PublicKey; +import org.apache.sshd.server.session.ServerSession; + + +/******************************************************************************* + ** Protocol authentication boundary for public-key credentials. + *******************************************************************************/ +public interface FileGatewayAuthenticatorInterface +{ + /******************************************************************************* + ** Resolve and authenticate a protocol credential. + *******************************************************************************/ + boolean authenticate(String username, PublicKey key, ServerSession session); +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/sftp/FileGatewayProtocolServer.java b/src/main/java/com/kingsrook/qbits/filegateway/sftp/FileGatewayProtocolServer.java new file mode 100644 index 0000000..b8a6dc7 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/sftp/FileGatewayProtocolServer.java @@ -0,0 +1,46 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.sftp; + + +import java.io.IOException; + + +/******************************************************************************* + ** Lifecycle of a gateway protocol listener. + *******************************************************************************/ +public interface FileGatewayProtocolServer +{ + /******************************************************************************* + ** Start listening. + *******************************************************************************/ + void start() throws IOException; + + /******************************************************************************* + ** Close the listener and live sessions. + *******************************************************************************/ + void stop() throws IOException; + + /******************************************************************************* + ** Number of open sessions. + *******************************************************************************/ + int activeSessions(); +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/sftp/GatewaySession.java b/src/main/java/com/kingsrook/qbits/filegateway/sftp/GatewaySession.java new file mode 100644 index 0000000..5cb043a --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/sftp/GatewaySession.java @@ -0,0 +1,129 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.sftp; + + +import java.io.IOException; +import java.net.InetSocketAddress; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.util.UUID; +import com.kingsrook.qbits.filegateway.permissions.FileGatewayPermissions; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleDispatcher; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInProcessSessions; +import org.apache.sshd.common.AttributeRepository.AttributeKey; +import org.apache.sshd.server.session.ServerSession; + + +/******************************************************************************* + ** One SSH connection, its audit identity and refreshable in-process credentials. + *******************************************************************************/ +public class GatewaySession +{ + public static final AttributeKey ATTRIBUTE = new AttributeKey<>(); + public static final AttributeKey TOKEN = new AttributeKey<>(); + public final String id = UUID.randomUUID().toString(); + public final ServerSession ssh; + public final String remoteIp; + public Integer connectionId; + public QRecord account; + public QRecord key; + public long operations; + public String disconnectReason = "Client disconnected"; + private Instant mintedAt; + private final QInstance instance; + private final Clock clock; + + + + /******************************************************************************* + ** Bind an accepted SSH connection. + *******************************************************************************/ + public GatewaySession(QInstance instance, ServerSession ssh, Clock clock) + { + this.instance = instance; + this.ssh = ssh; + this.clock = clock; + this.remoteIp = ((InetSocketAddress) ssh.getClientAddress()).getAddress().getHostAddress(); + } + + + + /******************************************************************************* + ** Mint via the host authentication module and refresh permissions every five minutes. + *******************************************************************************/ + public synchronized String token() throws QException + { + if(account == null || ssh.isClosing() || !ssh.isOpen()) + { + throw (new QException("Unauthenticated connection")); + } + if(mintedAt == null || !mintedAt.plus(Duration.ofMinutes(5)).isAfter(clock.instant())) + { + var module = new QAuthenticationModuleDispatcher().getQModule(instance.getAuthentication()); + var user = new AuditWriter(instance).system(() -> module.createAutomatedSessionForUser(instance, account.getValueString("userId"))); + if(user == null || !user.hasPermission(FileGatewayPermissions.CONNECT)) + { + revoke(); + throw (new QException("no file_gateway.connect")); + } + String next = UiApiInProcessSessions.register(user, Duration.ofMinutes(10)); + UiApiInProcessSessions.revoke(ssh.getAttribute(TOKEN)); + ssh.setAttribute(TOKEN, next); + mintedAt = clock.instant(); + } + return (ssh.getAttribute(TOKEN)); + } + + + + /******************************************************************************* + ** Release credentials immediately on disconnect or revocation. + *******************************************************************************/ + public synchronized void revoke() + { + UiApiInProcessSessions.revoke(ssh.removeAttribute(TOKEN)); + mintedAt = null; + } + + + + /******************************************************************************* + ** Count operations locally and charge the shared account minute window. + *******************************************************************************/ + public synchronized boolean operationAllowed() throws IOException + { + operations++; + Integer maximum = account.getValueInteger("maxOperationsPerMinute"); + try + { + return (new AuditWriter(instance).operationAllowed(account.getValueInteger("id"), clock.instant(), maximum == null ? 600 : maximum)); + } + catch(QException e) + { + throw (new IOException("Unable to verify account operation limit", e)); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/sftp/SftpProtocolServer.java b/src/main/java/com/kingsrook/qbits/filegateway/sftp/SftpProtocolServer.java new file mode 100644 index 0000000..b256d3d --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/sftp/SftpProtocolServer.java @@ -0,0 +1,355 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.sftp; + + +import java.io.IOException; +import java.nio.file.FileSystem; +import java.nio.file.Path; +import java.security.KeyPair; +import java.time.Clock; +import java.time.Duration; +import java.util.List; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; +import com.kingsrook.qbits.filegateway.FileGatewayQBitConfig; +import com.kingsrook.qbits.filegateway.storage.GatewayObjectStore; +import com.kingsrook.qbits.filegateway.storage.ObjectStorageGatewayStore; +import com.kingsrook.qbits.filegateway.upload.UploadJobs; +import com.kingsrook.qbits.filegateway.vfs.GatewayFileSystem; +import com.kingsrook.qbits.filegateway.vfs.GatewayUiApiClient; +import com.kingsrook.qbits.filegateway.vfs.GeneratedDownloads; +import com.kingsrook.qbits.filegateway.vfs.VirtualTree; +import com.kingsrook.qqq.backend.core.logging.QLogger; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInvoker; +import org.apache.sshd.common.cipher.BuiltinCiphers; +import org.apache.sshd.common.file.FileSystemFactory; +import org.apache.sshd.common.kex.BuiltinDHFactories; +import org.apache.sshd.common.keyprovider.KeyPairProvider; +import org.apache.sshd.common.mac.BuiltinMacs; +import org.apache.sshd.common.session.Session; +import org.apache.sshd.common.session.SessionContext; +import org.apache.sshd.common.session.SessionListener; +import org.apache.sshd.common.signature.BuiltinSignatures; +import org.apache.sshd.core.CoreModuleProperties; +import org.apache.sshd.server.ServerBuilder; +import org.apache.sshd.server.SshServer; +import org.apache.sshd.server.auth.pubkey.UserAuthPublicKeyFactory; +import org.apache.sshd.server.forward.RejectAllForwardingFilter; +import org.apache.sshd.server.session.ServerSession; +import org.apache.sshd.sftp.server.SftpSubsystemFactory; + + +/******************************************************************************* + ** Public-key-only SFTP with explicit modern algorithms and an entirely virtual filesystem. + *******************************************************************************/ +public class SftpProtocolServer implements FileGatewayProtocolServer +{ + private static final QLogger LOG = QLogger.getLogger(SftpProtocolServer.class); + private final SshServer server; + private final SshKeyAuthenticator authenticator; + private final Duration pollInterval; + private ScheduledExecutorService poller; + private final AuditWriter heartbeatAudit; + private final Clock heartbeatClock; + private final String heartbeatPod; + private final String hostFingerprint; + private java.time.Instant startedAt; + + + + /******************************************************************************* + ** Configure the listener, security policy and injected runtime dependencies. + *******************************************************************************/ + public SftpProtocolServer(QInstance instance, FileGatewayQBitConfig config, UiApiInvoker invoker, KeyPair hostKey, Clock clock, Duration pollInterval, String podName) + { + this(instance, config, invoker, hostKey, clock, pollInterval, podName, new ObjectStorageGatewayStore(instance, config.getFileGatewayBucketAlias())); + } + + + + /******************************************************************************* + ** Inject storage for transfer tests; production uses the object-storage implementation. + *******************************************************************************/ + public SftpProtocolServer(QInstance instance, FileGatewayQBitConfig config, UiApiInvoker invoker, KeyPair hostKey, Clock clock, Duration pollInterval, String podName, GatewayObjectStore store) + { + this.pollInterval = pollInterval; + heartbeatAudit = new AuditWriter(instance); + heartbeatClock = clock; + heartbeatPod = podName == null ? java.util.UUID.randomUUID().toString() : podName; + hostFingerprint = org.apache.sshd.common.config.keys.KeyUtils.getFingerPrint(org.apache.sshd.common.digest.BuiltinDigests.sha256, hostKey.getPublic()); + AuditWriter audit = new AuditWriter(instance); + var outbox = new com.kingsrook.qbits.filegateway.outbox.OutboxService(instance, store, config, clock); + com.kingsrook.qbits.filegateway.api.FileGatewayOutbox.setProvider(outbox); + authenticator = new SshKeyAuthenticator(instance, config, audit, clock, podName); + server = SshServer.setUpDefaultServer(); + server.setPort(config.getSftpPort()); + server.setKeyPairProvider(KeyPairProvider.wrap(hostKey)); + server.setCipherFactories(List.of(BuiltinCiphers.cc20p1305_openssh, BuiltinCiphers.aes128gcm, BuiltinCiphers.aes256gcm, BuiltinCiphers.aes128ctr, BuiltinCiphers.aes192ctr, BuiltinCiphers.aes256ctr)); + server.setMacFactories(List.of(BuiltinMacs.hmacsha256etm, BuiltinMacs.hmacsha512etm)); + server.setKeyExchangeFactories(List.of(BuiltinDHFactories.curve25519, BuiltinDHFactories.curve25519_libssh, BuiltinDHFactories.ecdhp256, BuiltinDHFactories.ecdhp384, BuiltinDHFactories.ecdhp521, BuiltinDHFactories.dhg16_512, BuiltinDHFactories.dhg18_512) + .stream().filter(BuiltinDHFactories::isSupported).map(ServerBuilder.DH2KEX).toList()); + server.setSignatureFactories(List.of(BuiltinSignatures.ed25519, BuiltinSignatures.nistp256, BuiltinSignatures.nistp384, BuiltinSignatures.nistp521, BuiltinSignatures.rsaSHA256, BuiltinSignatures.rsaSHA512)); + server.setPasswordAuthenticator(null); + server.setKeyboardInteractiveAuthenticator(null); + server.setGSSAuthenticator(null); + server.setHostBasedAuthenticator(null); + server.setUserAuthFactories(List.of(UserAuthPublicKeyFactory.INSTANCE)); + server.setPublickeyAuthenticator(authenticator); + server.setShellFactory(null); + server.setCommandFactory(null); + server.setForwardingFilter(RejectAllForwardingFilter.INSTANCE); + server.setAgentFactory(null); + server.setSubsystemFactories(List.of(new SftpSubsystemFactory.Builder().withSftpErrorStatusDataHandler(new org.apache.sshd.sftp.server.SftpErrorStatusDataHandler() + { + /******************************************************************************* + ** Preserve actionable gateway and UI API errors in SFTP failure responses. + *******************************************************************************/ + @Override + public String resolveErrorMessage(org.apache.sshd.sftp.server.SftpSubsystemEnvironment subsystem, int id, Throwable error, int status, int command, Object... args) + { + return (error instanceof IOException && error.getMessage() != null ? error.getMessage() + : org.apache.sshd.sftp.common.SftpHelper.resolveStatusMessage(status)); + } + }).withFileSystemAccessor(new org.apache.sshd.sftp.server.SftpFileSystemAccessor() + { + /******************************************************************************* + ** MINA must use our seekable channels, not the native FileChannel default. + *******************************************************************************/ + @Override + public java.nio.channels.SeekableByteChannel openFile(org.apache.sshd.sftp.server.SftpSubsystemProxy subsystem, org.apache.sshd.sftp.server.FileHandle handle, Path path, String handleId, + java.util.Set options, java.nio.file.attribute.FileAttribute... attributes) throws IOException + { + return (java.nio.file.Files.newByteChannel(path, options, attributes)); + } + }).build())); + CoreModuleProperties.IDLE_TIMEOUT.set(server, Duration.ofMinutes(15)); + CoreModuleProperties.AUTH_TIMEOUT.set(server, Duration.ofSeconds(30)); + server.setFileSystemFactory(new FileSystemFactory() + { + /******************************************************************************* + ** Create the session virtual root. + *******************************************************************************/ + @Override + public Path getUserHomeDir(SessionContext session) throws IOException + { + return (createFileSystem(session).getPath("/")); + } + + /******************************************************************************* + ** No native filesystem is consulted. + *******************************************************************************/ + @Override + public FileSystem createFileSystem(SessionContext session) + { + GatewaySession state = session.getAttribute(GatewaySession.ATTRIBUTE); + GatewayUiApiClient client = new GatewayUiApiClient(invoker, state); + return (new GatewayFileSystem(new VirtualTree(client, state, audit, clock).withUploads(new UploadJobs(audit, store, config, clock), store) + .withDownloads(new GeneratedDownloads(client, state, audit, config, clock, instance)).withOutbox(outbox))); + } + }); + server.addSessionListener(new SessionListener() + { + /******************************************************************************* + ** Persist accept before authentication. + *******************************************************************************/ + @Override + public void sessionCreated(Session session) + { + try + { + authenticator.accepted((ServerSession) session); + } + catch(Exception e) + { + LOG.warn("Unable to audit SFTP accept"); + session.close(true); + } + } + + /******************************************************************************* + ** Login bookkeeping starts only after proof of private-key possession. + *******************************************************************************/ + @Override + public void sessionEvent(Session session, Event event) + { + if(event == Event.Authenticated) + { + try + { + authenticator.authenticated((ServerSession) session); + } + catch(Exception e) + { + LOG.warn("Unable to finalize SFTP authentication"); + session.close(true); + } + } + } + + + + /******************************************************************************* + ** Always revoke the in-process token. + *******************************************************************************/ + @Override + public void sessionClosed(Session session) + { + try + { + authenticator.closed((ServerSession) session); + } + catch(Exception e) + { + LOG.warn("Unable to finalize SFTP connection audit"); + } + } + }); + } + + + + /******************************************************************************* + ** Start the listener and revocation heartbeat. + *******************************************************************************/ + @Override + public void start() throws IOException + { + server.start(); + startedAt = heartbeatClock.instant(); + try + { + heartbeat(); + } + catch(com.kingsrook.qqq.backend.core.exceptions.QException e) + { + server.stop(true); + throw (new IOException("Unable to publish gateway heartbeat", e)); + } + poller = Executors.newSingleThreadScheduledExecutor(runnable -> + { + Thread thread = new Thread(runnable, "file-gateway-revocation"); + thread.setDaemon(true); + return (thread); + }); + poller.scheduleWithFixedDelay(() -> + { + try + { + authenticator.poll(); + heartbeat(); + } + catch(Exception e) + { + LOG.warn("Unable to verify live SFTP sessions; closing them"); + server.getActiveSessions().forEach(session -> session.close(true)); + } + }, pollInterval.toMillis(), pollInterval.toMillis(), TimeUnit.MILLISECONDS); + } + + + + /******************************************************************************* + ** Publish host identity to the database for app homes in other deployments. + *******************************************************************************/ + public void heartbeat() throws com.kingsrook.qqq.backend.core.exceptions.QException + { + var rows = heartbeatAudit.query(com.kingsrook.qbits.filegateway.metadata.FileGatewayTables.SERVER, + UploadJobs.equal("podName", heartbeatPod)); + var values = new com.kingsrook.qqq.backend.core.model.data.QRecord().withValue("podName", heartbeatPod) + .withValue("hostKeyFingerprint", hostFingerprint).withValue("port", server.getPort()) + .withValue("startedAt", startedAt).withValue("lastSeenAt", heartbeatClock.instant()); + if(rows.isEmpty()) + { + heartbeatAudit.insert(com.kingsrook.qbits.filegateway.metadata.FileGatewayTables.SERVER, values); + } + else + { + heartbeatAudit.update(com.kingsrook.qbits.filegateway.metadata.FileGatewayTables.SERVER, values.withValue("id", rows.getFirst().getValue("id"))); + } + } + + + + /******************************************************************************* + ** Stop polling, revoke sessions and stop the listener. + *******************************************************************************/ + @Override + public void stop() throws IOException + { + if(poller != null) + { + poller.shutdownNow(); + } + server.stop(true); + try + { + authenticator.shutdown(); + } + catch(com.kingsrook.qqq.backend.core.exceptions.QException e) + { + throw (new IOException("Unable to finalize gateway shutdown", e)); + } + } + + + + /******************************************************************************* + ** Count active SSH connections. + *******************************************************************************/ + @Override + public int activeSessions() + { + return (server.getActiveSessions().size()); + } + + + + /******************************************************************************* + ** Bound port, including ephemeral test ports. + *******************************************************************************/ + public int getPort() + { + return (server.getPort()); + } + + + + /******************************************************************************* + ** Server security configuration for inspection. + *******************************************************************************/ + public SshServer getServer() + { + return (server); + } + + + + /******************************************************************************* + ** Live revocation hook for deterministic tests. + *******************************************************************************/ + public SshKeyAuthenticator getAuthenticator() + { + return (authenticator); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/sftp/SshKeyAuthenticator.java b/src/main/java/com/kingsrook/qbits/filegateway/sftp/SshKeyAuthenticator.java new file mode 100644 index 0000000..3015709 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/sftp/SshKeyAuthenticator.java @@ -0,0 +1,435 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.sftp; + + +import java.security.PublicKey; +import java.time.Clock; +import java.time.Instant; +import java.util.ArrayDeque; +import java.util.Deque; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import com.kingsrook.qbits.filegateway.FileGatewayQBitConfig; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qqq.backend.core.alerts.AlertSignal; +import com.kingsrook.qqq.backend.core.alerts.QAlerts; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.logging.QLogger; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import org.apache.sshd.common.config.keys.KeyUtils; +import org.apache.sshd.common.digest.BuiltinDigests; +import org.apache.sshd.server.auth.pubkey.PublickeyAuthenticator; +import org.apache.sshd.server.session.ServerSession; + + +/******************************************************************************* + ** Database-backed SSH key authentication, account limits, IP windows and live revocation. + *******************************************************************************/ +public class SshKeyAuthenticator implements PublickeyAuthenticator, FileGatewayAuthenticatorInterface +{ + private static final QLogger LOG = QLogger.getLogger(SshKeyAuthenticator.class); + private final QInstance instance; + private final FileGatewayQBitConfig config; + private final AuditWriter audit; + private final Clock clock; + private final String podName; + private final Map sessions = new ConcurrentHashMap<>(); + private final Map> failures = new HashMap<>(); + private final Map> connections = new HashMap<>(); + private final Map ipLocks = new HashMap<>(); + private final Map ipLockAccounts = new HashMap<>(); + private final Map failureAccounts = new HashMap<>(); + + + + /******************************************************************************* + ** Bind authentication and audit dependencies. + *******************************************************************************/ + public SshKeyAuthenticator(QInstance instance, FileGatewayQBitConfig config, AuditWriter audit, Clock clock, String podName) + { + this.instance = instance; + this.config = config; + this.audit = audit; + this.clock = clock; + this.podName = podName; + } + + + + /******************************************************************************* + ** Audit TCP accept and enforce the pre-authentication connection rate. + *******************************************************************************/ + public synchronized void accepted(ServerSession ssh) throws QException + { + GatewaySession state = new GatewaySession(instance, ssh, clock); + ssh.setAttribute(GatewaySession.ATTRIBUTE, state); + state.connectionId = audit.insert(FileGatewayTables.CONNECTION_LOG, new QRecord() + .withValue("sessionId", state.id).withValue("remoteIp", state.remoteIp).withValue("podName", podName) + .withValue("connectedAt", clock.instant()).withValue("lastSeenAt", clock.instant()) + .withValue("clientChannel", "sftp").withValue("outcome", "AUTH_FAILED")).getValueInteger("id"); + sessions.put(state.id, state); + prune(connections, clock.instant().minusSeconds(60)); + Deque window = connections.computeIfAbsent(state.remoteIp, ignored -> new ArrayDeque<>()); + window.addLast(clock.instant()); + if(window.size() > config.getMaxConnectionsPerIpPerMinute()) + { + outcome(state, "LIMIT", "Connection rate limit exceeded"); + ssh.close(true); + } + } + + + + /******************************************************************************* + ** Validate the account and offered public key, then mint the user session. + *******************************************************************************/ + @Override + public synchronized boolean authenticate(String username, PublicKey publicKey, ServerSession ssh) + { + GatewaySession state = ssh.getAttribute(GatewaySession.ATTRIBUTE); + if(state == null || ssh.isClosing()) + { + return (false); + } + try + { + return (audit.withAccountLock(username, () -> authenticateLocked(username, publicKey, ssh, state))); + } + catch(Exception e) + { + LOG.warn("SFTP authentication service failed", e); + state.revoke(); + state.disconnectReason = "Authentication service unavailable"; + ssh.close(true); + return (false); + } + } + + + + /******************************************************************************* + ** Account checks and reservations share one database transaction across pods. + *******************************************************************************/ + private boolean authenticateLocked(String username, PublicKey publicKey, ServerSession ssh, GatewaySession state) throws QException + { + if(state.account != null) + { + return (username.equals(state.account.getValueString("username")) && state.key != null + && KeyUtils.getFingerPrint(BuiltinDigests.sha256, publicKey).equals(state.key.getValueString("fingerprint"))); + } + QRecord account = first(FileGatewayTables.ACCOUNT, "username", username); + String fingerprint = KeyUtils.getFingerPrint(BuiltinDigests.sha256, publicKey); + audit.update(FileGatewayTables.CONNECTION_LOG, new QRecord().withValue("id", state.connectionId) + .withValue("username", username).withValue("keyFingerprint", fingerprint).withValue("clientVersion", ssh.getClientVersion()) + .withValue("accountId", account == null ? null : account.getValue("id"))); + Instant now = clock.instant(); + if(account != null && account.getValue("lockedUntil") == null && username.equals(ipLockAccounts.get(state.remoteIp))) + { + ipLocks.remove(state.remoteIp); + ipLockAccounts.remove(state.remoteIp); + failures.remove(state.remoteIp); + failureAccounts.remove(state.remoteIp); + } + ipLocks.entrySet().removeIf(entry -> !entry.getValue().isAfter(now)); + ipLockAccounts.keySet().retainAll(ipLocks.keySet()); + if(ipLocks.containsKey(state.remoteIp) || (account != null && (future(account, "lockedUntil") || "LOCKED".equals(account.getValueString("status"))))) + { + return (outcome(state, "LOCKED_OUT", "Authentication locked")); + } + if(account == null) + { + return (failed(state, null, "AUTH_FAILED", "Unknown account or key")); + } + if(!"ACTIVE".equals(account.getValueString("status"))) + { + return (failed(state, account, "DISABLED", "Account disabled")); + } + if(expired(account, "expiresAt")) + { + return (failed(state, account, "EXPIRED", "Account expired")); + } + if(!CidrMatcher.matches(state.remoteIp, account.getValueString("allowedCidrs"))) + { + return (failed(state, account, "IP_DENIED", "Address outside allowlist")); + } + QRecord key = first(FileGatewayTables.SSH_KEY, "fingerprint", fingerprint); + if(key == null || !account.getValue("id").equals(key.getValue("accountId")) || key.getValue("revokedAt") != null) + { + return (failed(state, account, "AUTH_FAILED", "Unknown or revoked key")); + } + if(expired(key, "expiresAt")) + { + return (failed(state, account, "EXPIRED", "Key expired")); + } + int limit = account.getValueInteger("maxConcurrentSessions") == null ? config.getDefaultMaxConcurrentSessions() : account.getValueInteger("maxConcurrentSessions"); + //////////////////////////////////////////////////////////////////////// + // only live rows: connected, not disconnected, seen in the last 3 min // + //////////////////////////////////////////////////////////////////////// + QQueryFilter liveFilter = filter("accountId", account.getValue("id")) + .withCriteria(new QFilterCriteria("outcome", QCriteriaOperator.EQUALS, "CONNECTED")) + .withCriteria(new QFilterCriteria("disconnectedAt", QCriteriaOperator.IS_BLANK)) + .withCriteria(new QFilterCriteria("lastSeenAt", QCriteriaOperator.GREATER_THAN, now.minusSeconds(180))); + long active = audit.query(FileGatewayTables.CONNECTION_LOG, liveFilter).size(); + if(active >= limit) + { + return (outcome(state, "LIMIT", "Concurrent session limit exceeded")); + } + state.account = account; + state.key = key; + try + { + state.token(); + } + catch(QException e) + { + state.account = null; + state.key = null; + return (failed(state, account, "AUTH_FAILED", "no file_gateway.connect")); + } + outcome(state, "CONNECTED", null); + return (true); + } + + + + /******************************************************************************* + ** MINA has verified the signature, not just accepted an unsigned key probe. + *******************************************************************************/ + public synchronized void authenticated(ServerSession ssh) throws QException + { + GatewaySession state = ssh.getAttribute(GatewaySession.ATTRIBUTE); + if(state == null || state.account == null) + { + throw (new QException("Missing authenticated gateway identity")); + } + QRecord account = state.account; + QRecord key = state.key; + Instant now = clock.instant(); + audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")) + .withValue("failedLoginCount", 0).withValue("failedLoginWindowAt", null) + .withValue("lastLoginAt", now).withValue("lastLoginIp", state.remoteIp)); + audit.update(FileGatewayTables.SSH_KEY, new QRecord().withValue("id", key.getValue("id")) + .withValue("lastUsedAt", now).withValue("lastUsedIp", state.remoteIp)); + } + + + + /******************************************************************************* + ** Persist account failure windows and apply the per-IP lockout. + *******************************************************************************/ + private boolean failed(GatewaySession state, QRecord account, String result, String reason) throws QException + { + Instant now = clock.instant(); + prune(failures, now.minus(config.getLockoutWindow())); + failureAccounts.keySet().retainAll(failures.keySet()); + Deque window = failures.computeIfAbsent(state.remoteIp, ignored -> new ArrayDeque<>()); + String username = account == null ? "" : account.getValueString("username"); + if(window.isEmpty()) + { + failureAccounts.put(state.remoteIp, username); + } + else if(!username.equals(failureAccounts.get(state.remoteIp))) + { + failureAccounts.put(state.remoteIp, ""); + } + window.addLast(now); + boolean lock = window.size() >= config.getLockoutThreshold(); + if(lock) + { + ipLocks.put(state.remoteIp, now.plus(config.getLockoutDuration())); + ipLockAccounts.put(state.remoteIp, failureAccounts.get(state.remoteIp)); + } + if(account != null) + { + Instant start = account.getValueInstant("failedLoginWindowAt"); + Integer oldCount = account.getValueInteger("failedLoginCount"); + int count = start == null || !start.plus(config.getLockoutWindow()).isAfter(now) ? 1 : (oldCount == null ? 0 : oldCount) + 1; + QRecord update = new QRecord().withValue("id", account.getValue("id")).withValue("failedLoginCount", count) + .withValue("failedLoginWindowAt", count == 1 ? now : start); + if(count >= config.getLockoutThreshold()) + { + update.setValue("lockedUntil", now.plus(config.getLockoutDuration())); + lock = true; + } + audit.update(FileGatewayTables.ACCOUNT, update); + } + if(lock) + { + audit.system(() -> + { + QAlerts.raise(new AlertSignal("filegateway.lockout").withDedupKey(account == null ? state.remoteIp : account.getValueString("id")) + .withSummary("File gateway authentication locked")); + return (null); + }); + } + return (outcome(state, result, reason)); + } + + + + /******************************************************************************* + ** Record the authentication outcome without credentials. + *******************************************************************************/ + private boolean outcome(GatewaySession state, String outcome, String reason) throws QException + { + audit.update(FileGatewayTables.CONNECTION_LOG, new QRecord().withValue("id", state.connectionId) + .withValue("outcome", outcome).withValue("disconnectReason", reason).withValue("lastSeenAt", clock.instant())); + if(reason != null) + { + state.disconnectReason = reason; + } + return (false); + } + + + + /******************************************************************************* + ** Revoke credentials and finalize connection accounting. + *******************************************************************************/ + public synchronized void closed(ServerSession ssh) throws QException + { + GatewaySession state = ssh.removeAttribute(GatewaySession.ATTRIBUTE); + if(state == null) + { + return; + } + state.revoke(); + sessions.remove(state.id); + if(state.account != null && !ssh.isAuthenticated()) + { + outcome(state, "AUTH_FAILED", "Public key signature was not verified"); + } + if(state.connectionId != null) + { + audit.update(FileGatewayTables.CONNECTION_LOG, new QRecord().withValue("id", state.connectionId) + .withValue("disconnectedAt", clock.instant()).withValue("disconnectReason", state.disconnectReason) + .withValue("bytesIn", 0L).withValue("bytesOut", 0L) + .withValue("operationCount", state.operations)); + } + } + + + + /******************************************************************************* + ** Finish any callbacks still queued when MINA has stopped accepting work. + *******************************************************************************/ + public synchronized void shutdown() throws QException + { + for(GatewaySession state : List.copyOf(sessions.values())) + { + closed(state.ssh); + } + } + + + + /******************************************************************************* + ** Re-read live accounts and keys and heartbeat open connection rows. + *******************************************************************************/ + public synchronized void poll() throws QException + { + for(GatewaySession state : List.copyOf(sessions.values())) + { + if(state.account == null) + { + continue; + } + QRecord account = first(FileGatewayTables.ACCOUNT, "id", state.account.getValue("id")); + QRecord key = first(FileGatewayTables.SSH_KEY, "id", state.key.getValue("id")); + QRecord connection = first(FileGatewayTables.CONNECTION_LOG, "id", state.connectionId); + if(account == null || key == null || !"ACTIVE".equals(account.getValueString("status")) + || expired(account, "expiresAt") || expired(key, "expiresAt") || key.getValue("revokedAt") != null + || connection == null || connection.getValue("disconnectedAt") != null || connection.getValue("endRequestedAt") != null) + { + state.disconnectReason = connection != null && connection.getValueString("disconnectReason") != null + ? connection.getValueString("disconnectReason") : "Account or key revoked"; + state.revoke(); + state.ssh.close(true); + } + else + { + state.account = account; + audit.update(FileGatewayTables.CONNECTION_LOG, new QRecord().withValue("id", state.connectionId) + .withValue("lastSeenAt", clock.instant()).withValue("operationCount", state.operations)); + } + } + } + + + + /******************************************************************************* + ** Find gateway-owned state. + *******************************************************************************/ + private QRecord first(String table, String field, java.io.Serializable value) throws QException + { + return (audit.query(table, filter(field, value)).stream().findFirst().orElse(null)); + } + + + + /******************************************************************************* + ** Equality query for owned tables. + *******************************************************************************/ + private QQueryFilter filter(String field, java.io.Serializable value) + { + return (new QQueryFilter(new QFilterCriteria(field, QCriteriaOperator.EQUALS, value))); + } + + + + /******************************************************************************* + ** Whether a timestamp lies in the future. + *******************************************************************************/ + private boolean future(QRecord row, String field) + { + Instant value = row.getValueInstant(field); + return (value != null && value.isAfter(clock.instant())); + } + + + + /******************************************************************************* + ** Whether an optional expiry has passed. + *******************************************************************************/ + private boolean expired(QRecord row, String field) + { + Instant value = row.getValueInstant(field); + return (value != null && !value.isAfter(clock.instant())); + } + + + + /******************************************************************************* + ** Bound in-memory IP windows to their active interval. + *******************************************************************************/ + private void prune(Map> windows, Instant cutoff) + { + windows.values().forEach(window -> window.removeIf(instant -> !instant.isAfter(cutoff))); + windows.entrySet().removeIf(entry -> entry.getValue().isEmpty()); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/storage/GatewayObjectStore.java b/src/main/java/com/kingsrook/qbits/filegateway/storage/GatewayObjectStore.java new file mode 100644 index 0000000..ab1f824 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/storage/GatewayObjectStore.java @@ -0,0 +1,62 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.storage; + +import java.io.IOException; +import java.io.InputStream; + +/******************************************************************************* + ** Streaming storage boundary. Production uses the object-storage system API. + *******************************************************************************/ +public interface GatewayObjectStore +{ + /******************************************************************************* + ** Completed immutable object identity, measured while streaming. + *******************************************************************************/ + record Receipt(long size, String sha256) + { + } + + /******************************************************************************* + ** Consume an unknown-length stream; a read failure must abort multipart upload. + *******************************************************************************/ + Receipt put(String key, InputStream content) throws IOException; + + /******************************************************************************* + ** Open a fresh stream owned by the caller. + *******************************************************************************/ + InputStream open(String key) throws IOException; + + /******************************************************************************* + ** Open an inclusive byte range, owned by the caller. + *******************************************************************************/ + InputStream openRange(String key, long start, long end) throws IOException; + + /******************************************************************************* + ** Copy within the configured bucket. + *******************************************************************************/ + void copy(String source, String destination) throws IOException; + + /******************************************************************************* + ** Permanently remove an aborted or quarantined source object. + *******************************************************************************/ + void delete(String key) throws IOException; +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/storage/GeneratedReadChannel.java b/src/main/java/com/kingsrook/qbits/filegateway/storage/GeneratedReadChannel.java new file mode 100644 index 0000000..e4b3791 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/storage/GeneratedReadChannel.java @@ -0,0 +1,268 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.storage; + + +import java.io.IOException; +import java.io.InputStream; +import java.nio.ByteBuffer; +import java.nio.channels.ClosedChannelException; +import java.nio.channels.NonWritableChannelException; +import java.nio.channels.SeekableByteChannel; + + +/******************************************************************************* + ** A generated stream with a fixed-size rolling window; never regenerates or spools. + *******************************************************************************/ +public final class GeneratedReadChannel implements SeekableByteChannel +{ + /******************************************************************************* + ** Charge newly consumed bytes and finalize the transfer when its handle closes. + *******************************************************************************/ + public interface Audit + { + /******************************************************************************* + ** Enforce limits before exposing the newly generated bytes to the client. + *******************************************************************************/ + void charge(int bytes) throws IOException; + + /******************************************************************************* + ** Record the final outcome, including stream and seek errors. + *******************************************************************************/ + void finish(IOException failure) throws IOException; + } + + private final InputStream input; + private final byte[] window; + private final Audit audit; + private long end; + private long position; + private boolean eof; + private boolean open = true; + private IOException failure; + + + /******************************************************************************* + ** The supplied stream is owned by this channel from construction through close. + *******************************************************************************/ + public GeneratedReadChannel(InputStream input, int windowBytes, Audit audit) + { + if(windowBytes < 1) + { + throw (new IllegalArgumentException("Read-ahead window must be positive")); + } + this.input = input; + this.window = new byte[windowBytes]; + this.audit = audit; + } + + + /******************************************************************************* + ** Retain the most recent window, including skipped bytes for pipelined reads. + *******************************************************************************/ + @Override + public synchronized int read(ByteBuffer destination) throws IOException + { + checkOpen(); + int total = 0; + while(destination.hasRemaining()) + { + int count = readChunk(destination); + if(count < 0) + { + return (total == 0 ? -1 : total); + } + total += count; + } + return (total); + } + + + /******************************************************************************* + ** Fill one ring segment; the public read fills its request to avoid false MINA EOF. + *******************************************************************************/ + private int readChunk(ByteBuffer destination) throws IOException + { + checkOpen(); + if(failure != null) + { + throw (failure); + } + if(!destination.hasRemaining()) + { + return (0); + } + try + { + if(eof && position >= end) + { + return (-1); + } + if(position < Math.max(0, end - window.length) || position - end >= window.length) + { + throw (new IOException("Read outside the generated-file window at " + position + " (stream offset " + end + "); download sequentially")); + } + while(position >= end && !eof) + { + int offset = (int) (end % window.length); + int length = (int) Math.min(Math.min(65536, window.length - offset), position - end + Math.min(destination.remaining(), window.length)); + int count = input.readNBytes(window, offset, length); + if(count == 0) + { + eof = true; + } + else if(count > 0) + { + audit.charge(count); + end += count; + } + } + if(position >= end) + { + return (-1); + } + int offset = (int) (position % window.length); + int count = (int) Math.min(Math.min(destination.remaining(), end - position), window.length - offset); + destination.put(window, offset, count); + position += count; + return (count); + } + catch(IOException e) + { + failure = e; + try + { + input.close(); + } + catch(IOException closeError) + { + e.addSuppressed(closeError); + } + throw (e); + } + } + + + /******************************************************************************* + ** Generated content is read-only. + *******************************************************************************/ + @Override + public int write(ByteBuffer source) + { + throw (new NonWritableChannelException()); + } + + + /******************************************************************************* + ** Current logical read offset. + *******************************************************************************/ + @Override + public synchronized long position() throws IOException + { + checkOpen(); + return (position); + } + + + /******************************************************************************* + ** Validate window membership on read, when an actual request is made. + *******************************************************************************/ + @Override + public synchronized SeekableByteChannel position(long value) throws IOException + { + checkOpen(); + if(value < 0) + { + throw (new IllegalArgumentException("Negative read offset")); + } + position = value; + return (this); + } + + + /******************************************************************************* + ** Generated size is unknown; consumers must read until EOF. + *******************************************************************************/ + @Override + public long size() throws IOException + { + checkOpen(); + return (0); + } + + + /******************************************************************************* + ** Generated content cannot be truncated. + *******************************************************************************/ + @Override + public SeekableByteChannel truncate(long size) + { + throw (new NonWritableChannelException()); + } + + + /******************************************************************************* + ** Handle state. + *******************************************************************************/ + @Override + public boolean isOpen() + { + return (open); + } + + + /******************************************************************************* + ** Always close the HTTP stream and finalize exactly one transfer row. + *******************************************************************************/ + @Override + public synchronized void close() throws IOException + { + if(open) + { + open = false; + try + { + input.close(); + } + catch(IOException e) + { + failure = e; + throw (e); + } + finally + { + audit.finish(failure); + } + } + } + + + /******************************************************************************* + ** Reject operations after close. + *******************************************************************************/ + private void checkOpen() throws ClosedChannelException + { + if(!open) + { + throw (new ClosedChannelException()); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/storage/ObjectReadChannel.java b/src/main/java/com/kingsrook/qbits/filegateway/storage/ObjectReadChannel.java new file mode 100644 index 0000000..9cf2e3b --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/storage/ObjectReadChannel.java @@ -0,0 +1,247 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.storage; + +import java.io.IOException; +import java.io.InputStream; +import java.nio.ByteBuffer; +import java.nio.channels.ClosedChannelException; +import java.nio.channels.NonWritableChannelException; +import java.nio.channels.SeekableByteChannel; + +/******************************************************************************* + ** Seekable range reads with bounded per-request memory and prompt response cleanup. + *******************************************************************************/ +public final class ObjectReadChannel implements SeekableByteChannel +{ + /******************************************************************************* + ** Open an inclusive range of an immutable object or generated sidecar. + *******************************************************************************/ + @FunctionalInterface + public interface Ranges + { + /******************************************************************************* + ** Caller closes the response stream. + *******************************************************************************/ + InputStream open(long start, long end) throws IOException; + } + + /******************************************************************************* + ** Persist one transfer outcome when the handle closes. + *******************************************************************************/ + @FunctionalInterface + public interface Completion + { + /******************************************************************************* + ** Report actual bytes sent and any read failure. + *******************************************************************************/ + void done(long bytes, boolean failed) throws IOException; + } + + private final long size; + private final Ranges ranges; + private final Completion completion; + private long bytesRead; + private Completion fullDownload; + private long contiguousEnd; + private final java.util.TreeMap readRanges = new java.util.TreeMap<>(); + private boolean failed; + private long position; + private boolean open = true; + + /******************************************************************************* + ** Bind a stored size and ranged stream supplier. + *******************************************************************************/ + public ObjectReadChannel(long size, Ranges ranges) + { + this(size, ranges, (bytes, failure) -> + { + // Direct channel users may omit a transfer audit callback. + }); + } + + /******************************************************************************* + ** Bind an audited object read. + *******************************************************************************/ + public ObjectReadChannel(long size, Ranges ranges, Completion completion) + { + this.size = size; + this.ranges = ranges; + this.completion = completion; + } + + /******************************************************************************* + ** At most 64 KiB is buffered, regardless of object size or seek distance. + *******************************************************************************/ + @Override + public int read(ByteBuffer destination) throws IOException + { + checkOpen(); + if(!destination.hasRemaining()) + { + return (0); + } + if(position >= size) + { + return (-1); + } + int length = (int) Math.min(Math.min(destination.remaining(), 65536), size - position); + try(InputStream input = ranges.open(position, position + length - 1)) + { + byte[] bytes = input.readNBytes(length); + if(bytes.length == 0) + { + throw (new IOException("Stored object ended before its recorded size")); + } + destination.put(bytes); + recordRange(position, position + bytes.length); + position += bytes.length; + bytesRead += bytes.length; + return (bytes.length); + } + catch(IOException e) + { + failed = true; + throw (e); + } + } + + /******************************************************************************* + ** Stored objects are immutable. + *******************************************************************************/ + @Override + public int write(ByteBuffer source) + { + throw (new NonWritableChannelException()); + } + + /******************************************************************************* + ** Current read offset. + *******************************************************************************/ + @Override + public long position() throws IOException + { + checkOpen(); + return (position); + } + + /******************************************************************************* + ** Seek without reading preceding bytes. + *******************************************************************************/ + @Override + public SeekableByteChannel position(long value) throws IOException + { + checkOpen(); + if(value < 0) + { + throw (new IllegalArgumentException("Negative read offset")); + } + position = value; + return (this); + } + + /******************************************************************************* + ** Persisted object length. + *******************************************************************************/ + @Override + public long size() + { + return (size); + } + + /******************************************************************************* + ** Stored objects cannot be truncated. + *******************************************************************************/ + @Override + public SeekableByteChannel truncate(long value) + { + throw (new NonWritableChannelException()); + } + + /******************************************************************************* + ** Handle state. + *******************************************************************************/ + @Override + public boolean isOpen() + { + return (open); + } + + /******************************************************************************* + ** No stream is retained between reads. + *******************************************************************************/ + @Override + public void close() throws IOException + { + if(open) + { + open = false; + completion.done(bytesRead, failed); + if(!failed && contiguousEnd >= size && fullDownload != null) + { + fullDownload.done(bytesRead, false); + } + } + } + + /******************************************************************************* + ** Optional acknowledgment, only after all bytes have been read on this handle. + *******************************************************************************/ + public ObjectReadChannel withFullDownload(Completion callback) + { + fullDownload = callback; + return (this); + } + + + + /******************************************************************************* + ** Bounded interval tracking avoids counting repeated ranged reads as a complete file. + *******************************************************************************/ + private void recordRange(long start, long end) + { + if(start <= contiguousEnd) + { + contiguousEnd = Math.max(contiguousEnd, end); + } + else if(readRanges.size() < 4096) + { + readRanges.merge(start, end, Math::max); + } + while(!readRanges.isEmpty() && readRanges.firstKey() <= contiguousEnd) + { + contiguousEnd = Math.max(contiguousEnd, readRanges.pollFirstEntry().getValue()); + } + } + + + + /******************************************************************************* + ** Reject use after close. + *******************************************************************************/ + private void checkOpen() throws ClosedChannelException + { + if(!open) + { + throw (new ClosedChannelException()); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/storage/ObjectStorageGatewayStore.java b/src/main/java/com/kingsrook/qbits/filegateway/storage/ObjectStorageGatewayStore.java new file mode 100644 index 0000000..35b361f --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/storage/ObjectStorageGatewayStore.java @@ -0,0 +1,149 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.storage; + +import java.io.FilterInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.security.DigestInputStream; +import java.security.MessageDigest; +import java.util.HexFormat; +import com.kingsrook.qbits.filegateway.sftp.AuditWriter; +import com.kingsrook.qbits.objectstorage.api.CopyRequest; +import com.kingsrook.qbits.objectstorage.api.DeleteRequest; +import com.kingsrook.qbits.objectstorage.api.ObjectStorage; +import com.kingsrook.qbits.objectstorage.api.PutRequest; +import com.kingsrook.qbits.objectstorage.api.Range; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; + +/******************************************************************************* + ** Only the allow-listed object-storage API is used, under a named system caller. + *******************************************************************************/ +public final class ObjectStorageGatewayStore implements GatewayObjectStore +{ + private final AuditWriter context; + private final String alias; + + /******************************************************************************* + ** Bind the host instance and bucket alias without touching storage at startup. + *******************************************************************************/ + public ObjectStorageGatewayStore(QInstance instance, String alias) + { + context = new AuditWriter(instance); + this.alias = alias; + } + + /******************************************************************************* + ** Unknown-length puts use object-storage's bounded multipart implementation. + *******************************************************************************/ + @Override + public Receipt put(String key, InputStream content) throws IOException + { + try + { + MessageDigest digest = MessageDigest.getInstance("SHA-256"); + var result = context.system(() -> ObjectStorage.asSystem("file-gateway", key).bucket(alias) + .put(PutRequest.of(key, new DigestInputStream(content, digest)))); + return (new Receipt(result.getBytes(), HexFormat.of().formatHex(digest.digest()))); + } + catch(Exception e) + { + throw (new IOException("Object upload failed", e)); + } + } + + /******************************************************************************* + ** Stream the complete object. + *******************************************************************************/ + @Override + public InputStream open(String key) throws IOException + { + return (read(key, Range.all())); + } + + /******************************************************************************* + ** Stream only the requested range. + *******************************************************************************/ + @Override + public InputStream openRange(String key, long start, long end) throws IOException + { + return (read(key, Range.bytes(start, end))); + } + + /******************************************************************************* + ** Closing the returned stream also releases its object response. + *******************************************************************************/ + private InputStream read(String key, Range range) throws IOException + { + try + { + var object = context.system(() -> ObjectStorage.asSystem("file-gateway", key).bucket(alias).get(key, range)); + return (new FilterInputStream(object.getInputStream()) + { + @Override + public void close() throws IOException + { + object.close(); + } + }); + } + catch(Exception e) + { + throw (new IOException("Object read failed", e)); + } + } + + /******************************************************************************* + ** Server-side quarantine copy. + *******************************************************************************/ + @Override + public void copy(String source, String destination) throws IOException + { + try + { + context.system(() -> + { + ObjectStorage.asSystem("file-gateway", source).bucket(alias).copy(CopyRequest.of(source, destination)); + return (null); + }); + } + catch(Exception e) + { + throw (new IOException("Object copy failed", e)); + } + } + + /******************************************************************************* + ** Remove all versions of a gateway-owned object. + *******************************************************************************/ + @Override + public void delete(String key) throws IOException + { + try + { + context.system(() -> ObjectStorage.asSystem("file-gateway", key).bucket(alias).delete(DeleteRequest.of(key).permanent())); + } + catch(Exception e) + { + throw (new IOException("Object deletion failed", e)); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/upload/BulkLoadConsumer.java b/src/main/java/com/kingsrook/qbits/filegateway/upload/BulkLoadConsumer.java new file mode 100644 index 0000000..8d578f1 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/upload/BulkLoadConsumer.java @@ -0,0 +1,209 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.upload; + +import java.io.IOException; +import java.util.LinkedHashMap; +import java.util.Map; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.kingsrook.qbits.filegateway.api.FileAcceptance; +import com.kingsrook.qbits.filegateway.api.FileGatewayConsumer; +import com.kingsrook.qbits.filegateway.api.InboundFile; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiRequest; +import com.kingsrook.qqq.openapi.model.HttpMethod; + +/******************************************************************************* + ** Drives the real bulk-insert UI process as the user; never inserts application rows directly. + *******************************************************************************/ +public final class BulkLoadConsumer implements FileGatewayConsumer +{ + private final WorkerUserSession session; + private final QRecord job; + private QRecord outcome; + + /******************************************************************************* + ** One consumer instance per delivery, holding only that process's outcome. + *******************************************************************************/ + public BulkLoadConsumer(WorkerUserSession session, QRecord job) + { + this.session = session; + this.job = job; + } + + /******************************************************************************* + ** Stable built-in registry name. + *******************************************************************************/ + @Override + public String getName() + { + return ("bulk-load"); + } + + /******************************************************************************* + ** Follow returned frontend steps; the API resumes backend steps after each one. + *******************************************************************************/ + @Override + public FileAcceptance accept(InboundFile file) throws Exception + { + String process = job.getValueString("tableName") + ".bulkInsert"; + JsonNode profiles = session.invoke(new UiApiRequest().withMethod(HttpMethod.POST).withPath("/table/{tableName}/query") + .withPathParam("tableName", "savedBulkLoadProfile").withJsonBody(Map.of("filter", UploadJobs.equal("id", job.getValue("savedBulkLoadProfileId"))))).path("records"); + if(profiles.size() != 1 || !job.getValueString("tableName").equals(profiles.get(0).path("values").path("tableName").asText())) + { + return (failed("Saved bulk load profile is unavailable")); + } + JsonNode mapping = new ObjectMapper().readTree(profiles.get(0).path("values").path("mappingJson").asText()); + Map values = new LinkedHashMap<>(); + values.put("savedBulkLoadProfileId", job.getValue("savedBulkLoadProfileId")); + values.put("version", mapping.path("version").asText()); + values.put("layout", mapping.path("layout").asText()); + values.put("hasHeaderRow", mapping.path("hasHeaderRow").asBoolean()); + values.put("fieldListJSON", mapping.path("fieldList").toString()); + for(JsonNode field : mapping.path("fieldList")) + { + if(field.path("doValueMapping").asBoolean() && field.path("valueMappings").isEmpty()) + { + return (failed("this profile needs value mappings saved in it")); + } + } + JsonNode response = session.invoke(new UiApiRequest().withMethod(HttpMethod.POST).withPath("/processes/{processName}/init") + .withPathParam("processName", process).withFormParam("values", new ObjectMapper().writeValueAsString(Map.of("tableName", job.getValueString("tableName")))) + .withMultipartFile("theFile", file.fileName(), "application/octet-stream", file.streamSupplier())); + String uuid = response.path("processUUID").asText(); + job.withValue("processUuid", uuid); + for(int steps = 0; steps < 20; steps++) + { + response = await(process, uuid, response); + if("ERROR".equals(response.path("type").asText())) + { + return (failed(response.path("userFacingError").asText("Bulk load process failed"))); + } + String next = response.path("nextStep").asText(); + if(next.equals("valueMapping")) + { + return (failed("this profile needs value mappings saved in it")); + } + if(response.path("values").has("processResults")) + { + summarize(response.path("values").path("processResults")); + return (FileAcceptance.accepted(uuid)); + } + if(!java.util.List.of("upload", "fileMapping", "review").contains(next)) + { + return (failed("Unexpected bulk load step: " + next)); + } + Map stepValues = next.equals("fileMapping") ? values : Map.of(); + UiApiRequest step = new UiApiRequest().withMethod(HttpMethod.POST).withPath("/processes/{processName}/{processUUID}/step/{stepName}") + .withPathParam("processName", process).withPathParam("processUUID", uuid).withPathParam("stepName", next); + if(!stepValues.isEmpty()) + { + // the invoker sends form fields as multipart for this multipart-only operation (qqq#1170) + step.withFormParam("values", new ObjectMapper().writeValueAsString(stepValues)); + } + response = session.invoke(step); + } + return (failed("Bulk load exceeded its step limit")); + } + + /******************************************************************************* + ** Asynchronous UI API steps retain their job UUID across RUNNING responses. + *******************************************************************************/ + private JsonNode await(String process, String uuid, JsonNode response) throws Exception + { + String async = response.path("jobUUID").asText(); + for(int poll = 0; poll < 3600 && ("JOB_STARTED".equals(response.path("type").asText()) || "RUNNING".equals(response.path("type").asText())); poll++) + { + Thread.sleep(100); + response = session.invoke(new UiApiRequest().withMethod(HttpMethod.GET).withPath("/processes/{processName}/{processUUID}/status/{jobUUID}") + .withPathParam("processName", process).withPathParam("processUUID", uuid).withPathParam("jobUUID", async)); + } + if("RUNNING".equals(response.path("type").asText())) + { + throw (new IOException("Bulk load status timed out")); + } + return (response); + } + + /******************************************************************************* + ** Preserve the API's detailed summary (including row examples) for the result artifact. + *******************************************************************************/ + private void summarize(JsonNode lines) + { + int inserted = 0; + int errors = 0; + StringBuilder summary = new StringBuilder(); + StringBuilder csv = new StringBuilder("row,field,message\r\n"); + for(JsonNode line : lines) + { + String status = line.path("status").asText(); + int count = line.path("count").asInt(); + String message = line.path("message").asText(); + if("OK".equals(status)) + { + inserted += count; + } + if("ERROR".equals(status)) + { + errors += count; + } + summary.append(count).append(" ").append(message).append("\n"); + csv.append(",,").append(csvCell(count + " " + message)).append("\r\n"); + for(JsonNode bullet : line.path("bulletsOfText")) + { + String detail = bullet.asText(); + summary.append(detail).append("\n"); + var row = java.util.regex.Pattern.compile("(?i)(?:row[^0-9]*)?([0-9]+)").matcher(detail); + csv.append(row.find() ? row.group(1) : "").append(",,").append(csvCell(message + " " + detail)).append("\r\n"); + } + } + outcome = new QRecord().withValue("status", errors == 0 ? "SUCCEEDED" : inserted == 0 ? "FAILED" : "PARTIAL") + .withValue("insertedCount", inserted).withValue("errorCount", errors).withValue("resultSummary", summary.toString()).withValue("_resultsCsv", csv.toString()); + } + + /******************************************************************************* + ** Quote CSV cells, including embedded commas, newlines and quotes. + *******************************************************************************/ + private String csvCell(String value) + { + return ("\"" + value.replace("\"", "\"\"") + "\""); + } + + + + /******************************************************************************* + ** Process errors are terminal rather than silently replaying an executed import. + *******************************************************************************/ + private FileAcceptance failed(String message) + { + outcome = new QRecord().withValue("status", "FAILED").withValue("errorMessage", message).withValue("resultSummary", message); + return (FileAcceptance.accepted(job.getValueString("processUuid"))); + } + + /******************************************************************************* + ** Worker persists counts and status along with the consumer reference. + *******************************************************************************/ + public QRecord outcome() + { + return (outcome); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/upload/UploadChannel.java b/src/main/java/com/kingsrook/qbits/filegateway/upload/UploadChannel.java new file mode 100644 index 0000000..261ab65 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/upload/UploadChannel.java @@ -0,0 +1,304 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.upload; + +import java.io.FilterInputStream; +import java.io.IOException; +import java.io.PipedInputStream; +import java.io.PipedOutputStream; +import java.nio.ByteBuffer; +import java.nio.channels.ClosedChannelException; +import java.nio.channels.NonReadableChannelException; +import java.nio.channels.SeekableByteChannel; +import java.util.TreeMap; +import java.util.concurrent.CompletableFuture; +import com.kingsrook.qbits.filegateway.storage.GatewayObjectStore; + +/******************************************************************************* + ** Sequential storage stream with a bounded SFTP reorder window and backpressure. + *******************************************************************************/ +public final class UploadChannel implements SeekableByteChannel +{ + /******************************************************************************* + ** Persist reservations and terminal state before acknowledging SFTP operations. + *******************************************************************************/ + public interface Lifecycle + { + /******************************************************************************* + ** Atomically reserve the current high-water size against account limits. + *******************************************************************************/ + void reserve(long size) throws IOException; + + /******************************************************************************* + ** Commit a completed object. + *******************************************************************************/ + void complete(GatewayObjectStore.Receipt receipt) throws IOException; + + /******************************************************************************* + ** Persist an aborted transfer and its reason. + *******************************************************************************/ + void abort(String message, boolean limit) throws IOException; + } + + private final GatewayObjectStore store; + private final String key; + private final Lifecycle lifecycle; + private final long window; + private final PipedInputStream input; + private final PipedOutputStream output; + private final CompletableFuture result = new CompletableFuture<>(); + private final TreeMap pending = new TreeMap<>(); + private volatile boolean aborted; + private boolean open = true; + private long position; + private long contiguous; + private long size; + + /******************************************************************************* + ** A virtual thread consumes the pipe; only bounded buffers hold upload content. + *******************************************************************************/ + public UploadChannel(GatewayObjectStore store, String key, int bufferSize, long window, Lifecycle lifecycle) throws IOException + { + this.store = store; + this.key = key; + this.lifecycle = lifecycle; + this.window = window; + input = new PipedInputStream(bufferSize); + output = new PipedOutputStream(input); + Thread.startVirtualThread(() -> + { + try(var source = new FilterInputStream(input) + { + @Override + public int read(byte[] bytes, int offset, int length) throws IOException + { + if(aborted) + { + throw (new IOException("Upload aborted")); + } + int count = super.read(bytes, offset, length); + if(aborted) + { + throw (new IOException("Upload aborted")); + } + return (count); + } + }) + { + var receipt = store.put(key, source); + result.complete(receipt); + } + catch(Exception e) + { + result.completeExceptionally(e); + } + finally + { + if(aborted) + { + try + { + store.delete(key); + } + catch(IOException ignored) + { + // The caller also deletes after waiting for completion. + } + } + } + }); + } + + /******************************************************************************* + ** Fail closed and release the streaming producer before deleting the object. + *******************************************************************************/ + private void fail(String message, boolean limit) throws IOException + { + aborted = true; + open = false; + pending.clear(); + output.close(); + input.close(); + try + { + result.join(); + } + catch(RuntimeException ignored) + { + // A failing input is how the storage client aborts multipart uploads. + } + try + { + store.delete(key); + } + finally + { + lifecycle.abort(message, limit); + } + } + + /******************************************************************************* + ** Writes behind the committed prefix and overlapping blocks are unsupported. + *******************************************************************************/ + @Override + public synchronized int write(ByteBuffer source) throws IOException + { + checkOpen(); + int count = source.remaining(); + if(count == 0) + { + return (0); + } + try + { + long end = Math.addExact(position, count); + var before = pending.floorEntry(position); + var after = pending.ceilingEntry(position); + if(position < contiguous || (before != null && before.getKey() + before.getValue().length > position) + || (after != null && after.getKey() < end) || end - contiguous > window) + { + throw (new IOException("Upload exceeds reorder window or overlaps an earlier write")); + } + lifecycle.reserve(Math.max(size, end)); + byte[] bytes = new byte[count]; + source.get(bytes); + pending.put(position, bytes); + size = Math.max(size, end); + position = end; + while(pending.containsKey(contiguous)) + { + byte[] next = pending.remove(contiguous); + output.write(next); + contiguous += next.length; + } + return (count); + } + catch(Exception e) + { + boolean limit = e instanceof UploadLimitException; + fail(limit ? e.getMessage() : "Upload stream failed", limit); + throw (new IOException(limit ? e.getMessage() : "Upload stream failed", e)); + } + } + + /******************************************************************************* + ** Complete only a contiguous upload; close of a temporary file does not queue it. + *******************************************************************************/ + @Override + public synchronized void close() throws IOException + { + if(!open) + { + return; + } + try + { + if(!pending.isEmpty()) + { + throw (new IOException("Upload has missing blocks")); + } + output.close(); + lifecycle.complete(result.join()); + open = false; + } + catch(Exception e) + { + fail("Upload did not complete", false); + throw (new IOException("Upload did not complete", e)); + } + } + + /******************************************************************************* + ** Verify this handle is usable. + *******************************************************************************/ + private void checkOpen() throws ClosedChannelException + { + if(!open) + { + throw (new ClosedChannelException()); + } + } + + /******************************************************************************* + ** Upload handles cannot read. + *******************************************************************************/ + @Override + public int read(ByteBuffer destination) + { + throw (new NonReadableChannelException()); + } + + /******************************************************************************* + ** Current SFTP write offset. + *******************************************************************************/ + @Override + public synchronized long position() throws IOException + { + checkOpen(); + return (position); + } + + /******************************************************************************* + ** SFTP clients set the offset before each write. + *******************************************************************************/ + @Override + public synchronized SeekableByteChannel position(long value) throws IOException + { + checkOpen(); + if(value < 0) + { + throw (new IllegalArgumentException("Negative upload offset")); + } + position = value; + return (this); + } + + /******************************************************************************* + ** Highest received offset. + *******************************************************************************/ + @Override + public synchronized long size() + { + return (size); + } + + /******************************************************************************* + ** Arbitrary truncation would invalidate a streamed object. + *******************************************************************************/ + @Override + public SeekableByteChannel truncate(long value) throws IOException + { + if(value != size) + { + throw (new IOException("Upload truncation is unsupported")); + } + return (this); + } + + /******************************************************************************* + ** Handle lifecycle. + *******************************************************************************/ + @Override + public boolean isOpen() + { + return (open); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/upload/UploadJobWorker.java b/src/main/java/com/kingsrook/qbits/filegateway/upload/UploadJobWorker.java new file mode 100644 index 0000000..917ef3b --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/upload/UploadJobWorker.java @@ -0,0 +1,482 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.upload; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.UncheckedIOException; +import java.nio.charset.StandardCharsets; +import java.time.Clock; +import java.time.Duration; +import java.util.List; +import java.util.Objects; +import java.util.concurrent.Executors; +import java.util.concurrent.ScheduledExecutorService; +import java.util.concurrent.TimeUnit; +import com.kingsrook.qbits.filegateway.FileGatewayQBitConfig; +import com.kingsrook.qbits.filegateway.api.FileAcceptance; +import com.kingsrook.qbits.filegateway.api.FileGatewayConsumer; +import com.kingsrook.qbits.filegateway.api.FileGatewayConsumersMetaData; +import com.kingsrook.qbits.filegateway.api.FileGatewayScannerInterface; +import com.kingsrook.qbits.filegateway.api.InboundFile; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qbits.filegateway.sftp.AuditWriter; +import com.kingsrook.qbits.filegateway.storage.GatewayObjectStore; +import com.kingsrook.qqq.backend.core.actions.customizers.QCodeLoader; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.logging.QLogger; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInvoker; + +/******************************************************************************* + ** Bounded, at-least-once upload delivery with fenced claims and deterministic polling. + *******************************************************************************/ +public class UploadJobWorker implements AutoCloseable +{ + private static final QLogger LOG = QLogger.getLogger(UploadJobWorker.class); + private static final long[] RETRY_SECONDS = {60, 300, 900, 3600, 21600}; + private final QInstance instance; + private final UiApiInvoker invoker; + private final FileGatewayQBitConfig config; + private final GatewayObjectStore store; + private final AuditWriter audit; + private final Clock clock; + private final String pod; + private java.util.function.Function environment = System::getenv; + private volatile boolean stopped; + private ScheduledExecutorService pool; + + /******************************************************************************* + ** No threads are started until the enabled runtime explicitly starts this worker. + *******************************************************************************/ + public UploadJobWorker(QInstance instance, UiApiInvoker invoker, FileGatewayQBitConfig config, GatewayObjectStore store, Clock clock, String pod) + { + this.instance = instance; + this.invoker = invoker; + this.config = config; + this.store = store; + this.clock = clock; + this.pod = pod == null ? java.util.UUID.randomUUID().toString() : pod; + audit = new AuditWriter(instance); + } + + /******************************************************************************* + ** Share the host-key environment seam; secret values are never persisted or logged. + *******************************************************************************/ + public UploadJobWorker withEnvironment(java.util.function.Function environment) + { + this.environment = environment; + return (this); + } + + + + /******************************************************************************* + ** Each scheduled lane handles one job at a time, without an unbounded work queue. + *******************************************************************************/ + public void start(Duration interval) + { + if(!config.getSftpEnabled()) + { + return; + } + pool = Executors.newScheduledThreadPool(config.getUploadWorkerThreads(), runnable -> + { + Thread thread = new Thread(runnable, "file-gateway-upload"); + thread.setDaemon(true); + return (thread); + }); + for(int lane = 0; lane < config.getUploadWorkerThreads(); lane++) + { + pool.scheduleWithFixedDelay(() -> + { + try + { + runOnce(); + } + catch(Exception e) + { + LOG.warn("Upload worker could not persist job state"); + } + }, interval.toMillis(), interval.toMillis(), TimeUnit.MILLISECONDS); + } + } + + /******************************************************************************* + ** Reconcile stale leases and synchronously deliver at most one eligible job. + *******************************************************************************/ + public boolean runOnce() throws QException + { + if(stopped) + { + return (false); + } + requeueStale(); + for(QRecord candidate : audit.query(FileGatewayTables.UPLOAD_JOB, UploadJobs.equal("status", "QUEUED"))) + { + if(candidate.getValueInstant("nextAttemptAt") != null && candidate.getValueInstant("nextAttemptAt").isAfter(clock.instant())) + { + continue; + } + QRecord claimed = claim(candidate); + if(claimed != null) + { + deliver(claimed); + return (true); + } + } + return (false); + } + + /******************************************************************************* + ** The expected status is checked inside the account transaction; competing pods cannot both win. + *******************************************************************************/ + public QRecord claim(QRecord candidate) throws QException + { + return (audit.withAccountLock("id", candidate.getValue("accountId"), () -> + { + QRecord current = find(candidate); + if(stopped || !"QUEUED".equals(current.getValueString("status")) + || (current.getValueInstant("nextAttemptAt") != null && current.getValueInstant("nextAttemptAt").isAfter(clock.instant()))) + { + return (null); + } + QRecord changes = new QRecord().withValue("id", current.getValue("id")).withValue("status", "RUNNING") + .withValue("claimedBy", pod).withValue("claimedAt", clock.instant()).withValue("startedAt", clock.instant()) + .withValue("attempts", attempts(current) + 1); + return (audit.transitionUpload(changes, "QUEUED") ? find(current) : null); + })); + } + + /******************************************************************************* + ** Requeue abandoned leases up to three starts, fencing late completions by claim time. + *******************************************************************************/ + public void requeueStale() throws QException + { + for(QRecord candidate : audit.query(FileGatewayTables.UPLOAD_JOB, UploadJobs.equal("status", "RUNNING"))) + { + audit.withAccountLock("id", candidate.getValue("accountId"), () -> + { + QRecord job = find(candidate); + if("RUNNING".equals(job.getValueString("status")) && job.getValueInstant("claimedAt") != null + && !job.getValueInstant("claimedAt").plus(config.getJobStaleAfter()).isAfter(clock.instant())) + { + QRecord changes = new QRecord().withValue("id", job.getValue("id")) + .withValue("status", attempts(job) >= 3 ? "FAILED" : "QUEUED").withValue("claimedBy", null).withValue("claimedAt", null) + .withValue("errorMessage", "Upload worker lease expired").withValue("nextAttemptAt", clock.instant()); + if(attempts(job) >= 3) + { + changes.withValue("finishedAt", clock.instant()); + } + audit.update(FileGatewayTables.UPLOAD_JOB, changes); + QRecord account = audit.query(FileGatewayTables.ACCOUNT, UploadJobs.equal("id", job.getValue("accountId"))).getFirst(); + UploadNotifications.send(audit, job, changes, account); + } + return (null); + }); + } + } + + /******************************************************************************* + ** Fresh user context is scoped to each delivery and always restored and revoked. + *******************************************************************************/ + private void deliver(QRecord job) throws QException + { + var previous = QContext.capture(); + try + { + QRecord account = audit.query(FileGatewayTables.ACCOUNT, UploadJobs.equal("id", job.getValue("accountId"))).getFirst(); + if(!"ACTIVE".equals(account.getValueString("status")) + || (account.getValueInstant("expiresAt") != null && !account.getValueInstant("expiresAt").isAfter(clock.instant()))) + { + finish(job, new QRecord().withValue("status", "REJECTED").withValue("errorMessage", "Gateway account is unavailable"), account); + return; + } + try(var user = new WorkerUserSession(instance, invoker, account.getValueString("userId"), "upload-" + job.getValue("id"), clock)) + { + InboundFile file = inbound(job, account); + if(config.getScanner() != null) + { + var scan = QCodeLoader.getAdHoc(FileGatewayScannerInterface.class, config.getScanner()).scan(file); + if(scan == FileGatewayScannerInterface.Result.INFECTED) + { + String quarantine = "file-gateway/quarantine/" + job.getValue("id") + "/" + job.getValueString("fileName"); + store.copy(file.storageKey(), quarantine); + store.delete(file.storageKey()); + finish(job, new QRecord().withValue("status", "REJECTED").withValue("storageKey", quarantine) + .withValue("errorMessage", "Scanner rejected infected file"), account); + return; + } + if(scan != FileGatewayScannerInterface.Result.CLEAN) + { + retry(job, "Scanner unavailable", null); + return; + } + } + if(Boolean.TRUE.equals(account.getValueBoolean("pgpDecryptUploads")) && job.getValueString("fileName").toLowerCase(java.util.Locale.ROOT).matches(".*[.](pgp|gpg)$")) + { + try + { + decrypt(job, account); + file = inbound(job, account); + } + catch(Exception e) + { + finish(job, new QRecord().withValue("status", "REJECTED").withValue("errorMessage", "PGP decryption failed: invalid ciphertext, key, integrity or size limit"), account); + return; + } + } + FileGatewayConsumer consumer = consumer(job, user); + if(consumer == null) + { + finish(job, new QRecord().withValue("status", "REJECTED").withValue("errorMessage", "Unknown consumer: " + job.getValueString("consumerName")), account); + return; + } + FileAcceptance acceptance = consumer.accept(file); + if(acceptance.getStatus() == FileAcceptance.Status.RETRY_LATER) + { + retry(job, acceptance.getReason(), acceptance.getRetryAfter()); + return; + } + QRecord outcome = consumer instanceof BulkLoadConsumer bulk ? bulk.outcome() : new QRecord() + .withValue("status", acceptance.getStatus() == FileAcceptance.Status.ACCEPTED ? "SUCCEEDED" : "REJECTED") + .withValue("errorMessage", acceptance.getReason()); + outcome.withValue("consumerReference", acceptance.getConsumerReference()).withValue("processUuid", job.getValue("processUuid")); + finish(job, outcome, account); + } + } + catch(Exception e) + { + retry(job, "Consumer delivery failed", null); + } + finally + { + QContext.init(previous); + } + } + + /******************************************************************************* + ** Keep the original for the archive and persist authenticated plaintext for retries. + *******************************************************************************/ + private void decrypt(QRecord job, QRecord account) throws IOException, QException + { + if(job.getValue("plaintextStorageKey") != null) + { + return; + } + String key = "file-gateway/plaintext/" + job.getValue("id") + "/" + java.util.UUID.randomUUID(); + String pass = config.getPgpPassphraseEnvVarName() == null ? null : environment.apply(config.getPgpPassphraseEnvVarName()); + char[] password = pass == null ? new char[0] : pass.toCharArray(); + try(var source = store.open(job.getValueString("storageKey")); + var plaintext = com.kingsrook.qbits.filegateway.pgp.PgpStreams.decrypt(source, + config.getPgpPrivateKeyEnvVarName() == null ? null : environment.apply(config.getPgpPrivateKeyEnvVarName()), password, + (account.getValueInteger("maxFileSizeMb") == null ? config.getDefaultMaxFileSizeMb() : account.getValueInteger("maxFileSizeMb")) * UploadJobs.MIB)) + { + var receipt = store.put(key, plaintext); + QRecord changes = new QRecord().withValue("plaintextStorageKey", key).withValue("plaintextSha256", receipt.sha256()).withValue("plaintextSizeBytes", receipt.size()); + if(!persistOwned(job, changes)) + { + throw (new IOException("Upload claim expired during PGP decryption")); + } + job.getValues().putAll(changes.getValues()); + } + catch(Exception e) + { + store.delete(key); + throw (new IOException("PGP decryption failed", e)); + } + finally + { + java.util.Arrays.fill(password, '\0'); + } + } + + + + /******************************************************************************* + ** Registry lookup is by declared name, with the built-in bulk-load consumer reserved. + *******************************************************************************/ + private FileGatewayConsumer consumer(QRecord job, WorkerUserSession user) + { + String name = job.getValueString("consumerName"); + if("bulk-load".equals(name)) + { + return (new BulkLoadConsumer(user, job)); + } + for(var reference : FileGatewayConsumersMetaData.ofOrWithNew(instance).getConsumers()) + { + FileGatewayConsumer consumer = QCodeLoader.getAdHoc(FileGatewayConsumer.class, reference); + if(name.equals(consumer.getName())) + { + return (consumer); + } + } + return (null); + } + + /******************************************************************************* + ** Immutable delivery envelope; each openStream gets an independent object stream. + *******************************************************************************/ + private InboundFile inbound(QRecord job, QRecord account) throws QException + { + String endpointName = job.getValue("aliasId") == null ? null + : audit.query(FileGatewayTables.ALIAS, UploadJobs.equal("id", job.getValue("aliasId"))).getFirst().getValueString("name"); + boolean plaintext = job.getValue("plaintextStorageKey") != null; + String storageKey = job.getValueString(plaintext ? "plaintextStorageKey" : "storageKey"); + String filename = job.getValueString("fileName"); + if(plaintext) + { + filename = filename.substring(0, filename.length() - 4); + } + return (new InboundFile(job.getValueInteger("id"), endpointName, account.getValueString("username"), account.getValueString("userId"), + job.getValueString("remoteAddress"), job.getValueString("path"), filename, job.getValueLong(plaintext ? "plaintextSizeBytes" : "sizeBytes"), + job.getValueString(plaintext ? "plaintextSha256" : "sha256"), job.getValueInstant("createDate"), config.getFileGatewayBucketAlias(), storageKey, () -> + { + try + { + return (store.open(storageKey)); + } + catch(IOException e) + { + throw (new UncheckedIOException(e)); + } + })); + } + + /******************************************************************************* + ** Retry delays have a five-attempt cap; a consumer can request a longer delay. + *******************************************************************************/ + private void retry(QRecord job, String message, Duration requested) throws QException + { + if(stopped) + { + return; + } + if(attempts(job) >= 5) + { + QRecord account = audit.query(FileGatewayTables.ACCOUNT, UploadJobs.equal("id", job.getValue("accountId"))).getFirst(); + try + { + finish(job, new QRecord().withValue("status", "FAILED").withValue("errorMessage", message), account); + } + catch(IOException e) + { + throw (new QException("Unable to finish failed upload", e)); + } + return; + } + long delay = RETRY_SECONDS[Math.min(Math.max(0, attempts(job) - 1), RETRY_SECONDS.length - 1)]; + if(requested != null) + { + delay = Math.max(delay, requested.toSeconds()); + } + persistOwned(job, new QRecord().withValue("status", "QUEUED").withValue("errorMessage", message) + .withValue("nextAttemptAt", clock.instant().plusSeconds(delay)).withValue("claimedBy", null).withValue("claimedAt", null)); + } + + /******************************************************************************* + ** Results are bounded by the process summary, never by the original upload size. + *******************************************************************************/ + private void finish(QRecord job, QRecord outcome, QRecord account) throws QException, IOException + { + String summary = outcome.getValueString("resultSummary"); + if(summary == null) + { + summary = outcome.getValueString("errorMessage"); + } + if(summary == null) + { + summary = outcome.getValueString("status"); + } + String key = "file-gateway/results/" + job.getValue("accountId") + "/" + job.getValue("id") + "/" + java.util.UUID.randomUUID() + ".csv"; + byte[] csv = ("row,field,message\r\n,,\"" + summary.replace("\"", "\"\"") + "\"\r\n").getBytes(StandardCharsets.UTF_8); + String detailed = outcome.getValueString("_resultsCsv"); + outcome.getValues().remove("_resultsCsv"); + if(detailed != null) + { + csv = detailed.getBytes(StandardCharsets.UTF_8); + } + store.put(key, new ByteArrayInputStream(csv)); + outcome.withValue("resultSummary", summary).withValue("resultsStorageKey", key).withValue("resultsSizeBytes", (long) csv.length) + .withValue("finishedAt", clock.instant()).withValue("nextAttemptAt", null).withValue("claimedBy", null).withValue("claimedAt", null); + if(persistOwned(job, outcome)) + { + UploadNotifications.send(audit, job, outcome, account); + } + else + { + store.delete(key); + } + } + + /******************************************************************************* + ** Late workers cannot overwrite a requeued or newly claimed job. + *******************************************************************************/ + private boolean persistOwned(QRecord job, QRecord changes) throws QException + { + return (audit.withAccountLock("id", job.getValue("accountId"), () -> + { + QRecord current = find(job); + if(!"RUNNING".equals(current.getValueString("status")) || !Objects.equals(pod, current.getValueString("claimedBy")) + || !Objects.equals(job.getValueInstant("claimedAt"), current.getValueInstant("claimedAt"))) + { + return (false); + } + return (audit.transitionUpload(changes.withValue("id", job.getValue("id")), "RUNNING")); + })); + } + + /******************************************************************************* + ** Reload a gateway job inside a short bookkeeping transaction. + *******************************************************************************/ + private QRecord find(QRecord job) throws QException + { + return (audit.query(FileGatewayTables.UPLOAD_JOB, UploadJobs.equal("id", job.getValue("id"))).getFirst()); + } + + /******************************************************************************* + ** Null attempts on older rows count as zero. + *******************************************************************************/ + private int attempts(QRecord job) + { + return (job.getValueInteger("attempts") == null ? 0 : job.getValueInteger("attempts")); + } + + /******************************************************************************* + ** Interrupt in-flight delivery and release only this pod's leases. + *******************************************************************************/ + @Override + public void close() throws QException + { + stopped = true; + if(pool != null) + { + pool.shutdownNow(); + } + for(QRecord job : audit.query(FileGatewayTables.UPLOAD_JOB, UploadJobs.equal("claimedBy", pod))) + { + persistOwned(job, new QRecord().withValue("status", "QUEUED").withValue("claimedBy", null).withValue("claimedAt", null)); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/upload/UploadJobs.java b/src/main/java/com/kingsrook/qbits/filegateway/upload/UploadJobs.java new file mode 100644 index 0000000..fcb136f --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/upload/UploadJobs.java @@ -0,0 +1,336 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.upload; + +import java.io.IOException; +import java.nio.channels.SeekableByteChannel; +import java.time.Clock; +import java.time.ZoneOffset; +import java.time.format.DateTimeFormatter; +import java.util.List; +import java.util.Objects; +import java.util.UUID; +import com.kingsrook.qbits.filegateway.FileGatewayQBitConfig; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qbits.filegateway.sftp.AuditWriter; +import com.kingsrook.qbits.filegateway.sftp.GatewaySession; +import com.kingsrook.qbits.filegateway.storage.GatewayObjectStore; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.data.QRecord; + +/******************************************************************************* + ** Durable upload lifecycle. Account row transactions serialize quotas and renames across pods. + *******************************************************************************/ +public class UploadJobs +{ + public static final long MIB = 1024L * 1024L; + private final AuditWriter audit; + private final GatewayObjectStore store; + private final FileGatewayQBitConfig config; + private final Clock clock; + + /******************************************************************************* + ** Share storage and persistence between the virtual tree and workers. + *******************************************************************************/ + public UploadJobs(AuditWriter audit, GatewayObjectStore store, FileGatewayQBitConfig config, Clock clock) + { + this.audit = audit; + this.store = store; + this.config = config; + this.clock = clock; + } + + /******************************************************************************* + ** The target was resolved through the user's metadata; endpoint routing can reuse this seam. + *******************************************************************************/ + public SeekableByteChannel receive(GatewaySession session, String path, String table, Integer profile) throws IOException + { + return (receive(session, path, table, profile, null)); + } + + + + /******************************************************************************* + ** Persist the resolved endpoint with the upload before any bytes are accepted. + *******************************************************************************/ + public SeekableByteChannel receive(GatewaySession session, String path, String table, Integer profile, QRecord alias) throws IOException + { + String name = path.substring(path.lastIndexOf('/') + 1); + if(name.isBlank() || name.contains(":") || name.contains("\\") || name.chars().anyMatch(Character::isISOControl)) + { + throw (new IOException("Invalid upload filename")); + } + if(!session.operationAllowed()) + { + throw (new IOException("Operation limit exceeded")); + } + try + { + String date = DateTimeFormatter.ofPattern("yyyy/MM/dd").withZone(ZoneOffset.UTC).format(clock.instant()); + String key = "file-gateway/uploads/" + session.account.getValue("id") + "/" + date + "/" + UUID.randomUUID() + "/" + name; + QRecord job = audit.insert(FileGatewayTables.UPLOAD_JOB, new QRecord() + .withValue("accountId", session.account.getValue("id")).withValue("connectionLogId", session.connectionId) + .withValue("remoteAddress", session.remoteIp).withValue("tableName", table).withValue("savedBulkLoadProfileId", profile) + .withValue("aliasId", alias == null ? null : alias.getValue("id")) + .withValue("consumerName", alias != null && "INBOX".equals(alias.getValueString("kind")) ? alias.getValueString("consumerName") : "bulk-load").withValue("path", path).withValue("fileName", name).withValue("storageKey", key) + .withValue("status", "RECEIVING").withValue("sizeBytes", 0L).withValue("attempts", 0).withValue("createDate", clock.instant())); + long started = System.nanoTime(); + return (new UploadChannel(store, key, Math.toIntExact(config.getUploadPartSizeMb() * MIB), config.getUploadReorderWindowMb() * MIB, new UploadChannel.Lifecycle() + { + @Override + public void reserve(long size) throws IOException + { + reserveBytes(job, size); + } + + @Override + public void complete(GatewayObjectStore.Receipt receipt) throws IOException + { + try + { + if(session.ssh.isClosing() || !session.ssh.isOpen()) + { + throw (new IOException("Connection ended before upload close")); + } + job.withValue("sizeBytes", receipt.size()).withValue("sha256", receipt.sha256()); + audit.update(FileGatewayTables.UPLOAD_JOB, new QRecord().withValue("id", job.getValue("id")) + .withValue("sizeBytes", receipt.size()).withValue("sha256", receipt.sha256())); + if(!temporary(name)) + { + queue(job, path, name); + } + log(session, job, "OK", null, started); + } + catch(QException e) + { + throw (new IOException("Unable to finalize upload", e)); + } + } + + @Override + public void abort(String message, boolean limit) throws IOException + { + try + { + QRecord outcome = new QRecord().withValue("id", job.getValue("id")) + .withValue("status", limit ? "REJECTED" : "ABORTED").withValue("errorMessage", message).withValue("finishedAt", clock.instant()); + audit.update(FileGatewayTables.UPLOAD_JOB, outcome); + UploadNotifications.send(audit, job, outcome, session.account); + log(session, job, limit ? "LIMIT" : "ERROR", message, started); + } + catch(QException e) + { + throw (new IOException("Unable to audit aborted upload", e)); + } + } + })); + } + catch(QException e) + { + throw (new IOException("Unable to create upload job", e)); + } + } + + /******************************************************************************* + ** Charge the high-water byte count under an account lock, including concurrent uploads. + *******************************************************************************/ + private void reserveBytes(QRecord job, long size) throws IOException + { + try + { + String rejection = audit.withAccountLock("id", job.getValue("accountId"), () -> + { + QRecord account = audit.query(FileGatewayTables.ACCOUNT, equal("id", job.getValue("accountId"))).getFirst(); + Integer maximum = account.getValueInteger("maxFileSizeMb"); + if(size > (maximum == null ? config.getDefaultMaxFileSizeMb() : maximum) * MIB) + { + return ("File size limit exceeded"); + } + Integer daily = account.getValueInteger("maxUploadMbPerDay"); + if(daily != null) + { + var midnight = clock.instant().atZone(ZoneOffset.UTC).toLocalDate().atStartOfDay(ZoneOffset.UTC).toInstant(); + long used = 0; + for(QRecord other : audit.query(FileGatewayTables.UPLOAD_JOB, equal("accountId", job.getValue("accountId")) + .withCriteria(new QFilterCriteria("createDate", QCriteriaOperator.GREATER_THAN_OR_EQUALS, midnight)))) + { + if(!Objects.equals(other.getValue("id"), job.getValue("id"))) + { + used += other.getValueLong("sizeBytes") == null ? 0 : other.getValueLong("sizeBytes"); + } + } + if(size > daily * MIB - used) + { + return ("Daily upload quota exceeded"); + } + } + audit.update(FileGatewayTables.UPLOAD_JOB, new QRecord().withValue("id", job.getValue("id")).withValue("sizeBytes", size)); + return (null); + }); + if(rejection != null) + { + throw (new UploadLimitException(rejection)); + } + job.withValue("sizeBytes", size); + } + catch(QException e) + { + throw (new IOException("Unable to reserve upload quota", e)); + } + } + + /******************************************************************************* + ** Close and rename share one idempotent transition, with target-aware deduplication. + *******************************************************************************/ + public void queue(QRecord job, String path, String name) throws QException + { + audit.withAccountLock("id", job.getValue("accountId"), () -> + { + QRecord current = audit.query(FileGatewayTables.UPLOAD_JOB, equal("id", job.getValue("id"))).getFirst(); + if(!"RECEIVING".equals(current.getValueString("status")) || current.getValueString("sha256") == null) + { + return (null); + } + QRecord update = new QRecord().withValue("id", job.getValue("id")).withValue("path", path).withValue("fileName", name); + if(temporary(name)) + { + audit.update(FileGatewayTables.UPLOAD_JOB, update); + return (null); + } + update.withValue("status", "QUEUED").withValue("queuedAt", clock.instant()); + QRecord account = audit.query(FileGatewayTables.ACCOUNT, equal("id", job.getValue("accountId"))).getFirst(); + if(Boolean.TRUE.equals(account.getValueBoolean("dedupeUploads"))) + { + for(QRecord candidate : audit.query(FileGatewayTables.UPLOAD_JOB, equal("accountId", job.getValue("accountId")) + .withCriteria(new QFilterCriteria("sha256", QCriteriaOperator.EQUALS, current.getValue("sha256"))) + .withCriteria(new QFilterCriteria("status", QCriteriaOperator.EQUALS, "SUCCEEDED")) + .withCriteria(new QFilterCriteria("finishedAt", QCriteriaOperator.GREATER_THAN_OR_EQUALS, clock.instant().minus(config.getDedupeWindow()))))) + { + if(Objects.equals(current.getValue("tableName"), candidate.getValue("tableName")) + && Objects.equals(current.getValue("savedBulkLoadProfileId"), candidate.getValue("savedBulkLoadProfileId")) + && Objects.equals(current.getValue("aliasId"), candidate.getValue("aliasId")) + && Objects.equals(current.getValue("consumerName"), candidate.getValue("consumerName"))) + { + update.withValue("status", "DUPLICATE").withValue("duplicateOfJobId", candidate.getValue("id")) + .withValue("finishedAt", clock.instant()).withValue("resultSummary", "Skipped: duplicate of upload " + candidate.getValue("id")); + break; + } + } + } + audit.update(FileGatewayTables.UPLOAD_JOB, update); + return (null); + }); + } + + /******************************************************************************* + ** Closed temporary files can be renamed only within their original upload folder. + *******************************************************************************/ + public void rename(Integer accountId, String source, String target) throws IOException + { + if(!source.substring(0, source.lastIndexOf('/')).equals(target.substring(0, target.lastIndexOf('/')))) + { + throw (new IOException("Rename must stay inside one upload folder")); + } + String name = target.substring(target.lastIndexOf('/') + 1); + if(name.isBlank() || name.contains(":") || name.contains("\\") || name.chars().anyMatch(Character::isISOControl)) + { + throw (new IOException("Invalid upload filename")); + } + try + { + var rows = audit.query(FileGatewayTables.UPLOAD_JOB, equal("accountId", accountId) + .withCriteria(new QFilterCriteria("path", QCriteriaOperator.EQUALS, source)) + .withCriteria(new QFilterCriteria("status", QCriteriaOperator.EQUALS, "RECEIVING"))); + if(rows.size() != 1 || rows.getFirst().getValueString("sha256") == null || !temporary(rows.getFirst().getValueString("fileName"))) + { + throw (new IOException("Only a completed temporary upload can be renamed")); + } + queue(rows.getFirst(), target, target.substring(target.lastIndexOf('/') + 1)); + } + catch(QException e) + { + throw (new IOException("Unable to rename upload", e)); + } + } + + /******************************************************************************* + ** Account-scoped records for a visible table. + *******************************************************************************/ + public List forTable(Integer accountId, String table) throws IOException + { + try + { + return (audit.query(FileGatewayTables.UPLOAD_JOB, equal("accountId", accountId) + .withCriteria(new QFilterCriteria("tableName", QCriteriaOperator.EQUALS, table)))); + } + catch(QException e) + { + throw (new IOException("Unable to list upload jobs", e)); + } + } + + /******************************************************************************* + ** Acknowledging either artifact hides both results and the sidecar. + *******************************************************************************/ + public void acknowledge(QRecord job) throws IOException + { + try + { + audit.update(FileGatewayTables.UPLOAD_JOB, new QRecord().withValue("id", job.getValue("id")).withValue("acknowledgedAt", clock.instant())); + } + catch(QException e) + { + throw (new IOException("Unable to acknowledge upload", e)); + } + } + + /******************************************************************************* + ** Configured temporary filename suffixes. + *******************************************************************************/ + public boolean temporary(String name) + { + return (config.getTemporaryUploadSuffixes().stream().anyMatch(name::endsWith)); + } + + /******************************************************************************* + ** Common equality filter for gateway bookkeeping only. + *******************************************************************************/ + public static QQueryFilter equal(String field, java.io.Serializable value) + { + return (new QQueryFilter(new QFilterCriteria(field, QCriteriaOperator.EQUALS, value))); + } + + /******************************************************************************* + ** Upload audit includes measured bytes, digest and the original connection identity. + *******************************************************************************/ + private void log(GatewaySession session, QRecord job, String outcome, String message, long started) throws QException + { + audit.insert(FileGatewayTables.TRANSFER_LOG, new QRecord().withValue("accountId", session.account.getValue("id")) + .withValue("connectionLogId", session.connectionId).withValue("operation", "UPLOAD").withValue("path", job.getValue("path")) + .withValue("objectKind", "PROFILE").withValue("objectId", job.getValueString("savedBulkLoadProfileId")) + .withValue("uploadJobId", job.getValue("id")).withValue("bytes", job.getValue("sizeBytes")).withValue("sha256", job.getValue("sha256")) + .withValue("durationMs", (System.nanoTime() - started) / 1_000_000).withValue("outcome", outcome).withValue("message", message) + .withValue("clientChannel", "sftp/" + session.id)); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/upload/UploadLimitException.java b/src/main/java/com/kingsrook/qbits/filegateway/upload/UploadLimitException.java new file mode 100644 index 0000000..47060d7 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/upload/UploadLimitException.java @@ -0,0 +1,37 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.upload; + +import java.io.IOException; + +/******************************************************************************* + ** An upload exceeded the account's byte budget. + *******************************************************************************/ +public class UploadLimitException extends IOException +{ + /******************************************************************************* + ** Safe reason to expose through SFTP and the transfer audit. + *******************************************************************************/ + public UploadLimitException(String message) + { + super(message); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/upload/UploadNotifications.java b/src/main/java/com/kingsrook/qbits/filegateway/upload/UploadNotifications.java new file mode 100644 index 0000000..8667c8d --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/upload/UploadNotifications.java @@ -0,0 +1,80 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.upload; + +import java.util.List; +import java.util.Locale; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qbits.filegateway.sftp.AuditWriter; +import com.kingsrook.qqq.backend.core.alerts.AlertLink; +import com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData; +import com.kingsrook.qqq.backend.core.alerts.NotificationAudience; +import com.kingsrook.qqq.backend.core.alerts.NotificationSignal; +import com.kingsrook.qqq.backend.core.alerts.QAlerts; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.data.QRecord; + +/******************************************************************************* + ** Core notification signals for terminal upload outcomes, including transfer limits. + *******************************************************************************/ +public final class UploadNotifications +{ + /******************************************************************************* + ** Static signal publisher. + *******************************************************************************/ + private UploadNotifications() + { + } + + /******************************************************************************* + ** No alerts producer is a no-op; service account messages go to their owner. + *******************************************************************************/ + public static void send(AuditWriter audit, QRecord job, QRecord outcome, QRecord account) throws QException + { + String status = outcome.getValueString("status"); + if(!List.of("FAILED", "PARTIAL", "REJECTED").contains(status) + && !("SUCCEEDED".equals(status) && Boolean.TRUE.equals(account.getValueBoolean("notifyOnSuccess")))) + { + return; + } + audit.system(() -> + { + var alerts = AlertsInstanceMetaData.of(QContext.getQInstance()); + if(alerts == null || alerts.getSink() == null) + { + return (null); + } + String recipient = "SERVICE".equals(account.getValueString("accountType")) ? account.getValueString("ownerUserId") : account.getValueString("userId"); + if(recipient == null || recipient.isBlank()) + { + return (null); + } + QAlerts.notify(new NotificationSignal().withTitle("Upload " + status.toLowerCase(Locale.ROOT)) + .withBody(outcome.getValueString("resultSummary") == null ? outcome.getValueString("errorMessage") : outcome.getValueString("resultSummary")) + .withSourceName("file-gateway").withSourceKey(job.getValueString("id")) + .withSourceTableName(FileGatewayTables.UPLOAD_JOB).withSourceRecordId(job.getValueString("id")) + .withAudience(NotificationAudience.user(recipient)) + .withLink(AlertLink.toRecord(FileGatewayTables.UPLOAD_JOB, job.getValue("id"), "Upload job"))); + return (null); + }); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/upload/WorkerUserSession.java b/src/main/java/com/kingsrook/qbits/filegateway/upload/WorkerUserSession.java new file mode 100644 index 0000000..f456d4d --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/upload/WorkerUserSession.java @@ -0,0 +1,114 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.upload; + +import java.io.IOException; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import com.fasterxml.jackson.databind.JsonNode; +import com.kingsrook.qbits.filegateway.permissions.FileGatewayPermissions; +import com.kingsrook.qbits.filegateway.sftp.AuditWriter; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleDispatcher; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiCredentials; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInProcessSessions; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInvoker; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiRequest; + +/******************************************************************************* + ** Independent, short-lived account-user credentials for an upload worker. + *******************************************************************************/ +public final class WorkerUserSession implements AutoCloseable +{ + private final QInstance instance; + private final UiApiInvoker invoker; + private final String userId; + private final String detail; + private final Clock clock; + private String token; + private Instant minted; + + /******************************************************************************* + ** The caller owns restoration of its QContext after the job. + *******************************************************************************/ + public WorkerUserSession(QInstance instance, UiApiInvoker invoker, String userId, String detail, Clock clock) throws QException + { + this.instance = instance; + this.invoker = invoker; + this.userId = userId; + this.detail = detail; + this.clock = clock; + refresh(); + } + + /******************************************************************************* + ** Re-mint permissions even after the originating SSH session has disconnected. + *******************************************************************************/ + private void refresh() throws QException + { + if(minted == null || !minted.plusSeconds(300).isAfter(clock.instant())) + { + var module = new QAuthenticationModuleDispatcher().getQModule(instance.getAuthentication()); + var user = new AuditWriter(instance).system(() -> module.createAutomatedSessionForUser(instance, userId)); + if(user == null || !user.hasPermission(FileGatewayPermissions.CONNECT)) + { + throw (new QException("Missing file_gateway.connect")); + } + user.setClientChannel("sftp/" + detail); + String replacement = UiApiInProcessSessions.register(user, Duration.ofMinutes(10)); + UiApiInProcessSessions.revoke(token); + token = replacement; + minted = clock.instant(); + QContext.init(instance, user); + } + } + + /******************************************************************************* + ** Only published operations, with fixed channel identity and user credentials. + *******************************************************************************/ + public JsonNode invoke(UiApiRequest request) throws IOException, QException + { + refresh(); + if(invoker.findOperation(request.getMethod(), request.getPath()).isEmpty()) + { + throw (new IOException("Gateway requires a spec-defined UI API operation")); + } + var response = invoker.invoke(request.withCredentials(UiApiCredentials.ofInProcessSession(token)) + .withInvokedBy("file-gateway").withClientDetail(detail)); + if(!response.isSuccess()) + { + throw (new IOException("UI API request failed with status " + response.getStatus())); + } + return (response.getJson()); + } + + /******************************************************************************* + ** Never retain worker credentials after a job. + *******************************************************************************/ + @Override + public void close() + { + UiApiInProcessSessions.revoke(token); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/vfs/GatewayFileAttributes.java b/src/main/java/com/kingsrook/qbits/filegateway/vfs/GatewayFileAttributes.java new file mode 100644 index 0000000..2d568b8 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/vfs/GatewayFileAttributes.java @@ -0,0 +1,173 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.vfs; + + +import java.nio.file.attribute.FileTime; +import java.nio.file.attribute.GroupPrincipal; +import java.nio.file.attribute.PosixFileAttributes; +import java.nio.file.attribute.PosixFilePermission; +import java.nio.file.attribute.UserPrincipal; +import java.util.Set; + + +/******************************************************************************* + ** Virtual attributes: generated exports have size zero; stored objects report persisted size. + *******************************************************************************/ +public record GatewayFileAttributes(VirtualTree.Node node, FileTime time) implements PosixFileAttributes +{ + + + + /******************************************************************************* + ** Virtual attribute lastModifiedTime. + *******************************************************************************/ + @Override + public FileTime lastModifiedTime() + { + return (time); + } + + + + /******************************************************************************* + ** Virtual attribute lastAccessTime. + *******************************************************************************/ + @Override + public FileTime lastAccessTime() + { + return (time); + } + + + + /******************************************************************************* + ** Virtual attribute creationTime. + *******************************************************************************/ + @Override + public FileTime creationTime() + { + return (time); + } + + + + /******************************************************************************* + ** Virtual attribute isRegularFile. + *******************************************************************************/ + @Override + public boolean isRegularFile() + { + return (!node.directory()); + } + + + + /******************************************************************************* + ** Virtual attribute isDirectory. + *******************************************************************************/ + @Override + public boolean isDirectory() + { + return (node.directory()); + } + + + + /******************************************************************************* + ** Virtual attribute isSymbolicLink. + *******************************************************************************/ + @Override + public boolean isSymbolicLink() + { + return (false); + } + + + + /******************************************************************************* + ** Virtual attribute isOther. + *******************************************************************************/ + @Override + public boolean isOther() + { + return (false); + } + + + + /******************************************************************************* + ** Virtual attribute size. + *******************************************************************************/ + @Override + public long size() + { + return (node.size()); + } + + + + /******************************************************************************* + ** Virtual attribute fileKey. + *******************************************************************************/ + @Override + public Object fileKey() + { + return (node.path()); + } + + + + /******************************************************************************* + ** Virtual attribute owner. + *******************************************************************************/ + @Override + public UserPrincipal owner() + { + return (() -> "gateway"); + } + + + + /******************************************************************************* + ** Virtual attribute group. + *******************************************************************************/ + @Override + public GroupPrincipal group() + { + return (() -> "gateway"); + } + + + + /******************************************************************************* + ** Virtual attribute permissions. + *******************************************************************************/ + @Override + public Set permissions() + { + if(node.directory() && "PROFILE".equals(node.kind()) && node.objectId() != null) + { + return (Set.of(PosixFilePermission.OWNER_READ, PosixFilePermission.OWNER_WRITE, PosixFilePermission.OWNER_EXECUTE)); + } + return (node.directory() ? Set.of(PosixFilePermission.OWNER_READ, PosixFilePermission.OWNER_EXECUTE) : Set.of(PosixFilePermission.OWNER_READ)); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/vfs/GatewayFileSystem.java b/src/main/java/com/kingsrook/qbits/filegateway/vfs/GatewayFileSystem.java new file mode 100644 index 0000000..89b7413 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/vfs/GatewayFileSystem.java @@ -0,0 +1,201 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.vfs; + + +import java.nio.file.ClosedFileSystemException; +import java.nio.file.FileStore; +import java.nio.file.FileSystem; +import java.nio.file.Path; +import java.nio.file.PathMatcher; +import java.nio.file.WatchService; +import java.nio.file.attribute.UserPrincipalLookupService; +import java.nio.file.spi.FileSystemProvider; +import java.util.List; +import java.util.Set; + + +/******************************************************************************* + ** Session-private NIO filesystem over a UI API virtual tree, with no native mounts. + *******************************************************************************/ +public class GatewayFileSystem extends FileSystem +{ + private final VirtualTree tree; + private final GatewayFileSystemProvider provider; + private boolean open = true; + + + + /******************************************************************************* + ** Create a purely virtual filesystem. + *******************************************************************************/ + public GatewayFileSystem(VirtualTree tree) + { + this.tree = tree; + this.provider = new GatewayFileSystemProvider(this); + } + + + + /******************************************************************************* + ** Access the live virtual tree. + *******************************************************************************/ + public VirtualTree tree() + { + if(!open) + { + throw (new ClosedFileSystemException()); + } + return (tree); + } + + + + /******************************************************************************* + ** Virtual filesystem provider. + *******************************************************************************/ + @Override + public FileSystemProvider provider() + { + return (provider); + } + + + + /******************************************************************************* + ** Virtual filesystem close. + *******************************************************************************/ + @Override + public void close() + { + open = false; + } + + + + /******************************************************************************* + ** Virtual filesystem isOpen. + *******************************************************************************/ + @Override + public boolean isOpen() + { + return (open); + } + + + + /******************************************************************************* + ** Virtual filesystem isReadOnly. + *******************************************************************************/ + @Override + public boolean isReadOnly() + { + return (false); + } + + + + /******************************************************************************* + ** Virtual filesystem getSeparator. + *******************************************************************************/ + @Override + public String getSeparator() + { + return ("/"); + } + + + + /******************************************************************************* + ** Virtual filesystem getRootDirectories. + *******************************************************************************/ + @Override + public Iterable getRootDirectories() + { + return (List.of(getPath("/"))); + } + + + + /******************************************************************************* + ** Virtual filesystem getFileStores. + *******************************************************************************/ + @Override + public Iterable getFileStores() + { + return (List.of()); + } + + + + /******************************************************************************* + ** Virtual filesystem supportedFileAttributeViews. + *******************************************************************************/ + @Override + public Set supportedFileAttributeViews() + { + return (Set.of("basic", "posix")); + } + + + + /******************************************************************************* + ** Virtual filesystem getPath. + *******************************************************************************/ + @Override + public Path getPath(String first, String... more) + { + return (new GatewayPath(this, first + (more.length == 0 ? "" : "/" + String.join("/", more)))); + } + + + + /******************************************************************************* + ** Virtual filesystem getPathMatcher. + *******************************************************************************/ + @Override + public PathMatcher getPathMatcher(String syntaxAndPattern) + { + throw (new UnsupportedOperationException("Path matching is unavailable")); + } + + + + /******************************************************************************* + ** Virtual filesystem getUserPrincipalLookupService. + *******************************************************************************/ + @Override + public UserPrincipalLookupService getUserPrincipalLookupService() + { + throw (new UnsupportedOperationException("Native users are unavailable")); + } + + + + /******************************************************************************* + ** Virtual filesystem newWatchService. + *******************************************************************************/ + @Override + public WatchService newWatchService() + { + throw (new UnsupportedOperationException("Watching is unavailable")); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/vfs/GatewayFileSystemProvider.java b/src/main/java/com/kingsrook/qbits/filegateway/vfs/GatewayFileSystemProvider.java new file mode 100644 index 0000000..13d9507 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/vfs/GatewayFileSystemProvider.java @@ -0,0 +1,388 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.vfs; + + +import java.io.IOException; +import java.net.URI; +import java.nio.channels.SeekableByteChannel; +import java.nio.file.AccessMode; +import java.nio.file.CopyOption; +import java.nio.file.DirectoryStream; +import java.nio.file.FileStore; +import java.nio.file.FileSystem; +import java.nio.file.FileSystemAlreadyExistsException; +import java.nio.file.LinkOption; +import java.nio.file.OpenOption; +import java.nio.file.Path; +import java.nio.file.ProviderMismatchException; +import java.nio.file.StandardOpenOption; +import java.nio.file.attribute.BasicFileAttributes; +import java.nio.file.attribute.FileAttribute; +import java.nio.file.attribute.FileAttributeView; +import java.nio.file.attribute.FileTime; +import java.nio.file.spi.FileSystemProvider; +import java.time.Instant; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.Set; + + +/******************************************************************************* + ** NIO operations resolve against the virtual tree and its explicitly supported upload artifacts. + *******************************************************************************/ +public class GatewayFileSystemProvider extends FileSystemProvider +{ + private final GatewayFileSystem fileSystem; + + + + /******************************************************************************* + ** Bind one virtual filesystem. + *******************************************************************************/ + public GatewayFileSystemProvider(GatewayFileSystem fileSystem) + { + this.fileSystem = fileSystem; + } + + + + /******************************************************************************* + ** Canonical virtual path in this session. + *******************************************************************************/ + private String name(Path path) + { + if(path.getFileSystem() != fileSystem) + { + throw (new ProviderMismatchException()); + } + return (path.toAbsolutePath().normalize().toString()); + } + + + + /******************************************************************************* + ** Virtual provider getScheme. + *******************************************************************************/ + @Override + public String getScheme() + { + return ("gateway"); + } + + + + /******************************************************************************* + ** Virtual provider newFileSystem. + *******************************************************************************/ + @Override + public FileSystem newFileSystem(URI uri, Map env) + { + throw (new FileSystemAlreadyExistsException()); + } + + + + /******************************************************************************* + ** Virtual provider getFileSystem. + *******************************************************************************/ + @Override + public FileSystem getFileSystem(URI uri) + { + return (fileSystem); + } + + + + /******************************************************************************* + ** Virtual provider getPath. + *******************************************************************************/ + @Override + public Path getPath(URI uri) + { + return (fileSystem.getPath(uri.getPath())); + } + + + + /******************************************************************************* + ** Virtual provider newByteChannel. + *******************************************************************************/ + @Override + public SeekableByteChannel newByteChannel(Path path, Set options, FileAttribute... attrs) throws IOException + { + boolean write = options.contains(StandardOpenOption.WRITE) || options.contains(StandardOpenOption.APPEND) || options.contains(StandardOpenOption.CREATE) || options.contains(StandardOpenOption.CREATE_NEW); + if(options.contains(StandardOpenOption.APPEND)) + { + fileSystem.tree().deny(name(path), "UPLOAD", "Appending to stored uploads is unsupported"); + } + return (write ? fileSystem.tree().write(name(path)) : fileSystem.tree().read(name(path))); + } + + + + /******************************************************************************* + ** Virtual provider newDirectoryStream. + *******************************************************************************/ + @Override + public DirectoryStream newDirectoryStream(Path dir, DirectoryStream.Filter filter) throws IOException + { + List entries = new ArrayList<>(); + for(VirtualTree.Node node : fileSystem.tree().list(name(dir))) + { + Path path = fileSystem.getPath(node.path()); + if(filter.accept(path)) + { + entries.add(path); + } + } + return (new GatewayDirectoryStream(entries)); + } + + + + /******************************************************************************* + ** Virtual provider createDirectory. + *******************************************************************************/ + @Override + public void createDirectory(Path dir, FileAttribute... attrs) throws IOException + { + fileSystem.tree().deny(name(dir), "UPLOAD", "mkdir is not supported"); + } + + + + /******************************************************************************* + ** Virtual provider delete. + *******************************************************************************/ + @Override + public void delete(Path path) throws IOException + { + fileSystem.tree().remove(name(path)); + } + + + + /******************************************************************************* + ** Virtual provider copy. + *******************************************************************************/ + @Override + public void copy(Path source, Path target, CopyOption... options) throws IOException + { + fileSystem.tree().deny(name(target), "UPLOAD", "Copy is not supported"); + } + + + + /******************************************************************************* + ** Virtual provider move. + *******************************************************************************/ + @Override + public void move(Path source, Path target, CopyOption... options) throws IOException + { + fileSystem.tree().rename(name(source), name(target)); + } + + + + /******************************************************************************* + ** Virtual provider isSameFile. + *******************************************************************************/ + @Override + public boolean isSameFile(Path path, Path path2) + { + return (name(path).equals(name(path2))); + } + + + + /******************************************************************************* + ** Virtual provider isHidden. + *******************************************************************************/ + @Override + public boolean isHidden(Path path) + { + return (false); + } + + + + /******************************************************************************* + ** Virtual provider getFileStore. + *******************************************************************************/ + @Override + public FileStore getFileStore(Path path) + { + throw (new UnsupportedOperationException("No native store")); + } + + + + /******************************************************************************* + ** Virtual provider checkAccess. + *******************************************************************************/ + @Override + public void checkAccess(Path path, AccessMode... modes) throws IOException + { + fileSystem.tree().stat(name(path)); + for(AccessMode mode : modes) + { + if(mode == AccessMode.WRITE) + { + VirtualTree.Node node = fileSystem.tree().stat(name(path)); + if(!node.directory() || !"PROFILE".equals(node.kind()) || node.objectId() == null) + { + fileSystem.tree().deny(name(path), "UPLOAD", "Writes require a profile folder"); + } + } + } + } + + + + /******************************************************************************* + ** Virtual provider getFileAttributeView. + *******************************************************************************/ + @Override + public V getFileAttributeView(Path path, Class type, LinkOption... options) + { + return (null); + } + + + + /******************************************************************************* + ** Virtual provider readAttributes. + *******************************************************************************/ + @Override + public A readAttributes(Path path, Class type, LinkOption... options) throws IOException + { + return (type.cast(new GatewayFileAttributes(fileSystem.tree().stat(name(path)), FileTime.from(Instant.now())))); + } + + + + /******************************************************************************* + ** Virtual provider readAttributes. + *******************************************************************************/ + @Override + public Map readAttributes(Path path, String attributes, LinkOption... options) throws IOException + { + var attrs = readAttributes(path, GatewayFileAttributes.class, options); + Map result = new HashMap<>(); + result.put("isDirectory", attrs.isDirectory()); + result.put("isRegularFile", attrs.isRegularFile()); + result.put("isSymbolicLink", false); + result.put("isOther", false); + result.put("size", attrs.size()); + result.put("fileKey", attrs.fileKey()); + result.put("lastModifiedTime", attrs.lastModifiedTime()); + result.put("lastAccessTime", attrs.lastAccessTime()); + result.put("creationTime", attrs.creationTime()); + result.put("permissions", attrs.permissions()); + result.put("owner", attrs.owner()); + result.put("group", attrs.group()); + return (result); + } + + + + /******************************************************************************* + ** Virtual provider setAttribute. + *******************************************************************************/ + @Override + public void setAttribute(Path path, String attribute, Object value, LinkOption... options) throws IOException + { + fileSystem.tree().deny(name(path), "UPLOAD", "setattr is not supported"); + } + + + + /******************************************************************************* + ** Virtual provider createSymbolicLink. + *******************************************************************************/ + @Override + public void createSymbolicLink(Path link, Path target, FileAttribute... attrs) throws IOException + { + fileSystem.tree().deny(name(link), "UPLOAD", "Links are not supported"); + } + + + + /******************************************************************************* + ** Virtual provider createLink. + *******************************************************************************/ + @Override + public void createLink(Path link, Path existing) throws IOException + { + fileSystem.tree().deny(name(link), "UPLOAD", "Links are not supported"); + } + + + /******************************************************************************* + ** Single-use directory snapshot held entirely in memory. + *******************************************************************************/ + private static class GatewayDirectoryStream implements DirectoryStream + { + private final List entries; + private boolean used; + + + + /******************************************************************************* + ** Directory snapshot lifecycle. + *******************************************************************************/ + GatewayDirectoryStream(List entries) + { + this.entries = entries; + } + + + + /******************************************************************************* + ** Directory snapshot lifecycle. + *******************************************************************************/ + @Override + public Iterator iterator() + { + if(used) + { + throw (new IllegalStateException("Directory stream is closed or already iterated")); + } + used = true; + return (entries.iterator()); + } + + + + /******************************************************************************* + ** Directory snapshot lifecycle. + *******************************************************************************/ + @Override + public void close() + { + used = true; + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/vfs/GatewayPath.java b/src/main/java/com/kingsrook/qbits/filegateway/vfs/GatewayPath.java new file mode 100644 index 0000000..921eb56 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/vfs/GatewayPath.java @@ -0,0 +1,371 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.vfs; + + +import java.io.File; +import java.io.IOException; +import java.net.URI; +import java.nio.file.FileSystem; +import java.nio.file.LinkOption; +import java.nio.file.Path; +import java.nio.file.ProviderMismatchException; +import java.nio.file.WatchEvent; +import java.nio.file.WatchKey; +import java.nio.file.WatchService; +import java.util.ArrayDeque; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Deque; +import java.util.Iterator; +import java.util.List; + + +/******************************************************************************* + ** A path made solely from virtual name segments; never delegates to a native filesystem. + *******************************************************************************/ +public final class GatewayPath implements Path +{ + private final GatewayFileSystem fileSystem; + private final String value; + + + + /******************************************************************************* + ** Construct a virtual path. + *******************************************************************************/ + public GatewayPath(GatewayFileSystem fileSystem, String value) + { + this.fileSystem = fileSystem; + this.value = value.replaceAll("/+", "/").replaceAll("(?<=.)/$", ""); + } + + + + /******************************************************************************* + ** Individual name segments. + *******************************************************************************/ + private List parts() + { + return (value.isEmpty() || "/".equals(value) ? List.of() : Arrays.asList(value.replaceFirst("^/", "").split("/"))); + } + + + + /******************************************************************************* + ** Virtual path getFileSystem. + *******************************************************************************/ + @Override + public FileSystem getFileSystem() + { + return (fileSystem); + } + + + + /******************************************************************************* + ** Virtual path isAbsolute. + *******************************************************************************/ + @Override + public boolean isAbsolute() + { + return (value.startsWith("/")); + } + + + + /******************************************************************************* + ** Virtual path getRoot. + *******************************************************************************/ + @Override + public Path getRoot() + { + return (isAbsolute() ? new GatewayPath(fileSystem, "/") : null); + } + + + + /******************************************************************************* + ** Virtual path getFileName. + *******************************************************************************/ + @Override + public Path getFileName() + { + List parts = parts(); + return (parts.isEmpty() ? null : new GatewayPath(fileSystem, parts.getLast())); + } + + + + /******************************************************************************* + ** Virtual path getParent. + *******************************************************************************/ + @Override + public Path getParent() + { + int slash = value.lastIndexOf('/'); + return (slash < 0 || "/".equals(value) ? null : new GatewayPath(fileSystem, slash == 0 ? "/" : value.substring(0, slash))); + } + + + + /******************************************************************************* + ** Virtual path getNameCount. + *******************************************************************************/ + @Override + public int getNameCount() + { + return (parts().size()); + } + + + + /******************************************************************************* + ** Virtual path getName. + *******************************************************************************/ + @Override + public Path getName(int index) + { + return (new GatewayPath(fileSystem, parts().get(index))); + } + + + + /******************************************************************************* + ** Virtual path subpath. + *******************************************************************************/ + @Override + public Path subpath(int beginIndex, int endIndex) + { + return (new GatewayPath(fileSystem, String.join("/", parts().subList(beginIndex, endIndex)))); + } + + + + /******************************************************************************* + ** Virtual path startsWith. + *******************************************************************************/ + @Override + public boolean startsWith(Path other) + { + return (other instanceof GatewayPath p && p.fileSystem == fileSystem && (value.equals(p.value) || value.startsWith(p.value.endsWith("/") ? p.value : p.value + "/"))); + } + + + + /******************************************************************************* + ** Virtual path endsWith. + *******************************************************************************/ + @Override + public boolean endsWith(Path other) + { + return (other instanceof GatewayPath p && p.fileSystem == fileSystem && (value.equals(p.value) || (!p.isAbsolute() && value.endsWith("/" + p.value)))); + } + + + + /******************************************************************************* + ** Virtual path normalize. + *******************************************************************************/ + @Override + public Path normalize() + { + Deque names = new ArrayDeque<>(); + for(String part : parts()) + { + if(part.equals(".") || part.isEmpty()) + { + continue; + } + if(part.equals("..") && !names.isEmpty() && !names.getLast().equals("..")) + { + names.removeLast(); + } + else if(!part.equals("..") || !isAbsolute()) + { + names.addLast(part); + } + } + return (new GatewayPath(fileSystem, (isAbsolute() ? "/" : "") + String.join("/", names))); + } + + + + /******************************************************************************* + ** Virtual path resolve. + *******************************************************************************/ + @Override + public Path resolve(Path other) + { + if(!(other instanceof GatewayPath p) || p.fileSystem != fileSystem) + { + throw (new ProviderMismatchException()); + } + return (other.isAbsolute() ? other : new GatewayPath(fileSystem, value + (value.endsWith("/") || value.isEmpty() ? "" : "/") + other)); + } + + + + /******************************************************************************* + ** Virtual path relativize. + *******************************************************************************/ + @Override + public Path relativize(Path other) + { + if(!(other instanceof GatewayPath p) || p.fileSystem != fileSystem || isAbsolute() != p.isAbsolute()) + { + throw (new IllegalArgumentException("Incompatible virtual paths")); + } + List left = ((GatewayPath) normalize()).parts(); + List right = ((GatewayPath) p.normalize()).parts(); + int common = 0; + while(common < left.size() && common < right.size() && left.get(common).equals(right.get(common))) + { + common++; + } + List result = new ArrayList<>(); + for(int i = common; i < left.size(); i++) + { + result.add(".."); + } + result.addAll(right.subList(common, right.size())); + return (new GatewayPath(fileSystem, String.join("/", result))); + } + + + + /******************************************************************************* + ** Virtual path toUri. + *******************************************************************************/ + @Override + public URI toUri() + { + try + { + return (new URI("gateway", null, toAbsolutePath().toString(), null)); + } + catch(java.net.URISyntaxException e) + { + throw (new IllegalArgumentException(e)); + } + } + + + + /******************************************************************************* + ** Virtual path toAbsolutePath. + *******************************************************************************/ + @Override + public Path toAbsolutePath() + { + return (isAbsolute() ? this : new GatewayPath(fileSystem, "/" + value)); + } + + + + /******************************************************************************* + ** Virtual path toRealPath. + *******************************************************************************/ + @Override + public Path toRealPath(LinkOption... options) throws IOException + { + Path path = toAbsolutePath().normalize(); + fileSystem.tree().stat(path.toString()); + return (path); + } + + + + /******************************************************************************* + ** Virtual path toFile. + *******************************************************************************/ + @Override + public File toFile() + { + throw (new UnsupportedOperationException("Virtual paths have no native file")); + } + + + + /******************************************************************************* + ** Virtual path register. + *******************************************************************************/ + @Override + public WatchKey register(WatchService watcher, WatchEvent.Kind[] events, WatchEvent.Modifier... modifiers) + { + throw (new UnsupportedOperationException("Virtual paths cannot be watched")); + } + + + + /******************************************************************************* + ** Virtual path iterator. + *******************************************************************************/ + @Override + public Iterator iterator() + { + return (parts().stream().map(name -> (Path) new GatewayPath(fileSystem, name)).iterator()); + } + + + + /******************************************************************************* + ** Virtual path compareTo. + *******************************************************************************/ + @Override + public int compareTo(Path other) + { + return (value.compareTo(other.toString())); + } + + + + /******************************************************************************* + ** Virtual path equals. + *******************************************************************************/ + @Override + public boolean equals(Object other) + { + return (other instanceof GatewayPath path && path.fileSystem == fileSystem && path.value.equals(value)); + } + + + + /******************************************************************************* + ** Virtual path hashCode. + *******************************************************************************/ + @Override + public int hashCode() + { + return (31 * System.identityHashCode(fileSystem) + value.hashCode()); + } + + + + /******************************************************************************* + ** Virtual path toString. + *******************************************************************************/ + @Override + public String toString() + { + return (value); + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/vfs/GatewayUiApiClient.java b/src/main/java/com/kingsrook/qbits/filegateway/vfs/GatewayUiApiClient.java new file mode 100644 index 0000000..5da1abe --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/vfs/GatewayUiApiClient.java @@ -0,0 +1,163 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.vfs; + + +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.AccessDeniedException; +import java.nio.file.NoSuchFileException; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.node.JsonNodeFactory; +import com.kingsrook.qbits.filegateway.sftp.GatewaySession; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiCredentials; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInvoker; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiRequest; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiStreamResponse; +import com.kingsrook.qqq.openapi.model.HttpMethod; + + +/******************************************************************************* + ** Every visible object comes from a spec-defined UI API operation as the signed-in user. + *******************************************************************************/ +public class GatewayUiApiClient +{ + private final UiApiInvoker invoker; + private final GatewaySession session; + + + + /******************************************************************************* + ** Bind this client to one SFTP identity. + *******************************************************************************/ + public GatewayUiApiClient(UiApiInvoker invoker, GatewaySession session) + { + this.invoker = invoker; + this.session = session; + } + + + + /******************************************************************************* + ** Get the permission-filtered app tree. + *******************************************************************************/ + public JsonNode metadata() throws IOException + { + return (invoke(new UiApiRequest().withMethod(HttpMethod.GET).withPath("/metaData"))); + } + + + + /******************************************************************************* + ** Query visible saved objects of a table. + *******************************************************************************/ + public JsonNode savedObjects(String source, String table) throws IOException + { + var records = JsonNodeFactory.instance.arrayNode(); + int pageSize = 250; + for(int skip = 0; ; skip += pageSize) + { + QQueryFilter filter = new QQueryFilter(new QFilterCriteria("tableName", QCriteriaOperator.EQUALS, table)) + .withOrderBy(new QFilterOrderBy("id", true)).withLimit(pageSize).withSkip(skip); + JsonNode page = invoke(new UiApiRequest().withMethod(HttpMethod.POST).withPath("/table/{tableName}/query") + .withPathParam("tableName", source).withJsonBody(java.util.Map.of("filter", filter))).path("records"); + page.forEach(records::add); + if(page.size() < pageSize) + { + return (records); + } + } + } + + + + /******************************************************************************* + ** Apply the fixed channel identity, credentials and contract check on every call. + *******************************************************************************/ + public JsonNode invoke(UiApiRequest request) throws IOException + { + if(invoker.findOperation(request.getMethod(), request.getPath()).isEmpty()) + { + throw (new IOException("Gateway requires a spec-defined UI API operation")); + } + try + { + var response = invoker.invoke(request.withCredentials(UiApiCredentials.ofInProcessSession(session.token())) + .withInvokedBy("file-gateway").withClientDetail(session.id)); + if(response.getStatus() == 404) + { + throw (new NoSuchFileException(request.getPath())); + } + if(response.getStatus() == 401 || response.getStatus() == 403) + { + throw (new AccessDeniedException(request.getPath(), null, response.getBody())); + } + if(!response.isSuccess()) + { + throw (new IOException("UI API request failed with status " + response.getStatus() + ": " + response.getBody())); + } + return (response.getJson()); + } + catch(com.kingsrook.qqq.backend.core.exceptions.QException e) + { + throw (new IOException("User session unavailable", e)); + } + } + + + + /******************************************************************************* + ** Open a binary response as this user, retaining only a bounded error body. + *******************************************************************************/ + public UiApiStreamResponse stream(UiApiRequest request) throws IOException + { + if(invoker.findOperation(request.getMethod(), request.getPath()).isEmpty()) + { + throw (new IOException("Gateway requires a spec-defined UI API operation")); + } + try + { + UiApiStreamResponse response = invoker.invokeForStream(request.withCredentials(UiApiCredentials.ofInProcessSession(session.token())) + .withInvokedBy("file-gateway").withClientDetail(session.id)); + if(response.isSuccess()) + { + return (response); + } + try(response) + { + String error = new String(response.getInputStream().readNBytes(65536), StandardCharsets.UTF_8); + if(response.getStatus() == 401 || response.getStatus() == 403) + { + throw (new AccessDeniedException(request.getPath(), null, error)); + } + throw (new IOException("UI API request failed with status " + response.getStatus() + ": " + error)); + } + } + catch(com.kingsrook.qqq.backend.core.exceptions.QException e) + { + throw (new IOException("User session unavailable", e)); + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/vfs/GeneratedDownloads.java b/src/main/java/com/kingsrook/qbits/filegateway/vfs/GeneratedDownloads.java new file mode 100644 index 0000000..db5097c --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/vfs/GeneratedDownloads.java @@ -0,0 +1,342 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.vfs; + + +import java.io.IOException; +import java.nio.channels.SeekableByteChannel; +import java.nio.file.AccessDeniedException; +import java.nio.file.NoSuchFileException; +import java.time.Clock; +import java.time.ZoneOffset; +import java.util.List; +import java.util.Map; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.node.ObjectNode; +import com.kingsrook.qbits.filegateway.FileGatewayQBitConfig; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qbits.filegateway.sftp.AuditWriter; +import com.kingsrook.qbits.filegateway.sftp.GatewaySession; +import com.kingsrook.qbits.filegateway.storage.GeneratedReadChannel; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiRequest; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiStreamResponse; +import com.kingsrook.qqq.openapi.model.HttpMethod; + + +/******************************************************************************* + ** Generate user-authorized exports and reports through the UI API, with durable quotas. + *******************************************************************************/ +public final class GeneratedDownloads +{ + private final GatewayUiApiClient client; + private final GatewaySession session; + private final AuditWriter audit; + private final FileGatewayQBitConfig config; + private final Clock clock; + private final boolean reportsAvailable; + private final ObjectMapper mapper = new ObjectMapper(); + + + /******************************************************************************* + ** Inspect only host configuration for report storage; visibility still uses user metadata. + *******************************************************************************/ + public GeneratedDownloads(GatewayUiApiClient client, GatewaySession session, AuditWriter audit, FileGatewayQBitConfig config, Clock clock, QInstance instance) + { + this.client = client; + this.session = session; + this.audit = audit; + this.config = config; + this.clock = clock; + var process = instance.getProcess("renderSavedReport"); + reportsAvailable = process != null && process.getInputFields().stream().anyMatch(field -> "storageTableName".equals(field.getName()) + && field.getDefaultValue() != null && instance.getTable(field.getDefaultValue().toString()) != null); + } + + + /******************************************************************************* + ** A render process without its configured storage table cannot provide downloads. + *******************************************************************************/ + public boolean reportsAvailable() + { + return (reportsAvailable); + } + + + /******************************************************************************* + ** Start generation at open, owning the HTTP stream until the SFTP handle closes. + *******************************************************************************/ + public SeekableByteChannel open(VirtualTree.Node node) throws IOException + { + Transfer transfer = new Transfer(node); + try + { + if(!session.operationAllowed()) + { + transfer.limit = true; + throw (new IOException("Operation limit exceeded")); + } + int window = Math.toIntExact(config.getDownloadReadAheadMb() * 1024L * 1024L); + if(window < 1) + { + throw (new IOException("Read-ahead window must be positive")); + } + String prefix = node.path().substring(0, node.path().lastIndexOf('/')); + prefix = prefix.substring(0, prefix.lastIndexOf('/')); + String table = node.job() == null ? prefix.substring(prefix.lastIndexOf('/') + 1) : node.job().getValueString("tableName"); + JsonNode record = null; + for(JsonNode row : client.savedObjects("VIEW".equals(node.kind()) ? "savedView" : "savedReport", table)) + { + if(node.objectId().equals(row.path("values").path("id").asText())) + { + record = row.path("values"); + } + } + if(record == null) + { + throw (new NoSuchFileException(node.path())); + } + UiApiStreamResponse response = "VIEW".equals(node.kind()) ? export(table, record, transfer.format) : report(node.objectId(), transfer.format); + return (new GeneratedReadChannel(response.getInputStream(), window, transfer)); + } + catch(IOException | RuntimeException e) + { + IOException failure = e instanceof IOException io ? io : new IOException("Unable to generate download", e); + transfer.finish(failure); + throw (failure); + } + } + + + /******************************************************************************* + ** Use the Next/Material saved-view schema, retaining filter ordering and visible columns. + *******************************************************************************/ + private UiApiStreamResponse export(String table, JsonNode record, String format) throws IOException + { + JsonNode view = mapper.readTree(record.path("viewJson").asText()); + if(view == null || !view.isObject()) + { + throw (new IOException("Saved view has invalid viewJson")); + } + ObjectNode body = mapper.createObjectNode().put("format", format).put("includeHeaderRow", true); + JsonNode filter = view.has("queryFilter") ? view.path("queryFilter") : view.path("filter"); + if(filter.isObject()) + { + ObjectNode query = filter.deepCopy(); + query.remove(List.of("skip", "limit")); + body.set("filter", query); + } + JsonNode columns = view.path("queryColumns").path("columns"); + if(columns.isArray()) + { + var names = body.putArray("fieldNames"); + for(JsonNode column : columns) + { + String name = column.path("name").asText(); + if(column.path("isVisible").asBoolean(true) && !name.isEmpty() && !name.equals("__check__")) + { + names.add(name); + } + } + if(names.isEmpty()) + { + throw (new IOException("Saved view has no visible export columns")); + } + } + else if(view.path("fieldNames").isArray()) + { + body.set("fieldNames", view.path("fieldNames")); + } + return (client.stream(new UiApiRequest().withMethod(HttpMethod.POST).withPath("/table/{tableName}/export") + .withPathParam("tableName", table).withJsonBody(body))); + } + + + /******************************************************************************* + ** Select one record, submit the format input, then follow asynchronous render status. + *******************************************************************************/ + private UiApiStreamResponse report(String id, String format) throws IOException + { + String values = mapper.writeValueAsString(Map.of("reportFormat", format)); + JsonNode response = client.invoke(new UiApiRequest().withMethod(HttpMethod.POST).withPath("/processes/{processName}/init") + .withPathParam("processName", "renderSavedReport").withFormParam("recordsParam", "recordIds").withFormParam("recordIds", id) + .withFormParam("values", values)); + String uuid = response.path("processUUID").asText(); + String job = response.path("jobUUID").asText(); + for(int poll = 0; poll < 3600; poll++) + { + String type = response.path("type").asText(); + if("ERROR".equals(type)) + { + throw (new IOException(response.path("userFacingError").asText("Report generation failed"))); + } + if("JOB_STARTED".equals(type) || "RUNNING".equals(type)) + { + if(response.hasNonNull("jobUUID")) + { + job = response.path("jobUUID").asText(); + } + try + { + Thread.sleep(100); + } + catch(InterruptedException e) + { + Thread.currentThread().interrupt(); + throw (new IOException("Report generation interrupted", e)); + } + response = client.invoke(new UiApiRequest().withMethod(HttpMethod.GET).withPath("/processes/{processName}/{processUUID}/status/{jobUUID}") + .withPathParam("processName", "renderSavedReport").withPathParam("processUUID", uuid).withPathParam("jobUUID", job)); + continue; + } + JsonNode output = response.path("values"); + if(output.hasNonNull("storageReference") && output.hasNonNull("downloadFileName") && output.hasNonNull("storageTableName")) + { + return (client.stream(new UiApiRequest().withMethod(HttpMethod.GET).withPath("/download/{file}") + .withPathParam("file", output.path("downloadFileName").asText()) + .withQueryParam("storageTableName", output.path("storageTableName").asText()) + .withQueryParam("storageReference", output.path("storageReference").asText()))); + } + if(!"input".equals(response.path("nextStep").asText())) + { + throw (new IOException("Unexpected report step: " + response.path("nextStep").asText())); + } + response = client.invoke(new UiApiRequest().withMethod(HttpMethod.POST).withPath("/processes/{processName}/{processUUID}/step/{stepName}") + .withPathParam("processName", "renderSavedReport").withPathParam("processUUID", uuid).withPathParam("stepName", "input") + .withFormParam("values", values)); + } + throw (new IOException("Report generation timed out")); + } + + + /******************************************************************************* + ** One durable row per handle, updated under the account lock as bytes flow. + *******************************************************************************/ + private final class Transfer implements GeneratedReadChannel.Audit + { + private final long began = System.nanoTime(); + private final String format; + private final QRecord row; + private long bytes; + private boolean limit; + + /******************************************************************************* + ** Default unfinished transfers to ERROR, including a process terminated mid-read. + *******************************************************************************/ + private Transfer(VirtualTree.Node node) throws IOException + { + format = node.job() == null ? node.path().substring(node.path().lastIndexOf('.') + 1) : node.job().getValueString("format"); + try + { + row = audit.insert(FileGatewayTables.TRANSFER_LOG, new QRecord().withValue("accountId", session.account.getValue("id")) + .withValue("connectionLogId", session.connectionId).withValue("operation", "DOWNLOAD").withValue("path", node.path()) + .withValue("objectKind", node.kind()).withValue("objectId", node.objectId()).withValue("format", format) + .withValue("bytes", 0L).withValue("outcome", "ERROR").withValue("clientChannel", "sftp/" + session.id)); + } + catch(QException e) + { + throw (new IOException("Unable to persist download audit", e)); + } + } + + + /******************************************************************************* + ** Filter by account, operation and UTC day; include in-progress and failed transfers. + *******************************************************************************/ + @Override + public void charge(int count) throws IOException + { + try + { + boolean allowed = audit.withAccountLock("id", session.account.getValue("id"), () -> + { + long maximum = config.getMaxGeneratedFileMb() * 1024L * 1024L; + Long dailyMb = session.account.getValueLong("maxDownloadMbPerDay"); + long total = 0; + if(dailyMb != null) + { + var start = clock.instant().atZone(ZoneOffset.UTC).toLocalDate().atStartOfDay(ZoneOffset.UTC).toInstant(); + for(int skip = 0; ; skip += 250) + { + var rows = audit.query(FileGatewayTables.TRANSFER_LOG, new QQueryFilter() + .withCriteria(new QFilterCriteria("accountId", QCriteriaOperator.EQUALS, session.account.getValue("id"))) + .withCriteria(new QFilterCriteria("operation", QCriteriaOperator.EQUALS, "DOWNLOAD")) + .withCriteria(new QFilterCriteria("createDate", QCriteriaOperator.GREATER_THAN_OR_EQUALS, start)) + .withOrderBy(new QFilterOrderBy("id", true)).withSkip(skip).withLimit(250)); + for(QRecord record : rows) + { + Long value = record.getValueLong("bytes"); + total += value == null ? 0 : value; + } + if(rows.size() < 250) + { + break; + } + } + } + if(bytes + count > maximum || (dailyMb != null && total + count > dailyMb * 1024L * 1024L)) + { + return (false); + } + audit.update(FileGatewayTables.TRANSFER_LOG, new QRecord().withValue("id", row.getValue("id")).withValue("bytes", bytes + count)); + return (true); + }); + if(!allowed) + { + limit = true; + throw (new IOException("Generated file or daily download limit exceeded")); + } + bytes += count; + } + catch(QException e) + { + throw (new IOException("Unable to enforce download limits", e)); + } + } + + + /******************************************************************************* + ** Preserve API denial and limit outcomes instead of collapsing them to generic errors. + *******************************************************************************/ + @Override + public void finish(IOException failure) throws IOException + { + try + { + audit.update(FileGatewayTables.TRANSFER_LOG, new QRecord().withValue("id", row.getValue("id")) + .withValue("durationMs", (System.nanoTime() - began) / 1_000_000) + .withValue("outcome", limit ? "LIMIT" : failure instanceof AccessDeniedException ? "DENIED" : failure == null ? "OK" : "ERROR") + .withValue("message", failure == null ? null : failure.getMessage())); + } + catch(QException e) + { + throw (new IOException("Unable to finalize download audit", e)); + } + } + } +} diff --git a/src/main/java/com/kingsrook/qbits/filegateway/vfs/VirtualTree.java b/src/main/java/com/kingsrook/qbits/filegateway/vfs/VirtualTree.java new file mode 100644 index 0000000..83329e4 --- /dev/null +++ b/src/main/java/com/kingsrook/qbits/filegateway/vfs/VirtualTree.java @@ -0,0 +1,894 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway.vfs; + + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.nio.channels.SeekableByteChannel; +import java.nio.charset.StandardCharsets; +import java.nio.file.AccessDeniedException; +import java.nio.file.NoSuchFileException; +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import com.fasterxml.jackson.databind.JsonNode; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qbits.filegateway.sftp.AuditWriter; +import com.kingsrook.qbits.filegateway.sftp.GatewaySession; +import com.kingsrook.qbits.filegateway.storage.GatewayObjectStore; +import com.kingsrook.qbits.filegateway.storage.ObjectReadChannel; +import com.kingsrook.qbits.filegateway.upload.UploadJobs; +import com.kingsrook.qqq.backend.core.model.data.QRecord; + + +/******************************************************************************* + ** Permission-filtered virtual app and table tree with per-session in-memory caches. + *******************************************************************************/ +public class VirtualTree +{ + /******************************************************************************* + ** A virtual directory or generated file. + *******************************************************************************/ + public record Node(String path, boolean directory, String kind, String objectId, long size, QRecord job) + { + /******************************************************************************* + ** Metadata-backed nodes have no stored size or job. + *******************************************************************************/ + public Node(String path, boolean directory, String kind, String objectId) + { + this(path, directory, kind, objectId, 0, null); + } + } + + private final GatewayUiApiClient client; + private final GatewaySession session; + private final AuditWriter audit; + private final Clock clock; + private UploadJobs uploads; + private GatewayObjectStore store; + private GeneratedDownloads downloads; + private com.kingsrook.qbits.filegateway.outbox.OutboxService outbox; + private JsonNode metadata; + private Instant metadataAt; + private final Map saved = new HashMap<>(); + private final Map savedAt = new HashMap<>(); + + + + /******************************************************************************* + ** Create a session-private tree and caches. + *******************************************************************************/ + public VirtualTree(GatewayUiApiClient client, GatewaySession session, AuditWriter audit, Clock clock) + { + this.client = client; + this.session = session; + this.audit = audit; + this.clock = clock; + } + + + + /******************************************************************************* + ** Attach the runtime's upload pipeline; endpoint targets will reuse this service. + *******************************************************************************/ + public VirtualTree withUploads(UploadJobs uploads, GatewayObjectStore store) + { + this.uploads = uploads; + this.store = store; + return (this); + } + + + /******************************************************************************* + ** Attach generated downloads and the host's report-storage availability. + *******************************************************************************/ + public VirtualTree withDownloads(GeneratedDownloads downloads) + { + this.downloads = downloads; + return (this); + } + + + + /******************************************************************************* + ** Use the same provider in the web deployment and the virtual outbox folders. + *******************************************************************************/ + public VirtualTree withOutbox(com.kingsrook.qbits.filegateway.outbox.OutboxService outbox) + { + this.outbox = outbox; + return (this); + } + + + + /******************************************************************************* + ** Cache user metadata for sixty seconds. + *******************************************************************************/ + private JsonNode metadata() throws IOException + { + if(metadataAt == null || !metadataAt.plusSeconds(60).isAfter(clock.instant())) + { + metadata = client.metadata(); + metadataAt = clock.instant(); + } + return (metadata); + } + + + + /******************************************************************************* + ** Cache saved objects for thirty seconds. + *******************************************************************************/ + private JsonNode saved(String source, String table) throws IOException + { + String key = source + "/" + table; + if(!savedAt.containsKey(key) || !savedAt.get(key).plusSeconds(30).isAfter(clock.instant())) + { + saved.put(key, client.savedObjects(source, table)); + savedAt.put(key, clock.instant()); + } + return (saved.get(key)); + } + + + + /******************************************************************************* + ** Resolve only through visible parents; hidden paths are indistinguishable from missing ones. + *******************************************************************************/ + public synchronized Node stat(String path) throws IOException + { + sessionToken(); + if("/".equals(path)) + { + return (new Node("/", true, "APP", null)); + } + int split = path.lastIndexOf('/'); + String parent = split == 0 ? "/" : path.substring(0, split); + return (children(parent).stream().filter(node -> node.path().equals(path)).findFirst() + .orElseThrow(() -> new NoSuchFileException(path))); + } + + + + /******************************************************************************* + ** Refresh permissions even when a listing uses cached metadata. + *******************************************************************************/ + private void sessionToken() throws IOException + { + try + { + session.token(); + } + catch(Exception e) + { + throw (new AccessDeniedException("User session unavailable")); + } + } + + + + /******************************************************************************* + ** Audit each directory open once. + *******************************************************************************/ + public synchronized List list(String path) throws IOException + { + long started = System.nanoTime(); + try + { + sessionToken(); + if(!session.operationAllowed()) + { + log(path, "LIST", "LIMIT", null, started); + throw (new IOException("Operation limit exceeded")); + } + List nodes = children(path); + log(path, "LIST", "OK", null, started); + return (nodes); + } + catch(NoSuchFileException e) + { + log(path, "LIST", "NOT_FOUND", null, started); + throw (e); + } + catch(AccessDeniedException e) + { + log(path, "LIST", "DENIED", null, started); + throw (e); + } + } + + + + /******************************************************************************* + ** Resolve permitted app and table folders from UI API metadata. + *******************************************************************************/ + private List children(String path) throws IOException + { + List nodes = rawChildren(path); + if(Boolean.TRUE.equals(session.account.getValueBoolean("pgpEncryptDownloads"))) + { + return (nodes.stream().map(node -> node.directory() || "TEMPORARY".equals(node.kind()) ? node + : new Node(node.path() + ".pgp", false, node.kind(), node.objectId(), 0, node.job())).toList()); + } + return (nodes); + } + + + + /******************************************************************************* + ** Resolve canonical plaintext paths before applying download encryption names. + *******************************************************************************/ + private List rawChildren(String path) throws IOException + { + JsonNode meta = metadata(); + List result = new ArrayList<>(); + List endpoints = endpointChildren(path, meta); + if(!endpoints.isEmpty() || endpointMount(path, meta) != null) + { + result.addAll(endpoints); + if(!"/".equals(path)) + { + return (result); + } + } + if("ENDPOINTS_ONLY".equals(session.account.getValueString("rootMode"))) + { + if("/".equals(path)) + { + return (result); + } + throw (new NoSuchFileException(path)); + } + if("/".equals(path)) + { + addAppChildren(result, "", meta.path("appTree"), meta); + return (result); + } + String[] segments = path.substring(1).split("/"); + JsonNode children = meta.path("appTree"); + String prefix = ""; + String table = null; + int index = 0; + for(; index < segments.length; index++) + { + JsonNode found = null; + for(JsonNode child : children) + { + if(segments[index].equals(child.path("name").asText())) + { + found = child; + break; + } + } + if(found == null) + { + throw (new NoSuchFileException(path)); + } + prefix += "/" + segments[index]; + if("TABLE".equalsIgnoreCase(found.path("type").asText()) && meta.path("tables").has(segments[index])) + { + table = segments[index++]; + break; + } + if(!"APP".equalsIgnoreCase(found.path("type").asText())) + { + throw (new NoSuchFileException(path)); + } + children = found.path("children"); + } + if(table == null) + { + addAppChildren(result, prefix, children, meta); + return (result); + } + List folders = new ArrayList<>(List.of("results", "archive")); + boolean profiles = meta.path("tables").path(table).path("insertPermission").asBoolean(false) && meta.path("tables").has("savedBulkLoadProfile") && meta.path("processes").has(table + ".bulkInsert"); + if(profiles && !saved("savedBulkLoadProfile", table).isEmpty()) + { + folders.add("upload"); + } + if(meta.path("tables").has("savedView")) + { + folders.add("views"); + } + if(downloads != null && downloads.reportsAvailable() && meta.path("tables").has("savedReport") && meta.path("processes").has("renderSavedReport")) + { + folders.add("reports"); + } + if(index == segments.length) + { + for(String folder : folders) + { + result.add(new Node(prefix + "/" + folder, true, folderKind(folder), null)); + } + return (result); + } + String folder = segments[index++]; + if(!folders.contains(folder)) + { + throw (new NoSuchFileException(path)); + } + if("results".equals(folder) || "archive".equals(folder)) + { + return (artifacts(prefix, table, path, folder)); + } + String source = "upload".equals(folder) ? "savedBulkLoadProfile" : "views".equals(folder) ? "savedView" : "savedReport"; + JsonNode records = saved(source, table); + Map counts = new HashMap<>(); + for(JsonNode row : records) + { + counts.merge(slug(row.path("values").path("label").asText()), 1, Integer::sum); + } + for(JsonNode row : records) + { + String id = row.path("values").path("id").asText(); + String name = slug(row.path("values").path("label").asText()); + if(counts.get(name) > 1) + { + name += "-" + id; + } + String item = prefix + "/" + folder + "/" + name; + if("upload".equals(folder)) + { + result.add(new Node(item, true, "PROFILE", id)); + } + else + { + String formats = session.account.getValueString("formats"); + for(String format : (formats == null ? "csv,xlsx" : formats).split("[,\\s]+")) + { + if(List.of("csv", "xlsx", "json").contains(format)) + { + result.add(new Node(item + "." + format, false, folderKind(folder), id)); + } + } + } + } + if(index == segments.length) + { + return (result); + } + if(index + 1 == segments.length && "upload".equals(folder) && result.stream().anyMatch(node -> node.path().equals(path))) + { + return (artifacts(prefix, table, path, "upload")); + } + throw (new NoSuchFileException(path)); + } + + + + /******************************************************************************* + ** Endpoint targets are resolved from the same user-scoped saved-object queries. + *******************************************************************************/ + private List visibleEndpoints(JsonNode meta) throws IOException + { + try + { + List visible = new ArrayList<>(); + for(QRecord alias : audit.query(FileGatewayTables.ALIAS, UploadJobs.equal("isActive", true))) + { + if(alias.getValue("accountId") != null && !java.util.Objects.equals(alias.getValue("accountId"), session.account.getValue("id"))) + { + continue; + } + String kind = alias.getValueString("kind"); + if(List.of("INBOX", "OUTBOX").contains(kind)) + { + visible.add(alias); + continue; + } + String table = alias.getValueString("tableName"); + if(table == null || !meta.path("tables").has(table)) + { + continue; + } + String source = "BULK_LOAD".equals(kind) ? "savedBulkLoadProfile" : "VIEW".equals(kind) ? "savedView" : "savedReport"; + if(!meta.path("tables").has(source) + || ("BULK_LOAD".equals(kind) && (!meta.path("tables").path(table).path("insertPermission").asBoolean(false) || !meta.path("processes").has(table + ".bulkInsert"))) + || ("REPORT".equals(kind) && (downloads == null || !downloads.reportsAvailable() || !meta.path("processes").has("renderSavedReport")))) + { + continue; + } + for(JsonNode row : saved(source, table)) + { + if(row.path("values").path("id").asText().equals(alias.getValueString(source + "Id"))) + { + alias.withValue("_label", row.path("values").path("label").asText()); + visible.add(alias); + break; + } + } + } + return (visible); + } + catch(com.kingsrook.qqq.backend.core.exceptions.QException e) + { + throw (new IOException("Unable to resolve endpoints", e)); + } + } + + + + /******************************************************************************* + ** An exact visible mount, including an empty upload or outbox folder. + *******************************************************************************/ + private QRecord endpointMount(String path, JsonNode meta) throws IOException + { + return (visibleEndpoints(meta).stream().filter(alias -> path.equals(alias.getValueString("path"))).findFirst().orElse(null)); + } + + + + /******************************************************************************* + ** Mount ancestors reveal only the next segment, preserving endpoint chroots. + *******************************************************************************/ + private List endpointChildren(String path, JsonNode meta) throws IOException + { + List nodes = new ArrayList<>(); + String prefix = "/".equals(path) ? "/" : path + "/"; + for(QRecord alias : visibleEndpoints(meta)) + { + String mount = alias.getValueString("path"); + if(mount.startsWith(prefix)) + { + String suffix = mount.substring(prefix.length()).split("/")[0]; + String child = prefix + suffix; + if(nodes.stream().noneMatch(node -> child.equals(node.path()))) + { + nodes.add(new Node(child, true, "ENDPOINT", alias.getValueString("id"), 0, mount.equals(child) ? alias : null)); + } + } + if(!mount.equals(path)) + { + continue; + } + String kind = alias.getValueString("kind"); + if(List.of("BULK_LOAD", "INBOX").contains(kind)) + { + nodes.addAll(artifacts("", alias.getValueString("tableName"), path, "upload")); + } + else if("OUTBOX".equals(kind) && outbox != null) + { + try + { + for(QRecord row : outbox.list(alias.getValueInteger("id"))) + { + nodes.add(new Node(prefix + row.getValueString("fileName"), false, "OUTBOX", row.getValueString("id"), row.getValueLong("sizeBytes"), row)); + } + } + catch(com.kingsrook.qqq.backend.core.exceptions.QException e) + { + throw (new IOException("Unable to list outbox", e)); + } + } + else if(List.of("VIEW", "REPORT").contains(kind)) + { + String format = alias.getValueString("format"); + String formats = session.account.getValueString("formats"); + if(format != null && List.of((formats == null ? "csv,xlsx" : formats).split("[,\\s]+")).contains(format)) + { + String name = alias.getValueString("fileName"); + if(name == null || name.isBlank()) + { + name = slug(alias.getValueString("_label")) + "." + format; + } + nodes.add(new Node(prefix + name, false, kind, alias.getValueString("VIEW".equals(kind) ? "savedViewId" : "savedReportId"), 0, alias)); + } + } + } + return (nodes); + } + + + + /******************************************************************************* + ** Sidecars and results expire after seven days; archive objects remain until retention. + *******************************************************************************/ + private List artifacts(String prefix, String table, String path, String folder) throws IOException + { + List nodes = new ArrayList<>(); + if("results".equals(folder) && !path.equals(prefix + "/results")) + { + throw (new NoSuchFileException(path)); + } + if(uploads == null) + { + return (nodes); + } + for(QRecord job : uploads.forTable(session.account.getValueInteger("id"), table)) + { + String status = job.getValueString("status"); + boolean active = List.of("RECEIVING", "QUEUED", "RUNNING").contains(status); + boolean visible = job.getValueInstant("acknowledgedAt") == null + && job.getValueInstant("createDate").plusSeconds(7 * 86400).isAfter(clock.instant()); + String name = job.getValueString("fileName"); + String id = job.getValueString("id"); + if("upload".equals(folder) && visible && job.getValueString("path").startsWith(path + "/")) + { + String suffix = active ? ".processing" : List.of("SUCCEEDED", "DUPLICATE").contains(status) ? ".done" : ".error"; + String item = path + "/" + name + suffix; + nodes.removeIf(node -> node.path().equals(item)); + nodes.add(new Node(item, false, "SIDECAR", id, summary(job).length, job)); + if("RECEIVING".equals(status) && uploads.temporary(name)) + { + nodes.add(new Node(path + "/" + name, false, "TEMPORARY", id, job.getValueLong("sizeBytes"), job)); + } + } + if("results".equals(folder) && visible && job.getValueString("resultsStorageKey") != null) + { + nodes.add(new Node(prefix + "/results/" + id + "-" + name + ".results.csv", false, "RESULT", id, + job.getValueLong("resultsSizeBytes") == null ? 0 : job.getValueLong("resultsSizeBytes"), job)); + } + if("archive".equals(folder) && !active && job.getValueString("sha256") != null + && job.getValueString("storageKey") != null && !job.getValueString("storageKey").startsWith("file-gateway/quarantine/")) + { + String date = job.getValueInstant("createDate").atZone(ZoneOffset.UTC).toLocalDate().toString(); + String directory = prefix + "/archive/" + date; + if(path.equals(prefix + "/archive")) + { + if(nodes.stream().noneMatch(node -> node.path().equals(directory))) + { + nodes.add(new Node(directory, true, "ARCHIVE", null)); + } + } + else if(path.equals(directory)) + { + nodes.add(new Node(directory + "/" + id + "-" + name, false, "ARCHIVE", id, job.getValueLong("sizeBytes"), job)); + } + } + } + return (nodes); + } + + + + /******************************************************************************* + ** Sidecar text is generated solely from durable job state. + *******************************************************************************/ + private byte[] summary(QRecord job) + { + String text = job.getValueString("status") + "\n"; + if(job.getValueString("resultSummary") != null) + { + text += job.getValueString("resultSummary") + "\n"; + } + else if(job.getValueString("errorMessage") != null) + { + text += job.getValueString("errorMessage") + "\n"; + } + return (text.getBytes(StandardCharsets.UTF_8)); + } + + + + /******************************************************************************* + ** Writes require a visible profile, never a guessed table or profile identifier. + *******************************************************************************/ + public SeekableByteChannel write(String path) throws IOException + { + String parent = path.substring(0, path.lastIndexOf('/')); + Node target = stat(parent); + if(uploads != null && target.job() != null && "ENDPOINT".equals(target.kind()) + && List.of("BULK_LOAD", "INBOX").contains(target.job().getValueString("kind"))) + { + QRecord alias = target.job(); + return (uploads.receive(session, path, alias.getValueString("tableName"), alias.getValueInteger("savedBulkLoadProfileId"), alias)); + } + if(uploads == null || !target.directory() || !"PROFILE".equals(target.kind()) || target.objectId() == null) + { + deny(path, "UPLOAD", "Uploads require a bulk load profile folder"); + } + String table = parent.substring(0, parent.lastIndexOf("/upload/")); + table = table.substring(table.lastIndexOf('/') + 1); + return (uploads.receive(session, path, table, Integer.valueOf(target.objectId()))); + } + + + + /******************************************************************************* + ** Open generated views/reports or stored artifacts without using the filesystem. + *******************************************************************************/ + public SeekableByteChannel read(String path) throws IOException + { + Node node = stat(path); + if(Boolean.TRUE.equals(session.account.getValueBoolean("pgpEncryptDownloads")) && !node.directory() && path.endsWith(".pgp")) + { + String plainPath = path.substring(0, path.length() - 4); + Node plain = rawChildren(plainPath.substring(0, plainPath.lastIndexOf('/'))).stream() + .filter(candidate -> candidate.path().equals(plainPath)).findFirst().orElseThrow(() -> new NoSuchFileException(path)); + return (encrypted(plain)); + } + return (readNode(node)); + } + + + + /******************************************************************************* + ** Encrypt lazily into the existing fixed rolling window, without a producer thread. + *******************************************************************************/ + private SeekableByteChannel encrypted(Node node) throws IOException + { + SeekableByteChannel source = readNode(node); + try + { + var encrypted = com.kingsrook.qbits.filegateway.pgp.PgpStreams.encrypt(java.nio.channels.Channels.newInputStream(source), + session.account.getValueString("pgpPublicKey"), node.path().substring(node.path().lastIndexOf('/') + 1)); + boolean[] complete = {false}; + var tracked = new java.io.FilterInputStream(encrypted) + { + @Override + public int read(byte[] bytes, int offset, int length) throws IOException + { + int count = in.read(bytes, offset, length); + if(count < 0) + { + complete[0] = true; + } + return (count); + } + }; + return (new com.kingsrook.qbits.filegateway.storage.GeneratedReadChannel(tracked, 65536, new com.kingsrook.qbits.filegateway.storage.GeneratedReadChannel.Audit() + { + @Override + public void charge(int bytes) + { + // The underlying transfer accounts for plaintext bytes and applies its limits. + } + + @Override + public void finish(IOException failure) throws IOException + { + if(complete[0] && failure == null && "OUTBOX".equals(node.kind())) + { + try + { + outbox.downloaded(node.job()); + } + catch(com.kingsrook.qqq.backend.core.exceptions.QException e) + { + throw (new IOException("Unable to acknowledge encrypted outbox", e)); + } + } + } + })); + } + catch(IOException | RuntimeException e) + { + source.close(); + throw (e); + } + } + + + + /******************************************************************************* + ** Open a resolved plaintext node through its original authorized delivery path. + *******************************************************************************/ + private SeekableByteChannel readNode(Node node) throws IOException + { + String path = node.path(); + if(downloads != null && !node.directory() && List.of("VIEW", "REPORT").contains(node.kind())) + { + return (downloads.open(node)); + } + if(node.job() == null || node.directory() || "TEMPORARY".equals(node.kind())) + { + deny(path, "DOWNLOAD", "This file is not downloadable"); + } + if(!session.operationAllowed()) + { + throw (new IOException("Operation limit exceeded")); + } + long began = System.nanoTime(); + ObjectReadChannel.Completion completion = (bytes, failed) -> downloadLog(node, bytes, failed, began); + if("SIDECAR".equals(node.kind())) + { + byte[] content = summary(node.job()); + return (new ObjectReadChannel(content.length, (start, end) -> new ByteArrayInputStream(content, (int) start, (int) (end - start + 1)), completion)); + } + String key = node.job().getValueString("RESULT".equals(node.kind()) ? "resultsStorageKey" : "storageKey"); + ObjectReadChannel channel = new ObjectReadChannel(node.size(), (start, end) -> store.openRange(key, start, end), completion); + if("OUTBOX".equals(node.kind()) && !Boolean.TRUE.equals(session.account.getValueBoolean("pgpEncryptDownloads"))) + { + channel.withFullDownload((bytes, failed) -> + { + try + { + outbox.downloaded(node.job()); + } + catch(com.kingsrook.qqq.backend.core.exceptions.QException e) + { + throw (new IOException("Unable to acknowledge outbox download", e)); + } + }); + } + return (channel); + } + + + + /******************************************************************************* + ** Record actual object-download bytes, including ranged reads, once per handle. + *******************************************************************************/ + private void downloadLog(Node node, long bytes, boolean failed, long began) throws IOException + { + try + { + audit.insert(FileGatewayTables.TRANSFER_LOG, new QRecord().withValue("accountId", session.account.getValue("id")) + .withValue("connectionLogId", session.connectionId).withValue("operation", "DOWNLOAD").withValue("path", node.path()) + .withValue("objectKind", "SIDECAR".equals(node.kind()) ? "RESULT" : node.kind()).withValue("objectId", node.objectId()) + .withValue("uploadJobId", "OUTBOX".equals(node.kind()) ? null : node.job().getValue("id")).withValue("bytes", bytes) + .withValue("sha256", "ARCHIVE".equals(node.kind()) ? node.job().getValue("sha256") : null) + .withValue("durationMs", (System.nanoTime() - began) / 1_000_000).withValue("outcome", failed ? "ERROR" : "OK") + .withValue("clientChannel", "sftp/" + session.id)); + } + catch(Exception e) + { + throw (new IOException("Unable to persist download audit", e)); + } + } + + + + /******************************************************************************* + ** Acknowledge visible job artifacts; archive removal remains denied. + *******************************************************************************/ + public void remove(String path) throws IOException + { + Node node = stat(path); + if("OUTBOX".equals(node.kind()) && outbox != null) + { + try + { + outbox.remove(node.job()); + log(path, "REMOVE", "OK", null, System.nanoTime()); + return; + } + catch(com.kingsrook.qqq.backend.core.exceptions.QException e) + { + throw (new IOException("Unable to remove outbox file", e)); + } + } + if(node.job() == null || !List.of("SIDECAR", "RESULT").contains(node.kind())) + { + deny(path, "REMOVE", "Only sidecars and results can be acknowledged"); + } + uploads.acknowledge(node.job()); + } + + + + /******************************************************************************* + ** Only completed temporary uploads may be renamed within the same visible folder. + *******************************************************************************/ + public void rename(String source, String target) throws IOException + { + Node node = stat(source); + if(!"TEMPORARY".equals(node.kind())) + { + deny(source, "RENAME", "Only temporary uploads can be renamed"); + } + stat(target.substring(0, target.lastIndexOf('/'))); + uploads.rename(session.account.getValueInteger("id"), source, target); + } + + + + /******************************************************************************* + ** Expose only app and readable table nodes. + *******************************************************************************/ + private void addAppChildren(List result, String prefix, JsonNode children, JsonNode meta) + { + for(JsonNode child : children) + { + String type = child.path("type").asText().toUpperCase(Locale.ROOT); + String name = child.path("name").asText(); + if(("APP".equals(type) || (!prefix.isEmpty() && "TABLE".equals(type) && meta.path("tables").has(name))) + && !name.isEmpty() && !name.contains("/") && !name.equals(".") && !name.equals("..")) + { + result.add(new Node(prefix + "/" + name, true, type, name)); + } + } + } + + + + /******************************************************************************* + ** Map virtual folders to persisted audit kinds. + *******************************************************************************/ + private String folderKind(String folder) + { + return (switch(folder) + { + case "upload" -> "PROFILE"; + case "views" -> "VIEW"; + case "reports" -> "REPORT"; + case "results" -> "RESULT"; + default -> "ARCHIVE"; + }); + } + + + + /******************************************************************************* + ** Stable lowercase label slug, at most eighty characters before a collision suffix. + *******************************************************************************/ + public static String slug(String label) + { + String value = label.toLowerCase(Locale.ROOT).replaceAll("[^a-z0-9]+", "-").replaceAll("^-|-$", ""); + if(value.length() > 80) + { + value = value.substring(0, 80); + } + return (value.isEmpty() ? "item" : value); + } + + + + /******************************************************************************* + ** Audit and deny unsupported operations. + *******************************************************************************/ + public void deny(String path, String operation, String message) throws IOException + { + session.operationAllowed(); + log(path, operation, "DENIED", message, System.nanoTime()); + throw (new AccessDeniedException(path, null, message)); + } + + + + /******************************************************************************* + ** Persist the operation through the system audit writer. + *******************************************************************************/ + private void log(String path, String operation, String outcome, String message, long started) throws IOException + { + String kind = "TABLE"; + String id = null; + try + { + Node node = stat(path); + kind = List.of("SIDECAR", "TEMPORARY").contains(node.kind()) ? "RESULT" : node.kind(); + id = node.objectId(); + } + catch(IOException ignored) + { + // Missing targets have no object id. + } + try + { + audit.insert(FileGatewayTables.TRANSFER_LOG, new QRecord().withValue("accountId", session.account.getValue("id")) + .withValue("connectionLogId", session.connectionId).withValue("operation", operation).withValue("path", path) + .withValue("objectKind", kind).withValue("objectId", id).withValue("bytes", 0L) + .withValue("durationMs", (System.nanoTime() - started) / 1_000_000).withValue("outcome", outcome) + .withValue("message", message).withValue("clientChannel", "sftp/" + session.id)); + } + catch(Exception e) + { + throw (new IOException("Unable to persist operation audit", e)); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/ApiContractTest.java b/src/test/java/com/kingsrook/qbits/filegateway/ApiContractTest.java new file mode 100644 index 0000000..e96e928 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/ApiContractTest.java @@ -0,0 +1,136 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + +import java.io.ByteArrayInputStream; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Duration; +import java.time.Instant; +import com.kingsrook.qbits.filegateway.api.FileAcceptance; +import com.kingsrook.qbits.filegateway.api.FileGatewayConsumer; +import com.kingsrook.qbits.filegateway.api.FileGatewayConsumersMetaData; +import com.kingsrook.qbits.filegateway.api.FileGatewayOutbox; +import com.kingsrook.qbits.filegateway.api.FileGatewayUnavailableException; +import com.kingsrook.qbits.filegateway.api.InboundFile; +import com.kingsrook.qbits.filegateway.api.OutboxFileRef; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/******************************************************************************* + ** Independently usable consumer and outbox contracts. + *******************************************************************************/ +class ApiContractTest +{ + /******************************************************************************* + ** The public API may depend on core, but never gateway implementation types. + *******************************************************************************/ + @Test + void apiImportsAreIsolated() throws Exception + { + try(var sources = Files.list(Path.of("src/main/java/com/kingsrook/qbits/filegateway/api"))) + { + for(Path source : sources.filter(p -> p.toString().endsWith(".java")).toList()) + { + String text = Files.readString(source); + assertThat(text).as(source.toString()).doesNotContainPattern("com\\.kingsrook\\.qbits\\.filegateway\\.(?!api[.;])"); + } + } + } + + + + /******************************************************************************* + ** Fresh streams, immutable decisions, consumer registration and provider dispatch. + *******************************************************************************/ + @Test + void consumerAndOutboxContract() throws Exception + { + var file = new InboundFile(1, "orders", "alice", "A", "127.0.0.1", "/orders/a.csv", "a.csv", 3, "sha", Instant.EPOCH, + "bucket", "key", () -> new ByteArrayInputStream(new byte[] {1, 2, 3})); + assertThat(file.openStream()).isNotSameAs(file.openStream()); + assertThat(file.openStream().readAllBytes()).containsExactly(1, 2, 3); + assertThat(file.remoteAddress()).isEqualTo("127.0.0.1"); + FileGatewayConsumer consumer = new FileGatewayConsumer() + { + /******************************************************************************* + ** Get name. + *******************************************************************************/ + public String getName() + { + return ("test"); + } + + + + /******************************************************************************* + ** Accept. + *******************************************************************************/ + public FileAcceptance accept(InboundFile inbound) + { + return (FileAcceptance.accepted(inbound.sha256())); + } + }; + assertThat(consumer.getName()).isEqualTo("test"); + var accepted = consumer.accept(file); + assertThat(accepted.getStatus()).isEqualTo(FileAcceptance.Status.ACCEPTED); + assertThat(accepted.getConsumerReference()).isEqualTo("sha"); + assertThat(accepted.getReason()).isNull(); + var rejected = FileAcceptance.rejected("invalid"); + assertThat(rejected.getStatus()).isEqualTo(FileAcceptance.Status.REJECTED); + assertThat(rejected.getReason()).isEqualTo("invalid"); + var retry = FileAcceptance.retryLater("busy", Duration.ofMinutes(1)); + assertThat(retry.getStatus()).isEqualTo(FileAcceptance.Status.RETRY_LATER); + assertThat(retry.getRetryAfter()).isEqualTo(Duration.ofMinutes(1)); + var ref = new OutboxFileRef(2, "orders", "/orders/b.csv", "b.csv", 3, "sha", Instant.EPOCH); + consumer.onDownloaded(ref); + FileGatewayOutbox.setProvider(null); + assertThat(FileGatewayOutbox.isAvailable()).isFalse(); + assertThatThrownBy(() -> FileGatewayOutbox.put("orders", "b.csv", file.openStream(), 3)).isInstanceOf(FileGatewayUnavailableException.class); + try + { + FileGatewayOutbox.setProvider((endpoint, name, stream, size) -> + { + assertThat(endpoint).isEqualTo("orders"); + assertThat(name).isEqualTo("b.csv"); + assertThat(size).isEqualTo(3); + return (ref); + }); + assertThat(FileGatewayOutbox.isAvailable()).isTrue(); + assertThat(FileGatewayOutbox.put("orders", "b.csv", file.openStream(), 3)).isSameAs(ref); + } + finally + { + FileGatewayOutbox.setProvider(null); + } + var instance = new QInstance(); + var registry = FileGatewayConsumersMetaData.ofOrWithNew(instance); + var code = new QCodeReference(consumer.getClass()); + assertThat(registry.withConsumer(code)).isSameAs(registry); + assertThat(registry.getName()).isEqualTo("fileGatewayConsumers"); + assertThat(FileGatewayConsumersMetaData.ofOrWithNew(instance)).isSameAs(registry); + assertThat(registry.getConsumers()).containsExactly(code); + assertThatThrownBy(() -> registry.getConsumers().clear()).isInstanceOf(UnsupportedOperationException.class); + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/CidrMatcherTest.java b/src/test/java/com/kingsrook/qbits/filegateway/CidrMatcherTest.java new file mode 100644 index 0000000..0b69688 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/CidrMatcherTest.java @@ -0,0 +1,59 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + +import com.kingsrook.qbits.filegateway.sftp.CidrMatcher; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.CsvSource; +import static org.assertj.core.api.Assertions.assertThat; + +/******************************************************************************* + ** IPv4 and IPv6 boundaries, multiple networks and numeric-only fail-closed parsing. + *******************************************************************************/ +class CidrMatcherTest +{ + /******************************************************************************* + ** Networks never require hostname resolution or accept invalid prefix lengths. + *******************************************************************************/ + @ParameterizedTest + @CsvSource({ + "192.0.2.255,192.0.2.0/24,true", + "192.0.3.0,192.0.2.0/24,false", + "127.0.0.1,0.0.0.0/0,true", + "127.0.0.1,127.0.0.1/32,true", + "127.0.0.1,127.0.0.1/33,false", + "127.0.0.1,127.0.0.1/-1,false", + "127.0.0.1,deadbeef/32,false", + "127.0.0.1,127.0.0/24,false", + "127.0.0.1,'192.0.2.0/24,127.0.0.0/8',true", + "2001:db8::ffff,2001:db8::/32,true", + "2001:db9::1,2001:db8::/32,false", + "::1,::1/128,true", + "::1,::1/129,false" + }) + /******************************************************************************* + ** Matches numeric networks. + *******************************************************************************/ + void matchesNumericNetworks(String address, String cidrs, boolean allowed) + { + assertThat(CidrMatcher.matches(address, cidrs)).isEqualTo(allowed); + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/DownloadIntegrationTest.java b/src/test/java/com/kingsrook/qbits/filegateway/DownloadIntegrationTest.java new file mode 100644 index 0000000..e7f5482 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/DownloadIntegrationTest.java @@ -0,0 +1,360 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + + +import java.nio.charset.StandardCharsets; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.security.FieldSecurityLock; +import com.kingsrook.qqq.backend.core.model.metadata.security.QSecurityKeyType; +import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleDispatcher; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiCredentials; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInProcessSessions; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiRequest; +import com.kingsrook.qqq.openapi.model.HttpMethod; +import org.apache.poi.xssf.usermodel.XSSFWorkbook; +import org.apache.sshd.common.config.keys.KeyUtils; +import org.apache.sshd.sftp.client.SftpClient; +import org.apache.sshd.sftp.client.SftpClientFactory; +import org.apache.sshd.sftp.common.SftpConstants; +import org.apache.sshd.sftp.common.SftpException; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Real SFTP downloads against strict user-authorized UI exports and report processes. + *******************************************************************************/ +@ExtendWith(NoDiskExtension.class) +class DownloadIntegrationTest +{ + private static final String VIEW = """ + {"queryFilter":{"criteria":[{"fieldName":"quantity","operator":"GREATER_THAN","values":[1]}], + "orderBys":[{"fieldName":"quantity","isAscending":false}],"skip":30,"limit":1}, + "queryColumns":{"columns":[{"name":"__check__","isVisible":true},{"name":"quantity","isVisible":true}, + {"name":"name","isVisible":true},{"name":"id","isVisible":false}]}} + """; + + + /******************************************************************************* + ** Store the same JSON used by Next UI, including selection and hidden columns. + *******************************************************************************/ + private QRecord view(FileGatewayTestServer h, String json) throws Exception + { + return (h.audit.system(() -> new InsertAction().execute(new InsertInput("savedView").withRecord(new QRecord() + .withValue("tableName", "alpha").withValue("label", "Selected rows").withValue("userId", "A").withValue("viewJson", json))).getRecords().getFirst())); + } + + + /******************************************************************************* + ** Read until EOF regardless of the generated file's zero stat size. + *******************************************************************************/ + private byte[] read(SftpClient client, String path) throws Exception + { + try(var input = client.read(path)) + { + return (input.readAllBytes()); + } + } + + + /******************************************************************************* + ** Filter, column order, ordering, three formats, hidden tables and transfer identity. + *******************************************************************************/ + @Test + void viewFormatsPermissionsAndAudit() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + h.seedRows(3); + QRecord saved = view(h, VIEW); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + var account = h.account("alice", "A"); + h.audit.update("file_gateway_account", new QRecord().withValue("id", account.getValue("id")).withValue("formats", "csv,xlsx,json")); + h.key(account, key); + var bob = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(h.account("bob", "B"), bob); + try(var session = h.login("alice", key); var client = SftpClientFactory.instance().createSftpClient(session); + var deniedSession = h.login("bob", bob); var denied = SftpClientFactory.instance().createSftpClient(deniedSession)) + { + String path = "/testapp/alpha/views/selected-rows."; + assertThat(client.stat(path + "csv").getSize()).isZero(); + String csv = new String(read(client, path + "csv"), StandardCharsets.UTF_8); + assertThat(csv.replace("\r\n", "\n")).isEqualTo("\"Quantity\",\"Name\"\n\"3\",\"Row 3 " + "x".repeat(50) + "\"\n\"2\",\"Row 2 " + "x".repeat(50) + "\"\n"); + try(var workbook = new XSSFWorkbook(new java.io.ByteArrayInputStream(read(client, path + "xlsx")))) + { + assertThat(workbook.getSheetAt(0).getRow(0).getCell(0).getStringCellValue()).isEqualTo("Quantity"); + assertThat(workbook.getSheetAt(0).getRow(1).getCell(0).getNumericCellValue()).isEqualTo(3); + } + var json = new ObjectMapper().readTree(read(client, path + "json")); + assertThat(json.toString()).contains("Row 3", "Row 2").doesNotContain("Row 1"); + assertThatThrownBy(() -> denied.stat(path + "csv")).isInstanceOfSatisfying(SftpException.class, + e -> assertThat(e.getStatus()).isEqualTo(SftpConstants.SSH_FX_NO_SUCH_FILE)); + assertThatThrownBy(() -> read(denied, path + "csv")).isInstanceOfSatisfying(SftpException.class, + e -> assertThat(e.getStatus()).isEqualTo(SftpConstants.SSH_FX_NO_SUCH_FILE)); + } + var downloads = h.audit.query("file_gateway_transfer_log", new QQueryFilter()).stream() + .filter(r -> "DOWNLOAD".equals(r.getValueString("operation"))).toList(); + assertThat(downloads).hasSize(3).allSatisfy(row -> + { + assertThat(row.getValueString("outcome")).isEqualTo("OK"); + assertThat(row.getValueString("objectKind")).isEqualTo("VIEW"); + assertThat(row.getValueString("objectId")).isEqualTo(saved.getValueString("id")); + assertThat(row.getValueLong("bytes")).isPositive(); + assertThat(row.getValueLong("durationMs")).isNotNegative(); + assertThat(row.getValueString("format")).isIn("csv", "xlsx", "json"); + }); + assertThat(h.calls).allSatisfy(call -> + { + assertThat(call.channel()).startsWith("sftp/"); + assertThat(call.invokedBy()).isEqualTo("file-gateway"); + }); + } + } + + + /******************************************************************************* + ** FieldSecurityLock DENY removes the protected value exactly as a direct user export. + *******************************************************************************/ + @Test + void fieldSecurityLockMatchesDirectUserExport() throws Exception + { + try(var h = new FileGatewayTestServer(Clock.systemUTC(), instance -> + { + instance.setEnforcePermissionsInCoreActions(true); + instance.addSecurityKeyType(new QSecurityKeyType().withName("privateField").withAllAccessKeyName("privateFieldAll")); + instance.getTable("alpha").getField("name").setFieldSecurityLock(new FieldSecurityLock() + .withSecurityKeyType("privateField").withDefaultBehavior(FieldSecurityLock.Behavior.DENY).withOverrideValues(List.of("reader"))); + })) + { + h.seedRows(3); + view(h, "{\"queryFilter\":{}}"); + h.audit.system(() -> new InsertAction().execute(new InsertInput("savedView").withRecord(new QRecord() + .withValue("tableName", "alpha").withValue("label", "Denied field").withValue("userId", "A").withValue("viewJson", VIEW)))); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(h.account("alice", "A"), key); + var user = h.audit.system(() -> new QAuthenticationModuleDispatcher().getQModule(h.instance.getAuthentication()).createAutomatedSessionForUser(h.instance, "A")); + String token = UiApiInProcessSessions.register(user, Duration.ofMinutes(5)); + try(var session = h.login("alice", key); var client = SftpClientFactory.instance().createSftpClient(session)) + { + var direct = h.runtime.getInvoker().invoke(new UiApiRequest().withMethod(HttpMethod.POST).withPath("/table/{tableName}/export") + .withPathParam("tableName", "alpha").withCredentials(UiApiCredentials.ofInProcessSession(token)).withClientDetail("direct-comparison") + .withJsonBody(Map.of("format", "csv", "filter", Map.of(), "includeHeaderRow", true))); + assertThat(direct.isSuccess()).as(direct.getBody()).isTrue(); + byte[] actual = read(client, "/testapp/alpha/views/selected-rows.csv"); + assertThat(new String(actual, StandardCharsets.UTF_8)).doesNotContain("Row "); + assertThat(actual).isEqualTo(direct.getBody().getBytes(StandardCharsets.UTF_8)); + assertThatThrownBy(() -> read(client, "/testapp/alpha/views/denied-field.csv")).isInstanceOf(SftpException.class); + } + finally + { + UiApiInProcessSessions.revoke(token); + } + } + } + + + /******************************************************************************* + ** The daily quota excludes prior UTC days, other accounts and non-download operations. + *******************************************************************************/ + @Test + void dailyQuotaUsesFilteredUtcHistory() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + h.seedRows(3); + view(h, "{}"); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + var account = h.account("alice", "A"); + h.audit.update("file_gateway_account", new QRecord().withValue("id", account.getValue("id")).withValue("maxDownloadMbPerDay", 1)); + h.seedTransfer(account, "DOWNLOAD", 5_000_000L, Instant.now().minusSeconds(86400)); + h.seedTransfer(account, "UPLOAD", 5_000_000L, Instant.now()); + h.seedTransfer(h.account("other", "A"), "DOWNLOAD", 5_000_000L, Instant.now()); + h.key(account, key); + try(var session = h.login("alice", key); var client = SftpClientFactory.instance().createSftpClient(session)) + { + assertThat(read(client, "/testapp/alpha/views/selected-rows.csv")).isNotEmpty(); + assertThatThrownBy(() -> read(client, "/testapp/alpha/views/selected-rows.json")).isInstanceOfSatisfying(SftpException.class, + e -> assertThat(e.getStatus()).isEqualTo(SftpConstants.SSH_FX_NO_SUCH_FILE)); + } + } + } + + + /******************************************************************************* + ** Non-success stream responses preserve their text and distinguish denied transfers. + *******************************************************************************/ + @ParameterizedTest + @ValueSource(ints = {403, 500}) + void apiErrorsAreAudited(int status) throws Exception + { + try(var h = new FileGatewayTestServer()) + { + view(h, "{}"); + h.exportFailureStatus = status; + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(h.account("alice", "A"), key); + try(var session = h.login("alice", key); var client = SftpClientFactory.instance().createSftpClient(session)) + { + assertThatThrownBy(() -> read(client, "/testapp/alpha/views/selected-rows.csv")) + .isInstanceOf(SftpException.class).hasMessageContaining("Export failure test"); + } + assertThat(h.audit.query("file_gateway_transfer_log", new QQueryFilter())).anySatisfy(row -> + { + assertThat(row.getValueString("operation")).isEqualTo("DOWNLOAD"); + assertThat(row.getValueString("outcome")).isEqualTo(status == 403 ? "DENIED" : "ERROR"); + assertThat(row.getValueString("message")).contains("Export failure test"); + assertThat(row.getValueLong("bytes")).isZero(); + }); + } + } + + + /******************************************************************************* + ** Real renderSavedReport input/execute/output and storage download as the SSH user. + *******************************************************************************/ + @Test + void savedReportDownload() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + h.seedRows(3); + QRecord report = h.audit.system(() -> new InsertAction().execute(new InsertInput("savedReport").withRecord(new QRecord() + .withValue("tableName", "alpha").withValue("label", "Daily").withValue("userId", "A") + .withValue("columnsJson", "{\"columns\":[{\"name\":\"quantity\"}]}") + .withValue("queryFilterJson", "{\"criteria\":[{\"fieldName\":\"quantity\",\"operator\":\"EQUALS\",\"values\":[2]}]}"))).getRecords().getFirst()); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(h.account("alice", "A"), key); + try(var session = h.login("alice", key); var client = SftpClientFactory.instance().createSftpClient(session)) + { + String csv = new String(read(client, "/testapp/alpha/reports/daily.csv"), StandardCharsets.UTF_8); + assertThat(csv.replace("\r\n", "\n")).isEqualTo("\"Quantity\"\n\"2\"\n"); + } + assertThat(h.calls).anySatisfy(call -> assertThat(call.path()).contains("/processes/renderSavedReport/init")) + .anySatisfy(call -> assertThat(call.path()).contains("/download/")) + .allSatisfy(call -> assertThat(call.channel()).startsWith("sftp/")); + assertThat(h.audit.query("file_gateway_transfer_log", new QQueryFilter())).anySatisfy(row -> + { + assertThat(row.getValueString("objectKind")).isEqualTo("REPORT"); + assertThat(row.getValueString("objectId")).isEqualTo(report.getValueString("id")); + assertThat(row.getValueString("outcome")).isEqualTo("OK"); + }); + } + } + + + /******************************************************************************* + ** Missing configured storage hides the report folder, including guessed paths. + *******************************************************************************/ + @Test + void missingReportStorageHidesFolder() throws Exception + { + try(var h = new FileGatewayTestServer(Clock.systemUTC(), instance -> instance.getTables().remove("reportStorage"))) + { + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(h.account("alice", "A"), key); + try(var session = h.login("alice", key); var client = SftpClientFactory.instance().createSftpClient(session)) + { + List names = new ArrayList<>(); + client.readDir("/testapp/alpha").forEach(entry -> names.add(entry.getFilename())); + assertThat(names).doesNotContain("reports"); + assertThatThrownBy(() -> client.stat("/testapp/alpha/reports")).isInstanceOfSatisfying(SftpException.class, + e -> assertThat(e.getStatus()).isEqualTo(SftpConstants.SSH_FX_NO_SUCH_FILE)); + } + } + } + + + /******************************************************************************* + ** Large exports are consumed with fixed buffers; caps fail during read and audit LIMIT. + *******************************************************************************/ + @ParameterizedTest + @ValueSource(strings = {"large", "generated", "daily"}) + void streamingAndLimits(String scenario) throws Exception + { + try(var h = new FileGatewayTestServer()) + { + h.seedRows(50000); + view(h, "{\"fieldNames\":[\"name\"],\"queryFilter\":{\"orderBys\":[{\"fieldName\":\"id\",\"isAscending\":true}]}}"); + h.config.withDownloadReadAheadMb(1).withMaxGeneratedFileMb(scenario.equals("generated") ? 1 : 20); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + var account = h.account("alice", "A"); + if(scenario.equals("daily")) + { + h.audit.update("file_gateway_account", new QRecord().withValue("id", account.getValue("id")).withValue("maxDownloadMbPerDay", 1)); + h.audit.insert("file_gateway_transfer_log", new QRecord().withValue("accountId", account.getValue("id")) + .withValue("operation", "DOWNLOAD").withValue("bytes", 900000L).withValue("outcome", "OK") + .withValue("path", "/earlier.csv").withValue("clientChannel", "sftp/earlier")); + } + h.key(account, key); + try(var session = h.login("alice", key); var client = SftpClientFactory.instance().createSftpClient(session)) + { + if(scenario.equals("large")) + { + try(var input = client.read("/testapp/alpha/views/selected-rows.csv")) + { + byte[] buffer = new byte[32768]; + long bytes = 0; + int lines = 0; + for(int count; (count = input.read(buffer)) >= 0; ) + { + bytes += count; + for(int i = 0; i < count; i++) + { + if(buffer[i] == '\n') + { + lines++; + } + } + } + assertThat(bytes).isGreaterThan(3_000_000); + assertThat(lines).isEqualTo(50001); + } + } + else + { + assertThatThrownBy(() -> read(client, "/testapp/alpha/views/selected-rows.csv")).isInstanceOf(java.io.IOException.class) + .hasMessageContaining("limit"); + } + } + assertThat(h.audit.query("file_gateway_transfer_log", new QQueryFilter())).anySatisfy(row -> + { + assertThat(row.getValueString("objectKind")).isEqualTo("VIEW"); + assertThat(row.getValueString("outcome")).isEqualTo(scenario.equals("large") ? "OK" : "LIMIT"); + assertThat(row.getValueLong("bytes")).isPositive().isLessThanOrEqualTo(scenario.equals("large") ? 4_000_000L : 1_048_576L); + }); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/EntityContractTest.java b/src/test/java/com/kingsrook/qbits/filegateway/EntityContractTest.java new file mode 100644 index 0000000..09da93d --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/EntityContractTest.java @@ -0,0 +1,99 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + +import java.lang.reflect.Modifier; +import java.time.Instant; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qqq.backend.core.model.data.QField; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.data.QRecordEntity; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import static org.assertj.core.api.Assertions.assertThat; + +/******************************************************************************* + ** The stable schema and record entities must round-trip every persisted field. + *******************************************************************************/ +class EntityContractTest +{ + /******************************************************************************* + ** Verify annotations, metadata completeness and fluent accessors against QRecord conversion. + *******************************************************************************/ + @ParameterizedTest + @ValueSource(strings = {"Account", "SshKey", "Alias", "UploadJob", "OutboxFile", "TransferLog", "ConnectionLog", "Server"}) + void entityRoundTrip(String suffix) throws Exception + { + Class type = Class.forName("com.kingsrook.qbits.filegateway.model.FileGateway" + suffix); + var entity = (QRecordEntity) type.getConstructor().newInstance(); + String tableName = (String) type.getField("TABLE_NAME").get(null); + var table = FileGatewayTables.tables(new FileGatewayQBitConfig()).stream().filter(t -> t.getName().equals(tableName)).findFirst().orElseThrow(); + for(var field : type.getDeclaredFields()) + { + if(Modifier.isStatic(field.getModifiers())) + { + continue; + } + assertThat(field.getAnnotation(QField.class)).isNotNull(); + assertThat(table.getField(field.getName())).as(field.getName()).isNotNull(); + Object value = field.getType() == String.class ? "value-" + field.getName() : field.getType() == Instant.class ? Instant.parse("2026-10-13T12:34:56Z") + : field.getType() == Boolean.class ? true : field.getType() == Long.class ? (Object) 12345678901L : 17; + String capitalized = Character.toUpperCase(field.getName().charAt(0)) + field.getName().substring(1); + assertThat(type.getMethod("with" + capitalized, field.getType()).invoke(entity, value)).isSameAs(entity); + assertThat(type.getMethod("get" + capitalized).invoke(entity)).isEqualTo(value); + } + QRecord record = entity.toQRecord(); + var restored = (QRecordEntity) type.getConstructor(QRecord.class).newInstance(record); + assertThat(restored.toQRecord().getValues()).containsAllEntriesOf(record.getValues()); + assertThat(record.getValues().keySet()).containsExactlyInAnyOrderElementsOf(table.getFields().keySet()); + } + + + + /******************************************************************************* + ** All documented defaults are available before any runtime has started. + *******************************************************************************/ + @Test + void configurationDefaultsAndFluentOptions() throws Exception + { + var config = new FileGatewayQBitConfig(); + assertThat(config.getSftpEnabled()).isFalse(); + assertThat(config.getSftpPort()).isEqualTo(2222); + assertThat(config.getAllowGeneratedHostKey()).isFalse(); + assertThat(config.getHostKeyEnvVarName()).isEqualTo("FILE_GATEWAY_SSH_HOST_KEY"); + assertThat(config.getDefaultMaxConcurrentSessions()).isEqualTo(4); + assertThat(config.getDefaultMaxOperationsPerMinute()).isEqualTo(600); + assertThat(config.getLockoutThreshold()).isEqualTo(5); + assertThat(config.getTemporaryUploadSuffixes()).containsExactly(".filepart", ".tmp", ".part", ".partial"); + for(var method : FileGatewayQBitConfig.class.getDeclaredMethods()) + { + if(!method.getName().startsWith("with")) + { + continue; + } + var getter = FileGatewayQBitConfig.class.getMethod("get" + method.getName().substring(4)); + Object value = getter.invoke(config); + assertThat(method.invoke(config, value)).isSameAs(config); + assertThat(getter.invoke(config)).isEqualTo(value); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/FileGatewayDemoServer.java b/src/test/java/com/kingsrook/qbits/filegateway/FileGatewayDemoServer.java new file mode 100644 index 0000000..58d9ea0 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/FileGatewayDemoServer.java @@ -0,0 +1,159 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2026. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + + +import java.time.Clock; +import java.time.Instant; +import java.time.temporal.ChronoUnit; +import java.util.List; +import java.util.Random; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import org.apache.sshd.common.config.keys.KeyUtils; + + +/******************************************************************************* + ** Demo server for screenshots: the full test instance with the Next UI on + ** port 8099 (or the first argument), MOCK auth granting every permission to + ** browser sessions, and realistic gateway data. Test scope only. + *******************************************************************************/ +public final class FileGatewayDemoServer +{ + private static final List FILES = List.of("orders-2026-10-13.csv", "inventory.xlsx", "returns.csv", "price-update.csv", "partners.csv", "shipments-week41.csv"); + + + + /******************************************************************************* + ** Utility class. + *******************************************************************************/ + private FileGatewayDemoServer() + { + } + + + + /******************************************************************************* + ** Start, seed and keep running until killed. + *******************************************************************************/ + public static void main(String[] args) throws Exception + { + int port = args.length > 0 ? Integer.parseInt(args[0]) : 8099; + FileGatewayTestServer.TestAuthentication.grantAllToUiSessions = true; + FileGatewayTestServer server = start(port); + System.out.println("File gateway demo: http://localhost:" + port + "/app/file_gateway"); + Thread.currentThread().join(); + server.close(); + } + + + + /******************************************************************************* + ** Start the server and seed demo data. + *******************************************************************************/ + static FileGatewayTestServer start(int port) throws Exception + { + FileGatewayTestServer server = new FileGatewayTestServer(Clock.systemUTC(), instance -> + { + }, port, true, true); + seed(server); + return (server); + } + + + + /******************************************************************************* + ** Seed accounts, keys, endpoints, jobs, transfers, connections and outbox files. + *******************************************************************************/ + static void seed(FileGatewayTestServer h) throws Exception + { + Random random = new Random(42); + Instant now = Instant.now(); + + QRecord acme = h.audit.insert(FileGatewayTables.ACCOUNT, new QRecord().withValue("username", "acme-orders").withValue("label", "Acme Corp (orders feed)") + .withValue("accountType", "SERVICE").withValue("userId", "acme-svc").withValue("ownerUserId", "jamie").withValue("allowedCidrs", "203.0.113.0/24")); + QRecord globex = h.audit.insert(FileGatewayTables.ACCOUNT, new QRecord().withValue("username", "globex").withValue("label", "Globex Logistics") + .withValue("accountType", "SERVICE").withValue("userId", "globex-svc").withValue("ownerUserId", "jamie")); + QRecord jamie = h.audit.insert(FileGatewayTables.ACCOUNT, new QRecord().withValue("username", "jamie").withValue("label", "Jamie Rivera") + .withValue("accountType", "USER").withValue("userId", "jamie")); + h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", globex.getValue("id")).withValue("failedLoginCount", 5) + .withValue("lockedUntil", now.plus(25, ChronoUnit.MINUTES))); + List accounts = List.of(acme, globex, jamie); + + for(QRecord account : accounts) + { + h.key(account, KeyUtils.generateKeyPair("ssh-ed25519", 256)); + } + + QRecord inbox = h.audit.insert(FileGatewayTables.ALIAS, new QRecord().withValue("name", "acme-orders-in").withValue("path", "/acme/orders-in") + .withValue("kind", "INBOX").withValue("consumerName", "bulk-load").withValue("accountId", acme.getValue("id")).withValue("description", "Acme drops nightly order files here")); + QRecord outbox = h.audit.insert(FileGatewayTables.ALIAS, new QRecord().withValue("name", "globex-acks").withValue("path", "/globex/acks") + .withValue("kind", "OUTBOX").withValue("accountId", globex.getValue("id")).withValue("description", "Acknowledgements Globex picks up")); + h.audit.insert(FileGatewayTables.ALIAS, new QRecord().withValue("name", "alpha-daily").withValue("path", "/reports/alpha-daily") + .withValue("kind", "VIEW").withValue("tableName", "alpha").withValue("format", "csv").withValue("fileName", "alpha-daily.csv")); + + List statuses = List.of("SUCCEEDED", "SUCCEEDED", "SUCCEEDED", "SUCCEEDED", "PARTIAL", "FAILED", "REJECTED", "DUPLICATE", "QUEUED", "RUNNING"); + for(int i = 0; i < 40; i++) + { + QRecord account = accounts.get(i % accounts.size()); + String status = statuses.get(random.nextInt(statuses.size())); + String file = FILES.get(random.nextInt(FILES.size())); + Instant finished = now.minus(random.nextInt(7 * 24 * 60), ChronoUnit.MINUTES); + int inserted = "FAILED".equals(status) || "REJECTED".equals(status) ? 0 : 50 + random.nextInt(900); + int errors = "PARTIAL".equals(status) ? 1 + random.nextInt(12) : 0; + h.audit.insert(FileGatewayTables.UPLOAD_JOB, new QRecord().withValue("accountId", account.getValue("id")) + .withValue("aliasId", account == acme ? inbox.getValue("id") : null).withValue("tableName", "alpha").withValue("consumerName", "bulk-load") + .withValue("path", "/testapp/alpha/upload/orders/" + file).withValue("fileName", file).withValue("sizeBytes", 20_000L + random.nextInt(4_000_000)) + .withValue("sha256", Long.toHexString(random.nextLong()) + Long.toHexString(random.nextLong())).withValue("status", status) + .withValue("insertedCount", inserted).withValue("errorCount", errors).withValue("remoteAddress", "203.0.113." + (10 + random.nextInt(40))) + .withValue("errorMessage", "FAILED".equals(status) ? "this profile needs value mappings saved in it" : "REJECTED".equals(status) ? "Virus scan: infected" : null) + .withValue("queuedAt", finished.minusSeconds(40)).withValue("startedAt", finished.minusSeconds(30)) + .withValue("finishedAt", List.of("QUEUED", "RUNNING").contains(status) ? null : finished).withValue("attempts", 1)); + } + + for(int i = 0; i < 60; i++) + { + QRecord account = accounts.get(random.nextInt(accounts.size())); + String op = List.of("LIST", "UPLOAD", "DOWNLOAD", "LIST", "DOWNLOAD").get(random.nextInt(5)); + h.seedTransfer(account, op, "LIST".equals(op) ? 0 : 10_000L + random.nextInt(8_000_000), now.minus(random.nextInt(6 * 24 * 60), ChronoUnit.MINUTES)); + } + + for(int i = 0; i < 12; i++) + { + QRecord account = accounts.get(i % accounts.size()); + boolean live = i < 2; + Instant start = now.minus(live ? 5 + i * 7 : 60 + random.nextInt(5000), ChronoUnit.MINUTES); + h.audit.insert(FileGatewayTables.CONNECTION_LOG, new QRecord().withValue("accountId", account.getValue("id")).withValue("username", account.getValueString("username")) + .withValue("remoteIp", "203.0.113." + (10 + i)).withValue("clientVersion", "SSH-2.0-OpenSSH_9.8").withValue("outcome", i == 5 ? "AUTH_FAILED" : "CONNECTED") + .withValue("sessionId", "demo-" + i).withValue("podName", "file-gateway-" + (i % 2 == 0 ? "7f9c4-abcde" : "7f9c4-fghij")).withValue("connectedAt", start) + .withValue("lastSeenAt", live ? now : start.plus(4, ChronoUnit.MINUTES)).withValue("disconnectedAt", live ? null : start.plus(4, ChronoUnit.MINUTES)) + .withValue("bytesIn", (long) random.nextInt(9_000_000)).withValue("bytesOut", (long) random.nextInt(2_000_000)).withValue("operationCount", (long) (3 + random.nextInt(40))) + .withValue("clientChannel", "sftp")); + } + + for(String name : List.of("ack-1041.csv", "ack-1042.csv", "ack-1043.csv")) + { + h.store.put("file-gateway/outbox/" + outbox.getValue("id") + "/" + name, new java.io.ByteArrayInputStream("order,status\n1041,received\n".getBytes(java.nio.charset.StandardCharsets.UTF_8))); + h.audit.insert(FileGatewayTables.OUTBOX_FILE, new QRecord().withValue("aliasId", outbox.getValue("id")).withValue("fileName", name).withValue("sizeBytes", 26L) + .withValue("storageKey", "file-gateway/outbox/" + outbox.getValue("id") + "/" + name).withValue("status", name.endsWith("3.csv") ? "READY" : "DOWNLOADED")); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/FileGatewayDemoServerTest.java b/src/test/java/com/kingsrook/qbits/filegateway/FileGatewayDemoServerTest.java new file mode 100644 index 0000000..a7422ee --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/FileGatewayDemoServerTest.java @@ -0,0 +1,60 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2026. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + + +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; + + +/******************************************************************************* + ** The demo server seeds and serves (smoke test; the demo itself is manual). + *******************************************************************************/ +class FileGatewayDemoServerTest +{ + /******************************************************************************* + ** Reset the demo-only grant. + *******************************************************************************/ + @AfterEach + void reset() + { + FileGatewayTestServer.TestAuthentication.grantAllToUiSessions = false; + } + + + + /******************************************************************************* + ** Start on a free port, check seeded data, stop. + *******************************************************************************/ + @Test + void startsAndSeeds() throws Exception + { + try(FileGatewayTestServer server = FileGatewayDemoServer.start(0)) + { + assertThat(server.audit.query(FileGatewayTables.ACCOUNT, new QQueryFilter())).hasSize(3); + assertThat(server.audit.query(FileGatewayTables.UPLOAD_JOB, new QQueryFilter())).hasSize(40); + assertThat(server.audit.query(FileGatewayTables.OUTBOX_FILE, new QQueryFilter())).hasSize(3); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/FileGatewayQBitProducerTest.java b/src/test/java/com/kingsrook/qbits/filegateway/FileGatewayQBitProducerTest.java index 30005c6..ab82d9c 100644 --- a/src/test/java/com/kingsrook/qbits/filegateway/FileGatewayQBitProducerTest.java +++ b/src/test/java/com/kingsrook/qbits/filegateway/FileGatewayQBitProducerTest.java @@ -20,74 +20,96 @@ package com.kingsrook.qbits.filegateway; - -import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput; +import java.util.List; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; import com.kingsrook.qqq.backend.core.model.metadata.QInstance; -import com.kingsrook.qqq.backend.core.model.metadata.producers.MetaDataCustomizerInterface; -import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData; -import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.ClientChannelsMetaData; import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; import static org.assertj.core.api.Assertions.assertThat; - /******************************************************************************* - ** Verify the scaffold's registration and configuration contract. + ** Registration, channel walls and contingent endpoint pickers. *******************************************************************************/ class FileGatewayQBitProducerTest { /******************************************************************************* - ** Produce the identity and exact config into an otherwise empty instance. + ** All stable metadata belongs to one configured qbit. *******************************************************************************/ @Test - void producesQBitWithoutTables() + void registersTablesAppPickersAndWalls() throws Exception { - QInstance instance = new QInstance(); - FileGatewayQBitConfig config = new FileGatewayQBitConfig(); - FileGatewayQBitProducer producer = new FileGatewayQBitProducer().withQBitConfig(config); - MetaDataProducerMultiOutput output = producer.produce(instance); - QBitMetaData qBit = instance.getQBits().get("com.kingsrook.qbits:qbit-file-gateway"); - - assertThat(qBit).isNotNull(); - assertThat(qBit.getGroupId()).isEqualTo("com.kingsrook.qbits"); - assertThat(qBit.getArtifactId()).isEqualTo("qbit-file-gateway"); - assertThat(qBit.getVersion()).isEqualTo("0.1.0"); - assertThat(qBit.getConfig()).isSameAs(config); + var instance = new QInstance(); + var config = new FileGatewayQBitConfig().withBackendName("host-db").withTableMetaDataCustomizer((i, t) -> t.withLabel("Custom " + t.getLabel())); + var producer = new FileGatewayQBitProducer().withQBitConfig(config).withNamespace("test"); + producer.produce(instance).addSelfToInstance(instance); + assertThat(instance.getTables().keySet()).containsExactlyInAnyOrderElementsOf(FileGatewayTables.ALL); + assertThat(instance.getQBits().get("com.kingsrook.qbits:qbit-file-gateway:test").getConfig()).isSameAs(config); assertThat(producer.getQBitConfig()).isSameAs(config); - assertThat(producer.isEnabled()).isTrue(); - assertThat(producer.getNamespace()).isNull(); - assertThat(output.getEach(QBitMetaData.class)).containsExactly(qBit); - assertThat(output.getEach(QTableMetaData.class)).isEmpty(); - assertThat(instance.getTables()).isEmpty(); - assertThat(config.getSftpEnabled()).isFalse(); - assertThat(config.getSftpPort()).isEqualTo(2222); - assertThat(config.getTableMetaDataCustomizer()).isNull(); + assertThat(producer.getNamespace()).isEqualTo("test"); + assertThat(instance.getApps()).containsOnlyKeys("file_gateway"); + assertThat(instance.getApp("file_gateway").getSections().stream().flatMap(s -> s.getTables().stream()).toList()) + .containsExactlyElementsOf(FileGatewayTables.MAIN); + assertThat(instance.getJoins()).hasSize(6); + assertThat(instance.getWidgets()).hasSize(16); + assertThat(instance.getPossibleValueSources()).hasSize(20); + instance.getTables().values().forEach(t -> + { + assertThat(t.getBackendName()).isEqualTo("host-db"); + assertThat(t.getLabel()).startsWith("Custom "); + assertThat(t.getField("createDate").getBackendName()).isEqualTo("create_date"); + }); + var channels = ClientChannelsMetaData.ofOrWithNew(instance); + assertThat(channels.getChannels()).containsKey("sftp"); + for(String name : List.of("file_gateway_account", "authentik_user", "audit", "api_key", "credential", "secret", "password", "permission", "role", "user_session", "webhooks_endpoint")) + { + assertThat(channels.isDenied("sftp", name, null)).as(name).isTrue(); + } + assertThat(channels.isDenied("assistant", "file_gateway_ssh_key", null)).isTrue(); + assertThat(channels.isDenied("sftp", "orders", null)).isFalse(); + assertThat(instance.getTable(FileGatewayTables.ALIAS).getField("tableName").getPossibleValueSourceName()).isEqualTo("tables"); } /******************************************************************************* - ** Configured instances may be namespaced; unconfigured scanning is disabled. + ** Package scanning never enables an unconfigured producer. *******************************************************************************/ @Test - void supportsNamespacedConfiguration() + void disabledWithoutConfig() throws Exception { - assertThat(new FileGatewayQBitProducer().isEnabled()).isFalse(); - MetaDataCustomizerInterface customizer = (instance, table) -> table; - FileGatewayQBitConfig config = new FileGatewayQBitConfig() - .withTableMetaDataCustomizer(customizer) - .withSftpEnabled(true) - .withSftpPort(2022); - FileGatewayQBitProducer producer = new FileGatewayQBitProducer() - .withQBitConfig(config) - .withNamespace("test"); - QInstance instance = new QInstance(); - producer.produce(instance); - - assertThat(producer.getNamespace()).isEqualTo("test"); - assertThat(instance.getQBits().get("com.kingsrook.qbits:qbit-file-gateway:test").getConfig()).isSameAs(config); - assertThat(config.getTableMetaDataCustomizer()).isSameAs(customizer); - assertThat(config.getSftpEnabled()).isTrue(); - assertThat(config.getSftpPort()).isEqualTo(2022); + var instance = new QInstance(); + var producer = new FileGatewayQBitProducer(); + assertThat(producer.isEnabled()).isFalse(); + producer.produce(instance).addSelfToInstance(instance); assertThat(instance.getTables()).isEmpty(); + assertThat(instance.getQBits()).isEmpty(); + assertThat(FileGatewayRuntime.start(instance, null)).isNull(); + new FileGatewayQBitProducer().withQBitConfig(new FileGatewayQBitConfig()).produce(instance); + assertThat(FileGatewayRuntime.start(instance, null, name -> null)).isNull(); + } + + + + /******************************************************************************* + ** Each dependent endpoint field has exactly the specified input criterion. + *******************************************************************************/ + @ParameterizedTest + @ValueSource(strings = {"savedBulkLoadProfile", "savedView", "savedReport"}) + void contingentPickerMetadata(String source) throws Exception + { + var instance = new QInstance(); + new FileGatewayQBitProducer().withQBitConfig(new FileGatewayQBitConfig()).produce(instance); + var field = instance.getTable(FileGatewayTables.ALIAS).getField(source + "Id"); + assertThat(field.getPossibleValueSourceName()).isEqualTo(source); + var filter = field.getPossibleValueSourceFilter(); + assertThat(filter.getCriteria()).singleElement().satisfies(c -> + { + assertThat(c.getFieldName()).isEqualTo("tableName"); + assertThat(c.getOperator().name()).isEqualTo("EQUALS"); + assertThat(c.getValues()).containsExactly("${input.tableName}"); + }); + assertThat(filter.getSubFilters()).isNullOrEmpty(); } } diff --git a/src/test/java/com/kingsrook/qbits/filegateway/FileGatewayTestServer.java b/src/test/java/com/kingsrook/qbits/filegateway/FileGatewayTestServer.java new file mode 100644 index 0000000..34e14a2 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/FileGatewayTestServer.java @@ -0,0 +1,420 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + + +import java.io.Serializable; +import java.nio.file.Files; +import java.nio.file.Path; +import java.security.KeyPair; +import java.sql.Connection; +import java.time.Clock; +import java.time.Duration; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import com.kingsrook.qbits.filegateway.sftp.AuditWriter; +import com.kingsrook.qqq.backend.core.actions.tables.InsertAction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.instances.AbstractQQQApplication; +import com.kingsrook.qqq.backend.core.instances.QInstanceEnricher; +import com.kingsrook.qqq.backend.core.instances.QInstanceValidator; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType; +import com.kingsrook.qqq.backend.core.model.metadata.QInstance; +import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType; +import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppSection; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel; +import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules; +import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData; +import com.kingsrook.qqq.backend.core.model.metadata.tables.SectionFactory; +import com.kingsrook.qqq.backend.core.model.metadata.tables.TablesPossibleValueSourceMetaDataProvider; +import com.kingsrook.qqq.backend.core.model.savedbulkloadprofiles.SavedBulkLoadProfileMetaDataProvider; +import com.kingsrook.qqq.backend.core.model.savedreports.SavedReportsMetaDataProvider; +import com.kingsrook.qqq.backend.core.model.savedviews.SavedViewsMetaDataProvider; +import com.kingsrook.qqq.backend.core.model.session.QSession; +import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession; +import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleCustomizerInterface; +import com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData; +import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSTableBackendDetails; +import com.kingsrook.qqq.middleware.javalin.QApplicationJavalinServer; +import com.kingsrook.qqq.middleware.javalin.specs.v1.MiddlewareVersionV1; +import org.apache.sshd.client.SshClient; +import org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier; +import org.apache.sshd.client.session.ClientSession; +import org.apache.sshd.common.config.keys.KeyUtils; +import org.apache.sshd.common.config.keys.PublicKeyEntry; + + +/******************************************************************************* + ** Real strict UI API server, in-memory H2 and real MINA clients for gateway acceptance tests. + *******************************************************************************/ +public class FileGatewayTestServer implements AutoCloseable +{ + public final MemoryGatewayObjectStore store = new MemoryGatewayObjectStore(); + public final QInstance instance = new QInstance(); + public final FileGatewayQBitConfig config = new FileGatewayQBitConfig().withSftpEnabled(true).withSftpPort(0); + public final AuditWriter audit = new AuditWriter(instance); + public final QApplicationJavalinServer http; + public final FileGatewayRuntime runtime; + public final SshClient client; + public final java.util.List calls = new java.util.concurrent.CopyOnWriteArrayList<>(); + public volatile int exportFailureStatus; + + /******************************************************************************* + ** Non-secret request evidence for the invoker contract assertions. + *******************************************************************************/ + public record ApiCall(String method, String path, String channel, String invokedBy) + { + } + private final Connection keeper; + public static final Map> GRANTS = new HashMap<>(); + + + + /******************************************************************************* + ** Start the complete in-memory test stack. + *******************************************************************************/ + public FileGatewayTestServer() throws Exception + { + this(Clock.systemUTC()); + } + + + + /******************************************************************************* + ** Controllable time for expiry, session refresh and metadata caching. + *******************************************************************************/ + public FileGatewayTestServer(Clock clock) throws Exception + { + this(clock, instance -> + { + // Most acceptance tests use the default application metadata. + }); + } + + + /******************************************************************************* + ** Customize application metadata before validation and server startup. + *******************************************************************************/ + public FileGatewayTestServer(Clock clock, java.util.function.Consumer customize) throws Exception + { + this(clock, customize, 0, false, true); + } + + + + /******************************************************************************* + ** Demo and web-only tests can serve Next without starting an SFTP listener. + *******************************************************************************/ + public FileGatewayTestServer(Clock clock, java.util.function.Consumer customize, int port, boolean frontend, boolean sftp) throws Exception + { + config.withSftpEnabled(sftp); + ConnectionManager.resetConnectionProviders(); + GRANTS.clear(); + GRANTS.put("A", Set.of("file_gateway.connect", "alpha.read", "alpha.insert", "beta.read", "beta.insert")); + GRANTS.put("B", Set.of("file_gateway.connect", "beta.read", "beta.insert")); + GRANTS.put("NONE", Set.of("beta.read")); + instance.withInstanceDefaultAuthentication(new QAuthenticationMetaData().withName("test").withType(QAuthenticationType.MOCK) + .withCustomizer(new QCodeReference(TestAuthentication.class))); + RDBMSBackendMetaData backend = new RDBMSBackendMetaData().withName("rdbms").withVendor("h2").withHostName("mem") + .withDatabaseName("fg" + UUID.randomUUID().toString().replace("-", "")).withUsername("sa"); + backend.setJdbcUrl(ConnectionManager.getJdbcUrl(backend) + ";DATABASE_TO_UPPER=FALSE"); + instance.addBackend(backend); + keeper = new ConnectionManager().getConnection(backend); + for(String sql : Files.readString(Path.of("docs/ddl/file_gateway.sql")).split(";")) + { + if(!sql.isBlank()) + { + keeper.createStatement().execute(sql); + } + } + for(String name : List.of("alpha", "beta")) + { + instance.addTable(new QTableMetaData().withName(name).withLabel(name).withBackendName("rdbms") + .withPrimaryKeyField("id").withField(new QFieldMetaData("id", QFieldType.INTEGER)) + .withField(new QFieldMetaData("name", QFieldType.STRING).withIsRequired(true)) + .withField(new QFieldMetaData("quantity", QFieldType.INTEGER)) + .withSection(SectionFactory.defaultT1("id", "name", "quantity")) + .withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.READ_INSERT_EDIT_DELETE_PERMISSIONS).withPermissionBaseName(name))); + } + instance.addApp(new QAppMetaData().withName("testapp").withLabel("Test App") + .withChild(instance.getTable("alpha")).withChild(instance.getTable("beta")) + .withSection(new QAppSection().withName("data").withLabel("Data").withTables(List.of("alpha", "beta")))); + instance.addPossibleValueSource(TablesPossibleValueSourceMetaDataProvider.defineTablesPossibleValueSource(instance)); + new SavedBulkLoadProfileMetaDataProvider().defineAll(instance, "rdbms", null); + new SavedViewsMetaDataProvider().defineAll(instance, "rdbms", null); + new SavedReportsMetaDataProvider().defineAll(instance, "rdbms", "rdbms", null); + new FileGatewayQBitProducer().withQBitConfig(config).produce(instance); + new com.kingsrook.qqq.backend.core.model.audits.AuditsMetaDataProvider().withClientChannelField(true).defineAll(instance, "rdbms", null); + instance.getTable("alpha").withAuditRules(new com.kingsrook.qqq.backend.core.model.metadata.audits.QAuditRules() + .withAuditLevel(com.kingsrook.qqq.backend.core.model.metadata.audits.AuditLevel.RECORD)); + TestAlertSink.notifications.clear(); + com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData.ofOrWithNew(instance).withSink(new QCodeReference(TestAlertSink.class)); + for(QTableMetaData table : instance.getTables().values()) + { + if(!table.getName().startsWith("file_gateway_")) + { + table.withBackendDetails(new RDBMSTableBackendDetails().withTableName(table.getName())); + for(QFieldMetaData field : table.getFields().values()) + { + field.setBackendName(QInstanceEnricher.inferBackendName(field.getName())); + } + createTable(table); + } + } + instance.addBackend(new com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData() + .withName("memory").withBackendType(com.kingsrook.qqq.backend.core.modules.backend.implementations.memory.MemoryBackendModule.class)); + instance.getTable("reportStorage").setBackendName("memory"); + instance.addTable(new QTableMetaData().withName("testUploadArchive").withBackendName("memory").withPrimaryKeyField("reference") + .withField(new QFieldMetaData("reference", QFieldType.STRING)).withField(new QFieldMetaData("contents", QFieldType.BLOB))); + instance.withSupplementalMetaData(new com.kingsrook.qqq.middleware.javalin.QJavalinMetaData().withUploadedFileArchiveTableName("testUploadArchive")); + customize.accept(instance); + new QInstanceEnricher(instance).enrich(); + new QInstanceValidator().validate(instance); + QContext.init(instance, new QSystemUserSession(instance)); + http = new QApplicationJavalinServer(new AbstractQQQApplication() + { + /******************************************************************************* + ** Define q instance. + *******************************************************************************/ + @Override + public QInstance defineQInstance() + { + return (instance); + } + }).withPort(port).withServeFrontendMaterialDashboard(false).withServeFrontendNext(frontend) + .withJavalinConfigCustomizer(j -> j.routes.before(ctx -> + { + calls.add(new ApiCall(ctx.method().name(), ctx.path(), ctx.header("X-QQQ-Client"), ctx.header("X-QQQ-Invoked-By"))); + if(exportFailureStatus != 0 && ctx.path().endsWith("/export")) + { + throw (new io.javalin.http.HttpResponseException(exportFailureStatus, "Export failure test")); + } + })) + .withStrictUiApiRoutes(true).withMiddlewareVersionList(List.of(new MiddlewareVersionV1())); + http.start(); + runtime = sftp ? FileGatewayRuntime.startForTest(instance, http, config, KeyUtils.generateKeyPair("ecdsa-sha2-nistp256", 256), clock, Duration.ofHours(1), store) : null; + if(!sftp) + { + com.kingsrook.qbits.filegateway.api.FileGatewayOutbox.setProvider(new com.kingsrook.qbits.filegateway.outbox.OutboxService(instance, store, config, clock)); + } + client = SshClient.setUpDefaultClient(); + client.setServerKeyVerifier(AcceptAllServerKeyVerifier.INSTANCE); + client.start(); + } + + + /******************************************************************************* + ** Seed many export rows without generating unrelated insert audits. + *******************************************************************************/ + public void seedRows(int count) throws Exception + { + try(var statement = keeper.prepareStatement("INSERT INTO alpha (id, name, quantity) VALUES (?, ?, ?)")) + { + for(int i = 1; i <= count; i++) + { + statement.setInt(1, i); + statement.setString(2, "Row " + i + " " + "x".repeat(50)); + statement.setInt(3, i); + statement.addBatch(); + if(i % 1000 == 0) + { + statement.executeBatch(); + } + } + statement.executeBatch(); + } + } + + + /******************************************************************************* + ** Seed quota history with an exact UTC timestamp rather than the insertion clock. + *******************************************************************************/ + public void seedTransfer(QRecord account, String operation, long bytes, java.time.Instant when) throws Exception + { + QRecord row = audit.insert("file_gateway_transfer_log", new QRecord().withValue("accountId", account.getValue("id")) + .withValue("operation", operation).withValue("bytes", bytes).withValue("outcome", "OK") + .withValue("path", "/history.csv").withValue("clientChannel", "sftp/history")); + try(var statement = keeper.prepareStatement("UPDATE file_gateway_transfer_log SET create_date = ? WHERE id = ?")) + { + statement.setObject(1, when.atOffset(java.time.ZoneOffset.UTC).toLocalDateTime()); + statement.setInt(2, row.getValueInteger("id")); + statement.executeUpdate(); + } + } + + + + /******************************************************************************* + ** Create core test tables using their actual metadata. + *******************************************************************************/ + private void createTable(QTableMetaData table) throws Exception + { + StringBuilder sql = new StringBuilder("CREATE TABLE ").append(table.getName()).append(" ("); + boolean first = true; + for(QFieldMetaData field : table.getFields().values()) + { + if(!first) + { + sql.append(", "); + } + first = false; + String type = switch(field.getType()) + { + case INTEGER -> "INTEGER"; + case LONG -> "BIGINT"; + case DECIMAL -> "DECIMAL"; + case BOOLEAN -> "BOOLEAN"; + case DATE_TIME -> "TIMESTAMP"; + case DATE -> "DATE"; + case TIME -> "TIME"; + case BLOB -> "BINARY LARGE OBJECT"; + default -> "VARCHAR(65535)"; + }; + sql.append(field.getBackendName()).append(" ").append(type); + if(field.getName().equals(table.getPrimaryKeyField()) && (field.getType() == QFieldType.INTEGER || field.getType() == QFieldType.LONG)) + { + sql.append(" GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY"); + } + } + keeper.createStatement().execute(sql.append(")").toString()); + } + + + + /******************************************************************************* + ** Insert an active account with schema defaults. + *******************************************************************************/ + public QRecord account(String username, String userId) throws Exception + { + return (audit.insert("file_gateway_account", new QRecord().withValue("username", username).withValue("label", username).withValue("userId", userId))); + } + + + + /******************************************************************************* + ** Insert a generated key through the real validation customizer. + *******************************************************************************/ + public QRecord key(QRecord account, KeyPair pair) throws Exception + { + return (audit.insert("file_gateway_ssh_key", new QRecord().withValue("accountId", account.getValue("id")) + .withValue("label", "Test key").withValue("publicKey", PublicKeyEntry.toString(pair.getPublic())))); + } + + + + /******************************************************************************* + ** Authenticate a real MINA SSH client. + *******************************************************************************/ + public ClientSession login(String username, KeyPair key) throws Exception + { + ClientSession session = client.connect(username, "127.0.0.1", runtime.getProtocol().getPort()).verify(Duration.ofSeconds(10)).getSession(); + session.addPublicKeyIdentity(key); + try + { + session.auth().verify(Duration.ofSeconds(10)); + return (session); + } + catch(Exception e) + { + session.close(true).await(Duration.ofSeconds(5)); + throw (e); + } + } + + + + /******************************************************************************* + ** Seed visible core saved-object rows under their owner. + *******************************************************************************/ + public QRecord saved(String source, String table, String label) throws Exception + { + return (audit.system(() -> new InsertAction().execute(new InsertInput(source).withRecord(new QRecord() + .withValue("tableName", table).withValue("label", label).withValue("userId", "A").withValue("viewJson", "{}") + .withValue("mappingJson", "{}").withValue("columnsJson", "{\"columns\":[{\"name\":\"id\"}]}"))).getRecords().get(0))); + } + + + + /******************************************************************************* + ** Close sessions and all services. + *******************************************************************************/ + @Override + public void close() throws Exception + { + client.stop(); + if(runtime != null) + { + runtime.stop(); + } + http.stop(); + keeper.close(); + QContext.clear(); + } + + + + /******************************************************************************* + ** MOCK automation permissions differ by user id. + *******************************************************************************/ + public static class TestAuthentication implements QAuthenticationModuleCustomizerInterface + { + /** Demo only: browser (MOCK) sessions get every permission in the instance. */ + public static volatile boolean grantAllToUiSessions = false; + + + + /******************************************************************************* + ** Browser sessions: nothing, unless the demo server switches it on. + *******************************************************************************/ + @Override + public void customizeSession(QInstance instance, QSession session, Map context) + { + if(grantAllToUiSessions) + { + java.util.Set all = new java.util.HashSet<>(com.kingsrook.qqq.backend.core.actions.permissions.PermissionsHelper.getAllAvailablePermissionNames(instance)); + all.addAll(com.kingsrook.qbits.filegateway.permissions.FileGatewayPermissions.ALL); + session.setPermissions(all); + } + } + + + + /******************************************************************************* + ** Customize automated session for user. + *******************************************************************************/ + @Override + public void customizeAutomatedSessionForUser(QInstance instance, QSession session, Serializable userId) + { + session.getUser().setFullName("Test user " + userId); + session.setPermissions(GRANTS.getOrDefault(userId.toString(), Set.of())); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/GatewayLifecycleTest.java b/src/test/java/com/kingsrook/qbits/filegateway/GatewayLifecycleTest.java new file mode 100644 index 0000000..66659e6 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/GatewayLifecycleTest.java @@ -0,0 +1,464 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + +import java.io.ByteArrayOutputStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.AccessDeniedException; +import java.nio.file.Files; +import java.nio.file.attribute.BasicFileAttributes; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.time.ZoneId; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.List; +import java.util.Set; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.Executors; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qbits.filegateway.sftp.GatewaySession; +import com.kingsrook.qbits.filegateway.vfs.GatewayFileSystem; +import com.kingsrook.qbits.filegateway.vfs.VirtualTree; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInProcessSessions; +import org.apache.sshd.client.session.ClientSession; +import org.apache.sshd.common.config.keys.KeyUtils; +import org.apache.sshd.common.config.keys.writer.openssh.OpenSSHKeyPairResourceWriter; +import org.apache.sshd.sftp.client.SftpClient; +import org.apache.sshd.sftp.client.SftpClientFactory; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/******************************************************************************* + ** Time, multiple gateway listeners, runtime configuration and NIO boundaries. + *******************************************************************************/ +@ExtendWith(NoDiskExtension.class) +class GatewayLifecycleTest +{ + /******************************************************************************* + ** Advance time deterministically without changing the machine clock. + *******************************************************************************/ + static class MutableClock extends Clock + { + private Instant now = Instant.now(); + /******************************************************************************* + ** Get zone. + *******************************************************************************/ + public ZoneId getZone() + { + return (ZoneOffset.UTC); + } + + + + /******************************************************************************* + ** With zone. + *******************************************************************************/ + public Clock withZone(ZoneId zone) + { + return (this); + } + + + + /******************************************************************************* + ** Instant. + *******************************************************************************/ + public Instant instant() + { + return (now); + } + + + + /******************************************************************************* + ** Advance. + *******************************************************************************/ + void advance(Duration duration) + { + now = now.plus(duration); + } + } + + + + /******************************************************************************* + ** Server-side session for assertions without exposing credentials in logs. + *******************************************************************************/ + private GatewaySession state(FileGatewayTestServer h) + { + return (h.runtime.getProtocol().getServer().getActiveSessions().iterator().next().getAttribute(GatewaySession.ATTRIBUTE)); + } + + + + /******************************************************************************* + ** Read names while retaining the actual opendir protocol path. + *******************************************************************************/ + private List names(SftpClient client, String path) throws Exception + { + var names = new ArrayList(); + for(var entry : client.readDir(path)) + { + names.add(entry.getFilename()); + } + return (names); + } + + + + /******************************************************************************* + ** Re-mint at five minutes, revoke the prior token, and enforce changed grants. + *******************************************************************************/ + @Test + void refreshPermissionsAndCaches() throws Exception + { + var clock = new MutableClock(); + try(var h = new FileGatewayTestServer(clock)) + { + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(h.account("alice", "A"), key); + try(var session = h.login("alice", key); var sftp = SftpClientFactory.instance().createSftpClient(session)) + { + assertThat(names(sftp, "/testapp")).contains("alpha", "beta"); + int count = h.calls.size(); + names(sftp, "/testapp"); + assertThat(h.calls).hasSize(count); + String token = state(h).token(); + assertThat(UiApiInProcessSessions.resolve(token)).isPresent(); + FileGatewayTestServer.GRANTS.put("A", Set.of("file_gateway.connect", "beta.read")); + clock.advance(Duration.ofMinutes(5)); + assertThat(names(sftp, "/testapp")).doesNotContain("alpha").contains("beta"); + assertThat(state(h).token().equals(token)).isFalse(); + assertThat(UiApiInProcessSessions.resolve(token)).isEmpty(); + FileGatewayTestServer.GRANTS.put("A", Set.of()); + clock.advance(Duration.ofMinutes(5)); + assertThatThrownBy(() -> names(sftp, "/")).isInstanceOf(Exception.class); + } + } + } + + + + /******************************************************************************* + ** Each saved-object list is cached for thirty seconds and paged past 250 rows. + *******************************************************************************/ + @Test + void savedObjectsPaginationAndSlugCollisions() throws Exception + { + var clock = new MutableClock(); + try(var h = new FileGatewayTestServer(clock)) + { + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(h.account("alice", "A"), key); + h.saved("savedView", "alpha", "Duplicate!"); + h.saved("savedView", "alpha", "Duplicate?"); + try(var session = h.login("alice", key); var sftp = SftpClientFactory.instance().createSftpClient(session)) + { + assertThat(names(sftp, "/testapp/alpha/views")).contains("duplicate-1.csv", "duplicate-2.xlsx"); + h.saved("savedView", "alpha", "Fresh"); + assertThat(names(sftp, "/testapp/alpha/views")).doesNotContain("fresh.csv"); + clock.advance(Duration.ofSeconds(31)); + assertThat(names(sftp, "/testapp/alpha/views")).contains("fresh.csv"); + var rows = new ArrayList(); + for(int i = 0; i < 251; i++) + { + rows.add(new QRecord().withValue("tableName", "alpha").withValue("label", "Paged " + i).withValue("userId", "A").withValue("viewJson", "{}")); + } + h.audit.system(() -> new com.kingsrook.qqq.backend.core.actions.tables.InsertAction().execute( + new com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput("savedView").withRecords(rows))); + clock.advance(Duration.ofSeconds(31)); + assertThat(names(sftp, "/testapp/alpha/views")).contains("paged-250.csv").hasSize(510); + } + } + assertThat(VirtualTree.slug(" !!! ")).isEqualTo("item"); + assertThat(VirtualTree.slug("A".repeat(100))).hasSize(80); + } + + + + /******************************************************************************* + ** Revocation covers accounts, keys and explicit session termination, with heartbeats. + *******************************************************************************/ + @ParameterizedTest + @ValueSource(strings = {"disabled", "expired", "keyExpired", "end"}) + void pollEndsRevokedSessions(String reason) throws Exception + { + var clock = new MutableClock(); + try(var h = new FileGatewayTestServer(clock)) + { + var pair = KeyUtils.generateKeyPair("ssh-ed25519", 256); + var account = h.account("alice", "A"); + var key = h.key(account, pair); + try(var session = h.login("alice", pair)) + { + var state = state(h); + String token = state.token(); + clock.advance(Duration.ofSeconds(60)); + h.runtime.getProtocol().getAuthenticator().poll(); + var connection = h.audit.query(FileGatewayTables.CONNECTION_LOG, new QQueryFilter()).getFirst(); + assertThat(connection.getValueInstant("lastSeenAt")).isAfter(connection.getValueInstant("connectedAt")); + switch(reason) + { + case "disabled" -> h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")).withValue("status", "DISABLED")); + case "expired" -> h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")).withValue("expiresAt", clock.instant().minusSeconds(1))); + case "keyExpired" -> h.audit.update(FileGatewayTables.SSH_KEY, new QRecord().withValue("id", key.getValue("id")).withValue("expiresAt", clock.instant().minusSeconds(1))); + case "end" -> + { + com.kingsrook.qqq.backend.core.context.QContext.init(h.instance, + new com.kingsrook.qqq.backend.core.model.session.QSession().withPermissions(Set.of("file_gateway.admin"))); + new com.kingsrook.qbits.filegateway.metadata.FileGatewayProcesses.Apply().runOnePage( + new com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput() + .withProcessName(com.kingsrook.qbits.filegateway.metadata.FileGatewayProcesses.END_SESSION).withRecords(List.of(connection)), + new com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput()); + } + } + h.runtime.getProtocol().getAuthenticator().poll(); + session.waitFor(Set.of(ClientSession.ClientSessionEvent.CLOSED), Duration.ofSeconds(5)); + assertThat(session.isClosed()).isTrue(); + assertThat(UiApiInProcessSessions.resolve(token)).isEmpty(); + assertThatThrownBy(state::token).isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QException.class); + } + } + } + + + + /******************************************************************************* + ** Competing listeners share the account row lock, never exceeding the database limit. + *******************************************************************************/ + @Test + void concurrentListenersAndRestartPreserveLockout() throws Exception + { + try(var h = new FileGatewayTestServer(); var second = FileGatewayRuntime.startForTest(h.instance, h.http, + h.config, KeyUtils.generateKeyPair("ecdsa-sha2-nistp256", 256), Clock.systemUTC(), Duration.ofHours(1)); + var executor = Executors.newFixedThreadPool(2)) + { + var pair = KeyUtils.generateKeyPair("ssh-ed25519", 256); + var account = h.account("alice", "A"); + h.key(account, pair); + h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")).withValue("maxConcurrentSessions", 1)); + var start = new CountDownLatch(1); + var tasks = new ArrayList>(); + for(var runtime : List.of(h.runtime, second)) + { + tasks.add(executor.submit(() -> + { + start.await(); + var session = h.client.connect("alice", "127.0.0.1", runtime.getProtocol().getPort()).verify(Duration.ofSeconds(5)).getSession(); + session.addPublicKeyIdentity(pair); + try + { + session.auth().verify(Duration.ofSeconds(5)); + return (session); + } + catch(Exception e) + { + session.close(true).await(); + return (null); + } + })); + } + start.countDown(); + var sessions = new ArrayList(); + for(var task : tasks) + { + var session = task.get(); + if(session != null) + { + sessions.add(session); + } + } + try + { + assertThat(sessions).hasSize(1); + } + finally + { + for(var session : sessions) + { + session.close(true).await(); + } + } + assertThat(h.audit.query(FileGatewayTables.CONNECTION_LOG, new QQueryFilter())).anySatisfy(r -> assertThat(r.getValueString("outcome")).isEqualTo("LIMIT")); + h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")).withValue("lockedUntil", Instant.now().plusSeconds(900))); + h.runtime.stop(); + try(var session = h.client.connect("alice", "127.0.0.1", second.getProtocol().getPort()).verify(Duration.ofSeconds(5)).getSession()) + { + session.addPublicKeyIdentity(pair); + assertThatThrownBy(() -> session.auth().verify(Duration.ofSeconds(5))).isInstanceOf(Exception.class); + } + assertThat(h.audit.query(FileGatewayTables.CONNECTION_LOG, new QQueryFilter())).anySatisfy(r -> assertThat(r.getValueString("outcome")).isEqualTo("LOCKED_OUT")); + } + } + + + + /******************************************************************************* + ** Connection limits apply before auth; operation limits reset on the next minute. + *******************************************************************************/ + @Test + void rateAndOperationLimits() throws Exception + { + var clock = new MutableClock(); + try(var h = new FileGatewayTestServer(clock)) + { + var pair = KeyUtils.generateKeyPair("ssh-ed25519", 256); + var account = h.account("alice", "A"); + h.key(account, pair); + h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")).withValue("maxOperationsPerMinute", 1)); + try(var session = h.login("alice", pair); var sftp = SftpClientFactory.instance().createSftpClient(session)) + { + names(sftp, "/"); + try(var other = h.login("alice", pair); var otherSftp = SftpClientFactory.instance().createSftpClient(other)) + { + assertThatThrownBy(() -> names(otherSftp, "/")).isInstanceOf(Exception.class); + } + assertThatThrownBy(() -> names(sftp, "/")).isInstanceOf(Exception.class); + clock.advance(Duration.ofSeconds(61)); + assertThat(names(sftp, "/")).contains("testapp"); + } + h.config.withMaxConnectionsPerIpPerMinute(1); + try(var first = h.login("alice", pair)) + { + assertThatThrownBy(() -> h.login("alice", pair)).isInstanceOf(Exception.class); + } + assertThat(h.audit.query(FileGatewayTables.TRANSFER_LOG, new QQueryFilter())).anySatisfy(r -> assertThat(r.getValueString("outcome")).isEqualTo("LIMIT")); + } + } + + + + /******************************************************************************* + ** Rotating usernames cannot evade the per-IP lock, which expires independently. + *******************************************************************************/ + @Test + void ipLockoutAcrossUsernames() throws Exception + { + var clock = new MutableClock(); + try(var h = new FileGatewayTestServer(clock)) + { + h.config.withLockoutThreshold(3); + var good = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(h.account("alice", "A"), good); + var bad = KeyUtils.generateKeyPair("ssh-ed25519", 256); + for(String username : List.of("unknown-one", "unknown-two", "unknown-three")) + { + assertThatThrownBy(() -> h.login(username, bad)).isInstanceOf(Exception.class); + } + assertThatThrownBy(() -> h.login("alice", good)).isInstanceOf(Exception.class); + assertThat(h.audit.query(FileGatewayTables.CONNECTION_LOG, new QQueryFilter())).anySatisfy(r -> assertThat(r.getValueString("outcome")).isEqualTo("LOCKED_OUT")); + clock.advance(Duration.ofMinutes(31)); + try(var session = h.login("alice", good)) + { + assertThat(session.isAuthenticated()).isTrue(); + } + } + } + + + + /******************************************************************************* + ** Host secrets are parsed from memory and missing secrets fail startup. + *******************************************************************************/ + @Test + void inMemoryHostKeys() throws Exception + { + var config = new FileGatewayQBitConfig().withHostKeyEnvVarName("TEST_HOST_KEY"); + assertThatThrownBy(() -> FileGatewayRuntime.hostKey(config, name -> null)).isInstanceOf(IllegalStateException.class); + assertThat(FileGatewayRuntime.hostKey(config.withAllowGeneratedHostKey(true), name -> null)).isNotNull(); + var pair = KeyUtils.generateKeyPair("ssh-ed25519", 256); + var output = new ByteArrayOutputStream(); + OpenSSHKeyPairResourceWriter.INSTANCE.writePrivateKey(pair, "test", null, output); + String pem = output.toString(StandardCharsets.UTF_8); + var parsed = FileGatewayRuntime.hostKey(config, name -> name.equals("TEST_HOST_KEY") ? pem : null); + assertThat(KeyUtils.compareKeyPairs(pair, parsed)).isTrue(); + assertThatThrownBy(() -> FileGatewayRuntime.hostKey(config, name -> "invalid")).isInstanceOf(Exception.class); + try(var h = new FileGatewayTestServer()) + { + h.runtime.stop(); + h.config.withHostKeyEnvVarName("TEST_HOST_KEY"); + try(var restarted = FileGatewayRuntime.start(h.instance, h.http, name -> name.equals("TEST_HOST_KEY") ? pem : "test-pod")) + { + assertThat(restarted.getProtocol().getPort()).isPositive(); + assertThat(restarted.getProtocol().activeSessions()).isZero(); + } + } + } + + + + /******************************************************************************* + ** NIO paths never escape to native paths; phase-a mutations are always denied. + *******************************************************************************/ + @Test + void virtualNioContract() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + var pair = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(h.account("alice", "A"), pair); + h.saved("savedView", "alpha", "Orders"); + try(var session = h.login("alice", pair)) + { + var state = state(h); + try(var fs = (GatewayFileSystem) h.runtime.getProtocol().getServer().getFileSystemFactory().createFileSystem(state.ssh)) + { + var path = fs.getPath("/testapp/alpha/views/orders.csv"); + assertThat(path.getFileName().toString()).isEqualTo("orders.csv"); + assertThat(path.getNameCount()).isEqualTo(4); + assertThat(path.getName(1).toString()).isEqualTo("alpha"); + assertThat(path.subpath(1, 3).toString()).isEqualTo("alpha/views"); + assertThat(path.startsWith("/testapp")).isTrue(); + assertThat(path.endsWith("views/orders.csv")).isTrue(); + assertThat(fs.getPath("/testapp").relativize(path).toString()).isEqualTo("alpha/views/orders.csv"); + assertThat(fs.getPath("../../testapp").toAbsolutePath().normalize().toString()).isEqualTo("/testapp"); + assertThat(fs.getPath("/testapp", "alpha").resolve("views/orders.csv")).isEqualTo(path); + assertThat(path.toRealPath()).isEqualTo(path); + assertThat(path.hashCode()).isEqualTo(fs.getPath(path.toString()).hashCode()); + assertThat(path.toUri().getScheme()).isEqualTo("gateway"); + assertThat(path.iterator()).toIterable().hasSize(4); + assertThatThrownBy(path::toFile).isInstanceOf(UnsupportedOperationException.class); + assertThat(Files.readAttributes(path, BasicFileAttributes.class).isRegularFile()).isTrue(); + assertThat(Files.readAttributes(fs.getPath("/"), BasicFileAttributes.class).isDirectory()).isTrue(); + assertThatThrownBy(() -> Files.setAttribute(path, "basic:lastModifiedTime", java.nio.file.attribute.FileTime.from(Instant.EPOCH))).isInstanceOf(AccessDeniedException.class); + assertThatThrownBy(() -> Files.createSymbolicLink(fs.getPath("/link"), path)).isInstanceOf(AccessDeniedException.class); + assertThatThrownBy(() -> Files.createLink(fs.getPath("/link"), path)).isInstanceOf(AccessDeniedException.class); + assertThatThrownBy(() -> Files.copy(path, fs.getPath("/copy"))).isInstanceOf(AccessDeniedException.class); + assertThat(fs.getFileStores()).isEmpty(); + assertThat(fs.getRootDirectories()).containsExactly(fs.getPath("/")); + assertThat(fs.isReadOnly()).isFalse(); + assertThat(fs.getSeparator()).isEqualTo("/"); + assertThatThrownBy(fs::newWatchService).isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(fs::getUserPrincipalLookupService).isInstanceOf(UnsupportedOperationException.class); + assertThatThrownBy(() -> fs.getPathMatcher("glob:*")).isInstanceOf(UnsupportedOperationException.class); + } + } + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/GeneratedReadChannelTest.java b/src/test/java/com/kingsrook/qbits/filegateway/GeneratedReadChannelTest.java new file mode 100644 index 0000000..97ccf01 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/GeneratedReadChannelTest.java @@ -0,0 +1,199 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.ByteBuffer; +import java.nio.channels.ClosedChannelException; +import java.nio.channels.NonWritableChannelException; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; +import com.kingsrook.qbits.filegateway.storage.GeneratedReadChannel; +import org.junit.jupiter.api.Test; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Fixed-window stream semantics independently of network packet boundaries and file size. + *******************************************************************************/ +class GeneratedReadChannelTest +{ + /******************************************************************************* + ** Capture charges and finalization without involving a database. + *******************************************************************************/ + private GeneratedReadChannel.Audit audit(AtomicInteger bytes, AtomicReference failure, AtomicInteger closes) + { + return (new GeneratedReadChannel.Audit() + { + /******************************************************************************* + ** Track each newly generated byte once, including forward read-ahead. + *******************************************************************************/ + public void charge(int count) + { + bytes.addAndGet(count); + } + + /******************************************************************************* + ** Capture final outcome. + *******************************************************************************/ + public void finish(IOException error) + { + failure.set(error); + closes.incrementAndGet(); + } + }); + } + + + /******************************************************************************* + ** Forward gaps and backward reads inside the window reuse bytes without regeneration. + *******************************************************************************/ + @Test + void reorderedReadsAndEviction() throws Exception + { + byte[] source = new byte[80]; + for(int i = 0; i < source.length; i++) + { + source[i] = (byte) i; + } + AtomicInteger bytes = new AtomicInteger(); + AtomicInteger closes = new AtomicInteger(); + AtomicReference failure = new AtomicReference<>(); + try(var channel = new GeneratedReadChannel(new ByteArrayInputStream(source), 16, audit(bytes, failure, closes))) + { + assertThat(channel.size()).isZero(); + channel.position(8); + ByteBuffer data = ByteBuffer.allocate(4); + assertThat(channel.read(data)).isEqualTo(4); + assertThat(data.array()).containsExactly(8, 9, 10, 11); + channel.position(0); + data = ByteBuffer.allocate(8); + assertThat(channel.read(data)).isEqualTo(8); + assertThat(data.array()).containsExactly(0, 1, 2, 3, 4, 5, 6, 7); + assertThat(bytes.get()).isEqualTo(12); + channel.position(12); + assertThat(channel.read(ByteBuffer.allocate(20))).isEqualTo(20); + assertThat(channel.position()).isEqualTo(32); + channel.position(16); + assertThat(channel.read(ByteBuffer.allocate(4))).isEqualTo(4); + channel.position(15); + assertThatThrownBy(() -> channel.read(ByteBuffer.allocate(1))).isInstanceOf(IOException.class).hasMessageContaining("download sequentially"); + assertThatThrownBy(() -> channel.read(ByteBuffer.allocate(1))).isInstanceOf(IOException.class); + } + assertThat(closes.get()).isEqualTo(1); + assertThat(failure.get()).hasMessageContaining("download sequentially"); + assertThat(bytes.get()).isEqualTo(32); + } + + + /******************************************************************************* + ** Tiny upstream packets cannot falsely signal EOF; only one fixed window is retained. + *******************************************************************************/ + @Test + void fullReadsEofAndLifecycle() throws Exception + { + AtomicInteger closes = new AtomicInteger(); + AtomicInteger streamCloses = new AtomicInteger(); + AtomicInteger bytes = new AtomicInteger(); + AtomicReference failure = new AtomicReference<>(); + InputStream input = new InputStream() + { + private int remaining = 100003; + + /******************************************************************************* + ** Supply bytes one at a time to simulate fragmented HTTP reads. + *******************************************************************************/ + public int read() + { + return (remaining-- > 0 ? 7 : -1); + } + + /******************************************************************************* + ** Verify stream ownership. + *******************************************************************************/ + public void close() + { + streamCloses.incrementAndGet(); + } + }; + var channel = new GeneratedReadChannel(input, 17, audit(bytes, failure, closes)); + ByteBuffer buffer = ByteBuffer.allocate(1024); + long total = 0; + int count; + while((count = channel.read(buffer.clear())) >= 0) + { + total += count; + assertThat(count).isEqualTo(total == 100003 ? 675 : 1024); + } + assertThat(total).isEqualTo(100003); + assertThat(bytes.get()).isEqualTo(100003); + assertThat(channel.read(ByteBuffer.allocate(0))).isZero(); + assertThatThrownBy(() -> channel.position(-1)).isInstanceOf(IllegalArgumentException.class); + assertThatThrownBy(() -> channel.write(buffer)).isInstanceOf(NonWritableChannelException.class); + assertThatThrownBy(() -> channel.truncate(0)).isInstanceOf(NonWritableChannelException.class); + channel.close(); + channel.close(); + assertThat(channel.isOpen()).isFalse(); + assertThat(closes.get()).isEqualTo(1); + assertThat(streamCloses.get()).isEqualTo(1); + assertThat(failure.get()).isNull(); + assertThatThrownBy(channel::position).isInstanceOf(ClosedChannelException.class); + assertThatThrownBy(channel::size).isInstanceOf(ClosedChannelException.class); + assertThatThrownBy(() -> channel.read(buffer)).isInstanceOf(ClosedChannelException.class); + } + + + /******************************************************************************* + ** Invalid windows, excessive forward seeks and underlying read failures are explicit. + *******************************************************************************/ + @Test + void failuresCloseAndAudit() throws Exception + { + AtomicInteger closes = new AtomicInteger(); + AtomicReference failure = new AtomicReference<>(); + var audit = audit(new AtomicInteger(), failure, closes); + assertThatThrownBy(() -> new GeneratedReadChannel(InputStream.nullInputStream(), 0, audit)).isInstanceOf(IllegalArgumentException.class); + try(var channel = new GeneratedReadChannel(InputStream.nullInputStream(), 8, audit)) + { + channel.position(8); + assertThatThrownBy(() -> channel.read(ByteBuffer.allocate(1))).hasMessageContaining("sequentially"); + } + try(var channel = new GeneratedReadChannel(new InputStream() + { + /******************************************************************************* + ** Simulate an HTTP transport failure. + *******************************************************************************/ + public int read() throws IOException + { + throw (new IOException("HTTP connection interrupted")); + } + }, 8, audit)) + { + assertThatThrownBy(() -> channel.read(ByteBuffer.allocate(1))).hasMessageContaining("HTTP connection interrupted"); + } + assertThat(failure.get()).hasMessageContaining("HTTP connection interrupted"); + assertThat(closes.get()).isEqualTo(2); + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/MaintenanceIntegrationTest.java b/src/test/java/com/kingsrook/qbits/filegateway/MaintenanceIntegrationTest.java new file mode 100644 index 0000000..4570961 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/MaintenanceIntegrationTest.java @@ -0,0 +1,157 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.List; +import java.util.Set; +import com.kingsrook.qbits.filegateway.maintenance.GatewayMaintenance; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayProcesses; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qbits.filegateway.upload.UploadJobs; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.session.QSession; +import org.apache.sshd.common.config.keys.KeyUtils; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/******************************************************************************* + ** Retention and administrative effects operate on durable shared database state. + *******************************************************************************/ +@ExtendWith(NoDiskExtension.class) +class MaintenanceIntegrationTest +{ + @Test + void everyRetentionRuleAndFreshRowsSurvive() throws Exception + { + Instant now = Instant.parse("2026-10-13T12:00:00Z"); + try(var h = new FileGatewayTestServer(Clock.fixed(now, ZoneOffset.UTC))) + { + var account = h.account("alice", "A"); + for(String status : List.of("SUCCEEDED", "RECEIVING", "RUNNING", "QUEUED")) + { + String key = "retention/" + status; + h.store.put(key, new java.io.ByteArrayInputStream(new byte[]{1})); + h.store.put(key + "-results", new java.io.ByteArrayInputStream(new byte[]{2})); + h.store.put(key + "-plain", new java.io.ByteArrayInputStream(new byte[]{3})); + var job = h.audit.insert(FileGatewayTables.UPLOAD_JOB, new QRecord().withValue("accountId", account.getValue("id")) + .withValue("status", status).withValue("fileName", status) + .withValue("storageKey", key).withValue("resultsStorageKey", key + "-results").withValue("plaintextStorageKey", key + "-plain")); + backdate(h, FileGatewayTables.UPLOAD_JOB, job, now.minusSeconds(100 * 86400L)); + } + for(String table : List.of(FileGatewayTables.TRANSFER_LOG, FileGatewayTables.CONNECTION_LOG)) + { + backdate(h, table, h.audit.insert(table, new QRecord().withValue("lastSeenAt", now.minusSeconds(181 * 86400L))), now.minusSeconds(181 * 86400L)); + backdate(h, table, h.audit.insert(table, new QRecord().withValue("lastSeenAt", now)), now); + } + h.audit.insert(FileGatewayTables.SERVER, new QRecord().withValue("podName", "dead-pod").withValue("lastSeenAt", now.minusSeconds(86401))); + var alias = PhaseEIntegrationTest.alias(h, "out", "/out", "OUTBOX", account); + h.store.put("outbox/old", new java.io.ByteArrayInputStream(new byte[]{4})); + var out = h.audit.insert(FileGatewayTables.OUTBOX_FILE, new QRecord().withValue("aliasId", alias.getValue("id")).withValue("status", "READY") + .withValue("storageKey", "outbox/old")); + backdate(h, FileGatewayTables.OUTBOX_FILE, out, now.minusSeconds(31 * 86400L)); + GatewayMaintenance.maintain(h.instance, h.config, h.store, Clock.fixed(now, ZoneOffset.UTC)); + var jobs = UploadIntegrationTest.jobs(h); + assertThat(jobs).anySatisfy(row -> + { + assertThat(row.getValueString("status")).isEqualTo("ABORTED"); + assertThat(row.getValue("storageKey")).isNull(); + }); + assertThat(h.store.objects).doesNotContainKeys("retention/SUCCEEDED", "retention/SUCCEEDED-results", "retention/SUCCEEDED-plain", "retention/RECEIVING", "outbox/old") + .containsKeys("retention/RUNNING", "retention/QUEUED"); + assertThat(h.audit.query(FileGatewayTables.OUTBOX_FILE, UploadJobs.equal("id", out.getValue("id"))).getFirst().getValueString("status")).isEqualTo("EXPIRED"); + assertThat(h.audit.query(FileGatewayTables.TRANSFER_LOG, new QQueryFilter())).hasSize(1); + assertThat(h.audit.query(FileGatewayTables.CONNECTION_LOG, new QQueryFilter())).hasSize(1); + assertThat(h.audit.query(FileGatewayTables.SERVER, new QQueryFilter())).hasSize(1); + GatewayMaintenance.maintain(h.instance, h.config, h.store, Clock.fixed(now, ZoneOffset.UTC)); + assertThat(GatewayMaintenance.metadata(h.config.withSchedulerName("scheduler")).getSchedule().getRepeatSeconds()).isEqualTo(86400); + } + } + + /******************************************************************************* + ** Inserts stamp createDate with the real time; move a row's createDate after. + *******************************************************************************/ + private static void backdate(FileGatewayTestServer h, String table, QRecord row, Instant createDate) throws Exception + { + h.audit.update(table, new QRecord().withValue("id", row.getValue("id")).withValue("createDate", createDate)); + assertThat(h.audit.query(table, UploadJobs.equal("id", row.getValue("id"))).getFirst().getValueInstant("createDate")).isEqualTo(createDate); + } + + + + @Test + void reviewResultAndLiveRevocation() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + var account = h.account("alice", "A"); + var keyPair = KeyUtils.generateKeyPair("ssh-ed25519", 256); + var key = h.key(account, keyPair); + QContext.init(h.instance, new QSession().withPermissions(Set.of("file_gateway.admin"))); + h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")).withValue("lockedUntil", Instant.now().plusSeconds(600)).withValue("failedLoginCount", 5)); + apply(h, FileGatewayProcesses.UNLOCK, account); + var unlocked = h.audit.query(FileGatewayTables.ACCOUNT, UploadJobs.equal("id", account.getValue("id"))).getFirst(); + assertThat(unlocked.getValue("lockedUntil")).isNull(); + assertThat(unlocked.getValueInteger("failedLoginCount")).isZero(); + try(var ssh = h.login("alice", keyPair)) + { + var row = h.audit.query(FileGatewayTables.CONNECTION_LOG, UploadJobs.equal("accountId", account.getValue("id"))).getFirst(); + apply(h, FileGatewayProcesses.END_SESSION, row); + assertThat(h.audit.query(FileGatewayTables.CONNECTION_LOG, UploadJobs.equal("id", row.getValue("id"))).getFirst().getValue("disconnectedAt")).isNull(); + h.runtime.getProtocol().getAuthenticator().poll(); + ssh.waitFor(java.util.EnumSet.of(org.apache.sshd.client.session.ClientSession.ClientSessionEvent.CLOSED), java.time.Duration.ofSeconds(5)); + assertThat(ssh.isClosed()).isTrue(); + } + try(var ssh = h.login("alice", keyPair)) + { + apply(h, FileGatewayProcesses.DISABLE, account); + h.runtime.getProtocol().getAuthenticator().poll(); + ssh.waitFor(java.util.EnumSet.of(org.apache.sshd.client.session.ClientSession.ClientSessionEvent.CLOSED), java.time.Duration.ofSeconds(5)); + assertThat(ssh.isClosed()).isTrue(); + } + h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")).withValue("status", "ACTIVE")); + apply(h, FileGatewayProcesses.REVOKE, key); + assertThatThrownBy(() -> h.login("alice", keyPair)).isInstanceOf(Exception.class); + } + } + + private void apply(FileGatewayTestServer h, String action, QRecord row) throws Exception + { + QContext.init(h.instance, new QSession().withPermissions(Set.of("file_gateway.admin"))); + var input = new RunBackendStepInput().withProcessName(action).withRecords(List.of(row)); + var review = new FileGatewayProcesses.Review(); + var preview = new RunBackendStepOutput(); + review.runOnePage(input, preview); + assertThat(review.getProcessSummary(preview, false)).hasSize(1); + assertThat(review.getProcessSummary(preview, true)).hasSize(1); + var result = new RunBackendStepOutput(); + new FileGatewayProcesses.Apply().runOnePage(input.withRecords(preview.getRecords()), result); + assertThat(result.getRecords()).hasSize(1); + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/MemoryGatewayObjectStore.java b/src/test/java/com/kingsrook/qbits/filegateway/MemoryGatewayObjectStore.java new file mode 100644 index 0000000..df16768 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/MemoryGatewayObjectStore.java @@ -0,0 +1,95 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.security.MessageDigest; +import java.util.Arrays; +import java.util.HexFormat; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import com.kingsrook.qbits.filegateway.storage.GatewayObjectStore; + +/******************************************************************************* + ** Test-only object storage. + *******************************************************************************/ +public class MemoryGatewayObjectStore implements GatewayObjectStore +{ + public final Map objects = new ConcurrentHashMap<>(); + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public Receipt put(String key, InputStream content) throws IOException + { + try + { + byte[] bytes = content.readAllBytes(); + objects.put(key, bytes); + return (new Receipt(bytes.length, HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(bytes)))); + } + catch(java.security.NoSuchAlgorithmException e) + { + throw new IOException(e); + } + } + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public InputStream open(String key) throws IOException + { + byte[] bytes = objects.get(key); + if(bytes == null) + { + throw (new IOException("Object missing")); + } + return (new ByteArrayInputStream(bytes)); + } + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public InputStream openRange(String key, long start, long end) throws IOException + { + try(var in = open(key)) + { + in.skipNBytes(start); + return (new ByteArrayInputStream(in.readNBytes((int) (end - start + 1)))); + } + } + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public void copy(String source, String destination) throws IOException + { + try(var in = open(source)) + { + objects.put(destination, in.readAllBytes()); + } + } + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public void delete(String key) + { + objects.remove(key); + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/MetadataBehaviorTest.java b/src/test/java/com/kingsrook/qbits/filegateway/MetadataBehaviorTest.java new file mode 100644 index 0000000..1bdacd3 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/MetadataBehaviorTest.java @@ -0,0 +1,173 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + +import java.time.Instant; +import java.util.List; +import java.util.Map; +import java.util.Set; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayProcesses; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qbits.filegateway.permissions.FileGatewayPermissionChecker; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.exceptions.QException; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput; +import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.session.QSession; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiCredentials; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInProcessSessions; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInvoker; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiRequest; +import com.kingsrook.qqq.openapi.model.HttpMethod; +import org.apache.sshd.common.config.keys.KeyUtils; +import org.apache.sshd.common.config.keys.PublicKeyEntry; +import org.apache.sshd.common.digest.BuiltinDigests; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/******************************************************************************* + ** Real database customizers and strict UI API dependent pickers. + *******************************************************************************/ +@ExtendWith(NoDiskExtension.class) +class MetadataBehaviorTest +{ + /******************************************************************************* + ** Every contingent picker excludes records for the other table via the UI API. + *******************************************************************************/ + @ParameterizedTest + @ValueSource(strings = {"savedBulkLoadProfile", "savedView", "savedReport"}) + void contingentPickerBehavior(String source) throws Exception + { + try(var h = new FileGatewayTestServer()) + { + var own = h.saved(source, "alpha", "Alpha choice"); + var other = h.saved(source, "beta", "Beta choice"); + assertThat(own.getErrors()).isEmpty(); + assertThat(other.getErrors()).isEmpty(); + var session = new QSession().withPermissions(Set.of("file_gateway.admin")); + String token = UiApiInProcessSessions.register(session, java.time.Duration.ofMinutes(1)); + try + { + var response = UiApiInvoker.forServer(h.http, "next-ui").invoke(new UiApiRequest().withMethod(HttpMethod.POST) + .withPath("/table/{tableName}/possibleValues/{fieldName}").withPathParam("tableName", FileGatewayTables.ALIAS) + .withPathParam("fieldName", source + "Id").withCredentials(UiApiCredentials.ofInProcessSession(token)) + .withJsonBody(Map.of("values", Map.of("tableName", "alpha")))); + assertThat(response.getStatus()).as(response.getJson().toString()).isEqualTo(200); + assertThat(response.getJson().toString()).contains("Alpha choice").doesNotContain("Beta choice"); + } + finally + { + UiApiInProcessSessions.revoke(token); + } + } + } + + + + /******************************************************************************* + ** Validation runs on insert and partial update, computes metadata, and enforces uniqueness. + *******************************************************************************/ + @Test + void publicKeyValidation() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + var account = h.account("alice", "A"); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + var stored = h.key(account, key); + assertThat(stored.getValueString("fingerprint")).isEqualTo(KeyUtils.getFingerPrint(BuiltinDigests.sha256, key.getPublic())); + assertThat(stored.getValueString("keyType")).isEqualTo("ssh-ed25519"); + assertThat(stored.getValueInteger("keyBits")).isEqualTo(256); + assertThatThrownBy(() -> h.key(account, key)).isInstanceOf(QException.class); + for(String type : List.of("ssh-rsa", "ssh-dss")) + { + var weak = KeyUtils.generateKeyPair(type, type.equals("ssh-rsa") ? 2048 : 1024); + assertThatThrownBy(() -> h.key(account, weak)).isInstanceOf(QException.class); + } + assertThat(h.key(account, KeyUtils.generateKeyPair("ssh-rsa", 3072)).getValueInteger("keyBits")).isEqualTo(3072); + assertThat(h.key(account, KeyUtils.generateKeyPair("ecdsa-sha2-nistp384", 384)).getValueInteger("keyBits")).isEqualTo(384); + for(String text : List.of("not a key", "ssh-unknown invalid", "")) + { + assertThatThrownBy(() -> h.audit.insert(FileGatewayTables.SSH_KEY, new QRecord().withValue("accountId", account.getValue("id")).withValue("publicKey", text))).isInstanceOf(QException.class); + } + h.audit.update(FileGatewayTables.SSH_KEY, new QRecord().withValue("id", stored.getValue("id")).withValue("label", "Renamed").withValue("fingerprint", "forged")); + var read = h.audit.query(FileGatewayTables.SSH_KEY, new QQueryFilter()).getFirst(); + assertThat(read.getValueString("fingerprint")).isEqualTo(stored.getValueString("fingerprint")); + var weak = KeyUtils.generateKeyPair("ssh-rsa", 2048); + assertThatThrownBy(() -> h.audit.update(FileGatewayTables.SSH_KEY, new QRecord().withValue("id", stored.getValue("id")).withValue("publicKey", PublicKeyEntry.toString(weak.getPublic())))).isInstanceOf(QException.class); + assertThatThrownBy(() -> h.account("Bad name", "A")).isInstanceOf(QException.class); + assertThatThrownBy(() -> h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")).withValue("maxConcurrentSessions", 0))).isInstanceOf(QException.class); + } + } + + + + /******************************************************************************* + ** People can read logs but cannot write them, including administrators. + *******************************************************************************/ + @Test + void permissionsAndAdministrativeActions() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + var account = h.account("alice", "A"); + var key = h.key(account, KeyUtils.generateKeyPair("ssh-ed25519", 256)); + var checker = new FileGatewayPermissionChecker(); + QContext.init(h.instance, new QSession().withPermissions(Set.of("file_gateway.view"))); + checker.checkPermissionsThrowing(new QueryInput(FileGatewayTables.ACCOUNT), h.instance.getTable(FileGatewayTables.ACCOUNT)); + assertThatThrownBy(() -> checker.checkPermissionsThrowing(new UpdateInput(FileGatewayTables.ACCOUNT), h.instance.getTable(FileGatewayTables.ACCOUNT))).isInstanceOf(QException.class); + QContext.init(h.instance, new QSession().withPermissions(Set.of("file_gateway.admin"))); + checker.checkPermissionsThrowing(new InsertInput(FileGatewayTables.ACCOUNT), h.instance.getTable(FileGatewayTables.ACCOUNT)); + for(String table : FileGatewayTables.SYSTEM_WRITTEN) + { + assertThatThrownBy(() -> checker.checkPermissionsThrowing(new UpdateInput(table), h.instance.getTable(table))).isInstanceOf(QException.class); + } + h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")).withValue("lockedUntil", Instant.now().plusSeconds(900)).withValue("failedLoginCount", 5)); + var input = new RunBackendStepInput().withProcessName(FileGatewayProcesses.UNLOCK).withRecords(List.of(account)); + var review = new FileGatewayProcesses.Review(); + var preview = new RunBackendStepOutput(); + review.runOnePage(input, preview); + assertThat(review.getProcessSummary(preview, false)).hasSize(1); + assertThat(h.audit.query(FileGatewayTables.ACCOUNT, new QQueryFilter()).getFirst().getValue("lockedUntil")).isNotNull(); + new FileGatewayProcesses.Apply().runOnePage(input, new RunBackendStepOutput()); + var unlocked = h.audit.query(FileGatewayTables.ACCOUNT, new QQueryFilter()).getFirst(); + assertThat(unlocked.getValue("lockedUntil")).isNull(); + assertThat(unlocked.getValueInteger("failedLoginCount")).isZero(); + new FileGatewayProcesses.Apply().runOnePage(input.withProcessName(FileGatewayProcesses.DISABLE), new RunBackendStepOutput()); + assertThat(h.audit.query(FileGatewayTables.ACCOUNT, new QQueryFilter()).getFirst().getValueString("status")).isEqualTo("DISABLED"); + new FileGatewayProcesses.Apply().runOnePage(input.withProcessName(FileGatewayProcesses.REVOKE).withRecords(List.of(key)), new RunBackendStepOutput()); + assertThat(h.audit.query(FileGatewayTables.SSH_KEY, new QQueryFilter()).getFirst().getValue("revokedAt")).isNotNull(); + QContext.init(h.instance, new QSession()); + assertThatThrownBy(() -> new FileGatewayProcesses.Apply().runOnePage(input, new RunBackendStepOutput())).isInstanceOf(QException.class); + assertThatThrownBy(() -> checker.checkPermissionsThrowing(new QueryInput(FileGatewayTables.ACCOUNT), h.instance.getTable(FileGatewayTables.ACCOUNT))).isInstanceOf(QException.class); + assertThatThrownBy(() -> h.audit.query("alpha", new QQueryFilter())).isInstanceOf(IllegalArgumentException.class); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/NoDiskExtension.java b/src/test/java/com/kingsrook/qbits/filegateway/NoDiskExtension.java new file mode 100644 index 0000000..de13622 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/NoDiskExtension.java @@ -0,0 +1,90 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + + +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.attribute.FileTime; +import java.util.Map; +import java.util.TreeMap; +import org.junit.jupiter.api.extension.AfterEachCallback; +import org.junit.jupiter.api.extension.BeforeEachCallback; +import org.junit.jupiter.api.extension.ExtensionContext; +import static org.assertj.core.api.Assertions.assertThat; + + +/******************************************************************************* + ** Snapshots real disk boundaries around every SFTP test; target build products are excluded. + *******************************************************************************/ +public class NoDiskExtension implements BeforeEachCallback, AfterEachCallback +{ + private Map before; + + /******************************************************************************* + ** Before each. + *******************************************************************************/ + @Override + public void beforeEach(ExtensionContext context) throws Exception + { + Files.createDirectories(Path.of(System.getProperty("java.io.tmpdir"))); + Files.createDirectories(Path.of(System.getProperty("user.home"))); + before = snapshot(); + } + + + + /******************************************************************************* + ** After each. + *******************************************************************************/ + @Override + public void afterEach(ExtensionContext context) throws Exception + { + Map after = snapshot(); + assertThat(after.keySet()).as("Files created or removed outside the temporary area").isEqualTo(before.keySet()); + after.forEach((path, time) -> assertThat(time).as("Modified: %s", path).isEqualTo(before.get(path))); + try(var files = Files.list(Path.of(System.getProperty("java.io.tmpdir")))) + { + assertThat(files.toList()).isEmpty(); + } + } + + private Map snapshot() throws Exception + { + Map result = new TreeMap<>(); + Path working = Path.of("").toAbsolutePath(); + try(var files = Files.walk(working)) + { + for(Path file : files.filter(p -> !p.startsWith(working.resolve("target"))).toList()) + { + result.put(file.toString(), Files.getLastModifiedTime(file)); + } + } + try(var files = Files.list(Path.of(System.getProperty("user.home")))) + { + for(Path file : files.toList()) + { + result.put(file.toString(), Files.getLastModifiedTime(file)); + } + } + return (result); + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/PgpIntegrationTest.java b/src/test/java/com/kingsrook/qbits/filegateway/PgpIntegrationTest.java new file mode 100644 index 0000000..8af0647 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/PgpIntegrationTest.java @@ -0,0 +1,196 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + +import java.io.ByteArrayInputStream; +import java.io.ByteArrayOutputStream; +import java.nio.charset.StandardCharsets; +import java.security.KeyPairGenerator; +import java.util.Date; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qbits.filegateway.pgp.PgpStreams; +import com.kingsrook.qbits.filegateway.upload.UploadJobs; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import org.apache.sshd.common.config.keys.KeyUtils; +import org.apache.sshd.sftp.client.SftpClientFactory; +import org.bouncycastle.bcpg.ArmoredOutputStream; +import org.bouncycastle.bcpg.HashAlgorithmTags; +import org.bouncycastle.openpgp.PGPKeyRingGenerator; +import org.bouncycastle.openpgp.PGPPublicKey; +import org.bouncycastle.openpgp.PGPSignature; +import org.bouncycastle.openpgp.operator.bc.BcPGPContentSignerBuilder; +import org.bouncycastle.openpgp.operator.bc.BcPGPDigestCalculatorProvider; +import org.bouncycastle.openpgp.operator.jcajce.JcaPGPKeyPair; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/******************************************************************************* + ** Real transfer encryption/decryption with ephemeral test-only OpenPGP keys. + *******************************************************************************/ +@ExtendWith(NoDiskExtension.class) +class PgpIntegrationTest +{ + record Keys(String publicKey, String privateKey) + { + } + + static Keys keys() throws Exception + { + var generator = KeyPairGenerator.getInstance("RSA"); + generator.initialize(2048); + var signing = new JcaPGPKeyPair(PGPPublicKey.RSA_SIGN, generator.generateKeyPair(), new Date()); + var encryption = new JcaPGPKeyPair(PGPPublicKey.RSA_ENCRYPT, generator.generateKeyPair(), new Date()); + var rings = new PGPKeyRingGenerator(PGPSignature.POSITIVE_CERTIFICATION, signing, "ephemeral-test", + new BcPGPDigestCalculatorProvider().get(HashAlgorithmTags.SHA1), null, null, + new BcPGPContentSignerBuilder(PGPPublicKey.RSA_SIGN, HashAlgorithmTags.SHA256), null); + rings.addSubKey(encryption); + ByteArrayOutputStream publicBytes = new ByteArrayOutputStream(); + try(var armor = new ArmoredOutputStream(publicBytes)) + { + rings.generatePublicKeyRing().encode(armor); + } + ByteArrayOutputStream privateBytes = new ByteArrayOutputStream(); + try(var armor = new ArmoredOutputStream(privateBytes)) + { + rings.generateSecretKeyRing().encode(armor); + } + return (new Keys(publicBytes.toString(StandardCharsets.UTF_8), privateBytes.toString(StandardCharsets.UTF_8))); + } + + @Test + void encryptedBulkUploadInvalidCiphertextAndAccountValidation() throws Exception + { + Keys keys = keys(); + try(var h = new FileGatewayTestServer()) + { + UploadIntegrationTest.profile(h); + var account = h.account("alice", "A"); + assertThatThrownBy(() -> h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")) + .withValue("pgpPublicKey", "invalid"))).isInstanceOf(Exception.class); + h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")).withValue("pgpDecryptUploads", true)); + h.config.withPgpPrivateKeyEnvVarName("TEST_PGP").withPgpPassphraseEnvVarName("TEST_PASS"); + h.runtime.getWorker().withEnvironment(name -> "TEST_PGP".equals(name) ? keys.privateKey() : ""); + var sshKey = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(account, sshKey); + try(var ssh = h.login("alice", sshKey); var sftp = SftpClientFactory.instance().createSftpClient(ssh)) + { + byte[] ciphertext; + try(var input = PgpStreams.encrypt(new ByteArrayInputStream(UploadIntegrationTest.CSV), keys.publicKey(), "orders.csv")) + { + ciphertext = input.readAllBytes(); + } + try(var output = sftp.write(UploadIntegrationTest.FOLDER + "orders.csv.pgp")) + { + output.write(ciphertext); + } + h.runtime.getWorker().runOnce(); + var job = UploadIntegrationTest.jobs(h).getFirst(); + assertThat(job.getValueString("status")).isEqualTo("SUCCEEDED"); + assertThat(job.getValueInteger("insertedCount")).isEqualTo(2); + assertThat(job.getValueString("plaintextSha256")).isNotBlank().isNotEqualTo(job.getValueString("sha256")); + assertThat(h.store.objects.get(job.getValueString("plaintextStorageKey"))).isEqualTo(UploadIntegrationTest.CSV); + assertThat(h.store.objects.get(job.getValueString("storageKey"))).isEqualTo(ciphertext); + try(var output = sftp.write(UploadIntegrationTest.FOLDER + "bad.csv.gpg")) + { + output.write(new byte[]{1, 2, 3}); + } + h.runtime.getWorker().runOnce(); + assertThat(UploadIntegrationTest.jobs(h)).anySatisfy(row -> + { + assertThat(row.getValueString("status")).isEqualTo("REJECTED"); + assertThat(row.getValueString("errorMessage")).contains("PGP decryption failed"); + }); + } + } + } + + @Test + void encryptedGeneratedAndOutboxDownloadsMatchPlaintext() throws Exception + { + Keys keys = keys(); + try(var h = new FileGatewayTestServer()) + { + h.seedRows(2); + h.saved("savedView", "alpha", "Daily"); + var account = h.account("alice", "A"); + var sshKey = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(account, sshKey); + byte[] plain; + try(var ssh = h.login("alice", sshKey); var sftp = SftpClientFactory.instance().createSftpClient(ssh)) + { + plain = PhaseEIntegrationTest.read(sftp, "/testapp/alpha/views/daily.csv"); + } + h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")) + .withValue("pgpPublicKey", keys.publicKey()).withValue("pgpEncryptDownloads", true)); + var alias = PhaseEIntegrationTest.alias(h, "out", "/out", "OUTBOX", account); + h.audit.update(FileGatewayTables.ALIAS, new QRecord().withValue("id", alias.getValue("id")).withValue("consumerName", null)); + com.kingsrook.qbits.filegateway.api.FileGatewayOutbox.put("out", "reply.csv", new ByteArrayInputStream(plain), plain.length); + try(var ssh = h.login("alice", sshKey); var sftp = SftpClientFactory.instance().createSftpClient(ssh)) + { + assertThat(PhaseEIntegrationTest.names(sftp, "/testapp/alpha/views")).contains("daily.csv.pgp"); + for(String path : java.util.List.of("/testapp/alpha/views/daily.csv.pgp", "/out/reply.csv.pgp")) + { + byte[] encrypted = PhaseEIntegrationTest.read(sftp, path); + try(var decrypted = PgpStreams.decrypt(new ByteArrayInputStream(encrypted), keys.privateKey(), new char[0], 1_000_000)) + { + assertThat(decrypted.readAllBytes()).isEqualTo(plain); + } + } + assertThat(h.audit.query(FileGatewayTables.OUTBOX_FILE, new com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter()).getFirst().getValueString("status")) + .isEqualTo("DOWNLOADED"); + } + } + } + + @Test + void integrityAndPlaintextSizeAreEnforced() throws Exception + { + Keys keys = keys(); + byte[] bytes = new byte[160000]; + new java.util.Random(42).nextBytes(bytes); + byte[] encrypted; + try(var input = PgpStreams.encrypt(new ByteArrayInputStream(bytes), keys.publicKey(), "file.bin")) + { + encrypted = input.readAllBytes(); + } + try(var input = PgpStreams.decrypt(new ByteArrayInputStream(encrypted), keys.privateKey(), new char[0], bytes.length)) + { + assertThat(input.readAllBytes()).isEqualTo(bytes); + } + assertThatThrownBy(() -> + { + try(var input = PgpStreams.decrypt(new ByteArrayInputStream(encrypted), keys.privateKey(), new char[0], 10)) + { + input.readAllBytes(); + } + }).isInstanceOf(java.io.IOException.class); + encrypted[encrypted.length - 1] ^= 1; + assertThatThrownBy(() -> + { + try(var input = PgpStreams.decrypt(new ByteArrayInputStream(encrypted), keys.privateKey(), new char[0], bytes.length)) + { + input.readAllBytes(); + } + }).isInstanceOf(java.io.IOException.class); + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/PhaseEIntegrationTest.java b/src/test/java/com/kingsrook/qbits/filegateway/PhaseEIntegrationTest.java new file mode 100644 index 0000000..61c2483 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/PhaseEIntegrationTest.java @@ -0,0 +1,288 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + +import java.io.ByteArrayInputStream; +import java.nio.charset.StandardCharsets; +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.List; +import java.util.Set; +import com.kingsrook.qbits.filegateway.api.FileAcceptance; +import com.kingsrook.qbits.filegateway.api.FileGatewayConsumer; +import com.kingsrook.qbits.filegateway.api.FileGatewayConsumersMetaData; +import com.kingsrook.qbits.filegateway.api.FileGatewayOutbox; +import com.kingsrook.qbits.filegateway.api.InboundFile; +import com.kingsrook.qbits.filegateway.api.OutboxFileRef; +import com.kingsrook.qbits.filegateway.maintenance.GatewayMaintenance; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayHome; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables; +import com.kingsrook.qbits.filegateway.outbox.OutboxService; +import com.kingsrook.qbits.filegateway.upload.UploadJobs; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.dashboard.widgets.StatisticsData; +import com.kingsrook.qqq.backend.core.model.dashboard.widgets.TableData; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.session.QSession; +import org.apache.sshd.common.config.keys.KeyUtils; +import org.apache.sshd.sftp.client.SftpClient; +import org.apache.sshd.sftp.client.SftpClientFactory; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +/******************************************************************************* + ** Phase E acceptance uses real MINA sessions, strict UI API and the adopter DDL. + *******************************************************************************/ +@ExtendWith(NoDiskExtension.class) +class PhaseEIntegrationTest +{ + static InboundFile accepted; + static byte[] acceptedBytes; + static int downloaded; + static boolean failCallback; + + public static class Consumer implements FileGatewayConsumer + { + @Override + public String getName() + { + return ("partner"); + } + + @Override + public FileAcceptance accept(InboundFile file) throws Exception + { + accepted = file; + try(var stream = file.openStream()) + { + acceptedBytes = stream.readAllBytes(); + } + return (FileAcceptance.accepted("receipt")); + } + + @Override + public void onDownloaded(OutboxFileRef ref) throws Exception + { + if(failCallback) + { + throw (new Exception("retry test")); + } + downloaded++; + } + } + + static QRecord alias(FileGatewayTestServer h, String name, String path, String kind, QRecord account) throws Exception + { + return (h.audit.insert(FileGatewayTables.ALIAS, new QRecord().withValue("name", name).withValue("path", path).withValue("kind", kind) + .withValue("accountId", account == null ? null : account.getValue("id")).withValue("consumerName", "partner"))); + } + + static List names(SftpClient sftp, String path) throws Exception + { + List names = new ArrayList<>(); + for(var item : sftp.readDir(path)) + { + if(!List.of(".", "..").contains(item.getFilename())) + { + names.add(item.getFilename()); + } + } + return (names); + } + + static byte[] read(SftpClient sftp, String path) throws Exception + { + try(var input = sftp.read(path)) + { + return (input.readAllBytes()); + } + } + + @Test + void endpointsChrootVisibilityBulkAndInbox() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + FileGatewayConsumersMetaData.ofOrWithNew(h.instance).withConsumer(new QCodeReference(Consumer.class)); + var profile = UploadIntegrationTest.profile(h); + var service = h.audit.insert(FileGatewayTables.ACCOUNT, new QRecord().withValue("username", "service").withValue("userId", "A").withValue("accountType", "SERVICE")); + assertThat(service.getValueString("rootMode")).isEqualTo("ENDPOINTS_ONLY"); + var user = h.account("alice", "A"); + assertThat(user.getValueString("rootMode")).isEqualTo("FULL_TREE"); + var denied = h.account("bob", "B"); + var bulk = alias(h, "orders", "/acme/orders", "BULK_LOAD", null); + h.audit.update(FileGatewayTables.ALIAS, new QRecord().withValue("id", bulk.getValue("id")).withValue("tableName", "alpha") + .withValue("savedBulkLoadProfileId", profile.getValue("id"))); + alias(h, "inbox", "/acme/inbox", "INBOX", service); + alias(h, "private", "/private", "INBOX", user); + alias(h, "shared", "/shared", "OUTBOX", null); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(service, key); + var userKey = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(user, userKey); + var deniedKey = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(denied, deniedKey); + try(var ssh = h.login("service", key); var sftp = SftpClientFactory.instance().createSftpClient(ssh); + var aliceSsh = h.login("alice", userKey); var alice = SftpClientFactory.instance().createSftpClient(aliceSsh); + var bobSsh = h.login("bob", deniedKey); var bob = SftpClientFactory.instance().createSftpClient(bobSsh)) + { + assertThat(names(sftp, "/")).containsExactlyInAnyOrder("acme", "shared"); + assertThat(names(alice, "/")).contains("testapp", "acme", "private", "shared"); + assertThat(names(bob, "/")).doesNotContain("acme", "private"); + assertThatThrownBy(() -> sftp.stat("/testapp")).isInstanceOf(Exception.class); + try(var output = sftp.write("/acme/orders/order.csv")) + { + output.write(UploadIntegrationTest.CSV); + } + h.runtime.getWorker().runOnce(); + assertThat(UploadIntegrationTest.jobs(h).getFirst().getValueString("status")).isEqualTo("SUCCEEDED"); + try(var output = sftp.write("/acme/inbox/message.txt")) + { + output.write("hello partner".getBytes(StandardCharsets.UTF_8)); + } + h.runtime.getWorker().runOnce(); + assertThat(accepted.endpointName()).isEqualTo("inbox"); + assertThat(accepted.remoteAddress()).isEqualTo("127.0.0.1"); + assertThat(acceptedBytes).isEqualTo("hello partner".getBytes(StandardCharsets.UTF_8)); + } + } + } + + @Test + void endpointValidationAndView() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + var user = h.account("alice", "A"); + for(String path : List.of("relative", "/../escape", "/UPPER", "/a//b", "/a/./b", "/a/", "/testapp/in")) + { + assertThatThrownBy(() -> alias(h, "invalid", path, "INBOX", user)).isInstanceOf(Exception.class); + } + var mount = alias(h, "good", "/good", "INBOX", user); + assertThatThrownBy(() -> h.audit.update(FileGatewayTables.ALIAS, new QRecord().withValue("id", mount.getValue("id")).withValue("path", "/testapp"))).isInstanceOf(Exception.class); + assertThatThrownBy(() -> alias(h, "duplicate", "/good", "INBOX", user)).isInstanceOf(Exception.class); + h.seedRows(2); + var saved = h.saved("savedView", "alpha", "Daily Rows"); + var view = alias(h, "view", "/download", "VIEW", user); + h.audit.update(FileGatewayTables.ALIAS, new QRecord().withValue("id", view.getValue("id")).withValue("tableName", "alpha") + .withValue("savedViewId", saved.getValue("id")).withValue("format", "csv")); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(user, key); + try(var ssh = h.login("alice", key); var sftp = SftpClientFactory.instance().createSftpClient(ssh)) + { + assertThat(names(sftp, "/download")).containsExactly("daily-rows.csv"); + assertThat(read(sftp, "/download/daily-rows.csv")).isEqualTo(read(sftp, "/testapp/alpha/views/daily-rows.csv")); + } + } + } + + @Test + void outboxLifecycleAndRetry() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + downloaded = 0; + failCallback = false; + FileGatewayConsumersMetaData.ofOrWithNew(h.instance).withConsumer(new QCodeReference(Consumer.class)); + var account = h.account("alice", "A"); + alias(h, "out", "/out", "OUTBOX", account); + alias(h, "in", "/in", "INBOX", account); + assertThatThrownBy(() -> FileGatewayOutbox.put("missing", "x", new ByteArrayInputStream(new byte[0]), 0)).isInstanceOf(Exception.class); + assertThatThrownBy(() -> FileGatewayOutbox.put("in", "x", new ByteArrayInputStream(new byte[0]), 0)).isInstanceOf(Exception.class); + byte[] bytes = new byte[200000]; + new java.util.Random(7).nextBytes(bytes); + var ref = FileGatewayOutbox.put("out", "reply.bin", new ByteArrayInputStream(bytes), bytes.length); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(account, key); + try(var ssh = h.login("alice", key); var sftp = SftpClientFactory.instance().createSftpClient(ssh)) + { + assertThat(names(sftp, "/out")).containsExactly("reply.bin"); + try(var handle = sftp.open("/out/reply.bin")) + { + byte[] partial = new byte[100]; + for(int i = 0; i < 5; i++) + { + sftp.read(handle, 0, partial); + } + } + assertThat(downloaded).isZero(); + assertThat(read(sftp, "/out/reply.bin")).isEqualTo(bytes); + assertThat(downloaded).isEqualTo(1); + var row = h.audit.query(FileGatewayTables.OUTBOX_FILE, UploadJobs.equal("id", ref.outboxFileId())).getFirst(); + assertThat(row.getValueString("status")).isEqualTo("DOWNLOADED"); + assertThat(row.getValue("firstDownloadedAt")).isNotNull(); + assertThat(row.getValueInteger("downloadCount")).isEqualTo(1); + read(sftp, "/out/reply.bin"); + assertThat(downloaded).isEqualTo(1); + sftp.remove("/out/reply.bin"); + assertThat(names(sftp, "/out")).isEmpty(); + assertThat(h.audit.query(FileGatewayTables.OUTBOX_FILE, UploadJobs.equal("id", ref.outboxFileId())).getFirst().getValueString("status")).isEqualTo("REMOVED"); + failCallback = true; + FileGatewayOutbox.put("out", "retry.bin", new ByteArrayInputStream(bytes), bytes.length); + read(sftp, "/out/retry.bin"); + assertThat(downloaded).isEqualTo(1); + failCallback = false; + GatewayMaintenance.maintain(h.instance, h.config, h.store, Clock.systemUTC()); + assertThat(downloaded).isEqualTo(2); + } + } + } + + @Test + void outboxWithoutRunningSftpAndHomeAndIntegrations() throws Exception + { + try(var h = new FileGatewayTestServer(Clock.systemUTC(), instance -> { }, 0, false, false)) + { + assertThat(FileGatewayOutbox.isAvailable()).isTrue(); + FileGatewayOutbox.setProvider(new OutboxService(h.instance, h.store, h.config, Clock.systemUTC())); + var account = h.account("partner", "A"); + alias(h, "out", "/out", "OUTBOX", account); + FileGatewayOutbox.put("out", "file.txt", new ByteArrayInputStream(new byte[]{1}), 1); + Instant now = Instant.now(); + h.seedTransfer(account, "UPLOAD", 2097152L, now); + h.seedTransfer(account, "DOWNLOAD", 1048576L, now); + h.audit.insert(FileGatewayTables.UPLOAD_JOB, new QRecord().withValue("accountId", account.getValue("id")).withValue("fileName", "orders.csv") + .withValue("status", "FAILED").withValue("finishedAt", now).withValue("errorMessage", "Invalid quantity")); + QContext.init(h.instance, new QSession().withPermissions(Set.of("file_gateway.view"))); + FileGatewayHome home = new FileGatewayHome(); + assertThat(((Number) ((StatisticsData) home.data("transfers", now)).getCount()).intValue()).isEqualTo(2); + assertThat(((Number) ((StatisticsData) home.data("in", now)).getCount()).doubleValue()).isEqualTo(2); + assertThat(((Number) ((StatisticsData) home.data("out", now)).getCount()).doubleValue()).isEqualTo(1); + assertThat(((Number) ((StatisticsData) home.data("failures", now)).getCount()).intValue()).isEqualTo(1); + assertThat(((TableData) home.data("recent", now)).getRows()).hasSize(1); + assertThat(((TableData) home.data("partners", now)).getRows()).hasSize(1); + for(String widget : FileGatewayHome.KEYS) + { + assertThat(home.data(widget, now).getType()).isIn("statistics", "table"); + } + assertThat(h.instance.getWidgets().values()).allSatisfy(widget -> assertThat(widget.getType()).isNotEqualTo("html")); + assertThat(FileGatewayIntegrations.esbTableNames()).containsExactly(FileGatewayTables.ACCOUNT, FileGatewayTables.ALIAS, FileGatewayTables.UPLOAD_JOB); + assertThat(FileGatewayIntegrations.apiTableNames()).containsExactlyElementsOf(FileGatewayTables.ALL); + assertThat(FileGatewayIntegrations.quickSearchTables()).hasSize(3); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/SftpIntegrationTest.java b/src/test/java/com/kingsrook/qbits/filegateway/SftpIntegrationTest.java new file mode 100644 index 0000000..6593ac1 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/SftpIntegrationTest.java @@ -0,0 +1,300 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + + +import java.security.KeyPair; +import java.time.Duration; +import java.time.Instant; +import java.util.ArrayList; +import java.util.List; +import com.kingsrook.qbits.filegateway.sftp.CidrMatcher; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInProcessSessions; +import com.kingsrook.qqq.openapi.model.HttpMethod; +import org.apache.sshd.common.config.keys.KeyUtils; +import org.apache.sshd.sftp.client.SftpClient; +import org.apache.sshd.sftp.client.SftpClientFactory; +import org.apache.sshd.sftp.common.SftpConstants; +import org.apache.sshd.sftp.common.SftpException; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Real protocol acceptance tests for listing, authentication, auditing and no-disk operation. + *******************************************************************************/ +@ExtendWith(NoDiskExtension.class) +class SftpIntegrationTest +{ + /******************************************************************************* + ** Pair. + *******************************************************************************/ + private KeyPair pair() throws Exception + { + return (KeyUtils.generateKeyPair("ssh-ed25519", 256)); + } + + + + /******************************************************************************* + ** Names. + *******************************************************************************/ + private List names(SftpClient client, String path) throws Exception + { + List result = new ArrayList<>(); + for(SftpClient.DirEntry entry : client.readDir(path)) + { + if(!entry.getFilename().equals(".") && !entry.getFilename().equals("..")) + { + result.add(entry.getFilename()); + } + } + return (result); + } + + + + /******************************************************************************* + ** Permission filtered listings and audit. + *******************************************************************************/ + @Test + void permissionFilteredListingsAndAudit() throws Exception + { + try(var harness = new FileGatewayTestServer()) + { + KeyPair a = pair(); + KeyPair b = pair(); + harness.key(harness.account("alice", "A"), a); + harness.key(harness.account("bob", "B"), b); + assertThat(harness.saved("savedBulkLoadProfile", "alpha", "Orders In").getErrors()).isEmpty(); + assertThat(harness.saved("savedView", "alpha", "All Orders").getErrors()).isEmpty(); + assertThat(harness.saved("savedReport", "alpha", "Daily Orders").getErrors()).isEmpty(); + int tokens = UiApiInProcessSessions.size(); + try(var sa = harness.login("alice", a); var sb = harness.login("bob", b); + var ca = SftpClientFactory.instance().createSftpClient(sa); var cb = SftpClientFactory.instance().createSftpClient(sb)) + { + assertThat(names(ca, "/")).contains("testapp"); + assertThat(names(ca, "/testapp")).containsExactlyInAnyOrder("alpha", "beta"); + assertThat(names(cb, "/testapp")).containsExactly("beta"); + assertThatThrownBy(() -> cb.stat("/testapp/alpha")).isInstanceOfSatisfying(SftpException.class, + e -> assertThat(e.getStatus()).isEqualTo(SftpConstants.SSH_FX_NO_SUCH_FILE)); + assertThat(names(ca, "/testapp/alpha")).contains("upload", "views", "reports", "results", "archive"); + assertThat(names(ca, "/testapp/beta")).doesNotContain("upload"); + assertThat(names(ca, "/testapp/alpha/upload")).containsExactly("orders-in"); + assertThat(names(ca, "/testapp/alpha/upload/orders-in")).isEmpty(); + assertThat(names(ca, "/testapp/alpha/views")).containsExactlyInAnyOrder("all-orders.csv", "all-orders.xlsx"); + assertThat(names(ca, "/testapp/alpha/reports")).contains("daily-orders.csv", "daily-orders.xlsx"); + assertThat(ca.stat("/testapp/alpha/views/all-orders.csv").getSize()).isZero(); + try(var download = ca.read("/testapp/alpha/views/all-orders.csv")) + { + assertThat(download.readAllBytes()).isNotEmpty(); + } + assertThatThrownBy(() -> ca.write("/testapp/alpha/test.csv")).isInstanceOf(SftpException.class); + assertThatThrownBy(() -> ca.setStat("/testapp/alpha/views/all-orders.csv", new SftpClient.Attributes().modifyTime(0))).isInstanceOfSatisfying(SftpException.class, + e -> assertThat(e.getStatus()).isEqualTo(SftpConstants.SSH_FX_PERMISSION_DENIED)); + assertThatThrownBy(() -> names(cb, "/testapp/alpha")).isInstanceOf(java.io.UncheckedIOException.class).hasCauseInstanceOf(SftpException.class); + assertThatThrownBy(() -> ca.mkdir("/testapp/alpha/new")).isInstanceOf(SftpException.class); + assertThatThrownBy(() -> ca.remove("/testapp/alpha/views/all-orders.csv")).isInstanceOf(SftpException.class); + assertThatThrownBy(() -> ca.rename("/testapp/alpha/views/all-orders.csv", "/testapp/alpha/views/new.csv")).isInstanceOf(SftpException.class); + assertThat(harness.runtime.getInvoker().findOperation(HttpMethod.GET, "/metaData")).isPresent(); + assertThat(harness.runtime.getInvoker().findOperation(HttpMethod.POST, "/table/{tableName}/query")).isPresent(); + assertThat(harness.runtime.getInvoker().getClientChannel()).isEqualTo("sftp"); + } + assertThat(harness.calls).isNotEmpty().allSatisfy(call -> + { + String template = call.path().substring("/qqq/v1".length()).replaceAll("/table/[^/]+/(query|export)", "/table/{tableName}/$1"); + assertThat(harness.runtime.getInvoker().findOperation(HttpMethod.valueOf(call.method()), template)).isPresent(); + assertThat(call.channel()).startsWith("sftp/"); + assertThat(call.invokedBy()).isEqualTo("file-gateway"); + }); + harness.runtime.stop(); + assertThat(UiApiInProcessSessions.size()).isEqualTo(tokens); + var connections = harness.audit.query("file_gateway_connection_log", new QQueryFilter()); + assertThat(connections).hasSize(2).allSatisfy(row -> + { + assertThat(row.getValueString("clientChannel")).isEqualTo("sftp"); + assertThat(row.getValueString("outcome")).isEqualTo("CONNECTED"); + assertThat(row.getValue("disconnectedAt")).isNotNull(); + }); + var transfers = harness.audit.query("file_gateway_transfer_log", new QQueryFilter()); + assertThat(transfers).anySatisfy(row -> + { + assertThat(row.getValueString("operation")).isEqualTo("LIST"); + assertThat(row.getValueString("outcome")).isEqualTo("OK"); + }).anySatisfy(row -> assertThat(row.getValueString("outcome")).isEqualTo("DENIED")) + .allSatisfy(row -> assertThat(row.getValueString("clientChannel")).startsWith("sftp/")); + } + } + + + + /******************************************************************************* + ** Refuses invalid credentials with audit. + *******************************************************************************/ + @ParameterizedTest + @ValueSource(strings = {"unknown", "revoked", "expiredKey", "disabled", "expiredAccount", "ip", "permission"}) + void refusesInvalidCredentialsWithAudit(String scenario) throws Exception + { + try(var h = new FileGatewayTestServer()) + { + var key = pair(); + var account = h.account("alice", scenario.equals("permission") ? "NONE" : "A"); + var stored = h.key(account, key); + String outcome = "AUTH_FAILED"; + switch(scenario) + { + case "revoked" -> h.audit.update("file_gateway_ssh_key", new QRecord().withValue("id", stored.getValue("id")).withValue("revokedAt", Instant.now())); + case "expiredKey" -> + { + h.audit.update("file_gateway_ssh_key", new QRecord().withValue("id", stored.getValue("id")).withValue("expiresAt", Instant.now().minusSeconds(1))); + outcome = "EXPIRED"; + } + case "disabled" -> + { + h.audit.update("file_gateway_account", new QRecord().withValue("id", account.getValue("id")).withValue("status", "DISABLED")); + outcome = "DISABLED"; + } + case "expiredAccount" -> + { + h.audit.update("file_gateway_account", new QRecord().withValue("id", account.getValue("id")).withValue("expiresAt", Instant.now().minusSeconds(1))); + outcome = "EXPIRED"; + } + case "ip" -> + { + h.audit.update("file_gateway_account", new QRecord().withValue("id", account.getValue("id")).withValue("allowedCidrs", "192.0.2.0/24")); + outcome = "IP_DENIED"; + } + } + KeyPair offered = scenario.equals("unknown") ? pair() : key; + assertThatThrownBy(() -> h.login("alice", offered)).isInstanceOf(Exception.class); + assertThat(h.audit.query("file_gateway_connection_log", new QQueryFilter()).getFirst().getValueString("outcome")).isEqualTo(outcome); + } + } + + + + /******************************************************************************* + ** Lockout and session limit. + *******************************************************************************/ + @Test + void lockoutAndSessionLimit() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData.ofOrWithNew(h.instance) + .withSink(new com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference(RecordingAlerts.class)); + RecordingAlerts.signals.clear(); + var account = h.account("alice", "A"); + var good = pair(); + h.key(account, good); + var bad = pair(); + for(int i = 0; i < 5; i++) + { + assertThatThrownBy(() -> h.login("alice", bad)).isInstanceOf(Exception.class); + } + assertThatThrownBy(() -> h.login("alice", good)).isInstanceOf(Exception.class); + assertThat(RecordingAlerts.signals).containsExactly("filegateway.lockout"); + var rows = h.audit.query("file_gateway_connection_log", new QQueryFilter()); + assertThat(rows).anySatisfy(row -> assertThat(row.getValueString("outcome")).isEqualTo("LOCKED_OUT")); + assertThat(h.audit.query("file_gateway_account", new QQueryFilter()).getFirst().getValue("lockedUntil")).isNotNull(); + h.audit.update("file_gateway_account", new QRecord().withValue("id", account.getValue("id")).withValue("lockedUntil", null)); + try(var unlocked = h.login("alice", good)) + { + assertThat(unlocked.isAuthenticated()).isTrue(); + } + } + try(var h = new FileGatewayTestServer()) + { + var account = h.account("alice", "A"); + h.audit.update("file_gateway_account", new QRecord().withValue("id", account.getValue("id")).withValue("maxConcurrentSessions", 1)); + var good = pair(); + h.key(account, good); + try(var session = h.login("alice", good)) + { + assertThatThrownBy(() -> h.login("alice", good)).isInstanceOf(Exception.class); + assertThat(h.audit.query("file_gateway_connection_log", new QQueryFilter())).anySatisfy(row -> assertThat(row.getValueString("outcome")).isEqualTo("LIMIT")); + } + } + } + + + + /******************************************************************************* + ** Public key only and live revocation. + *******************************************************************************/ + @Test + void publicKeyOnlyAndLiveRevocation() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + var account = h.account("alice", "A"); + var pair = pair(); + var key = h.key(account, pair); + var server = h.runtime.getProtocol().getServer(); + assertThat(server.getPasswordAuthenticator()).isNull(); + assertThat(server.getKeyboardInteractiveAuthenticator()).isNull(); + assertThat(server.getUserAuthFactories()).extracting(factory -> factory.getName()).containsExactly("publickey"); + assertThat(server.getCipherFactories()).noneSatisfy(factory -> assertThat(factory.getName()).contains("cbc")); + assertThat(server.getMacFactories()).allSatisfy(factory -> assertThat(factory.getName()).contains("sha2")); + assertThat(server.getSignatureFactories()).noneSatisfy(factory -> assertThat(factory.getName()).isEqualTo("ssh-rsa")); + try(var password = h.client.connect("alice", "127.0.0.1", h.runtime.getProtocol().getPort()).verify(Duration.ofSeconds(5)).getSession()) + { + password.addPasswordIdentity("not-a-password"); + assertThatThrownBy(() -> password.auth().verify(Duration.ofSeconds(5))).isInstanceOf(Exception.class); + } + try(var session = h.login("alice", pair)) + { + h.audit.update("file_gateway_ssh_key", new QRecord().withValue("id", key.getValue("id")).withValue("revokedAt", Instant.now())); + h.runtime.getProtocol().getAuthenticator().poll(); + session.waitFor(java.util.Set.of(org.apache.sshd.client.session.ClientSession.ClientSessionEvent.CLOSED), Duration.ofSeconds(5)); + assertThat(session.isClosed()).isTrue(); + } + } + assertThat(CidrMatcher.matches("::1", "::1/128")).isTrue(); + assertThat(CidrMatcher.matches("127.0.0.1", "::1/128")).isFalse(); + assertThat(CidrMatcher.matches("127.0.0.1", "localhost/32")).isFalse(); + } + + + + /******************************************************************************* + ** Exercise the core alert facade without adding any dependency on qbit-alerts. + *******************************************************************************/ + public static class RecordingAlerts extends com.kingsrook.qqq.backend.core.alerts.LoggingAlertSink + { + static final List signals = new java.util.concurrent.CopyOnWriteArrayList<>(); + + /******************************************************************************* + ** Record alert type names only, without account or key material. + *******************************************************************************/ + @Override + public void raise(com.kingsrook.qqq.backend.core.alerts.AlertSignal signal) + { + signals.add(signal.getAlertTypeName()); + } + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/TestAlertSink.java b/src/test/java/com/kingsrook/qbits/filegateway/TestAlertSink.java new file mode 100644 index 0000000..90f7fd1 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/TestAlertSink.java @@ -0,0 +1,62 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + +import java.time.Instant; +import java.util.List; +import java.util.concurrent.CopyOnWriteArrayList; +import com.kingsrook.qqq.backend.core.alerts.AlertResolution; +import com.kingsrook.qqq.backend.core.alerts.AlertSignal; +import com.kingsrook.qqq.backend.core.alerts.AlertSinkInterface; +import com.kingsrook.qqq.backend.core.alerts.NotificationSignal; + +/******************************************************************************* + ** Captures real core notification dispatch. + *******************************************************************************/ +public class TestAlertSink implements AlertSinkInterface +{ + public static final List notifications = new CopyOnWriteArrayList<>(); + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public void raise(AlertSignal signal) + { + } + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public void resolve(AlertResolution resolution) + { + } + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public void heartbeat(String key, Instant instant) + { + } + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public void notify(NotificationSignal signal) + { + notifications.add(signal); + } +} diff --git a/src/test/java/com/kingsrook/qbits/filegateway/UploadIntegrationTest.java b/src/test/java/com/kingsrook/qbits/filegateway/UploadIntegrationTest.java new file mode 100644 index 0000000..57b754b --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/filegateway/UploadIntegrationTest.java @@ -0,0 +1,595 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.filegateway; + +import java.nio.charset.StandardCharsets; +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.time.ZoneId; +import java.time.ZoneOffset; +import java.util.ArrayList; +import java.util.List; +import java.util.Set; +import java.util.concurrent.CompletableFuture; +import com.kingsrook.qbits.filegateway.api.FileAcceptance; +import com.kingsrook.qbits.filegateway.api.FileGatewayConsumer; +import com.kingsrook.qbits.filegateway.api.FileGatewayConsumersMetaData; +import com.kingsrook.qbits.filegateway.api.FileGatewayScannerInterface; +import com.kingsrook.qbits.filegateway.api.InboundFile; +import com.kingsrook.qbits.filegateway.metadata.FileGatewayProcesses; +import com.kingsrook.qbits.filegateway.upload.UploadJobWorker; +import com.kingsrook.qbits.filegateway.upload.UploadJobs; +import com.kingsrook.qqq.backend.core.actions.tables.QueryAction; +import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction; +import com.kingsrook.qqq.backend.core.context.QContext; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput; +import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter; +import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput; +import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput; +import com.kingsrook.qqq.backend.core.model.data.QRecord; +import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference; +import com.kingsrook.qqq.backend.core.model.session.QSession; +import org.apache.sshd.common.config.keys.KeyUtils; +import org.apache.sshd.sftp.client.SftpClient; +import org.apache.sshd.sftp.client.SftpClientFactory; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + +@ExtendWith(NoDiskExtension.class) +/******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ +class UploadIntegrationTest +{ + static final String FOLDER = "/testapp/alpha/upload/orders-in/"; + static final byte[] CSV = "name\nAlice\nBob\n".getBytes(StandardCharsets.UTF_8); + + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + static QRecord profile(FileGatewayTestServer h) throws Exception + { + QRecord profile = h.saved("savedBulkLoadProfile", "alpha", "Orders In"); + h.audit.system(() -> new UpdateAction().execute(new UpdateInput("savedBulkLoadProfile").withRecords(List.of(new QRecord() + .withValue("id", profile.getValue("id")).withValue("mappingJson", "{\"version\":\"v1\",\"layout\":\"FLAT\",\"hasHeaderRow\":true,\"fieldList\":[{\"fieldName\":\"name\",\"headerName\":\"name\",\"columnIndex\":null,\"defaultValue\":null}]}"))))); + return (profile); + } + + /******************************************************************************* + ** Verify the upload contract through the real SFTP stack. + *******************************************************************************/ + @Test + void bulkInsertAndArtifacts() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + profile(h); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + var account = h.account("alice", "A"); + h.key(account, key); + h.audit.update("file_gateway_account", new QRecord().withValue("id", account.getValue("id")).withValue("notifyOnSuccess", true)); + try(var ssh = h.login("alice", key); var sftp = SftpClientFactory.instance().createSftpClient(ssh)) + { + try(var out = sftp.write(FOLDER + "orders.csv")) + { + out.write(CSV); + } + var jobs = h.audit.query("file_gateway_upload_job", new QQueryFilter()); + assertThat(jobs).hasSize(1); + assertThat(jobs.getFirst().getValueString("status")).isEqualTo("QUEUED"); + assertThat(h.runtime.getWorker().runOnce()).isTrue(); + QRecord job = h.audit.query("file_gateway_upload_job", new QQueryFilter()).getFirst(); + assertThat(job.getValueString("status")).withFailMessage("Job values: %s", job.getValues()).isEqualTo("SUCCEEDED"); + var rows = h.audit.system(() -> new QueryAction().execute(new QueryInput("alpha")).getRecords()); + assertThat(rows).hasSize(2); + assertThat(TestAlertSink.notifications).hasSize(1); + assertThat(TestAlertSink.notifications.getFirst().getAudience().getValues()).containsExactly("A"); + var audits = h.audit.system(() -> new QueryAction().execute(new QueryInput("audit")).getRecords()); + assertThat(audits).anySatisfy(row -> assertThat(row.getValueString("clientChannel")).withFailMessage("%s", row.getValues()).startsWith("sftp/")); + assertThat(job.getValueInteger("insertedCount")).isEqualTo(2); + try(var in = sftp.read(FOLDER + "orders.csv.done")) + { + assertThat(new String(in.readAllBytes(), StandardCharsets.UTF_8)).contains("2", "SUCCEEDED"); + } + String date = job.getValueInstant("createDate").atZone(ZoneOffset.UTC).toLocalDate().toString(); + try(var in = sftp.read("/testapp/alpha/archive/" + date + "/" + job.getValue("id") + "-orders.csv")) + { + assertThat(in.readAllBytes()).isEqualTo(CSV); + } + try(var in = sftp.read("/testapp/alpha/results/" + job.getValue("id") + "-orders.csv.results.csv")) + { + assertThat(in.readAllBytes()).isEqualTo(h.store.objects.get(job.getValueString("resultsStorageKey"))); + } + } + } + } + + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + static List jobs(FileGatewayTestServer h) throws Exception + { + return (h.audit.query("file_gateway_upload_job", new QQueryFilter())); + } + + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + static void put(SftpClient sftp, String name, byte[] bytes) throws Exception + { + try(var out = sftp.write(FOLDER + name)) + { + out.write(bytes); + } + } + + /******************************************************************************* + ** Verify the upload contract through the real SFTP stack. + *******************************************************************************/ + @Test + void temporaryRenameDedupeAndAcknowledgement() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + profile(h); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(h.account("alice", "A"), key); + try(var ssh = h.login("alice", key); var sftp = SftpClientFactory.instance().createSftpClient(ssh)) + { + put(sftp, "orders.csv.filepart", CSV); + assertThat(jobs(h).getFirst().getValueString("status")).isEqualTo("RECEIVING"); + assertThat(h.runtime.getWorker().runOnce()).isFalse(); + sftp.rename(FOLDER + "orders.csv.filepart", FOLDER + "orders.csv"); + assertThat(jobs(h)).hasSize(1); + assertThat(h.runtime.getWorker().runOnce()).isTrue(); + assertThat(jobs(h).getFirst().getValueString("status")).isEqualTo("SUCCEEDED"); + put(sftp, "orders.csv", CSV); + assertThat(jobs(h)).hasSize(2); + assertThat(jobs(h).getLast().getValueString("status")).isEqualTo("DUPLICATE"); + assertThat(h.runtime.getWorker().runOnce()).isFalse(); + try(var in = sftp.read(FOLDER + "orders.csv.done")) + { + assertThat(new String(in.readAllBytes(), StandardCharsets.UTF_8)).contains("Skipped"); + } + assertThat(h.audit.system(() -> new QueryAction().execute(new QueryInput("alpha")).getRecords())).hasSize(2); + sftp.remove(FOLDER + "orders.csv.done"); + assertThat(jobs(h).getLast().getValue("acknowledgedAt")).isNotNull(); + String result = "/testapp/alpha/results/" + jobs(h).getFirst().getValue("id") + "-orders.csv.results.csv"; + sftp.remove(result); + assertThatThrownBy(() -> sftp.stat(result)).isInstanceOf(Exception.class); + } + } + } + + /******************************************************************************* + ** Verify the upload contract through the real SFTP stack. + *******************************************************************************/ + @Test + void badRowAndMissingValueMappings() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + var profile = profile(h); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(h.account("alice", "A"), key); + try(var ssh = h.login("alice", key); var sftp = SftpClientFactory.instance().createSftpClient(ssh)) + { + h.audit.system(() -> new UpdateAction().execute(new UpdateInput("savedBulkLoadProfile").withRecords(List.of(new QRecord() + .withValue("id", profile.getValue("id")).withValue("mappingJson", "{\"version\":\"v1\",\"layout\":\"FLAT\",\"hasHeaderRow\":true,\"fieldList\":[{\"fieldName\":\"name\",\"headerName\":\"name\",\"columnIndex\":null,\"defaultValue\":null},{\"fieldName\":\"quantity\",\"headerName\":\"quantity\",\"columnIndex\":null,\"defaultValue\":null}]}"))))); + put(sftp, "bad.csv", "name,quantity\nAlice,1\nBob,bad\n".getBytes(StandardCharsets.UTF_8)); + h.runtime.getWorker().runOnce(); + var job = jobs(h).getFirst(); + assertThat(job.getValueString("status")).withFailMessage("%s", job.getValues()).isEqualTo("PARTIAL"); + assertThat(job.getValueInteger("insertedCount")).isEqualTo(1); + assertThat(job.getValueInteger("errorCount")).isEqualTo(1); + assertThat(new String(h.store.objects.get(job.getValueString("resultsStorageKey")), StandardCharsets.UTF_8)).contains("Quantity", "3"); + assertThat(TestAlertSink.notifications).hasSize(1); + h.audit.system(() -> new UpdateAction().execute(new UpdateInput("savedBulkLoadProfile").withRecords(List.of(new QRecord() + .withValue("id", profile.getValue("id")).withValue("mappingJson", "{\"version\":\"v1\",\"layout\":\"FLAT\",\"hasHeaderRow\":true,\"fieldList\":[{\"fieldName\":\"name\",\"doValueMapping\":true}]}"))))); + put(sftp, "mapping.csv", CSV); + h.runtime.getWorker().runOnce(); + assertThat(jobs(h).getLast().getValueString("status")).isEqualTo("FAILED"); + assertThat(jobs(h).getLast().getValueString("errorMessage")).contains("this profile needs value mappings saved in it"); + } + } + } + + /******************************************************************************* + ** Verify the upload contract through the real SFTP stack. + *******************************************************************************/ + @Test + void fileAndDailyLimitsDeletePartialObjects() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + profile(h); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + var account = h.account("alice", "A"); + h.key(account, key); + h.audit.update("file_gateway_account", new QRecord().withValue("id", account.getValue("id")).withValue("maxFileSizeMb", 1)); + try(var ssh = h.login("alice", key); var sftp = SftpClientFactory.instance().createSftpClient(ssh)) + { + assertThatThrownBy(() -> put(sftp, "large.csv", new byte[1024 * 1024 + 1])).isInstanceOf(Exception.class); + var job = jobs(h).getFirst(); + assertThat(job.getValueString("status")).isEqualTo("REJECTED"); + assertThat(h.store.objects).doesNotContainKey(job.getValueString("storageKey")); + assertThat(h.audit.query("file_gateway_transfer_log", new QQueryFilter())).anySatisfy(log -> assertThat(log.getValueString("outcome")).isEqualTo("LIMIT")); + h.audit.update("file_gateway_account", new QRecord().withValue("id", account.getValue("id")).withValue("maxUploadMbPerDay", 1)); + assertThatThrownBy(() -> put(sftp, "quota.csv", new byte[1024 * 1024])).isInstanceOf(Exception.class); + assertThat(jobs(h).getLast().getValueString("errorMessage")).contains("Daily upload quota"); + } + } + } + + /******************************************************************************* + ** Verify the upload contract through the real SFTP stack. + *******************************************************************************/ + @Test + void permissionDenialCreatesNoJob() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + profile(h); + FileGatewayTestServer.GRANTS.put("A", Set.of("file_gateway.connect", "alpha.read")); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(h.account("alice", "A"), key); + try(var ssh = h.login("alice", key); var sftp = SftpClientFactory.instance().createSftpClient(ssh)) + { + var names = new ArrayList(); + for(var entry : sftp.readDir("/testapp/alpha")) + { + names.add(entry.getFilename()); + } + assertThat(names).doesNotContain("upload"); + assertThatThrownBy(() -> put(sftp, "denied.csv", CSV)).isInstanceOf(Exception.class); + assertThat(jobs(h)).isEmpty(); + } + } + } + + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public static class Scanner implements FileGatewayScannerInterface + { + static Result next; + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public Result scan(InboundFile file) throws Exception + { + try(var in = file.openStream()) + { + assertThat(in.readAllBytes()).isEqualTo(CSV); + } + return (next); + } + } + + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public static class Consumer implements FileGatewayConsumer + { + static int count; + static FileAcceptance next; + static InboundFile received; + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public String getName() + { + return ("test-consumer"); + } + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public FileAcceptance accept(InboundFile file) throws Exception + { + count++; + received = file; + assertThat(QContext.getQSession().getClientChannel()).startsWith("sftp/"); + assertThat(QContext.getQSession().hasPermission("alpha.insert")).isTrue(); + try(var in = file.openStream()) + { + assertThat(in.readAllBytes()).isEqualTo(CSV); + } + return (next); + } + } + + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public static class MutableClock extends Clock + { + Instant now = Instant.parse("2026-10-14T03:00:00Z"); + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public Instant instant() + { + return (now); + } + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public ZoneId getZone() + { + return (ZoneOffset.UTC); + } + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + public Clock withZone(ZoneId zone) + { + return (this); + } + /******************************************************************************* + ** Test fixture operation. + *******************************************************************************/ + void advance(Duration by) + { + now = now.plus(by); + } + } + + /******************************************************************************* + ** Verify the upload contract through the real SFTP stack. + *******************************************************************************/ + @Test + void scanAndConsumerRetriesAfterDisconnect() throws Exception + { + var clock = new MutableClock(); + try(var h = new FileGatewayTestServer(clock)) + { + profile(h); + h.config.withScanner(new QCodeReference(Scanner.class)); + FileGatewayConsumersMetaData.ofOrWithNew(h.instance).withConsumer(new QCodeReference(Consumer.class)); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + var account = h.account("alice", "A"); + h.key(account, key); + try(var ssh = h.login("alice", key); var sftp = SftpClientFactory.instance().createSftpClient(ssh)) + { + put(sftp, "in.csv", CSV); + } + QRecord job = jobs(h).getFirst(); + h.audit.update("file_gateway_upload_job", new QRecord().withValue("id", job.getValue("id")).withValue("consumerName", "test-consumer")); + Scanner.next = FileGatewayScannerInterface.Result.ERROR; + Consumer.count = 0; + h.runtime.getWorker().runOnce(); + assertThat(jobs(h).getFirst().getValueInstant("nextAttemptAt")).isEqualTo(clock.instant().plusSeconds(60)); + assertThat(h.runtime.getWorker().runOnce()).isFalse(); + clock.advance(Duration.ofMinutes(1)); + Scanner.next = FileGatewayScannerInterface.Result.CLEAN; + Consumer.next = FileAcceptance.retryLater("Busy", Duration.ofSeconds(2)); + h.runtime.getWorker().runOnce(); + assertThat(Consumer.count).isEqualTo(1); + assertThat(Consumer.received.sha256()).isEqualTo(job.getValueString("sha256")); + assertThat(Consumer.received.remoteAddress()).isEqualTo("127.0.0.1"); + assertThat(Consumer.received.username()).isEqualTo("alice"); + assertThat(Consumer.received.endpointName()).isNull(); + assertThat(jobs(h).getFirst().getValueInstant("nextAttemptAt")).isEqualTo(clock.instant().plusSeconds(300)); + clock.advance(Duration.ofMinutes(5)); + Consumer.next = FileAcceptance.accepted("accepted-123"); + h.runtime.getWorker().runOnce(); + assertThat(jobs(h).getFirst().getValueString("status")).isEqualTo("SUCCEEDED"); + assertThat(jobs(h).getFirst().getValueString("consumerReference")).isEqualTo("accepted-123"); + } + } + + /******************************************************************************* + ** Verify the upload contract through the real SFTP stack. + *******************************************************************************/ + @Test + void quarantineUnknownConsumerAndRetryAction() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + profile(h); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(h.account("alice", "A"), key); + try(var ssh = h.login("alice", key); var sftp = SftpClientFactory.instance().createSftpClient(ssh)) + { + put(sftp, "infected.csv", CSV); + String original = jobs(h).getFirst().getValueString("storageKey"); + h.config.withScanner(new QCodeReference(Scanner.class)); + Scanner.next = FileGatewayScannerInterface.Result.INFECTED; + h.runtime.getWorker().runOnce(); + var rejected = jobs(h).getFirst(); + assertThat(rejected.getValueString("status")).isEqualTo("REJECTED"); + assertThat(h.store.objects).doesNotContainKey(original); + assertThat(h.store.objects.get(rejected.getValueString("storageKey"))).isEqualTo(CSV); + assertThat(rejected.getValueString("storageKey")).startsWith("file-gateway/quarantine/"); + h.config.withScanner(null); + put(sftp, "unknown.csv", CSV); + var job = jobs(h).getLast(); + h.audit.update("file_gateway_upload_job", new QRecord().withValue("id", job.getValue("id")).withValue("consumerName", "missing")); + h.runtime.getWorker().runOnce(); + job = jobs(h).getLast(); + assertThat(job.getValueString("status")).isEqualTo("REJECTED"); + assertThat(job.getValueString("errorMessage")).contains("Unknown consumer"); + QContext.init(h.instance, new QSession().withPermissions(Set.of("file_gateway.admin"))); + var input = new RunBackendStepInput().withProcessName(FileGatewayProcesses.RETRY_UPLOAD).withRecords(List.of(job)); + var review = new FileGatewayProcesses.Review(); + var output = new RunBackendStepOutput(); + review.runOnePage(input, output); + assertThat(review.getProcessSummary(output, false)).hasSize(1); + new FileGatewayProcesses.Apply().runOnePage(input, new RunBackendStepOutput()); + assertThat(jobs(h).getLast().getValueString("status")).isEqualTo("QUEUED"); + assertThat(jobs(h).getLast().getValueInteger("attempts")).isZero(); + } + } + } + + /******************************************************************************* + ** Verify the upload contract through the real SFTP stack. + *******************************************************************************/ + @Test + void exclusiveClaimsStaleRecoveryAndStop() throws Exception + { + var clock = new MutableClock(); + try(var h = new FileGatewayTestServer(clock)) + { + profile(h); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(h.account("alice", "A"), key); + try(var ssh = h.login("alice", key); var sftp = SftpClientFactory.instance().createSftpClient(ssh)) + { + put(sftp, "claim.csv", CSV); + } + var job = jobs(h).getFirst(); + var first = new UploadJobWorker(h.instance, h.runtime.getInvoker(), h.config, h.store, clock, "pod-a"); + var second = new UploadJobWorker(h.instance, h.runtime.getInvoker(), h.config, h.store, clock, "pod-b"); + var a = CompletableFuture.supplyAsync(() -> + { + try + { + return (first.claim(job)); + } + catch(Exception e) + { + throw new RuntimeException(e); + } + } + ); + var b = CompletableFuture.supplyAsync(() -> + { + try + { + return (second.claim(job)); + } + catch(Exception e) + { + throw new RuntimeException(e); + } + } + ); + assertThat((a.get() == null ? 0 : 1) + (b.get() == null ? 0 : 1)).isEqualTo(1); + clock.advance(Duration.ofMinutes(31)); + second.requeueStale(); + assertThat(jobs(h).getFirst().getValueString("status")).isEqualTo("QUEUED"); + assertThat(first.claim(jobs(h).getFirst())).isNotNull(); + first.close(); + assertThat(jobs(h).getFirst().getValueString("status")).isEqualTo("QUEUED"); + assertThat(first.runOnce()).isFalse(); + second.close(); + } + } + + + /******************************************************************************* + ** Verify the upload contract through the real SFTP stack. + *******************************************************************************/ + @Test + void reorderedWritesRangesAndMissingBlocks() throws Exception + { + try(var h = new FileGatewayTestServer()) + { + profile(h); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + h.key(h.account("alice", "A"), key); + try(var ssh = h.login("alice", key); var sftp = SftpClientFactory.instance().createSftpClient(ssh)) + { + try(var handle = sftp.open(FOLDER + "reordered.csv", SftpClient.OpenMode.Write, SftpClient.OpenMode.Create, SftpClient.OpenMode.Truncate)) + { + sftp.write(handle, 5, CSV, 5, CSV.length - 5); + sftp.write(handle, 0, CSV, 0, 5); + } + assertThat(h.store.objects.get(jobs(h).getFirst().getValueString("storageKey"))).isEqualTo(CSV); + h.runtime.getWorker().runOnce(); + var job = jobs(h).getFirst(); + String date = job.getValueInstant("createDate").atZone(ZoneOffset.UTC).toLocalDate().toString(); + String path = "/testapp/alpha/archive/" + date + "/" + job.getValue("id") + "-reordered.csv"; + assertThat(sftp.stat(path).getSize()).isEqualTo(CSV.length); + byte[] part = new byte[4]; + try(var handle = sftp.open(path, SftpClient.OpenMode.Read)) + { + assertThat(sftp.read(handle, 6, part)).isEqualTo(4); + assertThat(part).isEqualTo(java.util.Arrays.copyOfRange(CSV, 6, 10)); + } + assertThat(h.audit.query("file_gateway_transfer_log", new QQueryFilter())).anySatisfy(log -> + { + assertThat(log.getValueString("operation")).isEqualTo("DOWNLOAD"); + assertThat(log.getValueLong("bytes")).isEqualTo(4); + }); + var hole = sftp.open(FOLDER + "hole.csv", SftpClient.OpenMode.Write, SftpClient.OpenMode.Create); + sftp.write(hole, 5, CSV, 5, CSV.length - 5); + assertThatThrownBy(hole::close).isInstanceOf(Exception.class); + assertThat(jobs(h).getLast().getValueString("status")).isEqualTo("ABORTED"); + assertThat(h.store.objects).doesNotContainKey(jobs(h).getLast().getValueString("storageKey")); + h.config.withUploadReorderWindowMb(1); + try(var far = sftp.open(FOLDER + "far.csv", SftpClient.OpenMode.Write, SftpClient.OpenMode.Create)) + { + assertThatThrownBy(() -> sftp.write(far, 2 * 1024 * 1024, CSV, 0, 1)).isInstanceOf(Exception.class); + } + assertThat(jobs(h).getLast().getValueString("status")).isEqualTo("ABORTED"); + } + } + } + + /******************************************************************************* + ** Verify the upload contract through the real SFTP stack. + *******************************************************************************/ + @Test + void retriesExhaustAndServiceOwnerIsNotified() throws Exception + { + var clock = new MutableClock(); + try(var h = new FileGatewayTestServer(clock)) + { + profile(h); + var key = KeyUtils.generateKeyPair("ssh-ed25519", 256); + var account = h.account("service", "A"); + h.audit.update("file_gateway_account", new QRecord().withValue("id", account.getValue("id")) + .withValue("accountType", "SERVICE").withValue("ownerUserId", "OWNER").withValue("rootMode", "FULL_TREE")); + h.key(account, key); + try(var ssh = h.login("service", key); var sftp = SftpClientFactory.instance().createSftpClient(ssh)) + { + put(sftp, "retry.csv", CSV); + } + FileGatewayConsumersMetaData.ofOrWithNew(h.instance).withConsumer(new QCodeReference(Consumer.class)); + Consumer.next = FileAcceptance.retryLater("Busy", null); + Consumer.count = 0; + h.audit.update("file_gateway_upload_job", new QRecord().withValue("id", jobs(h).getFirst().getValue("id")).withValue("consumerName", "test-consumer")); + long[] delays = {60, 300, 900, 3600}; + for(int attempt = 0; attempt < 5; attempt++) + { + assertThat(h.runtime.getWorker().runOnce()).isTrue(); + if(attempt < 4) + { + assertThat(jobs(h).getFirst().getValueInstant("nextAttemptAt")).isEqualTo(clock.instant().plusSeconds(delays[attempt])); + clock.advance(Duration.ofSeconds(delays[attempt])); + } + } + assertThat(jobs(h).getFirst().getValueString("status")).isEqualTo("FAILED"); + assertThat(Consumer.count).isEqualTo(5); + assertThat(TestAlertSink.notifications).hasSize(1); + assertThat(TestAlertSink.notifications.getFirst().getAudience().getValues()).containsExactly("OWNER"); + assertThat(TestAlertSink.notifications.getFirst().getLink().getTableName()).isEqualTo("file_gateway_upload_job"); + assertThat(h.runtime.getWorker().runOnce()).isFalse(); + } + } + +} diff --git a/src/test/resources/log4j2.xml b/src/test/resources/log4j2.xml new file mode 100644 index 0000000..4128d71 --- /dev/null +++ b/src/test/resources/log4j2.xml @@ -0,0 +1,11 @@ + + + + + + + + + + +