feat: SFTP server, virtual tree, SSH-key auth and listing (phase a) #5

Merged
KofTwentyTwo merged 3 commits from feature/phase-a-sftp-server into develop 2026-10-14 14:17:01 +00:00
55 changed files with 12978 additions and 99 deletions

No files matched your search

+27 -4
View File
@@ -14,18 +14,41 @@ code and QInstance. Design: `docs/DESIGN.md`.
## Build commands
```bash
./mvnw compile
./mvnw test
./mvnw verify # Checkstyle; JaCoCo: 70% instructions, 90% classes
./mvnw -o -Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT compile
./mvnw -o -Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT test
./mvnw -o -Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT verify # Checkstyle; JaCoCo: 70% instructions, 90% classes
```
Use the Maven 3.9.11 wrapper (Java 21); Maven 3.10 cannot read the parent POM. On Windows use `./mvnw.cmd`.
Use the Maven 3.9.11 wrapper (Java 21); Maven 3.10 cannot read the parent POM. On Windows use `./mvnw.cmd` and quote the `-Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT` argument in PowerShell.
Keep the POM default at `4.1.0-SNAPSHOT` for CI; the private version prevents local artifact replacement.
## Architecture
Package: `com.kingsrook.qbits.filegateway`. Entry point: `FileGatewayQBitProducer implements
QBitMetaDataProducer<FileGatewayQBitConfig>`.
- `metadata/`: seven stable tables, enum/picker sources, joins, typed child widgets, key/account
validators and streamed Review/Result admin actions. Producer adds metadata to the QInstance.
- `model/`: QRecordEntity mappings; DDL in `docs/ddl/file_gateway.sql` uses UTC zone-less timestamps.
- `permissions/`: view/admin/connect constants and the custom permission checker; system-written
logs/jobs/outbox tables remain read-only to people.
- `FileGatewayRuntime`: host-owned startup after the UI API server, in-memory host-key parsing,
injectable environment and clock/test hooks. Production polling interval is 60 seconds.
- `sftp/`: public-key-only MINA listener, per-account transactional authentication, per-pod IP
windows, token refresh/revocation and system-session bookkeeping restricted to owned tables.
- `vfs/`: custom NIO filesystem; per-session UI API metadata/saved-object caches. Every API call
checks its operation spec and sends user credentials plus `sftp/<sessionId>` client detail.
- `api/`: isolated consumer/outbox contract. Never import gateway implementation packages here.
Phase (a) supports listing/stat and denies reads/mutations with audit. Upload/download workers,
endpoint routing/chroot, outbox delivery, Quick Search/ESB, preferences, retention and the app
home belong to later phases. The core-valid alert type is `filegateway.lockout`; the design's
underscore-only name is rejected by core validation.
Tests run a strict UI API server, H2 from the adopter DDL and real MINA clients. NoDiskExtension
snapshots the working tree (excluding target) and isolated test home, and asserts the test temp
directory is empty. The host supplies RDBMS; `qqq-backend-module-rdbms` must remain test-scoped.
## Invariants and naming
- Data access only through `/qqq/v1` (`UiApiInvoker`, channel `sftp`) as the signed-in user. The
+18 -1
View File
@@ -6,4 +6,21 @@ All notable changes are documented here, following [Keep a Changelog](https://ke
### Added
- Initial scaffold
- Phase (a) SSH-key-only SFTP runtime with in-memory host keys, explicit modern algorithms,
account/key expiry, numeric CIDR allowlists, transactional account lockout/session limits,
per-IP limits, user-session refresh and live revocation.
- Virtual listing/stat of permitted apps, tables, saved bulk-load profiles, views and reports
through the strict UI API invoker, with per-session caches and paged saved-object queries.
- Seven gateway tables and record entities, H2/PostgreSQL DDL, enum sources, dependent endpoint
pickers, joins, typed child lists, permissions and channel walls.
- SSH key/account validation and Review/Result actions to unlock/disable accounts, revoke keys
and end sessions; connection and operation auditing, including denied mutations.
- Isolated consumer/outbox API contract and complete configuration for subsequent phases.
- Real SFTP/strict UI API acceptance tests, no-disk guard, schema round trips and adoption docs.
### Compatibility
- Local invoker integration builds use `4.1.0-FG-SNAPSHOT`; CI retains the POM's
`4.1.0-SNAPSHOT` default. RDBMS remains a host-provided runtime dependency.
- Alert type `filegateway.lockout` uses the dotted syntax required by core validation.
- Uploads, downloads, endpoint routing/chroot, outbox delivery and app home remain later phases.
+27 -4
View File
@@ -14,18 +14,41 @@ code and QInstance. Design: `docs/DESIGN.md`.
## Build commands
```bash
./mvnw compile
./mvnw test
./mvnw verify # Checkstyle; JaCoCo: 70% instructions, 90% classes
./mvnw -o -Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT compile
./mvnw -o -Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT test
./mvnw -o -Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT verify # Checkstyle; JaCoCo: 70% instructions, 90% classes
```
Use the Maven 3.9.11 wrapper (Java 21); Maven 3.10 cannot read the parent POM. On Windows use `./mvnw.cmd`.
Use the Maven 3.9.11 wrapper (Java 21); Maven 3.10 cannot read the parent POM. On Windows use `./mvnw.cmd` and quote the `-Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT` argument in PowerShell.
Keep the POM default at `4.1.0-SNAPSHOT` for CI; the private version prevents local artifact replacement.
## Architecture
Package: `com.kingsrook.qbits.filegateway`. Entry point: `FileGatewayQBitProducer implements
QBitMetaDataProducer<FileGatewayQBitConfig>`.
- `metadata/`: seven stable tables, enum/picker sources, joins, typed child widgets, key/account
validators and streamed Review/Result admin actions. Producer adds metadata to the QInstance.
- `model/`: QRecordEntity mappings; DDL in `docs/ddl/file_gateway.sql` uses UTC zone-less timestamps.
- `permissions/`: view/admin/connect constants and the custom permission checker; system-written
logs/jobs/outbox tables remain read-only to people.
- `FileGatewayRuntime`: host-owned startup after the UI API server, in-memory host-key parsing,
injectable environment and clock/test hooks. Production polling interval is 60 seconds.
- `sftp/`: public-key-only MINA listener, per-account transactional authentication, per-pod IP
windows, token refresh/revocation and system-session bookkeeping restricted to owned tables.
- `vfs/`: custom NIO filesystem; per-session UI API metadata/saved-object caches. Every API call
checks its operation spec and sends user credentials plus `sftp/<sessionId>` client detail.
- `api/`: isolated consumer/outbox contract. Never import gateway implementation packages here.
Phase (a) supports listing/stat and denies reads/mutations with audit. Upload/download workers,
endpoint routing/chroot, outbox delivery, Quick Search/ESB, preferences, retention and the app
home belong to later phases. The core-valid alert type is `filegateway.lockout`; the design's
underscore-only name is rejected by core validation.
Tests run a strict UI API server, H2 from the adopter DDL and real MINA clients. NoDiskExtension
snapshots the working tree (excluding target) and isolated test home, and asserts the test temp
directory is empty. The host supplies RDBMS; `qqq-backend-module-rdbms` must remain test-scoped.
## Invariants and naming
- Data access only through `/qqq/v1` (`UiApiInvoker`, channel `sftp`) as the signed-in user. The
+153 -23
View File
@@ -1,18 +1,22 @@
# QBit File Gateway
An SFTP server for QQQ applications. Users and service accounts sign in with their SSH keys and see
the application as a folder tree of apps and tables. Uploading a file into a table's
`upload/<profile>/` folder runs a saved bulk-load profile as that user; getting a file from
`views/` or `reports/` exports a saved view or runs a saved report. The gateway reaches data only
through the `/qqq/v1` UI API with the signed-in user's own session (client channel `sftp`), so
permissions, masking and audits apply exactly as in the UI. Design: [docs/DESIGN.md](docs/DESIGN.md).
An SSH-key-only SFTP server that presents a QQQ application's permitted apps and tables as a
virtual folder tree. Application metadata and saved objects are read through the spec-defined
`/qqq/v1` UI API as the signed-in user, using `UiApiInvoker` on channel `sftp`.
This repository currently holds the scaffold; the gateway is built in phases described in the design.
Phase (a) implements authentication, listing/stat, account administration and audit. Uploads,
downloads, endpoint routing/chroot, outbox delivery, preferences and the app home follow in later
phases. File reads and every mutation are denied and audited. No native filesystem, host-key
file, home directory or spool is used by the gateway. See [the approved design](docs/DESIGN.md).
## Requirements
Java 21 and QQQ 4.1.0-SNAPSHOT (UI API invoker and client channels, until 4.1.0-RC.6). The Maven
wrapper pins Maven 3.9.11.
Java 21, Apache MINA SSHD 2.16.0, and the QQQ 4.1 invoker APIs. The host supplies the RDBMS
backend and core saved bulk-load profile, saved view and saved report metadata. The gateway's
RDBMS module dependency is test-scoped. Maven 3.9.11 is pinned by the wrapper.
Until the invoker core PR merges, local builds use the private `4.1.0-FG-SNAPSHOT` artifacts.
The POM default remains `4.1.0-SNAPSHOT` for CI, to be replaced by 4.1.0-RC.6 when released.
## Install
@@ -24,32 +28,158 @@ wrapper pins Maven 3.9.11.
</dependency>
```
## Adopting
Apply [docs/ddl/file_gateway.sql](docs/ddl/file_gateway.sql) to the host's database. It supports
H2 and PostgreSQL and stores UTC timestamps without a zone, matching QQQ's RDBMS binding.
The gateway does not create or migrate tables at runtime.
Register metadata while building the QInstance:
```java
FileGatewayQBitConfig config = new FileGatewayQBitConfig()
.withBackendName("rdbms")
.withSftpEnabled(Boolean.parseBoolean(System.getenv("FILE_GATEWAY_SFTP_ENABLED")))
.withHostKeyEnvVarName("FILE_GATEWAY_SSH_HOST_KEY");
new FileGatewayQBitProducer().withQBitConfig(config).produce(qInstance);
```
After starting the host's `QApplicationJavalinServer`, start SFTP only in the gateway deployment:
```java
FileGatewayRuntime gateway = null;
if(config.getSftpEnabled())
{
gateway = FileGatewayRuntime.start(qInstance, server);
}
// Call gateway.stop() during application shutdown when gateway is non-null.
```
Supply an unencrypted OpenSSH private host key as the value of the named environment variable.
It is parsed in memory; a missing key fails startup. `allowGeneratedHostKey` is for development
only and generates an ephemeral key with a warning. Each deployment uses the same host application
and QInstance; expose its SFTP listener separately from the UI HTTP port.
Create an ACTIVE account with a QQQ `userId`, then add an Ed25519, ECDSA, or RSA public key of at
least 3072 bits. Public-key validation runs on insert and update and computes its unique SHA-256
fingerprint, type and size. The host authentication module must assign the user's permissions
when `createAutomatedSessionForUser` is called, including `file_gateway.connect`.
## Tables and permissions
Planned tables (none are produced yet), all prefixed `file_gateway_`: accounts, SSH keys, aliases,
upload jobs, the transfer log and the connection log. Permissions use the full `file_gateway`
prefix.
All seven tables are registered, with camelCase QQQ fields mapped to snake_case database columns.
`backendName` selects the host backend; physical table names match the QQQ table names.
| Table | Purpose |
|---|---|
| `file_gateway_account` | User/service login, limits, CIDRs, expiry and lockout state |
| `file_gateway_ssh_key` | Account child keys and computed fingerprints |
| `file_gateway_alias` | Endpoint definitions; routing is a later phase |
| `file_gateway_upload_job` | Stable schema for the later upload worker |
| `file_gateway_outbox_file` | Stable schema for later outbound delivery |
| `file_gateway_transfer_log` | Listings and denied operations |
| `file_gateway_connection_log` | Accept, auth outcome, heartbeat and disconnect |
One File Gateway app lists Accounts, Endpoints, Upload Jobs, Transfers and Connections. Keys,
outbox files and related jobs/logs appear in typed child lists. Endpoint saved-object pickers
are filtered by the selected table. There is no HTML widget or navigation process.
| Permission | Grants |
|---|---|
| `file_gateway.connect` | SFTP login, checked again when the user session is refreshed |
| `file_gateway.view` | App and read access to gateway tables |
| `file_gateway.admin` | Read access and management of accounts, keys, endpoints and record actions |
System-written job/outbox/audit tables are read-only to people, including admins. Record actions
use the standard Review/Result flow: Unlock account, Disable account, Revoke key and End session.
End session is enforced by the owning listener's next revocation poll. Retry upload is phase (b).
The `sftp` channel cannot access gateway, Authentik, audit, credential, API-key, secret, password,
permission, role, user-session or webhook administration surfaces. The `assistant` channel also
cannot access gateway tables.
## Listing and authentication
The root lists visible apps; apps contain child apps and readable tables. Table folders include:
- `upload/<profile-slug>/` when a visible profile and the table's bulk-insert process exist;
- `views/<slug>.csv` and `.xlsx` (or account-enabled JSON);
- `reports/<slug>.<format>` when the saved-report table and render process are available;
- empty `results/` and `archive/` folders reserved for later phases.
Hidden paths return no-such-file. Slug collisions append record IDs. Saved-object queries are
paged. Metadata and saved objects are cached per connection for 60 and 30 seconds respectively.
Generated files report size zero and reads return "not yet supported" in phase (a).
Only public-key auth is advertised. Password, keyboard-interactive, GSSAPI, host-based auth,
shell, exec and forwarding are disabled. The algorithm lists exclude CBC, SHA-1 MAC/KEX and
`ssh-rsa` signatures. Idle timeout is 15 minutes; authentication timeout is 30 seconds.
Account failure counters, operation-rate windows and session reservations are serialized with a database row lock across
listeners. Per-IP failure and connection windows are held in each pod. Unlocking an account also
clears an IP lock caused solely by that account on its next attempt; mixed-user IP locks expire
independently. User sessions are re-minted after five minutes on the next operation, expire after
ten minutes, and are revoked on close. The 60-second poll closes disabled/expired/revoked sessions
and heartbeats connection rows; three-minute-old heartbeats no longer count as active sessions.
Lockouts raise core `QAlerts` signals. **Core compatibility:** the alert type is
`filegateway.lockout`, because the core validator rejects the design's underscore-only
`file_gateway_lockout`. Without an alert sink the core logging fallback is used.
## Configuration
`FileGatewayQBitConfig` holds the optional `tableMetaDataCustomizer`, `sftpEnabled` (only the
gateway deployment runs the server) and `sftpPort` (default 2222).
Every option has a getter and fluent `withX(...)` setter. Later-phase options are present in the
contract but do not enable transfers in phase (a).
## Adopting in another app
| Options | Defaults |
|---|---|
| `backendName`, `sftpEnabled`, `sftpPort` | `rdbms`, `false`, `2222` |
| `hostKeyEnvVarName`, `allowGeneratedHostKey` | `FILE_GATEWAY_SSH_HOST_KEY`, `false` |
| `lockoutThreshold`, `lockoutWindow`, `lockoutDuration` | 5, 15 minutes, 30 minutes |
| `maxConnectionsPerIpPerMinute` | 30 |
| `defaultMaxConcurrentSessions`, `defaultMaxOperationsPerMinute` | 4, 600 |
| `defaultMaxFileSizeMb` | 100 |
| `defaultMaxUploadMbPerDay`, `defaultMaxDownloadMbPerDay` | null (unlimited) |
| `fileGatewayBucketAlias` | `file-gateway` |
| `uploadPartSizeMb`, `uploadReorderWindowMb`, `downloadReadAheadMb`, `maxGeneratedFileMb` | 8, 64, 16, 1024 |
| `uploadWorkerThreads`, `dedupeWindow`, `jobStaleAfter` | 4, 7 days, 30 minutes |
| `temporaryUploadSuffixes` | `.filepart`, `.tmp`, `.part`, `.partial` |
| `scanner`, `pgpPrivateKeyEnvVarName`, `pgpPassphraseEnvVarName` | null |
| `logRetention`, `archiveRetention`, `outboxRetention`, `resultsRetention`, `receivingStaleAfter` | 180 days, 30 days, 30 days, 90 days, 1 hour |
| `tableMetaDataCustomizer` | null |
```java
new FileGatewayQBitProducer()
.withQBitConfig(new FileGatewayQBitConfig())
.produce(qInstance);
```
## Consumer and outbox API
## Build
The isolated `com.kingsrook.qbits.filegateway.api` package contains `FileGatewayConsumer`,
`InboundFile` (including remote address and a fresh stream supplier), `FileAcceptance`,
`OutboxFileRef`, and the consumer metadata registry. `FileGatewayOutbox` delegates to a registered
`FileGatewayOutboxProvider`; without one it reports unavailable and throws
`FileGatewayUnavailableException`. Phase (a) supplies the contract only; the later outbox runtime
will install the provider.
## Build and acceptance tests
```bash
./mvnw verify # Checkstyle; JaCoCo 70% instructions, 90% classes
./mvnw -o -Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT verify
```
On Windows, use `./mvnw.cmd`.
PowerShell:
```powershell
./mvnw.cmd -o '-Dqqq.snapshot.version=4.1.0-FG-SNAPSHOT' verify
```
The tests use H2 from the adopter DDL, a strict UI API server, generated in-memory SSH keys and
real MINA clients. They exercise permissions, auth failures, lockout, competing listeners,
revocation/refresh, pagination/caching, audit, key validation, contingent pickers, API isolation,
metadata registration and record serialization. Checkstyle and JaCoCo gates require 70%
instruction coverage and 90% class coverage.
Surefire isolates `user.home` under `target/fg-home` and `java.io.tmpdir` under `target/fg-tmp`.
The disk guard creates those directories before tests, compares working-tree and home snapshots,
and requires an empty temp directory after each protocol test. Build output under `target/`
is excluded from the working-tree snapshot.
## License
+1 -1
View File
@@ -390,7 +390,7 @@ Everything is streamed. Nothing is written to disk.
- **Lockout.**
- 5 failed auths within 15 minutes, per username or per IP, lock that username or IP for 30
minutes. All three numbers are configurable.
- A lock sets `locked_until` and raises an alert (`file_gateway_lockout`, `QAlerts.raise`).
- A lock sets `locked_until` and raises an alert (`filegateway.lockout`, `QAlerts.raise`; core alert-type names use dots, not the table prefix).
- The record action "Unlock account" clears it.
- Lockout state lives in the database, so it holds across pods.
- A per-IP limit on new connections (30 per minute) applies before auth.
+163
View File
@@ -0,0 +1,163 @@
-- File Gateway schema. PostgreSQL and H2. Timestamps are UTC, matching QQQ RDBMS binding.
CREATE TABLE file_gateway_account (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
create_date TIMESTAMP,
modify_date TIMESTAMP,
username VARCHAR(1024) NOT NULL UNIQUE,
label VARCHAR(1024),
account_type VARCHAR(1024),
user_id VARCHAR(1024),
owner_user_id VARCHAR(1024),
status VARCHAR(1024),
root_mode VARCHAR(1024),
allowed_cidrs TEXT,
expires_at TIMESTAMP,
formats VARCHAR(1024),
max_concurrent_sessions INTEGER,
max_operations_per_minute INTEGER,
max_file_size_mb INTEGER,
max_upload_mb_per_day INTEGER,
max_download_mb_per_day INTEGER,
dedupe_uploads BOOLEAN,
notify_on_success BOOLEAN,
pgp_public_key TEXT,
pgp_decrypt_uploads BOOLEAN,
pgp_encrypt_downloads BOOLEAN,
operation_window_at TIMESTAMP,
operation_window_count INTEGER,
failed_login_count INTEGER,
failed_login_window_at TIMESTAMP,
locked_until TIMESTAMP,
last_login_at TIMESTAMP,
last_login_ip VARCHAR(1024)
);
CREATE TABLE file_gateway_ssh_key (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
create_date TIMESTAMP,
modify_date TIMESTAMP,
account_id INTEGER,
label VARCHAR(1024),
public_key TEXT,
fingerprint VARCHAR(1024) NOT NULL UNIQUE,
key_type VARCHAR(1024),
key_bits INTEGER,
expires_at TIMESTAMP,
revoked_at TIMESTAMP,
last_used_at TIMESTAMP,
last_used_ip VARCHAR(1024)
);
CREATE TABLE file_gateway_alias (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
create_date TIMESTAMP,
modify_date TIMESTAMP,
name VARCHAR(1024) NOT NULL UNIQUE,
path VARCHAR(1024) NOT NULL UNIQUE,
kind VARCHAR(1024),
account_id INTEGER,
table_name VARCHAR(1024),
saved_bulk_load_profile_id INTEGER,
saved_view_id INTEGER,
saved_report_id INTEGER,
format VARCHAR(1024),
file_name VARCHAR(1024),
consumer_name VARCHAR(1024),
is_active BOOLEAN,
description TEXT
);
CREATE TABLE file_gateway_upload_job (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
create_date TIMESTAMP,
modify_date TIMESTAMP,
account_id INTEGER,
alias_id INTEGER,
connection_log_id INTEGER,
remote_address VARCHAR(1024),
table_name VARCHAR(1024),
saved_bulk_load_profile_id INTEGER,
consumer_name VARCHAR(1024),
consumer_reference VARCHAR(1024),
path VARCHAR(1024),
file_name VARCHAR(1024),
size_bytes BIGINT,
sha256 VARCHAR(1024),
storage_key VARCHAR(1024),
status VARCHAR(1024),
duplicate_of_job_id INTEGER,
process_uuid VARCHAR(1024),
inserted_count INTEGER,
error_count INTEGER,
result_summary TEXT,
results_storage_key VARCHAR(1024),
error_message TEXT,
attempts INTEGER,
next_attempt_at TIMESTAMP,
claimed_by VARCHAR(1024),
claimed_at TIMESTAMP,
queued_at TIMESTAMP,
started_at TIMESTAMP,
finished_at TIMESTAMP,
acknowledged_at TIMESTAMP
);
CREATE TABLE file_gateway_outbox_file (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
create_date TIMESTAMP,
modify_date TIMESTAMP,
alias_id INTEGER,
file_name VARCHAR(1024),
size_bytes BIGINT,
sha256 VARCHAR(1024),
storage_key VARCHAR(1024),
status VARCHAR(1024),
first_downloaded_at TIMESTAMP,
download_count INTEGER,
removed_at TIMESTAMP,
expires_at TIMESTAMP,
created_by_consumer VARCHAR(1024)
);
CREATE TABLE file_gateway_transfer_log (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
create_date TIMESTAMP,
modify_date TIMESTAMP,
account_id INTEGER,
connection_log_id INTEGER,
operation VARCHAR(1024),
path VARCHAR(1024),
object_kind VARCHAR(1024),
object_id VARCHAR(1024),
format VARCHAR(1024),
bytes BIGINT,
sha256 VARCHAR(1024),
duration_ms BIGINT,
outcome VARCHAR(1024),
message TEXT,
client_channel VARCHAR(1024),
upload_job_id INTEGER
);
CREATE TABLE file_gateway_connection_log (
id INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
create_date TIMESTAMP,
modify_date TIMESTAMP,
account_id INTEGER,
username VARCHAR(1024),
remote_ip VARCHAR(1024),
client_version VARCHAR(1024),
key_fingerprint VARCHAR(1024),
outcome VARCHAR(1024),
session_id VARCHAR(1024),
pod_name VARCHAR(1024),
connected_at TIMESTAMP,
last_seen_at TIMESTAMP,
disconnected_at TIMESTAMP,
disconnect_reason TEXT,
bytes_in BIGINT,
bytes_out BIGINT,
operation_count BIGINT,
client_channel VARCHAR(1024)
);
+27
View File
@@ -27,6 +27,7 @@
<maven.compiler.target>21</maven.compiler.target>
<jacoco.version>0.8.15</jacoco.version>
<h2.version>2.2.224</h2.version>
<sshd.version>2.16.0</sshd.version>
</properties>
<parent>
@@ -72,6 +73,22 @@
</scm>
<dependencies>
<!-- MINA's Ed25519 key decoder and signer. -->
<dependency>
<groupId>net.i2p.crypto</groupId>
<artifactId>eddsa</artifactId>
<version>0.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.sshd</groupId>
<artifactId>sshd-core</artifactId>
<version>${sshd.version}</version>
</dependency>
<dependency>
<groupId>org.apache.sshd</groupId>
<artifactId>sshd-sftp</artifactId>
<version>${sshd.version}</version>
</dependency>
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-backend-core</artifactId>
@@ -114,6 +131,16 @@
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<systemPropertyVariables>
<java.io.tmpdir>${project.build.directory}/fg-tmp</java.io.tmpdir>
<user.home>${project.build.directory}/fg-home</user.home>
</systemPropertyVariables>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
@@ -21,46 +21,77 @@
package com.kingsrook.qbits.filegateway;
import java.time.Duration;
import java.util.List;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.producers.MetaDataCustomizerInterface;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitConfig;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Configuration for the file-gateway qbit.
** Configuration shared by the admin application and gateway deployment.
*******************************************************************************/
public class FileGatewayQBitConfig implements QBitConfig
{
private MetaDataCustomizerInterface<QTableMetaData> tableMetaDataCustomizer;
private String backendName = "rdbms";
private boolean sftpEnabled = false;
private int sftpPort = 2222;
private int sftpPort = 2222;
private String hostKeyEnvVarName = "FILE_GATEWAY_SSH_HOST_KEY";
private boolean allowGeneratedHostKey = false;
private String fileGatewayBucketAlias = "file-gateway";
private int uploadPartSizeMb = 8;
private int uploadReorderWindowMb = 64;
private int downloadReadAheadMb = 16;
private int maxGeneratedFileMb = 1024;
private int uploadWorkerThreads = 4;
private Duration dedupeWindow = Duration.ofDays(7);
private Duration jobStaleAfter = Duration.ofMinutes(30);
private List<String> temporaryUploadSuffixes = List.of(".filepart", ".tmp", ".part", ".partial");
private int lockoutThreshold = 5;
private Duration lockoutWindow = Duration.ofMinutes(15);
private Duration lockoutDuration = Duration.ofMinutes(30);
private int maxConnectionsPerIpPerMinute = 30;
private int defaultMaxConcurrentSessions = 4;
private int defaultMaxOperationsPerMinute = 600;
private int defaultMaxFileSizeMb = 100;
private Integer defaultMaxUploadMbPerDay = null;
private Integer defaultMaxDownloadMbPerDay = null;
private QCodeReference scanner = null;
private String pgpPrivateKeyEnvVarName = null;
private String pgpPassphraseEnvVarName = null;
private Duration logRetention = Duration.ofDays(180);
private Duration archiveRetention = Duration.ofDays(30);
private Duration outboxRetention = Duration.ofDays(30);
private Duration resultsRetention = Duration.ofDays(90);
private Duration receivingStaleAfter = Duration.ofHours(1);
private MetaDataCustomizerInterface<QTableMetaData> tableMetaDataCustomizer = null;
/*******************************************************************************
** Getter for the optional table customizer.
** Getter for backendName.
*******************************************************************************/
@Override
public MetaDataCustomizerInterface<QTableMetaData> getTableMetaDataCustomizer()
public String getBackendName()
{
return (tableMetaDataCustomizer);
return (backendName);
}
/*******************************************************************************
** Fluent setter for the optional table customizer.
** Fluent setter for backendName.
*******************************************************************************/
public FileGatewayQBitConfig withTableMetaDataCustomizer(MetaDataCustomizerInterface<QTableMetaData> tableMetaDataCustomizer)
public FileGatewayQBitConfig withBackendName(String backendName)
{
this.tableMetaDataCustomizer = tableMetaDataCustomizer;
this.backendName = backendName;
return (this);
}
/*******************************************************************************
** Whether this process runs the SFTP server (only the gateway deployment does).
** Getter for sftpEnabled.
*******************************************************************************/
public boolean getSftpEnabled()
{
@@ -70,7 +101,7 @@ public class FileGatewayQBitConfig implements QBitConfig
/*******************************************************************************
** Fluent setter for whether this process runs the SFTP server.
** Fluent setter for sftpEnabled.
*******************************************************************************/
public FileGatewayQBitConfig withSftpEnabled(boolean sftpEnabled)
{
@@ -81,7 +112,7 @@ public class FileGatewayQBitConfig implements QBitConfig
/*******************************************************************************
** Getter for the SFTP listen port (default 2222).
** Getter for sftpPort.
*******************************************************************************/
public int getSftpPort()
{
@@ -91,11 +122,620 @@ public class FileGatewayQBitConfig implements QBitConfig
/*******************************************************************************
** Fluent setter for the SFTP listen port.
** Fluent setter for sftpPort.
*******************************************************************************/
public FileGatewayQBitConfig withSftpPort(int sftpPort)
{
this.sftpPort = sftpPort;
return (this);
}
/*******************************************************************************
** Getter for hostKeyEnvVarName.
*******************************************************************************/
public String getHostKeyEnvVarName()
{
return (hostKeyEnvVarName);
}
/*******************************************************************************
** Fluent setter for hostKeyEnvVarName.
*******************************************************************************/
public FileGatewayQBitConfig withHostKeyEnvVarName(String hostKeyEnvVarName)
{
this.hostKeyEnvVarName = hostKeyEnvVarName;
return (this);
}
/*******************************************************************************
** Getter for allowGeneratedHostKey.
*******************************************************************************/
public boolean getAllowGeneratedHostKey()
{
return (allowGeneratedHostKey);
}
/*******************************************************************************
** Fluent setter for allowGeneratedHostKey.
*******************************************************************************/
public FileGatewayQBitConfig withAllowGeneratedHostKey(boolean allowGeneratedHostKey)
{
this.allowGeneratedHostKey = allowGeneratedHostKey;
return (this);
}
/*******************************************************************************
** Getter for fileGatewayBucketAlias.
*******************************************************************************/
public String getFileGatewayBucketAlias()
{
return (fileGatewayBucketAlias);
}
/*******************************************************************************
** Fluent setter for fileGatewayBucketAlias.
*******************************************************************************/
public FileGatewayQBitConfig withFileGatewayBucketAlias(String fileGatewayBucketAlias)
{
this.fileGatewayBucketAlias = fileGatewayBucketAlias;
return (this);
}
/*******************************************************************************
** Getter for uploadPartSizeMb.
*******************************************************************************/
public int getUploadPartSizeMb()
{
return (uploadPartSizeMb);
}
/*******************************************************************************
** Fluent setter for uploadPartSizeMb.
*******************************************************************************/
public FileGatewayQBitConfig withUploadPartSizeMb(int uploadPartSizeMb)
{
this.uploadPartSizeMb = uploadPartSizeMb;
return (this);
}
/*******************************************************************************
** Getter for uploadReorderWindowMb.
*******************************************************************************/
public int getUploadReorderWindowMb()
{
return (uploadReorderWindowMb);
}
/*******************************************************************************
** Fluent setter for uploadReorderWindowMb.
*******************************************************************************/
public FileGatewayQBitConfig withUploadReorderWindowMb(int uploadReorderWindowMb)
{
this.uploadReorderWindowMb = uploadReorderWindowMb;
return (this);
}
/*******************************************************************************
** Getter for downloadReadAheadMb.
*******************************************************************************/
public int getDownloadReadAheadMb()
{
return (downloadReadAheadMb);
}
/*******************************************************************************
** Fluent setter for downloadReadAheadMb.
*******************************************************************************/
public FileGatewayQBitConfig withDownloadReadAheadMb(int downloadReadAheadMb)
{
this.downloadReadAheadMb = downloadReadAheadMb;
return (this);
}
/*******************************************************************************
** Getter for maxGeneratedFileMb.
*******************************************************************************/
public int getMaxGeneratedFileMb()
{
return (maxGeneratedFileMb);
}
/*******************************************************************************
** Fluent setter for maxGeneratedFileMb.
*******************************************************************************/
public FileGatewayQBitConfig withMaxGeneratedFileMb(int maxGeneratedFileMb)
{
this.maxGeneratedFileMb = maxGeneratedFileMb;
return (this);
}
/*******************************************************************************
** Getter for uploadWorkerThreads.
*******************************************************************************/
public int getUploadWorkerThreads()
{
return (uploadWorkerThreads);
}
/*******************************************************************************
** Fluent setter for uploadWorkerThreads.
*******************************************************************************/
public FileGatewayQBitConfig withUploadWorkerThreads(int uploadWorkerThreads)
{
this.uploadWorkerThreads = uploadWorkerThreads;
return (this);
}
/*******************************************************************************
** Getter for dedupeWindow.
*******************************************************************************/
public Duration getDedupeWindow()
{
return (dedupeWindow);
}
/*******************************************************************************
** Fluent setter for dedupeWindow.
*******************************************************************************/
public FileGatewayQBitConfig withDedupeWindow(Duration dedupeWindow)
{
this.dedupeWindow = dedupeWindow;
return (this);
}
/*******************************************************************************
** Getter for jobStaleAfter.
*******************************************************************************/
public Duration getJobStaleAfter()
{
return (jobStaleAfter);
}
/*******************************************************************************
** Fluent setter for jobStaleAfter.
*******************************************************************************/
public FileGatewayQBitConfig withJobStaleAfter(Duration jobStaleAfter)
{
this.jobStaleAfter = jobStaleAfter;
return (this);
}
/*******************************************************************************
** Getter for temporaryUploadSuffixes.
*******************************************************************************/
public List<String> getTemporaryUploadSuffixes()
{
return (temporaryUploadSuffixes);
}
/*******************************************************************************
** Fluent setter for temporaryUploadSuffixes.
*******************************************************************************/
public FileGatewayQBitConfig withTemporaryUploadSuffixes(List<String> temporaryUploadSuffixes)
{
this.temporaryUploadSuffixes = temporaryUploadSuffixes;
return (this);
}
/*******************************************************************************
** Getter for lockoutThreshold.
*******************************************************************************/
public int getLockoutThreshold()
{
return (lockoutThreshold);
}
/*******************************************************************************
** Fluent setter for lockoutThreshold.
*******************************************************************************/
public FileGatewayQBitConfig withLockoutThreshold(int lockoutThreshold)
{
this.lockoutThreshold = lockoutThreshold;
return (this);
}
/*******************************************************************************
** Getter for lockoutWindow.
*******************************************************************************/
public Duration getLockoutWindow()
{
return (lockoutWindow);
}
/*******************************************************************************
** Fluent setter for lockoutWindow.
*******************************************************************************/
public FileGatewayQBitConfig withLockoutWindow(Duration lockoutWindow)
{
this.lockoutWindow = lockoutWindow;
return (this);
}
/*******************************************************************************
** Getter for lockoutDuration.
*******************************************************************************/
public Duration getLockoutDuration()
{
return (lockoutDuration);
}
/*******************************************************************************
** Fluent setter for lockoutDuration.
*******************************************************************************/
public FileGatewayQBitConfig withLockoutDuration(Duration lockoutDuration)
{
this.lockoutDuration = lockoutDuration;
return (this);
}
/*******************************************************************************
** Getter for maxConnectionsPerIpPerMinute.
*******************************************************************************/
public int getMaxConnectionsPerIpPerMinute()
{
return (maxConnectionsPerIpPerMinute);
}
/*******************************************************************************
** Fluent setter for maxConnectionsPerIpPerMinute.
*******************************************************************************/
public FileGatewayQBitConfig withMaxConnectionsPerIpPerMinute(int maxConnectionsPerIpPerMinute)
{
this.maxConnectionsPerIpPerMinute = maxConnectionsPerIpPerMinute;
return (this);
}
/*******************************************************************************
** Getter for defaultMaxConcurrentSessions.
*******************************************************************************/
public int getDefaultMaxConcurrentSessions()
{
return (defaultMaxConcurrentSessions);
}
/*******************************************************************************
** Fluent setter for defaultMaxConcurrentSessions.
*******************************************************************************/
public FileGatewayQBitConfig withDefaultMaxConcurrentSessions(int defaultMaxConcurrentSessions)
{
this.defaultMaxConcurrentSessions = defaultMaxConcurrentSessions;
return (this);
}
/*******************************************************************************
** Getter for defaultMaxOperationsPerMinute.
*******************************************************************************/
public int getDefaultMaxOperationsPerMinute()
{
return (defaultMaxOperationsPerMinute);
}
/*******************************************************************************
** Fluent setter for defaultMaxOperationsPerMinute.
*******************************************************************************/
public FileGatewayQBitConfig withDefaultMaxOperationsPerMinute(int defaultMaxOperationsPerMinute)
{
this.defaultMaxOperationsPerMinute = defaultMaxOperationsPerMinute;
return (this);
}
/*******************************************************************************
** Getter for defaultMaxFileSizeMb.
*******************************************************************************/
public int getDefaultMaxFileSizeMb()
{
return (defaultMaxFileSizeMb);
}
/*******************************************************************************
** Fluent setter for defaultMaxFileSizeMb.
*******************************************************************************/
public FileGatewayQBitConfig withDefaultMaxFileSizeMb(int defaultMaxFileSizeMb)
{
this.defaultMaxFileSizeMb = defaultMaxFileSizeMb;
return (this);
}
/*******************************************************************************
** Getter for defaultMaxUploadMbPerDay.
*******************************************************************************/
public Integer getDefaultMaxUploadMbPerDay()
{
return (defaultMaxUploadMbPerDay);
}
/*******************************************************************************
** Fluent setter for defaultMaxUploadMbPerDay.
*******************************************************************************/
public FileGatewayQBitConfig withDefaultMaxUploadMbPerDay(Integer defaultMaxUploadMbPerDay)
{
this.defaultMaxUploadMbPerDay = defaultMaxUploadMbPerDay;
return (this);
}
/*******************************************************************************
** Getter for defaultMaxDownloadMbPerDay.
*******************************************************************************/
public Integer getDefaultMaxDownloadMbPerDay()
{
return (defaultMaxDownloadMbPerDay);
}
/*******************************************************************************
** Fluent setter for defaultMaxDownloadMbPerDay.
*******************************************************************************/
public FileGatewayQBitConfig withDefaultMaxDownloadMbPerDay(Integer defaultMaxDownloadMbPerDay)
{
this.defaultMaxDownloadMbPerDay = defaultMaxDownloadMbPerDay;
return (this);
}
/*******************************************************************************
** Getter for scanner.
*******************************************************************************/
public QCodeReference getScanner()
{
return (scanner);
}
/*******************************************************************************
** Fluent setter for scanner.
*******************************************************************************/
public FileGatewayQBitConfig withScanner(QCodeReference scanner)
{
this.scanner = scanner;
return (this);
}
/*******************************************************************************
** Getter for pgpPrivateKeyEnvVarName.
*******************************************************************************/
public String getPgpPrivateKeyEnvVarName()
{
return (pgpPrivateKeyEnvVarName);
}
/*******************************************************************************
** Fluent setter for pgpPrivateKeyEnvVarName.
*******************************************************************************/
public FileGatewayQBitConfig withPgpPrivateKeyEnvVarName(String pgpPrivateKeyEnvVarName)
{
this.pgpPrivateKeyEnvVarName = pgpPrivateKeyEnvVarName;
return (this);
}
/*******************************************************************************
** Getter for pgpPassphraseEnvVarName.
*******************************************************************************/
public String getPgpPassphraseEnvVarName()
{
return (pgpPassphraseEnvVarName);
}
/*******************************************************************************
** Fluent setter for pgpPassphraseEnvVarName.
*******************************************************************************/
public FileGatewayQBitConfig withPgpPassphraseEnvVarName(String pgpPassphraseEnvVarName)
{
this.pgpPassphraseEnvVarName = pgpPassphraseEnvVarName;
return (this);
}
/*******************************************************************************
** Getter for logRetention.
*******************************************************************************/
public Duration getLogRetention()
{
return (logRetention);
}
/*******************************************************************************
** Fluent setter for logRetention.
*******************************************************************************/
public FileGatewayQBitConfig withLogRetention(Duration logRetention)
{
this.logRetention = logRetention;
return (this);
}
/*******************************************************************************
** Getter for archiveRetention.
*******************************************************************************/
public Duration getArchiveRetention()
{
return (archiveRetention);
}
/*******************************************************************************
** Fluent setter for archiveRetention.
*******************************************************************************/
public FileGatewayQBitConfig withArchiveRetention(Duration archiveRetention)
{
this.archiveRetention = archiveRetention;
return (this);
}
/*******************************************************************************
** Getter for outboxRetention.
*******************************************************************************/
public Duration getOutboxRetention()
{
return (outboxRetention);
}
/*******************************************************************************
** Fluent setter for outboxRetention.
*******************************************************************************/
public FileGatewayQBitConfig withOutboxRetention(Duration outboxRetention)
{
this.outboxRetention = outboxRetention;
return (this);
}
/*******************************************************************************
** Getter for resultsRetention.
*******************************************************************************/
public Duration getResultsRetention()
{
return (resultsRetention);
}
/*******************************************************************************
** Fluent setter for resultsRetention.
*******************************************************************************/
public FileGatewayQBitConfig withResultsRetention(Duration resultsRetention)
{
this.resultsRetention = resultsRetention;
return (this);
}
/*******************************************************************************
** Getter for receivingStaleAfter.
*******************************************************************************/
public Duration getReceivingStaleAfter()
{
return (receivingStaleAfter);
}
/*******************************************************************************
** Fluent setter for receivingStaleAfter.
*******************************************************************************/
public FileGatewayQBitConfig withReceivingStaleAfter(Duration receivingStaleAfter)
{
this.receivingStaleAfter = receivingStaleAfter;
return (this);
}
/*******************************************************************************
** Getter for tableMetaDataCustomizer.
*******************************************************************************/
public MetaDataCustomizerInterface<QTableMetaData> getTableMetaDataCustomizer()
{
return (tableMetaDataCustomizer);
}
/*******************************************************************************
** Fluent setter for tableMetaDataCustomizer.
*******************************************************************************/
public FileGatewayQBitConfig withTableMetaDataCustomizer(MetaDataCustomizerInterface<QTableMetaData> tableMetaDataCustomizer)
{
this.tableMetaDataCustomizer = tableMetaDataCustomizer;
return (this);
}
}
@@ -21,17 +21,37 @@
package com.kingsrook.qbits.filegateway;
import java.util.List;
import com.kingsrook.qbits.filegateway.metadata.FileGatewayProcesses;
import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables;
import com.kingsrook.qbits.filegateway.permissions.FileGatewayPermissions;
import com.kingsrook.qqq.backend.core.alerts.QAlertTypeMetaData;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppSection;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.ClientChannelsMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaDataProducer;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
/*******************************************************************************
** Entry point for the file-gateway qbit; no tables are produced yet.
** Entry point for the file-gateway metadata; runtime startup belongs to the host.
*******************************************************************************/
public class FileGatewayQBitProducer implements QBitMetaDataProducer<FileGatewayQBitConfig>
{
/*******************************************************************************
** The sftp wall owns its list (core's qbit prefixes are transitional and the
** assistant may now read audits): partners never reach identity, audits,
** webhooks or this qbit's own tables, read or write. Core's generic secret,
** credential, permission and session patterns are added on top.
*******************************************************************************/
public static final java.util.List<String> SFTP_DENIED_PATTERNS = java.util.List.of(
"file_gateway_*", "audit*", "authentik_*", "webhooks_*", "worm_audit_*", "secured_data_*", "assistant_*");
public static final String GROUP_ID = "com.kingsrook.qbits";
public static final String ARTIFACT_ID = "qbit-file-gateway";
public static final String VERSION = "0.1.0";
@@ -108,13 +128,56 @@ public class FileGatewayQBitProducer implements QBitMetaDataProducer<FileGateway
/*******************************************************************************
** Register this qbit and return its metadata, without producing tables.
** Register tables, child lists, permissions, channel walls and the application.
*******************************************************************************/
@Override
public MetaDataProducerMultiOutput produce(QInstance qInstance)
public MetaDataProducerMultiOutput produce(QInstance qInstance) throws QException
{
if(!isEnabled())
{
return (new MetaDataProducerMultiOutput());
}
QBitMetaData qBit = getQBitMetaData();
qInstance.addQBit(qBit);
return (new MetaDataProducerMultiOutput().with(qBit));
MetaDataProducerMultiOutput output = new MetaDataProducerMultiOutput().with(qBit);
FileGatewayTables.possibleValueSources().forEach(output::add);
QAppMetaData app = new QAppMetaData().withName("file_gateway").withLabel("File Gateway")
.withIcon(new QIcon("folder_shared")).withPermissionRules(FileGatewayTables.permissionRules("file_gateway"));
for(QTableMetaData table : FileGatewayTables.tables(qBitConfig))
{
if(qBitConfig.getTableMetaDataCustomizer() != null)
{
table = qBitConfig.getTableMetaDataCustomizer().customizeMetaData(qInstance, table);
}
table.setSourceQBitName(qBit.getName());
output.add(table);
if(FileGatewayTables.MAIN.contains(table.getName()))
{
app.withChild(table);
}
}
app.withSection(new QAppSection().withName("configuration").withLabel("Configuration")
.withTables(List.of(FileGatewayTables.ACCOUNT, FileGatewayTables.ALIAS)));
app.withSection(new QAppSection().withName("activity").withLabel("Activity")
.withTables(List.of(FileGatewayTables.UPLOAD_JOB, FileGatewayTables.TRANSFER_LOG, FileGatewayTables.CONNECTION_LOG)));
FileGatewayTables.joins().forEach(output::add);
FileGatewayTables.childWidgets().forEach(output::add);
output.add(app);
FileGatewayProcesses.processes().forEach(output::add);
output.add(new QAlertTypeMetaData().withName("filegateway.lockout").withLabel("File Gateway Lockout")
.withSourceQBitName(qBit.getName()).withSubjectTableName(FileGatewayTables.ACCOUNT)
.withVisibilityPermission(FileGatewayPermissions.VIEW));
output.addSelfToInstance(qInstance);
ClientChannelsMetaData channels = ClientChannelsMetaData.ofOrWithNew(qInstance)
.withChannel("sftp", "SFTP file gateway").withDeniedPattern("sftp", "file_gateway_*")
.withDeniedPattern("assistant", "file_gateway_*");
for(String pattern : SFTP_DENIED_PATTERNS)
{
channels.withDeniedPattern("sftp", pattern);
}
for(String pattern : ClientChannelsMetaData.ASSISTANT_GENERIC_DENIED_PATTERNS)
{
channels.withDeniedPattern("sftp", pattern);
}
return (new MetaDataProducerMultiOutput());
}
}
@@ -0,0 +1,168 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.security.KeyPair;
import java.time.Clock;
import java.time.Duration;
import java.util.function.Function;
import com.kingsrook.qbits.filegateway.sftp.SftpProtocolServer;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.middleware.javalin.QApplicationJavalinServer;
import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInvoker;
import org.apache.sshd.common.NamedResource;
import org.apache.sshd.common.config.keys.KeyUtils;
import org.apache.sshd.common.util.security.SecurityUtils;
/*******************************************************************************
** Host-started SFTP runtime. Host private keys are parsed in memory and never persisted.
*******************************************************************************/
public final class FileGatewayRuntime implements AutoCloseable
{
private static final QLogger LOG = QLogger.getLogger(FileGatewayRuntime.class);
private final UiApiInvoker invoker;
private final SftpProtocolServer protocol;
/*******************************************************************************
** Bind the host UI API server to the SFTP channel.
*******************************************************************************/
private FileGatewayRuntime(QInstance instance, QApplicationJavalinServer server, FileGatewayQBitConfig config, KeyPair key, Clock clock, Duration interval, String podName)
{
invoker = UiApiInvoker.forServer(server, "sftp");
protocol = new SftpProtocolServer(instance, config, invoker, key, clock, interval, podName);
}
/*******************************************************************************
** Start only when the registered qbit configuration enables SFTP.
*******************************************************************************/
public static FileGatewayRuntime start(QInstance instance, QApplicationJavalinServer server) throws Exception
{
return (start(instance, server, System::getenv));
}
/*******************************************************************************
** Start with an injectable environment reader.
*******************************************************************************/
public static FileGatewayRuntime start(QInstance instance, QApplicationJavalinServer server, Function<String, String> environment) throws Exception
{
var qbit = instance.getQBits().get(FileGatewayQBitProducer.GROUP_ID + ":" + FileGatewayQBitProducer.ARTIFACT_ID);
if(qbit == null || !(qbit.getConfig() instanceof FileGatewayQBitConfig config) || !config.getSftpEnabled())
{
return (null);
}
FileGatewayRuntime runtime = new FileGatewayRuntime(instance, server, config, hostKey(config, environment), Clock.systemUTC(), Duration.ofSeconds(60), environment.apply("HOSTNAME"));
runtime.protocol.start();
return (runtime);
}
/*******************************************************************************
** Start with an explicit ephemeral port, in-memory host key and controllable clock.
*******************************************************************************/
public static FileGatewayRuntime startForTest(QInstance instance, QApplicationJavalinServer server, FileGatewayQBitConfig config, KeyPair key, Clock clock, Duration pollInterval) throws IOException
{
FileGatewayRuntime runtime = new FileGatewayRuntime(instance, server, config, key, clock, pollInterval, "test");
runtime.protocol.start();
return (runtime);
}
/*******************************************************************************
** Read an OpenSSH private key directly from memory.
*******************************************************************************/
public static KeyPair hostKey(FileGatewayQBitConfig config, Function<String, String> environment) throws Exception
{
String pem = environment.apply(config.getHostKeyEnvVarName());
if(pem == null || pem.isBlank())
{
if(!config.getAllowGeneratedHostKey())
{
throw (new IllegalStateException("SFTP host key environment variable is missing: " + config.getHostKeyEnvVarName()));
}
LOG.warn("Using a generated in-memory SFTP host key; do not use in production");
return (KeyUtils.generateKeyPair("ecdsa-sha2-nistp256", 256));
}
try(var stream = new ByteArrayInputStream(pem.getBytes(StandardCharsets.UTF_8)))
{
var keys = SecurityUtils.loadKeyPairIdentities(null, NamedResource.ofName("environment-host-key"), stream, null).iterator();
if(!keys.hasNext())
{
throw (new IllegalArgumentException("No SFTP host key in environment variable"));
}
return (keys.next());
}
}
/*******************************************************************************
** Stop the SFTP runtime.
*******************************************************************************/
public void stop() throws IOException
{
protocol.stop();
}
/*******************************************************************************
** Close this runtime.
*******************************************************************************/
@Override
public void close() throws IOException
{
stop();
}
/*******************************************************************************
** Access the protocol listener.
*******************************************************************************/
public SftpProtocolServer getProtocol()
{
return (protocol);
}
/*******************************************************************************
** The fixed sftp-channel invoker.
*******************************************************************************/
public UiApiInvoker getInvoker()
{
return (invoker);
}
}
@@ -0,0 +1,127 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.api;
import java.time.Duration;
/*******************************************************************************
** Consumer decision, including its reference or reason and retry delay.
*******************************************************************************/
public final class FileAcceptance
{
/*******************************************************************************
** Delivery decisions.
*******************************************************************************/
public enum Status
{
ACCEPTED, REJECTED, RETRY_LATER
}
private final Status status;
private final String consumerReference;
private final String reason;
private final Duration retryAfter;
/*******************************************************************************
** Create an immutable decision.
*******************************************************************************/
private FileAcceptance(Status status, String consumerReference, String reason, Duration retryAfter)
{
this.status = status;
this.consumerReference = consumerReference;
this.reason = reason;
this.retryAfter = retryAfter;
}
/*******************************************************************************
** Accept the delivery.
*******************************************************************************/
public static FileAcceptance accepted(String consumerReference)
{
return (new FileAcceptance(Status.ACCEPTED, consumerReference, null, null));
}
/*******************************************************************************
** Reject the delivery.
*******************************************************************************/
public static FileAcceptance rejected(String reason)
{
return (new FileAcceptance(Status.REJECTED, null, reason, null));
}
/*******************************************************************************
** Request a retry.
*******************************************************************************/
public static FileAcceptance retryLater(String reason, Duration retryAfter)
{
return (new FileAcceptance(Status.RETRY_LATER, null, reason, retryAfter));
}
/*******************************************************************************
** Getter for status.
*******************************************************************************/
public Status getStatus()
{
return (status);
}
/*******************************************************************************
** Getter for consumerReference.
*******************************************************************************/
public String getConsumerReference()
{
return (consumerReference);
}
/*******************************************************************************
** Getter for reason.
*******************************************************************************/
public String getReason()
{
return (reason);
}
/*******************************************************************************
** Getter for retryAfter.
*******************************************************************************/
public Duration getRetryAfter()
{
return (retryAfter);
}
}
@@ -0,0 +1,49 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.api;
/*******************************************************************************
** Consumer SPI for at-least-once inbound delivery and download notification.
*******************************************************************************/
public interface FileGatewayConsumer
{
/*******************************************************************************
** Stable lowercase consumer name.
*******************************************************************************/
String getName();
/*******************************************************************************
** Accept, reject or request a later retry.
*******************************************************************************/
FileAcceptance accept(InboundFile file) throws Exception;
/*******************************************************************************
** Called after the first complete download.
*******************************************************************************/
default void onDownloaded(OutboxFileRef ref) throws Exception
{
}
}
@@ -0,0 +1,79 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.api;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.QSupplementalInstanceMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
/*******************************************************************************
** Host-declared consumer implementations; contains no gateway runtime dependencies.
*******************************************************************************/
public class FileGatewayConsumersMetaData implements QSupplementalInstanceMetaData
{
public static final String NAME = "fileGatewayConsumers";
private final List<QCodeReference> consumers = new ArrayList<>();
/*******************************************************************************
** Metadata registry name.
*******************************************************************************/
public String getName()
{
return (NAME);
}
/*******************************************************************************
** Get or install the registry.
*******************************************************************************/
public static FileGatewayConsumersMetaData ofOrWithNew(QInstance instance)
{
return (QSupplementalInstanceMetaData.ofOrWithNew(instance, NAME, FileGatewayConsumersMetaData::new));
}
/*******************************************************************************
** Register a consumer implementation.
*******************************************************************************/
public FileGatewayConsumersMetaData withConsumer(QCodeReference consumer)
{
consumers.add(consumer);
return (this);
}
/*******************************************************************************
** Registered consumer implementations.
*******************************************************************************/
public List<QCodeReference> getConsumers()
{
return (List.copyOf(consumers));
}
}
@@ -0,0 +1,78 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.api;
import java.io.InputStream;
import com.kingsrook.qqq.backend.core.exceptions.QException;
/*******************************************************************************
** Optional outbox facade; the host runtime owns provider registration.
*******************************************************************************/
public final class FileGatewayOutbox
{
private static volatile FileGatewayOutboxProvider provider;
/*******************************************************************************
** Static facade.
*******************************************************************************/
private FileGatewayOutbox()
{
}
/*******************************************************************************
** Install or clear the runtime provider.
*******************************************************************************/
public static void setProvider(FileGatewayOutboxProvider provider)
{
FileGatewayOutbox.provider = provider;
}
/*******************************************************************************
** Whether the runtime supports outbox writes.
*******************************************************************************/
public static boolean isAvailable()
{
return (provider != null);
}
/*******************************************************************************
** Delegate to the installed provider.
*******************************************************************************/
public static OutboxFileRef put(String endpointName, String fileName, InputStream content, long sizeBytes) throws FileGatewayUnavailableException, QException
{
FileGatewayOutboxProvider current = provider;
if(current == null)
{
throw (new FileGatewayUnavailableException());
}
return (current.put(endpointName, fileName, content, sizeBytes));
}
}
@@ -0,0 +1,37 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.api;
import java.io.InputStream;
import com.kingsrook.qqq.backend.core.exceptions.QException;
/*******************************************************************************
** Runtime implementation registered with the outbox facade.
*******************************************************************************/
public interface FileGatewayOutboxProvider
{
/*******************************************************************************
** Stream content into the endpoint outbox.
*******************************************************************************/
OutboxFileRef put(String endpointName, String fileName, InputStream content, long sizeBytes) throws QException;
}
@@ -0,0 +1,41 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.api;
import com.kingsrook.qqq.backend.core.exceptions.QException;
/*******************************************************************************
** The outbox runtime has not been installed.
*******************************************************************************/
public class FileGatewayUnavailableException extends QException
{
/*******************************************************************************
** Create the unavailable error.
*******************************************************************************/
public FileGatewayUnavailableException()
{
super("File gateway outbox is unavailable");
}
}
@@ -0,0 +1,45 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.api;
import java.io.InputStream;
import java.time.Instant;
import java.util.function.Supplier;
/*******************************************************************************
** An inbound object; each openStream call supplies a fresh stream.
*******************************************************************************/
public record InboundFile(Integer uploadJobId, String endpointName, String username, String userId,
String remoteAddress, String path, String fileName, long sizeBytes, String sha256,
Instant receivedAt, String storageBucketAlias, String storageKey, Supplier<InputStream> streamSupplier)
{
/*******************************************************************************
** Open a fresh object-storage stream.
*******************************************************************************/
public InputStream openStream()
{
return (streamSupplier.get());
}
}
@@ -0,0 +1,33 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.api;
import java.time.Instant;
/*******************************************************************************
** Identity and metadata of a file waiting for partner pickup.
*******************************************************************************/
public record OutboxFileRef(Integer outboxFileId, String endpointName, String path, String fileName,
long sizeBytes, String sha256, Instant createdAt)
{
}
@@ -0,0 +1,57 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.metadata;
import java.util.List;
import java.util.Optional;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.statusmessages.BadInputStatusMessage;
/*******************************************************************************
** Reject account names that cannot be used as stable SSH login names.
*******************************************************************************/
public class AccountCustomizer implements TableCustomizerInterface
{
/*******************************************************************************
** Partial updates preserve the current username; supplied names must be valid.
*******************************************************************************/
@Override
public List<QRecord> preInsertOrUpdate(AbstractActionInput input, List<QRecord> records, boolean preview, Optional<List<QRecord>> oldRecords)
{
for(QRecord record : records)
{
if(record.getValues().containsKey("username") && (record.getValueString("username") == null || !record.getValueString("username").matches("[a-z0-9._-]{3,64}")))
{
record.addError(new BadInputStatusMessage("Username must be 3 to 64 lowercase letters, digits, dots, underscores or hyphens"));
}
for(String limit : List.of("maxConcurrentSessions", "maxOperationsPerMinute", "maxFileSizeMb", "maxUploadMbPerDay", "maxDownloadMbPerDay"))
{
if(record.getValueInteger(limit) != null && record.getValueInteger(limit) < 1)
{
record.addError(new BadInputStatusMessage(limit + " must be positive, or blank for an unlimited daily quota"));
}
}
}
return (records);
}
}
@@ -0,0 +1,508 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.metadata;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.PossibleValueEnum;
/*******************************************************************************
** Stable persisted values for the gateway schema.
*******************************************************************************/
public final class FileGatewayEnums
{
/*******************************************************************************
** Namespace for schema values.
*******************************************************************************/
private FileGatewayEnums()
{
}
/*******************************************************************************
** AccountType values.
*******************************************************************************/
public enum AccountType implements PossibleValueEnum<String>
{
USER("USER"), SERVICE("SERVICE");
private final String value;
/*******************************************************************************
** Construct a persisted value.
*******************************************************************************/
AccountType(String value)
{
this.value = value;
}
/*******************************************************************************
** Possible value id.
*******************************************************************************/
@Override
public String getPossibleValueId()
{
return (value);
}
/*******************************************************************************
** Possible value label.
*******************************************************************************/
@Override
public String getPossibleValueLabel()
{
return (value);
}
}
/*******************************************************************************
** AccountStatus values.
*******************************************************************************/
public enum AccountStatus implements PossibleValueEnum<String>
{
ACTIVE("ACTIVE"), DISABLED("DISABLED"), LOCKED("LOCKED");
private final String value;
/*******************************************************************************
** Construct a persisted value.
*******************************************************************************/
AccountStatus(String value)
{
this.value = value;
}
/*******************************************************************************
** Possible value id.
*******************************************************************************/
@Override
public String getPossibleValueId()
{
return (value);
}
/*******************************************************************************
** Possible value label.
*******************************************************************************/
@Override
public String getPossibleValueLabel()
{
return (value);
}
}
/*******************************************************************************
** RootMode values.
*******************************************************************************/
public enum RootMode implements PossibleValueEnum<String>
{
FULL_TREE("FULL_TREE"), ENDPOINTS_ONLY("ENDPOINTS_ONLY");
private final String value;
/*******************************************************************************
** Construct a persisted value.
*******************************************************************************/
RootMode(String value)
{
this.value = value;
}
/*******************************************************************************
** Possible value id.
*******************************************************************************/
@Override
public String getPossibleValueId()
{
return (value);
}
/*******************************************************************************
** Possible value label.
*******************************************************************************/
@Override
public String getPossibleValueLabel()
{
return (value);
}
}
/*******************************************************************************
** AliasKind values.
*******************************************************************************/
public enum AliasKind implements PossibleValueEnum<String>
{
BULK_LOAD("BULK_LOAD"), INBOX("INBOX"), VIEW("VIEW"), REPORT("REPORT"), OUTBOX("OUTBOX");
private final String value;
/*******************************************************************************
** Construct a persisted value.
*******************************************************************************/
AliasKind(String value)
{
this.value = value;
}
/*******************************************************************************
** Possible value id.
*******************************************************************************/
@Override
public String getPossibleValueId()
{
return (value);
}
/*******************************************************************************
** Possible value label.
*******************************************************************************/
@Override
public String getPossibleValueLabel()
{
return (value);
}
}
/*******************************************************************************
** UploadStatus values.
*******************************************************************************/
public enum UploadStatus implements PossibleValueEnum<String>
{
RECEIVING("RECEIVING"), QUEUED("QUEUED"), RUNNING("RUNNING"), SUCCEEDED("SUCCEEDED"), PARTIAL("PARTIAL"), FAILED("FAILED"), REJECTED("REJECTED"), DUPLICATE("DUPLICATE"), ABORTED("ABORTED");
private final String value;
/*******************************************************************************
** Construct a persisted value.
*******************************************************************************/
UploadStatus(String value)
{
this.value = value;
}
/*******************************************************************************
** Possible value id.
*******************************************************************************/
@Override
public String getPossibleValueId()
{
return (value);
}
/*******************************************************************************
** Possible value label.
*******************************************************************************/
@Override
public String getPossibleValueLabel()
{
return (value);
}
}
/*******************************************************************************
** OutboxStatus values.
*******************************************************************************/
public enum OutboxStatus implements PossibleValueEnum<String>
{
READY("READY"), DOWNLOADED("DOWNLOADED"), REMOVED("REMOVED"), EXPIRED("EXPIRED");
private final String value;
/*******************************************************************************
** Construct a persisted value.
*******************************************************************************/
OutboxStatus(String value)
{
this.value = value;
}
/*******************************************************************************
** Possible value id.
*******************************************************************************/
@Override
public String getPossibleValueId()
{
return (value);
}
/*******************************************************************************
** Possible value label.
*******************************************************************************/
@Override
public String getPossibleValueLabel()
{
return (value);
}
}
/*******************************************************************************
** Operation values.
*******************************************************************************/
public enum Operation implements PossibleValueEnum<String>
{
LIST("LIST"), UPLOAD("UPLOAD"), DOWNLOAD("DOWNLOAD"), RENAME("RENAME"), REMOVE("REMOVE");
private final String value;
/*******************************************************************************
** Construct a persisted value.
*******************************************************************************/
Operation(String value)
{
this.value = value;
}
/*******************************************************************************
** Possible value id.
*******************************************************************************/
@Override
public String getPossibleValueId()
{
return (value);
}
/*******************************************************************************
** Possible value label.
*******************************************************************************/
@Override
public String getPossibleValueLabel()
{
return (value);
}
}
/*******************************************************************************
** ObjectKind values.
*******************************************************************************/
public enum ObjectKind implements PossibleValueEnum<String>
{
APP("APP"), TABLE("TABLE"), PROFILE("PROFILE"), VIEW("VIEW"), REPORT("REPORT"), RESULT("RESULT"), ARCHIVE("ARCHIVE"), ENDPOINT("ENDPOINT"), OUTBOX("OUTBOX");
private final String value;
/*******************************************************************************
** Construct a persisted value.
*******************************************************************************/
ObjectKind(String value)
{
this.value = value;
}
/*******************************************************************************
** Possible value id.
*******************************************************************************/
@Override
public String getPossibleValueId()
{
return (value);
}
/*******************************************************************************
** Possible value label.
*******************************************************************************/
@Override
public String getPossibleValueLabel()
{
return (value);
}
}
/*******************************************************************************
** TransferOutcome values.
*******************************************************************************/
public enum TransferOutcome implements PossibleValueEnum<String>
{
OK("OK"), DENIED("DENIED"), NOT_FOUND("NOT_FOUND"), LIMIT("LIMIT"), ERROR("ERROR");
private final String value;
/*******************************************************************************
** Construct a persisted value.
*******************************************************************************/
TransferOutcome(String value)
{
this.value = value;
}
/*******************************************************************************
** Possible value id.
*******************************************************************************/
@Override
public String getPossibleValueId()
{
return (value);
}
/*******************************************************************************
** Possible value label.
*******************************************************************************/
@Override
public String getPossibleValueLabel()
{
return (value);
}
}
/*******************************************************************************
** ConnectionOutcome values.
*******************************************************************************/
public enum ConnectionOutcome implements PossibleValueEnum<String>
{
CONNECTED("CONNECTED"), AUTH_FAILED("AUTH_FAILED"), LOCKED_OUT("LOCKED_OUT"), IP_DENIED("IP_DENIED"), EXPIRED("EXPIRED"), DISABLED("DISABLED"), LIMIT("LIMIT");
private final String value;
/*******************************************************************************
** Construct a persisted value.
*******************************************************************************/
ConnectionOutcome(String value)
{
this.value = value;
}
/*******************************************************************************
** Possible value id.
*******************************************************************************/
@Override
public String getPossibleValueId()
{
return (value);
}
/*******************************************************************************
** Possible value label.
*******************************************************************************/
@Override
public String getPossibleValueLabel()
{
return (value);
}
}
/*******************************************************************************
** Format values.
*******************************************************************************/
public enum Format implements PossibleValueEnum<String>
{
CSV("csv"), XLSX("xlsx"), JSON("json");
private final String value;
/*******************************************************************************
** Construct a persisted value.
*******************************************************************************/
Format(String value)
{
this.value = value;
}
/*******************************************************************************
** Possible value id.
*******************************************************************************/
@Override
public String getPossibleValueId()
{
return (value);
}
/*******************************************************************************
** Possible value label.
*******************************************************************************/
@Override
public String getPossibleValueLabel()
{
return (value);
}
}
/*******************************************************************************
** KeyType values.
*******************************************************************************/
public enum KeyType implements PossibleValueEnum<String>
{
SSH_ED25519("ssh-ed25519"), ECDSA_SHA2_NISTP256("ecdsa-sha2-nistp256"), ECDSA_SHA2_NISTP384("ecdsa-sha2-nistp384"), ECDSA_SHA2_NISTP521("ecdsa-sha2-nistp521"), SSH_RSA("ssh-rsa");
private final String value;
/*******************************************************************************
** Construct a persisted value.
*******************************************************************************/
KeyType(String value)
{
this.value = value;
}
/*******************************************************************************
** Possible value id.
*******************************************************************************/
@Override
public String getPossibleValueId()
{
return (value);
}
/*******************************************************************************
** Possible value label.
*******************************************************************************/
@Override
public String getPossibleValueLabel()
{
return (value);
}
}
}
@@ -0,0 +1,207 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.metadata;
import java.io.Serializable;
import java.time.Instant;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import com.kingsrook.qbits.filegateway.permissions.FileGatewayPermissions;
import com.kingsrook.qbits.filegateway.sftp.AuditWriter;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLine;
import com.kingsrook.qqq.backend.core.model.actions.processes.ProcessSummaryLineInterface;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.actions.processes.Status;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractLoadStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractTransformStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ExtractViaQueryStep;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.ProcessSummaryProviderInterface;
import com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.StreamedETLWithFrontendProcess;
/*******************************************************************************
** Administrative record actions use the standard streamed Review and Result flow.
*******************************************************************************/
public final class FileGatewayProcesses
{
public static final String UNLOCK = "file_gateway_unlock_account";
public static final String DISABLE = "file_gateway_disable_account";
public static final String REVOKE = "file_gateway_revoke_key";
public static final String END_SESSION = "file_gateway_end_session";
/*******************************************************************************
** Metadata factory only.
*******************************************************************************/
private FileGatewayProcesses()
{
}
/*******************************************************************************
** Phase-a actions; retrying uploads belongs to the phase-b job worker.
*******************************************************************************/
public static List<QProcessMetaData> processes()
{
return (List.of(action(UNLOCK, "Unlock account", FileGatewayTables.ACCOUNT),
action(DISABLE, "Disable account", FileGatewayTables.ACCOUNT),
action(REVOKE, "Revoke key", FileGatewayTables.SSH_KEY),
action(END_SESSION, "End session", FileGatewayTables.CONNECTION_LOG)));
}
/*******************************************************************************
** A table-bound action, never a navigation entry.
*******************************************************************************/
private static QProcessMetaData action(String name, String label, String table)
{
Map<String, Serializable> values = new HashMap<>();
values.put(StreamedETLWithFrontendProcess.FIELD_SOURCE_TABLE, table);
values.put(StreamedETLWithFrontendProcess.FIELD_DESTINATION_TABLE, table);
values.put(StreamedETLWithFrontendProcess.FIELD_SUPPORTS_FULL_VALIDATION, true);
values.put(StreamedETLWithFrontendProcess.FIELD_DO_FULL_VALIDATION, true);
values.put(StreamedETLWithFrontendProcess.FIELD_PREVIEW_MESSAGE, label + " for the selected records.");
return (StreamedETLWithFrontendProcess.defineProcessMetaData(ExtractViaQueryStep.class, Review.class, Apply.class, values)
.withName(name).withLabel(label).withTableName(table).withPermissionRules(FileGatewayTables.permissionRules(name)));
}
/*******************************************************************************
** Direct step invocation must require the same permission as the process route.
*******************************************************************************/
private static void requireAdmin() throws QPermissionDeniedException
{
if(QContext.getQSession() == null || !QContext.getQSession().hasPermission(FileGatewayPermissions.ADMIN))
{
throw (new QPermissionDeniedException("Missing file_gateway.admin"));
}
}
/*******************************************************************************
** Preview selections without changing records.
*******************************************************************************/
public static class Review extends AbstractTransformStep implements ProcessSummaryProviderInterface
{
private final ProcessSummaryLine summary = new ProcessSummaryLine(Status.OK, 0, null);
/*******************************************************************************
** Review uses the framework's future and past tense summary lines.
*******************************************************************************/
public Review()
{
summary.setSingularFutureMessage("record will be updated");
summary.setPluralFutureMessage("records will be updated");
summary.setSingularPastMessage("record was updated");
summary.setPluralPastMessage("records were updated");
}
/*******************************************************************************
** Pass only authorized selections to the load phase.
*******************************************************************************/
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
requireAdmin();
for(QRecord record : input.getRecords())
{
summary.incrementCountAndAddPrimaryKey(record.getValue("id"));
output.getRecords().add(record);
}
}
/*******************************************************************************
** Standard Review and Result summary.
*******************************************************************************/
@Override
public ArrayList<ProcessSummaryLineInterface> getProcessSummary(RunBackendStepOutput output, boolean result)
{
return (new ArrayList<>(List.of(summary)));
}
}
/*******************************************************************************
** Apply the approved action to the gateway's own bookkeeping tables.
*******************************************************************************/
public static class Apply extends AbstractLoadStep
{
/*******************************************************************************
** Whitelist mutations, using sparse updates so concurrent account changes survive.
*******************************************************************************/
@Override
public void runOnePage(RunBackendStepInput input, RunBackendStepOutput output) throws QException
{
requireAdmin();
AuditWriter audit = new AuditWriter(QContext.getQInstance());
for(QRecord record : input.getRecords())
{
QRecord update = new QRecord().withValue("id", record.getValue("id"));
String table;
switch(input.getProcessName())
{
case UNLOCK ->
{
table = FileGatewayTables.ACCOUNT;
update.withValue("lockedUntil", null).withValue("failedLoginCount", 0).withValue("failedLoginWindowAt", null);
if("LOCKED".equals(record.getValueString("status")))
{
update.withValue("status", "ACTIVE");
}
}
case DISABLE ->
{
table = FileGatewayTables.ACCOUNT;
update.withValue("status", "DISABLED");
}
case REVOKE ->
{
table = FileGatewayTables.SSH_KEY;
update.withValue("revokedAt", Instant.now());
}
case END_SESSION ->
{
table = FileGatewayTables.CONNECTION_LOG;
update.withValue("disconnectedAt", Instant.now()).withValue("disconnectReason", "Ended by administrator");
}
default -> throw (new QException("Unsupported gateway action"));
}
audit.update(table, update);
output.getRecords().add(update);
}
}
}
}
@@ -0,0 +1,431 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.metadata;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qbits.filegateway.FileGatewayQBitConfig;
import com.kingsrook.qbits.filegateway.permissions.FileGatewayPermissionChecker;
import com.kingsrook.qqq.backend.core.actions.ai.AiFieldMetaData;
import com.kingsrook.qqq.backend.core.actions.ai.DataClassification;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizers;
import com.kingsrook.qqq.backend.core.actions.dashboard.widgets.ChildRecordListRenderer;
import com.kingsrook.qqq.backend.core.instances.QInstanceEnricher;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.dashboard.QWidgetMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinOn;
import com.kingsrook.qqq.backend.core.model.metadata.joins.JoinType;
import com.kingsrook.qqq.backend.core.model.metadata.joins.QJoinMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QIcon;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.possiblevalues.QPossibleValueSource;
import com.kingsrook.qqq.backend.core.model.metadata.tables.Capability;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.SectionFactory;
import com.kingsrook.qqq.backend.core.model.metadata.tables.UniqueKey;
/*******************************************************************************
** The seven gateway tables, their stable columns, pickers and child lists.
*******************************************************************************/
public final class FileGatewayTables
{
public static final String ACCOUNT = "file_gateway_account";
public static final String SSH_KEY = "file_gateway_ssh_key";
public static final String ALIAS = "file_gateway_alias";
public static final String UPLOAD_JOB = "file_gateway_upload_job";
public static final String OUTBOX_FILE = "file_gateway_outbox_file";
public static final String TRANSFER_LOG = "file_gateway_transfer_log";
public static final String CONNECTION_LOG = "file_gateway_connection_log";
public static final List<String> ALL = List.of(ACCOUNT, SSH_KEY, ALIAS, UPLOAD_JOB, OUTBOX_FILE, TRANSFER_LOG, CONNECTION_LOG);
public static final List<String> MAIN = List.of(ACCOUNT, ALIAS, UPLOAD_JOB, TRANSFER_LOG, CONNECTION_LOG);
public static final List<String> SYSTEM_WRITTEN = List.of(UPLOAD_JOB, OUTBOX_FILE, TRANSFER_LOG, CONNECTION_LOG);
/*******************************************************************************
** Metadata factories only.
*******************************************************************************/
private FileGatewayTables()
{
}
/*******************************************************************************
** Permission names use the full table or app name.
*******************************************************************************/
public static QPermissionRules permissionRules(String name)
{
return (new QPermissionRules().withLevel(PermissionLevel.READ_WRITE_PERMISSIONS).withPermissionBaseName(name)
.withCustomPermissionChecker(new QCodeReference(FileGatewayPermissionChecker.class)));
}
/*******************************************************************************
** Enums and gateway record pickers.
*******************************************************************************/
public static List<QPossibleValueSource> possibleValueSources()
{
List<QPossibleValueSource> sources = new ArrayList<>();
sources.add(QPossibleValueSource.newForEnum("file_gateway_account_type", FileGatewayEnums.AccountType.values()));
sources.add(QPossibleValueSource.newForEnum("file_gateway_account_status", FileGatewayEnums.AccountStatus.values()));
sources.add(QPossibleValueSource.newForEnum("file_gateway_root_mode", FileGatewayEnums.RootMode.values()));
sources.add(QPossibleValueSource.newForEnum("file_gateway_alias_kind", FileGatewayEnums.AliasKind.values()));
sources.add(QPossibleValueSource.newForEnum("file_gateway_upload_status", FileGatewayEnums.UploadStatus.values()));
sources.add(QPossibleValueSource.newForEnum("file_gateway_outbox_status", FileGatewayEnums.OutboxStatus.values()));
sources.add(QPossibleValueSource.newForEnum("file_gateway_operation", FileGatewayEnums.Operation.values()));
sources.add(QPossibleValueSource.newForEnum("file_gateway_object_kind", FileGatewayEnums.ObjectKind.values()));
sources.add(QPossibleValueSource.newForEnum("file_gateway_transfer_outcome", FileGatewayEnums.TransferOutcome.values()));
sources.add(QPossibleValueSource.newForEnum("file_gateway_connection_outcome", FileGatewayEnums.ConnectionOutcome.values()));
sources.add(QPossibleValueSource.newForEnum("file_gateway_format", FileGatewayEnums.Format.values()));
sources.add(QPossibleValueSource.newForEnum("file_gateway_key_type", FileGatewayEnums.KeyType.values()));
ALL.forEach(name -> sources.add(QPossibleValueSource.newForTable(name)));
return (sources);
}
/*******************************************************************************
** Build all seven tables.
*******************************************************************************/
public static List<QTableMetaData> tables(FileGatewayQBitConfig config)
{
return (List.of(account(config), ssh_key(config), alias(config), upload_job(config), outbox_file(config), transfer_log(config), connection_log(config)));
}
/*******************************************************************************
** Common identity and timestamps.
*******************************************************************************/
private static QTableMetaData base(FileGatewayQBitConfig config, String name, String label)
{
return (new QTableMetaData().withName(name).withLabel(label).withBackendName(config.getBackendName())
.withPrimaryKeyField("id")
.withPermissionRules(permissionRules(name))
.withField(field("id", QFieldType.INTEGER).withIsEditable(false))
.withField(field("createDate", QFieldType.DATE_TIME).withIsEditable(false))
.withField(field("modifyDate", QFieldType.DATE_TIME).withIsEditable(false)));
}
/*******************************************************************************
** Apply the host backend naming convention explicitly.
*******************************************************************************/
private static QFieldMetaData field(String name, QFieldType type)
{
return (new QFieldMetaData(name, type).withBackendName(QInstanceEnricher.inferBackendName(name)));
}
/*******************************************************************************
** Metadata for accounts.
*******************************************************************************/
private static QTableMetaData account(FileGatewayQBitConfig config)
{
QTableMetaData table = base(config, ACCOUNT, "Accounts")
.withRecordLabelFormatAndFields("%s", "username")
.withField(field("username", QFieldType.STRING).withIsRequired(true))
.withField(field("label", QFieldType.STRING))
.withField(field("accountType", QFieldType.STRING).withPossibleValueSourceName("file_gateway_account_type").withDefaultValue("USER").withIsRequired(true))
.withField(field("userId", QFieldType.STRING).withIsRequired(true))
.withField(field("ownerUserId", QFieldType.STRING))
.withField(field("status", QFieldType.STRING).withPossibleValueSourceName("file_gateway_account_status").withDefaultValue("ACTIVE").withIsRequired(true))
.withField(field("rootMode", QFieldType.STRING).withPossibleValueSourceName("file_gateway_root_mode").withDefaultValue("FULL_TREE").withIsRequired(true))
.withField(field("allowedCidrs", QFieldType.TEXT))
.withField(field("expiresAt", QFieldType.DATE_TIME))
.withField(field("formats", QFieldType.STRING).withDefaultValue("csv,xlsx"))
.withField(field("maxConcurrentSessions", QFieldType.INTEGER).withDefaultValue(config.getDefaultMaxConcurrentSessions()))
.withField(field("maxOperationsPerMinute", QFieldType.INTEGER).withDefaultValue(config.getDefaultMaxOperationsPerMinute()))
.withField(field("maxFileSizeMb", QFieldType.INTEGER).withDefaultValue(config.getDefaultMaxFileSizeMb()))
.withField(field("maxUploadMbPerDay", QFieldType.INTEGER).withDefaultValue(config.getDefaultMaxUploadMbPerDay()))
.withField(field("maxDownloadMbPerDay", QFieldType.INTEGER).withDefaultValue(config.getDefaultMaxDownloadMbPerDay()))
.withField(field("dedupeUploads", QFieldType.BOOLEAN).withDefaultValue(true))
.withField(field("notifyOnSuccess", QFieldType.BOOLEAN).withDefaultValue(false))
.withField(field("pgpPublicKey", QFieldType.TEXT))
.withField(field("pgpDecryptUploads", QFieldType.BOOLEAN).withDefaultValue(false))
.withField(field("pgpEncryptDownloads", QFieldType.BOOLEAN).withDefaultValue(false))
.withField(field("operationWindowAt", QFieldType.DATE_TIME).withIsEditable(false))
.withField(field("operationWindowCount", QFieldType.INTEGER).withDefaultValue(0).withIsEditable(false))
.withField(field("failedLoginCount", QFieldType.INTEGER).withDefaultValue(0).withIsEditable(false))
.withField(field("failedLoginWindowAt", QFieldType.DATE_TIME).withIsEditable(false))
.withField(field("lockedUntil", QFieldType.DATE_TIME).withIsEditable(false))
.withField(field("lastLoginAt", QFieldType.DATE_TIME).withIsEditable(false))
.withField(field("lastLoginIp", QFieldType.STRING).withIsEditable(false));
table.withUniqueKey(new UniqueKey("username"));
table.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(AccountCustomizer.class));
table.withCustomizer(TableCustomizers.PRE_UPDATE_RECORD, new QCodeReference(AccountCustomizer.class));
table.withSection(SectionFactory.defaultT1("id", "username", "label", "accountType", "userId", "ownerUserId"));
table.withSection(SectionFactory.defaultT2("status", "rootMode", "allowedCidrs", "expiresAt", "formats", "maxConcurrentSessions", "maxOperationsPerMinute", "maxFileSizeMb", "maxUploadMbPerDay", "maxDownloadMbPerDay", "dedupeUploads", "notifyOnSuccess", "pgpPublicKey", "pgpDecryptUploads", "pgpEncryptDownloads", "operationWindowAt", "operationWindowCount", "failedLoginCount", "failedLoginWindowAt", "lockedUntil", "lastLoginAt", "lastLoginIp"));
table.withSection(SectionFactory.defaultT3("createDate", "modifyDate"));
table.withSection(SectionFactory.customT2(SSH_KEY, new QIcon("list")).withWidgetName(ACCOUNT + "_children_" + SSH_KEY));
table.withSection(SectionFactory.customT2(UPLOAD_JOB, new QIcon("list")).withWidgetName(ACCOUNT + "_children_" + UPLOAD_JOB));
table.withSection(SectionFactory.customT2(CONNECTION_LOG, new QIcon("list")).withWidgetName(ACCOUNT + "_children_" + CONNECTION_LOG));
return (table);
}
/*******************************************************************************
** Metadata for ssh keys.
*******************************************************************************/
private static QTableMetaData ssh_key(FileGatewayQBitConfig config)
{
QTableMetaData table = base(config, SSH_KEY, "SSH Keys")
.withRecordLabelFormatAndFields("%s", "label")
.withField(field("accountId", QFieldType.INTEGER).withPossibleValueSourceName(ACCOUNT).withIsRequired(true))
.withField(field("label", QFieldType.STRING))
.withField(field("publicKey", QFieldType.TEXT).withIsRequired(true))
.withField(field("fingerprint", QFieldType.STRING).withIsEditable(false))
.withField(field("keyType", QFieldType.STRING).withPossibleValueSourceName("file_gateway_key_type").withIsEditable(false))
.withField(field("keyBits", QFieldType.INTEGER).withIsEditable(false))
.withField(field("expiresAt", QFieldType.DATE_TIME))
.withField(field("revokedAt", QFieldType.DATE_TIME))
.withField(field("lastUsedAt", QFieldType.DATE_TIME).withIsEditable(false))
.withField(field("lastUsedIp", QFieldType.STRING).withIsEditable(false));
table.withUniqueKey(new UniqueKey("fingerprint"));
table.withSection(SectionFactory.defaultT1("id", "accountId", "label", "publicKey", "fingerprint", "keyType"));
table.withSection(SectionFactory.defaultT2("keyBits", "expiresAt", "revokedAt", "lastUsedAt", "lastUsedIp"));
table.withSection(SectionFactory.defaultT3("createDate", "modifyDate"));
table.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(SshKeyCustomizer.class));
table.withCustomizer(TableCustomizers.PRE_UPDATE_RECORD, new QCodeReference(SshKeyCustomizer.class));
table.getField("publicKey").withSupplementalMetaData(new AiFieldMetaData().withClassification(DataClassification.CONFIDENTIAL));
return (table);
}
/*******************************************************************************
** Metadata for endpoints.
*******************************************************************************/
private static QTableMetaData alias(FileGatewayQBitConfig config)
{
QTableMetaData table = base(config, ALIAS, "Endpoints")
.withRecordLabelFormatAndFields("%s", "name")
.withField(field("name", QFieldType.STRING))
.withField(field("path", QFieldType.STRING))
.withField(field("kind", QFieldType.STRING).withPossibleValueSourceName("file_gateway_alias_kind"))
.withField(field("accountId", QFieldType.INTEGER).withPossibleValueSourceName(ACCOUNT))
.withField(field("tableName", QFieldType.STRING).withPossibleValueSourceName("tables"))
.withField(field("savedBulkLoadProfileId", QFieldType.INTEGER).withPossibleValueSourceName("savedBulkLoadProfile").withPossibleValueSourceFilter(new QQueryFilter(new QFilterCriteria("tableName", QCriteriaOperator.EQUALS, "${input.tableName}"))))
.withField(field("savedViewId", QFieldType.INTEGER).withPossibleValueSourceName("savedView").withPossibleValueSourceFilter(new QQueryFilter(new QFilterCriteria("tableName", QCriteriaOperator.EQUALS, "${input.tableName}"))))
.withField(field("savedReportId", QFieldType.INTEGER).withPossibleValueSourceName("savedReport").withPossibleValueSourceFilter(new QQueryFilter(new QFilterCriteria("tableName", QCriteriaOperator.EQUALS, "${input.tableName}"))))
.withField(field("format", QFieldType.STRING).withPossibleValueSourceName("file_gateway_format"))
.withField(field("fileName", QFieldType.STRING))
.withField(field("consumerName", QFieldType.STRING))
.withField(field("isActive", QFieldType.BOOLEAN).withDefaultValue(true))
.withField(field("description", QFieldType.TEXT));
table.withUniqueKey(new UniqueKey("name"));
table.withUniqueKey(new UniqueKey("path"));
table.withSection(SectionFactory.defaultT1("id", "name", "path", "kind", "accountId", "tableName"));
table.withSection(SectionFactory.defaultT2("savedBulkLoadProfileId", "savedViewId", "savedReportId", "format", "fileName", "consumerName", "isActive", "description"));
table.withSection(SectionFactory.defaultT3("createDate", "modifyDate"));
table.withSection(SectionFactory.customT2(UPLOAD_JOB, new QIcon("list")).withWidgetName(ALIAS + "_children_" + UPLOAD_JOB));
table.withSection(SectionFactory.customT2(OUTBOX_FILE, new QIcon("list")).withWidgetName(ALIAS + "_children_" + OUTBOX_FILE));
return (table);
}
/*******************************************************************************
** Metadata for upload jobs.
*******************************************************************************/
private static QTableMetaData upload_job(FileGatewayQBitConfig config)
{
QTableMetaData table = base(config, UPLOAD_JOB, "Upload Jobs")
.withRecordLabelFormatAndFields("%s", "id")
.withField(field("accountId", QFieldType.INTEGER).withPossibleValueSourceName(ACCOUNT))
.withField(field("aliasId", QFieldType.INTEGER).withPossibleValueSourceName(ALIAS))
.withField(field("connectionLogId", QFieldType.INTEGER).withPossibleValueSourceName(CONNECTION_LOG))
.withField(field("remoteAddress", QFieldType.STRING))
.withField(field("tableName", QFieldType.STRING).withPossibleValueSourceName("tables"))
.withField(field("savedBulkLoadProfileId", QFieldType.INTEGER).withPossibleValueSourceName("savedBulkLoadProfile"))
.withField(field("consumerName", QFieldType.STRING))
.withField(field("consumerReference", QFieldType.STRING))
.withField(field("path", QFieldType.STRING))
.withField(field("fileName", QFieldType.STRING))
.withField(field("sizeBytes", QFieldType.LONG))
.withField(field("sha256", QFieldType.STRING))
.withField(field("storageKey", QFieldType.STRING))
.withField(field("status", QFieldType.STRING).withPossibleValueSourceName("file_gateway_upload_status"))
.withField(field("duplicateOfJobId", QFieldType.INTEGER).withPossibleValueSourceName(UPLOAD_JOB))
.withField(field("processUuid", QFieldType.STRING))
.withField(field("insertedCount", QFieldType.INTEGER))
.withField(field("errorCount", QFieldType.INTEGER))
.withField(field("resultSummary", QFieldType.TEXT))
.withField(field("resultsStorageKey", QFieldType.STRING))
.withField(field("errorMessage", QFieldType.TEXT))
.withField(field("attempts", QFieldType.INTEGER))
.withField(field("nextAttemptAt", QFieldType.DATE_TIME))
.withField(field("claimedBy", QFieldType.STRING))
.withField(field("claimedAt", QFieldType.DATE_TIME))
.withField(field("queuedAt", QFieldType.DATE_TIME))
.withField(field("startedAt", QFieldType.DATE_TIME))
.withField(field("finishedAt", QFieldType.DATE_TIME))
.withField(field("acknowledgedAt", QFieldType.DATE_TIME));
table.withSection(SectionFactory.defaultT1("id", "accountId", "aliasId", "connectionLogId", "remoteAddress", "tableName"));
table.withSection(SectionFactory.defaultT2("savedBulkLoadProfileId", "consumerName", "consumerReference", "path", "fileName", "sizeBytes", "sha256", "storageKey", "status", "duplicateOfJobId", "processUuid", "insertedCount", "errorCount", "resultSummary", "resultsStorageKey", "errorMessage", "attempts", "nextAttemptAt", "claimedBy", "claimedAt", "queuedAt", "startedAt", "finishedAt", "acknowledgedAt"));
table.withSection(SectionFactory.defaultT3("createDate", "modifyDate"));
table.withSection(SectionFactory.customT2(TRANSFER_LOG, new QIcon("list")).withWidgetName(UPLOAD_JOB + "_children_" + TRANSFER_LOG));
table.withoutCapabilities(Capability.TABLE_INSERT, Capability.TABLE_UPDATE, Capability.TABLE_DELETE);
return (table);
}
/*******************************************************************************
** Metadata for outbox files.
*******************************************************************************/
private static QTableMetaData outbox_file(FileGatewayQBitConfig config)
{
QTableMetaData table = base(config, OUTBOX_FILE, "Outbox Files")
.withRecordLabelFormatAndFields("%s", "id")
.withField(field("aliasId", QFieldType.INTEGER).withPossibleValueSourceName(ALIAS))
.withField(field("fileName", QFieldType.STRING))
.withField(field("sizeBytes", QFieldType.LONG))
.withField(field("sha256", QFieldType.STRING))
.withField(field("storageKey", QFieldType.STRING))
.withField(field("status", QFieldType.STRING).withPossibleValueSourceName("file_gateway_outbox_status"))
.withField(field("firstDownloadedAt", QFieldType.DATE_TIME))
.withField(field("downloadCount", QFieldType.INTEGER))
.withField(field("removedAt", QFieldType.DATE_TIME))
.withField(field("expiresAt", QFieldType.DATE_TIME))
.withField(field("createdByConsumer", QFieldType.STRING));
table.withSection(SectionFactory.defaultT1("id", "aliasId", "fileName", "sizeBytes", "sha256", "storageKey"));
table.withSection(SectionFactory.defaultT2("status", "firstDownloadedAt", "downloadCount", "removedAt", "expiresAt", "createdByConsumer"));
table.withSection(SectionFactory.defaultT3("createDate", "modifyDate"));
table.withoutCapabilities(Capability.TABLE_INSERT, Capability.TABLE_UPDATE, Capability.TABLE_DELETE);
return (table);
}
/*******************************************************************************
** Metadata for transfers.
*******************************************************************************/
private static QTableMetaData transfer_log(FileGatewayQBitConfig config)
{
QTableMetaData table = base(config, TRANSFER_LOG, "Transfers")
.withRecordLabelFormatAndFields("%s", "id")
.withField(field("accountId", QFieldType.INTEGER).withPossibleValueSourceName(ACCOUNT))
.withField(field("connectionLogId", QFieldType.INTEGER).withPossibleValueSourceName(CONNECTION_LOG))
.withField(field("operation", QFieldType.STRING).withPossibleValueSourceName("file_gateway_operation"))
.withField(field("path", QFieldType.STRING))
.withField(field("objectKind", QFieldType.STRING).withPossibleValueSourceName("file_gateway_object_kind"))
.withField(field("objectId", QFieldType.STRING))
.withField(field("format", QFieldType.STRING).withPossibleValueSourceName("file_gateway_format"))
.withField(field("bytes", QFieldType.LONG))
.withField(field("sha256", QFieldType.STRING))
.withField(field("durationMs", QFieldType.LONG))
.withField(field("outcome", QFieldType.STRING).withPossibleValueSourceName("file_gateway_transfer_outcome"))
.withField(field("message", QFieldType.TEXT))
.withField(field("clientChannel", QFieldType.STRING))
.withField(field("uploadJobId", QFieldType.INTEGER).withPossibleValueSourceName(UPLOAD_JOB));
table.withSection(SectionFactory.defaultT1("id", "accountId", "connectionLogId", "operation", "path", "objectKind"));
table.withSection(SectionFactory.defaultT2("objectId", "format", "bytes", "sha256", "durationMs", "outcome", "message", "clientChannel", "uploadJobId"));
table.withSection(SectionFactory.defaultT3("createDate", "modifyDate"));
table.withoutCapabilities(Capability.TABLE_INSERT, Capability.TABLE_UPDATE, Capability.TABLE_DELETE);
return (table);
}
/*******************************************************************************
** Metadata for connections.
*******************************************************************************/
private static QTableMetaData connection_log(FileGatewayQBitConfig config)
{
QTableMetaData table = base(config, CONNECTION_LOG, "Connections")
.withRecordLabelFormatAndFields("%s", "id")
.withField(field("accountId", QFieldType.INTEGER).withPossibleValueSourceName(ACCOUNT))
.withField(field("username", QFieldType.STRING))
.withField(field("remoteIp", QFieldType.STRING))
.withField(field("clientVersion", QFieldType.STRING))
.withField(field("keyFingerprint", QFieldType.STRING))
.withField(field("outcome", QFieldType.STRING).withPossibleValueSourceName("file_gateway_connection_outcome"))
.withField(field("sessionId", QFieldType.STRING))
.withField(field("podName", QFieldType.STRING))
.withField(field("connectedAt", QFieldType.DATE_TIME))
.withField(field("lastSeenAt", QFieldType.DATE_TIME))
.withField(field("disconnectedAt", QFieldType.DATE_TIME))
.withField(field("disconnectReason", QFieldType.TEXT))
.withField(field("bytesIn", QFieldType.LONG))
.withField(field("bytesOut", QFieldType.LONG))
.withField(field("operationCount", QFieldType.LONG))
.withField(field("clientChannel", QFieldType.STRING));
table.withSection(SectionFactory.defaultT1("id", "accountId", "username", "remoteIp", "clientVersion", "keyFingerprint"));
table.withSection(SectionFactory.defaultT2("outcome", "sessionId", "podName", "connectedAt", "lastSeenAt", "disconnectedAt", "disconnectReason", "bytesIn", "bytesOut", "operationCount", "clientChannel"));
table.withSection(SectionFactory.defaultT3("createDate", "modifyDate"));
table.withoutCapabilities(Capability.TABLE_INSERT, Capability.TABLE_UPDATE, Capability.TABLE_DELETE);
return (table);
}
/*******************************************************************************
** Parent and child relationships.
*******************************************************************************/
public static List<QJoinMetaData> joins()
{
return (List.of(
new QJoinMetaData().withName(ACCOUNT + "_join_" + SSH_KEY).withType(JoinType.ONE_TO_MANY)
.withLeftTable(ACCOUNT).withRightTable(SSH_KEY).withJoinOn(new JoinOn("id", "accountId")),
new QJoinMetaData().withName(ACCOUNT + "_join_" + UPLOAD_JOB).withType(JoinType.ONE_TO_MANY)
.withLeftTable(ACCOUNT).withRightTable(UPLOAD_JOB).withJoinOn(new JoinOn("id", "accountId")),
new QJoinMetaData().withName(ACCOUNT + "_join_" + CONNECTION_LOG).withType(JoinType.ONE_TO_MANY)
.withLeftTable(ACCOUNT).withRightTable(CONNECTION_LOG).withJoinOn(new JoinOn("id", "accountId")),
new QJoinMetaData().withName(ALIAS + "_join_" + UPLOAD_JOB).withType(JoinType.ONE_TO_MANY)
.withLeftTable(ALIAS).withRightTable(UPLOAD_JOB).withJoinOn(new JoinOn("id", "aliasId")),
new QJoinMetaData().withName(ALIAS + "_join_" + OUTBOX_FILE).withType(JoinType.ONE_TO_MANY)
.withLeftTable(ALIAS).withRightTable(OUTBOX_FILE).withJoinOn(new JoinOn("id", "aliasId")),
new QJoinMetaData().withName(UPLOAD_JOB + "_join_" + TRANSFER_LOG).withType(JoinType.ONE_TO_MANY)
.withLeftTable(UPLOAD_JOB).withRightTable(TRANSFER_LOG).withJoinOn(new JoinOn("id", "uploadJobId"))));
}
/*******************************************************************************
** Typed child-record widgets; no home or HTML widget.
*******************************************************************************/
public static List<QWidgetMetaData> childWidgets()
{
List<QWidgetMetaData> widgets = new ArrayList<>();
for(QJoinMetaData join : joins())
{
widgets.add(ChildRecordListRenderer.widgetMetaDataBuilder(join)
.withName(join.getLeftTable() + "_children_" + join.getRightTable()).withLabel(join.getRightTable().replace("file_gateway_", "").replace("_", " "))
.withCanAddChildRecord(SSH_KEY.equals(join.getRightTable())).withMaxRows(100)
.getWidgetMetaData().withGridColumns(12).withPermissionRules(permissionRules(join.getRightTable())));
}
return (widgets);
}
}
@@ -0,0 +1,76 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.metadata;
import java.security.PublicKey;
import java.util.List;
import java.util.Map;
import java.util.Objects;
import java.util.Optional;
import com.kingsrook.qqq.backend.core.actions.customizers.TableCustomizerInterface;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.statusmessages.BadInputStatusMessage;
import org.apache.sshd.common.config.keys.KeyUtils;
import org.apache.sshd.common.config.keys.PublicKeyEntry;
import org.apache.sshd.common.digest.BuiltinDigests;
/*******************************************************************************
** Validates stored public keys and computes authoritative key metadata.
*******************************************************************************/
public class SshKeyCustomizer implements TableCustomizerInterface
{
/*******************************************************************************
** Parse and validate inserts and partial updates without logging key material.
*******************************************************************************/
@Override
public List<QRecord> preInsertOrUpdate(AbstractActionInput input, List<QRecord> records, boolean isPreview, Optional<List<QRecord>> oldRecords)
{
for(QRecord record : records)
{
QRecord old = oldRecords.orElse(List.of()).stream().filter(candidate -> Objects.equals(candidate.getValue("id"), record.getValue("id"))).findFirst().orElse(null);
String text = record.getValues().containsKey("publicKey") ? record.getValueString("publicKey") : old == null ? null : old.getValueString("publicKey");
try
{
PublicKey key = PublicKeyEntry.parsePublicKeyEntry(text).resolvePublicKey(null, Map.of(), null);
String type = KeyUtils.getKeyType(key);
int bits = KeyUtils.getKeySize(key);
if(!("ssh-ed25519".equals(type) || type.startsWith("ecdsa-sha2-nistp") || ("ssh-rsa".equals(type) && bits >= 3072)))
{
throw (new IllegalArgumentException("Unsupported key type or size"));
}
record.setValue("fingerprint", KeyUtils.getFingerPrint(BuiltinDigests.sha256, key));
record.setValue("keyType", type);
record.setValue("keyBits", bits);
}
catch(Exception e)
{
record.addError(new BadInputStatusMessage("Use an OpenSSH Ed25519, ECDSA, or RSA key of at least 3072 bits"));
}
}
return (records);
}
}
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,600 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.model;
import java.time.Instant;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.data.QField;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
/*******************************************************************************
** Record entity for endpoints.
*******************************************************************************/
public class FileGatewayAlias extends QRecordEntity
{
public static final String TABLE_NAME = "file_gateway_alias";
@QField
private Integer id;
@QField
private Instant createDate;
@QField
private Instant modifyDate;
@QField
private String name;
@QField
private String path;
@QField
private String kind;
@QField
private Integer accountId;
@QField
private String tableName;
@QField
private Integer savedBulkLoadProfileId;
@QField
private Integer savedViewId;
@QField
private Integer savedReportId;
@QField
private String format;
@QField
private String fileName;
@QField
private String consumerName;
@QField
private Boolean isActive;
@QField
private String description;
/*******************************************************************************
** Create an empty entity.
*******************************************************************************/
public FileGatewayAlias()
{
}
/*******************************************************************************
** Populate from a database record.
*******************************************************************************/
public FileGatewayAlias(QRecord record) throws QException
{
populateFromQRecord(record);
}
/*******************************************************************************
** Getter for id.
*******************************************************************************/
public Integer getId()
{
return (id);
}
/*******************************************************************************
** Fluent setter for id.
*******************************************************************************/
public FileGatewayAlias withId(Integer id)
{
this.id = id;
return (this);
}
/*******************************************************************************
** Getter for createDate.
*******************************************************************************/
public Instant getCreateDate()
{
return (createDate);
}
/*******************************************************************************
** Fluent setter for createDate.
*******************************************************************************/
public FileGatewayAlias withCreateDate(Instant createDate)
{
this.createDate = createDate;
return (this);
}
/*******************************************************************************
** Getter for modifyDate.
*******************************************************************************/
public Instant getModifyDate()
{
return (modifyDate);
}
/*******************************************************************************
** Fluent setter for modifyDate.
*******************************************************************************/
public FileGatewayAlias withModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
return (this);
}
/*******************************************************************************
** Getter for name.
*******************************************************************************/
public String getName()
{
return (name);
}
/*******************************************************************************
** Fluent setter for name.
*******************************************************************************/
public FileGatewayAlias withName(String name)
{
this.name = name;
return (this);
}
/*******************************************************************************
** Getter for path.
*******************************************************************************/
public String getPath()
{
return (path);
}
/*******************************************************************************
** Fluent setter for path.
*******************************************************************************/
public FileGatewayAlias withPath(String path)
{
this.path = path;
return (this);
}
/*******************************************************************************
** Getter for kind.
*******************************************************************************/
public String getKind()
{
return (kind);
}
/*******************************************************************************
** Fluent setter for kind.
*******************************************************************************/
public FileGatewayAlias withKind(String kind)
{
this.kind = kind;
return (this);
}
/*******************************************************************************
** Getter for accountId.
*******************************************************************************/
public Integer getAccountId()
{
return (accountId);
}
/*******************************************************************************
** Fluent setter for accountId.
*******************************************************************************/
public FileGatewayAlias withAccountId(Integer accountId)
{
this.accountId = accountId;
return (this);
}
/*******************************************************************************
** Getter for tableName.
*******************************************************************************/
public String getTableName()
{
return (tableName);
}
/*******************************************************************************
** Fluent setter for tableName.
*******************************************************************************/
public FileGatewayAlias withTableName(String tableName)
{
this.tableName = tableName;
return (this);
}
/*******************************************************************************
** Getter for savedBulkLoadProfileId.
*******************************************************************************/
public Integer getSavedBulkLoadProfileId()
{
return (savedBulkLoadProfileId);
}
/*******************************************************************************
** Fluent setter for savedBulkLoadProfileId.
*******************************************************************************/
public FileGatewayAlias withSavedBulkLoadProfileId(Integer savedBulkLoadProfileId)
{
this.savedBulkLoadProfileId = savedBulkLoadProfileId;
return (this);
}
/*******************************************************************************
** Getter for savedViewId.
*******************************************************************************/
public Integer getSavedViewId()
{
return (savedViewId);
}
/*******************************************************************************
** Fluent setter for savedViewId.
*******************************************************************************/
public FileGatewayAlias withSavedViewId(Integer savedViewId)
{
this.savedViewId = savedViewId;
return (this);
}
/*******************************************************************************
** Getter for savedReportId.
*******************************************************************************/
public Integer getSavedReportId()
{
return (savedReportId);
}
/*******************************************************************************
** Fluent setter for savedReportId.
*******************************************************************************/
public FileGatewayAlias withSavedReportId(Integer savedReportId)
{
this.savedReportId = savedReportId;
return (this);
}
/*******************************************************************************
** Getter for format.
*******************************************************************************/
public String getFormat()
{
return (format);
}
/*******************************************************************************
** Fluent setter for format.
*******************************************************************************/
public FileGatewayAlias withFormat(String format)
{
this.format = format;
return (this);
}
/*******************************************************************************
** Getter for fileName.
*******************************************************************************/
public String getFileName()
{
return (fileName);
}
/*******************************************************************************
** Fluent setter for fileName.
*******************************************************************************/
public FileGatewayAlias withFileName(String fileName)
{
this.fileName = fileName;
return (this);
}
/*******************************************************************************
** Getter for consumerName.
*******************************************************************************/
public String getConsumerName()
{
return (consumerName);
}
/*******************************************************************************
** Fluent setter for consumerName.
*******************************************************************************/
public FileGatewayAlias withConsumerName(String consumerName)
{
this.consumerName = consumerName;
return (this);
}
/*******************************************************************************
** Getter for isActive.
*******************************************************************************/
public Boolean getIsActive()
{
return (isActive);
}
/*******************************************************************************
** Fluent setter for isActive.
*******************************************************************************/
public FileGatewayAlias withIsActive(Boolean isActive)
{
this.isActive = isActive;
return (this);
}
/*******************************************************************************
** Getter for description.
*******************************************************************************/
public String getDescription()
{
return (description);
}
/*******************************************************************************
** Fluent setter for description.
*******************************************************************************/
public FileGatewayAlias withDescription(String description)
{
this.description = description;
return (this);
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setId(Integer id)
{
this.id = id;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setCreateDate(Instant createDate)
{
this.createDate = createDate;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setName(String name)
{
this.name = name;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setPath(String path)
{
this.path = path;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setKind(String kind)
{
this.kind = kind;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setAccountId(Integer accountId)
{
this.accountId = accountId;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setTableName(String tableName)
{
this.tableName = tableName;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setSavedBulkLoadProfileId(Integer savedBulkLoadProfileId)
{
this.savedBulkLoadProfileId = savedBulkLoadProfileId;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setSavedViewId(Integer savedViewId)
{
this.savedViewId = savedViewId;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setSavedReportId(Integer savedReportId)
{
this.savedReportId = savedReportId;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setFormat(String format)
{
this.format = format;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setFileName(String fileName)
{
this.fileName = fileName;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setConsumerName(String consumerName)
{
this.consumerName = consumerName;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setIsActive(Boolean isActive)
{
this.isActive = isActive;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setDescription(String description)
{
this.description = description;
}
}
@@ -0,0 +1,702 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.model;
import java.time.Instant;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.data.QField;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
/*******************************************************************************
** Record entity for connections.
*******************************************************************************/
public class FileGatewayConnectionLog extends QRecordEntity
{
public static final String TABLE_NAME = "file_gateway_connection_log";
@QField
private Integer id;
@QField
private Instant createDate;
@QField
private Instant modifyDate;
@QField
private Integer accountId;
@QField
private String username;
@QField
private String remoteIp;
@QField
private String clientVersion;
@QField
private String keyFingerprint;
@QField
private String outcome;
@QField
private String sessionId;
@QField
private String podName;
@QField
private Instant connectedAt;
@QField
private Instant lastSeenAt;
@QField
private Instant disconnectedAt;
@QField
private String disconnectReason;
@QField
private Long bytesIn;
@QField
private Long bytesOut;
@QField
private Long operationCount;
@QField
private String clientChannel;
/*******************************************************************************
** Create an empty entity.
*******************************************************************************/
public FileGatewayConnectionLog()
{
}
/*******************************************************************************
** Populate from a database record.
*******************************************************************************/
public FileGatewayConnectionLog(QRecord record) throws QException
{
populateFromQRecord(record);
}
/*******************************************************************************
** Getter for id.
*******************************************************************************/
public Integer getId()
{
return (id);
}
/*******************************************************************************
** Fluent setter for id.
*******************************************************************************/
public FileGatewayConnectionLog withId(Integer id)
{
this.id = id;
return (this);
}
/*******************************************************************************
** Getter for createDate.
*******************************************************************************/
public Instant getCreateDate()
{
return (createDate);
}
/*******************************************************************************
** Fluent setter for createDate.
*******************************************************************************/
public FileGatewayConnectionLog withCreateDate(Instant createDate)
{
this.createDate = createDate;
return (this);
}
/*******************************************************************************
** Getter for modifyDate.
*******************************************************************************/
public Instant getModifyDate()
{
return (modifyDate);
}
/*******************************************************************************
** Fluent setter for modifyDate.
*******************************************************************************/
public FileGatewayConnectionLog withModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
return (this);
}
/*******************************************************************************
** Getter for accountId.
*******************************************************************************/
public Integer getAccountId()
{
return (accountId);
}
/*******************************************************************************
** Fluent setter for accountId.
*******************************************************************************/
public FileGatewayConnectionLog withAccountId(Integer accountId)
{
this.accountId = accountId;
return (this);
}
/*******************************************************************************
** Getter for username.
*******************************************************************************/
public String getUsername()
{
return (username);
}
/*******************************************************************************
** Fluent setter for username.
*******************************************************************************/
public FileGatewayConnectionLog withUsername(String username)
{
this.username = username;
return (this);
}
/*******************************************************************************
** Getter for remoteIp.
*******************************************************************************/
public String getRemoteIp()
{
return (remoteIp);
}
/*******************************************************************************
** Fluent setter for remoteIp.
*******************************************************************************/
public FileGatewayConnectionLog withRemoteIp(String remoteIp)
{
this.remoteIp = remoteIp;
return (this);
}
/*******************************************************************************
** Getter for clientVersion.
*******************************************************************************/
public String getClientVersion()
{
return (clientVersion);
}
/*******************************************************************************
** Fluent setter for clientVersion.
*******************************************************************************/
public FileGatewayConnectionLog withClientVersion(String clientVersion)
{
this.clientVersion = clientVersion;
return (this);
}
/*******************************************************************************
** Getter for keyFingerprint.
*******************************************************************************/
public String getKeyFingerprint()
{
return (keyFingerprint);
}
/*******************************************************************************
** Fluent setter for keyFingerprint.
*******************************************************************************/
public FileGatewayConnectionLog withKeyFingerprint(String keyFingerprint)
{
this.keyFingerprint = keyFingerprint;
return (this);
}
/*******************************************************************************
** Getter for outcome.
*******************************************************************************/
public String getOutcome()
{
return (outcome);
}
/*******************************************************************************
** Fluent setter for outcome.
*******************************************************************************/
public FileGatewayConnectionLog withOutcome(String outcome)
{
this.outcome = outcome;
return (this);
}
/*******************************************************************************
** Getter for sessionId.
*******************************************************************************/
public String getSessionId()
{
return (sessionId);
}
/*******************************************************************************
** Fluent setter for sessionId.
*******************************************************************************/
public FileGatewayConnectionLog withSessionId(String sessionId)
{
this.sessionId = sessionId;
return (this);
}
/*******************************************************************************
** Getter for podName.
*******************************************************************************/
public String getPodName()
{
return (podName);
}
/*******************************************************************************
** Fluent setter for podName.
*******************************************************************************/
public FileGatewayConnectionLog withPodName(String podName)
{
this.podName = podName;
return (this);
}
/*******************************************************************************
** Getter for connectedAt.
*******************************************************************************/
public Instant getConnectedAt()
{
return (connectedAt);
}
/*******************************************************************************
** Fluent setter for connectedAt.
*******************************************************************************/
public FileGatewayConnectionLog withConnectedAt(Instant connectedAt)
{
this.connectedAt = connectedAt;
return (this);
}
/*******************************************************************************
** Getter for lastSeenAt.
*******************************************************************************/
public Instant getLastSeenAt()
{
return (lastSeenAt);
}
/*******************************************************************************
** Fluent setter for lastSeenAt.
*******************************************************************************/
public FileGatewayConnectionLog withLastSeenAt(Instant lastSeenAt)
{
this.lastSeenAt = lastSeenAt;
return (this);
}
/*******************************************************************************
** Getter for disconnectedAt.
*******************************************************************************/
public Instant getDisconnectedAt()
{
return (disconnectedAt);
}
/*******************************************************************************
** Fluent setter for disconnectedAt.
*******************************************************************************/
public FileGatewayConnectionLog withDisconnectedAt(Instant disconnectedAt)
{
this.disconnectedAt = disconnectedAt;
return (this);
}
/*******************************************************************************
** Getter for disconnectReason.
*******************************************************************************/
public String getDisconnectReason()
{
return (disconnectReason);
}
/*******************************************************************************
** Fluent setter for disconnectReason.
*******************************************************************************/
public FileGatewayConnectionLog withDisconnectReason(String disconnectReason)
{
this.disconnectReason = disconnectReason;
return (this);
}
/*******************************************************************************
** Getter for bytesIn.
*******************************************************************************/
public Long getBytesIn()
{
return (bytesIn);
}
/*******************************************************************************
** Fluent setter for bytesIn.
*******************************************************************************/
public FileGatewayConnectionLog withBytesIn(Long bytesIn)
{
this.bytesIn = bytesIn;
return (this);
}
/*******************************************************************************
** Getter for bytesOut.
*******************************************************************************/
public Long getBytesOut()
{
return (bytesOut);
}
/*******************************************************************************
** Fluent setter for bytesOut.
*******************************************************************************/
public FileGatewayConnectionLog withBytesOut(Long bytesOut)
{
this.bytesOut = bytesOut;
return (this);
}
/*******************************************************************************
** Getter for operationCount.
*******************************************************************************/
public Long getOperationCount()
{
return (operationCount);
}
/*******************************************************************************
** Fluent setter for operationCount.
*******************************************************************************/
public FileGatewayConnectionLog withOperationCount(Long operationCount)
{
this.operationCount = operationCount;
return (this);
}
/*******************************************************************************
** Getter for clientChannel.
*******************************************************************************/
public String getClientChannel()
{
return (clientChannel);
}
/*******************************************************************************
** Fluent setter for clientChannel.
*******************************************************************************/
public FileGatewayConnectionLog withClientChannel(String clientChannel)
{
this.clientChannel = clientChannel;
return (this);
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setId(Integer id)
{
this.id = id;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setCreateDate(Instant createDate)
{
this.createDate = createDate;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setAccountId(Integer accountId)
{
this.accountId = accountId;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setUsername(String username)
{
this.username = username;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setRemoteIp(String remoteIp)
{
this.remoteIp = remoteIp;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setClientVersion(String clientVersion)
{
this.clientVersion = clientVersion;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setKeyFingerprint(String keyFingerprint)
{
this.keyFingerprint = keyFingerprint;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setOutcome(String outcome)
{
this.outcome = outcome;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setSessionId(String sessionId)
{
this.sessionId = sessionId;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setPodName(String podName)
{
this.podName = podName;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setConnectedAt(Instant connectedAt)
{
this.connectedAt = connectedAt;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setLastSeenAt(Instant lastSeenAt)
{
this.lastSeenAt = lastSeenAt;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setDisconnectedAt(Instant disconnectedAt)
{
this.disconnectedAt = disconnectedAt;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setDisconnectReason(String disconnectReason)
{
this.disconnectReason = disconnectReason;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setBytesIn(Long bytesIn)
{
this.bytesIn = bytesIn;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setBytesOut(Long bytesOut)
{
this.bytesOut = bytesOut;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setOperationCount(Long operationCount)
{
this.operationCount = operationCount;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setClientChannel(String clientChannel)
{
this.clientChannel = clientChannel;
}
}
@@ -0,0 +1,532 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.model;
import java.time.Instant;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.data.QField;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
/*******************************************************************************
** Record entity for outbox files.
*******************************************************************************/
public class FileGatewayOutboxFile extends QRecordEntity
{
public static final String TABLE_NAME = "file_gateway_outbox_file";
@QField
private Integer id;
@QField
private Instant createDate;
@QField
private Instant modifyDate;
@QField
private Integer aliasId;
@QField
private String fileName;
@QField
private Long sizeBytes;
@QField
private String sha256;
@QField
private String storageKey;
@QField
private String status;
@QField
private Instant firstDownloadedAt;
@QField
private Integer downloadCount;
@QField
private Instant removedAt;
@QField
private Instant expiresAt;
@QField
private String createdByConsumer;
/*******************************************************************************
** Create an empty entity.
*******************************************************************************/
public FileGatewayOutboxFile()
{
}
/*******************************************************************************
** Populate from a database record.
*******************************************************************************/
public FileGatewayOutboxFile(QRecord record) throws QException
{
populateFromQRecord(record);
}
/*******************************************************************************
** Getter for id.
*******************************************************************************/
public Integer getId()
{
return (id);
}
/*******************************************************************************
** Fluent setter for id.
*******************************************************************************/
public FileGatewayOutboxFile withId(Integer id)
{
this.id = id;
return (this);
}
/*******************************************************************************
** Getter for createDate.
*******************************************************************************/
public Instant getCreateDate()
{
return (createDate);
}
/*******************************************************************************
** Fluent setter for createDate.
*******************************************************************************/
public FileGatewayOutboxFile withCreateDate(Instant createDate)
{
this.createDate = createDate;
return (this);
}
/*******************************************************************************
** Getter for modifyDate.
*******************************************************************************/
public Instant getModifyDate()
{
return (modifyDate);
}
/*******************************************************************************
** Fluent setter for modifyDate.
*******************************************************************************/
public FileGatewayOutboxFile withModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
return (this);
}
/*******************************************************************************
** Getter for aliasId.
*******************************************************************************/
public Integer getAliasId()
{
return (aliasId);
}
/*******************************************************************************
** Fluent setter for aliasId.
*******************************************************************************/
public FileGatewayOutboxFile withAliasId(Integer aliasId)
{
this.aliasId = aliasId;
return (this);
}
/*******************************************************************************
** Getter for fileName.
*******************************************************************************/
public String getFileName()
{
return (fileName);
}
/*******************************************************************************
** Fluent setter for fileName.
*******************************************************************************/
public FileGatewayOutboxFile withFileName(String fileName)
{
this.fileName = fileName;
return (this);
}
/*******************************************************************************
** Getter for sizeBytes.
*******************************************************************************/
public Long getSizeBytes()
{
return (sizeBytes);
}
/*******************************************************************************
** Fluent setter for sizeBytes.
*******************************************************************************/
public FileGatewayOutboxFile withSizeBytes(Long sizeBytes)
{
this.sizeBytes = sizeBytes;
return (this);
}
/*******************************************************************************
** Getter for sha256.
*******************************************************************************/
public String getSha256()
{
return (sha256);
}
/*******************************************************************************
** Fluent setter for sha256.
*******************************************************************************/
public FileGatewayOutboxFile withSha256(String sha256)
{
this.sha256 = sha256;
return (this);
}
/*******************************************************************************
** Getter for storageKey.
*******************************************************************************/
public String getStorageKey()
{
return (storageKey);
}
/*******************************************************************************
** Fluent setter for storageKey.
*******************************************************************************/
public FileGatewayOutboxFile withStorageKey(String storageKey)
{
this.storageKey = storageKey;
return (this);
}
/*******************************************************************************
** Getter for status.
*******************************************************************************/
public String getStatus()
{
return (status);
}
/*******************************************************************************
** Fluent setter for status.
*******************************************************************************/
public FileGatewayOutboxFile withStatus(String status)
{
this.status = status;
return (this);
}
/*******************************************************************************
** Getter for firstDownloadedAt.
*******************************************************************************/
public Instant getFirstDownloadedAt()
{
return (firstDownloadedAt);
}
/*******************************************************************************
** Fluent setter for firstDownloadedAt.
*******************************************************************************/
public FileGatewayOutboxFile withFirstDownloadedAt(Instant firstDownloadedAt)
{
this.firstDownloadedAt = firstDownloadedAt;
return (this);
}
/*******************************************************************************
** Getter for downloadCount.
*******************************************************************************/
public Integer getDownloadCount()
{
return (downloadCount);
}
/*******************************************************************************
** Fluent setter for downloadCount.
*******************************************************************************/
public FileGatewayOutboxFile withDownloadCount(Integer downloadCount)
{
this.downloadCount = downloadCount;
return (this);
}
/*******************************************************************************
** Getter for removedAt.
*******************************************************************************/
public Instant getRemovedAt()
{
return (removedAt);
}
/*******************************************************************************
** Fluent setter for removedAt.
*******************************************************************************/
public FileGatewayOutboxFile withRemovedAt(Instant removedAt)
{
this.removedAt = removedAt;
return (this);
}
/*******************************************************************************
** Getter for expiresAt.
*******************************************************************************/
public Instant getExpiresAt()
{
return (expiresAt);
}
/*******************************************************************************
** Fluent setter for expiresAt.
*******************************************************************************/
public FileGatewayOutboxFile withExpiresAt(Instant expiresAt)
{
this.expiresAt = expiresAt;
return (this);
}
/*******************************************************************************
** Getter for createdByConsumer.
*******************************************************************************/
public String getCreatedByConsumer()
{
return (createdByConsumer);
}
/*******************************************************************************
** Fluent setter for createdByConsumer.
*******************************************************************************/
public FileGatewayOutboxFile withCreatedByConsumer(String createdByConsumer)
{
this.createdByConsumer = createdByConsumer;
return (this);
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setId(Integer id)
{
this.id = id;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setCreateDate(Instant createDate)
{
this.createDate = createDate;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setAliasId(Integer aliasId)
{
this.aliasId = aliasId;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setFileName(String fileName)
{
this.fileName = fileName;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setSizeBytes(Long sizeBytes)
{
this.sizeBytes = sizeBytes;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setSha256(String sha256)
{
this.sha256 = sha256;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setStorageKey(String storageKey)
{
this.storageKey = storageKey;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setStatus(String status)
{
this.status = status;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setFirstDownloadedAt(Instant firstDownloadedAt)
{
this.firstDownloadedAt = firstDownloadedAt;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setDownloadCount(Integer downloadCount)
{
this.downloadCount = downloadCount;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setRemovedAt(Instant removedAt)
{
this.removedAt = removedAt;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setExpiresAt(Instant expiresAt)
{
this.expiresAt = expiresAt;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setCreatedByConsumer(String createdByConsumer)
{
this.createdByConsumer = createdByConsumer;
}
}
@@ -0,0 +1,498 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.model;
import java.time.Instant;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.data.QField;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
/*******************************************************************************
** Record entity for ssh keys.
*******************************************************************************/
public class FileGatewaySshKey extends QRecordEntity
{
public static final String TABLE_NAME = "file_gateway_ssh_key";
@QField
private Integer id;
@QField
private Instant createDate;
@QField
private Instant modifyDate;
@QField
private Integer accountId;
@QField
private String label;
@QField
private String publicKey;
@QField
private String fingerprint;
@QField
private String keyType;
@QField
private Integer keyBits;
@QField
private Instant expiresAt;
@QField
private Instant revokedAt;
@QField
private Instant lastUsedAt;
@QField
private String lastUsedIp;
/*******************************************************************************
** Create an empty entity.
*******************************************************************************/
public FileGatewaySshKey()
{
}
/*******************************************************************************
** Populate from a database record.
*******************************************************************************/
public FileGatewaySshKey(QRecord record) throws QException
{
populateFromQRecord(record);
}
/*******************************************************************************
** Getter for id.
*******************************************************************************/
public Integer getId()
{
return (id);
}
/*******************************************************************************
** Fluent setter for id.
*******************************************************************************/
public FileGatewaySshKey withId(Integer id)
{
this.id = id;
return (this);
}
/*******************************************************************************
** Getter for createDate.
*******************************************************************************/
public Instant getCreateDate()
{
return (createDate);
}
/*******************************************************************************
** Fluent setter for createDate.
*******************************************************************************/
public FileGatewaySshKey withCreateDate(Instant createDate)
{
this.createDate = createDate;
return (this);
}
/*******************************************************************************
** Getter for modifyDate.
*******************************************************************************/
public Instant getModifyDate()
{
return (modifyDate);
}
/*******************************************************************************
** Fluent setter for modifyDate.
*******************************************************************************/
public FileGatewaySshKey withModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
return (this);
}
/*******************************************************************************
** Getter for accountId.
*******************************************************************************/
public Integer getAccountId()
{
return (accountId);
}
/*******************************************************************************
** Fluent setter for accountId.
*******************************************************************************/
public FileGatewaySshKey withAccountId(Integer accountId)
{
this.accountId = accountId;
return (this);
}
/*******************************************************************************
** Getter for label.
*******************************************************************************/
public String getLabel()
{
return (label);
}
/*******************************************************************************
** Fluent setter for label.
*******************************************************************************/
public FileGatewaySshKey withLabel(String label)
{
this.label = label;
return (this);
}
/*******************************************************************************
** Getter for publicKey.
*******************************************************************************/
public String getPublicKey()
{
return (publicKey);
}
/*******************************************************************************
** Fluent setter for publicKey.
*******************************************************************************/
public FileGatewaySshKey withPublicKey(String publicKey)
{
this.publicKey = publicKey;
return (this);
}
/*******************************************************************************
** Getter for fingerprint.
*******************************************************************************/
public String getFingerprint()
{
return (fingerprint);
}
/*******************************************************************************
** Fluent setter for fingerprint.
*******************************************************************************/
public FileGatewaySshKey withFingerprint(String fingerprint)
{
this.fingerprint = fingerprint;
return (this);
}
/*******************************************************************************
** Getter for keyType.
*******************************************************************************/
public String getKeyType()
{
return (keyType);
}
/*******************************************************************************
** Fluent setter for keyType.
*******************************************************************************/
public FileGatewaySshKey withKeyType(String keyType)
{
this.keyType = keyType;
return (this);
}
/*******************************************************************************
** Getter for keyBits.
*******************************************************************************/
public Integer getKeyBits()
{
return (keyBits);
}
/*******************************************************************************
** Fluent setter for keyBits.
*******************************************************************************/
public FileGatewaySshKey withKeyBits(Integer keyBits)
{
this.keyBits = keyBits;
return (this);
}
/*******************************************************************************
** Getter for expiresAt.
*******************************************************************************/
public Instant getExpiresAt()
{
return (expiresAt);
}
/*******************************************************************************
** Fluent setter for expiresAt.
*******************************************************************************/
public FileGatewaySshKey withExpiresAt(Instant expiresAt)
{
this.expiresAt = expiresAt;
return (this);
}
/*******************************************************************************
** Getter for revokedAt.
*******************************************************************************/
public Instant getRevokedAt()
{
return (revokedAt);
}
/*******************************************************************************
** Fluent setter for revokedAt.
*******************************************************************************/
public FileGatewaySshKey withRevokedAt(Instant revokedAt)
{
this.revokedAt = revokedAt;
return (this);
}
/*******************************************************************************
** Getter for lastUsedAt.
*******************************************************************************/
public Instant getLastUsedAt()
{
return (lastUsedAt);
}
/*******************************************************************************
** Fluent setter for lastUsedAt.
*******************************************************************************/
public FileGatewaySshKey withLastUsedAt(Instant lastUsedAt)
{
this.lastUsedAt = lastUsedAt;
return (this);
}
/*******************************************************************************
** Getter for lastUsedIp.
*******************************************************************************/
public String getLastUsedIp()
{
return (lastUsedIp);
}
/*******************************************************************************
** Fluent setter for lastUsedIp.
*******************************************************************************/
public FileGatewaySshKey withLastUsedIp(String lastUsedIp)
{
this.lastUsedIp = lastUsedIp;
return (this);
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setId(Integer id)
{
this.id = id;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setCreateDate(Instant createDate)
{
this.createDate = createDate;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setAccountId(Integer accountId)
{
this.accountId = accountId;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setLabel(String label)
{
this.label = label;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setPublicKey(String publicKey)
{
this.publicKey = publicKey;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setFingerprint(String fingerprint)
{
this.fingerprint = fingerprint;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setKeyType(String keyType)
{
this.keyType = keyType;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setKeyBits(Integer keyBits)
{
this.keyBits = keyBits;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setExpiresAt(Instant expiresAt)
{
this.expiresAt = expiresAt;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setRevokedAt(Instant revokedAt)
{
this.revokedAt = revokedAt;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setLastUsedAt(Instant lastUsedAt)
{
this.lastUsedAt = lastUsedAt;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setLastUsedIp(String lastUsedIp)
{
this.lastUsedIp = lastUsedIp;
}
}
@@ -0,0 +1,634 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.model;
import java.time.Instant;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.data.QField;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
/*******************************************************************************
** Record entity for transfers.
*******************************************************************************/
public class FileGatewayTransferLog extends QRecordEntity
{
public static final String TABLE_NAME = "file_gateway_transfer_log";
@QField
private Integer id;
@QField
private Instant createDate;
@QField
private Instant modifyDate;
@QField
private Integer accountId;
@QField
private Integer connectionLogId;
@QField
private String operation;
@QField
private String path;
@QField
private String objectKind;
@QField
private String objectId;
@QField
private String format;
@QField
private Long bytes;
@QField
private String sha256;
@QField
private Long durationMs;
@QField
private String outcome;
@QField
private String message;
@QField
private String clientChannel;
@QField
private Integer uploadJobId;
/*******************************************************************************
** Create an empty entity.
*******************************************************************************/
public FileGatewayTransferLog()
{
}
/*******************************************************************************
** Populate from a database record.
*******************************************************************************/
public FileGatewayTransferLog(QRecord record) throws QException
{
populateFromQRecord(record);
}
/*******************************************************************************
** Getter for id.
*******************************************************************************/
public Integer getId()
{
return (id);
}
/*******************************************************************************
** Fluent setter for id.
*******************************************************************************/
public FileGatewayTransferLog withId(Integer id)
{
this.id = id;
return (this);
}
/*******************************************************************************
** Getter for createDate.
*******************************************************************************/
public Instant getCreateDate()
{
return (createDate);
}
/*******************************************************************************
** Fluent setter for createDate.
*******************************************************************************/
public FileGatewayTransferLog withCreateDate(Instant createDate)
{
this.createDate = createDate;
return (this);
}
/*******************************************************************************
** Getter for modifyDate.
*******************************************************************************/
public Instant getModifyDate()
{
return (modifyDate);
}
/*******************************************************************************
** Fluent setter for modifyDate.
*******************************************************************************/
public FileGatewayTransferLog withModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
return (this);
}
/*******************************************************************************
** Getter for accountId.
*******************************************************************************/
public Integer getAccountId()
{
return (accountId);
}
/*******************************************************************************
** Fluent setter for accountId.
*******************************************************************************/
public FileGatewayTransferLog withAccountId(Integer accountId)
{
this.accountId = accountId;
return (this);
}
/*******************************************************************************
** Getter for connectionLogId.
*******************************************************************************/
public Integer getConnectionLogId()
{
return (connectionLogId);
}
/*******************************************************************************
** Fluent setter for connectionLogId.
*******************************************************************************/
public FileGatewayTransferLog withConnectionLogId(Integer connectionLogId)
{
this.connectionLogId = connectionLogId;
return (this);
}
/*******************************************************************************
** Getter for operation.
*******************************************************************************/
public String getOperation()
{
return (operation);
}
/*******************************************************************************
** Fluent setter for operation.
*******************************************************************************/
public FileGatewayTransferLog withOperation(String operation)
{
this.operation = operation;
return (this);
}
/*******************************************************************************
** Getter for path.
*******************************************************************************/
public String getPath()
{
return (path);
}
/*******************************************************************************
** Fluent setter for path.
*******************************************************************************/
public FileGatewayTransferLog withPath(String path)
{
this.path = path;
return (this);
}
/*******************************************************************************
** Getter for objectKind.
*******************************************************************************/
public String getObjectKind()
{
return (objectKind);
}
/*******************************************************************************
** Fluent setter for objectKind.
*******************************************************************************/
public FileGatewayTransferLog withObjectKind(String objectKind)
{
this.objectKind = objectKind;
return (this);
}
/*******************************************************************************
** Getter for objectId.
*******************************************************************************/
public String getObjectId()
{
return (objectId);
}
/*******************************************************************************
** Fluent setter for objectId.
*******************************************************************************/
public FileGatewayTransferLog withObjectId(String objectId)
{
this.objectId = objectId;
return (this);
}
/*******************************************************************************
** Getter for format.
*******************************************************************************/
public String getFormat()
{
return (format);
}
/*******************************************************************************
** Fluent setter for format.
*******************************************************************************/
public FileGatewayTransferLog withFormat(String format)
{
this.format = format;
return (this);
}
/*******************************************************************************
** Getter for bytes.
*******************************************************************************/
public Long getBytes()
{
return (bytes);
}
/*******************************************************************************
** Fluent setter for bytes.
*******************************************************************************/
public FileGatewayTransferLog withBytes(Long bytes)
{
this.bytes = bytes;
return (this);
}
/*******************************************************************************
** Getter for sha256.
*******************************************************************************/
public String getSha256()
{
return (sha256);
}
/*******************************************************************************
** Fluent setter for sha256.
*******************************************************************************/
public FileGatewayTransferLog withSha256(String sha256)
{
this.sha256 = sha256;
return (this);
}
/*******************************************************************************
** Getter for durationMs.
*******************************************************************************/
public Long getDurationMs()
{
return (durationMs);
}
/*******************************************************************************
** Fluent setter for durationMs.
*******************************************************************************/
public FileGatewayTransferLog withDurationMs(Long durationMs)
{
this.durationMs = durationMs;
return (this);
}
/*******************************************************************************
** Getter for outcome.
*******************************************************************************/
public String getOutcome()
{
return (outcome);
}
/*******************************************************************************
** Fluent setter for outcome.
*******************************************************************************/
public FileGatewayTransferLog withOutcome(String outcome)
{
this.outcome = outcome;
return (this);
}
/*******************************************************************************
** Getter for message.
*******************************************************************************/
public String getMessage()
{
return (message);
}
/*******************************************************************************
** Fluent setter for message.
*******************************************************************************/
public FileGatewayTransferLog withMessage(String message)
{
this.message = message;
return (this);
}
/*******************************************************************************
** Getter for clientChannel.
*******************************************************************************/
public String getClientChannel()
{
return (clientChannel);
}
/*******************************************************************************
** Fluent setter for clientChannel.
*******************************************************************************/
public FileGatewayTransferLog withClientChannel(String clientChannel)
{
this.clientChannel = clientChannel;
return (this);
}
/*******************************************************************************
** Getter for uploadJobId.
*******************************************************************************/
public Integer getUploadJobId()
{
return (uploadJobId);
}
/*******************************************************************************
** Fluent setter for uploadJobId.
*******************************************************************************/
public FileGatewayTransferLog withUploadJobId(Integer uploadJobId)
{
this.uploadJobId = uploadJobId;
return (this);
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setId(Integer id)
{
this.id = id;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setCreateDate(Instant createDate)
{
this.createDate = createDate;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setModifyDate(Instant modifyDate)
{
this.modifyDate = modifyDate;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setAccountId(Integer accountId)
{
this.accountId = accountId;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setConnectionLogId(Integer connectionLogId)
{
this.connectionLogId = connectionLogId;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setOperation(String operation)
{
this.operation = operation;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setPath(String path)
{
this.path = path;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setObjectKind(String objectKind)
{
this.objectKind = objectKind;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setObjectId(String objectId)
{
this.objectId = objectId;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setFormat(String format)
{
this.format = format;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setBytes(Long bytes)
{
this.bytes = bytes;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setSha256(String sha256)
{
this.sha256 = sha256;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setDurationMs(Long durationMs)
{
this.durationMs = durationMs;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setOutcome(String outcome)
{
this.outcome = outcome;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setMessage(String message)
{
this.message = message;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setClientChannel(String clientChannel)
{
this.clientChannel = clientChannel;
}
/*******************************************************************************
** Setter used by QRecordEntity population.
*******************************************************************************/
public void setUploadJobId(Integer uploadJobId)
{
this.uploadJobId = uploadJobId;
}
}
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,63 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.permissions;
import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables;
import com.kingsrook.qqq.backend.core.actions.permissions.CustomPermissionChecker;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QPermissionDeniedException;
import com.kingsrook.qqq.backend.core.model.actions.AbstractActionInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.delete.DeleteInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.MetaDataWithPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.processes.QProcessMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.session.QSession;
/*******************************************************************************
** Read with view or admin; only administrators write configuration tables.
*******************************************************************************/
public class FileGatewayPermissionChecker implements CustomPermissionChecker
{
/*******************************************************************************
** Check the current user against the gateway permission model.
*******************************************************************************/
@Override
public void checkPermissionsThrowing(AbstractActionInput input, MetaDataWithPermissionRules metadata) throws QPermissionDeniedException
{
boolean write = metadata instanceof QProcessMetaData || input instanceof InsertInput || input instanceof UpdateInput || input instanceof DeleteInput;
if(write && metadata instanceof QTableMetaData table && FileGatewayTables.SYSTEM_WRITTEN.contains(table.getName()))
{
throw (new QPermissionDeniedException("This table is written only by the file gateway"));
}
QSession session = QContext.getQSession();
if(session == null || !(session.hasPermission(FileGatewayPermissions.ADMIN) || (!write && session.hasPermission(FileGatewayPermissions.VIEW))))
{
throw (new QPermissionDeniedException("Missing file gateway permission"));
}
}
}
@@ -0,0 +1,45 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.permissions;
import java.util.List;
/*******************************************************************************
** Permissions granted by the host application.
*******************************************************************************/
public final class FileGatewayPermissions
{
public static final String VIEW = "file_gateway.view";
public static final String ADMIN = "file_gateway.admin";
public static final String CONNECT = "file_gateway.connect";
public static final List<String> ALL = List.of(VIEW, ADMIN, CONNECT);
/*******************************************************************************
** Constants only.
*******************************************************************************/
private FileGatewayPermissions()
{
}
}
@@ -0,0 +1,232 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.sftp;
import java.io.Serializable;
import java.time.Instant;
import java.util.List;
import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables;
import com.kingsrook.qqq.backend.core.actions.QBackendTransaction;
import com.kingsrook.qqq.backend.core.actions.permissions.CorePermissionEnforcer;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.actions.tables.QueryAction;
import com.kingsrook.qqq.backend.core.actions.tables.UpdateAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
/*******************************************************************************
** System-session access limited to the gateway own tables; application data uses the UI API.
*******************************************************************************/
public class AuditWriter
{
private final QInstance instance;
private final ThreadLocal<QBackendTransaction> transaction = new ThreadLocal<>();
/*******************************************************************************
** Bind the gateway database instance.
*******************************************************************************/
public AuditWriter(QInstance instance)
{
this.instance = instance;
}
/*******************************************************************************
** Work within a short system-session scope.
*******************************************************************************/
@FunctionalInterface
public interface Work<T>
{
/*******************************************************************************
** Execute the database operation.
*******************************************************************************/
T run() throws QException;
}
/*******************************************************************************
** Preserve caller context while writing gateway-owned state.
*******************************************************************************/
public <T> T system(Work<T> work) throws QException
{
var previous = QContext.capture();
try
{
QContext.init(instance, new QSystemUserSession(instance));
try(var ignored = CorePermissionEnforcer.openAuthorizedScope())
{
return (work.run());
}
}
finally
{
QContext.init(previous);
}
}
/*******************************************************************************
** Serialize account authentication across pods with a transactional row update.
** Re-reading after acquiring the write lock sees committed counters and sessions.
*******************************************************************************/
public <T> T withAccountLock(String username, Work<T> work) throws QException
{
return (withAccountLock("username", username, work));
}
/*******************************************************************************
** Lock one account row without changing any user-managed fields.
*******************************************************************************/
private <T> T withAccountLock(String field, Serializable value, Work<T> work) throws QException
{
return (system(() ->
{
try(var tx = QBackendTransaction.openFor(new QueryInput(FileGatewayTables.ACCOUNT)))
{
transaction.set(tx);
try
{
var rows = query(FileGatewayTables.ACCOUNT, new QQueryFilter(new QFilterCriteria(field, QCriteriaOperator.EQUALS, value)));
if(!rows.isEmpty())
{
update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", rows.getFirst().getValue("id")).withValue("modifyDate", Instant.now()));
}
T result = work.run();
tx.commit();
return (result);
}
catch(Exception e)
{
tx.rollback();
throw (new QException("Gateway account transaction failed", e));
}
finally
{
transaction.remove();
}
}
}));
}
/*******************************************************************************
** Charge an account-wide minute window atomically, including other gateway pods.
*******************************************************************************/
public boolean operationAllowed(Integer accountId, Instant now, int maximum) throws QException
{
return (withAccountLock("id", accountId, () ->
{
var rows = query(FileGatewayTables.ACCOUNT, new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.EQUALS, accountId)));
if(rows.isEmpty())
{
throw (new QException("Gateway account no longer exists"));
}
QRecord account = rows.getFirst();
Instant window = account.getValueInstant("operationWindowAt");
Integer previous = account.getValueInteger("operationWindowCount");
boolean reset = window == null || !window.plusSeconds(60).isAfter(now);
int count = reset ? 1 : (previous == null ? 0 : previous) + 1;
update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", accountId)
.withValue("operationWindowAt", reset ? now : window).withValue("operationWindowCount", count));
return (count <= maximum);
}));
}
/*******************************************************************************
** Reject accidental use on application data.
*******************************************************************************/
private void checkTable(String table)
{
if(!FileGatewayTables.ALL.contains(table))
{
throw (new IllegalArgumentException("System access is limited to gateway tables"));
}
}
/*******************************************************************************
** Read gateway state.
*******************************************************************************/
public List<QRecord> query(String table, QQueryFilter filter) throws QException
{
checkTable(table);
return (system(() -> new QueryAction().execute(new QueryInput(table).withFilter(filter).withTransaction(transaction.get())).getRecords()));
}
/*******************************************************************************
** Insert and require successful persistence.
*******************************************************************************/
public QRecord insert(String table, QRecord record) throws QException
{
checkTable(table);
QRecord result = system(() -> new InsertAction().execute(new InsertInput(table).withRecord(record).withTransaction(transaction.get())).getRecords().get(0));
checkErrors(result);
return (result);
}
/*******************************************************************************
** Update gateway state.
*******************************************************************************/
public void update(String table, QRecord record) throws QException
{
checkTable(table);
QRecord result = system(() -> new UpdateAction().execute(new UpdateInput(table).withRecords(List.of(record)).withTransaction(transaction.get())).getRecords().get(0));
checkErrors(result);
}
/*******************************************************************************
** Fail closed on bookkeeping errors without printing records.
*******************************************************************************/
private void checkErrors(QRecord record) throws QException
{
if(record.getErrors() != null && !record.getErrors().isEmpty())
{
throw (new QException("Unable to persist gateway state"));
}
}
}
@@ -0,0 +1,105 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.sftp;
import java.net.InetAddress;
/*******************************************************************************
** Numeric IPv4 and IPv6 CIDR matching; hostnames never trigger DNS resolution.
*******************************************************************************/
public final class CidrMatcher
{
/*******************************************************************************
** Utility only.
*******************************************************************************/
private CidrMatcher()
{
}
/*******************************************************************************
** Match an address against comma or whitespace separated networks.
*******************************************************************************/
public static boolean matches(String address, String cidrs)
{
if(cidrs == null || cidrs.isBlank())
{
return (true);
}
try
{
byte[] candidate = numeric(address);
for(String cidr : cidrs.split("[,\\s]+"))
{
String[] parts = cidr.split("/", -1);
byte[] network = numeric(parts[0]);
int bits = parts.length == 1 ? network.length * 8 : Integer.parseInt(parts[1]);
if(parts.length > 2 || bits < 0 || bits > network.length * 8)
{
return (false);
}
if(candidate.length != network.length)
{
continue;
}
boolean match = true;
for(int i = 0; i < bits; i++)
{
int mask = 1 << (7 - i % 8);
if((candidate[i / 8] & mask) != (network[i / 8] & mask))
{
match = false;
break;
}
}
if(match)
{
return (true);
}
}
}
catch(Exception e)
{
return (false);
}
return (false);
}
/*******************************************************************************
** Parse numeric addresses only.
*******************************************************************************/
private static byte[] numeric(String address) throws Exception
{
if(address == null || !address.matches("[0-9a-fA-F:.]+") || (!address.contains(":") && !address.matches("[0-9]{1,3}(\\.[0-9]{1,3}){3}")))
{
throw (new IllegalArgumentException("Expected a numeric IP address"));
}
return (InetAddress.getByName(address).getAddress());
}
}
@@ -0,0 +1,37 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.sftp;
import java.security.PublicKey;
import org.apache.sshd.server.session.ServerSession;
/*******************************************************************************
** Protocol authentication boundary for public-key credentials.
*******************************************************************************/
public interface FileGatewayAuthenticatorInterface
{
/*******************************************************************************
** Resolve and authenticate a protocol credential.
*******************************************************************************/
boolean authenticate(String username, PublicKey key, ServerSession session);
}
@@ -0,0 +1,46 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.sftp;
import java.io.IOException;
/*******************************************************************************
** Lifecycle of a gateway protocol listener.
*******************************************************************************/
public interface FileGatewayProtocolServer
{
/*******************************************************************************
** Start listening.
*******************************************************************************/
void start() throws IOException;
/*******************************************************************************
** Close the listener and live sessions.
*******************************************************************************/
void stop() throws IOException;
/*******************************************************************************
** Number of open sessions.
*******************************************************************************/
int activeSessions();
}
@@ -0,0 +1,129 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.sftp;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.time.Clock;
import java.time.Duration;
import java.time.Instant;
import java.util.UUID;
import com.kingsrook.qbits.filegateway.permissions.FileGatewayPermissions;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleDispatcher;
import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInProcessSessions;
import org.apache.sshd.common.AttributeRepository.AttributeKey;
import org.apache.sshd.server.session.ServerSession;
/*******************************************************************************
** One SSH connection, its audit identity and refreshable in-process credentials.
*******************************************************************************/
public class GatewaySession
{
public static final AttributeKey<GatewaySession> ATTRIBUTE = new AttributeKey<>();
public static final AttributeKey<String> TOKEN = new AttributeKey<>();
public final String id = UUID.randomUUID().toString();
public final ServerSession ssh;
public final String remoteIp;
public Integer connectionId;
public QRecord account;
public QRecord key;
public long operations;
public String disconnectReason = "Client disconnected";
private Instant mintedAt;
private final QInstance instance;
private final Clock clock;
/*******************************************************************************
** Bind an accepted SSH connection.
*******************************************************************************/
public GatewaySession(QInstance instance, ServerSession ssh, Clock clock)
{
this.instance = instance;
this.ssh = ssh;
this.clock = clock;
this.remoteIp = ((InetSocketAddress) ssh.getClientAddress()).getAddress().getHostAddress();
}
/*******************************************************************************
** Mint via the host authentication module and refresh permissions every five minutes.
*******************************************************************************/
public synchronized String token() throws QException
{
if(account == null || ssh.isClosing() || !ssh.isOpen())
{
throw (new QException("Unauthenticated connection"));
}
if(mintedAt == null || !mintedAt.plus(Duration.ofMinutes(5)).isAfter(clock.instant()))
{
var module = new QAuthenticationModuleDispatcher().getQModule(instance.getAuthentication());
var user = new AuditWriter(instance).system(() -> module.createAutomatedSessionForUser(instance, account.getValueString("userId")));
if(user == null || !user.hasPermission(FileGatewayPermissions.CONNECT))
{
revoke();
throw (new QException("no file_gateway.connect"));
}
String next = UiApiInProcessSessions.register(user, Duration.ofMinutes(10));
UiApiInProcessSessions.revoke(ssh.getAttribute(TOKEN));
ssh.setAttribute(TOKEN, next);
mintedAt = clock.instant();
}
return (ssh.getAttribute(TOKEN));
}
/*******************************************************************************
** Release credentials immediately on disconnect or revocation.
*******************************************************************************/
public synchronized void revoke()
{
UiApiInProcessSessions.revoke(ssh.removeAttribute(TOKEN));
mintedAt = null;
}
/*******************************************************************************
** Count operations locally and charge the shared account minute window.
*******************************************************************************/
public synchronized boolean operationAllowed() throws IOException
{
operations++;
Integer maximum = account.getValueInteger("maxOperationsPerMinute");
try
{
return (new AuditWriter(instance).operationAllowed(account.getValueInteger("id"), clock.instant(), maximum == null ? 600 : maximum));
}
catch(QException e)
{
throw (new IOException("Unable to verify account operation limit", e));
}
}
}
@@ -0,0 +1,273 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.sftp;
import java.io.IOException;
import java.nio.file.FileSystem;
import java.nio.file.Path;
import java.security.KeyPair;
import java.time.Clock;
import java.time.Duration;
import java.util.List;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import com.kingsrook.qbits.filegateway.FileGatewayQBitConfig;
import com.kingsrook.qbits.filegateway.vfs.GatewayFileSystem;
import com.kingsrook.qbits.filegateway.vfs.GatewayUiApiClient;
import com.kingsrook.qbits.filegateway.vfs.VirtualTree;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInvoker;
import org.apache.sshd.common.cipher.BuiltinCiphers;
import org.apache.sshd.common.file.FileSystemFactory;
import org.apache.sshd.common.kex.BuiltinDHFactories;
import org.apache.sshd.common.keyprovider.KeyPairProvider;
import org.apache.sshd.common.mac.BuiltinMacs;
import org.apache.sshd.common.session.Session;
import org.apache.sshd.common.session.SessionContext;
import org.apache.sshd.common.session.SessionListener;
import org.apache.sshd.common.signature.BuiltinSignatures;
import org.apache.sshd.core.CoreModuleProperties;
import org.apache.sshd.server.ServerBuilder;
import org.apache.sshd.server.SshServer;
import org.apache.sshd.server.auth.pubkey.UserAuthPublicKeyFactory;
import org.apache.sshd.server.forward.RejectAllForwardingFilter;
import org.apache.sshd.server.session.ServerSession;
import org.apache.sshd.sftp.server.SftpSubsystemFactory;
/*******************************************************************************
** Public-key-only SFTP with explicit modern algorithms and an entirely virtual filesystem.
*******************************************************************************/
public class SftpProtocolServer implements FileGatewayProtocolServer
{
private static final QLogger LOG = QLogger.getLogger(SftpProtocolServer.class);
private final SshServer server;
private final SshKeyAuthenticator authenticator;
private final Duration pollInterval;
private ScheduledExecutorService poller;
/*******************************************************************************
** Configure the listener, security policy and injected runtime dependencies.
*******************************************************************************/
public SftpProtocolServer(QInstance instance, FileGatewayQBitConfig config, UiApiInvoker invoker, KeyPair hostKey, Clock clock, Duration pollInterval, String podName)
{
this.pollInterval = pollInterval;
AuditWriter audit = new AuditWriter(instance);
authenticator = new SshKeyAuthenticator(instance, config, audit, clock, podName);
server = SshServer.setUpDefaultServer();
server.setPort(config.getSftpPort());
server.setKeyPairProvider(KeyPairProvider.wrap(hostKey));
server.setCipherFactories(List.of(BuiltinCiphers.cc20p1305_openssh, BuiltinCiphers.aes128gcm, BuiltinCiphers.aes256gcm, BuiltinCiphers.aes128ctr, BuiltinCiphers.aes192ctr, BuiltinCiphers.aes256ctr));
server.setMacFactories(List.of(BuiltinMacs.hmacsha256etm, BuiltinMacs.hmacsha512etm));
server.setKeyExchangeFactories(List.of(BuiltinDHFactories.curve25519, BuiltinDHFactories.curve25519_libssh, BuiltinDHFactories.ecdhp256, BuiltinDHFactories.ecdhp384, BuiltinDHFactories.ecdhp521, BuiltinDHFactories.dhg16_512, BuiltinDHFactories.dhg18_512)
.stream().filter(BuiltinDHFactories::isSupported).map(ServerBuilder.DH2KEX).toList());
server.setSignatureFactories(List.of(BuiltinSignatures.ed25519, BuiltinSignatures.nistp256, BuiltinSignatures.nistp384, BuiltinSignatures.nistp521, BuiltinSignatures.rsaSHA256, BuiltinSignatures.rsaSHA512));
server.setPasswordAuthenticator(null);
server.setKeyboardInteractiveAuthenticator(null);
server.setGSSAuthenticator(null);
server.setHostBasedAuthenticator(null);
server.setUserAuthFactories(List.of(UserAuthPublicKeyFactory.INSTANCE));
server.setPublickeyAuthenticator(authenticator);
server.setShellFactory(null);
server.setCommandFactory(null);
server.setForwardingFilter(RejectAllForwardingFilter.INSTANCE);
server.setAgentFactory(null);
server.setSubsystemFactories(List.of(new SftpSubsystemFactory.Builder().build()));
CoreModuleProperties.IDLE_TIMEOUT.set(server, Duration.ofMinutes(15));
CoreModuleProperties.AUTH_TIMEOUT.set(server, Duration.ofSeconds(30));
server.setFileSystemFactory(new FileSystemFactory()
{
/*******************************************************************************
** Create the session virtual root.
*******************************************************************************/
@Override
public Path getUserHomeDir(SessionContext session) throws IOException
{
return (createFileSystem(session).getPath("/"));
}
/*******************************************************************************
** No native filesystem is consulted.
*******************************************************************************/
@Override
public FileSystem createFileSystem(SessionContext session)
{
GatewaySession state = session.getAttribute(GatewaySession.ATTRIBUTE);
return (new GatewayFileSystem(new VirtualTree(new GatewayUiApiClient(invoker, state), state, audit, clock)));
}
});
server.addSessionListener(new SessionListener()
{
/*******************************************************************************
** Persist accept before authentication.
*******************************************************************************/
@Override
public void sessionCreated(Session session)
{
try
{
authenticator.accepted((ServerSession) session);
}
catch(Exception e)
{
LOG.warn("Unable to audit SFTP accept");
session.close(true);
}
}
/*******************************************************************************
** Login bookkeeping starts only after proof of private-key possession.
*******************************************************************************/
@Override
public void sessionEvent(Session session, Event event)
{
if(event == Event.Authenticated)
{
try
{
authenticator.authenticated((ServerSession) session);
}
catch(Exception e)
{
LOG.warn("Unable to finalize SFTP authentication");
session.close(true);
}
}
}
/*******************************************************************************
** Always revoke the in-process token.
*******************************************************************************/
@Override
public void sessionClosed(Session session)
{
try
{
authenticator.closed((ServerSession) session);
}
catch(Exception e)
{
LOG.warn("Unable to finalize SFTP connection audit");
}
}
});
}
/*******************************************************************************
** Start the listener and revocation heartbeat.
*******************************************************************************/
@Override
public void start() throws IOException
{
server.start();
poller = Executors.newSingleThreadScheduledExecutor(runnable ->
{
Thread thread = new Thread(runnable, "file-gateway-revocation");
thread.setDaemon(true);
return (thread);
});
poller.scheduleWithFixedDelay(() ->
{
try
{
authenticator.poll();
}
catch(Exception e)
{
LOG.warn("Unable to verify live SFTP sessions; closing them");
server.getActiveSessions().forEach(session -> session.close(true));
}
}, pollInterval.toMillis(), pollInterval.toMillis(), TimeUnit.MILLISECONDS);
}
/*******************************************************************************
** Stop polling, revoke sessions and stop the listener.
*******************************************************************************/
@Override
public void stop() throws IOException
{
if(poller != null)
{
poller.shutdownNow();
}
server.stop(true);
try
{
authenticator.shutdown();
}
catch(com.kingsrook.qqq.backend.core.exceptions.QException e)
{
throw (new IOException("Unable to finalize gateway shutdown", e));
}
}
/*******************************************************************************
** Count active SSH connections.
*******************************************************************************/
@Override
public int activeSessions()
{
return (server.getActiveSessions().size());
}
/*******************************************************************************
** Bound port, including ephemeral test ports.
*******************************************************************************/
public int getPort()
{
return (server.getPort());
}
/*******************************************************************************
** Server security configuration for inspection.
*******************************************************************************/
public SshServer getServer()
{
return (server);
}
/*******************************************************************************
** Live revocation hook for deterministic tests.
*******************************************************************************/
public SshKeyAuthenticator getAuthenticator()
{
return (authenticator);
}
}
@@ -0,0 +1,435 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.sftp;
import java.security.PublicKey;
import java.time.Clock;
import java.time.Instant;
import java.util.ArrayDeque;
import java.util.Deque;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import com.kingsrook.qbits.filegateway.FileGatewayQBitConfig;
import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables;
import com.kingsrook.qqq.backend.core.alerts.AlertSignal;
import com.kingsrook.qqq.backend.core.alerts.QAlerts;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.logging.QLogger;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import org.apache.sshd.common.config.keys.KeyUtils;
import org.apache.sshd.common.digest.BuiltinDigests;
import org.apache.sshd.server.auth.pubkey.PublickeyAuthenticator;
import org.apache.sshd.server.session.ServerSession;
/*******************************************************************************
** Database-backed SSH key authentication, account limits, IP windows and live revocation.
*******************************************************************************/
public class SshKeyAuthenticator implements PublickeyAuthenticator, FileGatewayAuthenticatorInterface
{
private static final QLogger LOG = QLogger.getLogger(SshKeyAuthenticator.class);
private final QInstance instance;
private final FileGatewayQBitConfig config;
private final AuditWriter audit;
private final Clock clock;
private final String podName;
private final Map<String, GatewaySession> sessions = new ConcurrentHashMap<>();
private final Map<String, Deque<Instant>> failures = new HashMap<>();
private final Map<String, Deque<Instant>> connections = new HashMap<>();
private final Map<String, Instant> ipLocks = new HashMap<>();
private final Map<String, String> ipLockAccounts = new HashMap<>();
private final Map<String, String> failureAccounts = new HashMap<>();
/*******************************************************************************
** Bind authentication and audit dependencies.
*******************************************************************************/
public SshKeyAuthenticator(QInstance instance, FileGatewayQBitConfig config, AuditWriter audit, Clock clock, String podName)
{
this.instance = instance;
this.config = config;
this.audit = audit;
this.clock = clock;
this.podName = podName;
}
/*******************************************************************************
** Audit TCP accept and enforce the pre-authentication connection rate.
*******************************************************************************/
public synchronized void accepted(ServerSession ssh) throws QException
{
GatewaySession state = new GatewaySession(instance, ssh, clock);
ssh.setAttribute(GatewaySession.ATTRIBUTE, state);
state.connectionId = audit.insert(FileGatewayTables.CONNECTION_LOG, new QRecord()
.withValue("sessionId", state.id).withValue("remoteIp", state.remoteIp).withValue("podName", podName)
.withValue("connectedAt", clock.instant()).withValue("lastSeenAt", clock.instant())
.withValue("clientChannel", "sftp").withValue("outcome", "AUTH_FAILED")).getValueInteger("id");
sessions.put(state.id, state);
prune(connections, clock.instant().minusSeconds(60));
Deque<Instant> window = connections.computeIfAbsent(state.remoteIp, ignored -> new ArrayDeque<>());
window.addLast(clock.instant());
if(window.size() > config.getMaxConnectionsPerIpPerMinute())
{
outcome(state, "LIMIT", "Connection rate limit exceeded");
ssh.close(true);
}
}
/*******************************************************************************
** Validate the account and offered public key, then mint the user session.
*******************************************************************************/
@Override
public synchronized boolean authenticate(String username, PublicKey publicKey, ServerSession ssh)
{
GatewaySession state = ssh.getAttribute(GatewaySession.ATTRIBUTE);
if(state == null || ssh.isClosing())
{
return (false);
}
try
{
return (audit.withAccountLock(username, () -> authenticateLocked(username, publicKey, ssh, state)));
}
catch(Exception e)
{
LOG.warn("SFTP authentication service failed", e);
state.revoke();
state.disconnectReason = "Authentication service unavailable";
ssh.close(true);
return (false);
}
}
/*******************************************************************************
** Account checks and reservations share one database transaction across pods.
*******************************************************************************/
private boolean authenticateLocked(String username, PublicKey publicKey, ServerSession ssh, GatewaySession state) throws QException
{
if(state.account != null)
{
return (username.equals(state.account.getValueString("username")) && state.key != null
&& KeyUtils.getFingerPrint(BuiltinDigests.sha256, publicKey).equals(state.key.getValueString("fingerprint")));
}
QRecord account = first(FileGatewayTables.ACCOUNT, "username", username);
String fingerprint = KeyUtils.getFingerPrint(BuiltinDigests.sha256, publicKey);
audit.update(FileGatewayTables.CONNECTION_LOG, new QRecord().withValue("id", state.connectionId)
.withValue("username", username).withValue("keyFingerprint", fingerprint).withValue("clientVersion", ssh.getClientVersion())
.withValue("accountId", account == null ? null : account.getValue("id")));
Instant now = clock.instant();
if(account != null && account.getValue("lockedUntil") == null && username.equals(ipLockAccounts.get(state.remoteIp)))
{
ipLocks.remove(state.remoteIp);
ipLockAccounts.remove(state.remoteIp);
failures.remove(state.remoteIp);
failureAccounts.remove(state.remoteIp);
}
ipLocks.entrySet().removeIf(entry -> !entry.getValue().isAfter(now));
ipLockAccounts.keySet().retainAll(ipLocks.keySet());
if(ipLocks.containsKey(state.remoteIp) || (account != null && (future(account, "lockedUntil") || "LOCKED".equals(account.getValueString("status")))))
{
return (outcome(state, "LOCKED_OUT", "Authentication locked"));
}
if(account == null)
{
return (failed(state, null, "AUTH_FAILED", "Unknown account or key"));
}
if(!"ACTIVE".equals(account.getValueString("status")))
{
return (failed(state, account, "DISABLED", "Account disabled"));
}
if(expired(account, "expiresAt"))
{
return (failed(state, account, "EXPIRED", "Account expired"));
}
if(!CidrMatcher.matches(state.remoteIp, account.getValueString("allowedCidrs")))
{
return (failed(state, account, "IP_DENIED", "Address outside allowlist"));
}
QRecord key = first(FileGatewayTables.SSH_KEY, "fingerprint", fingerprint);
if(key == null || !account.getValue("id").equals(key.getValue("accountId")) || key.getValue("revokedAt") != null)
{
return (failed(state, account, "AUTH_FAILED", "Unknown or revoked key"));
}
if(expired(key, "expiresAt"))
{
return (failed(state, account, "EXPIRED", "Key expired"));
}
int limit = account.getValueInteger("maxConcurrentSessions") == null ? config.getDefaultMaxConcurrentSessions() : account.getValueInteger("maxConcurrentSessions");
////////////////////////////////////////////////////////////////////////
// only live rows: connected, not disconnected, seen in the last 3 min //
////////////////////////////////////////////////////////////////////////
QQueryFilter liveFilter = filter("accountId", account.getValue("id"))
.withCriteria(new QFilterCriteria("outcome", QCriteriaOperator.EQUALS, "CONNECTED"))
.withCriteria(new QFilterCriteria("disconnectedAt", QCriteriaOperator.IS_BLANK))
.withCriteria(new QFilterCriteria("lastSeenAt", QCriteriaOperator.GREATER_THAN, now.minusSeconds(180)));
long active = audit.query(FileGatewayTables.CONNECTION_LOG, liveFilter).size();
if(active >= limit)
{
return (outcome(state, "LIMIT", "Concurrent session limit exceeded"));
}
state.account = account;
state.key = key;
try
{
state.token();
}
catch(QException e)
{
state.account = null;
state.key = null;
return (failed(state, account, "AUTH_FAILED", "no file_gateway.connect"));
}
outcome(state, "CONNECTED", null);
return (true);
}
/*******************************************************************************
** MINA has verified the signature, not just accepted an unsigned key probe.
*******************************************************************************/
public synchronized void authenticated(ServerSession ssh) throws QException
{
GatewaySession state = ssh.getAttribute(GatewaySession.ATTRIBUTE);
if(state == null || state.account == null)
{
throw (new QException("Missing authenticated gateway identity"));
}
QRecord account = state.account;
QRecord key = state.key;
Instant now = clock.instant();
audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id"))
.withValue("failedLoginCount", 0).withValue("failedLoginWindowAt", null)
.withValue("lastLoginAt", now).withValue("lastLoginIp", state.remoteIp));
audit.update(FileGatewayTables.SSH_KEY, new QRecord().withValue("id", key.getValue("id"))
.withValue("lastUsedAt", now).withValue("lastUsedIp", state.remoteIp));
}
/*******************************************************************************
** Persist account failure windows and apply the per-IP lockout.
*******************************************************************************/
private boolean failed(GatewaySession state, QRecord account, String result, String reason) throws QException
{
Instant now = clock.instant();
prune(failures, now.minus(config.getLockoutWindow()));
failureAccounts.keySet().retainAll(failures.keySet());
Deque<Instant> window = failures.computeIfAbsent(state.remoteIp, ignored -> new ArrayDeque<>());
String username = account == null ? "" : account.getValueString("username");
if(window.isEmpty())
{
failureAccounts.put(state.remoteIp, username);
}
else if(!username.equals(failureAccounts.get(state.remoteIp)))
{
failureAccounts.put(state.remoteIp, "");
}
window.addLast(now);
boolean lock = window.size() >= config.getLockoutThreshold();
if(lock)
{
ipLocks.put(state.remoteIp, now.plus(config.getLockoutDuration()));
ipLockAccounts.put(state.remoteIp, failureAccounts.get(state.remoteIp));
}
if(account != null)
{
Instant start = account.getValueInstant("failedLoginWindowAt");
Integer oldCount = account.getValueInteger("failedLoginCount");
int count = start == null || !start.plus(config.getLockoutWindow()).isAfter(now) ? 1 : (oldCount == null ? 0 : oldCount) + 1;
QRecord update = new QRecord().withValue("id", account.getValue("id")).withValue("failedLoginCount", count)
.withValue("failedLoginWindowAt", count == 1 ? now : start);
if(count >= config.getLockoutThreshold())
{
update.setValue("lockedUntil", now.plus(config.getLockoutDuration()));
lock = true;
}
audit.update(FileGatewayTables.ACCOUNT, update);
}
if(lock)
{
audit.system(() ->
{
QAlerts.raise(new AlertSignal("filegateway.lockout").withDedupKey(account == null ? state.remoteIp : account.getValueString("id"))
.withSummary("File gateway authentication locked"));
return (null);
});
}
return (outcome(state, result, reason));
}
/*******************************************************************************
** Record the authentication outcome without credentials.
*******************************************************************************/
private boolean outcome(GatewaySession state, String outcome, String reason) throws QException
{
audit.update(FileGatewayTables.CONNECTION_LOG, new QRecord().withValue("id", state.connectionId)
.withValue("outcome", outcome).withValue("disconnectReason", reason).withValue("lastSeenAt", clock.instant()));
if(reason != null)
{
state.disconnectReason = reason;
}
return (false);
}
/*******************************************************************************
** Revoke credentials and finalize connection accounting.
*******************************************************************************/
public synchronized void closed(ServerSession ssh) throws QException
{
GatewaySession state = ssh.removeAttribute(GatewaySession.ATTRIBUTE);
if(state == null)
{
return;
}
state.revoke();
sessions.remove(state.id);
if(state.account != null && !ssh.isAuthenticated())
{
outcome(state, "AUTH_FAILED", "Public key signature was not verified");
}
if(state.connectionId != null)
{
audit.update(FileGatewayTables.CONNECTION_LOG, new QRecord().withValue("id", state.connectionId)
.withValue("disconnectedAt", clock.instant()).withValue("disconnectReason", state.disconnectReason)
.withValue("bytesIn", 0L).withValue("bytesOut", 0L)
.withValue("operationCount", state.operations));
}
}
/*******************************************************************************
** Finish any callbacks still queued when MINA has stopped accepting work.
*******************************************************************************/
public synchronized void shutdown() throws QException
{
for(GatewaySession state : List.copyOf(sessions.values()))
{
closed(state.ssh);
}
}
/*******************************************************************************
** Re-read live accounts and keys and heartbeat open connection rows.
*******************************************************************************/
public synchronized void poll() throws QException
{
for(GatewaySession state : List.copyOf(sessions.values()))
{
if(state.account == null)
{
continue;
}
QRecord account = first(FileGatewayTables.ACCOUNT, "id", state.account.getValue("id"));
QRecord key = first(FileGatewayTables.SSH_KEY, "id", state.key.getValue("id"));
QRecord connection = first(FileGatewayTables.CONNECTION_LOG, "id", state.connectionId);
if(account == null || key == null || !"ACTIVE".equals(account.getValueString("status"))
|| expired(account, "expiresAt") || expired(key, "expiresAt") || key.getValue("revokedAt") != null
|| connection == null || connection.getValue("disconnectedAt") != null)
{
state.disconnectReason = connection != null && connection.getValueString("disconnectReason") != null
? connection.getValueString("disconnectReason") : "Account or key revoked";
state.revoke();
state.ssh.close(true);
}
else
{
state.account = account;
audit.update(FileGatewayTables.CONNECTION_LOG, new QRecord().withValue("id", state.connectionId)
.withValue("lastSeenAt", clock.instant()).withValue("operationCount", state.operations));
}
}
}
/*******************************************************************************
** Find gateway-owned state.
*******************************************************************************/
private QRecord first(String table, String field, java.io.Serializable value) throws QException
{
return (audit.query(table, filter(field, value)).stream().findFirst().orElse(null));
}
/*******************************************************************************
** Equality query for owned tables.
*******************************************************************************/
private QQueryFilter filter(String field, java.io.Serializable value)
{
return (new QQueryFilter(new QFilterCriteria(field, QCriteriaOperator.EQUALS, value)));
}
/*******************************************************************************
** Whether a timestamp lies in the future.
*******************************************************************************/
private boolean future(QRecord row, String field)
{
Instant value = row.getValueInstant(field);
return (value != null && value.isAfter(clock.instant()));
}
/*******************************************************************************
** Whether an optional expiry has passed.
*******************************************************************************/
private boolean expired(QRecord row, String field)
{
Instant value = row.getValueInstant(field);
return (value != null && !value.isAfter(clock.instant()));
}
/*******************************************************************************
** Bound in-memory IP windows to their active interval.
*******************************************************************************/
private void prune(Map<String, Deque<Instant>> windows, Instant cutoff)
{
windows.values().forEach(window -> window.removeIf(instant -> !instant.isAfter(cutoff)));
windows.entrySet().removeIf(entry -> entry.getValue().isEmpty());
}
}
@@ -0,0 +1,169 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.vfs;
import java.nio.file.attribute.FileTime;
import java.nio.file.attribute.GroupPrincipal;
import java.nio.file.attribute.PosixFileAttributes;
import java.nio.file.attribute.PosixFilePermission;
import java.nio.file.attribute.UserPrincipal;
import java.util.Set;
/*******************************************************************************
** Virtual attributes: generated files have size zero and read-only permissions.
*******************************************************************************/
public record GatewayFileAttributes(VirtualTree.Node node, FileTime time) implements PosixFileAttributes
{
/*******************************************************************************
** Virtual attribute lastModifiedTime.
*******************************************************************************/
@Override
public FileTime lastModifiedTime()
{
return (time);
}
/*******************************************************************************
** Virtual attribute lastAccessTime.
*******************************************************************************/
@Override
public FileTime lastAccessTime()
{
return (time);
}
/*******************************************************************************
** Virtual attribute creationTime.
*******************************************************************************/
@Override
public FileTime creationTime()
{
return (time);
}
/*******************************************************************************
** Virtual attribute isRegularFile.
*******************************************************************************/
@Override
public boolean isRegularFile()
{
return (!node.directory());
}
/*******************************************************************************
** Virtual attribute isDirectory.
*******************************************************************************/
@Override
public boolean isDirectory()
{
return (node.directory());
}
/*******************************************************************************
** Virtual attribute isSymbolicLink.
*******************************************************************************/
@Override
public boolean isSymbolicLink()
{
return (false);
}
/*******************************************************************************
** Virtual attribute isOther.
*******************************************************************************/
@Override
public boolean isOther()
{
return (false);
}
/*******************************************************************************
** Virtual attribute size.
*******************************************************************************/
@Override
public long size()
{
return (0);
}
/*******************************************************************************
** Virtual attribute fileKey.
*******************************************************************************/
@Override
public Object fileKey()
{
return (node.path());
}
/*******************************************************************************
** Virtual attribute owner.
*******************************************************************************/
@Override
public UserPrincipal owner()
{
return (() -> "gateway");
}
/*******************************************************************************
** Virtual attribute group.
*******************************************************************************/
@Override
public GroupPrincipal group()
{
return (() -> "gateway");
}
/*******************************************************************************
** Virtual attribute permissions.
*******************************************************************************/
@Override
public Set<PosixFilePermission> permissions()
{
return (node.directory() ? Set.of(PosixFilePermission.OWNER_READ, PosixFilePermission.OWNER_EXECUTE) : Set.of(PosixFilePermission.OWNER_READ));
}
}
@@ -0,0 +1,201 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.vfs;
import java.nio.file.ClosedFileSystemException;
import java.nio.file.FileStore;
import java.nio.file.FileSystem;
import java.nio.file.Path;
import java.nio.file.PathMatcher;
import java.nio.file.WatchService;
import java.nio.file.attribute.UserPrincipalLookupService;
import java.nio.file.spi.FileSystemProvider;
import java.util.List;
import java.util.Set;
/*******************************************************************************
** Session-private NIO filesystem over a UI API virtual tree, with no native mounts.
*******************************************************************************/
public class GatewayFileSystem extends FileSystem
{
private final VirtualTree tree;
private final GatewayFileSystemProvider provider;
private boolean open = true;
/*******************************************************************************
** Create a purely virtual filesystem.
*******************************************************************************/
public GatewayFileSystem(VirtualTree tree)
{
this.tree = tree;
this.provider = new GatewayFileSystemProvider(this);
}
/*******************************************************************************
** Access the live virtual tree.
*******************************************************************************/
public VirtualTree tree()
{
if(!open)
{
throw (new ClosedFileSystemException());
}
return (tree);
}
/*******************************************************************************
** Virtual filesystem provider.
*******************************************************************************/
@Override
public FileSystemProvider provider()
{
return (provider);
}
/*******************************************************************************
** Virtual filesystem close.
*******************************************************************************/
@Override
public void close()
{
open = false;
}
/*******************************************************************************
** Virtual filesystem isOpen.
*******************************************************************************/
@Override
public boolean isOpen()
{
return (open);
}
/*******************************************************************************
** Virtual filesystem isReadOnly.
*******************************************************************************/
@Override
public boolean isReadOnly()
{
return (true);
}
/*******************************************************************************
** Virtual filesystem getSeparator.
*******************************************************************************/
@Override
public String getSeparator()
{
return ("/");
}
/*******************************************************************************
** Virtual filesystem getRootDirectories.
*******************************************************************************/
@Override
public Iterable<Path> getRootDirectories()
{
return (List.of(getPath("/")));
}
/*******************************************************************************
** Virtual filesystem getFileStores.
*******************************************************************************/
@Override
public Iterable<FileStore> getFileStores()
{
return (List.of());
}
/*******************************************************************************
** Virtual filesystem supportedFileAttributeViews.
*******************************************************************************/
@Override
public Set<String> supportedFileAttributeViews()
{
return (Set.of("basic", "posix"));
}
/*******************************************************************************
** Virtual filesystem getPath.
*******************************************************************************/
@Override
public Path getPath(String first, String... more)
{
return (new GatewayPath(this, first + (more.length == 0 ? "" : "/" + String.join("/", more))));
}
/*******************************************************************************
** Virtual filesystem getPathMatcher.
*******************************************************************************/
@Override
public PathMatcher getPathMatcher(String syntaxAndPattern)
{
throw (new UnsupportedOperationException("Path matching is unavailable"));
}
/*******************************************************************************
** Virtual filesystem getUserPrincipalLookupService.
*******************************************************************************/
@Override
public UserPrincipalLookupService getUserPrincipalLookupService()
{
throw (new UnsupportedOperationException("Native users are unavailable"));
}
/*******************************************************************************
** Virtual filesystem newWatchService.
*******************************************************************************/
@Override
public WatchService newWatchService()
{
throw (new UnsupportedOperationException("Watching is unavailable"));
}
}
@@ -0,0 +1,385 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.vfs;
import java.io.IOException;
import java.net.URI;
import java.nio.channels.SeekableByteChannel;
import java.nio.file.AccessMode;
import java.nio.file.CopyOption;
import java.nio.file.DirectoryStream;
import java.nio.file.FileStore;
import java.nio.file.FileSystem;
import java.nio.file.FileSystemAlreadyExistsException;
import java.nio.file.LinkOption;
import java.nio.file.OpenOption;
import java.nio.file.Path;
import java.nio.file.ProviderMismatchException;
import java.nio.file.StandardOpenOption;
import java.nio.file.attribute.BasicFileAttributes;
import java.nio.file.attribute.FileAttribute;
import java.nio.file.attribute.FileAttributeView;
import java.nio.file.attribute.FileTime;
import java.nio.file.spi.FileSystemProvider;
import java.time.Instant;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.Set;
/*******************************************************************************
** NIO operations resolve against the virtual tree; every mutation is denied and audited.
*******************************************************************************/
public class GatewayFileSystemProvider extends FileSystemProvider
{
private final GatewayFileSystem fileSystem;
/*******************************************************************************
** Bind one virtual filesystem.
*******************************************************************************/
public GatewayFileSystemProvider(GatewayFileSystem fileSystem)
{
this.fileSystem = fileSystem;
}
/*******************************************************************************
** Canonical virtual path in this session.
*******************************************************************************/
private String name(Path path)
{
if(path.getFileSystem() != fileSystem)
{
throw (new ProviderMismatchException());
}
return (path.toAbsolutePath().normalize().toString());
}
/*******************************************************************************
** Virtual provider getScheme.
*******************************************************************************/
@Override
public String getScheme()
{
return ("gateway");
}
/*******************************************************************************
** Virtual provider newFileSystem.
*******************************************************************************/
@Override
public FileSystem newFileSystem(URI uri, Map<String, ?> env)
{
throw (new FileSystemAlreadyExistsException());
}
/*******************************************************************************
** Virtual provider getFileSystem.
*******************************************************************************/
@Override
public FileSystem getFileSystem(URI uri)
{
return (fileSystem);
}
/*******************************************************************************
** Virtual provider getPath.
*******************************************************************************/
@Override
public Path getPath(URI uri)
{
return (fileSystem.getPath(uri.getPath()));
}
/*******************************************************************************
** Virtual provider newByteChannel.
*******************************************************************************/
@Override
public SeekableByteChannel newByteChannel(Path path, Set<? extends OpenOption> options, FileAttribute<?>... attrs) throws IOException
{
boolean write = options.contains(StandardOpenOption.WRITE) || options.contains(StandardOpenOption.APPEND) || options.contains(StandardOpenOption.CREATE) || options.contains(StandardOpenOption.CREATE_NEW);
if(!write)
{
fileSystem.tree().stat(name(path));
}
fileSystem.tree().deny(name(path), write ? "UPLOAD" : "DOWNLOAD", "not yet supported");
throw (new AssertionError());
}
/*******************************************************************************
** Virtual provider newDirectoryStream.
*******************************************************************************/
@Override
public DirectoryStream<Path> newDirectoryStream(Path dir, DirectoryStream.Filter<? super Path> filter) throws IOException
{
List<Path> entries = new ArrayList<>();
for(VirtualTree.Node node : fileSystem.tree().list(name(dir)))
{
Path path = fileSystem.getPath(node.path());
if(filter.accept(path))
{
entries.add(path);
}
}
return (new GatewayDirectoryStream(entries));
}
/*******************************************************************************
** Virtual provider createDirectory.
*******************************************************************************/
@Override
public void createDirectory(Path dir, FileAttribute<?>... attrs) throws IOException
{
fileSystem.tree().deny(name(dir), "UPLOAD", "mkdir is not supported");
}
/*******************************************************************************
** Virtual provider delete.
*******************************************************************************/
@Override
public void delete(Path path) throws IOException
{
fileSystem.tree().deny(name(path), "REMOVE", "Removal is not supported");
}
/*******************************************************************************
** Virtual provider copy.
*******************************************************************************/
@Override
public void copy(Path source, Path target, CopyOption... options) throws IOException
{
fileSystem.tree().deny(name(target), "UPLOAD", "Copy is not supported");
}
/*******************************************************************************
** Virtual provider move.
*******************************************************************************/
@Override
public void move(Path source, Path target, CopyOption... options) throws IOException
{
fileSystem.tree().deny(name(source), "RENAME", "Rename is not supported");
}
/*******************************************************************************
** Virtual provider isSameFile.
*******************************************************************************/
@Override
public boolean isSameFile(Path path, Path path2)
{
return (name(path).equals(name(path2)));
}
/*******************************************************************************
** Virtual provider isHidden.
*******************************************************************************/
@Override
public boolean isHidden(Path path)
{
return (false);
}
/*******************************************************************************
** Virtual provider getFileStore.
*******************************************************************************/
@Override
public FileStore getFileStore(Path path)
{
throw (new UnsupportedOperationException("No native store"));
}
/*******************************************************************************
** Virtual provider checkAccess.
*******************************************************************************/
@Override
public void checkAccess(Path path, AccessMode... modes) throws IOException
{
fileSystem.tree().stat(name(path));
for(AccessMode mode : modes)
{
if(mode == AccessMode.WRITE)
{
fileSystem.tree().deny(name(path), "UPLOAD", "Writes are not supported");
}
}
}
/*******************************************************************************
** Virtual provider getFileAttributeView.
*******************************************************************************/
@Override
public <V extends FileAttributeView> V getFileAttributeView(Path path, Class<V> type, LinkOption... options)
{
return (null);
}
/*******************************************************************************
** Virtual provider readAttributes.
*******************************************************************************/
@Override
public <A extends BasicFileAttributes> A readAttributes(Path path, Class<A> type, LinkOption... options) throws IOException
{
return (type.cast(new GatewayFileAttributes(fileSystem.tree().stat(name(path)), FileTime.from(Instant.now()))));
}
/*******************************************************************************
** Virtual provider readAttributes.
*******************************************************************************/
@Override
public Map<String, Object> readAttributes(Path path, String attributes, LinkOption... options) throws IOException
{
var attrs = readAttributes(path, GatewayFileAttributes.class, options);
Map<String, Object> result = new HashMap<>();
result.put("isDirectory", attrs.isDirectory());
result.put("isRegularFile", attrs.isRegularFile());
result.put("isSymbolicLink", false);
result.put("isOther", false);
result.put("size", 0L);
result.put("fileKey", attrs.fileKey());
result.put("lastModifiedTime", attrs.lastModifiedTime());
result.put("lastAccessTime", attrs.lastAccessTime());
result.put("creationTime", attrs.creationTime());
result.put("permissions", attrs.permissions());
result.put("owner", attrs.owner());
result.put("group", attrs.group());
return (result);
}
/*******************************************************************************
** Virtual provider setAttribute.
*******************************************************************************/
@Override
public void setAttribute(Path path, String attribute, Object value, LinkOption... options) throws IOException
{
fileSystem.tree().deny(name(path), "UPLOAD", "setattr is not supported");
}
/*******************************************************************************
** Virtual provider createSymbolicLink.
*******************************************************************************/
@Override
public void createSymbolicLink(Path link, Path target, FileAttribute<?>... attrs) throws IOException
{
fileSystem.tree().deny(name(link), "UPLOAD", "Links are not supported");
}
/*******************************************************************************
** Virtual provider createLink.
*******************************************************************************/
@Override
public void createLink(Path link, Path existing) throws IOException
{
fileSystem.tree().deny(name(link), "UPLOAD", "Links are not supported");
}
/*******************************************************************************
** Single-use directory snapshot held entirely in memory.
*******************************************************************************/
private static class GatewayDirectoryStream implements DirectoryStream<Path>
{
private final List<Path> entries;
private boolean used;
/*******************************************************************************
** Directory snapshot lifecycle.
*******************************************************************************/
GatewayDirectoryStream(List<Path> entries)
{
this.entries = entries;
}
/*******************************************************************************
** Directory snapshot lifecycle.
*******************************************************************************/
@Override
public Iterator<Path> iterator()
{
if(used)
{
throw (new IllegalStateException("Directory stream is closed or already iterated"));
}
used = true;
return (entries.iterator());
}
/*******************************************************************************
** Directory snapshot lifecycle.
*******************************************************************************/
@Override
public void close()
{
used = true;
}
}
}
@@ -0,0 +1,371 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.vfs;
import java.io.File;
import java.io.IOException;
import java.net.URI;
import java.nio.file.FileSystem;
import java.nio.file.LinkOption;
import java.nio.file.Path;
import java.nio.file.ProviderMismatchException;
import java.nio.file.WatchEvent;
import java.nio.file.WatchKey;
import java.nio.file.WatchService;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Deque;
import java.util.Iterator;
import java.util.List;
/*******************************************************************************
** A path made solely from virtual name segments; never delegates to a native filesystem.
*******************************************************************************/
public final class GatewayPath implements Path
{
private final GatewayFileSystem fileSystem;
private final String value;
/*******************************************************************************
** Construct a virtual path.
*******************************************************************************/
public GatewayPath(GatewayFileSystem fileSystem, String value)
{
this.fileSystem = fileSystem;
this.value = value.replaceAll("/+", "/").replaceAll("(?<=.)/$", "");
}
/*******************************************************************************
** Individual name segments.
*******************************************************************************/
private List<String> parts()
{
return (value.isEmpty() || "/".equals(value) ? List.of() : Arrays.asList(value.replaceFirst("^/", "").split("/")));
}
/*******************************************************************************
** Virtual path getFileSystem.
*******************************************************************************/
@Override
public FileSystem getFileSystem()
{
return (fileSystem);
}
/*******************************************************************************
** Virtual path isAbsolute.
*******************************************************************************/
@Override
public boolean isAbsolute()
{
return (value.startsWith("/"));
}
/*******************************************************************************
** Virtual path getRoot.
*******************************************************************************/
@Override
public Path getRoot()
{
return (isAbsolute() ? new GatewayPath(fileSystem, "/") : null);
}
/*******************************************************************************
** Virtual path getFileName.
*******************************************************************************/
@Override
public Path getFileName()
{
List<String> parts = parts();
return (parts.isEmpty() ? null : new GatewayPath(fileSystem, parts.getLast()));
}
/*******************************************************************************
** Virtual path getParent.
*******************************************************************************/
@Override
public Path getParent()
{
int slash = value.lastIndexOf('/');
return (slash < 0 || "/".equals(value) ? null : new GatewayPath(fileSystem, slash == 0 ? "/" : value.substring(0, slash)));
}
/*******************************************************************************
** Virtual path getNameCount.
*******************************************************************************/
@Override
public int getNameCount()
{
return (parts().size());
}
/*******************************************************************************
** Virtual path getName.
*******************************************************************************/
@Override
public Path getName(int index)
{
return (new GatewayPath(fileSystem, parts().get(index)));
}
/*******************************************************************************
** Virtual path subpath.
*******************************************************************************/
@Override
public Path subpath(int beginIndex, int endIndex)
{
return (new GatewayPath(fileSystem, String.join("/", parts().subList(beginIndex, endIndex))));
}
/*******************************************************************************
** Virtual path startsWith.
*******************************************************************************/
@Override
public boolean startsWith(Path other)
{
return (other instanceof GatewayPath p && p.fileSystem == fileSystem && (value.equals(p.value) || value.startsWith(p.value.endsWith("/") ? p.value : p.value + "/")));
}
/*******************************************************************************
** Virtual path endsWith.
*******************************************************************************/
@Override
public boolean endsWith(Path other)
{
return (other instanceof GatewayPath p && p.fileSystem == fileSystem && (value.equals(p.value) || (!p.isAbsolute() && value.endsWith("/" + p.value))));
}
/*******************************************************************************
** Virtual path normalize.
*******************************************************************************/
@Override
public Path normalize()
{
Deque<String> names = new ArrayDeque<>();
for(String part : parts())
{
if(part.equals(".") || part.isEmpty())
{
continue;
}
if(part.equals("..") && !names.isEmpty() && !names.getLast().equals(".."))
{
names.removeLast();
}
else if(!part.equals("..") || !isAbsolute())
{
names.addLast(part);
}
}
return (new GatewayPath(fileSystem, (isAbsolute() ? "/" : "") + String.join("/", names)));
}
/*******************************************************************************
** Virtual path resolve.
*******************************************************************************/
@Override
public Path resolve(Path other)
{
if(!(other instanceof GatewayPath p) || p.fileSystem != fileSystem)
{
throw (new ProviderMismatchException());
}
return (other.isAbsolute() ? other : new GatewayPath(fileSystem, value + (value.endsWith("/") || value.isEmpty() ? "" : "/") + other));
}
/*******************************************************************************
** Virtual path relativize.
*******************************************************************************/
@Override
public Path relativize(Path other)
{
if(!(other instanceof GatewayPath p) || p.fileSystem != fileSystem || isAbsolute() != p.isAbsolute())
{
throw (new IllegalArgumentException("Incompatible virtual paths"));
}
List<String> left = ((GatewayPath) normalize()).parts();
List<String> right = ((GatewayPath) p.normalize()).parts();
int common = 0;
while(common < left.size() && common < right.size() && left.get(common).equals(right.get(common)))
{
common++;
}
List<String> result = new ArrayList<>();
for(int i = common; i < left.size(); i++)
{
result.add("..");
}
result.addAll(right.subList(common, right.size()));
return (new GatewayPath(fileSystem, String.join("/", result)));
}
/*******************************************************************************
** Virtual path toUri.
*******************************************************************************/
@Override
public URI toUri()
{
try
{
return (new URI("gateway", null, toAbsolutePath().toString(), null));
}
catch(java.net.URISyntaxException e)
{
throw (new IllegalArgumentException(e));
}
}
/*******************************************************************************
** Virtual path toAbsolutePath.
*******************************************************************************/
@Override
public Path toAbsolutePath()
{
return (isAbsolute() ? this : new GatewayPath(fileSystem, "/" + value));
}
/*******************************************************************************
** Virtual path toRealPath.
*******************************************************************************/
@Override
public Path toRealPath(LinkOption... options) throws IOException
{
Path path = toAbsolutePath().normalize();
fileSystem.tree().stat(path.toString());
return (path);
}
/*******************************************************************************
** Virtual path toFile.
*******************************************************************************/
@Override
public File toFile()
{
throw (new UnsupportedOperationException("Virtual paths have no native file"));
}
/*******************************************************************************
** Virtual path register.
*******************************************************************************/
@Override
public WatchKey register(WatchService watcher, WatchEvent.Kind<?>[] events, WatchEvent.Modifier... modifiers)
{
throw (new UnsupportedOperationException("Virtual paths cannot be watched"));
}
/*******************************************************************************
** Virtual path iterator.
*******************************************************************************/
@Override
public Iterator<Path> iterator()
{
return (parts().stream().map(name -> (Path) new GatewayPath(fileSystem, name)).iterator());
}
/*******************************************************************************
** Virtual path compareTo.
*******************************************************************************/
@Override
public int compareTo(Path other)
{
return (value.compareTo(other.toString()));
}
/*******************************************************************************
** Virtual path equals.
*******************************************************************************/
@Override
public boolean equals(Object other)
{
return (other instanceof GatewayPath path && path.fileSystem == fileSystem && path.value.equals(value));
}
/*******************************************************************************
** Virtual path hashCode.
*******************************************************************************/
@Override
public int hashCode()
{
return (31 * System.identityHashCode(fileSystem) + value.hashCode());
}
/*******************************************************************************
** Virtual path toString.
*******************************************************************************/
@Override
public String toString()
{
return (value);
}
}
@@ -0,0 +1,126 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.vfs;
import java.io.IOException;
import java.nio.file.AccessDeniedException;
import java.nio.file.NoSuchFileException;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.node.JsonNodeFactory;
import com.kingsrook.qbits.filegateway.sftp.GatewaySession;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QCriteriaOperator;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterCriteria;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QFilterOrderBy;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiCredentials;
import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInvoker;
import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiRequest;
import com.kingsrook.qqq.openapi.model.HttpMethod;
/*******************************************************************************
** Every visible object comes from a spec-defined UI API operation as the signed-in user.
*******************************************************************************/
public class GatewayUiApiClient
{
private final UiApiInvoker invoker;
private final GatewaySession session;
/*******************************************************************************
** Bind this client to one SFTP identity.
*******************************************************************************/
public GatewayUiApiClient(UiApiInvoker invoker, GatewaySession session)
{
this.invoker = invoker;
this.session = session;
}
/*******************************************************************************
** Get the permission-filtered app tree.
*******************************************************************************/
public JsonNode metadata() throws IOException
{
return (invoke(new UiApiRequest().withMethod(HttpMethod.GET).withPath("/metaData")));
}
/*******************************************************************************
** Query visible saved objects of a table.
*******************************************************************************/
public JsonNode savedObjects(String source, String table) throws IOException
{
var records = JsonNodeFactory.instance.arrayNode();
int pageSize = 250;
for(int skip = 0; ; skip += pageSize)
{
QQueryFilter filter = new QQueryFilter(new QFilterCriteria("tableName", QCriteriaOperator.EQUALS, table))
.withOrderBy(new QFilterOrderBy("id", true)).withLimit(pageSize).withSkip(skip);
JsonNode page = invoke(new UiApiRequest().withMethod(HttpMethod.POST).withPath("/table/{tableName}/query")
.withPathParam("tableName", source).withJsonBody(java.util.Map.of("filter", filter))).path("records");
page.forEach(records::add);
if(page.size() < pageSize)
{
return (records);
}
}
}
/*******************************************************************************
** Apply the fixed channel identity, credentials and contract check on every call.
*******************************************************************************/
protected JsonNode invoke(UiApiRequest request) throws IOException
{
if(invoker.findOperation(request.getMethod(), request.getPath()).isEmpty())
{
throw (new IOException("Gateway requires a spec-defined UI API operation"));
}
try
{
var response = invoker.invoke(request.withCredentials(UiApiCredentials.ofInProcessSession(session.token()))
.withInvokedBy("file-gateway").withClientDetail(session.id));
if(response.getStatus() == 404)
{
throw (new NoSuchFileException(request.getPath()));
}
if(response.getStatus() == 401 || response.getStatus() == 403)
{
throw (new AccessDeniedException(request.getPath()));
}
if(!response.isSuccess())
{
throw (new IOException("UI API request failed with status " + response.getStatus()));
}
return (response.getJson());
}
catch(com.kingsrook.qqq.backend.core.exceptions.QException e)
{
throw (new IOException("User session unavailable", e));
}
}
}
@@ -0,0 +1,397 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway.vfs;
import java.io.IOException;
import java.nio.file.AccessDeniedException;
import java.nio.file.NoSuchFileException;
import java.time.Clock;
import java.time.Instant;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import com.fasterxml.jackson.databind.JsonNode;
import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables;
import com.kingsrook.qbits.filegateway.sftp.AuditWriter;
import com.kingsrook.qbits.filegateway.sftp.GatewaySession;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
/*******************************************************************************
** Permission-filtered virtual app and table tree with per-session in-memory caches.
*******************************************************************************/
public class VirtualTree
{
/*******************************************************************************
** A virtual directory or generated file.
*******************************************************************************/
public record Node(String path, boolean directory, String kind, String objectId)
{
}
private final GatewayUiApiClient client;
private final GatewaySession session;
private final AuditWriter audit;
private final Clock clock;
private JsonNode metadata;
private Instant metadataAt;
private final Map<String, JsonNode> saved = new HashMap<>();
private final Map<String, Instant> savedAt = new HashMap<>();
/*******************************************************************************
** Create a session-private tree and caches.
*******************************************************************************/
public VirtualTree(GatewayUiApiClient client, GatewaySession session, AuditWriter audit, Clock clock)
{
this.client = client;
this.session = session;
this.audit = audit;
this.clock = clock;
}
/*******************************************************************************
** Cache user metadata for sixty seconds.
*******************************************************************************/
private JsonNode metadata() throws IOException
{
if(metadataAt == null || !metadataAt.plusSeconds(60).isAfter(clock.instant()))
{
metadata = client.metadata();
metadataAt = clock.instant();
}
return (metadata);
}
/*******************************************************************************
** Cache saved objects for thirty seconds.
*******************************************************************************/
private JsonNode saved(String source, String table) throws IOException
{
String key = source + "/" + table;
if(!savedAt.containsKey(key) || !savedAt.get(key).plusSeconds(30).isAfter(clock.instant()))
{
saved.put(key, client.savedObjects(source, table));
savedAt.put(key, clock.instant());
}
return (saved.get(key));
}
/*******************************************************************************
** Resolve only through visible parents; hidden paths are indistinguishable from missing ones.
*******************************************************************************/
public synchronized Node stat(String path) throws IOException
{
sessionToken();
if("/".equals(path))
{
return (new Node("/", true, "APP", null));
}
int split = path.lastIndexOf('/');
String parent = split == 0 ? "/" : path.substring(0, split);
return (children(parent).stream().filter(node -> node.path().equals(path)).findFirst()
.orElseThrow(() -> new NoSuchFileException(path)));
}
/*******************************************************************************
** Refresh permissions even when a listing uses cached metadata.
*******************************************************************************/
private void sessionToken() throws IOException
{
try
{
session.token();
}
catch(Exception e)
{
throw (new AccessDeniedException("User session unavailable"));
}
}
/*******************************************************************************
** Audit each directory open once.
*******************************************************************************/
public synchronized List<Node> list(String path) throws IOException
{
long started = System.nanoTime();
try
{
sessionToken();
if(!session.operationAllowed())
{
log(path, "LIST", "LIMIT", null, started);
throw (new IOException("Operation limit exceeded"));
}
List<Node> nodes = children(path);
log(path, "LIST", "OK", null, started);
return (nodes);
}
catch(NoSuchFileException e)
{
log(path, "LIST", "NOT_FOUND", null, started);
throw (e);
}
catch(AccessDeniedException e)
{
log(path, "LIST", "DENIED", null, started);
throw (e);
}
}
/*******************************************************************************
** Resolve app hierarchy and phase-a table folders from UI API metadata.
*******************************************************************************/
private List<Node> children(String path) throws IOException
{
JsonNode meta = metadata();
List<Node> result = new ArrayList<>();
if("/".equals(path))
{
addAppChildren(result, "", meta.path("appTree"), meta);
return (result);
}
String[] segments = path.substring(1).split("/");
JsonNode children = meta.path("appTree");
String prefix = "";
String table = null;
int index = 0;
for(; index < segments.length; index++)
{
JsonNode found = null;
for(JsonNode child : children)
{
if(segments[index].equals(child.path("name").asText()))
{
found = child;
break;
}
}
if(found == null)
{
throw (new NoSuchFileException(path));
}
prefix += "/" + segments[index];
if("TABLE".equalsIgnoreCase(found.path("type").asText()) && meta.path("tables").has(segments[index]))
{
table = segments[index++];
break;
}
if(!"APP".equalsIgnoreCase(found.path("type").asText()))
{
throw (new NoSuchFileException(path));
}
children = found.path("children");
}
if(table == null)
{
addAppChildren(result, prefix, children, meta);
return (result);
}
List<String> folders = new ArrayList<>(List.of("results", "archive"));
boolean profiles = meta.path("tables").has("savedBulkLoadProfile") && meta.path("processes").has(table + ".bulkInsert");
if(profiles && !saved("savedBulkLoadProfile", table).isEmpty())
{
folders.add("upload");
}
if(meta.path("tables").has("savedView"))
{
folders.add("views");
}
if(meta.path("tables").has("savedReport") && meta.path("processes").has("renderSavedReport"))
{
folders.add("reports");
}
if(index == segments.length)
{
for(String folder : folders)
{
result.add(new Node(prefix + "/" + folder, true, folderKind(folder), null));
}
return (result);
}
String folder = segments[index++];
if(!folders.contains(folder))
{
throw (new NoSuchFileException(path));
}
if("results".equals(folder) || "archive".equals(folder))
{
if(index != segments.length)
{
throw (new NoSuchFileException(path));
}
return (result);
}
String source = "upload".equals(folder) ? "savedBulkLoadProfile" : "views".equals(folder) ? "savedView" : "savedReport";
JsonNode records = saved(source, table);
Map<String, Integer> counts = new HashMap<>();
for(JsonNode row : records)
{
counts.merge(slug(row.path("values").path("label").asText()), 1, Integer::sum);
}
for(JsonNode row : records)
{
String id = row.path("values").path("id").asText();
String name = slug(row.path("values").path("label").asText());
if(counts.get(name) > 1)
{
name += "-" + id;
}
String item = prefix + "/" + folder + "/" + name;
if("upload".equals(folder))
{
result.add(new Node(item, true, "PROFILE", id));
}
else
{
String formats = session.account.getValueString("formats");
for(String format : (formats == null ? "csv,xlsx" : formats).split("[,\\s]+"))
{
if(List.of("csv", "xlsx", "json").contains(format))
{
result.add(new Node(item + "." + format, false, folderKind(folder), id));
}
}
}
}
if(index == segments.length)
{
return (result);
}
if(index + 1 == segments.length && "upload".equals(folder) && result.stream().anyMatch(node -> node.path().equals(path)))
{
return (List.of());
}
throw (new NoSuchFileException(path));
}
/*******************************************************************************
** Expose only app and readable table nodes.
*******************************************************************************/
private void addAppChildren(List<Node> result, String prefix, JsonNode children, JsonNode meta)
{
for(JsonNode child : children)
{
String type = child.path("type").asText().toUpperCase(Locale.ROOT);
String name = child.path("name").asText();
if(("APP".equals(type) || (!prefix.isEmpty() && "TABLE".equals(type) && meta.path("tables").has(name)))
&& !name.isEmpty() && !name.contains("/") && !name.equals(".") && !name.equals(".."))
{
result.add(new Node(prefix + "/" + name, true, type, name));
}
}
}
/*******************************************************************************
** Map virtual folders to persisted audit kinds.
*******************************************************************************/
private String folderKind(String folder)
{
return (switch(folder)
{
case "upload" -> "PROFILE";
case "views" -> "VIEW";
case "reports" -> "REPORT";
case "results" -> "RESULT";
default -> "ARCHIVE";
});
}
/*******************************************************************************
** Stable lowercase label slug, at most eighty characters before a collision suffix.
*******************************************************************************/
public static String slug(String label)
{
String value = label.toLowerCase(Locale.ROOT).replaceAll("[^a-z0-9]+", "-").replaceAll("^-|-$", "");
if(value.length() > 80)
{
value = value.substring(0, 80);
}
return (value.isEmpty() ? "item" : value);
}
/*******************************************************************************
** Audit and deny unsupported operations.
*******************************************************************************/
public void deny(String path, String operation, String message) throws IOException
{
session.operationAllowed();
log(path, operation, "DENIED", message, System.nanoTime());
throw (new AccessDeniedException(path, null, message));
}
/*******************************************************************************
** Persist the operation through the system audit writer.
*******************************************************************************/
private void log(String path, String operation, String outcome, String message, long started) throws IOException
{
String kind = "TABLE";
String id = null;
try
{
Node node = stat(path);
kind = node.kind();
id = node.objectId();
}
catch(IOException ignored)
{
// Missing targets have no object id.
}
try
{
audit.insert(FileGatewayTables.TRANSFER_LOG, new QRecord().withValue("accountId", session.account.getValue("id"))
.withValue("connectionLogId", session.connectionId).withValue("operation", operation).withValue("path", path)
.withValue("objectKind", kind).withValue("objectId", id).withValue("bytes", 0L)
.withValue("durationMs", (System.nanoTime() - started) / 1_000_000).withValue("outcome", outcome)
.withValue("message", message).withValue("clientChannel", "sftp/" + session.id));
}
catch(Exception e)
{
throw (new IOException("Unable to persist operation audit", e));
}
}
}
@@ -0,0 +1,136 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway;
import java.io.ByteArrayInputStream;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.time.Instant;
import com.kingsrook.qbits.filegateway.api.FileAcceptance;
import com.kingsrook.qbits.filegateway.api.FileGatewayConsumer;
import com.kingsrook.qbits.filegateway.api.FileGatewayConsumersMetaData;
import com.kingsrook.qbits.filegateway.api.FileGatewayOutbox;
import com.kingsrook.qbits.filegateway.api.FileGatewayUnavailableException;
import com.kingsrook.qbits.filegateway.api.InboundFile;
import com.kingsrook.qbits.filegateway.api.OutboxFileRef;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import org.junit.jupiter.api.Test;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Independently usable consumer and outbox contracts.
*******************************************************************************/
class ApiContractTest
{
/*******************************************************************************
** The public API may depend on core, but never gateway implementation types.
*******************************************************************************/
@Test
void apiImportsAreIsolated() throws Exception
{
try(var sources = Files.list(Path.of("src/main/java/com/kingsrook/qbits/filegateway/api")))
{
for(Path source : sources.filter(p -> p.toString().endsWith(".java")).toList())
{
String text = Files.readString(source);
assertThat(text).as(source.toString()).doesNotContainPattern("com\\.kingsrook\\.qbits\\.filegateway\\.(?!api[.;])");
}
}
}
/*******************************************************************************
** Fresh streams, immutable decisions, consumer registration and provider dispatch.
*******************************************************************************/
@Test
void consumerAndOutboxContract() throws Exception
{
var file = new InboundFile(1, "orders", "alice", "A", "127.0.0.1", "/orders/a.csv", "a.csv", 3, "sha", Instant.EPOCH,
"bucket", "key", () -> new ByteArrayInputStream(new byte[] {1, 2, 3}));
assertThat(file.openStream()).isNotSameAs(file.openStream());
assertThat(file.openStream().readAllBytes()).containsExactly(1, 2, 3);
assertThat(file.remoteAddress()).isEqualTo("127.0.0.1");
FileGatewayConsumer consumer = new FileGatewayConsumer()
{
/*******************************************************************************
** Get name.
*******************************************************************************/
public String getName()
{
return ("test");
}
/*******************************************************************************
** Accept.
*******************************************************************************/
public FileAcceptance accept(InboundFile inbound)
{
return (FileAcceptance.accepted(inbound.sha256()));
}
};
assertThat(consumer.getName()).isEqualTo("test");
var accepted = consumer.accept(file);
assertThat(accepted.getStatus()).isEqualTo(FileAcceptance.Status.ACCEPTED);
assertThat(accepted.getConsumerReference()).isEqualTo("sha");
assertThat(accepted.getReason()).isNull();
var rejected = FileAcceptance.rejected("invalid");
assertThat(rejected.getStatus()).isEqualTo(FileAcceptance.Status.REJECTED);
assertThat(rejected.getReason()).isEqualTo("invalid");
var retry = FileAcceptance.retryLater("busy", Duration.ofMinutes(1));
assertThat(retry.getStatus()).isEqualTo(FileAcceptance.Status.RETRY_LATER);
assertThat(retry.getRetryAfter()).isEqualTo(Duration.ofMinutes(1));
var ref = new OutboxFileRef(2, "orders", "/orders/b.csv", "b.csv", 3, "sha", Instant.EPOCH);
consumer.onDownloaded(ref);
FileGatewayOutbox.setProvider(null);
assertThat(FileGatewayOutbox.isAvailable()).isFalse();
assertThatThrownBy(() -> FileGatewayOutbox.put("orders", "b.csv", file.openStream(), 3)).isInstanceOf(FileGatewayUnavailableException.class);
try
{
FileGatewayOutbox.setProvider((endpoint, name, stream, size) ->
{
assertThat(endpoint).isEqualTo("orders");
assertThat(name).isEqualTo("b.csv");
assertThat(size).isEqualTo(3);
return (ref);
});
assertThat(FileGatewayOutbox.isAvailable()).isTrue();
assertThat(FileGatewayOutbox.put("orders", "b.csv", file.openStream(), 3)).isSameAs(ref);
}
finally
{
FileGatewayOutbox.setProvider(null);
}
var instance = new QInstance();
var registry = FileGatewayConsumersMetaData.ofOrWithNew(instance);
var code = new QCodeReference(consumer.getClass());
assertThat(registry.withConsumer(code)).isSameAs(registry);
assertThat(registry.getName()).isEqualTo("fileGatewayConsumers");
assertThat(FileGatewayConsumersMetaData.ofOrWithNew(instance)).isSameAs(registry);
assertThat(registry.getConsumers()).containsExactly(code);
assertThatThrownBy(() -> registry.getConsumers().clear()).isInstanceOf(UnsupportedOperationException.class);
}
}
@@ -0,0 +1,59 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway;
import com.kingsrook.qbits.filegateway.sftp.CidrMatcher;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.CsvSource;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** IPv4 and IPv6 boundaries, multiple networks and numeric-only fail-closed parsing.
*******************************************************************************/
class CidrMatcherTest
{
/*******************************************************************************
** Networks never require hostname resolution or accept invalid prefix lengths.
*******************************************************************************/
@ParameterizedTest
@CsvSource({
"192.0.2.255,192.0.2.0/24,true",
"192.0.3.0,192.0.2.0/24,false",
"127.0.0.1,0.0.0.0/0,true",
"127.0.0.1,127.0.0.1/32,true",
"127.0.0.1,127.0.0.1/33,false",
"127.0.0.1,127.0.0.1/-1,false",
"127.0.0.1,deadbeef/32,false",
"127.0.0.1,127.0.0/24,false",
"127.0.0.1,'192.0.2.0/24,127.0.0.0/8',true",
"2001:db8::ffff,2001:db8::/32,true",
"2001:db9::1,2001:db8::/32,false",
"::1,::1/128,true",
"::1,::1/129,false"
})
/*******************************************************************************
** Matches numeric networks.
*******************************************************************************/
void matchesNumericNetworks(String address, String cidrs, boolean allowed)
{
assertThat(CidrMatcher.matches(address, cidrs)).isEqualTo(allowed);
}
}
@@ -0,0 +1,99 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway;
import java.lang.reflect.Modifier;
import java.time.Instant;
import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables;
import com.kingsrook.qqq.backend.core.model.data.QField;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.data.QRecordEntity;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** The stable schema and record entities must round-trip every persisted field.
*******************************************************************************/
class EntityContractTest
{
/*******************************************************************************
** Verify annotations, metadata completeness and fluent accessors against QRecord conversion.
*******************************************************************************/
@ParameterizedTest
@ValueSource(strings = {"Account", "SshKey", "Alias", "UploadJob", "OutboxFile", "TransferLog", "ConnectionLog"})
void entityRoundTrip(String suffix) throws Exception
{
Class<?> type = Class.forName("com.kingsrook.qbits.filegateway.model.FileGateway" + suffix);
var entity = (QRecordEntity) type.getConstructor().newInstance();
String tableName = (String) type.getField("TABLE_NAME").get(null);
var table = FileGatewayTables.tables(new FileGatewayQBitConfig()).stream().filter(t -> t.getName().equals(tableName)).findFirst().orElseThrow();
for(var field : type.getDeclaredFields())
{
if(Modifier.isStatic(field.getModifiers()))
{
continue;
}
assertThat(field.getAnnotation(QField.class)).isNotNull();
assertThat(table.getField(field.getName())).as(field.getName()).isNotNull();
Object value = field.getType() == String.class ? "value-" + field.getName() : field.getType() == Instant.class ? Instant.parse("2026-10-13T12:34:56Z")
: field.getType() == Boolean.class ? true : field.getType() == Long.class ? (Object) 12345678901L : 17;
String capitalized = Character.toUpperCase(field.getName().charAt(0)) + field.getName().substring(1);
assertThat(type.getMethod("with" + capitalized, field.getType()).invoke(entity, value)).isSameAs(entity);
assertThat(type.getMethod("get" + capitalized).invoke(entity)).isEqualTo(value);
}
QRecord record = entity.toQRecord();
var restored = (QRecordEntity) type.getConstructor(QRecord.class).newInstance(record);
assertThat(restored.toQRecord().getValues()).containsAllEntriesOf(record.getValues());
assertThat(record.getValues().keySet()).containsExactlyInAnyOrderElementsOf(table.getFields().keySet());
}
/*******************************************************************************
** All documented defaults are available before any runtime has started.
*******************************************************************************/
@Test
void configurationDefaultsAndFluentOptions() throws Exception
{
var config = new FileGatewayQBitConfig();
assertThat(config.getSftpEnabled()).isFalse();
assertThat(config.getSftpPort()).isEqualTo(2222);
assertThat(config.getAllowGeneratedHostKey()).isFalse();
assertThat(config.getHostKeyEnvVarName()).isEqualTo("FILE_GATEWAY_SSH_HOST_KEY");
assertThat(config.getDefaultMaxConcurrentSessions()).isEqualTo(4);
assertThat(config.getDefaultMaxOperationsPerMinute()).isEqualTo(600);
assertThat(config.getLockoutThreshold()).isEqualTo(5);
assertThat(config.getTemporaryUploadSuffixes()).containsExactly(".filepart", ".tmp", ".part", ".partial");
for(var method : FileGatewayQBitConfig.class.getDeclaredMethods())
{
if(!method.getName().startsWith("with"))
{
continue;
}
var getter = FileGatewayQBitConfig.class.getMethod("get" + method.getName().substring(4));
Object value = getter.invoke(config);
assertThat(method.invoke(config, value)).isSameAs(config);
assertThat(getter.invoke(config)).isEqualTo(value);
}
}
}
@@ -20,74 +20,96 @@
package com.kingsrook.qbits.filegateway;
import com.kingsrook.qqq.backend.core.model.metadata.MetaDataProducerMultiOutput;
import java.util.List;
import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.producers.MetaDataCustomizerInterface;
import com.kingsrook.qqq.backend.core.model.metadata.qbits.QBitMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.ClientChannelsMetaData;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** Verify the scaffold's registration and configuration contract.
** Registration, channel walls and contingent endpoint pickers.
*******************************************************************************/
class FileGatewayQBitProducerTest
{
/*******************************************************************************
** Produce the identity and exact config into an otherwise empty instance.
** All stable metadata belongs to one configured qbit.
*******************************************************************************/
@Test
void producesQBitWithoutTables()
void registersTablesAppPickersAndWalls() throws Exception
{
QInstance instance = new QInstance();
FileGatewayQBitConfig config = new FileGatewayQBitConfig();
FileGatewayQBitProducer producer = new FileGatewayQBitProducer().withQBitConfig(config);
MetaDataProducerMultiOutput output = producer.produce(instance);
QBitMetaData qBit = instance.getQBits().get("com.kingsrook.qbits:qbit-file-gateway");
assertThat(qBit).isNotNull();
assertThat(qBit.getGroupId()).isEqualTo("com.kingsrook.qbits");
assertThat(qBit.getArtifactId()).isEqualTo("qbit-file-gateway");
assertThat(qBit.getVersion()).isEqualTo("0.1.0");
assertThat(qBit.getConfig()).isSameAs(config);
var instance = new QInstance();
var config = new FileGatewayQBitConfig().withBackendName("host-db").withTableMetaDataCustomizer((i, t) -> t.withLabel("Custom " + t.getLabel()));
var producer = new FileGatewayQBitProducer().withQBitConfig(config).withNamespace("test");
producer.produce(instance).addSelfToInstance(instance);
assertThat(instance.getTables().keySet()).containsExactlyInAnyOrderElementsOf(FileGatewayTables.ALL);
assertThat(instance.getQBits().get("com.kingsrook.qbits:qbit-file-gateway:test").getConfig()).isSameAs(config);
assertThat(producer.getQBitConfig()).isSameAs(config);
assertThat(producer.isEnabled()).isTrue();
assertThat(producer.getNamespace()).isNull();
assertThat(output.getEach(QBitMetaData.class)).containsExactly(qBit);
assertThat(output.getEach(QTableMetaData.class)).isEmpty();
assertThat(instance.getTables()).isEmpty();
assertThat(config.getSftpEnabled()).isFalse();
assertThat(config.getSftpPort()).isEqualTo(2222);
assertThat(config.getTableMetaDataCustomizer()).isNull();
assertThat(producer.getNamespace()).isEqualTo("test");
assertThat(instance.getApps()).containsOnlyKeys("file_gateway");
assertThat(instance.getApp("file_gateway").getSections().stream().flatMap(s -> s.getTables().stream()).toList())
.containsExactlyElementsOf(FileGatewayTables.MAIN);
assertThat(instance.getJoins()).hasSize(6);
assertThat(instance.getWidgets()).hasSize(6);
assertThat(instance.getPossibleValueSources()).hasSize(19);
instance.getTables().values().forEach(t ->
{
assertThat(t.getBackendName()).isEqualTo("host-db");
assertThat(t.getLabel()).startsWith("Custom ");
assertThat(t.getField("createDate").getBackendName()).isEqualTo("create_date");
});
var channels = ClientChannelsMetaData.ofOrWithNew(instance);
assertThat(channels.getChannels()).containsKey("sftp");
for(String name : List.of("file_gateway_account", "authentik_user", "audit", "api_key", "credential", "secret", "password", "permission", "role", "user_session", "webhooks_endpoint"))
{
assertThat(channels.isDenied("sftp", name, null)).as(name).isTrue();
}
assertThat(channels.isDenied("assistant", "file_gateway_ssh_key", null)).isTrue();
assertThat(channels.isDenied("sftp", "orders", null)).isFalse();
assertThat(instance.getTable(FileGatewayTables.ALIAS).getField("tableName").getPossibleValueSourceName()).isEqualTo("tables");
}
/*******************************************************************************
** Configured instances may be namespaced; unconfigured scanning is disabled.
** Package scanning never enables an unconfigured producer.
*******************************************************************************/
@Test
void supportsNamespacedConfiguration()
void disabledWithoutConfig() throws Exception
{
assertThat(new FileGatewayQBitProducer().isEnabled()).isFalse();
MetaDataCustomizerInterface<QTableMetaData> customizer = (instance, table) -> table;
FileGatewayQBitConfig config = new FileGatewayQBitConfig()
.withTableMetaDataCustomizer(customizer)
.withSftpEnabled(true)
.withSftpPort(2022);
FileGatewayQBitProducer producer = new FileGatewayQBitProducer()
.withQBitConfig(config)
.withNamespace("test");
QInstance instance = new QInstance();
producer.produce(instance);
assertThat(producer.getNamespace()).isEqualTo("test");
assertThat(instance.getQBits().get("com.kingsrook.qbits:qbit-file-gateway:test").getConfig()).isSameAs(config);
assertThat(config.getTableMetaDataCustomizer()).isSameAs(customizer);
assertThat(config.getSftpEnabled()).isTrue();
assertThat(config.getSftpPort()).isEqualTo(2022);
var instance = new QInstance();
var producer = new FileGatewayQBitProducer();
assertThat(producer.isEnabled()).isFalse();
producer.produce(instance).addSelfToInstance(instance);
assertThat(instance.getTables()).isEmpty();
assertThat(instance.getQBits()).isEmpty();
assertThat(FileGatewayRuntime.start(instance, null)).isNull();
new FileGatewayQBitProducer().withQBitConfig(new FileGatewayQBitConfig()).produce(instance);
assertThat(FileGatewayRuntime.start(instance, null, name -> null)).isNull();
}
/*******************************************************************************
** Each dependent endpoint field has exactly the specified input criterion.
*******************************************************************************/
@ParameterizedTest
@ValueSource(strings = {"savedBulkLoadProfile", "savedView", "savedReport"})
void contingentPickerMetadata(String source) throws Exception
{
var instance = new QInstance();
new FileGatewayQBitProducer().withQBitConfig(new FileGatewayQBitConfig()).produce(instance);
var field = instance.getTable(FileGatewayTables.ALIAS).getField(source + "Id");
assertThat(field.getPossibleValueSourceName()).isEqualTo(source);
var filter = field.getPossibleValueSourceFilter();
assertThat(filter.getCriteria()).singleElement().satisfies(c ->
{
assertThat(c.getFieldName()).isEqualTo("tableName");
assertThat(c.getOperator().name()).isEqualTo("EQUALS");
assertThat(c.getValues()).containsExactly("${input.tableName}");
});
assertThat(filter.getSubFilters()).isNullOrEmpty();
}
}
@@ -0,0 +1,306 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway;
import java.io.Serializable;
import java.nio.file.Files;
import java.nio.file.Path;
import java.security.KeyPair;
import java.sql.Connection;
import java.time.Clock;
import java.time.Duration;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.UUID;
import com.kingsrook.qbits.filegateway.sftp.AuditWriter;
import com.kingsrook.qqq.backend.core.actions.tables.InsertAction;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.instances.AbstractQQQApplication;
import com.kingsrook.qqq.backend.core.instances.QInstanceEnricher;
import com.kingsrook.qqq.backend.core.instances.QInstanceValidator;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.metadata.QAuthenticationType;
import com.kingsrook.qqq.backend.core.model.metadata.QInstance;
import com.kingsrook.qqq.backend.core.model.metadata.authentication.QAuthenticationMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.fields.QFieldType;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.layout.QAppSection;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.PermissionLevel;
import com.kingsrook.qqq.backend.core.model.metadata.permissions.QPermissionRules;
import com.kingsrook.qqq.backend.core.model.metadata.tables.QTableMetaData;
import com.kingsrook.qqq.backend.core.model.metadata.tables.SectionFactory;
import com.kingsrook.qqq.backend.core.model.metadata.tables.TablesPossibleValueSourceMetaDataProvider;
import com.kingsrook.qqq.backend.core.model.savedbulkloadprofiles.SavedBulkLoadProfileMetaDataProvider;
import com.kingsrook.qqq.backend.core.model.savedreports.SavedReportsMetaDataProvider;
import com.kingsrook.qqq.backend.core.model.savedviews.SavedViewsMetaDataProvider;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.backend.core.model.session.QSystemUserSession;
import com.kingsrook.qqq.backend.core.modules.authentication.QAuthenticationModuleCustomizerInterface;
import com.kingsrook.qqq.backend.module.rdbms.jdbc.ConnectionManager;
import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSBackendMetaData;
import com.kingsrook.qqq.backend.module.rdbms.model.metadata.RDBMSTableBackendDetails;
import com.kingsrook.qqq.middleware.javalin.QApplicationJavalinServer;
import com.kingsrook.qqq.middleware.javalin.specs.v1.MiddlewareVersionV1;
import org.apache.sshd.client.SshClient;
import org.apache.sshd.client.keyverifier.AcceptAllServerKeyVerifier;
import org.apache.sshd.client.session.ClientSession;
import org.apache.sshd.common.config.keys.KeyUtils;
import org.apache.sshd.common.config.keys.PublicKeyEntry;
/*******************************************************************************
** Real strict UI API server, in-memory H2 and real MINA clients for gateway acceptance tests.
*******************************************************************************/
public class FileGatewayTestServer implements AutoCloseable
{
public final QInstance instance = new QInstance();
public final FileGatewayQBitConfig config = new FileGatewayQBitConfig().withSftpEnabled(true).withSftpPort(0);
public final AuditWriter audit = new AuditWriter(instance);
public final QApplicationJavalinServer http;
public final FileGatewayRuntime runtime;
public final SshClient client;
public final java.util.List<ApiCall> calls = new java.util.concurrent.CopyOnWriteArrayList<>();
/*******************************************************************************
** Non-secret request evidence for the invoker contract assertions.
*******************************************************************************/
public record ApiCall(String method, String path, String channel, String invokedBy)
{
}
private final Connection keeper;
public static final Map<String, Set<String>> GRANTS = new HashMap<>();
/*******************************************************************************
** Start the complete in-memory test stack.
*******************************************************************************/
public FileGatewayTestServer() throws Exception
{
this(Clock.systemUTC());
}
/*******************************************************************************
** Controllable time for expiry, session refresh and metadata caching.
*******************************************************************************/
public FileGatewayTestServer(Clock clock) throws Exception
{
ConnectionManager.resetConnectionProviders();
GRANTS.clear();
GRANTS.put("A", Set.of("file_gateway.connect", "alpha.read", "alpha.insert", "beta.read", "beta.insert"));
GRANTS.put("B", Set.of("file_gateway.connect", "beta.read", "beta.insert"));
GRANTS.put("NONE", Set.of("beta.read"));
instance.withInstanceDefaultAuthentication(new QAuthenticationMetaData().withName("test").withType(QAuthenticationType.MOCK)
.withCustomizer(new QCodeReference(TestAuthentication.class)));
RDBMSBackendMetaData backend = new RDBMSBackendMetaData().withName("rdbms").withVendor("h2").withHostName("mem")
.withDatabaseName("fg" + UUID.randomUUID().toString().replace("-", "")).withUsername("sa");
backend.setJdbcUrl(ConnectionManager.getJdbcUrl(backend) + ";DATABASE_TO_UPPER=FALSE");
instance.addBackend(backend);
keeper = new ConnectionManager().getConnection(backend);
for(String sql : Files.readString(Path.of("docs/ddl/file_gateway.sql")).split(";"))
{
if(!sql.isBlank())
{
keeper.createStatement().execute(sql);
}
}
for(String name : List.of("alpha", "beta"))
{
instance.addTable(new QTableMetaData().withName(name).withLabel(name).withBackendName("rdbms")
.withPrimaryKeyField("id").withField(new QFieldMetaData("id", QFieldType.INTEGER))
.withField(new QFieldMetaData("name", QFieldType.STRING))
.withSection(SectionFactory.defaultT1("id", "name"))
.withPermissionRules(new QPermissionRules().withLevel(PermissionLevel.READ_INSERT_EDIT_DELETE_PERMISSIONS).withPermissionBaseName(name)));
}
instance.addApp(new QAppMetaData().withName("testapp").withLabel("Test App")
.withChild(instance.getTable("alpha")).withChild(instance.getTable("beta"))
.withSection(new QAppSection().withName("data").withLabel("Data").withTables(List.of("alpha", "beta"))));
instance.addPossibleValueSource(TablesPossibleValueSourceMetaDataProvider.defineTablesPossibleValueSource(instance));
new SavedBulkLoadProfileMetaDataProvider().defineAll(instance, "rdbms", null);
new SavedViewsMetaDataProvider().defineAll(instance, "rdbms", null);
new SavedReportsMetaDataProvider().defineAll(instance, "rdbms", "rdbms", null);
new FileGatewayQBitProducer().withQBitConfig(config).produce(instance);
for(QTableMetaData table : instance.getTables().values())
{
if(!table.getName().startsWith("file_gateway_"))
{
table.withBackendDetails(new RDBMSTableBackendDetails().withTableName(table.getName()));
for(QFieldMetaData field : table.getFields().values())
{
field.setBackendName(QInstanceEnricher.inferBackendName(field.getName()));
}
createTable(table);
}
}
new QInstanceEnricher(instance).enrich();
new QInstanceValidator().validate(instance);
QContext.init(instance, new QSystemUserSession(instance));
http = new QApplicationJavalinServer(new AbstractQQQApplication()
{
/*******************************************************************************
** Define q instance.
*******************************************************************************/
@Override
public QInstance defineQInstance()
{
return (instance);
}
}).withPort(0).withServeFrontendMaterialDashboard(false).withServeFrontendNext(false)
.withJavalinConfigCustomizer(j -> j.routes.before(ctx -> calls.add(new ApiCall(ctx.method().name(), ctx.path(), ctx.header("X-QQQ-Client"), ctx.header("X-QQQ-Invoked-By")))))
.withStrictUiApiRoutes(true).withMiddlewareVersionList(List.of(new MiddlewareVersionV1()));
http.start();
runtime = FileGatewayRuntime.startForTest(instance, http, config, KeyUtils.generateKeyPair("ecdsa-sha2-nistp256", 256), clock, Duration.ofHours(1));
client = SshClient.setUpDefaultClient();
client.setServerKeyVerifier(AcceptAllServerKeyVerifier.INSTANCE);
client.start();
}
/*******************************************************************************
** Create core test tables using their actual metadata.
*******************************************************************************/
private void createTable(QTableMetaData table) throws Exception
{
StringBuilder sql = new StringBuilder("CREATE TABLE ").append(table.getName()).append(" (");
boolean first = true;
for(QFieldMetaData field : table.getFields().values())
{
if(!first)
{
sql.append(", ");
}
first = false;
String type = switch(field.getType())
{
case INTEGER -> "INTEGER";
case LONG -> "BIGINT";
case DECIMAL -> "DECIMAL";
case BOOLEAN -> "BOOLEAN";
case DATE_TIME -> "TIMESTAMP";
case DATE -> "DATE";
case TIME -> "TIME";
case BLOB -> "BINARY LARGE OBJECT";
default -> "VARCHAR(65535)";
};
sql.append(field.getBackendName()).append(" ").append(type);
if(field.getName().equals(table.getPrimaryKeyField()) && field.getType() == QFieldType.INTEGER)
{
sql.append(" GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY");
}
}
keeper.createStatement().execute(sql.append(")").toString());
}
/*******************************************************************************
** Insert an active account with schema defaults.
*******************************************************************************/
public QRecord account(String username, String userId) throws Exception
{
return (audit.insert("file_gateway_account", new QRecord().withValue("username", username).withValue("label", username).withValue("userId", userId)));
}
/*******************************************************************************
** Insert a generated key through the real validation customizer.
*******************************************************************************/
public QRecord key(QRecord account, KeyPair pair) throws Exception
{
return (audit.insert("file_gateway_ssh_key", new QRecord().withValue("accountId", account.getValue("id"))
.withValue("label", "Test key").withValue("publicKey", PublicKeyEntry.toString(pair.getPublic()))));
}
/*******************************************************************************
** Authenticate a real MINA SSH client.
*******************************************************************************/
public ClientSession login(String username, KeyPair key) throws Exception
{
ClientSession session = client.connect(username, "127.0.0.1", runtime.getProtocol().getPort()).verify(Duration.ofSeconds(10)).getSession();
session.addPublicKeyIdentity(key);
try
{
session.auth().verify(Duration.ofSeconds(10));
return (session);
}
catch(Exception e)
{
session.close(true).await(Duration.ofSeconds(5));
throw (e);
}
}
/*******************************************************************************
** Seed visible core saved-object rows under their owner.
*******************************************************************************/
public QRecord saved(String source, String table, String label) throws Exception
{
return (audit.system(() -> new InsertAction().execute(new InsertInput(source).withRecord(new QRecord()
.withValue("tableName", table).withValue("label", label).withValue("userId", "A").withValue("viewJson", "{}")
.withValue("mappingJson", "{}").withValue("columnsJson", "{\"columns\":[{\"name\":\"id\"}]}"))).getRecords().get(0)));
}
/*******************************************************************************
** Close sessions and all services.
*******************************************************************************/
@Override
public void close() throws Exception
{
client.stop();
runtime.stop();
http.stop();
keeper.close();
QContext.clear();
}
/*******************************************************************************
** MOCK automation permissions differ by user id.
*******************************************************************************/
public static class TestAuthentication implements QAuthenticationModuleCustomizerInterface
{
/*******************************************************************************
** Customize automated session for user.
*******************************************************************************/
@Override
public void customizeAutomatedSessionForUser(QInstance instance, QSession session, Serializable userId)
{
session.setPermissions(GRANTS.getOrDefault(userId.toString(), Set.of()));
}
}
}
@@ -0,0 +1,464 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway;
import java.io.ByteArrayOutputStream;
import java.nio.charset.StandardCharsets;
import java.nio.file.AccessDeniedException;
import java.nio.file.Files;
import java.nio.file.attribute.BasicFileAttributes;
import java.time.Clock;
import java.time.Duration;
import java.time.Instant;
import java.time.ZoneId;
import java.time.ZoneOffset;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.Executors;
import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables;
import com.kingsrook.qbits.filegateway.sftp.GatewaySession;
import com.kingsrook.qbits.filegateway.vfs.GatewayFileSystem;
import com.kingsrook.qbits.filegateway.vfs.VirtualTree;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInProcessSessions;
import org.apache.sshd.client.session.ClientSession;
import org.apache.sshd.common.config.keys.KeyUtils;
import org.apache.sshd.common.config.keys.writer.openssh.OpenSSHKeyPairResourceWriter;
import org.apache.sshd.sftp.client.SftpClient;
import org.apache.sshd.sftp.client.SftpClientFactory;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Time, multiple gateway listeners, runtime configuration and NIO boundaries.
*******************************************************************************/
@ExtendWith(NoDiskExtension.class)
class GatewayLifecycleTest
{
/*******************************************************************************
** Advance time deterministically without changing the machine clock.
*******************************************************************************/
static class MutableClock extends Clock
{
private Instant now = Instant.now();
/*******************************************************************************
** Get zone.
*******************************************************************************/
public ZoneId getZone()
{
return (ZoneOffset.UTC);
}
/*******************************************************************************
** With zone.
*******************************************************************************/
public Clock withZone(ZoneId zone)
{
return (this);
}
/*******************************************************************************
** Instant.
*******************************************************************************/
public Instant instant()
{
return (now);
}
/*******************************************************************************
** Advance.
*******************************************************************************/
void advance(Duration duration)
{
now = now.plus(duration);
}
}
/*******************************************************************************
** Server-side session for assertions without exposing credentials in logs.
*******************************************************************************/
private GatewaySession state(FileGatewayTestServer h)
{
return (h.runtime.getProtocol().getServer().getActiveSessions().iterator().next().getAttribute(GatewaySession.ATTRIBUTE));
}
/*******************************************************************************
** Read names while retaining the actual opendir protocol path.
*******************************************************************************/
private List<String> names(SftpClient client, String path) throws Exception
{
var names = new ArrayList<String>();
for(var entry : client.readDir(path))
{
names.add(entry.getFilename());
}
return (names);
}
/*******************************************************************************
** Re-mint at five minutes, revoke the prior token, and enforce changed grants.
*******************************************************************************/
@Test
void refreshPermissionsAndCaches() throws Exception
{
var clock = new MutableClock();
try(var h = new FileGatewayTestServer(clock))
{
var key = KeyUtils.generateKeyPair("ssh-ed25519", 256);
h.key(h.account("alice", "A"), key);
try(var session = h.login("alice", key); var sftp = SftpClientFactory.instance().createSftpClient(session))
{
assertThat(names(sftp, "/testapp")).contains("alpha", "beta");
int count = h.calls.size();
names(sftp, "/testapp");
assertThat(h.calls).hasSize(count);
String token = state(h).token();
assertThat(UiApiInProcessSessions.resolve(token)).isPresent();
FileGatewayTestServer.GRANTS.put("A", Set.of("file_gateway.connect", "beta.read"));
clock.advance(Duration.ofMinutes(5));
assertThat(names(sftp, "/testapp")).doesNotContain("alpha").contains("beta");
assertThat(state(h).token().equals(token)).isFalse();
assertThat(UiApiInProcessSessions.resolve(token)).isEmpty();
FileGatewayTestServer.GRANTS.put("A", Set.of());
clock.advance(Duration.ofMinutes(5));
assertThatThrownBy(() -> names(sftp, "/")).isInstanceOf(Exception.class);
}
}
}
/*******************************************************************************
** Each saved-object list is cached for thirty seconds and paged past 250 rows.
*******************************************************************************/
@Test
void savedObjectsPaginationAndSlugCollisions() throws Exception
{
var clock = new MutableClock();
try(var h = new FileGatewayTestServer(clock))
{
var key = KeyUtils.generateKeyPair("ssh-ed25519", 256);
h.key(h.account("alice", "A"), key);
h.saved("savedView", "alpha", "Duplicate!");
h.saved("savedView", "alpha", "Duplicate?");
try(var session = h.login("alice", key); var sftp = SftpClientFactory.instance().createSftpClient(session))
{
assertThat(names(sftp, "/testapp/alpha/views")).contains("duplicate-1.csv", "duplicate-2.xlsx");
h.saved("savedView", "alpha", "Fresh");
assertThat(names(sftp, "/testapp/alpha/views")).doesNotContain("fresh.csv");
clock.advance(Duration.ofSeconds(31));
assertThat(names(sftp, "/testapp/alpha/views")).contains("fresh.csv");
var rows = new ArrayList<QRecord>();
for(int i = 0; i < 251; i++)
{
rows.add(new QRecord().withValue("tableName", "alpha").withValue("label", "Paged " + i).withValue("userId", "A").withValue("viewJson", "{}"));
}
h.audit.system(() -> new com.kingsrook.qqq.backend.core.actions.tables.InsertAction().execute(
new com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput("savedView").withRecords(rows)));
clock.advance(Duration.ofSeconds(31));
assertThat(names(sftp, "/testapp/alpha/views")).contains("paged-250.csv").hasSize(510);
}
}
assertThat(VirtualTree.slug(" !!! ")).isEqualTo("item");
assertThat(VirtualTree.slug("A".repeat(100))).hasSize(80);
}
/*******************************************************************************
** Revocation covers accounts, keys and explicit session termination, with heartbeats.
*******************************************************************************/
@ParameterizedTest
@ValueSource(strings = {"disabled", "expired", "keyExpired", "end"})
void pollEndsRevokedSessions(String reason) throws Exception
{
var clock = new MutableClock();
try(var h = new FileGatewayTestServer(clock))
{
var pair = KeyUtils.generateKeyPair("ssh-ed25519", 256);
var account = h.account("alice", "A");
var key = h.key(account, pair);
try(var session = h.login("alice", pair))
{
var state = state(h);
String token = state.token();
clock.advance(Duration.ofSeconds(60));
h.runtime.getProtocol().getAuthenticator().poll();
var connection = h.audit.query(FileGatewayTables.CONNECTION_LOG, new QQueryFilter()).getFirst();
assertThat(connection.getValueInstant("lastSeenAt")).isAfter(connection.getValueInstant("connectedAt"));
switch(reason)
{
case "disabled" -> h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")).withValue("status", "DISABLED"));
case "expired" -> h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")).withValue("expiresAt", clock.instant().minusSeconds(1)));
case "keyExpired" -> h.audit.update(FileGatewayTables.SSH_KEY, new QRecord().withValue("id", key.getValue("id")).withValue("expiresAt", clock.instant().minusSeconds(1)));
case "end" ->
{
com.kingsrook.qqq.backend.core.context.QContext.init(h.instance,
new com.kingsrook.qqq.backend.core.model.session.QSession().withPermissions(Set.of("file_gateway.admin")));
new com.kingsrook.qbits.filegateway.metadata.FileGatewayProcesses.Apply().runOnePage(
new com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput()
.withProcessName(com.kingsrook.qbits.filegateway.metadata.FileGatewayProcesses.END_SESSION).withRecords(List.of(connection)),
new com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput());
}
}
h.runtime.getProtocol().getAuthenticator().poll();
session.waitFor(Set.of(ClientSession.ClientSessionEvent.CLOSED), Duration.ofSeconds(5));
assertThat(session.isClosed()).isTrue();
assertThat(UiApiInProcessSessions.resolve(token)).isEmpty();
assertThatThrownBy(state::token).isInstanceOf(com.kingsrook.qqq.backend.core.exceptions.QException.class);
}
}
}
/*******************************************************************************
** Competing listeners share the account row lock, never exceeding the database limit.
*******************************************************************************/
@Test
void concurrentListenersAndRestartPreserveLockout() throws Exception
{
try(var h = new FileGatewayTestServer(); var second = FileGatewayRuntime.startForTest(h.instance, h.http,
h.config, KeyUtils.generateKeyPair("ecdsa-sha2-nistp256", 256), Clock.systemUTC(), Duration.ofHours(1));
var executor = Executors.newFixedThreadPool(2))
{
var pair = KeyUtils.generateKeyPair("ssh-ed25519", 256);
var account = h.account("alice", "A");
h.key(account, pair);
h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")).withValue("maxConcurrentSessions", 1));
var start = new CountDownLatch(1);
var tasks = new ArrayList<java.util.concurrent.Future<ClientSession>>();
for(var runtime : List.of(h.runtime, second))
{
tasks.add(executor.submit(() ->
{
start.await();
var session = h.client.connect("alice", "127.0.0.1", runtime.getProtocol().getPort()).verify(Duration.ofSeconds(5)).getSession();
session.addPublicKeyIdentity(pair);
try
{
session.auth().verify(Duration.ofSeconds(5));
return (session);
}
catch(Exception e)
{
session.close(true).await();
return (null);
}
}));
}
start.countDown();
var sessions = new ArrayList<ClientSession>();
for(var task : tasks)
{
var session = task.get();
if(session != null)
{
sessions.add(session);
}
}
try
{
assertThat(sessions).hasSize(1);
}
finally
{
for(var session : sessions)
{
session.close(true).await();
}
}
assertThat(h.audit.query(FileGatewayTables.CONNECTION_LOG, new QQueryFilter())).anySatisfy(r -> assertThat(r.getValueString("outcome")).isEqualTo("LIMIT"));
h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")).withValue("lockedUntil", Instant.now().plusSeconds(900)));
h.runtime.stop();
try(var session = h.client.connect("alice", "127.0.0.1", second.getProtocol().getPort()).verify(Duration.ofSeconds(5)).getSession())
{
session.addPublicKeyIdentity(pair);
assertThatThrownBy(() -> session.auth().verify(Duration.ofSeconds(5))).isInstanceOf(Exception.class);
}
assertThat(h.audit.query(FileGatewayTables.CONNECTION_LOG, new QQueryFilter())).anySatisfy(r -> assertThat(r.getValueString("outcome")).isEqualTo("LOCKED_OUT"));
}
}
/*******************************************************************************
** Connection limits apply before auth; operation limits reset on the next minute.
*******************************************************************************/
@Test
void rateAndOperationLimits() throws Exception
{
var clock = new MutableClock();
try(var h = new FileGatewayTestServer(clock))
{
var pair = KeyUtils.generateKeyPair("ssh-ed25519", 256);
var account = h.account("alice", "A");
h.key(account, pair);
h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")).withValue("maxOperationsPerMinute", 1));
try(var session = h.login("alice", pair); var sftp = SftpClientFactory.instance().createSftpClient(session))
{
names(sftp, "/");
try(var other = h.login("alice", pair); var otherSftp = SftpClientFactory.instance().createSftpClient(other))
{
assertThatThrownBy(() -> names(otherSftp, "/")).isInstanceOf(Exception.class);
}
assertThatThrownBy(() -> names(sftp, "/")).isInstanceOf(Exception.class);
clock.advance(Duration.ofSeconds(61));
assertThat(names(sftp, "/")).contains("testapp");
}
h.config.withMaxConnectionsPerIpPerMinute(1);
try(var first = h.login("alice", pair))
{
assertThatThrownBy(() -> h.login("alice", pair)).isInstanceOf(Exception.class);
}
assertThat(h.audit.query(FileGatewayTables.TRANSFER_LOG, new QQueryFilter())).anySatisfy(r -> assertThat(r.getValueString("outcome")).isEqualTo("LIMIT"));
}
}
/*******************************************************************************
** Rotating usernames cannot evade the per-IP lock, which expires independently.
*******************************************************************************/
@Test
void ipLockoutAcrossUsernames() throws Exception
{
var clock = new MutableClock();
try(var h = new FileGatewayTestServer(clock))
{
h.config.withLockoutThreshold(3);
var good = KeyUtils.generateKeyPair("ssh-ed25519", 256);
h.key(h.account("alice", "A"), good);
var bad = KeyUtils.generateKeyPair("ssh-ed25519", 256);
for(String username : List.of("unknown-one", "unknown-two", "unknown-three"))
{
assertThatThrownBy(() -> h.login(username, bad)).isInstanceOf(Exception.class);
}
assertThatThrownBy(() -> h.login("alice", good)).isInstanceOf(Exception.class);
assertThat(h.audit.query(FileGatewayTables.CONNECTION_LOG, new QQueryFilter())).anySatisfy(r -> assertThat(r.getValueString("outcome")).isEqualTo("LOCKED_OUT"));
clock.advance(Duration.ofMinutes(31));
try(var session = h.login("alice", good))
{
assertThat(session.isAuthenticated()).isTrue();
}
}
}
/*******************************************************************************
** Host secrets are parsed from memory and missing secrets fail startup.
*******************************************************************************/
@Test
void inMemoryHostKeys() throws Exception
{
var config = new FileGatewayQBitConfig().withHostKeyEnvVarName("TEST_HOST_KEY");
assertThatThrownBy(() -> FileGatewayRuntime.hostKey(config, name -> null)).isInstanceOf(IllegalStateException.class);
assertThat(FileGatewayRuntime.hostKey(config.withAllowGeneratedHostKey(true), name -> null)).isNotNull();
var pair = KeyUtils.generateKeyPair("ssh-ed25519", 256);
var output = new ByteArrayOutputStream();
OpenSSHKeyPairResourceWriter.INSTANCE.writePrivateKey(pair, "test", null, output);
String pem = output.toString(StandardCharsets.UTF_8);
var parsed = FileGatewayRuntime.hostKey(config, name -> name.equals("TEST_HOST_KEY") ? pem : null);
assertThat(KeyUtils.compareKeyPairs(pair, parsed)).isTrue();
assertThatThrownBy(() -> FileGatewayRuntime.hostKey(config, name -> "invalid")).isInstanceOf(Exception.class);
try(var h = new FileGatewayTestServer())
{
h.runtime.stop();
h.config.withHostKeyEnvVarName("TEST_HOST_KEY");
try(var restarted = FileGatewayRuntime.start(h.instance, h.http, name -> name.equals("TEST_HOST_KEY") ? pem : "test-pod"))
{
assertThat(restarted.getProtocol().getPort()).isPositive();
assertThat(restarted.getProtocol().activeSessions()).isZero();
}
}
}
/*******************************************************************************
** NIO paths never escape to native paths; phase-a mutations are always denied.
*******************************************************************************/
@Test
void virtualNioContract() throws Exception
{
try(var h = new FileGatewayTestServer())
{
var pair = KeyUtils.generateKeyPair("ssh-ed25519", 256);
h.key(h.account("alice", "A"), pair);
h.saved("savedView", "alpha", "Orders");
try(var session = h.login("alice", pair))
{
var state = state(h);
try(var fs = (GatewayFileSystem) h.runtime.getProtocol().getServer().getFileSystemFactory().createFileSystem(state.ssh))
{
var path = fs.getPath("/testapp/alpha/views/orders.csv");
assertThat(path.getFileName().toString()).isEqualTo("orders.csv");
assertThat(path.getNameCount()).isEqualTo(4);
assertThat(path.getName(1).toString()).isEqualTo("alpha");
assertThat(path.subpath(1, 3).toString()).isEqualTo("alpha/views");
assertThat(path.startsWith("/testapp")).isTrue();
assertThat(path.endsWith("views/orders.csv")).isTrue();
assertThat(fs.getPath("/testapp").relativize(path).toString()).isEqualTo("alpha/views/orders.csv");
assertThat(fs.getPath("../../testapp").toAbsolutePath().normalize().toString()).isEqualTo("/testapp");
assertThat(fs.getPath("/testapp", "alpha").resolve("views/orders.csv")).isEqualTo(path);
assertThat(path.toRealPath()).isEqualTo(path);
assertThat(path.hashCode()).isEqualTo(fs.getPath(path.toString()).hashCode());
assertThat(path.toUri().getScheme()).isEqualTo("gateway");
assertThat(path.iterator()).toIterable().hasSize(4);
assertThatThrownBy(path::toFile).isInstanceOf(UnsupportedOperationException.class);
assertThat(Files.readAttributes(path, BasicFileAttributes.class).isRegularFile()).isTrue();
assertThat(Files.readAttributes(fs.getPath("/"), BasicFileAttributes.class).isDirectory()).isTrue();
assertThatThrownBy(() -> Files.setAttribute(path, "basic:lastModifiedTime", java.nio.file.attribute.FileTime.from(Instant.EPOCH))).isInstanceOf(AccessDeniedException.class);
assertThatThrownBy(() -> Files.createSymbolicLink(fs.getPath("/link"), path)).isInstanceOf(AccessDeniedException.class);
assertThatThrownBy(() -> Files.createLink(fs.getPath("/link"), path)).isInstanceOf(AccessDeniedException.class);
assertThatThrownBy(() -> Files.copy(path, fs.getPath("/copy"))).isInstanceOf(AccessDeniedException.class);
assertThat(fs.getFileStores()).isEmpty();
assertThat(fs.getRootDirectories()).containsExactly(fs.getPath("/"));
assertThat(fs.isReadOnly()).isTrue();
assertThat(fs.getSeparator()).isEqualTo("/");
assertThatThrownBy(fs::newWatchService).isInstanceOf(UnsupportedOperationException.class);
assertThatThrownBy(fs::getUserPrincipalLookupService).isInstanceOf(UnsupportedOperationException.class);
assertThatThrownBy(() -> fs.getPathMatcher("glob:*")).isInstanceOf(UnsupportedOperationException.class);
}
}
}
}
}
@@ -0,0 +1,173 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway;
import java.time.Instant;
import java.util.List;
import java.util.Map;
import java.util.Set;
import com.kingsrook.qbits.filegateway.metadata.FileGatewayProcesses;
import com.kingsrook.qbits.filegateway.metadata.FileGatewayTables;
import com.kingsrook.qbits.filegateway.permissions.FileGatewayPermissionChecker;
import com.kingsrook.qqq.backend.core.context.QContext;
import com.kingsrook.qqq.backend.core.exceptions.QException;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepInput;
import com.kingsrook.qqq.backend.core.model.actions.processes.RunBackendStepOutput;
import com.kingsrook.qqq.backend.core.model.actions.tables.insert.InsertInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput;
import com.kingsrook.qqq.backend.core.model.actions.tables.update.UpdateInput;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.backend.core.model.session.QSession;
import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiCredentials;
import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInProcessSessions;
import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInvoker;
import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiRequest;
import com.kingsrook.qqq.openapi.model.HttpMethod;
import org.apache.sshd.common.config.keys.KeyUtils;
import org.apache.sshd.common.config.keys.PublicKeyEntry;
import org.apache.sshd.common.digest.BuiltinDigests;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Real database customizers and strict UI API dependent pickers.
*******************************************************************************/
@ExtendWith(NoDiskExtension.class)
class MetadataBehaviorTest
{
/*******************************************************************************
** Every contingent picker excludes records for the other table via the UI API.
*******************************************************************************/
@ParameterizedTest
@ValueSource(strings = {"savedBulkLoadProfile", "savedView", "savedReport"})
void contingentPickerBehavior(String source) throws Exception
{
try(var h = new FileGatewayTestServer())
{
var own = h.saved(source, "alpha", "Alpha choice");
var other = h.saved(source, "beta", "Beta choice");
assertThat(own.getErrors()).isEmpty();
assertThat(other.getErrors()).isEmpty();
var session = new QSession().withPermissions(Set.of("file_gateway.admin"));
String token = UiApiInProcessSessions.register(session, java.time.Duration.ofMinutes(1));
try
{
var response = UiApiInvoker.forServer(h.http, "next-ui").invoke(new UiApiRequest().withMethod(HttpMethod.POST)
.withPath("/table/{tableName}/possibleValues/{fieldName}").withPathParam("tableName", FileGatewayTables.ALIAS)
.withPathParam("fieldName", source + "Id").withCredentials(UiApiCredentials.ofInProcessSession(token))
.withJsonBody(Map.of("values", Map.of("tableName", "alpha"))));
assertThat(response.getStatus()).as(response.getJson().toString()).isEqualTo(200);
assertThat(response.getJson().toString()).contains("Alpha choice").doesNotContain("Beta choice");
}
finally
{
UiApiInProcessSessions.revoke(token);
}
}
}
/*******************************************************************************
** Validation runs on insert and partial update, computes metadata, and enforces uniqueness.
*******************************************************************************/
@Test
void publicKeyValidation() throws Exception
{
try(var h = new FileGatewayTestServer())
{
var account = h.account("alice", "A");
var key = KeyUtils.generateKeyPair("ssh-ed25519", 256);
var stored = h.key(account, key);
assertThat(stored.getValueString("fingerprint")).isEqualTo(KeyUtils.getFingerPrint(BuiltinDigests.sha256, key.getPublic()));
assertThat(stored.getValueString("keyType")).isEqualTo("ssh-ed25519");
assertThat(stored.getValueInteger("keyBits")).isEqualTo(256);
assertThatThrownBy(() -> h.key(account, key)).isInstanceOf(QException.class);
for(String type : List.of("ssh-rsa", "ssh-dss"))
{
var weak = KeyUtils.generateKeyPair(type, type.equals("ssh-rsa") ? 2048 : 1024);
assertThatThrownBy(() -> h.key(account, weak)).isInstanceOf(QException.class);
}
assertThat(h.key(account, KeyUtils.generateKeyPair("ssh-rsa", 3072)).getValueInteger("keyBits")).isEqualTo(3072);
assertThat(h.key(account, KeyUtils.generateKeyPair("ecdsa-sha2-nistp384", 384)).getValueInteger("keyBits")).isEqualTo(384);
for(String text : List.of("not a key", "ssh-unknown invalid", ""))
{
assertThatThrownBy(() -> h.audit.insert(FileGatewayTables.SSH_KEY, new QRecord().withValue("accountId", account.getValue("id")).withValue("publicKey", text))).isInstanceOf(QException.class);
}
h.audit.update(FileGatewayTables.SSH_KEY, new QRecord().withValue("id", stored.getValue("id")).withValue("label", "Renamed").withValue("fingerprint", "forged"));
var read = h.audit.query(FileGatewayTables.SSH_KEY, new QQueryFilter()).getFirst();
assertThat(read.getValueString("fingerprint")).isEqualTo(stored.getValueString("fingerprint"));
var weak = KeyUtils.generateKeyPair("ssh-rsa", 2048);
assertThatThrownBy(() -> h.audit.update(FileGatewayTables.SSH_KEY, new QRecord().withValue("id", stored.getValue("id")).withValue("publicKey", PublicKeyEntry.toString(weak.getPublic())))).isInstanceOf(QException.class);
assertThatThrownBy(() -> h.account("Bad name", "A")).isInstanceOf(QException.class);
assertThatThrownBy(() -> h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")).withValue("maxConcurrentSessions", 0))).isInstanceOf(QException.class);
}
}
/*******************************************************************************
** People can read logs but cannot write them, including administrators.
*******************************************************************************/
@Test
void permissionsAndAdministrativeActions() throws Exception
{
try(var h = new FileGatewayTestServer())
{
var account = h.account("alice", "A");
var key = h.key(account, KeyUtils.generateKeyPair("ssh-ed25519", 256));
var checker = new FileGatewayPermissionChecker();
QContext.init(h.instance, new QSession().withPermissions(Set.of("file_gateway.view")));
checker.checkPermissionsThrowing(new QueryInput(FileGatewayTables.ACCOUNT), h.instance.getTable(FileGatewayTables.ACCOUNT));
assertThatThrownBy(() -> checker.checkPermissionsThrowing(new UpdateInput(FileGatewayTables.ACCOUNT), h.instance.getTable(FileGatewayTables.ACCOUNT))).isInstanceOf(QException.class);
QContext.init(h.instance, new QSession().withPermissions(Set.of("file_gateway.admin")));
checker.checkPermissionsThrowing(new InsertInput(FileGatewayTables.ACCOUNT), h.instance.getTable(FileGatewayTables.ACCOUNT));
for(String table : FileGatewayTables.SYSTEM_WRITTEN)
{
assertThatThrownBy(() -> checker.checkPermissionsThrowing(new UpdateInput(table), h.instance.getTable(table))).isInstanceOf(QException.class);
}
h.audit.update(FileGatewayTables.ACCOUNT, new QRecord().withValue("id", account.getValue("id")).withValue("lockedUntil", Instant.now().plusSeconds(900)).withValue("failedLoginCount", 5));
var input = new RunBackendStepInput().withProcessName(FileGatewayProcesses.UNLOCK).withRecords(List.of(account));
var review = new FileGatewayProcesses.Review();
var preview = new RunBackendStepOutput();
review.runOnePage(input, preview);
assertThat(review.getProcessSummary(preview, false)).hasSize(1);
assertThat(h.audit.query(FileGatewayTables.ACCOUNT, new QQueryFilter()).getFirst().getValue("lockedUntil")).isNotNull();
new FileGatewayProcesses.Apply().runOnePage(input, new RunBackendStepOutput());
var unlocked = h.audit.query(FileGatewayTables.ACCOUNT, new QQueryFilter()).getFirst();
assertThat(unlocked.getValue("lockedUntil")).isNull();
assertThat(unlocked.getValueInteger("failedLoginCount")).isZero();
new FileGatewayProcesses.Apply().runOnePage(input.withProcessName(FileGatewayProcesses.DISABLE), new RunBackendStepOutput());
assertThat(h.audit.query(FileGatewayTables.ACCOUNT, new QQueryFilter()).getFirst().getValueString("status")).isEqualTo("DISABLED");
new FileGatewayProcesses.Apply().runOnePage(input.withProcessName(FileGatewayProcesses.REVOKE).withRecords(List.of(key)), new RunBackendStepOutput());
assertThat(h.audit.query(FileGatewayTables.SSH_KEY, new QQueryFilter()).getFirst().getValue("revokedAt")).isNotNull();
QContext.init(h.instance, new QSession());
assertThatThrownBy(() -> new FileGatewayProcesses.Apply().runOnePage(input, new RunBackendStepOutput())).isInstanceOf(QException.class);
assertThatThrownBy(() -> checker.checkPermissionsThrowing(new QueryInput(FileGatewayTables.ACCOUNT), h.instance.getTable(FileGatewayTables.ACCOUNT))).isInstanceOf(QException.class);
assertThatThrownBy(() -> h.audit.query("alpha", new QQueryFilter())).isInstanceOf(IllegalArgumentException.class);
}
}
}
@@ -0,0 +1,90 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.attribute.FileTime;
import java.util.Map;
import java.util.TreeMap;
import org.junit.jupiter.api.extension.AfterEachCallback;
import org.junit.jupiter.api.extension.BeforeEachCallback;
import org.junit.jupiter.api.extension.ExtensionContext;
import static org.assertj.core.api.Assertions.assertThat;
/*******************************************************************************
** Snapshots real disk boundaries around every SFTP test; target build products are excluded.
*******************************************************************************/
public class NoDiskExtension implements BeforeEachCallback, AfterEachCallback
{
private Map<String, FileTime> before;
/*******************************************************************************
** Before each.
*******************************************************************************/
@Override
public void beforeEach(ExtensionContext context) throws Exception
{
Files.createDirectories(Path.of(System.getProperty("java.io.tmpdir")));
Files.createDirectories(Path.of(System.getProperty("user.home")));
before = snapshot();
}
/*******************************************************************************
** After each.
*******************************************************************************/
@Override
public void afterEach(ExtensionContext context) throws Exception
{
Map<String, FileTime> after = snapshot();
assertThat(after.keySet()).as("Files created or removed outside the temporary area").isEqualTo(before.keySet());
after.forEach((path, time) -> assertThat(time).as("Modified: %s", path).isEqualTo(before.get(path)));
try(var files = Files.list(Path.of(System.getProperty("java.io.tmpdir"))))
{
assertThat(files.toList()).isEmpty();
}
}
private Map<String, FileTime> snapshot() throws Exception
{
Map<String, FileTime> result = new TreeMap<>();
Path working = Path.of("").toAbsolutePath();
try(var files = Files.walk(working))
{
for(Path file : files.filter(p -> !p.startsWith(working.resolve("target"))).toList())
{
result.put(file.toString(), Files.getLastModifiedTime(file));
}
}
try(var files = Files.list(Path.of(System.getProperty("user.home"))))
{
for(Path file : files.toList())
{
result.put(file.toString(), Files.getLastModifiedTime(file));
}
}
return (result);
}
}
@@ -0,0 +1,297 @@
/*
* QQQ - Low-code Application Framework for Engineers.
* Copyright (C) 2021-2025. Kingsrook, LLC
* 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States
* contact@kingsrook.com
* https://github.com/Kingsrook/
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package com.kingsrook.qbits.filegateway;
import java.security.KeyPair;
import java.time.Duration;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import com.kingsrook.qbits.filegateway.sftp.CidrMatcher;
import com.kingsrook.qqq.backend.core.model.actions.tables.query.QQueryFilter;
import com.kingsrook.qqq.backend.core.model.data.QRecord;
import com.kingsrook.qqq.middleware.javalin.specs.invoke.UiApiInProcessSessions;
import com.kingsrook.qqq.openapi.model.HttpMethod;
import org.apache.sshd.common.config.keys.KeyUtils;
import org.apache.sshd.sftp.client.SftpClient;
import org.apache.sshd.sftp.client.SftpClientFactory;
import org.apache.sshd.sftp.common.SftpConstants;
import org.apache.sshd.sftp.common.SftpException;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
/*******************************************************************************
** Real protocol acceptance tests for listing, authentication, auditing and no-disk operation.
*******************************************************************************/
@ExtendWith(NoDiskExtension.class)
class SftpIntegrationTest
{
/*******************************************************************************
** Pair.
*******************************************************************************/
private KeyPair pair() throws Exception
{
return (KeyUtils.generateKeyPair("ssh-ed25519", 256));
}
/*******************************************************************************
** Names.
*******************************************************************************/
private List<String> names(SftpClient client, String path) throws Exception
{
List<String> result = new ArrayList<>();
for(SftpClient.DirEntry entry : client.readDir(path))
{
if(!entry.getFilename().equals(".") && !entry.getFilename().equals(".."))
{
result.add(entry.getFilename());
}
}
return (result);
}
/*******************************************************************************
** Permission filtered listings and audit.
*******************************************************************************/
@Test
void permissionFilteredListingsAndAudit() throws Exception
{
try(var harness = new FileGatewayTestServer())
{
KeyPair a = pair();
KeyPair b = pair();
harness.key(harness.account("alice", "A"), a);
harness.key(harness.account("bob", "B"), b);
assertThat(harness.saved("savedBulkLoadProfile", "alpha", "Orders In").getErrors()).isEmpty();
assertThat(harness.saved("savedView", "alpha", "All Orders").getErrors()).isEmpty();
assertThat(harness.saved("savedReport", "alpha", "Daily Orders").getErrors()).isEmpty();
int tokens = UiApiInProcessSessions.size();
try(var sa = harness.login("alice", a); var sb = harness.login("bob", b);
var ca = SftpClientFactory.instance().createSftpClient(sa); var cb = SftpClientFactory.instance().createSftpClient(sb))
{
assertThat(names(ca, "/")).contains("testapp");
assertThat(names(ca, "/testapp")).containsExactlyInAnyOrder("alpha", "beta");
assertThat(names(cb, "/testapp")).containsExactly("beta");
assertThatThrownBy(() -> cb.stat("/testapp/alpha")).isInstanceOfSatisfying(SftpException.class,
e -> assertThat(e.getStatus()).isEqualTo(SftpConstants.SSH_FX_NO_SUCH_FILE));
assertThat(names(ca, "/testapp/alpha")).contains("upload", "views", "reports", "results", "archive");
assertThat(names(ca, "/testapp/beta")).doesNotContain("upload");
assertThat(names(ca, "/testapp/alpha/upload")).containsExactly("orders-in");
assertThat(names(ca, "/testapp/alpha/upload/orders-in")).isEmpty();
assertThat(names(ca, "/testapp/alpha/views")).containsExactlyInAnyOrder("all-orders.csv", "all-orders.xlsx");
assertThat(names(ca, "/testapp/alpha/reports")).contains("daily-orders.csv", "daily-orders.xlsx");
assertThat(ca.stat("/testapp/alpha/views/all-orders.csv").getSize()).isZero();
assertThatThrownBy(() -> ca.read("/testapp/alpha/views/all-orders.csv")).isInstanceOf(SftpException.class);
assertThatThrownBy(() -> ca.write("/testapp/alpha/upload/orders-in/test.csv")).isInstanceOf(SftpException.class);
assertThatThrownBy(() -> ca.setStat("/testapp/alpha/views/all-orders.csv", new SftpClient.Attributes().modifyTime(0))).isInstanceOfSatisfying(SftpException.class,
e -> assertThat(e.getStatus()).isEqualTo(SftpConstants.SSH_FX_PERMISSION_DENIED));
assertThatThrownBy(() -> names(cb, "/testapp/alpha")).isInstanceOf(java.io.UncheckedIOException.class).hasCauseInstanceOf(SftpException.class);
assertThatThrownBy(() -> ca.mkdir("/testapp/alpha/new")).isInstanceOf(SftpException.class);
assertThatThrownBy(() -> ca.remove("/testapp/alpha/views/all-orders.csv")).isInstanceOf(SftpException.class);
assertThatThrownBy(() -> ca.rename("/testapp/alpha/views/all-orders.csv", "/testapp/alpha/views/new.csv")).isInstanceOf(SftpException.class);
assertThat(harness.runtime.getInvoker().findOperation(HttpMethod.GET, "/metaData")).isPresent();
assertThat(harness.runtime.getInvoker().findOperation(HttpMethod.POST, "/table/{tableName}/query")).isPresent();
assertThat(harness.runtime.getInvoker().getClientChannel()).isEqualTo("sftp");
}
assertThat(harness.calls).isNotEmpty().allSatisfy(call ->
{
String template = call.path().substring("/qqq/v1".length()).replaceAll("/table/[^/]+/query", "/table/{tableName}/query");
assertThat(harness.runtime.getInvoker().findOperation(HttpMethod.valueOf(call.method()), template)).isPresent();
assertThat(call.channel()).startsWith("sftp/");
assertThat(call.invokedBy()).isEqualTo("file-gateway");
});
harness.runtime.stop();
assertThat(UiApiInProcessSessions.size()).isEqualTo(tokens);
var connections = harness.audit.query("file_gateway_connection_log", new QQueryFilter());
assertThat(connections).hasSize(2).allSatisfy(row ->
{
assertThat(row.getValueString("clientChannel")).isEqualTo("sftp");
assertThat(row.getValueString("outcome")).isEqualTo("CONNECTED");
assertThat(row.getValue("disconnectedAt")).isNotNull();
});
var transfers = harness.audit.query("file_gateway_transfer_log", new QQueryFilter());
assertThat(transfers).anySatisfy(row ->
{
assertThat(row.getValueString("operation")).isEqualTo("LIST");
assertThat(row.getValueString("outcome")).isEqualTo("OK");
}).anySatisfy(row -> assertThat(row.getValueString("outcome")).isEqualTo("DENIED"))
.allSatisfy(row -> assertThat(row.getValueString("clientChannel")).startsWith("sftp/"));
}
}
/*******************************************************************************
** Refuses invalid credentials with audit.
*******************************************************************************/
@ParameterizedTest
@ValueSource(strings = {"unknown", "revoked", "expiredKey", "disabled", "expiredAccount", "ip", "permission"})
void refusesInvalidCredentialsWithAudit(String scenario) throws Exception
{
try(var h = new FileGatewayTestServer())
{
var key = pair();
var account = h.account("alice", scenario.equals("permission") ? "NONE" : "A");
var stored = h.key(account, key);
String outcome = "AUTH_FAILED";
switch(scenario)
{
case "revoked" -> h.audit.update("file_gateway_ssh_key", new QRecord().withValue("id", stored.getValue("id")).withValue("revokedAt", Instant.now()));
case "expiredKey" ->
{
h.audit.update("file_gateway_ssh_key", new QRecord().withValue("id", stored.getValue("id")).withValue("expiresAt", Instant.now().minusSeconds(1)));
outcome = "EXPIRED";
}
case "disabled" ->
{
h.audit.update("file_gateway_account", new QRecord().withValue("id", account.getValue("id")).withValue("status", "DISABLED"));
outcome = "DISABLED";
}
case "expiredAccount" ->
{
h.audit.update("file_gateway_account", new QRecord().withValue("id", account.getValue("id")).withValue("expiresAt", Instant.now().minusSeconds(1)));
outcome = "EXPIRED";
}
case "ip" ->
{
h.audit.update("file_gateway_account", new QRecord().withValue("id", account.getValue("id")).withValue("allowedCidrs", "192.0.2.0/24"));
outcome = "IP_DENIED";
}
}
KeyPair offered = scenario.equals("unknown") ? pair() : key;
assertThatThrownBy(() -> h.login("alice", offered)).isInstanceOf(Exception.class);
assertThat(h.audit.query("file_gateway_connection_log", new QQueryFilter()).getFirst().getValueString("outcome")).isEqualTo(outcome);
}
}
/*******************************************************************************
** Lockout and session limit.
*******************************************************************************/
@Test
void lockoutAndSessionLimit() throws Exception
{
try(var h = new FileGatewayTestServer())
{
com.kingsrook.qqq.backend.core.alerts.AlertsInstanceMetaData.ofOrWithNew(h.instance)
.withSink(new com.kingsrook.qqq.backend.core.model.metadata.code.QCodeReference(RecordingAlerts.class));
RecordingAlerts.signals.clear();
var account = h.account("alice", "A");
var good = pair();
h.key(account, good);
var bad = pair();
for(int i = 0; i < 5; i++)
{
assertThatThrownBy(() -> h.login("alice", bad)).isInstanceOf(Exception.class);
}
assertThatThrownBy(() -> h.login("alice", good)).isInstanceOf(Exception.class);
assertThat(RecordingAlerts.signals).containsExactly("filegateway.lockout");
var rows = h.audit.query("file_gateway_connection_log", new QQueryFilter());
assertThat(rows).anySatisfy(row -> assertThat(row.getValueString("outcome")).isEqualTo("LOCKED_OUT"));
assertThat(h.audit.query("file_gateway_account", new QQueryFilter()).getFirst().getValue("lockedUntil")).isNotNull();
h.audit.update("file_gateway_account", new QRecord().withValue("id", account.getValue("id")).withValue("lockedUntil", null));
try(var unlocked = h.login("alice", good))
{
assertThat(unlocked.isAuthenticated()).isTrue();
}
}
try(var h = new FileGatewayTestServer())
{
var account = h.account("alice", "A");
h.audit.update("file_gateway_account", new QRecord().withValue("id", account.getValue("id")).withValue("maxConcurrentSessions", 1));
var good = pair();
h.key(account, good);
try(var session = h.login("alice", good))
{
assertThatThrownBy(() -> h.login("alice", good)).isInstanceOf(Exception.class);
assertThat(h.audit.query("file_gateway_connection_log", new QQueryFilter())).anySatisfy(row -> assertThat(row.getValueString("outcome")).isEqualTo("LIMIT"));
}
}
}
/*******************************************************************************
** Public key only and live revocation.
*******************************************************************************/
@Test
void publicKeyOnlyAndLiveRevocation() throws Exception
{
try(var h = new FileGatewayTestServer())
{
var account = h.account("alice", "A");
var pair = pair();
var key = h.key(account, pair);
var server = h.runtime.getProtocol().getServer();
assertThat(server.getPasswordAuthenticator()).isNull();
assertThat(server.getKeyboardInteractiveAuthenticator()).isNull();
assertThat(server.getUserAuthFactories()).extracting(factory -> factory.getName()).containsExactly("publickey");
assertThat(server.getCipherFactories()).noneSatisfy(factory -> assertThat(factory.getName()).contains("cbc"));
assertThat(server.getMacFactories()).allSatisfy(factory -> assertThat(factory.getName()).contains("sha2"));
assertThat(server.getSignatureFactories()).noneSatisfy(factory -> assertThat(factory.getName()).isEqualTo("ssh-rsa"));
try(var password = h.client.connect("alice", "127.0.0.1", h.runtime.getProtocol().getPort()).verify(Duration.ofSeconds(5)).getSession())
{
password.addPasswordIdentity("not-a-password");
assertThatThrownBy(() -> password.auth().verify(Duration.ofSeconds(5))).isInstanceOf(Exception.class);
}
try(var session = h.login("alice", pair))
{
h.audit.update("file_gateway_ssh_key", new QRecord().withValue("id", key.getValue("id")).withValue("revokedAt", Instant.now()));
h.runtime.getProtocol().getAuthenticator().poll();
session.waitFor(java.util.Set.of(org.apache.sshd.client.session.ClientSession.ClientSessionEvent.CLOSED), Duration.ofSeconds(5));
assertThat(session.isClosed()).isTrue();
}
}
assertThat(CidrMatcher.matches("::1", "::1/128")).isTrue();
assertThat(CidrMatcher.matches("127.0.0.1", "::1/128")).isFalse();
assertThat(CidrMatcher.matches("127.0.0.1", "localhost/32")).isFalse();
}
/*******************************************************************************
** Exercise the core alert facade without adding any dependency on qbit-alerts.
*******************************************************************************/
public static class RecordingAlerts extends com.kingsrook.qqq.backend.core.alerts.LoggingAlertSink
{
static final List<String> signals = new java.util.concurrent.CopyOnWriteArrayList<>();
/*******************************************************************************
** Record alert type names only, without account or key material.
*******************************************************************************/
@Override
public void raise(com.kingsrook.qqq.backend.core.alerts.AlertSignal signal)
{
signals.add(signal.getAlertTypeName());
}
}
}
+11
View File
@@ -0,0 +1,11 @@
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%level %logger{1}: %msg%n"/>
</Console>
</Appenders>
<Loggers>
<Root level="WARN"><AppenderRef ref="Console"/></Root>
</Loggers>
</Configuration>