diff --git a/AGENTS.md b/AGENTS.md index e393622..08bd02d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -35,7 +35,7 @@ This file provides guidance to Codex when working with code in this repository. ## Dependencies - Java 21. -- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-RC.4` ahead of it, as in qbit-quick-search (ADR-0007). Remove that activation when the parent moves to 4.1. +- Parent `com.kingsrook:qbit-build-parent:2.0.0` (qqq 4.0.0). The default-active `qqq-snapshot` profile imports `qqq-bom-pom:4.1.0-RC.5` ahead of it, as in qbit-quick-search (ADR-0007). Remove that activation when the parent moves to 4.1. - Tests: JUnit 5, AssertJ. ## Release diff --git a/CHANGELOG.md b/CHANGELOG.md index 55a46f8..4d695e1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -15,6 +15,11 @@ All notable changes to this project are documented here. The format follows Keep - `ObjectStorageSchema`: PostgreSQL DDL generated from the table metadata. - Conformance suite against MinIO and Garage (Testcontainers), service, index and action tests, and the index and actions on PostgreSQL. +### Changed +- Raised the QQQ BOM baseline to released `4.1.0-RC.5` and delegated S3 client construction to core `S3ClientFactory.buildClient`. Removed local HTTP client and trust-manager implementations; existing endpoint, region, addressing, checksum, TLS/CA, credentials and timeout settings remain supported. +- Clients remain owned by each connection runtime and are rebuilt on reset/reinstall. Retry behavior now follows core/SDK defaults instead of forcing `ADAPTIVE_V2`; hosts can select the SDK retry mode through `AWS_RETRY_MODE` or `aws.retryMode`. +- Retained local presigning (including a separate endpoint/key). A request-credentials bridge preserves static session tokens because RC.5 core metadata/factory has no session-token field or credentials-provider hook. This bridge and the existing `${file.PATH}` resolver are marked for removal by RC.7 under issue #10. +- Added the filesystem module with its unused SFTP/SSH stack excluded. Resolved dependencies otherwise only change QQQ RC.4 to RC.5 and jsoup 1.23.1 to 1.23.2; see [dependency details](docs/S3-FACTORY-MIGRATION.md). + ### Notes -- Until QQQ core's S3 module on AWS SDK v2 (qqq#1095, RC.5) is released, the qbit uses a small internal SDK v2 client factory. - Deliberate differences from the design text: Test connection is two actions (read-only, and with write tests) so the write variant can require `object_storage.admin`; a folder record has no subfolder child list (core does not allow a self join as an exposed join); there is no `withTableNamePrefix` (the `object_storage_` prefix is fixed by the naming rule). diff --git a/README.md b/README.md index 8f9e2d5..4ead1a2 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ S3-compatible storage as a capability for any QQQ app, and an admin app to run i ## Requirements - Java 21. -- QQQ 4.1 (built against `4.1.0-RC.4`). Until QQQ core's S3 module on AWS SDK v2 (qqq#1095, RC.5) is released, the qbit carries a small internal SDK v2 client factory (`ConnectionRuntime`); it will be replaced by core's `S3ClientFactory` without changing the public API. +- QQQ `4.1.0-RC.5` (released BOM), including `qqq-backend-module-filesystem`. `ConnectionRuntime` maps connection settings to core's `S3BackendMetaData` and uses `S3ClientFactory.buildClient`; core owns HTTP, TLS and checksum setup. The unused SFTP/SSH dependency is excluded. See the [dependency diff](docs/S3-FACTORY-MIGRATION.md). - A QQQ backend for the qbit's tables (PostgreSQL is the intended store). `ObjectStorageSchema.postgresScript(config)` generates the DDL from the same metadata the app runs on, for your Liquibase or Flyway migration. ## Install @@ -45,6 +45,8 @@ new ObjectStorageQBitProducer() Connections and buckets come from this code only. Secrets are **references** (`${env.X}`, `${prop.X}`, `${file.PATH}`) that must match `allowedSecretRefPatterns` (default `env.S3_*`, `env.STORAGE_*`, `file./var/run/secrets/storage/*`); no table stores a secret value. At startup the qbit upserts one `object_storage_connection` and one `object_storage_bucket` row per entry, so the tables always show the real configuration. +Client settings take effect when the qbit configuration is reinstalled or the runtime is reset. Presigning keeps its separate public endpoint and credentials. Retry behavior now follows core/SDK defaults; hosts requiring adaptive retries can set `AWS_RETRY_MODE=adaptive` or `aws.retryMode=adaptive`. Static session tokens remain supported through a small local bridge pending core support. + ## Use ```java diff --git a/docs/S3-FACTORY-MIGRATION.md b/docs/S3-FACTORY-MIGRATION.md new file mode 100644 index 0000000..d4a97ea --- /dev/null +++ b/docs/S3-FACTORY-MIGRATION.md @@ -0,0 +1,86 @@ +# Core S3 factory migration (issue #10, F3) + +Core source was inspected at local QRun-IO/qqq tag `v4.1.0-RC.5`: +`qqq-backend-module-filesystem/src/main/java/com/kingsrook/qqq/backend/module/filesystem/s3/{utils/S3ClientFactory,model/metadata/S3BackendMetaData}.java`. + +## Ownership and host behavior + +`ConnectionRuntime` resolves secret/setting references and maps endpoint, region, +path style, checksum mode, TLS verification, extra CA PEM, static keys, and both +HTTP timeouts into `S3BackendMetaData`. Default-chain and web-identity modes leave +keys unset so core uses the SDK default credentials chain. Provider-specific qbit +defaults and the existing 5-second connect / 60-second read timeouts are preserved. +Core handles custom CA trust in addition to the JVM trust store. + +The qbit calls uncached `buildClient`, because it owns and closes each client. +`ObjectStorageRuntime` caches by connection name and resets clients on +configuration install/reset; `settingsHash` is an informational table value, +not the cache key. This migration keeps that lifecycle and avoids sharing a +core-cached client that the qbit would later close. Configuration changes take +effect on reinstall/reset, not through mutation of an already cached client. + +Core does not expose retry configuration. Requests now use core/SDK retry +defaults instead of the former forced `ADAPTIVE_V2`. Hosts needing adaptive +retries can set the SDK's `AWS_RETRY_MODE=adaptive` or `aws.retryMode=adaptive`. +The public qbit configuration API is unchanged. + +## Remaining local code + +- Core RC.5 has no session-token field or credentials-provider customization + in `S3BackendMetaData` / `S3ClientFactory.buildClient`. The narrow + `SessionCredentialsClient` bridge adds session credentials to requests while + retaining their overrides; core still creates the client and HTTP transport. + Core needs session-token metadata (including enrichment and cache-key support) + and `AwsSessionCredentials` construction, or a caller-provided credentials hook. + Marker: `STAND-IN(core=QRun-IO/qqq#1165, removeBy=4.1.0-RC.7)`. +- Core has no presigner factory. The local presigner retains its separate public + endpoint and least-privilege keys, along with existing session-token handling. +- `SecretRefs` file resolution is unchanged and carries the same RC.7 marker. + +## Resolved dependency diff + +Captured using the Maven 3.9.11 wrapper's `-B dependency:tree` before and after +this change. Coordinates below are normalized to omit tree indentation and +parent ordering. Full raw trees are in `target/dependency-tree-{before,after}.txt` +for this working session (generated build files, not committed). + +```diff +- com.kingsrook.qqq:qqq-backend-core:jar:4.1.0-RC.4:compile +- com.kingsrook.qqq:qqq-backend-module-postgres:jar:4.1.0-RC.4:test +- com.kingsrook.qqq:qqq-backend-module-rdbms:jar:4.1.0-RC.4:test +- com.kingsrook.qqq:qqq-middleware-javalin:jar:4.1.0-RC.4:test +- com.kingsrook.qqq:qqq-openapi:jar:4.1.0-RC.4:test +- org.jsoup:jsoup:jar:1.23.1:compile ++ com.kingsrook.qqq:qqq-backend-core:jar:4.1.0-RC.5:compile ++ com.kingsrook.qqq:qqq-backend-module-filesystem:jar:4.1.0-RC.5:compile ++ com.kingsrook.qqq:qqq-backend-module-postgres:jar:4.1.0-RC.5:test ++ com.kingsrook.qqq:qqq-backend-module-rdbms:jar:4.1.0-RC.5:test ++ com.kingsrook.qqq:qqq-middleware-javalin:jar:4.1.0-RC.5:test ++ com.kingsrook.qqq:qqq-openapi:jar:4.1.0-RC.5:test ++ org.jsoup:jsoup:jar:1.23.2:compile +``` + +The only new resolved artifact is `qqq-backend-module-filesystem`. It declares +`org.apache.sshd:sshd-sftp:2.16.0`, which would bring `sshd-core` and `sshd-common`; +that unused SFTP stack is excluded at the qbit dependency. No SSH libraries or +AWS SDK v1 artifacts are present in the resulting tree. AWS SDK v2 remains +`2.41.34`; `commons-lang3:3.20.0` was already present and is unchanged. The +filesystem module's test dependencies do not become runtime dependencies. + +## Verification in this worktree + +Java 21.0.12.1, Maven wrapper 3.9.11; `./mvnw.cmd -B verify`: + +- Checkstyle: zero violations; production and test compilation succeeded. +- Surefire: 119 tests reported, 32 passed, 87 skipped, zero failures/errors. + All eight new `ConnectionRuntimeTest` tests passed, including actual HTTP + signing with a session token on GET, PUT and pagination. +- Docker Desktop access is denied at `npipe:////./pipe/dockerDesktopLinuxEngine`. + MinIO, Garage, standalone-service and PostgreSQL integration coverage could + not run. Testcontainers also reported its default `docker_engine` pipe unavailable. +- The full gate **failed** at JaCoCo: instructions 33.68% (required 70%), classes + 70.75% (required 90%). Coverage thresholds and Docker test policy are unchanged. +- No unrelated RC.5 compatibility fixes were needed. `git diff --check` passed. + +Rerun the full verification gate in an environment with Docker access before +merging. No changes were committed or pushed. diff --git a/pom.xml b/pom.xml index 04b75f8..5af803c 100644 --- a/pom.xml +++ b/pom.xml @@ -80,6 +80,17 @@ + + com.kingsrook.qqq + qqq-backend-module-filesystem + + + + org.apache.sshd + sshd-sftp + + + software.amazon.awssdk s3 @@ -228,14 +239,14 @@ - + qqq-snapshot true - 4.1.0-RC.4 + 4.1.0-RC.5 diff --git a/src/main/java/com/kingsrook/qbits/objectstorage/internal/ConnectionRuntime.java b/src/main/java/com/kingsrook/qbits/objectstorage/internal/ConnectionRuntime.java index f5f31d6..739ce7b 100644 --- a/src/main/java/com/kingsrook/qbits/objectstorage/internal/ConnectionRuntime.java +++ b/src/main/java/com/kingsrook/qbits/objectstorage/internal/ConnectionRuntime.java @@ -21,57 +21,38 @@ package com.kingsrook.qbits.objectstorage.internal; -import javax.net.ssl.TrustManager; -import javax.net.ssl.TrustManagerFactory; -import javax.net.ssl.X509TrustManager; -import java.io.ByteArrayInputStream; import java.net.URI; -import java.nio.charset.StandardCharsets; -import java.security.KeyStore; -import java.security.cert.Certificate; -import java.security.cert.CertificateException; -import java.security.cert.CertificateFactory; -import java.security.cert.X509Certificate; -import java.time.Duration; -import java.util.Collection; +import java.util.function.Function; import com.kingsrook.qbits.objectstorage.config.AddressingStyle; -import com.kingsrook.qbits.objectstorage.config.ChecksumMode; import com.kingsrook.qbits.objectstorage.config.CredentialsMode; import com.kingsrook.qbits.objectstorage.config.StorageConnectionConfig; -import com.kingsrook.qqq.backend.core.logging.QLogger; import com.kingsrook.qqq.backend.core.utils.StringUtils; +import com.kingsrook.qqq.backend.module.filesystem.s3.model.metadata.S3BackendMetaData; +import com.kingsrook.qqq.backend.module.filesystem.s3.model.metadata.S3ChecksumMode; +import com.kingsrook.qqq.backend.module.filesystem.s3.utils.S3ClientFactory; import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider; import software.amazon.awssdk.auth.credentials.AwsSessionCredentials; import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider; import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; -import software.amazon.awssdk.core.checksums.RequestChecksumCalculation; -import software.amazon.awssdk.core.checksums.ResponseChecksumValidation; -import software.amazon.awssdk.core.retry.RetryMode; -import software.amazon.awssdk.http.SdkHttpClient; -import software.amazon.awssdk.http.SdkHttpConfigurationOption; -import software.amazon.awssdk.http.apache.ApacheHttpClient; +import software.amazon.awssdk.awscore.AwsRequestOverrideConfiguration; import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.DelegatingS3Client; import software.amazon.awssdk.services.s3.S3Client; -import software.amazon.awssdk.services.s3.S3ClientBuilder; import software.amazon.awssdk.services.s3.S3Configuration; +import software.amazon.awssdk.services.s3.model.S3Request; import software.amazon.awssdk.services.s3.presigner.S3Presigner; -import software.amazon.awssdk.utils.AttributeMap; -import static com.kingsrook.qqq.backend.core.logging.LogUtils.logPair; /******************************************************************************* ** One connection's live state: the settings, one cached SDK v2 S3Client and ** S3Presigner, and the rate limiter. ** - ** STAND-IN: this is the small internal client factory used until QQQ core's - ** S3ClientFactory (qqq #1095, RC.5) is released; replace it then. Nothing - ** outside this package depends on how the client is built. + ** Core owns S3 client construction. This runtime owns its uncached client and + ** closes it on reset/reinstall; the presigner remains local. *******************************************************************************/ public class ConnectionRuntime { - private static final QLogger LOG = QLogger.getLogger(ConnectionRuntime.class); - private final StorageConnectionConfig config; private final RateLimiter rateLimiter; private final String settingsHash; @@ -119,7 +100,7 @@ public class ConnectionRuntime /******************************************************************************* - ** Hash of the settings that shape the client (clients rebuild when it changes). + ** Display hash of connection settings; runtime reset/reinstall rebuilds clients. *******************************************************************************/ public String getSettingsHash() { @@ -217,36 +198,41 @@ public class ConnectionRuntime *******************************************************************************/ private S3Client buildClient() { - S3ClientBuilder builder = S3Client.builder() - .credentialsProvider(credentials(config.getAccessKeyRef(), config.getSecretKeyRef())) - .httpClient(httpClient()) - .forcePathStyle(pathStyle()) - .overrideConfiguration(o -> o.retryStrategy(RetryMode.ADAPTIVE_V2)); - - String endpoint = SecretRefs.resolveSetting(config.getEndpoint()); - if(StringUtils.hasContent(endpoint)) + S3BackendMetaData backend = backendMetaData(); + String token = config.getCredentialsMode() == CredentialsMode.STATIC_REFS ? SecretRefs.resolve(config.getSessionTokenRef()) : null; + AwsCredentialsProvider sessionCredentials = null; + if(token != null && backend.getAccessKey() != null && backend.getSecretKey() != null) { - builder.endpointOverride(URI.create(endpoint)); - } - String region = effectiveRegion(); - if(region != null) - { - builder.region(Region.of(region)); + sessionCredentials = StaticCredentialsProvider.create(AwsSessionCredentials.create(backend.getAccessKey(), backend.getSecretKey(), token)); } - if(config.effectiveChecksumMode() == ChecksumMode.WHEN_REQUIRED) - { - builder.requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED); - builder.responseChecksumValidation(ResponseChecksumValidation.WHEN_REQUIRED); - } - else - { - builder.requestChecksumCalculation(RequestChecksumCalculation.WHEN_SUPPORTED); - builder.responseChecksumValidation(ResponseChecksumValidation.WHEN_SUPPORTED); - } + S3Client coreClient = S3ClientFactory.buildClient(backend); + return (sessionCredentials == null ? coreClient : new SessionCredentialsClient(coreClient, sessionCredentials)); + } - LOG.debug("Building S3 client", logPair("connection", config.getName()), logPair("endpoint", endpoint), logPair("checksumMode", config.effectiveChecksumMode())); - return (builder.build()); + + + /******************************************************************************* + ** Resolve references before passing the connection settings to core. No bucket + ** is mapped: one connection serves multiple buckets. Presigning stays local. + *******************************************************************************/ + S3BackendMetaData backendMetaData() + { + S3BackendMetaData backend = new S3BackendMetaData() + .withEndpoint(SecretRefs.resolveSetting(config.getEndpoint())) + .withRegion(effectiveRegion()) + .withPathStyleAccess(pathStyle()) + .withChecksumMode(S3ChecksumMode.valueOf(config.effectiveChecksumMode().name())) + .withTlsVerify(config.getTlsVerify()) + .withCaCertificatesPem(SecretRefs.resolve(config.getCaBundleRef())) + .withConnectTimeoutMillis(config.getConnectTimeoutMs()) + .withSocketTimeoutMillis(config.getReadTimeoutMs()); + if(config.getCredentialsMode() == CredentialsMode.STATIC_REFS) + { + backend.withAccessKey(SecretRefs.resolve(config.getAccessKeyRef())) + .withSecretKey(SecretRefs.resolve(config.getSecretKeyRef())); + } + return (backend); } @@ -305,139 +291,41 @@ public class ConnectionRuntime /******************************************************************************* - ** Apache HTTP client with the connection's timeouts and TLS settings. + ** STAND-IN(core=QRun-IO/qqq#1165, removeBy=4.1.0-RC.7) + ** RC.5 metadata/factory lacks a session token or credentials-provider hook. + ** Override request credentials only; core still owns the client and transport. *******************************************************************************/ - private SdkHttpClient httpClient() + private static class SessionCredentialsClient extends DelegatingS3Client { - ApacheHttpClient.Builder builder = ApacheHttpClient.builder() - .connectionTimeout(Duration.ofMillis(config.getConnectTimeoutMs())) - .socketTimeout(Duration.ofMillis(config.getReadTimeoutMs())); - - if(Boolean.FALSE.equals(config.getTlsVerify())) - { - LOG.warn("TLS verification is disabled for a storage connection; use only for development", logPair("connection", config.getName())); - return (builder.buildWithDefaults(AttributeMap.builder().put(SdkHttpConfigurationOption.TRUST_ALL_CERTIFICATES, Boolean.TRUE).build())); - } - - String caPem = SecretRefs.resolve(config.getCaBundleRef()); - if(StringUtils.hasContent(caPem)) - { - TrustManager[] trustManagers = trustManagers(caPem); - builder.tlsTrustManagersProvider(() -> trustManagers); - } - return (builder.build()); - } + private final AwsCredentialsProvider credentialsProvider; - /******************************************************************************* - ** Trust the given PEM bundle in addition to the JVM default trust store. - *******************************************************************************/ - static TrustManager[] trustManagers(String pem) - { - try - { - CertificateFactory certificateFactory = CertificateFactory.getInstance("X.509"); - Collection certificates = certificateFactory.generateCertificates(new ByteArrayInputStream(pem.getBytes(StandardCharsets.UTF_8))); - if(certificates.isEmpty()) - { - throw (new IllegalArgumentException("The CA bundle contains no certificates")); - } - KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType()); - keyStore.load(null, null); - int index = 0; - for(Certificate certificate : certificates) - { - keyStore.setCertificateEntry("object-storage-ca-" + index++, certificate); - } - X509TrustManager custom = x509(keyStore); - X509TrustManager fallback = x509(null); - return (new TrustManager[] { new Composite(custom, fallback) }); - } - catch(IllegalArgumentException e) - { - throw (e); - } - catch(Exception e) - { - throw (new IllegalArgumentException("Could not load the CA bundle for a storage connection", e)); - } - } - - - - /******************************************************************************* - ** The X509 trust manager of a key store (null: the JVM default). - *******************************************************************************/ - private static X509TrustManager x509(KeyStore keyStore) throws Exception - { - TrustManagerFactory factory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm()); - factory.init(keyStore); - for(TrustManager trustManager : factory.getTrustManagers()) - { - if(trustManager instanceof X509TrustManager x) - { - return (x); - } - } - throw (new IllegalStateException("No X509TrustManager available")); - } - - - - /******************************************************************************* - ** Trusts a chain if either the private-CA or the default trust manager does. - *******************************************************************************/ - private record Composite(X509TrustManager custom, X509TrustManager fallback) implements X509TrustManager - { /******************************************************************************* - ** Client certs are not used. + ** Wrap a core-built client with the connection's session credentials. *******************************************************************************/ - @Override - public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException + private SessionCredentialsClient(S3Client delegate, AwsCredentialsProvider credentialsProvider) { - try - { - custom.checkClientTrusted(chain, authType); - } - catch(CertificateException e) - { - fallback.checkClientTrusted(chain, authType); - } + super(delegate); + this.credentialsProvider = credentialsProvider; } /******************************************************************************* - ** Server certs. + ** Preserve request overrides and the concrete request type while adding the + ** session token to signing, including streaming and paginated operations. *******************************************************************************/ @Override - public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException + @SuppressWarnings("unchecked") // toBuilder().build() retains the concrete request type T. + protected ReturnT invokeOperation(T request, Function operation) { - try - { - custom.checkServerTrusted(chain, authType); - } - catch(CertificateException e) - { - fallback.checkServerTrusted(chain, authType); - } - } - - - - /******************************************************************************* - ** Accepted issuers of both. - *******************************************************************************/ - @Override - public X509Certificate[] getAcceptedIssuers() - { - X509Certificate[] a = custom.getAcceptedIssuers(); - X509Certificate[] b = fallback.getAcceptedIssuers(); - X509Certificate[] all = new X509Certificate[a.length + b.length]; - System.arraycopy(a, 0, all, 0, a.length); - System.arraycopy(b, 0, all, a.length, b.length); - return (all); + AwsRequestOverrideConfiguration.Builder overrides = request.overrideConfiguration() + .map(AwsRequestOverrideConfiguration::toBuilder).orElseGet(AwsRequestOverrideConfiguration::builder); + T signedRequest = (T) request.toBuilder() + .overrideConfiguration(overrides.credentialsProvider(credentialsProvider).build()) + .build(); + return (operation.apply(signedRequest)); } } } diff --git a/src/main/java/com/kingsrook/qbits/objectstorage/internal/SecretRefs.java b/src/main/java/com/kingsrook/qbits/objectstorage/internal/SecretRefs.java index ce8aa1f..e160d13 100644 --- a/src/main/java/com/kingsrook/qbits/objectstorage/internal/SecretRefs.java +++ b/src/main/java/com/kingsrook/qbits/objectstorage/internal/SecretRefs.java @@ -36,7 +36,8 @@ import com.kingsrook.qqq.backend.core.utils.StringUtils; ** the allow-list. Resolved values live only in memory, never in a record, a log ** line or a process value. ** - ** STAND-IN: ${file.PATH} moves to QMetaDataVariableInterpreter in QQQ core + ** STAND-IN(core=QRun-IO/qqq#1165, removeBy=4.1.0-RC.7) + ** ${file.PATH} moves to QMetaDataVariableInterpreter in QQQ core ** (spec section 8.2); then this class delegates entirely. *******************************************************************************/ public final class SecretRefs diff --git a/src/test/java/com/kingsrook/qbits/objectstorage/internal/ConnectionRuntimeTest.java b/src/test/java/com/kingsrook/qbits/objectstorage/internal/ConnectionRuntimeTest.java new file mode 100644 index 0000000..2285c34 --- /dev/null +++ b/src/test/java/com/kingsrook/qbits/objectstorage/internal/ConnectionRuntimeTest.java @@ -0,0 +1,320 @@ +/* + * QQQ - Low-code Application Framework for Engineers. + * Copyright (C) 2021-2025. Kingsrook, LLC + * 651 N Broad St Ste 205 # 6917 | Middletown DE 19709 | United States + * contact@kingsrook.com + * https://github.com/Kingsrook/ + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.kingsrook.qbits.objectstorage.internal; + + +import java.net.InetSocketAddress; +import java.net.URI; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Duration; +import java.util.ArrayList; +import java.util.List; +import com.kingsrook.qbits.objectstorage.config.AddressingStyle; +import com.kingsrook.qbits.objectstorage.config.ChecksumMode; +import com.kingsrook.qbits.objectstorage.config.CredentialsMode; +import com.kingsrook.qbits.objectstorage.config.ProviderType; +import com.kingsrook.qbits.objectstorage.config.StorageConnectionConfig; +import com.kingsrook.qqq.backend.module.filesystem.s3.model.metadata.S3BackendMetaData; +import com.kingsrook.qqq.backend.module.filesystem.s3.model.metadata.S3ChecksumMode; +import com.sun.net.httpserver.Headers; +import com.sun.net.httpserver.HttpServer; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider; +import software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider; +import software.amazon.awssdk.core.checksums.RequestChecksumCalculation; +import software.amazon.awssdk.core.checksums.ResponseChecksumValidation; +import software.amazon.awssdk.core.sync.RequestBody; +import software.amazon.awssdk.regions.Region; +import software.amazon.awssdk.services.s3.S3Client; +import software.amazon.awssdk.services.s3.model.GetObjectRequest; +import software.amazon.awssdk.services.s3.model.ListObjectsV2Request; +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; + + +/******************************************************************************* + ** Mapping and lifecycle checks plus real SDK signing against a loopback server. + *******************************************************************************/ +class ConnectionRuntimeTest +{ + /******************************************************************************* + ** Every core-supported setting is mapped after resolving references. Session + ** tokens and presign settings have no core fields and are exercised below. + *******************************************************************************/ + @Test + void testBackendMetaDataCarriesEverySetting(@TempDir Path dir) throws Exception + { + StorageConnectionConfig config = new StorageConnectionConfig("mapping") + .withEndpoint(reference(dir, "endpoint", "https://storage.example.test")) + .withRegion(reference(dir, "region", "eu-west-1")) + .withAddressingStyle(AddressingStyle.VIRTUAL_HOST) + .withChecksumMode(ChecksumMode.WHEN_SUPPORTED) + .withTlsVerify(false) + .withCaBundleRef(reference(dir, "ca", "test-pem-bundle")) + .withCredentialsMode(CredentialsMode.STATIC_REFS) + .withAccessKeyRef(reference(dir, "access", "test-access")) + .withSecretKeyRef(reference(dir, "secret", "test-secret")) + .withConnectTimeoutMs(1234) + .withReadTimeoutMs(5678); + S3BackendMetaData backend = new ConnectionRuntime(config).backendMetaData(); + assertThat(backend.getEndpoint()).isEqualTo("https://storage.example.test"); + assertThat(backend.getRegion()).isEqualTo("eu-west-1"); + assertThat(backend.getPathStyleAccess()).isFalse(); + assertThat(backend.getChecksumMode()).isEqualTo(S3ChecksumMode.WHEN_SUPPORTED); + assertThat(backend.getTlsVerify()).isFalse(); + assertThat(backend.getCaCertificatesPem()).isEqualTo("test-pem-bundle"); + assertThat(backend.getAccessKey()).isEqualTo("test-access"); + assertThat(backend.getSecretKey()).isEqualTo("test-secret"); + assertThat(backend.getConnectTimeoutMillis()).isEqualTo(1234); + assertThat(backend.getSocketTimeoutMillis()).isEqualTo(5678); + assertThat(backend.getBucketName()).isNull(); + } + + + + /******************************************************************************* + ** Provider defaults remain the qbit's defaults, including its 5s/60s timeouts. + *******************************************************************************/ + @Test + void testBackendDefaults() + { + S3BackendMetaData compatible = new ConnectionRuntime(new StorageConnectionConfig("compatible") + .withEndpoint("http://localhost:9000")).backendMetaData(); + assertThat(compatible.getRegion()).isEqualTo("us-east-1"); + assertThat(compatible.getPathStyleAccess()).isTrue(); + assertThat(compatible.getChecksumMode()).isEqualTo(S3ChecksumMode.WHEN_REQUIRED); + assertThat(compatible.getTlsVerify()).isTrue(); + assertThat(compatible.getCaCertificatesPem()).isNull(); + assertThat(compatible.getConnectTimeoutMillis()).isEqualTo(5000); + assertThat(compatible.getSocketTimeoutMillis()).isEqualTo(60000); + + S3BackendMetaData aws = new ConnectionRuntime(new StorageConnectionConfig("aws").withProviderType(ProviderType.AWS_S3)).backendMetaData(); + assertThat(aws.getEndpoint()).isNull(); + assertThat(aws.getRegion()).isNull(); + assertThat(aws.getPathStyleAccess()).isFalse(); + assertThat(aws.getChecksumMode()).isEqualTo(S3ChecksumMode.WHEN_SUPPORTED); + } + + + + /******************************************************************************* + ** Default-chain and web-identity modes must ignore configured static refs. + *******************************************************************************/ + @Test + void testChainCredentialsIgnoreStaticRefs(@TempDir Path dir) throws Exception + { + for(CredentialsMode mode : List.of(CredentialsMode.DEFAULT_CHAIN, CredentialsMode.WEB_IDENTITY)) + { + ConnectionRuntime runtime = new ConnectionRuntime(config(dir).withCredentialsMode(mode) + .withSessionTokenRef(reference(dir, "token", "test-session"))); + try + { + assertThat(runtime.backendMetaData().getAccessKey()).isNull(); + assertThat(runtime.backendMetaData().getSecretKey()).isNull(); + assertThat(runtime.client().serviceClientConfiguration().credentialsProvider()).isInstanceOf(DefaultCredentialsProvider.class); + } + finally + { + runtime.close(); + } + } + } + + + + /******************************************************************************* + ** Missing static refs preserve the prior fallback to the SDK default chain. + *******************************************************************************/ + @Test + void testMissingStaticCredentialsFallBack(@TempDir Path dir) throws Exception + { + ConnectionRuntime runtime = new ConnectionRuntime(config(dir).withSecretKeyRef(null)); + try + { + assertThat(runtime.client().serviceClientConfiguration().credentialsProvider()).isInstanceOf(DefaultCredentialsProvider.class); + } + finally + { + runtime.close(); + } + } + + + + /******************************************************************************* + ** Clients belong to a runtime, never core's shared cache; close rebuilds from + ** changed settings and does not affect another connection with equal settings. + *******************************************************************************/ + @Test + void testClientOwnershipAndRebuild(@TempDir Path dir) throws Exception + { + StorageConnectionConfig config = config(dir); + ConnectionRuntime runtime = new ConnectionRuntime(config); + ConnectionRuntime sibling = new ConnectionRuntime(config); + try + { + S3Client first = runtime.client(); + assertThat(runtime.client()).isSameAs(first); + assertThat(sibling.client()).isNotSameAs(first); + assertThat(first.serviceClientConfiguration().endpointOverride()).contains(URI.create("http://localhost:9000")); + assertThat(first.serviceClientConfiguration().region()).isEqualTo(Region.US_EAST_1); + assertThat(((AwsCredentialsProvider) first.serviceClientConfiguration().credentialsProvider()).resolveCredentials().accessKeyId()).isEqualTo("test-access"); + assertThat(first.serviceClientConfiguration().requestChecksumCalculation()).isEqualTo(RequestChecksumCalculation.WHEN_REQUIRED); + assertThat(first.serviceClientConfiguration().responseChecksumValidation()).isEqualTo(ResponseChecksumValidation.WHEN_REQUIRED); + runtime.close(); + config.withRegion("eu-west-1").withChecksumMode(ChecksumMode.WHEN_SUPPORTED); + assertThat(runtime.client()).isNotSameAs(first); + assertThat(runtime.client().serviceClientConfiguration().region()).isEqualTo(Region.EU_WEST_1); + assertThat(runtime.client().serviceClientConfiguration().requestChecksumCalculation()).isEqualTo(RequestChecksumCalculation.WHEN_SUPPORTED); + assertThat(sibling.client().serviceClientConfiguration().region()).isEqualTo(Region.US_EAST_1); + } + finally + { + runtime.close(); + sibling.close(); + } + } + + + + /******************************************************************************* + ** Session credentials sign streaming reads/writes and paginator requests. + ** Existing request headers survive the bridge's credentials override. + *******************************************************************************/ + @Test + void testSessionSigningAndRequestOverrides(@TempDir Path dir) throws Exception + { + List requests = new ArrayList<>(); + HttpServer server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0); + server.createContext("/", exchange -> + { + requests.add(exchange.getRequestHeaders()); + exchange.getRequestBody().readAllBytes(); + byte[] response = (exchange.getRequestURI().getQuery() != null && exchange.getRequestURI().getQuery().contains("list-type") + ? "false" : "data").getBytes(StandardCharsets.UTF_8); + exchange.sendResponseHeaders(200, response.length); + exchange.getResponseBody().write(response); + exchange.close(); + }); + server.start(); + ConnectionRuntime runtime = new ConnectionRuntime(config(dir) + .withEndpoint("http://127.0.0.1:" + server.getAddress().getPort()) + .withSessionTokenRef(reference(dir, "token", "test-session"))); + try + { + GetObjectRequest request = GetObjectRequest.builder().bucket("bucket").key("key") + .overrideConfiguration(o -> o.putHeader("x-test-preserved", "yes")).build(); + assertThat(runtime.client().getObjectAsBytes(request).asUtf8String()).isEqualTo("data"); + runtime.client().putObject(o -> o.bucket("bucket").key("key"), RequestBody.fromString("upload")); + assertThat(runtime.client().listObjectsV2Paginator(ListObjectsV2Request.builder().bucket("bucket").build()).stream().count()).isEqualTo(1); + assertThat(requests).hasSize(3); + for(Headers headers : requests) + { + assertThat(headers.getFirst("x-amz-security-token")).isEqualTo("test-session"); + assertThat(headers.getFirst("Authorization")).contains("Credential=test-access/"); + } + assertThat(requests.getFirst().getFirst("x-test-preserved")).isEqualTo("yes"); + assertThat(request.overrideConfiguration().orElseThrow().credentialsProvider()).isEmpty(); + } + finally + { + runtime.close(); + server.stop(0); + } + } + + + + /******************************************************************************* + ** Presigning keeps its separate public endpoint/key and session-token support. + *******************************************************************************/ + @Test + void testPresignerRemainsLocal(@TempDir Path dir) throws Exception + { + ConnectionRuntime runtime = new ConnectionRuntime(config(dir) + .withPresignEndpoint("https://public.example.test") + .withPresignAccessKeyRef(reference(dir, "presign-access", "presign-access")) + .withPresignSecretKeyRef(reference(dir, "presign-secret", "presign-secret")) + .withSessionTokenRef(reference(dir, "token", "test-session"))); + try + { + assertThat(runtime.presigner()).isSameAs(runtime.presigner()); + String url = runtime.presigner().presignGetObject(p -> p.signatureDuration(Duration.ofMinutes(5)) + .getObjectRequest(g -> g.bucket("bucket").key("key"))).url().toString(); + assertThat(url).startsWith("https://public.example.test/bucket/key?") + .contains("X-Amz-Credential=presign-access%2F", "X-Amz-Security-Token=test-session"); + assertThat(runtime.presignCredentials()).containsExactly("presign-access", "presign-secret", "us-east-1", "https://public.example.test"); + } + finally + { + runtime.close(); + } + } + + + + /******************************************************************************* + ** Core handles invalid CA bundles and the explicit TLS-verification opt-out. + *******************************************************************************/ + @Test + void testTlsSettingsReachCore(@TempDir Path dir) throws Exception + { + StorageConnectionConfig config = config(dir).withCaBundleRef(reference(dir, "ca", "invalid-pem")); + ConnectionRuntime runtime = new ConnectionRuntime(config); + assertThatThrownBy(runtime::client).isInstanceOf(IllegalArgumentException.class).hasMessageContaining("caCertificatesPem"); + config.withTlsVerify(false); + try + { + assertThat(runtime.client()).isNotNull(); + } + finally + { + runtime.close(); + } + } + + + + /******************************************************************************* + ** Deterministic test connection without any external credentials or service. + *******************************************************************************/ + private static StorageConnectionConfig config(Path dir) throws Exception + { + return (new StorageConnectionConfig("test").withEndpoint("http://localhost:9000") + .withAccessKeyRef(reference(dir, "access", "test-access")) + .withSecretKeyRef(reference(dir, "secret", "test-secret"))); + } + + + + /******************************************************************************* + ** Mounted-file reference for test data, scoped to JUnit's temporary directory. + *******************************************************************************/ + private static String reference(Path dir, String name, String value) throws Exception + { + Path file = dir.resolve(name); + Files.writeString(file, value); + return ("${file." + file + "}"); + } +}