Compare commits

...
Author SHA1 Message Date
james.maesandCodex 061f7b8944 ci: use central source scanning
Co-Authored-By: Codex <noreply@openai.com>
2026-10-21 11:38:46 -05:00
james.maesandCodex 2b75c6beb9 ci: adopt central orb 0.7.12 and stable gates (#36)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 03:36:27 -05:00
james.maesandCodex 12ae927c33 ci: align canonical shared orb workers (#35)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 01:33:08 -05:00
james.maesandCodex 6bcc28e332 ci: adopt shared orb 0.7.11 (#34)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 00:58:52 -05:00
james.maesandCodex cc89f73680 ci: centralize All builds smoke checks and releases (#33)
* ci: centralize All builds smoke checks and releases

Use orb 0.7.9 and its immutable shared release worker. Build once, preserve core/full verification and conditional public GHCR delivery, and consume public Circle provenance without exporting a token.

Co-Authored-By: Codex <noreply@openai.com>

* fix: use shared SDK v2 S3 support for MinIO

Replace obsolete SDK v1 client overrides with upstream endpoint and path-style metadata. Preserve endpoint coverage for every filesystem action and local HTTP listing/write assertions.

Co-Authored-By: Codex <noreply@openai.com>

* test: verify the HttpOnly OIDC session contract

Current QQQ omits the session UUID from JSON for HttpOnly cookies. Require the nonempty HttpOnly cookie and reject exposed UUIDs, retaining authenticated backend and permission checks.

Co-Authored-By: Codex <noreply@openai.com>

* fix: accept leading-dash Mongo credentials in Compose

Use explicit username/password option assignments in health and replica-set initialization. Reproduced the original failure and verified authenticated ping with the corrected arguments against mongo:7 and a leading-dash fixture password.

Co-Authored-By: Codex <noreply@openai.com>

---------

Co-authored-by: Codex <noreply@openai.com>
2026-10-20 21:56:33 -05:00
james.maes e35f1f6b99 Merge pull request #28 from QRun-IO/feature/GH-992-smoke-diagnostics
ci(GH-992): retain sanitized full smoke diagnostics
2026-10-01 21:29:33 -05:00
james.maes cedc3023c8 ci(GH-992): identify failures in sequential image pulls
Retain each selected service, image and sanitized pull result before full startup. Refs QRun-IO/qqq#992.
2026-10-01 21:22:00 -05:00
james.maes 281d1bdb07 ci(GH-992): retain sanitized full smoke diagnostics
Record startup, service and artifact identities before private cleanup. Refs QRun-IO/qqq#992; original hosted failure remains unexplained.
2026-10-01 20:57:16 -05:00
james.maes 9a56ec64e2 Merge pull request #27 from QRun-IO/feature/GH-921-qqq-all-rc-path
ci: prepare exact RC1 distribution delivery
2026-10-01 20:24:39 -05:00
james.maes 308830a06a fix: preserve SFTP target in release Compose bundle
Use long volume syntax for Compose 2.38.2 rendering and assert that the variable survives. Refs QRun-IO/qqq#921.
2026-10-01 20:18:36 -05:00
james.maes 1a8d393d03 ci: prepare exact RC1 distribution delivery
Extend the existing release gates and ship an image Compose bundle. Refs QRun-IO/qqq#921.
2026-10-01 20:12:51 -05:00
james.maes d709359317 Merge pull request #26 from QRun-IO/feature/GH-10-next-rc8
feat: pin Next UI RC8 in the stack BOM (#10)
2026-10-01 07:08:27 -05:00
james.maes aafc96a839 feat(GH-10): pin public Next UI RC8 in the stack BOM 2026-10-01 06:58:05 -05:00
james.maes 59e606418c Merge pull request #25 from QRun-IO/feature/GH-10-packaged-notices
fix(packaging): include project notices in JAR and image (#24)
2026-10-01 00:01:21 -05:00
james.maes 6d981933b9 fix(packaging): include project notices in JAR and image (#24) 2026-09-30 23:56:07 -05:00
james.maes de378a70ae Merge pull request #23 from QRun-IO/feature/GH-10-current-release-docs
docs(#10): align release instructions with approved decisions
2026-09-30 23:10:40 -05:00
29 changed files with 1140 additions and 393 deletions

No files matched your search

+10 -24
View File
@@ -1,28 +1,14 @@
---
version: 2.1
setup: true
orbs:
qqq-orb: kingsrook/qqq-orb@0.6.8
commands:
configure-package-install:
steps:
- run:
name: Keep package installation unattended
command: |
# These build machines are discarded after the job.
# Use Ubuntu's archive mirror, which also carries the signed security suite.
sudo sed -i 's|http://security.ubuntu.com/ubuntu|https://archive.ubuntu.com/ubuntu|g' /etc/apt/sources.list
sudo install -d /etc/needrestart/conf.d
printf '%s\n' '$nrconf{restart} = "l";' | sudo tee /etc/needrestart/conf.d/99-qqq-ci.conf >/dev/null
qqq-orb: kingsrook/qqq-orb@0.7.12
workflows:
# Build and unit tests on every branch push. Dependencies are public
# (Maven Central and the Central snapshot repository), so no context is needed.
test_only:
setup:
jobs:
- qqq-orb/mvn_test_only:
pre-steps:
- configure-package-install
- qqq-orb/generate_pipeline:
orb_version: 0.7.12
filters:
branches:
ignore: automation/qbit-rc-pending
tags:
only: /.*/
+23
View File
@@ -0,0 +1,23 @@
name: CodeQL
'on':
push:
branches:
- main
- develop
- release/**
- hotfix/**
pull_request: {}
schedule:
- cron: 41 16 * * 0
workflow_dispatch: {}
permissions:
contents: read
actions: read
security-events: write
packages: read
concurrency:
group: codeql-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
analyze:
uses: QRun-IO/qqq-orb/.github/workflows/qbit-codeql.yml@c288a8864fe9054f68410e1d78f5863aea4c4d22
-48
View File
@@ -1,48 +0,0 @@
name: Publish develop image
on:
push:
branches: [develop]
workflow_dispatch:
permissions:
contents: read
packages: write
jobs:
publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Check public package visibility
id: package_visibility
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
if visibility="$(gh api /orgs/QRun-IO/packages/container/qqq-all --jq .visibility 2>/dev/null)" && [ "$visibility" = public ]; then
echo 'publish=true' >> "$GITHUB_OUTPUT"
echo 'The qqq-all container package is public; develop image publication is enabled.' >> "$GITHUB_STEP_SUMMARY"
else
echo 'publish=false' >> "$GITHUB_OUTPUT"
echo 'Skipped develop image publication: the qqq-all container package is not yet public or cannot be read. An owner must bootstrap the package and set it to Public before C9 enables publication.' >> "$GITHUB_STEP_SUMMARY"
fi
- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: '21'
cache: maven
- name: Verify application
run: mvn -B -ntp verify
- uses: docker/login-action@v3
if: steps.package_visibility.outputs.publish == 'true'
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build and publish develop image
if: steps.package_visibility.outputs.publish == 'true'
run: |
docker build \
--label org.opencontainers.image.source=https://github.com/QRun-IO/qqq-all \
--tag ghcr.io/qrun-io/qqq-all:develop .
docker push ghcr.io/qrun-io/qqq-all:develop
+38 -109
View File
@@ -1,124 +1,53 @@
name: Release 4.1.0
on:
'on':
workflow_dispatch:
inputs:
version:
description: Reviewed source version (GA or exact coordinated RC)
required: true
type: choice
default: 4.1.0
options:
- 4.1.0
- 4.1.0-RC.1
release_sha:
description: Full develop commit SHA already tagged 4.1.0
description: Full develop commit SHA already signed and tagged with the selected
version
required: true
type: string
smoke_run_id:
description: Successful Smoke workflow run ID for that exact develop SHA
required: true
required: false
type: string
default: ''
smoke_evidence:
description: Evidence provider; github preserves the original Smoke workflow
validation
type: choice
default: circle
options:
- github
- circle
required: true
circle_workflow_id:
description: Exact successful Circle workflow UUID when circle evidence is
selected
type: string
required: false
default: ''
permissions:
actions: read
contents: write
packages: write
concurrency:
group: qqq-all-release-4.1.0
cancel-in-progress: false
checks: read
jobs:
release:
runs-on: ubuntu-latest
timeout-minutes: 60
environment: qqq-all-release
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
RELEASE_SHA: ${{ inputs.release_sha }}
SMOKE_RUN_ID: ${{ inputs.smoke_run_id }}
RELEASE_ENABLED: ${{ vars.QQQ_ALL_RELEASE_ENABLED }}
LICENSE_CLEARED: ${{ vars.QQQ_ALL_RELEASE_LICENSE_APPROVED }}
VERSION: 4.1.0
TAG: 4.1.0
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
persist-credentials: false
- name: Check manual release controls and exact tag
shell: bash
run: |
set -euo pipefail
test "$GITHUB_REF" = refs/heads/develop
test "$RELEASE_ENABLED" = true
test "$LICENSE_CLEARED" = true
[[ "$RELEASE_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$SMOKE_RUN_ID" =~ ^[0-9]+$ ]]
test "$(git rev-parse HEAD)" = "$GITHUB_SHA"
test "$RELEASE_SHA" = "$GITHUB_SHA"
test "$(git cat-file -t "refs/tags/$TAG")" = tag
test "$(git rev-parse "refs/tags/$TAG^{commit}")" = "$RELEASE_SHA"
- name: Verify upstream GA, exact-commit smoke, Central POMs, and public GHCR
shell: bash
run: |
set -euo pipefail
ga_state="$(gh api repos/QRun-IO/qqq/releases/tags/v4.1.0 \
--jq 'if (.tag_name == "v4.1.0" and .draft == false and .prerelease == false) then "ready" else "blocked" end')"
test "$ga_state" = ready
gh api "repos/$GITHUB_REPOSITORY/actions/runs/$SMOKE_RUN_ID" > "$RUNNER_TEMP/smoke-run.json"
gh api "repos/$GITHUB_REPOSITORY/actions/runs/$SMOKE_RUN_ID/jobs?per_page=100" > "$RUNNER_TEMP/smoke-jobs.json"
central="https://repo.maven.apache.org/maven2/com/kingsrook/qqq"
for artifact in qqq-all-parent qqq-all-bom; do
curl --fail --silent --show-error \
"$central/$artifact/$VERSION/$artifact-$VERSION.pom" \
--output "$RUNNER_TEMP/$artifact.pom"
done
python3 scripts/release_gate.py --root . --version "$VERSION" --sha "$RELEASE_SHA" \
--smoke-run "$RUNNER_TEMP/smoke-run.json" \
--smoke-jobs "$RUNNER_TEMP/smoke-jobs.json" \
--published-parent "$RUNNER_TEMP/qqq-all-parent.pom" \
--published-bom "$RUNNER_TEMP/qqq-all-bom.pom"
test "$(gh api /orgs/QRun-IO/packages/container/qqq-all --jq .visibility)" = public
gh api --paginate 'orgs/QRun-IO/packages/container/qqq-all/versions?per_page=100' \
--jq '.[] | .metadata.container.tags[]?' > "$RUNNER_TEMP/ghcr-tags.txt"
if grep -Fxq "$VERSION" "$RUNNER_TEMP/ghcr-tags.txt"; then
echo "GHCR tag $VERSION already exists" >&2
exit 1
fi
- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: '21'
cache: maven
- name: Verify release reactor
run: mvn -B -ntp verify
- name: Recheck develop, annotated tag, and release absence before publication
shell: bash
run: |
set -euo pipefail
test "$(gh api "repos/$GITHUB_REPOSITORY/branches/develop" --jq .commit.sha)" = "$RELEASE_SHA"
test "$(git ls-remote origin "refs/tags/$TAG^{}" | cut -f1)" = "$RELEASE_SHA"
if release_check="$(gh api "repos/$GITHUB_REPOSITORY/releases/tags/$TAG" 2>&1)"; then
echo "GitHub release $TAG already exists" >&2
exit 1
elif [[ "$release_check" != *"HTTP 404"* ]]; then
echo "$release_check" >&2
exit 1
fi
- uses: docker/login-action@v3
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Publish versioned image
shell: bash
run: |
set -euo pipefail
image="ghcr.io/qrun-io/qqq-all:$VERSION"
docker build \
--label org.opencontainers.image.source=https://github.com/QRun-IO/qqq-all \
--label "org.opencontainers.image.version=$VERSION" \
--label "org.opencontainers.image.revision=$RELEASE_SHA" \
--tag "$image" .
docker push "$image"
printf 'Published image: `%s`\n' "$image" >> "$GITHUB_STEP_SUMMARY"
- name: Publish GitHub release with Compose asset
shell: bash
run: |
set -euo pipefail
gh release create "$TAG" compose.yaml --repo "$GITHUB_REPOSITORY" \
--verify-tag --title "qqq-all $VERSION" --notes-from-tag
uses: QRun-IO/qqq-orb/.github/workflows/qqq-all-release.yml@0f33ddadf88b9a1b940b8883782fe2000ee1d2bd
with:
version: ${{ inputs.version }}
release_sha: ${{ inputs.release_sha }}
smoke_run_id: ${{ inputs.smoke_run_id }}
smoke_evidence: ${{ inputs.smoke_evidence }}
circle_workflow_id: ${{ inputs.circle_workflow_id }}
secrets:
CIRCLECI_TOKEN: ${{ secrets.CIRCLECI_TOKEN }}
-47
View File
@@ -1,47 +0,0 @@
name: Smoke
on:
pull_request:
push:
branches: [develop]
workflow_dispatch:
permissions:
contents: read
jobs:
core:
runs-on: ubuntu-latest
timeout-minutes: 25
env:
PYTHONDONTWRITEBYTECODE: "1"
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: "21"
cache: maven
- name: Check smoke assertions
run: python3 -m unittest discover -s scripts -p 'test_*.py' -v
- name: Verify and package core app
run: mvn -B -ntp -pl qqq-all-app -am verify
- name: Smoke packaged core over HTTP
run: python3 scripts/run_core_smoke.py qqq-all-app/target/qqq-all-app.jar
full:
runs-on: ubuntu-latest
timeout-minutes: 45
env:
PYTHONDONTWRITEBYTECODE: "1"
steps:
- uses: actions/checkout@v4
- uses: actions/setup-java@v4
with:
distribution: temurin
java-version: "21"
cache: maven
- name: Package full app for Docker image
run: mvn -B -ntp -pl qqq-all-app -am -DskipTests package
- name: Smoke full Compose over HTTP
run: python3 scripts/run_full_smoke.py
+6
View File
@@ -0,0 +1,6 @@
profile: all
orb_version: 0.7.12
default_branch: develop
publish:
snapshot: true
publish_context: qqq-maven-registry-credentials
+1
View File
@@ -10,6 +10,7 @@ RUN apt-get update \
WORKDIR /app
COPY --chown=qqq:qqq qqq-all-app/target/qqq-all-app.jar /app/qqq-all.jar
COPY --chown=qqq:qqq LICENSE NOTICE /app/
USER 10001:10001
EXPOSE 8080
ENV QQQ_ALL_DATA_DIR=/app/data
+8 -1
View File
@@ -2,6 +2,8 @@
A runnable reference app for QQQ 4.1 with two profiles: `core` uses local storage and embedded Artemis; `full` adds external storage, RabbitMQ, OpenSearch quick search, Keycloak, Mailpit, and the included qbits. It is a local demo under construction while QQQ 4.1 finishes. See the [design](docs/design.md), [plan](docs/plan.md), and [epic #1](https://github.com/QRun-IO/qqq-all/issues/1).
The stack pins public Next **1.0.0-RC.8** for application testing. The owner approved an RC carrying its documented Firefox/WebKit/tablet failures, report/query and input parity gaps, sub-path hosting, real Google Drive verification, and visual-approval limitations. These remain open; passing qqq-all HTTP smoke checks do not establish browser parity or GA acceptance. See the [RC8 scope and approval](docs/release.md#next-rc8-integration).
## Quick start
Install Java 21 and Maven, then build the app and its BOM from the repository root:
@@ -39,10 +41,15 @@ Full adds sample tables on PostgreSQL, MySQL, MongoDB, MinIO S3, and SFTP. It in
To stop the full stack, run `docker compose --profile full down`. Add `--volumes` only when you intend to erase demo data. For a fresh Artemis volume, the configured login is required; old volumes created for anonymous access should be recreated. This reference stack uses demo credentials and is not a production deployment.
The [develop image workflow](.github/workflows/publish-image.yml) verifies the build and publishes to GHCR only after an organization owner bootstraps the container package and sets its visibility to public. Until then, image publication is skipped and local Compose remains available.
The [canonical QQQ pipeline](.circleci/config.yml), configured by [.qqq-ci.yml](.qqq-ci.yml), builds once and runs core/full smoke checks through the shared orb. Its develop-image job publishes to GHCR only after the container package exists and its visibility is public. Until then, image publication is skipped and local Compose remains available.
The [4.1.0 release procedure](docs/release.md) describes the separate Maven parent/BOM publication path and the manually gated, exact-commit [release workflow](.github/workflows/release.yml). The workflow will not release the current snapshot build.
## Contributing and license
See [CONTRIBUTING.md](CONTRIBUTING.md), the [Code of Conduct](CODE_OF_CONDUCT.md), and the [security policy](SECURITY.md). Licensed under Apache-2.0; see [LICENSE](LICENSE) and [NOTICE](NOTICE).
The application JAR includes exact copies of these project documents under
`META-INF/qqq-all/`, separate from dependency notices. The container also includes
them at `/app/LICENSE` and `/app/NOTICE`. `PackagedNoticesIT` checks the shaded JAR's
project documents against the root files during the existing `mvn verify` lifecycle.
+4 -2
View File
@@ -124,7 +124,7 @@ services:
- mongo-config:/data/configdb
- ./infra/mongo/start.sh:/start.sh:ro
healthcheck:
test: ["CMD-SHELL", "mongosh --quiet -u \"$$MONGO_INITDB_ROOT_USERNAME\" -p \"$$MONGO_INITDB_ROOT_PASSWORD\" --authenticationDatabase admin --eval 'db.adminCommand({ping:1}).ok' | grep -q 1"]
test: ["CMD-SHELL", "mongosh --quiet --username=\"$$MONGO_INITDB_ROOT_USERNAME\" --password=\"$$MONGO_INITDB_ROOT_PASSWORD\" --authenticationDatabase admin --eval 'db.adminCommand({ping:1}).ok' | grep -q 1"]
interval: 5s
timeout: 5s
retries: 20
@@ -181,7 +181,9 @@ services:
profiles: [full]
command: "${SFTP_USER:-qqq}:${SFTP_PASSWORD:?set SFTP_PASSWORD}:1001:1001:upload"
volumes:
- sftp-data:/home/${SFTP_USER:-qqq}/upload
- type: volume
source: sftp-data
target: /home/${SFTP_USER:-qqq}/upload
healthcheck:
test: ["CMD-SHELL", "pgrep sshd >/dev/null"]
interval: 5s
+23 -5
View File
@@ -1,11 +1,29 @@
# 4.1.0 release preparation
This procedure prepares a release; this repository currently remains on `4.1.0-SNAPSHOT`. The [manual release workflow](../.github/workflows/release.yml) only publishes the versioned GHCR image and a GitHub release containing `compose.yaml`. It does **not** deploy Maven artifacts or create a tag. Publication is authorized once the release gates pass. Keep the workflow disabled until Apache-2.0 alignment, public artifact verification and GHCR visibility are complete.
This procedure prepares a release; this repository currently remains on `4.1.0-SNAPSHOT`. The [manual release workflow](../.github/workflows/release.yml) publishes the versioned GHCR image and a GitHub release containing an image-based Compose bundle plus `compose.yaml`. It does **not** deploy Maven artifacts or create a tag. Publication is authorized once the release gates pass. Keep the workflow disabled until Apache-2.0 alignment, public artifact verification and GHCR visibility are complete.
## Exact coordinated RC under QQQ #921
The same workflow accepts only `4.1.0` (default GA path) or exact `4.1.0-RC.1`. For the [coordinated RC chain](https://github.com/QRun-IO/qqq/issues/921), first publish and verify the accepted Next candidate, QQQ `v4.1.0-RC.1`, and the corresponding build parent/eight QBits. Then review actual source pins: root `revision` and BOM `qqq.version` both `4.1.0-RC.1`, explicit accepted Next version, and immutable published QBit versions. No command-line version overrides substitute for source edits. The current RC8/snapshot pins are intentionally unchanged and cannot pass release validation. Acceptance limitations belong to the exact accepted pair; the RC8 notes below are historical evidence, not automatic approval for a newer pair.
Use the existing `release-central` profile and the parent/BOM-only command below to stage the exact RC POMs for **manual** Central review (`autoPublish=false`). Its enforcers accept only GA or exact RC1, require the QQQ pin to equal the source revision, and reject snapshot/range/LATEST dependency pins. After both POMs are public and the reviewed source is merged, require a successful CircleCI `verify-and-smoke` workflow with `build-and-test`, `core` and `full` on that exact develop SHA and an annotated signed tag named `4.1.0-RC.1` at it. The workflow requires GitHub's tag verification to report `verified=true`, `reason=valid`, and checks the exact local/remote tag object and commit. Public QQQ/Next release tags must match their pins and prerelease state; first-party BOM inputs (including recursive public parents) must resolve anonymously, with qbit-build-parent importing this QQQ version. The full reactor also rejects transitive snapshot dependencies. Public POM comparison includes the explicit Next pin. Existing GHCR visibility, protected environment, license/enable controls, destination-absence checks and live ref rechecks remain required.
When those gates and owner authorization are satisfied, dispatch **Release 4.1.0** on develop with `version=4.1.0-RC.1`, the full `release_sha`, `smoke_evidence=circle`, and its successful `circle_workflow_id`. RC publication uses `--prerelease --latest=false`; it never becomes the latest GA release. The default `version=4.1.0` still requires upstream non-prerelease GA and follows the separate GA requirements below. Do not dispatch while any pin or external prerequisite is pending.
## Image Compose bundle
The workflow attaches `qqq-all-<version>.tar.gz` and its rendered `compose.yaml`. Download/extract the **whole bundle**, retain the `infra` directory beside Compose, and follow its README. Both app profiles use the exact pushed GHCR digest and have no app build context. The bundle includes the tracked PostgreSQL/MySQL/Mongo initialization, Keycloak realm, MinIO Dockerfile, `.env.example`, LICENSE/NOTICE and `release.json` source/image identity. It excludes local `.env`, application source and Maven outputs. Full-profile MinIO still builds locally and needs network access; other service images are unchanged. The standalone Compose asset alone is insufficient for full-profile operation. Configure a unique Compose project and demo credentials, build only MinIO for full, then start with `--no-build`. Verify anonymous image pull and real core/full startup from the extracted bundle after publication; config-rendering tests are not runtime acceptance.
## Next RC8 integration
The stack BOM explicitly pins public `com.kingsrook.qqq:qqq-frontend-next:1.0.0-RC.8`, independently of the framework snapshot's dashboard pin. Its [release source](https://github.com/QRun-IO/qqq-frontend-next/commit/cd3d4731d1910e68e994da9c74a8c28a2f1b7c1c) is `cd3d4731d1910e68e994da9c74a8c28a2f1b7c1c`; the public JAR SHA-256 is `238c9ae80beea336717be759219f0d65a1d614e8e7d6a145a16374cc392a73a1`. The owner [approved a QQQ RC with these documented limitations](https://github.com/QRun-IO/qqq/issues/798#issuecomment-5930629521): Firefox/WebKit/tablet failures, incomplete report/query and input parity, sub-path hosting, real Google Drive verification, and visual approval. See the [RC8 release notes](https://github.com/QRun-IO/qqq-frontend-next/releases/tag/v1.0.0-RC.8) for the retained failures and incomplete runs.
This approval applies to exact RC8 and does not turn those limitations into passing acceptance. The successor contracts `core.widget.data_bag_viewer`, `core.widget.pivot_table_setup`, `core.widget.filter_and_columns_setup`, `core.widget.row_builder`, `core.widget.script_viewer`, `core.widget.blocks`, and `dashboard.acceptance` remain pending and excluded from verified counts, with Next 1.0 final / QQQ 4.1 GA as the follow-up target. qqq-all's package and HTTP smoke checks verify integration and asset delivery, not browser parity. This repository remains a snapshot integration build; the separate GA procedure below and other release/security/public-consumer gates remain in force.
## Prerequisites and Maven artifacts
1. QQQ must publish a non-draft, non-prerelease `v4.1.0` GitHub release with ESB and the Next dashboard included. Release `qbit-build-parent` against the 4.1 BOM, then publish all eight qbits against that parent. This work occurs in their own repositories.
2. In a separate reviewed change to this repository, set root `revision` and the BOM's `qqq.version` to `4.1.0`; pin each qbit to a published non-snapshot version. Run `mvn -B -ntp verify`, the core and full smoke checks, and review the resolved dependencies and effective POMs. Do not use `-Drevision=4.1.0` against a snapshot source tree as a substitute for reviewed pins.
2. In a separate reviewed change to this repository, set root `revision` and the BOM's `qqq.version` to `4.1.0`; replace the explicit Next RC8 pin with the accepted Next 1.0 final version and pin each qbit to a published non-snapshot version. Run `mvn -B -ntp verify`, the core and full smoke checks, and review the resolved dependencies and effective POMs. Do not use `-Drevision=4.1.0` against a snapshot source tree as a substitute for reviewed pins.
3. Stage **both** `com.kingsrook.qqq:qqq-all-parent:4.1.0` and `com.kingsrook.qqq:qqq-all-bom:4.1.0` using the opt-in `release-central` Maven profile. The profile runs the release-version/dependency checks, signs the POMs with GPG, and uses the Central Publishing Maven plugin with `autoPublish=false`. It is only for the parent and BOM reactor; the app and BOM test JAR are excluded. POM packaging needs no source or Javadoc JARs. Configure the owner's `central` Portal user token in a private Maven `settings.xml` and a trusted GPG key outside the repository. Do not place credentials in this project or CI.
```bash
@@ -19,10 +37,10 @@ The owner selected Apache-2.0 for first-party code in [QQQ #923](https://github.
## Exact commit and manual dispatch
After the release POMs are public, merge the reviewed version-pin change and the C7 smoke workflow to `develop`. Require a successful **push** run of `Smoke` on the exact intended commit; both `core` and `full` jobs must pass. Review the run's full SHA and run ID in Actions. Create an **annotated** `4.1.0` tag at that exact `develop` commit through the normal signed-tag process. The workflow checks that the local tag is annotated and that the remote tag resolves to the same SHA; it does not verify a tag signature, so the operator must do that separately. Do not move the tag or advance `develop` before dispatch.
After the release POMs are public, merge the version-pin change to `develop`. Require a successful webhook-triggered CircleCI `verify-and-smoke` workflow on the exact intended commit; `build-and-test`, `core` and `full` must all pass. Record its full SHA and workflow UUID. The shared verifier checks public Circle records against the latest trusted GitHub CircleCI check; it needs no exported Circle token for this public repository. Create an **annotated**, signed `4.1.0` tag at that exact `develop` commit. The workflow checks GitHub's valid signature verification, the tag object and its exact commit, including live remote refs. Do not move the tag or advance `develop` before dispatch.
Configure a protected `qqq-all-release` GitHub environment with required reviewers and branch restriction to `develop`. After Apache alignment and the publication preconditions pass, set environment variables `QQQ_ALL_RELEASE_LICENSE_APPROVED=true` and `QQQ_ALL_RELEASE_ENABLED=true`. Unset them to close the gate. These variables and the environment approval are operational controls, not substitutes for the release review.
Configure a protected `qqq-all-release` GitHub environment with no required human reviewers and branch restriction to `develop`. After Apache alignment and the publication preconditions pass, set environment variables `QQQ_ALL_RELEASE_LICENSE_APPROVED=true` and `QQQ_ALL_RELEASE_ENABLED=true`. Unset them to close the gate. These variables are automated release controls; enable them only after the documented artifact and license prerequisites are established.
Dispatch **Release 4.1.0** on `develop` with `release_sha` (the full 40-character commit SHA) and `smoke_run_id` (the successful push run ID). The job requires that checkout, `develop`, the annotated tag, and the smoke run all refer to that SHA. It checks upstream GA, both public Central POMs, public GHCR visibility, and the absence of a `4.1.0` GHCR tag; runs `mvn -B -ntp verify`; then rechecks the live `develop` and remote tag refs before publication. Only then does it build and push `ghcr.io/qrun-io/qqq-all:4.1.0` and create the GitHub `4.1.0` release with `compose.yaml` attached.
Dispatch **Release 4.1.0** on `develop` with `version=4.1.0` (default), `release_sha` (the full 40-character commit SHA) `smoke_evidence=circle`, and `circle_workflow_id` (the successful Circle workflow UUID). The job requires that checkout, `develop`, the annotated signed tag, and the smoke run all refer to that SHA. It checks upstream GA, both public Central POMs and immutable upstream inputs, public GHCR visibility, and destination absence; verifies the reactor with a fresh public-only cache and rejects transitive snapshots; then rechecks the live `develop` and remote tag refs before publication. Only then does it build and push `ghcr.io/qrun-io/qqq-all:4.1.0` and create the GitHub `4.1.0` release with the Compose bundle and rendered `compose.yaml` attached.
The image push and GitHub release creation are separate API operations, not an atomic transaction. If the release step fails after the image push, inspect the image digest and workflow logs before recovery; never reuse or overwrite the version tag blindly. Verify the public image pull, release asset, and both Maven POMs from an unauthenticated environment after publication. No workflow in this preparation branch publishes or tags anything by itself.
+1 -1
View File
@@ -2,7 +2,7 @@
set -euo pipefail
for attempt in {1..30}; do
if mongosh --host mongo --quiet -u "${MONGO_USER}" -p "${MONGO_PASSWORD}" --authenticationDatabase admin \
if mongosh --host mongo --quiet --username="${MONGO_USER}" --password="${MONGO_PASSWORD}" --authenticationDatabase admin \
--eval 'try { rs.status().ok } catch (e) { if (e.codeName === "NotYetInitialized") { rs.initiate({_id: process.env.MONGO_REPLICA_SET, members: [{_id: 0, host: "mongo:27017"}]}).ok } else { throw e } }' | grep -q 1; then
exit 0
fi
+3 -3
View File
@@ -62,15 +62,15 @@
<version>3.5.0</version>
<executions>
<execution>
<id>require-ga-revision</id>
<id>require-release-revision</id>
<phase>validate</phase>
<goals><goal>enforce</goal></goals>
<configuration>
<rules>
<requireProperty>
<property>revision</property>
<regex>^4[.]1[.]0$</regex>
<message>Central release requires a reviewed 4.1.0 source revision.</message>
<regex>^4[.]1[.]0(-RC[.]1)?$</regex>
<message>Central release requires reviewed 4.1.0 or exact 4.1.0-RC.1 source pins.</message>
</requireProperty>
<requireReleaseDeps/>
</rules>
+15
View File
@@ -60,6 +60,20 @@
<build>
<finalName>qqq-all-app</finalName>
<resources>
<resource>
<directory>src/main/resources</directory>
</resource>
<resource>
<directory>${project.basedir}/..</directory>
<targetPath>META-INF/qqq-all</targetPath>
<filtering>false</filtering>
<includes>
<include>LICENSE</include>
<include>NOTICE</include>
</includes>
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
@@ -93,6 +107,7 @@
<configuration>
<systemPropertyVariables>
<qqqAllJar>${project.build.directory}/${project.build.finalName}.jar</qqqAllJar>
<qqqAllRoot>${project.basedir}/..</qqqAllRoot>
</systemPropertyVariables>
</configuration>
<executions>
@@ -11,16 +11,10 @@ import com.kingsrook.qqq.backend.module.filesystem.s3.model.metadata.S3BackendMe
/** S3 metadata with an explicit endpoint for MinIO. */
public final class MinioBackendMetaData extends S3BackendMetaData
{
private final String endpoint;
public MinioBackendMetaData(String endpoint)
{
this.endpoint = endpoint;
setEndpoint(endpoint);
setPathStyleAccess(true);
setBackendType(MinioBackendModule.class);
}
public String getEndpoint()
{
return endpoint;
}
}
@@ -6,29 +6,11 @@
*/
package com.kingsrook.qqq.all;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.client.builder.AwsClientBuilder;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.S3ObjectSummary;
import com.kingsrook.qqq.backend.core.actions.interfaces.CountInterface;
import com.kingsrook.qqq.backend.core.actions.interfaces.DeleteInterface;
import com.kingsrook.qqq.backend.core.actions.interfaces.InsertInterface;
import com.kingsrook.qqq.backend.core.actions.interfaces.QStorageInterface;
import com.kingsrook.qqq.backend.core.actions.interfaces.QueryInterface;
import com.kingsrook.qqq.backend.core.model.metadata.QBackendMetaData;
import com.kingsrook.qqq.backend.core.modules.backend.QBackendModuleDispatcher;
import com.kingsrook.qqq.backend.module.filesystem.base.actions.AbstractBaseFilesystemAction;
import com.kingsrook.qqq.backend.module.filesystem.s3.S3BackendModule;
import com.kingsrook.qqq.backend.module.filesystem.s3.actions.AbstractS3Action;
import com.kingsrook.qqq.backend.module.filesystem.s3.actions.S3CountAction;
import com.kingsrook.qqq.backend.module.filesystem.s3.actions.S3DeleteAction;
import com.kingsrook.qqq.backend.module.filesystem.s3.actions.S3InsertAction;
import com.kingsrook.qqq.backend.module.filesystem.s3.actions.S3QueryAction;
import com.kingsrook.qqq.backend.module.filesystem.s3.actions.S3StorageAction;
/** Routes every S3 operation to the configured MinIO endpoint. */
/** Uses the shared S3 actions with MinIO endpoint metadata. */
public final class MinioBackendModule extends S3BackendModule
{
static
@@ -47,92 +29,4 @@ public final class MinioBackendModule extends S3BackendModule
{
return MinioBackendMetaData.class;
}
private static AmazonS3 client(QBackendMetaData metadata)
{
MinioBackendMetaData minio = (MinioBackendMetaData) metadata;
return AmazonS3ClientBuilder.standard()
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(minio.getEndpoint(), minio.getRegion()))
.withPathStyleAccessEnabled(true)
.withCredentials(new AWSStaticCredentialsProvider(new BasicAWSCredentials(minio.getAccessKey(), minio.getSecretKey())))
.build();
}
@Override
public AbstractBaseFilesystemAction<S3ObjectSummary> getActionBase()
{
return new AbstractS3Action()
{
@Override
protected AmazonS3 buildAmazonS3ClientFromBackendMetaData(QBackendMetaData metadata)
{
return client(metadata);
}
};
}
@Override
public QueryInterface getQueryInterface()
{
return new S3QueryAction()
{
@Override
protected AmazonS3 buildAmazonS3ClientFromBackendMetaData(QBackendMetaData metadata)
{
return client(metadata);
}
};
}
@Override
public InsertInterface getInsertInterface()
{
return new S3InsertAction()
{
@Override
protected AmazonS3 buildAmazonS3ClientFromBackendMetaData(QBackendMetaData metadata)
{
return client(metadata);
}
};
}
@Override
public CountInterface getCountInterface()
{
return new S3CountAction()
{
@Override
protected AmazonS3 buildAmazonS3ClientFromBackendMetaData(QBackendMetaData metadata)
{
return client(metadata);
}
};
}
@Override
public DeleteInterface getDeleteInterface()
{
return new S3DeleteAction()
{
@Override
protected AmazonS3 buildAmazonS3ClientFromBackendMetaData(QBackendMetaData metadata)
{
return client(metadata);
}
};
}
@Override
public QStorageInterface getStorageInterface()
{
return new S3StorageAction()
{
@Override
protected AmazonS3 buildAmazonS3ClientFromBackendMetaData(QBackendMetaData metadata)
{
return client(metadata);
}
};
}
}
@@ -44,7 +44,7 @@ class MinioBackendTest
action.preAction(backend);
S3Utils utils = utils(action);
assertEquals("http://minio.example.test:9000/qqq-demo/example.json",
utils.getAmazonS3().getUrl("qqq-demo", "example.json").toString());
utils.getS3Client().utilities().getUrl(request -> request.bucket("qqq-demo").key("example.json")).toString());
}
@Test
@@ -88,8 +88,8 @@ class MinioBackendTest
AbstractS3Action query = (AbstractS3Action) module.getQueryInterface();
query.preAction(backend);
assertEquals("documents/example.json", utils(query).getAmazonS3()
.listObjectsV2("qqq-demo").getObjectSummaries().getFirst().getKey());
assertEquals("documents/example.json", utils(query).getS3Client()
.listObjectsV2(request -> request.bucket("qqq-demo")).contents().getFirst().key());
AbstractS3Action insert = (AbstractS3Action) module.getInsertInterface();
insert.preAction(backend);
@@ -0,0 +1,40 @@
/*
* Copyright 2026 Kingsrook, LLC
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0
*/
package com.kingsrook.qqq.all;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.jar.JarFile;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
/** Verifies this project's notices in the actual shaded application archive. */
class PackagedNoticesIT
{
@ParameterizedTest
@ValueSource(strings = {"LICENSE", "NOTICE"})
void packagedProjectDocumentMatchesRoot(String document) throws Exception
{
Path rootDocument = Path.of(System.getProperty("qqqAllRoot"), document);
try(JarFile jar = new JarFile(System.getProperty("qqqAllJar")))
{
String name = "META-INF/qqq-all/" + document;
var entry = jar.getJarEntry(name);
assertNotNull(entry, "Missing project document: " + name);
assertEquals(1L, jar.stream().filter(candidate -> candidate.getName().equals(name)).count(),
"Project document must occur exactly once: " + name);
try(var input = jar.getInputStream(entry))
{
assertArrayEquals(Files.readAllBytes(rootDocument), input.readAllBytes(),
"Packaged document must preserve root bytes: " + document);
}
}
}
}
@@ -97,6 +97,7 @@ class StackBomTest
List<String> missing = required.stream().filter(key -> !managedVersions.containsKey(key)).toList();
assertEquals(List.of(), missing, "Artifacts the stack BOM does not manage");
assertEquals("3.9.0", managedVersions.get("com.rabbitmq.jms:rabbitmq-jms"));
assertEquals("1.0.0-RC.8", managedVersions.get("com.kingsrook.qqq:qqq-frontend-next"));
}
+19 -10
View File
@@ -18,6 +18,7 @@
<properties>
<qqq.version>4.1.0-SNAPSHOT</qqq.version>
<qqq-frontend-next.version>1.0.0-RC.8</qqq-frontend-next.version>
<!-- qBits: develop builds on qbit-build-parent 2.0.0; none has a release on that parent yet. -->
<qbit-quick-search.version>0.2.1-SNAPSHOT</qbit-quick-search.version>
@@ -55,16 +56,17 @@
<rules>
<requireProperty>
<property>qqq.version</property>
<regex>^4[.]1[.]0$</regex>
<regex>^\Q${revision}\E$</regex>
</requireProperty>
<requireProperty><property>qbit-quick-search.version</property><regex>^(?!.*SNAPSHOT).+$</regex></requireProperty>
<requireProperty><property>qbit-user-role-permissions.version</property><regex>^(?!.*SNAPSHOT).+$</regex></requireProperty>
<requireProperty><property>qbit-customizable-table-views.version</property><regex>^(?!.*SNAPSHOT).+$</regex></requireProperty>
<requireProperty><property>qbit-standard-process-trace.version</property><regex>^(?!.*SNAPSHOT).+$</regex></requireProperty>
<requireProperty><property>qbit-webhooks.version</property><regex>^(?!.*SNAPSHOT).+$</regex></requireProperty>
<requireProperty><property>qbit-workflows.version</property><regex>^(?!.*SNAPSHOT).+$</regex></requireProperty>
<requireProperty><property>qbit-geo-data.version</property><regex>^(?!.*SNAPSHOT).+$</regex></requireProperty>
<requireProperty><property>qbit-sftp-data-integration.version</property><regex>^(?!.*SNAPSHOT).+$</regex></requireProperty>
<requireProperty><property>qqq-frontend-next.version</property><regex>^[0-9]+[.][0-9]+[.][0-9]+(-RC[.][1-9][0-9]*)?$</regex></requireProperty>
<requireProperty><property>qbit-quick-search.version</property><regex>^[0-9]+[.][0-9]+[.][0-9]+(-RC[.][1-9][0-9]*)?$</regex></requireProperty>
<requireProperty><property>qbit-user-role-permissions.version</property><regex>^[0-9]+[.][0-9]+[.][0-9]+(-RC[.][1-9][0-9]*)?$</regex></requireProperty>
<requireProperty><property>qbit-customizable-table-views.version</property><regex>^[0-9]+[.][0-9]+[.][0-9]+(-RC[.][1-9][0-9]*)?$</regex></requireProperty>
<requireProperty><property>qbit-standard-process-trace.version</property><regex>^[0-9]+[.][0-9]+[.][0-9]+(-RC[.][1-9][0-9]*)?$</regex></requireProperty>
<requireProperty><property>qbit-webhooks.version</property><regex>^[0-9]+[.][0-9]+[.][0-9]+(-RC[.][1-9][0-9]*)?$</regex></requireProperty>
<requireProperty><property>qbit-workflows.version</property><regex>^[0-9]+[.][0-9]+[.][0-9]+(-RC[.][1-9][0-9]*)?$</regex></requireProperty>
<requireProperty><property>qbit-geo-data.version</property><regex>^[0-9]+[.][0-9]+[.][0-9]+(-RC[.][1-9][0-9]*)?$</regex></requireProperty>
<requireProperty><property>qbit-sftp-data-integration.version</property><regex>^[0-9]+[.][0-9]+[.][0-9]+(-RC[.][1-9][0-9]*)?$</regex></requireProperty>
</rules>
</configuration>
</execution>
@@ -77,7 +79,7 @@
<dependencyManagement>
<dependencies>
<!-- QQQ modules, qqq-esb, and qqq-frontend-next. -->
<!-- QQQ modules and qqq-esb. -->
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-bom-pom</artifactId>
@@ -86,6 +88,13 @@
<scope>import</scope>
</dependency>
<!-- Keep the accepted dashboard candidate independent of framework snapshots. -->
<dependency>
<groupId>com.kingsrook.qqq</groupId>
<artifactId>qqq-frontend-next</artifactId>
<version>${qqq-frontend-next.version}</version>
</dependency>
<!-- qBits -->
<dependency>
<groupId>com.kingsrook.qbits</groupId>
+99
View File
@@ -0,0 +1,99 @@
#!/usr/bin/env python3
"""Package digest-pinned Compose with the tracked full-profile infrastructure."""
import argparse
import json
import os
import re
import shutil
import subprocess
import tarfile
from pathlib import Path
from release_gate import validate_version
def create_bundle(root, version, sha, image, output):
validate_version(version)
if not re.fullmatch(r"[0-9a-f]{40}", sha):
raise ValueError("Source SHA must be a full commit ID")
if not re.fullmatch(r"ghcr\.io/qrun-io/qqq-all@sha256:[0-9a-f]{64}", image):
raise ValueError("Release image must be the qqq-all GHCR digest")
environment = {key: value for key, value in os.environ.items() if not key.startswith("COMPOSE_")}
rendered = subprocess.run(
["docker", "compose", "--env-file", ".env.example", "--profile", "*", "-f", "compose.yaml",
"config", "--no-interpolate", "--no-env-resolution", "--no-path-resolution", "--format", "json"],
cwd=root, env=environment, capture_output=True, text=True, check=True,
)
compose = json.loads(rendered.stdout)
compose.pop("x-app-image", None)
for name in ("app-core", "app-full"):
app = compose["services"][name]
app.pop("build", None)
app["image"] = image
app["pull_policy"] = "always"
compose["services"]["minio"]["image"] = "qqq-all-minio:" + version
name = "qqq-all-" + version
bundle = output / name
bundle.mkdir(parents=True, exist_ok=False)
(bundle / "compose.yaml").write_text(json.dumps(compose, indent=2) + "\n")
tracked = subprocess.check_output(["git", "-C", str(root), "ls-files", "--", "infra"], text=True).splitlines()
for filename in (".env.example", "LICENSE", "NOTICE", *tracked):
source = root / filename
if source.is_symlink() or not source.is_file():
raise ValueError(f"Bundle input must be a regular file: {filename}")
target = bundle / filename
target.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(source, target)
(bundle / "release.json").write_text(json.dumps(
{"version": version, "source_sha": sha, "image": image}, indent=2) + "\n")
(bundle / "README.md").write_text(f"""# qqq-all {version}
Extract this complete bundle and run commands from this directory. The application uses
`{image}` (source `{sha}`). No application source, Maven or app build is required.
Docker with Compose v2 is required. Copy `.env.example` to `.env` and replace
the demonstration passwords before sharing the environment. All fixture values remain
demo-only; retain the infra directory next to compose.yaml.
```bash
cp .env.example .env
docker compose -p qqq-all-demo --profile core up -d --wait --no-build
```
For the full profile, stop core first. The included MinIO Dockerfile still builds locally
and needs network access; PostgreSQL/MySQL/Mongo initialization and the Keycloak realm
use the bundled infra files. External service images also need network access.
```bash
docker compose -p qqq-all-demo --profile core down
docker compose -p qqq-all-demo --profile full build minio
docker compose -p qqq-all-demo --profile full up -d --wait --no-build
```
Use a distinct Compose project name for each installation. The application listens on
http://localhost:8080 by default. The full-profile identity provider uses
http://keycloak.localhost:8081; that hostname must resolve to loopback on the client.
Ports and fixture settings are in `.env.example`. `docker compose ... down` retains
named data volumes; remove volumes only when intentionally resetting that installation.
The standalone compose.yaml asset needs this bundle's infra and environment template
for full-profile operation. See release.json for the exact application identity.
""")
archive = output / (name + ".tar.gz")
with tarfile.open(archive, "x:gz") as handle:
handle.add(bundle, arcname=name)
return archive
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--root", type=Path, required=True)
parser.add_argument("--version", required=True)
parser.add_argument("--sha", required=True)
parser.add_argument("--image", required=True)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
print(create_bundle(args.root.resolve(), args.version, args.sha, args.image, args.output.resolve()))
if __name__ == "__main__":
main()
+124 -11
View File
@@ -1,9 +1,12 @@
"""Fail-closed checks for the manual 4.1.0 release workflow."""
"""Fail-closed checks for the manual 4.1.0 GA / exact RC.1 release workflow."""
import argparse
import hashlib
import json
import re
import sys
import urllib.error
import urllib.request
from pathlib import Path
from xml.etree import ElementTree
@@ -13,6 +16,16 @@ QBITS = (
"standard-process-trace", "webhooks", "workflows", "geo-data",
"sftp-data-integration",
)
RELEASE_VERSIONS = ("4.1.0", "4.1.0-RC.1")
def immutable_version(value):
return re.fullmatch(r"[0-9]+(?:\.[0-9]+){2,3}(?:-RC\.[1-9][0-9]*)?", value) is not None
def validate_version(version):
if version not in RELEASE_VERSIONS:
raise GateError("This release workflow is limited to 4.1.0 and 4.1.0-RC.1")
class GateError(ValueError):
@@ -64,7 +77,7 @@ def managed_dependencies(path):
reference = re.fullmatch(r"\$\{([^}]+)\}", version)
if reference:
version = values.get(reference.group(1), "")
if not version or "${" in version or "SNAPSHOT" in version.upper():
if not immutable_version(version):
raise GateError(f"Unreleased managed dependency in {path}: "
f"{field(dependency, 'artifactId')}")
managed.append((field(dependency, "groupId"), field(dependency, "artifactId"),
@@ -84,6 +97,9 @@ def validate_managed_bom(path, version):
if entries.get(("com.kingsrook.qqq", "qqq-bom-pom")) != (version, "pom", "import"):
raise GateError(f"QQQ BOM import differs from {version} in {path}")
values = properties(path)
if entries.get(("com.kingsrook.qqq", "qqq-frontend-next")) != (
values.get("qqq-frontend-next.version"), "jar", "compile"):
raise GateError(f"Managed Next differs from its release pin in {path}")
for name in QBITS:
artifact = f"qbit-{name}"
if entries.get(("com.kingsrook.qbits", artifact)) != (
@@ -102,14 +118,14 @@ def validate_published_poms(parent_path, bom_path, version, local_bom):
raise GateError(f"Published {artifact} POM is not the {version} release")
published = properties(bom_path)
if published.get("qqq.version") != version:
raise GateError("Published BOM does not import QQQ 4.1.0")
raise GateError(f"Published BOM does not import QQQ {version}")
for name in QBITS:
key = f"qbit-{name}.version"
value = published.get(key, "")
if not value or "SNAPSHOT" in value.upper():
if not immutable_version(value):
raise GateError(f"Published BOM {key} is not a release")
local = properties(local_bom)
for key in ("qqq.version", *(f"qbit-{name}.version" for name in QBITS)):
for key in ("qqq.version", "qqq-frontend-next.version", *(f"qbit-{name}.version" for name in QBITS)):
if published.get(key) != local.get(key):
raise GateError(f"Published BOM {key} differs from release source")
if validate_managed_bom(bom_path, version) != validate_managed_bom(local_bom, version):
@@ -117,20 +133,19 @@ def validate_published_poms(parent_path, bom_path, version, local_bom):
def validate_poms(root, version):
if version != "4.1.0":
raise GateError("This release workflow is limited to 4.1.0")
validate_version(version)
parent = properties(root / "pom.xml")
bom = properties(root / "qqq-all-bom/pom.xml")
if parent.get("revision") != version or bom.get("qqq.version") != version:
raise GateError("Parent revision and QQQ BOM pin must both equal 4.1.0")
raise GateError(f"Parent revision and QQQ BOM pin must both equal {version}")
for name in QBITS:
key = f"qbit-{name}.version"
value = bom.get(key, "")
if not value or "SNAPSHOT" in value.upper():
if not immutable_version(value):
raise GateError(f"{key} must be a published non-snapshot version")
for key, value in bom.items():
if key.endswith(".version") and "SNAPSHOT" in value.upper():
raise GateError(f"{key} still points to a snapshot")
if key.endswith(".version") and not immutable_version(value):
raise GateError(f"{key} must be an immutable release version")
validate_managed_bom(root / "qqq-all-bom/pom.xml", version)
@@ -156,6 +171,94 @@ def validate_smoke(run, jobs, sha):
raise GateError("Smoke run must include successful core and full jobs")
def validate_upstream_release(release, version):
if (not immutable_version(version) or release.get("tag_name") != "v" + version
or release.get("draft") is not False
or release.get("prerelease") is not ("-RC." in version)):
raise GateError(f"Upstream release must be public v{version} with matching prerelease state")
def validate_tag(tag, version, sha, tag_sha):
validate_version(version)
if not all(re.fullmatch(r"[0-9a-f]{40}", value) for value in (sha, tag_sha)):
raise GateError("Tag and commit IDs must be full Git SHAs")
if (tag.get("sha") != tag_sha or tag.get("tag") != version
or tag.get("object", {}).get("type") != "commit"
or tag.get("object", {}).get("sha") != sha
or tag.get("verification", {}).get("verified") is not True
or tag.get("verification", {}).get("reason") != "valid"):
raise GateError("Release tag must be signed, verified, annotated and point to the exact release commit")
def download_public_inputs(bom, version, destination, central="https://repo.maven.apache.org/maven2"):
"""Fetch the declared first-party inputs and their parents without credentials."""
validate_version(version)
destination.mkdir(parents=True, exist_ok=True)
receipts = []
visited = set()
def fetch(group, artifact, pin, kind):
coordinate = (group, artifact, pin, kind)
if coordinate in visited:
return
if (not re.fullmatch(r"[A-Za-z0-9_-]+(?:\.[A-Za-z0-9_-]+)*", group)
or not re.fullmatch(r"[A-Za-z0-9_-]+", artifact) or not immutable_version(pin)):
raise GateError(f"Invalid or mutable public coordinate: {coordinate}")
visited.add(coordinate)
relative = f"{group.replace('.', '/')}/{artifact}/{pin}/{artifact}-{pin}.{kind}"
url = central + "/" + relative
try:
with urllib.request.urlopen(url, timeout=30) as response:
data = response.read()
except (urllib.error.URLError, OSError) as error:
if isinstance(error, urllib.error.HTTPError):
error.close()
raise GateError(f"Public input unavailable: {relative}") from error
if not data:
raise GateError(f"Empty public input: {relative}")
path = destination / relative
path.parent.mkdir(parents=True, exist_ok=True)
path.write_bytes(data)
if kind == "pom":
if coordinates(path) != (group, artifact, pin):
raise GateError(f"Public POM coordinates differ: {relative}")
root = ElementTree.fromstring(data)
ns = xml_namespace(root)
parent = root.find(f"{ns}parent")
if parent is not None:
parent_group = parent.findtext(f"{ns}groupId", "")
parent_pin = parent.findtext(f"{ns}version", "")
if not immutable_version(parent_pin):
raise GateError(f"Mutable parent in {relative}")
if parent_group.startswith("com.kingsrook"):
fetch(parent_group, parent.findtext(f"{ns}artifactId", ""), parent_pin, "pom")
if artifact == "qbit-build-parent":
imports = root.findall(f"{ns}dependencyManagement/{ns}dependencies/{ns}dependency")
props = root.find(f"{ns}properties")
values = {child.tag.removeprefix(ns): child.text for child in props} if props is not None else {}
qqq_imports = [dep for dep in imports if dep.findtext(f"{ns}groupId") == "com.kingsrook.qqq"
and dep.findtext(f"{ns}artifactId") == "qqq-bom-pom"]
if len(qqq_imports) != 1:
raise GateError("Public qbit-build-parent must import the candidate QQQ BOM")
dep = qqq_imports[0]
actual = dep.findtext(f"{ns}version", "")
actual = values.get(actual[2:-1], "") if actual.startswith("${") else actual
if (actual != version or dep.findtext(f"{ns}type") != "pom"
or dep.findtext(f"{ns}scope") != "import"):
raise GateError("Public qbit-build-parent QQQ import differs from the candidate")
receipts.append({"url": url, "sha256": hashlib.sha256(data).hexdigest(), "bytes": len(data)})
for group, artifact, pin, kind, _, classifier in validate_managed_bom(bom, version):
if group.startswith("com.kingsrook"):
if classifier or kind not in ("jar", "pom"):
raise GateError(f"Unsupported first-party release input: {artifact}")
fetch(group, artifact, pin, "pom")
if kind == "jar":
fetch(group, artifact, pin, "jar")
(destination / "receipts.json").write_text(json.dumps(receipts, indent=2) + "\n")
return receipts
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--root", type=Path, required=True)
@@ -165,13 +268,23 @@ def main():
parser.add_argument("--smoke-jobs", type=Path, required=True)
parser.add_argument("--published-parent", type=Path, required=True)
parser.add_argument("--published-bom", type=Path, required=True)
parser.add_argument("--upstream-release", type=Path, required=True)
parser.add_argument("--next-release", type=Path, required=True)
parser.add_argument("--tag-object", type=Path, required=True)
parser.add_argument("--tag-sha", required=True)
parser.add_argument("--public-inputs", type=Path, required=True)
args = parser.parse_args()
try:
validate_poms(args.root, args.version)
validate_upstream_release(json.loads(args.upstream_release.read_text()), args.version)
validate_upstream_release(json.loads(args.next_release.read_text()),
properties(args.root / "qqq-all-bom/pom.xml")["qqq-frontend-next.version"])
validate_tag(json.loads(args.tag_object.read_text()), args.version, args.sha, args.tag_sha)
validate_smoke(json.loads(args.smoke_run.read_text()),
json.loads(args.smoke_jobs.read_text()), args.sha)
validate_published_poms(args.published_parent, args.published_bom, args.version,
args.root / "qqq-all-bom/pom.xml")
download_public_inputs(args.root / "qqq-all-bom/pom.xml", args.version, args.public_inputs)
except (GateError, OSError, ElementTree.ParseError, json.JSONDecodeError) as error:
print(f"Release gate failed: {error}", file=sys.stderr)
return 1
+23 -7
View File
@@ -12,6 +12,7 @@ import urllib.parse
from pathlib import Path
from smoke import check_full
from smoke_diagnostics import Diagnostics
ROOT = Path(__file__).resolve().parent.parent
@@ -95,6 +96,7 @@ def main():
environment = os.environ.copy()
environment.update(values)
os.environ.update(values)
diagnostics = Diagnostics(ROOT / "target/smoke-diagnostics", environment)
with tempfile.TemporaryDirectory(prefix="qqq-all-full-smoke-") as directory:
env_file = Path(directory) / "compose.env"
env_file.write_text("".join(f"{key}={value}\n" for key, value in values.items()))
@@ -103,10 +105,12 @@ def main():
"-f", str(ROOT / "compose.yaml"), "-f", str(ROOT / "scripts/compose.smoke.yaml"),
"--profile", "full"]
try:
started = subprocess.run(command + ["up", "-d", "--build", "--wait", "--wait-timeout", "360"],
cwd=ROOT, env=environment, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL, timeout=480, check=False)
if started.returncode:
diagnostics.jar_hashes(ROOT)
diagnostics.pull(command, ROOT, environment)
started = diagnostics.run(command + ["up", "-d", "--build", "--wait", "--wait-timeout", "360"],
ROOT, environment, timeout=480)
diagnostics.write("startup.json", started)
if started["exit_code"] != 0:
raise AssertionError("full Compose stack did not become healthy")
prepare_sftp_import_directory(command, environment, values.get("SFTP_USER", "qqq"))
check_full(
@@ -115,10 +119,22 @@ def main():
lambda index, query: query_opensearch(command, environment, index, query),
120,
)
except (AssertionError, OSError, ValueError, subprocess.TimeoutExpired) as error:
diagnostics.write("failure.json", {"type": type(error).__name__,
"message": diagnostics.sanitize(str(error))})
raise AssertionError(diagnostics.sanitize(str(error))) from None
finally:
subprocess.run(command + ["down", "--volumes", "--remove-orphans"],
cwd=ROOT, env=environment, stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL, timeout=120, check=False)
try:
diagnostics.collect(command, ROOT, environment)
except Exception as error:
# Evidence failure must not suppress the smoke error or private-stack cleanup.
print(f"smoke diagnostics collection failed: {type(error).__name__}", file=sys.stderr)
finally:
cleanup = diagnostics.run(command + ["down", "--volumes", "--remove-orphans"],
ROOT, environment, timeout=120)
diagnostics.write("cleanup.json", cleanup)
if cleanup["exit_code"] != 0 and sys.exc_info()[0] is None:
raise AssertionError("private Compose cleanup failed; see sanitized diagnostics")
if __name__ == "__main__":
+11 -2
View File
@@ -119,6 +119,16 @@ class SmokeClient:
raise AssertionError(f"missing expected {table} {field}")
def require_oidc_session(client, result):
# Next uses an HttpOnly cookie; current QQQ intentionally omits its UUID from JSON.
cookies = [cookie for cookie in client.cookies if cookie.name == "sessionUUID"]
if (len(cookies) != 1 or not cookies[0].value
or not cookies[0].has_nonstandard_attr("HttpOnly")):
raise AssertionError("QQQ did not establish an HttpOnly OIDC session")
if "uuid" in result:
raise AssertionError("QQQ exposed the HttpOnly session UUID in JSON")
def allow_loopback_http_session(client):
parsed = urllib.parse.urlsplit(client.base_url)
if parsed.scheme != "http" or parsed.hostname not in ("127.0.0.1", "localhost"):
@@ -273,8 +283,7 @@ def oidc_login(client, keycloak_url, username, password):
result = client.json("/manageSession", {
"code": parameters["code"][0], "redirectUri": callback, "codeVerifier": verifier
})
if not result.get("uuid") or not any(cookie.name == "sessionUUID" for cookie in client.cookies):
raise AssertionError("QQQ did not establish an OIDC session")
require_oidc_session(client, result)
if urllib.parse.urlsplit(client.base_url).scheme == "http":
allow_loopback_http_session(client)
except (OSError, KeyError, ValueError, urllib.error.HTTPError) as error:
+177
View File
@@ -0,0 +1,177 @@
#!/usr/bin/env python3
"""Small, sanitized receipts for the private full-smoke stack only."""
import base64
import hashlib
import json
import os
import re
import subprocess
import sys
import time
import urllib.parse
from pathlib import Path
LIMIT = 65536
CONTAINER_FORMAT = ('{"id":{{json .Id}},"service":{{json (index .Config.Labels '
'"com.docker.compose.service")}},"image_id":{{json .Image}},'
'"status":{{json .State.Status}},"exit_code":{{json .State.ExitCode}},'
'"oom_killed":{{json .State.OOMKilled}},"error":{{json .State.Error}},'
'"health":{{if index .State "Health"}}{{json (index .State "Health")}}{{else}}null{{end}}}')
IMAGE_FORMAT = ('{"id":{{json .Id}},"repo_digests":{{json .RepoDigests}},'
'"architecture":{{json .Architecture}},"os":{{json .Os}}}')
class Diagnostics:
def __init__(self, directory, environment):
self.directory = directory
self.secrets = set()
for key, value in environment.items():
if value and re.search(r'PASSWORD|PASSWD|SECRET|TOKEN|KEYFILE|ACCESS_KEY|PRIVATE_KEY|CREDENTIAL|AUTHORIZATION', key, re.I):
self.secrets.update((value, urllib.parse.quote(value, safe=''),
urllib.parse.quote_plus(value),
base64.b64encode(value.encode()).decode()))
def sanitize(self, text):
if isinstance(text, bytes):
text = text.decode('utf-8', errors='replace')
text = text or ''
text = re.sub(r'\x1b\[[0-?]*[ -/]*[@-~]', '', text)
for secret in sorted(self.secrets, key=len, reverse=True):
text = text.replace(secret, '[REDACTED]')
text = re.sub(r'(?i)(\b[\w.-]{0,80}(?:password|passwd|secret|token|authorization|credential|api[_-]?key|access[_-]?key)\b["\s]*[:=]\s*)("[^"]*"|\x27[^\x27]*\x27|[^\s,;]+)', r'\1[REDACTED]', text)
text = re.sub(r'(?i)\b(Bearer|Basic)\s+[^\s,;]+', r'\1 [REDACTED]', text)
text = re.sub(r'(?i)\b([a-z][a-z0-9+.-]*://)[^/\s@]+@', r'\1[REDACTED]@', text)
if len(text) > LIMIT:
text = '[...truncated...]\n' + text[-(LIMIT - 18):]
return text
def write(self, name, data):
# Redact strings before JSON encoding, so escaped secrets cannot escape filtering.
def clean(value):
if isinstance(value, str):
return self.sanitize(value)
if isinstance(value, dict):
return {key: clean(item) for key, item in value.items()}
if isinstance(value, list):
return [clean(item) for item in value]
return value
try:
self.directory.mkdir(parents=True, exist_ok=True)
path = self.directory / name
path.write_text(json.dumps(clean(data), indent=2) + '\n')
path.chmod(0o600)
except OSError as error:
print(f"smoke diagnostics write failed: {type(error).__name__}", file=sys.stderr)
def run(self, args, root, environment, timeout=15):
try:
result = subprocess.run(args, cwd=root, env=environment, capture_output=True,
text=True, timeout=timeout, check=False)
return {'exit_code': result.returncode,
'output': self.sanitize((result.stdout or '') + (result.stderr or ''))}
except subprocess.TimeoutExpired as error:
return {'exit_code': None, 'error': 'TimeoutExpired',
'output': self.sanitize(self.sanitize(error.stdout) + self.sanitize(error.stderr))}
except OSError as error:
return {'exit_code': None, 'error': type(error).__name__, 'output': ''}
def jar_hashes(self, root):
data = {'dependencies': []}
manifest = root / 'qqq-all-app/target/smoke-classpath.txt'
if manifest.is_file():
for name in sorted(set(manifest.read_text().strip().split(os.pathsep))):
path = Path(name)
match = re.search(r'/com/kingsrook/(qqq|qbits)/([^/]+)/([^/]+)/[^/]+\.jar$', path.as_posix())
if not match:
continue
group, artifact, version = match.groups()
entry = {'coordinate': f'com.kingsrook.{group}:{artifact}:{version}', 'file': path.name}
if path.is_file():
with path.open('rb') as stream:
entry['sha256'] = hashlib.file_digest(stream, 'sha256').hexdigest()
else:
entry['error'] = 'resolved JAR missing'
data['dependencies'].append(entry)
else:
data['error'] = 'resolved classpath missing; run the full-smoke Maven command'
app = root / 'qqq-all-app/target/qqq-all-app.jar'
if app.is_file():
with app.open('rb') as stream:
data['application_sha256'] = hashlib.file_digest(stream, 'sha256').hexdigest()
self.write('jars.json', data)
def pull(self, command, root, environment):
receipt = {'pulls': []}
try:
# Keep the uninterpolated model in memory only; retain no service environment.
model = subprocess.run(command + ['config', '--format', 'json', '--no-interpolate',
'--no-env-resolution'], cwd=root, env=environment,
capture_output=True, text=True, timeout=15, check=False)
if model.returncode:
raise ValueError('Compose image selection failed')
services = json.loads(model.stdout)['services']
if not isinstance(services, dict) or not 1 <= len(services) <= 32:
raise ValueError('unexpected service inventory')
selected = [(name, service['image']) for name, service in sorted(services.items())
if 'build' not in service]
if any(not re.fullmatch(r'[a-zA-Z0-9][a-zA-Z0-9_.-]*', name)
or not isinstance(image, str) or '${' in image for name, image in selected):
raise ValueError('unresolved service image')
except (KeyError, TypeError, ValueError, OSError, subprocess.TimeoutExpired) as error:
receipt['selection_error'] = type(error).__name__
self.write('pulls.json', receipt)
raise AssertionError('could not select Compose service images; see pulls.json') from None
deadline = time.monotonic() + 480
for service, image in selected:
first = image.split('/')[0]
registry = first if '/' in image and ('.' in first or ':' in first or first == 'localhost') else 'docker.io'
remaining = deadline - time.monotonic()
if remaining <= 0:
result = {'exit_code': None, 'error': 'pull-stage budget exhausted', 'output': ''}
else:
result = self.run(command + ['--parallel', '1', '--progress', 'plain', 'pull',
'--ignore-buildable', '--policy', 'missing', service],
root, environment, timeout=min(120, remaining))
receipt['pulls'].append({'service': service, 'image': image,
'registry': registry, 'result': result})
self.write('pulls.json', receipt)
if result['exit_code'] != 0:
raise AssertionError(f'Compose pull failed for {service} ({image}); see pulls.json')
def collect(self, command, root, environment):
deadline = time.monotonic() + 60
def run(args):
remaining = deadline - time.monotonic()
if remaining <= 0:
return {'exit_code': None, 'error': 'collection budget exhausted', 'output': ''}
return self.run(args, root, environment, timeout=min(15, remaining))
inventory = run(command + ['ps', '-a', '-q'])
ids = [line for line in inventory['output'].splitlines()
if re.fullmatch(r'[a-f0-9]{12,64}', line)][:32]
services, images = [], {}
for container in ids:
result = run(['docker', 'inspect', '--format', CONTAINER_FORMAT, container])
try:
raw = json.loads(result['output'])
state = {key: raw.get(key) for key in ('id', 'service', 'image_id', 'status',
'exit_code', 'oom_killed', 'error')}
health = raw.get('health') or {}
state['health'] = {'status': health.get('Status'),
'checks': [{key: check.get(key) for key in ('Start', 'End', 'ExitCode', 'Output')}
for check in health.get('Log', [])[-5:]]}
services.append(state)
image = state['image_id']
if isinstance(image, str) and re.fullmatch(r'sha256:[a-f0-9]{64}', image) and image not in images:
found = run(['docker', 'image', 'inspect', '--format', IMAGE_FORMAT, image])
try:
details = json.loads(found['output'])
images[image] = {key: details.get(key) for key in ('id', 'repo_digests', 'architecture', 'os')}
except (ValueError, TypeError, AttributeError):
images[image] = {'error': 'image identity unavailable', 'command': found}
except (ValueError, TypeError, AttributeError):
services.append({'id': container, 'error': 'container state unavailable', 'command': result})
self.write(f'container-{container[:12]}.json',
run(['docker', 'logs', '--timestamps', '--tail', '100', container]))
self.write('services.json', {'inventory': inventory, 'services': services, 'images': images})
+76
View File
@@ -0,0 +1,76 @@
"""Exercise the delivered archive with Compose, without running any containers."""
import json
import os
import subprocess
import tarfile
import tempfile
import unittest
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
IMAGE = "ghcr.io/qrun-io/qqq-all@sha256:" + "a" * 64
class ReleaseBundleTest(unittest.TestCase):
def test_bundle_uses_digest_and_keeps_full_profile_infra_without_app_build(self):
with tempfile.TemporaryDirectory() as directory:
result = subprocess.run(["python3", str(ROOT / "scripts/release_bundle.py"),
"--root", str(ROOT), "--version", "4.1.0-RC.1",
"--sha", "b" * 40, "--image", IMAGE, "--output", directory],
capture_output=True, text=True)
self.assertEqual(0, result.returncode, result.stderr)
archive = Path(directory) / "qqq-all-4.1.0-RC.1.tar.gz"
extracted = Path(directory) / "extracted"
with tarfile.open(archive) as bundle:
names = bundle.getnames()
self.assertIn("qqq-all-4.1.0-RC.1/infra/minio/Dockerfile", names)
self.assertIn("qqq-all-4.1.0-RC.1/infra/keycloak/realm.json", names)
self.assertIn("qqq-all-4.1.0-RC.1/.env.example", names)
self.assertNotIn("qqq-all-4.1.0-RC.1/.env", names)
self.assertNotIn("qqq-all-4.1.0-RC.1/Dockerfile", names)
bundle.extractall(extracted, filter="data")
root = extracted / "qqq-all-4.1.0-RC.1"
bundled = json.loads((root / "compose.yaml").read_text())
self.assertEqual("/home/${SFTP_USER:-qqq}/upload",
bundled["services"]["sftp"]["volumes"][0]["target"])
environment = {key: value for key, value in os.environ.items() if not key.startswith("COMPOSE_")}
result = subprocess.run(["docker", "compose", "--env-file", ".env.example", "--profile", "*",
"-f", "compose.yaml", "config", "--format", "json"],
cwd=root, env=environment, capture_output=True, text=True)
self.assertEqual(0, result.returncode, result.stderr)
config = json.loads(result.stdout)
original_result = subprocess.run(["docker", "compose", "--env-file", ".env.example", "--profile", "*",
"-f", "compose.yaml", "config", "--format", "json"],
cwd=ROOT, env=environment, capture_output=True, text=True, check=True)
original = json.loads(original_result.stdout.replace(str(ROOT.resolve()), str(root.resolve())))
for name, service in original["services"].items():
if name in ("app-core", "app-full"):
service.pop("build")
service.update(image=IMAGE, pull_policy="always")
if name == "minio":
service["image"] = "qqq-all-minio:4.1.0-RC.1"
self.assertEqual(service, config["services"][name], name)
for service in ("app-core", "app-full"):
self.assertEqual(IMAGE, config["services"][service]["image"])
self.assertNotIn("build", config["services"][service])
self.assertEqual((root / "infra/minio").resolve(), Path(config["services"]["minio"]["build"]["context"]).resolve())
for service in config["services"].values():
for volume in service.get("volumes", []):
if volume["type"] == "bind":
self.assertTrue(Path(volume["source"]).exists(), volume)
manifest = json.loads((root / "release.json").read_text())
self.assertEqual("4.1.0-RC.1", manifest["version"])
self.assertEqual("b" * 40, manifest["source_sha"])
self.assertEqual(IMAGE, manifest["image"])
def test_bundle_rejects_mutable_image_and_unapproved_version(self):
with tempfile.TemporaryDirectory() as directory:
for version, image in (("4.1.0-RC.2", IMAGE), ("4.1.0-RC.1", "ghcr.io/qrun-io/qqq-all:develop"),
("4.1.0", "ghcr.io/other/qqq-all@sha256:" + "a" * 64)):
result = subprocess.run(["python3", str(ROOT / "scripts/release_bundle.py"),
"--root", str(ROOT), "--version", version, "--sha", "b" * 40,
"--image", image, "--output", directory], capture_output=True, text=True)
self.assertNotEqual(0, result.returncode)
self.assertEqual([], list(Path(directory).iterdir()))
+8
View File
@@ -26,6 +26,10 @@ def management(qqq_version="${qqq.version}", qbit_version=None):
f"<artifactId>qbit-{name}</artifactId><version>{qbit_version(name)}</version></dependency>"
for name in QBITS
)
dependencies += (
"<dependency><groupId>com.kingsrook.qqq</groupId><artifactId>qqq-frontend-next</artifactId>"
"<version>1.0.0-RC.8</version></dependency>"
)
return f"<dependencyManagement><dependencies>{dependencies}</dependencies></dependencyManagement>"
@@ -44,6 +48,7 @@ class ReleaseGateTest(unittest.TestCase):
(self.root / "qqq-all-bom/pom.xml").write_text(
"<project><properties>"
f"<qqq.version>{qqq_version}</qqq.version>"
"<qqq-frontend-next.version>1.0.0-RC.8</qqq-frontend-next.version>"
+ "".join(f"<qbit-{name}.version>{qbit_version}</qbit-{name}.version>" for name in QBITS)
+ "</properties>" + management() + "</project>"
)
@@ -59,6 +64,7 @@ class ReleaseGateTest(unittest.TestCase):
"<groupId>com.kingsrook.qqq</groupId>"
"<artifactId>qqq-all-parent</artifactId><version>4.1.0</version></parent>"
"<artifactId>qqq-all-bom</artifactId><properties><qqq.version>4.1.0</qqq.version>"
"<qqq-frontend-next.version>1.0.0-RC.8</qqq-frontend-next.version>"
+ "".join(f"<qbit-{name}.version>1.2.3</qbit-{name}.version>" for name in QBITS)
+ "</properties>" + managed + "</project>")
return parent, bom
@@ -97,6 +103,8 @@ class ReleaseGateTest(unittest.TestCase):
({**run, "head_sha": "b" * 40}, jobs),
({**run, "conclusion": "failure"}, jobs),
({**run, "path": ".github/workflows/other.yml"}, jobs),
({**run, "event": "workflow_dispatch"}, jobs),
({**run, "head_branch": "main"}, jobs),
(run, {"jobs": [jobs["jobs"][0]]}),
(run, {"jobs": [jobs["jobs"][0], {"name": "full", "conclusion": "skipped"}]}),
):
+151
View File
@@ -0,0 +1,151 @@
"""RC publication must retain the GA boundary and fail before any upload."""
import functools
import http.server
import tempfile
import threading
import unittest
from pathlib import Path
import release_gate
from release_gate import GateError
import test_release_gate
RC = "4.1.0-RC.1"
TAG_SHA = "b" * 40
RELEASE_SHA = "a" * 40
class QuietHandler(http.server.SimpleHTTPRequestHandler):
def log_message(self, format, *args):
pass
class ReleaseCandidateTest(unittest.TestCase):
def setUp(self):
self.fixture = test_release_gate.ReleaseGateTest()
self.fixture.setUp()
self.addCleanup(self.fixture.doCleanups)
self.root = self.fixture.root
def test_exact_rc_and_ga_are_allowed_but_other_candidates_are_not(self):
for version in ("4.1.0", RC):
self.fixture.write_poms(version, version, "1.2.3-RC.1")
release_gate.validate_poms(self.root, version)
for version in ("4.1.0-RC.2", "4.1.0-SNAPSHOT", "4.1.1", "LATEST", ""):
with self.subTest(version=version), self.assertRaises(GateError):
self.fixture.write_poms(version, version, "1.2.3")
release_gate.validate_poms(self.root, version)
def test_rc_rejects_wrong_framework_pin_and_mutable_dependencies(self):
for pin in ("4.1.0", "4.1.0-RC.2", "4.1.0-SNAPSHOT"):
self.fixture.write_poms(RC, pin, "1.2.3")
with self.assertRaises(GateError):
release_gate.validate_poms(self.root, RC)
for pin in ("LATEST", "RELEASE", "[1.0,2.0)", "1.2.3-SNAPSHOT", "${unknown}"):
self.fixture.write_poms(RC, RC, pin)
with self.subTest(pin=pin), self.assertRaises(GateError):
release_gate.validate_poms(self.root, RC)
self.fixture.write_poms(RC, RC, "1.2.3")
bom = self.root / "qqq-all-bom/pom.xml"
bom.write_text(bom.read_text().replace("1.0.0-RC.8", "LATEST"))
with self.assertRaises(GateError):
release_gate.validate_poms(self.root, RC)
def test_public_rc_distribution_poms_must_match_source_including_next(self):
self.fixture.write_poms(RC, RC, "1.2.3")
parent, bom = self.fixture.published_poms(resolved=True)
for path in (parent, bom):
path.write_text(path.read_text().replace("4.1.0", RC))
release_gate.validate_published_poms(parent, bom, RC, self.root / "qqq-all-bom/pom.xml")
original = bom.read_text()
for changed in (original.replace("1.0.0-RC.8", "1.0.0-RC.9"),
original.replace(RC, "4.1.0-RC.2"),
original.replace("1.2.3", "[1,2)")):
bom.write_text(changed)
with self.assertRaises(GateError):
release_gate.validate_published_poms(parent, bom, RC, self.root / "qqq-all-bom/pom.xml")
def test_upstream_rc_requires_exact_public_prerelease_and_ga_requires_final(self):
for version, prerelease in ((RC, True), ("4.1.0", False)):
release = {"tag_name": "v" + version, "draft": False, "prerelease": prerelease}
release_gate.validate_upstream_release(release, version)
for changed in ({**release, "draft": True}, {**release, "prerelease": not prerelease},
{**release, "tag_name": "v4.1.0-RC.2"}, {}):
with self.subTest(changed=changed), self.assertRaises(GateError):
release_gate.validate_upstream_release(changed, version)
def test_tag_requires_verified_signature_exact_name_object_and_commit(self):
tag = {"sha": TAG_SHA, "tag": RC, "object": {"type": "commit", "sha": RELEASE_SHA},
"verification": {"verified": True, "reason": "valid"}}
release_gate.validate_tag(tag, RC, RELEASE_SHA, TAG_SHA)
mutations = [
{**tag, "tag": "4.1.0"}, {**tag, "sha": "c" * 40},
{**tag, "object": {"type": "commit", "sha": "c" * 40}},
{**tag, "object": {"type": "tag", "sha": RELEASE_SHA}},
{**tag, "verification": {"verified": False, "reason": "unsigned"}},
{**tag, "verification": {"verified": True, "reason": "unknown_key"}}, {},
]
for changed in mutations:
with self.subTest(changed=changed), self.assertRaises(GateError):
release_gate.validate_tag(changed, RC, RELEASE_SHA, TAG_SHA)
def test_public_download_fails_on_missing_or_wrong_coordinates_and_parent(self):
self.fixture.write_poms(RC, RC, "1.2.3")
with tempfile.TemporaryDirectory() as directory:
central = Path(directory)
dependencies = [("com.kingsrook.qqq", "qqq-bom-pom", RC, "pom"),
("com.kingsrook.qqq", "qqq-frontend-next", "1.0.0-RC.8", "jar")]
dependencies += [("com.kingsrook.qbits", "qbit-" + name, "1.2.3", "jar")
for name in release_gate.QBITS]
for group, artifact, version, kind in dependencies:
folder = central / group.replace(".", "/") / artifact / version
folder.mkdir(parents=True)
stem = folder / (artifact + "-" + version)
Path(str(stem) + ".pom").write_text(
f"<project><groupId>{group}</groupId><artifactId>{artifact}</artifactId>"
f"<version>{version}</version></project>")
if kind == "jar":
Path(str(stem) + ".jar").write_bytes(b"published fixture jar")
handler = functools.partial(QuietHandler, directory=directory)
server = http.server.ThreadingHTTPServer(("127.0.0.1", 0), handler)
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
self.addCleanup(server.server_close)
self.addCleanup(server.shutdown)
base_url = f"http://127.0.0.1:{server.server_port}"
dest = self.root / "public"
bom = self.root / "qqq-all-bom/pom.xml"
receipts = release_gate.download_public_inputs(bom, RC, dest, base_url)
self.assertEqual(19, len(receipts))
self.assertTrue(all(len(row["sha256"]) == 64 for row in receipts))
public_pom = central / "com/kingsrook/qqq/qqq-frontend-next/1.0.0-RC.8/qqq-frontend-next-1.0.0-RC.8.pom"
original = public_pom.read_text()
public_pom.write_text(original.replace("1.0.0-RC.8", "1.0.0-RC.9"))
with self.assertRaises(GateError):
release_gate.download_public_inputs(bom, RC, dest, base_url)
public_pom.write_text(original.replace("</project>", "<parent><groupId>com.kingsrook</groupId>"
"<artifactId>qbit-build-parent</artifactId><version>LATEST</version></parent></project>"))
with self.assertRaises(GateError):
release_gate.download_public_inputs(bom, RC, dest, base_url)
public_pom.write_text(original.replace("</project>", "<parent><groupId>com.kingsrook</groupId>"
"<artifactId>qbit-build-parent</artifactId><version>2.1.0-RC.1</version></parent></project>"))
parent = central / "com/kingsrook/qbit-build-parent/2.1.0-RC.1/qbit-build-parent-2.1.0-RC.1.pom"
parent.parent.mkdir(parents=True)
parent.write_text("<project><groupId>com.kingsrook</groupId><artifactId>qbit-build-parent</artifactId>"
"<version>2.1.0-RC.1</version><dependencyManagement><dependencies><dependency>"
"<groupId>com.kingsrook.qqq</groupId><artifactId>qqq-bom-pom</artifactId>"
"<version>4.0.0</version><type>pom</type><scope>import</scope>"
"</dependency></dependencies></dependencyManagement></project>")
with self.assertRaises(GateError):
release_gate.download_public_inputs(bom, RC, dest, base_url)
parent.write_text(parent.read_text().replace("4.0.0", RC))
self.assertEqual(20, len(release_gate.download_public_inputs(bom, RC, dest, base_url)))
public_pom.unlink()
with self.assertRaises(GateError):
release_gate.download_public_inputs(bom, RC, dest, base_url)
if __name__ == "__main__":
unittest.main()
+18 -5
View File
@@ -148,10 +148,12 @@ class SmokeTest(unittest.TestCase):
pom = ET.parse(ROOT / "qqq-all-app" / "pom.xml")
final_name = pom.findtext(".//{http://maven.apache.org/POM/4.0.0}build/"
"{http://maven.apache.org/POM/4.0.0}finalName")
workflow = (ROOT / ".github" / "workflows" / "smoke.yml").read_text()
command = re.search(r"run: python3 scripts/run_core_smoke.py (\S+)", workflow)
self.assertIsNotNone(command)
self.assertEqual(f"qqq-all-app/target/{final_name}.jar", command.group(1))
# The shared all profile owns the smoke command; this repository owns
# the Maven artifact it consumes. Keep that cross-repository contract.
self.assertEqual("qqq-all-app", final_name)
manifest = (ROOT / ".qqq-ci.yml").read_text()
self.assertRegex(manifest, r"(?m)^profile: all$")
def test_standalone_full_smoke_requires_explicit_search_endpoint(self):
result = subprocess.run(
@@ -192,6 +194,7 @@ class SmokeTest(unittest.TestCase):
with (mock.patch.object(run_full_smoke, "ROOT", root),
mock.patch.object(run_full_smoke, "demo_environment", return_value=values),
mock.patch.object(run_full_smoke, "published_port", return_value=20003),
mock.patch.object(run_full_smoke.Diagnostics, "pull"),
mock.patch.object(run_full_smoke.subprocess, "run",
return_value=subprocess.CompletedProcess([], 0)),
mock.patch.object(run_full_smoke, "check_full", side_effect=verify_credentials) as check,
@@ -233,7 +236,7 @@ class SmokeTest(unittest.TestCase):
run_full_smoke.prepare_sftp_import_directory(["docker", "compose"], {}, "qqq")
def test_oidc_cookie_is_sent_only_to_loopback_http_app(self):
from smoke import allow_loopback_http_session
from smoke import allow_loopback_http_session, require_oidc_session
class SecureSession(DemoHandler):
def do_GET(self):
@@ -257,7 +260,17 @@ class SmokeTest(unittest.TestCase):
thread.start()
try:
client = SmokeClient(f"http://127.0.0.1:{server.server_port}")
with self.assertRaisesRegex(AssertionError, "HttpOnly OIDC session"):
require_oidc_session(client, {})
client.request("/login")
require_oidc_session(client, {})
with self.assertRaisesRegex(AssertionError, "exposed"):
require_oidc_session(client, {"uuid": "test-session"})
cookie = next(iter(client.cookies))
cookie._rest.clear()
with self.assertRaisesRegex(AssertionError, "HttpOnly OIDC session"):
require_oidc_session(client, {})
cookie.set_nonstandard_attr("HttpOnly", None)
with self.assertRaisesRegex(AssertionError, "HTTP 403"):
client.request("/protected")
allow_loopback_http_session(client)
+255
View File
@@ -0,0 +1,255 @@
#!/usr/bin/env python3
"""Failure evidence must survive cleanup without leaking runtime credentials."""
import hashlib
import json
import os
import subprocess
import tempfile
import unittest
from pathlib import Path
from unittest import mock
import run_full_smoke
class DiagnosticsTest(unittest.TestCase):
def test_startup_failure_retains_redacted_evidence_before_cleanup(self):
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
(root / 'compose.yaml').touch()
values = {'QQQ_ALL_PORT': '20001', 'KEYCLOAK_PORT': '20002',
'POSTGRES_PASSWORD': 'private-password-123'}
cleaned = []
def run(args, **kwargs):
if 'config' in args:
return subprocess.CompletedProcess(args, 0, json.dumps({'services': {'db': {'image': 'postgres:16-alpine'}}}), '')
if 'up' in args:
return subprocess.CompletedProcess(args, 1, 'postgres unhealthy private-password-123', '')
if 'down' in args:
receipt = root / 'target/smoke-diagnostics/startup.json'
self.assertTrue(receipt.is_file(), 'diagnostics must exist before teardown')
cleaned.append(True)
return subprocess.CompletedProcess(args, 0, '', '')
with (mock.patch.object(run_full_smoke, 'ROOT', root),
mock.patch.object(run_full_smoke, 'demo_environment', return_value=values),
mock.patch.object(run_full_smoke.subprocess, 'run', side_effect=run),
mock.patch.dict(os.environ)):
with self.assertRaisesRegex(AssertionError, 'did not become healthy'):
run_full_smoke.main()
self.assertTrue(cleaned)
output = (root / 'target/smoke-diagnostics/startup.json').read_text()
self.assertIn('postgres unhealthy', output)
self.assertNotIn('private-password-123', output)
self.assertEqual(1, json.loads(output)['exit_code'])
def test_http_failure_stays_sanitized_when_collection_and_cleanup_fail(self):
self.diagnostics_module()
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
(root / 'compose.yaml').touch()
values = {'QQQ_ALL_PORT': '20001', 'KEYCLOAK_PORT': '20002',
'DB_PASSWORD': 'hidden-fixture'}
def run(args, **kwargs):
if 'config' in args:
return subprocess.CompletedProcess(args, 0, json.dumps({'services': {'db': {'image': 'postgres:16-alpine'}}}), '')
if 'down' in args:
raise subprocess.TimeoutExpired(args, 120, output='hidden-fixture')
return subprocess.CompletedProcess(args, 0, '', '')
with (mock.patch.object(run_full_smoke, 'ROOT', root),
mock.patch.object(run_full_smoke, 'demo_environment', return_value=values),
mock.patch.object(run_full_smoke.subprocess, 'run', side_effect=run),
mock.patch.object(run_full_smoke.Diagnostics, 'collect', side_effect=ValueError('hidden-fixture')),
mock.patch.object(run_full_smoke, 'check_full', side_effect=AssertionError('HTTP failed hidden-fixture')),
mock.patch.dict(os.environ)):
with self.assertRaisesRegex(AssertionError, r'HTTP failed \[REDACTED\]'):
run_full_smoke.main()
text = ''.join(f.read_text() for f in (root / 'target/smoke-diagnostics').glob('*.json'))
self.assertNotIn('hidden-fixture', text)
self.assertIn('TimeoutExpired', text)
def test_startup_timeout_keeps_partial_sanitized_output_and_tears_down(self):
self.diagnostics_module()
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
(root / 'compose.yaml').touch()
values = {'QQQ_ALL_PORT': '20001', 'KEYCLOAK_PORT': '20002',
'DB_PASSWORD': 'hidden-fixture'}
cleanup = []
def run(args, **kwargs):
if 'config' in args:
return subprocess.CompletedProcess(args, 0, json.dumps({'services': {'db': {'image': 'postgres:16-alpine'}}}), '')
if 'up' in args:
raise subprocess.TimeoutExpired(args, 480, output=b'postgres unhealthy hidden-fixture')
if 'down' in args:
cleanup.append(True)
return subprocess.CompletedProcess(args, 0, '', '')
with (mock.patch.object(run_full_smoke, 'ROOT', root),
mock.patch.object(run_full_smoke, 'demo_environment', return_value=values),
mock.patch.object(run_full_smoke.subprocess, 'run', side_effect=run),
mock.patch.dict(os.environ)):
with self.assertRaisesRegex(AssertionError, 'did not become healthy'):
run_full_smoke.main()
self.assertTrue(cleanup)
receipt = json.loads((root / 'target/smoke-diagnostics/startup.json').read_text())
self.assertEqual('TimeoutExpired', receipt['error'])
self.assertIn('postgres unhealthy', receipt['output'])
self.assertNotIn('hidden-fixture', receipt['output'])
def test_pull_failure_names_service_image_registry_and_stops_before_up(self):
module = self.diagnostics_module()
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
(root / 'compose.yaml').touch()
values = {'QQQ_ALL_PORT': '20001', 'KEYCLOAK_PORT': '20002',
'DB_PASSWORD': 'pull-secret'}
calls = []
model = {'services': {
'app-full': {'image': 'qqq-all:local', 'build': {'context': '.'}},
'minio-init': {'image': 'public.ecr.aws/aws-cli/aws-cli:2.27.41',
'environment': {'DO_NOT_RETAIN': 'raw-config-fixture'}},
'sftp': {'image': 'atmoz/sftp:alpine'}}}
def run(args, **kwargs):
calls.append(args)
if 'config' in args:
self.assertIn('--no-interpolate', args)
self.assertIn('--no-env-resolution', args)
return subprocess.CompletedProcess(args, 0, json.dumps(model), '')
if 'pull' in args:
self.assertEqual('minio-init', args[-1])
self.assertNotIn('--ignore-pull-failures', args)
self.assertNotIn('--include-deps', args)
return subprocess.CompletedProcess(args, 1, '', 'toomanyrequests: Rate exceeded pull-secret')
return subprocess.CompletedProcess(args, 0, '', '')
with (mock.patch.object(run_full_smoke, 'ROOT', root),
mock.patch.object(run_full_smoke, 'demo_environment', return_value=values),
mock.patch.object(run_full_smoke.subprocess, 'run', side_effect=run),
mock.patch.object(run_full_smoke, 'check_full'),
mock.patch.dict(os.environ)):
with self.assertRaisesRegex(AssertionError, 'minio-init'):
run_full_smoke.main()
self.assertFalse(any('up' in args for args in calls))
receipt = json.loads((root / 'target/smoke-diagnostics/pulls.json').read_text())
failed = receipt['pulls'][-1]
self.assertEqual('public.ecr.aws', failed['registry'])
self.assertEqual(model['services']['minio-init']['image'], failed['image'])
self.assertEqual(1, failed['result']['exit_code'])
self.assertIn('Rate exceeded', failed['result']['output'])
text = json.dumps(receipt)
self.assertNotIn('pull-secret', text)
self.assertNotIn('raw-config-fixture', text)
def test_successful_pulls_use_same_images_without_building_or_starting(self):
module = self.diagnostics_module()
with tempfile.TemporaryDirectory() as directory:
evidence = module.Diagnostics(Path(directory), {})
self.assertTrue(hasattr(evidence, 'pull'), 'explicit pull diagnostics not implemented')
calls = []
model = {'services': {'postgres': {'image': 'postgres:16-alpine'},
'minio': {'image': 'qqq-all-minio:local', 'build': {'context': './infra/minio'}},
'mongo': {'image': 'mongo:7'}}}
def run(args, **kwargs):
calls.append(args)
output = json.dumps(model) if 'config' in args else 'pulled'
return subprocess.CompletedProcess(args, 0, output, '')
with mock.patch.object(module.subprocess, 'run', side_effect=run):
evidence.pull(['docker', 'compose', '-p', 'private'], Path(directory), {})
pulls = [args for args in calls if 'pull' in args]
self.assertEqual(['mongo', 'postgres'], [args[-1] for args in pulls])
for args in pulls:
self.assertEqual('1', args[args.index('--parallel') + 1])
self.assertEqual('missing', args[args.index('--policy') + 1])
self.assertFalse(any('up' in args or 'build' in args for args in calls))
receipt = json.loads((Path(directory) / 'pulls.json').read_text())
self.assertEqual(['mongo:7', 'postgres:16-alpine'], [r['image'] for r in receipt['pulls']])
self.assertEqual(['docker.io', 'docker.io'], [r['registry'] for r in receipt['pulls']])
def diagnostics_module(self):
try:
import smoke_diagnostics
except ImportError:
self.fail('sanitized diagnostics collector is not implemented')
return smoke_diagnostics
def test_redacts_secret_forms_and_bounds_output_after_redaction(self):
module = self.diagnostics_module()
with tempfile.TemporaryDirectory() as directory:
evidence = module.Diagnostics(Path(directory), {'DB_PASSWORD': 'a+/secret',
'GITHUB_TOKEN': 'token-1234'})
output = evidence.sanitize('x' * 90000 + '\npassword="unknown-value" '
'postgres://u:uri-secret@db/ Bearer bearer-secret '
'a%2B%2Fsecret a+/secret token-1234 access_token=generated-token client_secret=generated-secret')
for secret in ['unknown-value', 'uri-secret', 'bearer-secret',
'a%2B%2Fsecret', 'a+/secret', 'token-1234', 'generated-token', 'generated-secret']:
self.assertFalse(secret in output, "credential remained in sanitized output")
self.assertLessEqual(len(output), 65536)
self.assertIn('[REDACTED]', output)
def test_hashes_only_resolved_first_party_jars_not_other_cached_versions(self):
module = self.diagnostics_module()
with tempfile.TemporaryDirectory() as directory:
root = Path(directory)
jar = root / 'repository/com/kingsrook/qqq/qqq-esb/4.1.0-SNAPSHOT/qqq-esb.jar'
jar.parent.mkdir(parents=True)
jar.write_bytes(b'actual resolved bytes')
(jar.parent / 'unused.jar').write_bytes(b'unused')
manifest = root / 'qqq-all-app/target/smoke-classpath.txt'
manifest.parent.mkdir(parents=True)
manifest.write_text(str(jar))
evidence = module.Diagnostics(root / 'target/smoke-diagnostics', {})
evidence.jar_hashes(root)
data = json.loads((evidence.directory / 'jars.json').read_text())
self.assertEqual([{'coordinate': 'com.kingsrook.qqq:qqq-esb:4.1.0-SNAPSHOT',
'file': 'qqq-esb.jar',
'sha256': hashlib.sha256(b'actual resolved bytes').hexdigest()}], data['dependencies'])
self.assertNotIn(str(root), json.dumps(data))
def test_container_collection_is_selected_redacted_and_survives_command_failure(self):
module = self.diagnostics_module()
with tempfile.TemporaryDirectory() as directory:
evidence = module.Diagnostics(Path(directory), {'DB_PASSWORD': 'secret-fixture'})
container = 'a' * 64
completed = 'd' * 64
image = 'sha256:' + 'b' * 64
commands = []
def run(args, **kwargs):
commands.append(args)
if 'ps' in args:
return subprocess.CompletedProcess(args, 0, container + '\n' + completed + '\n', '')
if 'inspect' in args and 'image' not in args:
self.assertIn('--format', args)
if args[-1] == completed:
return subprocess.CompletedProcess(args, 0, json.dumps({
'id': completed, 'service': 'mongo-init', 'image_id': image,
'status': 'exited', 'exit_code': 0, 'oom_killed': False,
'error': '', 'health': None}), '')
return subprocess.CompletedProcess(args, 0, json.dumps({
'id': container, 'service': 'postgres', 'image_id': image,
'status': 'exited', 'exit_code': 1, 'oom_killed': False,
'error': '', 'health': {'Status': 'unhealthy', 'Log': [
{'ExitCode': 1, 'Output': 'failed secret-fixture'}]}}), '')
if 'image' in args:
return subprocess.CompletedProcess(args, 0, json.dumps({'id': image,
'repo_digests': ['postgres@sha256:' + 'c' * 64],
'architecture': 'amd64', 'os': 'linux'}), '')
if 'logs' in args:
raise subprocess.TimeoutExpired(args, 15, output='secret-fixture')
self.fail('unexpected command')
with mock.patch.object(module.subprocess, 'run', side_effect=run):
evidence.collect(['docker', 'compose', '-p', 'private'], Path(directory), {})
text = '\n'.join(f.read_text() for f in Path(directory).glob('*.json'))
self.assertNotIn('secret-fixture', text)
self.assertIn('unhealthy', text)
self.assertIn('amd64', text)
states = json.loads((Path(directory) / 'services.json').read_text())['services']
self.assertEqual('mongo-init', states[1]['service'])
self.assertEqual(0, states[1]['exit_code'])
self.assertIsNone(states[1]['health']['status'])
self.assertIn('TimeoutExpired', text)
self.assertFalse(any('config' in c or '--volumes' in c for c in commands))
if __name__ == '__main__':
unittest.main()