Compare commits

...
Author SHA1 Message Date
james.maesandCodex 77cdf931c2 fix(ci): carry refreshed snapshot dependencies with build artifacts
Overlay mutable external SNAPSHOT dependencies as well as locally installed reactor outputs so downstream jobs do not read older snapshots from the immutable cache.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: James Maes <james@kof22.com>
2026-10-19 11:02:15 -05:00
james.maesandCodex d7c278b344 fix(ci): preserve required Java PR check name
Give the differently parameterized production build a distinct name so CircleCI retains build-and-test for pull requests.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: James Maes <james@kof22.com>
2026-10-19 10:46:38 -05:00
james.maesandCodex c8fd974bfd perf(ci): bound Maven caches and remove duplicate setup
Keep restore and save keys stable across release version rewrites, avoid inherited cross-version caches, skip unused OWASP setup, upload test results once and pack the orb once before gated publication.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: James Maes <james@kof22.com>
2026-10-19 10:40:43 -05:00
james.maes 6cd50b0a90 Merge pull request #44 from KofTwentyTwo/release/0.3.8-backmerge
chore(release): back-merge Munitor 0.3.8
2026-10-19 09:37:14 -05:00
james.maes 5d4a9ea420 Merge pull request #43 from KofTwentyTwo/release/0.3.8
chore(release): publish Munitor 0.3.8
2026-10-19 09:28:24 -05:00
james.maesandCodex f402fb8347 chore(release): prepare Munitor 0.3.8
Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: James Maes <james@kof22.com>
2026-10-19 09:21:59 -05:00
james.maes 11d5704b47 Merge pull request #42 from KofTwentyTwo/feature/fresh-supplemental-images
fix(build): refresh supplemental image package layers
2026-10-19 09:21:26 -05:00
james.maesandCodex 576de90e03 fix(build): refresh supplemental image package layers
Force fresh base-image metadata and package upgrades for supplemental images while retaining enforced vulnerability scans before publication.

Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: James Maes <james@kof22.com>
2026-10-19 09:13:40 -05:00
james.maes 9678be0cff Merge pull request #41 from KofTwentyTwo/release/0.3.7-backmerge
chore(release): backmerge Munitor 0.3.7
2026-10-19 08:29:57 -05:00
james.maes baca55bd77 Merge pull request #40 from KofTwentyTwo/release/0.3.7
chore(release): publish Munitor 0.3.7
2026-10-19 08:22:15 -05:00
james.maesandCodex 01176862d4 chore(release): prepare Munitor 0.3.7
Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: James Maes <james@kof22.com>
2026-10-19 08:04:03 -05:00
james.maes f7487b356a Merge pull request #39 from KofTwentyTwo/feature/fresh-java-runtime
fix: refresh generated JVM runtime packages on every build
2026-10-19 08:02:29 -05:00
james.maesandCodex 21517b25e7 fix: refresh generated JVM runtime packages on every build
Co-Authored-By: Codex <noreply@openai.com>
Signed-off-by: James Maes <james@kof22.com>
2026-10-19 07:50:52 -05:00
james.maes 34ddd25ac9 Merge pull request #38 from KofTwentyTwo/feature/scan-gates-enforce
fix(scan): make security-scan and sast-scan fail on findings
2026-10-19 07:48:49 -05:00
james.maesandClaude Opus 5.5 4aeccb801e fix(scan): make the security-scan and sast-scan gates fail on findings
- run_trivy.sh: `trivy fs` ran without --exit-code, so it always exited 0
  and security-scan passed with HIGH/CRITICAL findings in the report. Add
  --exit-code 1 and print the findings table on failure.
- Semgrep 1.61.1 cannot parse the current `auto` ruleset, so sast-scan ran
  zero rules on zero files and passed. Move the executor to
  semgrep/semgrep:1.181.0, and fail the job when the report shows no files
  scanned or a rule configuration error (even in warn-only mode). Print a
  findings summary in the job output.
- tests/test_scan_gates.sh covers both gates with tool stubs.

Consumers pick this up when they bump their pinned orb version; repos with
existing HIGH/CRITICAL or Semgrep findings will need to triage them then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:42:51 -05:00
james.maes a3d680f130 Merge pull request #37 from KofTwentyTwo/main
chore: back-merge main into develop after 0.3.6
2026-10-13 17:24:24 -05:00
james.maes 57dffdd9a3 Merge pull request #36 from KofTwentyTwo/release/0.3.6
Release Munitor 0.3.6
2026-10-13 17:17:42 -05:00
james.maesandClaude Opus 5.5 9e2c45e79a docs(readme): adoption examples use 0.3.6
Signed-off-by: James Maes <james@kof22.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 17:11:51 -05:00
james.maes fbeec65cb5 Merge pull request #35 from KofTwentyTwo/feature/supplemental-push-boolean
fix: push supplemental images on release builds again (0.3.5 regression)
2026-10-13 17:11:47 -05:00
james.maesandClaude Opus 5.5 69898922be fix: push supplemental images on release builds again
CircleCI renders the boolean push parameter as 1/0 in the step
environment, but build_push_supplemental.sh only pushed when
PUSH_IMAGES was the string "true". Since 0.3.5 every release build
built and scanned supplemental images and never pushed them, so
deployments referencing them hit ImagePullBackOff (kof22 0.1.75).

PUSH_IMAGES now accepts true/1/yes/on and false/0/no/off/empty; unset
still means push, and an unrecognised value fails before any build.
Tests cover release-mode pushes (1, true, TRUE) of every supplemental
image, PR-mode non-pushes (0, false, empty), and the invalid value.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: James Maes <james@kof22.com>
2026-10-13 17:05:34 -05:00
james.maes 59eb5294fe Merge pull request #34 from KofTwentyTwo/main
chore: back-merge main into develop after 0.3.5
2026-10-13 15:44:21 -05:00
james.maes 16079c5ca8 Merge pull request #33 from KofTwentyTwo/release/0.3.5
Release Munitor 0.3.5
2026-10-13 15:37:54 -05:00
james.maesandClaude Opus 5.5 578c7ea0eb docs(readme): adoption examples use 0.3.5
Signed-off-by: James Maes <james@kof22.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 15:31:55 -05:00
james.maes 53c04937b0 Merge pull request #32 from KofTwentyTwo/feature/pr-image-scan
feat: scan container images in PR pipelines
2026-10-13 14:42:28 -05:00
james.maesandClaude Opus 5.5 b09ced056f feat: scan container images in PR pipelines
New munitor/docker_scan job (image-scan) in the pr-checks workflow of
java-webapp, gradle-webapp, node-api and node-webapp: builds the app
image and every supplemental image, runs the same Trivy 0.69.3
HIGH/CRITICAL --exit-code 1 scan as the release pipeline (same
.trivyignore), stores the JSON results, and never pushes.
build_push_supplemental gains push (default true, unchanged behavior).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 14:18:41 -05:00
james.maes fb232fb6fe Merge pull request #31 from KofTwentyTwo/main
chore: back-merge main into develop after 0.3.4
2026-10-12 06:24:42 -05:00
james.maes 687944e872 Merge pull request #30 from KofTwentyTwo/release/0.3.4
Release Munitor 0.3.4
2026-10-12 06:19:46 -05:00
39 changed files with 1140 additions and 202 deletions

No files matched your search

+11 -11
View File
@@ -52,27 +52,21 @@ workflows:
- test:
filters: *filters
test-deploy:
jobs:
- orb-tools/pack:
filters: *tag-only
- orb-tools/publish:
orb_name: kof22/munitor
vcs_type: << pipeline.project.type >>
pub_type: production
requires:
- orb-tools/lint
- orb-tools/pack
- orb-tools/review
- test
context:
- orb-publishing
filters: *tag-only
# develop owns the shared dev:snapshot tag. Feature branches publish only an
# immutable dev:<sha> tag so open branches cannot overwrite each other's
# snapshot (issue #26).
snapshot:
jobs:
- orb-tools/pack:
filters: *filters
# All publishers consume the same packed workspace after all checks pass.
# Only develop updates dev:snapshot; feature branches publish dev:<sha>.
- orb-tools/publish:
name: publish-develop-snapshot
orb_name: kof22/munitor
@@ -80,7 +74,10 @@ workflows:
pub_type: dev
dev_tags: "dev:${CIRCLE_SHA1},dev:snapshot"
requires:
- orb-tools/lint
- orb-tools/pack
- orb-tools/review
- test
context:
- orb-publishing
filters:
@@ -94,7 +91,10 @@ workflows:
pub_type: dev
dev_tags: "dev:${CIRCLE_SHA1}"
requires:
- orb-tools/lint
- orb-tools/pack
- orb-tools/review
- test
context:
- orb-publishing
filters:
+1
View File
@@ -332,6 +332,7 @@ These components are shared across all modules -- you don't need to recreate the
| `secrets_scan` job | Gitleaks (all modules) |
| `sast_scan` job | Semgrep (all modules, opt-in) |
| `docker_build_push` job | Build + health check + Trivy scan + GHCR push |
| `docker_scan` job | PR image build + Trivy scan (including supplemental images), JSON artifacts, no publishing |
| `health_check` command | Container health check smoke test (all Docker modules) |
| `update_cd_repo` job | GitOps CD repo update |
| `github_release` job | Annotated git tags + GitHub Releases (auto-enabled with `contexts.github`) |
+1 -1
View File
@@ -30,7 +30,7 @@ lint:
test-scripts:
@echo "Running script tests..."
@for f in tests/test_*.sh; do \
if [ -f "$$f" ]; then \
if [ -f "$$f" ] && [ "$$f" != "tests/test_generate_config.sh" ]; then \
echo " Running $$f"; \
bash "$$f" || exit 1; \
fi; \
+15 -6
View File
@@ -6,7 +6,7 @@ Latin: *munitor* (fortifier, protector) -- KofTwentyTwo's unified CircleCI orb f
| --- | --- |
| **Tier** | Internal until classified -- listed under *Awaiting classification* in the KofTwentyTwo standards [MAINTAINERS.md](https://github.com/KofTwentyTwo/standards/blob/main/MAINTAINERS.md); Product-tier requirements are being adopted |
| **Standards** | Targets [KofTwentyTwo standards](https://github.com/KofTwentyTwo/standards) 1.0.0 |
| **Orb** | [`kof22/munitor`](https://circleci.com/developer/orbs/orb/kof22/munitor), latest stable `0.3.3` |
| **Orb** | [`kof22/munitor`](https://circleci.com/developer/orbs/orb/kof22/munitor), latest stable `0.3.8` |
| **Security** | [SECURITY.md](SECURITY.md) |
| **Contributing** | [CONTRIBUTING.md](CONTRIBUTING.md) |
@@ -70,16 +70,16 @@ Munitor validates branch names and will fail on non-conforming names.
Munitor is a public CircleCI orb: any CircleCI organization can use it without forking. This checklist covers what a new consumer needs, for example a Java/Maven web service. A QQQ application should also read the [QQQ adoption checklist](https://github.com/QRun-IO/qqq/blob/develop/docs/migration/adoption-checklist.adoc) (section 7 covers release tooling).
**Version.** Use `kof22/munitor@0.3.3` (or `@0.3` to float on the 0.3 line) in `.circleci/config.yml`, and the same value as `orb_version` in `.munitor.yml`. Keep the setup orb and the generated pipeline on the same line.
**Version.** Use `kof22/munitor@0.3.8` (or `@0.3` to float on the 0.3 line) in `.circleci/config.yml`, and the same value as `orb_version` in `.munitor.yml`. Keep the setup orb and the generated pipeline on the same line.
### 1. Repository files
- `.circleci/config.yml`: the setup config from [Quick Start](#quick-start), pinned to `kof22/munitor@0.3.3`.
- `.circleci/config.yml`: the setup config from [Quick Start](#quick-start), pinned to `kof22/munitor@0.3.8`.
- `.munitor.yml`: pick a [pipeline type](#pipeline-types). A Java service is `java-webapp`. KofTwentyTwo's Website-Backend is a working example; a generalized version:
```yaml
pipeline: java-webapp
orb_version: "0.3.3"
orb_version: "0.3.8"
image_name: my-service
java_version: "21"
docker:
@@ -372,7 +372,7 @@ contexts:
**What runs:** Maven build/test, Checkstyle, SpotBugs, PMD, JaCoCo coverage, OWASP + Trivy security, Docker build/push (Dockerfile auto-generated), gitleaks. OWASP dependency check requires the `nvd` context; if omitted, the security scan runs without NVD database enrichment.
**Dockerfile:** Auto-generated production-ready image using Eclipse Temurin JRE Alpine. Includes non-root user (uid 1001), JVM container tuning flags, Docker HEALTHCHECK, and configurable port via `health.port`. The pre-built JAR from CI is copied directly (no redundant Maven build stage).
**Dockerfile:** Auto-generated production-ready image using Eclipse Temurin JRE Alpine. Includes non-root user (uid 1001), JVM container tuning flags, Docker HEALTHCHECK, and configurable port via `health.port`. The pre-built JAR from CI is copied directly (no redundant Maven build stage). PR and release image builds pull the current base and bypass Docker layer caching for generated Java/Gradle runtime images, ensuring the Alpine package upgrade runs on every build. Node builds retain their layer cache. Trivy gates remain required.
### `terraform`
@@ -490,7 +490,7 @@ Each workflow runs on specific branches and includes jobs based on your config:
| Workflow | Branches | Jobs |
|----------|----------|------|
| `pr-checks` | `feature/*`, `hotfix/*` | Build, test, lint, coverage, security scans |
| `pr-checks` | `feature/*`, `hotfix/*` | Build, test, lint, coverage, security scans, container image scan (Docker modules) |
| `develop` | `develop` | All pr-checks + Docker build/push + CD update (dev env) |
| `staging` | `staging` | All pr-checks + Docker build/push + CD update (staging env) |
| `release-candidate` | `release/*` | All pr-checks + Docker, CD (staging), SonarCloud, SBOM, GitHub Release (pre-release) |
@@ -498,11 +498,20 @@ Each workflow runs on specific branches and includes jobs based on your config:
For `validate-cd-repo` and `argocd-apps`, all three workflows (`pr-checks`, `develop`, `release`) run the same 5-job set: yaml-lint, kustomize-validate, kubesec-scan, kube-linter, secrets-scan.
For `java-webapp`, `gradle-webapp`, `node-api`, and `node-webapp`, PRs run
`image-scan` after `build-and-test`: build the application and any supplemental
images locally, then fail on HIGH/CRITICAL CVEs with the same Trivy policy and
repository-root `.trivyignore` as release builds. JSON reports are saved as
artifacts. This job does not log in, publish images, reserve tags, or update CD.
Release candidates already scan through `docker-build-push` and do not run a
second image scan job.
### Job Dependency Graph (node-api pr-checks)
```
checkout
--> build-and-test
--> image-scan (local build + Trivy, including supplemental images)
--> code-quality
--> coverage
--> security-scan
+41
View File
@@ -0,0 +1,41 @@
# Fast Java CI
Backend ticket work is paused while this performance change is delivered.
## Measured baseline
Website-Backend PR311, CircleCI jobs6314–6318 on2026-10-19:
- Build/test job:30m15s. Compilation49s; tests14m24s; Java installation4m27s.
- Maven cache48GiB: restore5m32s, save4m19s.
- Quality, coverage and security each restore the same48GiB again (5–6min).
- Coverage computation itself takes5.4s. Quality tools take57s combined.
- OWASP is disabled in this consumer, but security still installs Java and restores Maven.
- Test results are uploaded by both the test command and its enclosing job:777 results appear twice.
Evidence: coordinator cache/cicd-baseline.json, cicd-test-timings.json and cicd-github-friction.md.
## First implementation
1. Replace the old broad Maven fallback with an exact v2 key based on checked-out dependency definitions. Keep that key stable across release-version rewrites and invalidate for nested POM or Maven configuration changes. Never restore the48GiB v1 cache.
2. Skip Java/Maven/NVD setup when OWASP is disabled; preserve Trivy.
3. Upload Maven test results once.
4. Pack the orb once per pipeline; publish only after its existing checks pass, preserving branch/tag behavior and required check names.
5. Run template tests only in their dedicated Make target, rather than also discovering them in the generic script loop.
6. Run Checkstyle, SpotBugs, PMD and coverage on the existing build worker for generated PR/develop/staging/release workflows. Keep the standalone jobs available for custom consumers and preserve production behavior. Generated downstream jobs depend on the completed build gate.
7. Render each immutable test fixture once per test run; retain all277 assertions.
Maven caches are dependency reuse, not artifact handoff. A real three-module experiment failed SpotBugs with current target outputs and an older installed-module cache. Preserve locally installed version directories, mutable SNAPSHOT dependencies and ancestor metadata in a workspace archive, then overlay it after dependency-cache restore. The experiment passes for release versions, snapshots and classifier dependencies. Maven Resolver identifies local installs with empty repository IDs in `_remote.repositories`; this implementation depends on that Maven3 format. Do not replace this archive with the entire local repository.
Cold dependency downloads on changed POMs are deliberate: no cross-version fallback accumulation. Do not delete caches, artifacts, tags or change branch protections.
## Verification and rollout
- Focused behavioral/config regressions, then required make all and BATS suites once the combined patch is complete.
- Publish immutable feature orb; run a backend feature canary with both pins matching that SHA.
- Compare real cache size and build/check durations against the baseline. No claimed speedup before measurement; do not merge the consumer until tests and all existing security gates pass.
- Release the proven orb and adopt its stable version through normal PRs. Track the already-running backend release separately; do not resume paused ticket work.
## Follow-up decisions after measurement
If tests dominate after cache repair, inspect measured slow fixtures and isolate/parallelize only tests proven independent. Exact-source artifact promotion is a separate future change requiring verified source identity; do not skip merged-source validation on the strength of an earlier PR run. Avoid a new orchestration framework or speculative configuration options.
+59
View File
@@ -1,5 +1,53 @@
## Fast Java CI (2026-10-19)
Active user priority: fast, agile, lightweight CI/CD; backend ticket goal paused.
Measured48GiB Maven cache restored four times and saved once per backend PR.
Work on feature/lean-java-ci: stable exact dependency cache keys with new namespace,
conditional OWASP setup, single test-result upload, and one orb pack per pipeline.
The first patch passed make all and15BATS. Further simplification combines quality
and coverage with the build, renders fixtures once, and carries installed reactor
artifacts in a small workspace archive. A real multi-module experiment proved this
artifact handoff necessary, including snapshot/classifier coverage. Expanded make
all and20BATS pass, including all277template assertions. Real consumer timing and
stable publication/adoption remain pending. No deployed speedup claimed.
See docs/PLAN-lean-java-ci.md.
## Supplemental runtime freshness (2026-10-19)
Backend PR311 image scan6307 failed because db-backup reused a cached apk-upgrade layer (libexpat2.8.5-r0, fixed2.9.0-r0). Supplemental builds now use --pull --no-cache in both PR scan-only and release publishing paths. Trivy and immutable-tag guards remain unchanged. Targeted tests cover both Dockerfile forms, fresh-build flags, scan-before-push ordering and scan failure propagation. Validation pending; no release or consumer adoption yet.
# Munitor session state
## JVM runtime package freshness (2026-10-19)
Website-Backend release6258 and PR scans6266/6277 reused a cached apk upgrade
layer containing vulnerable libexpat. The shared docker_build command now pulls
the base and bypasses layer caching for generated Java/Gradle runtimes, while
Node caching and all Trivy gates remain intact. Narrow behavioral tests cover
both pipeline types, alternate/missing manifests, tags, provenance, and both
PR/release job routes. All eight focused shell cases passed via the coordinating
agent; ShellCheck and git diff --check passed. Full make/BATS/orb validation and
real-image scanning remain pending. Stable release and consumer adoption remain
pending; no unchanged vulnerable image should be retried.
## PR container scanning (2026-10-13)
Uncommitted work on `feature/pr-image-scan` adds `docker_scan` after
`build-and-test` in the four application PR templates. It restores the existing
feature/hotfix version metadata, builds locally, and applies the release Trivy
HIGH/CRITICAL policy from the repository root. Supplemental images use
`push: false`; the command defaults to publishing for existing callers. Main
and supplemental JSON reports are artifacts. Release candidates already scan
through `docker_build_push`, so no duplicate job was added.
Verified: native CircleCI orb pack/validate, ShellCheck on all source scripts
and the changed tests, template structure/filters/parameter wiring, packed
template synchronization, and `git diff --check`. Requested make and BATS runs
were attempted but Git Bash cannot start in this sandbox (CreateFileMapping,
Win32 error 5); yamllint is not installed. New behavioral tests therefore need
a Linux/CI run, including real Docker/Trivy integration. No commit or publish.
The Second Brain vault is outside writable roots; continuity is recorded here.
## Release 0.3.3 (2026-10-11)
PR22 (00f349c) fixes release version collisions. Website-Backend #65/#66 (0.1.24)
@@ -66,3 +114,14 @@ after Java installation. APT now receives noninteractive debconf and automatic
needrestart settings; a stubbed integration test verifies both APT invocations.
These fixes require normal feature-to-develop review and stable tag publication
from main before website consumers can select the new version. Nothing deployed.
Runtime freshness validation: coordinating agent completed make all (lint, pack, orb validation, scripts,277template assertions) and all7BATS tests in Ubuntu. Real consumer image scan remains required after release.
## Release 0.3.7 preparation (2026-10-19)
Includes merged PR38 (filesystem Trivy findings fail CI; current Semgrep image and zero-scan/configuration failures) and PR39 (pull fresh runtime bases and bypass cached JVM package-upgrade layers). Both feature PRs passed all CircleCI gates; PR39 also passed full local make all and seven BATS tests. Publish the stable orb only from a signed v0.3.7 tag on the reviewed main merge. Website-Backend must update both orb pins and pass real image/SAST scans before rollout. No vulnerability waiver or immutable-tag overwrite is part of this release.
Validation completed: focused scan tests12/12, full make all (lint, packed orb validation, script/template tests), and7BATS passed under WSL/Nix on2026-10-19. Real backend supplemental image scans remain required after stable release adoption.
## Release candidate 0.3.8 (2026-10-19)
Promotes PR42 supplemental build freshness fix (develop11d5704) to main. Full local make all and seven BATS passed; all PR42 CI checks passed. Release metadata changes only; source/tests match the verified feature. Stable publication still requires the approved main merge and signed v0.3.8 tag. Backend adoption and real supplemental scans remain pending.
+9
View File
@@ -2,6 +2,15 @@
## Pending
- [ ] Validate and publish fast Java CI patch, canary actual cache size/timings, then adopt stable orb (docs/PLAN-lean-java-ci.md).
- [ ] Validate and release supplemental-image freshness fix, then adopt it in backend PR311 and verify all supplemental Trivy scans.
- [ ] Release JVM runtime cache fix and adopt the stable orb in Website-Backend; verify fresh image Trivy before release.
- [ ] Run PR image-scan script/template/BATS tests on Linux and verify a consumer PR with Docker/Trivy before publishing.
- [x] Add PR-only image scans for the four application templates, supplemental scan-only mode, JSON artifacts, and regression tests (uncommitted).
- [x] Review publishing CI fixes and merge PR19 to develop.
- [x] Publish v0.3.2 from main.
- [ ] Publish v0.3.3 (atomic release versions); bump Website-Backend/Frontend pins to 0.3.3.
+11 -3
View File
@@ -1,8 +1,12 @@
description: >
Build, scan, and push supplemental Docker images alongside the main application image.
Each supplemental image shares the same version tag. Requires docker_push_ghcr to have
run first (for registry auth and Trivy installation).
Each supplemental image shares the same version tag. Requires run_trivy_image first
for Trivy installation, and docker_push_ghcr for registry auth only when push is true.
parameters:
push:
type: boolean
default: true
description: Publish scanned images to the registry. Disable for PR scans.
supplemental_images:
type: string
default: "[]"
@@ -16,9 +20,13 @@ parameters:
description: Container registry URL.
steps:
- run:
name: Build and push supplemental images
name: Build and scan supplemental images (optionally push)
environment:
PUSH_IMAGES: << parameters.push >>
SUPPLEMENTAL_IMAGES_JSON: << parameters.supplemental_images >>
IMAGE_NAME: << parameters.image_name >>
REGISTRY: << parameters.registry >>
command: << include(scripts/build_push_supplemental.sh) >>
- store_artifacts:
path: /tmp/trivy-supplemental-results
destination: trivy-supplemental-scans
+8 -3
View File
@@ -1,7 +1,12 @@
description: >
Restore the Maven local repository cache based on pom.xml checksum.
Restore only the cache for the checked-out Maven dependency definitions.
steps:
- run:
name: Identify Maven dependency cache
command: <<include(scripts/maven_cache_key.sh)>>
- restore_cache:
keys:
- maven-v1-{{ checksum "pom.xml" }}
- maven-v1-
- maven-v2-{{ checksum ".munitor-maven-cache-key" }}
- run:
name: Restore this build's installed Maven artifacts
command: <<include(scripts/maven_workspace.sh)>>
+7 -2
View File
@@ -1,7 +1,12 @@
description: >
Save the Maven local repository to cache for future builds.
Save the Maven repository under the same immutable key used at restore time.
steps:
- save_cache:
key: maven-v1-{{ checksum "pom.xml" }}
key: maven-v2-{{ checksum ".munitor-maven-cache-key" }}
paths:
- ~/.m2/repository
- run:
name: Preserve installed Maven artifacts for this workspace
environment:
MAVEN_WORKSPACE_ACTION: save
command: <<include(scripts/maven_workspace.sh)>>
+2 -2
View File
@@ -3,8 +3,8 @@ description: >
parameters:
version:
type: string
default: "1.61.1"
default: "1.181.0"
description: Semgrep image version tag.
docker:
- image: returntocorp/semgrep:<< parameters.version >>
- image: semgrep/semgrep:<< parameters.version >>
resource_class: medium
+49
View File
@@ -0,0 +1,49 @@
description: >
Build and scan application and supplemental images without publishing.
Requires the source and version metadata workspace from build-and-test.
parameters:
image_name:
type: string
description: Docker image name (without registry prefix).
registry:
type: string
default: ghcr.io/KofTwentyTwo
description: Container registry URL.
resource_class:
type: enum
enum:
- medium
- large
default: large
description: Machine resource class.
docker_layer_caching:
type: boolean
default: true
description: Enable Docker Layer Caching for faster builds.
supplemental_images:
type: string
default: "[]"
description: JSON array of supplemental images to build and scan.
executor:
name: machine
resource_class: << parameters.resource_class >>
docker_layer_caching: << parameters.docker_layer_caching >>
steps:
- attach_workspace:
at: .
- run:
name: Restore version metadata
command: <<include(scripts/restore_version_metadata.sh)>>
- generate_dockerfile
- docker_build:
image_name: << parameters.image_name >>
registry: << parameters.registry >>
- restore_trivy_cache
- run_trivy_image:
image_tag: << parameters.registry >>/<< parameters.image_name >>:${DOCKER_TAG}
- save_trivy_cache
- build_push_supplemental:
supplemental_images: << parameters.supplemental_images >>
image_name: << parameters.image_name >>
registry: << parameters.registry >>
push: false
+22 -2
View File
@@ -3,6 +3,14 @@ description: >
version, builds, tests, saves cache, and persists the workspace for
downstream jobs.
parameters:
quality_checks:
type: boolean
default: false
description: Run quality and coverage checks on the build worker before sharing artifacts.
min_instruction:
type: string
default: "70"
description: Minimum instruction coverage when quality_checks is enabled.
java_version:
type: string
default: "21"
@@ -29,8 +37,20 @@ steps:
- mvn_set_version
- mvn_build
- mvn_test
- store_test_results:
path: target/surefire-reports
- when:
condition: << parameters.quality_checks >>
steps:
- mvn_checkstyle
- mvn_spotbugs
- mvn_pmd
- mvn_jacoco:
min_instruction: << parameters.min_instruction >>
- store_artifacts:
path: target/checkstyle-result.xml
destination: checkstyle-result.xml
- store_artifacts:
path: target/site/jacoco
destination: coverage-report
- run:
name: Persist version metadata
command: <<include(scripts/persist_version_metadata.sh)>>
+3 -3
View File
@@ -23,12 +23,12 @@ executor:
steps:
- attach_workspace:
at: .
- install_java:
version: << parameters.java_version >>
- restore_maven_cache
- when:
condition: << parameters.owasp >>
steps:
- install_java:
version: << parameters.java_version >>
- restore_maven_cache
- restore_cache:
keys:
- nvd-cache-v1-{{ .Branch }}
+30 -9
View File
@@ -27,6 +27,19 @@ IMAGE="${IMAGE_NAME:?IMAGE_NAME not set}"
REGISTRY="${REGISTRY:-ghcr.io/KofTwentyTwo}"
VERSION="${PROJECT_VERSION:?PROJECT_VERSION not set -- run export_version_vars first}"
ENV_TAG="${DOCKER_ENV_TAG:-}"
# CircleCI renders a boolean orb parameter as "1"/"0" in a step environment, not
# "true"/"false", so accept both spellings. Unset means push (release builds).
case "$(echo "${PUSH_IMAGES-true}" | tr '[:upper:]' '[:lower:]')" in
true | 1 | yes | on) PUSH_IMAGES=true ;;
false | 0 | no | off | "") PUSH_IMAGES=false ;;
*)
echo "ERROR: PUSH_IMAGES must be true/false (or 1/0), got '${PUSH_IMAGES}'" >&2
exit 1
;;
esac
# run_trivy_image installs here in a previous CircleCI step.
export PATH="${HOME}/bin:${PATH}"
mkdir -p /tmp/trivy-supplemental-results
COUNT=$(echo "${IMAGES_JSON}" | jq 'length')
echo "Found ${COUNT} supplemental image(s) to build."
@@ -70,32 +83,40 @@ for i in $(seq 0 $((COUNT - 1))); do
# Build
echo "Building ${FULL_TAG} ..."
BUILD_ARGS=(-f "${DOCKERFILE}" -t "${FULL_TAG}")
# Refresh base images and rerun package upgrades in supplemental Dockerfiles.
BUILD_ARGS=(--pull --no-cache -f "${DOCKERFILE}" -t "${FULL_TAG}")
if [[ -n "${CIRCLE_SHA1:-}" ]]; then
BUILD_ARGS+=(--label "org.opencontainers.image.revision=${CIRCLE_SHA1}")
fi
docker build "${BUILD_ARGS[@]}" .
if [[ -n "${ENV_TAG}" ]]; then
if [[ "${PUSH_IMAGES}" == "true" && -n "${ENV_TAG}" ]]; then
ENV_FULL_TAG="${REGISTRY}/${DERIVED_NAME}:${ENV_TAG}"
docker tag "${FULL_TAG}" "${ENV_FULL_TAG}"
fi
# Trivy scan
echo "Scanning ${FULL_TAG} with Trivy ..."
trivy image --severity HIGH,CRITICAL --exit-code 1 "${FULL_TAG}"
trivy image --format json --output "/tmp/trivy-supplemental-results/${NAME}.json" \
--severity HIGH,CRITICAL --exit-code 1 "${FULL_TAG}" || {
EXIT_CODE=$?
trivy image --severity HIGH,CRITICAL "${FULL_TAG}"
exit "${EXIT_CODE}"
}
# Push
echo "Pushing ${FULL_TAG} ..."
docker push "${FULL_TAG}"
if [[ "${PUSH_IMAGES}" == "true" ]]; then
echo "Pushing ${FULL_TAG} ..."
docker push "${FULL_TAG}"
if [[ -n "${ENV_TAG}" ]]; then
echo "Pushing ${ENV_FULL_TAG} ..."
docker push "${ENV_FULL_TAG}"
if [[ -n "${ENV_TAG}" ]]; then
echo "Pushing ${ENV_FULL_TAG} ..."
docker push "${ENV_FULL_TAG}"
fi
fi
echo "--- Done: ${DERIVED_NAME} ---"
done
echo ""
echo "All supplemental images built, scanned, and pushed."
echo "All supplemental images built and scanned (push=${PUSH_IMAGES})."
+10 -1
View File
@@ -23,7 +23,16 @@ munitor_header "docker_build (${FULL_IMAGE}:${VERSION})"
munitor_check_tool docker --version
# Build with version tag; add env tag when set (e.g., develop, staging)
BUILD_CMD=(docker build -f "${DOCKERFILE}" -t "${FULL_IMAGE}:${VERSION}")
BUILD_CMD=(docker build --pull -f "${DOCKERFILE}" -t "${FULL_IMAGE}:${VERSION}")
# Generated JVM images only copy a prebuilt JAR. Re-run their package upgrades:
# a cached apk upgrade layer can retain vulnerable packages indefinitely.
CONFIG_FILE="${MUNITOR_CONFIG:-.munitor.yml}"
if [[ -f "${CONFIG_FILE}" ]]; then
munitor_check_tool yq --version
case "$(yq -r '.pipeline' "${CONFIG_FILE}")" in
java-webapp|gradle-webapp) BUILD_CMD+=(--no-cache) ;;
esac
fi
if [[ -n "${ENV_TAG}" ]]; then
BUILD_CMD+=(-t "${FULL_IMAGE}:${ENV_TAG}")
fi
+19
View File
@@ -0,0 +1,19 @@
#!/usr/bin/env bash
set -euo pipefail
MUNITOR_HELPERS="${MUNITOR_HELPERS:-$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)/munitor_helpers.sh}"
# shellcheck source=munitor_helpers.sh
if [[ -f "${MUNITOR_HELPERS}" ]]; then
source "${MUNITOR_HELPERS}"
elif ! type munitor_header &>/dev/null; then
munitor_header() { echo "=== Munitor: ${1:-unknown} ==="; }
fi
munitor_header "maven_cache_key"
# Hash the checked-out dependency definitions, not release-version rewrites.
# The same index travels with the workspace to downstream Maven jobs.
git ls-files -s -- pom.xml '**/pom.xml' '.mvn/*' > .munitor-maven-cache-key
if [[ ! -s .munitor-maven-cache-key ]]; then
echo "ERROR: Maven cache requires tracked dependency definitions." >&2
exit 1
fi
+47
View File
@@ -0,0 +1,47 @@
#!/usr/bin/env bash
set -euo pipefail
MUNITOR_HELPERS="${MUNITOR_HELPERS:-$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)/munitor_helpers.sh}"
# shellcheck source=munitor_helpers.sh
if [[ -f "${MUNITOR_HELPERS}" ]]; then
source "${MUNITOR_HELPERS}"
elif ! type munitor_header &>/dev/null; then
munitor_header() { echo "=== Munitor: ${1:-unknown} ==="; }
fi
munitor_header "maven_workspace"
archive="${PWD}/.munitor-maven-installed.tgz"
repository="${HOME}/.m2/repository"
case "${MAVEN_WORKSPACE_ACTION:-restore}" in
save)
# Maven marks local installs with an empty repository ID (artifact>=).
# Carry build outputs and mutable SNAPSHOT dependencies separately from the cache.
files=$(mktemp)
trap 'rm -f "${files}"' EXIT
cd "${repository}"
while IFS= read -r -d '' tracking; do
if [[ "${tracking}" == */*-SNAPSHOT/_remote.repositories ]] ||
grep -qE '^[^#].*>=[[:space:]]*$' "${tracking}"; then
directory=$(dirname "${tracking}")
printf '%s\0' "${directory}" >> "${files}"
while [[ "${directory}" != . ]]; do
directory=$(dirname "${directory}")
if [[ -f "${directory}/maven-metadata-local.xml" ]]; then
printf '%s\0' "${directory}/maven-metadata-local.xml" >> "${files}"
fi
done
fi
done < <(find . -name _remote.repositories -type f -print0)
tar -czf "${archive}" --null -T "${files}"
;;
restore)
if [[ -f "${archive}" ]]; then
mkdir -p "${repository}"
tar -xzf "${archive}" -C "${repository}"
fi
;;
*)
echo "ERROR: Unknown Maven workspace action." >&2
exit 1
;;
esac
+68 -69
View File
@@ -358,6 +358,8 @@ workflows:
jobs:
- munitor/mvn_build_and_test:
name: build-and-test
quality_checks: true
min_instruction: "${MUNITOR_COVERAGE_MIN}"
java_version: "${MUNITOR_JAVA_VERSION}"
context:
- ${MUNITOR_CONTEXT_GITHUB}
@@ -366,6 +368,20 @@ workflows:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/docker_scan:
name: image-scan
image_name: ${MUNITOR_IMAGE_NAME}
registry: ${MUNITOR_DOCKER_REGISTRY}
##IF_SUPPLEMENTAL##
supplemental_images: '${MUNITOR_SUPPLEMENTAL_IMAGES_JSON}'
##ENDIF_SUPPLEMENTAL##
requires:
- build-and-test
filters:
branches:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/secrets_scan:
name: secrets-scan
filters:
@@ -381,27 +397,6 @@ workflows:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/mvn_code_quality:
name: code-quality
java_version: "${MUNITOR_JAVA_VERSION}"
requires:
- build-and-test
filters:
branches:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/mvn_coverage:
name: coverage
java_version: "${MUNITOR_JAVA_VERSION}"
min_instruction: "${MUNITOR_COVERAGE_MIN}"
requires:
- build-and-test
filters:
branches:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/security_scan:
name: security-scan
java_version: "${MUNITOR_JAVA_VERSION}"
@@ -439,6 +434,8 @@ workflows:
jobs:
- munitor/mvn_build_and_test:
name: build-and-test
quality_checks: true
min_instruction: "${MUNITOR_COVERAGE_MIN}"
java_version: "${MUNITOR_JAVA_VERSION}"
context:
- ${MUNITOR_CONTEXT_GITHUB}
@@ -459,25 +456,6 @@ workflows:
branches:
only:
- /release\/.*/
- munitor/mvn_code_quality:
name: code-quality
java_version: "${MUNITOR_JAVA_VERSION}"
requires:
- build-and-test
filters:
branches:
only:
- /release\/.*/
- munitor/mvn_coverage:
name: coverage
java_version: "${MUNITOR_JAVA_VERSION}"
min_instruction: "${MUNITOR_COVERAGE_MIN}"
requires:
- build-and-test
filters:
branches:
only:
- /release\/.*/
- munitor/security_scan:
name: security-scan
java_version: "${MUNITOR_JAVA_VERSION}"
@@ -512,7 +490,6 @@ workflows:
sonar_project_key: ${MUNITOR_SONAR_PROJECT_KEY}
requires:
- build-and-test
- coverage
context:
- ${MUNITOR_CONTEXT_SONAR}
filters:
@@ -533,8 +510,7 @@ workflows:
supplemental_images: '${MUNITOR_SUPPLEMENTAL_IMAGES_JSON}'
##ENDIF_SUPPLEMENTAL##
requires:
- code-quality
- coverage
- build-and-test
- secrets-scan
- sast-scan
- security-scan
@@ -557,8 +533,7 @@ workflows:
context:
- ${MUNITOR_CONTEXT_GITHUB}
requires:
- code-quality
- coverage
- build-and-test
- secrets-scan
- sast-scan
- security-scan
@@ -613,7 +588,7 @@ workflows:
production:
jobs:
- munitor/mvn_build_and_test:
name: build-and-test
name: production-build-and-test
java_version: "${MUNITOR_JAVA_VERSION}"
context:
- ${MUNITOR_CONTEXT_GITHUB}
@@ -633,7 +608,7 @@ workflows:
supplemental_images: '${MUNITOR_SUPPLEMENTAL_IMAGES_JSON}'
##ENDIF_SUPPLEMENTAL##
requires:
- build-and-test
- production-build-and-test
context:
- ${MUNITOR_CONTEXT_REGISTRY}
filters:
@@ -694,6 +669,20 @@ workflows:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/docker_scan:
name: image-scan
image_name: ${MUNITOR_IMAGE_NAME}
registry: ${MUNITOR_DOCKER_REGISTRY}
##IF_SUPPLEMENTAL##
supplemental_images: '${MUNITOR_SUPPLEMENTAL_IMAGES_JSON}'
##ENDIF_SUPPLEMENTAL##
requires:
- build-and-test
filters:
branches:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/secrets_scan:
name: secrets-scan
filters:
@@ -1037,6 +1026,20 @@ workflows:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/docker_scan:
name: image-scan
image_name: ${MUNITOR_IMAGE_NAME}
registry: ${MUNITOR_DOCKER_REGISTRY}
##IF_SUPPLEMENTAL##
supplemental_images: '${MUNITOR_SUPPLEMENTAL_IMAGES_JSON}'
##ENDIF_SUPPLEMENTAL##
requires:
- build-and-test
filters:
branches:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/secrets_scan:
name: secrets-scan
filters:
@@ -1406,6 +1409,20 @@ workflows:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/docker_scan:
name: image-scan
image_name: ${MUNITOR_IMAGE_NAME}
registry: ${MUNITOR_DOCKER_REGISTRY}
##IF_SUPPLEMENTAL##
supplemental_images: '${MUNITOR_SUPPLEMENTAL_IMAGES_JSON}'
##ENDIF_SUPPLEMENTAL##
requires:
- build-and-test
filters:
branches:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/secrets_scan:
name: secrets-scan
filters:
@@ -2516,6 +2533,8 @@ cat > /tmp/munitor/templates/partials/java-webapp-deploy.yml.tpl << 'MUNITOR_PAR
jobs:
- munitor/mvn_build_and_test:
name: build-and-test
quality_checks: true
min_instruction: "${MUNITOR_COVERAGE_MIN}"
java_version: "${MUNITOR_JAVA_VERSION}"
context:
- ${MUNITOR_CONTEXT_GITHUB}
@@ -2533,23 +2552,6 @@ cat > /tmp/munitor/templates/partials/java-webapp-deploy.yml.tpl << 'MUNITOR_PAR
filters:
branches:
only: __BRANCH_FILTER__
- munitor/mvn_code_quality:
name: code-quality
java_version: "${MUNITOR_JAVA_VERSION}"
requires:
- build-and-test
filters:
branches:
only: __BRANCH_FILTER__
- munitor/mvn_coverage:
name: coverage
java_version: "${MUNITOR_JAVA_VERSION}"
min_instruction: "${MUNITOR_COVERAGE_MIN}"
requires:
- build-and-test
filters:
branches:
only: __BRANCH_FILTER__
- munitor/security_scan:
name: security-scan
java_version: "${MUNITOR_JAVA_VERSION}"
@@ -2582,7 +2584,6 @@ cat > /tmp/munitor/templates/partials/java-webapp-deploy.yml.tpl << 'MUNITOR_PAR
sonar_project_key: ${MUNITOR_SONAR_PROJECT_KEY}
requires:
- build-and-test
- coverage
context:
- ${MUNITOR_CONTEXT_SONAR}
filters:
@@ -2600,8 +2601,7 @@ cat > /tmp/munitor/templates/partials/java-webapp-deploy.yml.tpl << 'MUNITOR_PAR
supplemental_images: '${MUNITOR_SUPPLEMENTAL_IMAGES_JSON}'
##ENDIF_SUPPLEMENTAL##
requires:
- code-quality
- coverage
- build-and-test
- secrets-scan
- sast-scan
- security-scan
@@ -2623,8 +2623,7 @@ cat > /tmp/munitor/templates/partials/java-webapp-deploy.yml.tpl << 'MUNITOR_PAR
context:
- ${MUNITOR_CONTEXT_GITHUB}
requires:
- code-quality
- coverage
- build-and-test
- secrets-scan
- sast-scan
- security-scan
+34 -11
View File
@@ -15,22 +15,45 @@ munitor_header "run_semgrep"
CONFIG="${SEMGREP_CONFIG:-auto}"
FAIL_ON_FINDINGS="${SEMGREP_FAIL_ON_FINDINGS:-true}"
REPORT="${SEMGREP_REPORT:-/tmp/semgrep-results.json}"
echo "Running Semgrep SAST scan (config: ${CONFIG}, fail_on_findings: ${FAIL_ON_FINDINGS})..."
EXIT_CODE=0
semgrep ci \
--config "${CONFIG}" \
--json \
--output /tmp/semgrep-results.json || {
EXIT_CODE=$?
if [[ "${FAIL_ON_FINDINGS}" == "true" ]]; then
echo "ERROR: Semgrep found issues."
echo "Review the report at: /tmp/semgrep-results.json"
exit "${EXIT_CODE}"
else
echo "WARNING: Semgrep found issues (warn-only mode, not failing build)."
echo "Review the report at: /tmp/semgrep-results.json"
fi
}
--output "${REPORT}" || EXIT_CODE=$?
# Summarise the report, and fail a scan that ran no rules or scanned no files:
# semgrep exits 0 when the downloaded rules fail to parse (an outdated semgrep),
# which used to pass this job without scanning anything.
python3 - "${REPORT}" <<'PY'
import json, sys
report = json.load(open(sys.argv[1]))
scanned = len(report.get("paths", {}).get("scanned", []))
results = report.get("results", [])
errors = report.get("errors", [])
rule_errors = [e for e in errors if "rule" in str(e.get("type", "")).lower()
or "config" in str(e.get("message", "")).lower()]
print(f"Semgrep: {scanned} files scanned, {len(results)} findings, {len(errors)} errors.")
for r in results[:50]:
print(f" {r.get('check_id')} {r.get('path')}:{r.get('start', {}).get('line')}")
if scanned == 0 or rule_errors:
for e in rule_errors[:5]:
print(f" error: {str(e.get('message', e))[:300]}")
print("ERROR: Semgrep did not run its rules (no files scanned or invalid rule configuration).")
sys.exit(2)
PY
if [[ "${EXIT_CODE}" -ne 0 ]]; then
if [[ "${FAIL_ON_FINDINGS}" == "true" ]]; then
echo "ERROR: Semgrep found issues."
echo "Review the report at: ${REPORT}"
exit "${EXIT_CODE}"
fi
echo "WARNING: Semgrep found issues (warn-only mode, not failing build)."
echo "Review the report at: ${REPORT}"
fi
echo "Semgrep scan completed."
+4
View File
@@ -36,13 +36,17 @@ fi
echo "Running Trivy filesystem scan..."
# --exit-code 1: without it trivy exits 0 even when it reports HIGH/CRITICAL
# findings, so this job could never fail.
trivy fs \
--format json \
--output /tmp/trivy-results.json \
--severity HIGH,CRITICAL \
--exit-code 1 \
. || {
EXIT_CODE=$?
echo "Trivy found vulnerabilities. See /tmp/trivy-results.json"
trivy fs --severity HIGH,CRITICAL . || true
exit "${EXIT_CODE}"
}
@@ -16,6 +16,20 @@ workflows:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/docker_scan:
name: image-scan
image_name: ${MUNITOR_IMAGE_NAME}
registry: ${MUNITOR_DOCKER_REGISTRY}
##IF_SUPPLEMENTAL##
supplemental_images: '${MUNITOR_SUPPLEMENTAL_IMAGES_JSON}'
##ENDIF_SUPPLEMENTAL##
requires:
- build-and-test
filters:
branches:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/secrets_scan:
name: secrets-scan
filters:
+22 -47
View File
@@ -8,6 +8,8 @@ workflows:
jobs:
- munitor/mvn_build_and_test:
name: build-and-test
quality_checks: true
min_instruction: "${MUNITOR_COVERAGE_MIN}"
java_version: "${MUNITOR_JAVA_VERSION}"
context:
- ${MUNITOR_CONTEXT_GITHUB}
@@ -16,6 +18,20 @@ workflows:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/docker_scan:
name: image-scan
image_name: ${MUNITOR_IMAGE_NAME}
registry: ${MUNITOR_DOCKER_REGISTRY}
##IF_SUPPLEMENTAL##
supplemental_images: '${MUNITOR_SUPPLEMENTAL_IMAGES_JSON}'
##ENDIF_SUPPLEMENTAL##
requires:
- build-and-test
filters:
branches:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/secrets_scan:
name: secrets-scan
filters:
@@ -31,27 +47,6 @@ workflows:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/mvn_code_quality:
name: code-quality
java_version: "${MUNITOR_JAVA_VERSION}"
requires:
- build-and-test
filters:
branches:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/mvn_coverage:
name: coverage
java_version: "${MUNITOR_JAVA_VERSION}"
min_instruction: "${MUNITOR_COVERAGE_MIN}"
requires:
- build-and-test
filters:
branches:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/security_scan:
name: security-scan
java_version: "${MUNITOR_JAVA_VERSION}"
@@ -89,6 +84,8 @@ workflows:
jobs:
- munitor/mvn_build_and_test:
name: build-and-test
quality_checks: true
min_instruction: "${MUNITOR_COVERAGE_MIN}"
java_version: "${MUNITOR_JAVA_VERSION}"
context:
- ${MUNITOR_CONTEXT_GITHUB}
@@ -109,25 +106,6 @@ workflows:
branches:
only:
- /release\/.*/
- munitor/mvn_code_quality:
name: code-quality
java_version: "${MUNITOR_JAVA_VERSION}"
requires:
- build-and-test
filters:
branches:
only:
- /release\/.*/
- munitor/mvn_coverage:
name: coverage
java_version: "${MUNITOR_JAVA_VERSION}"
min_instruction: "${MUNITOR_COVERAGE_MIN}"
requires:
- build-and-test
filters:
branches:
only:
- /release\/.*/
- munitor/security_scan:
name: security-scan
java_version: "${MUNITOR_JAVA_VERSION}"
@@ -162,7 +140,6 @@ workflows:
sonar_project_key: ${MUNITOR_SONAR_PROJECT_KEY}
requires:
- build-and-test
- coverage
context:
- ${MUNITOR_CONTEXT_SONAR}
filters:
@@ -183,8 +160,7 @@ workflows:
supplemental_images: '${MUNITOR_SUPPLEMENTAL_IMAGES_JSON}'
##ENDIF_SUPPLEMENTAL##
requires:
- code-quality
- coverage
- build-and-test
- secrets-scan
- sast-scan
- security-scan
@@ -207,8 +183,7 @@ workflows:
context:
- ${MUNITOR_CONTEXT_GITHUB}
requires:
- code-quality
- coverage
- build-and-test
- secrets-scan
- sast-scan
- security-scan
@@ -263,7 +238,7 @@ workflows:
production:
jobs:
- munitor/mvn_build_and_test:
name: build-and-test
name: production-build-and-test
java_version: "${MUNITOR_JAVA_VERSION}"
context:
- ${MUNITOR_CONTEXT_GITHUB}
@@ -283,7 +258,7 @@ workflows:
supplemental_images: '${MUNITOR_SUPPLEMENTAL_IMAGES_JSON}'
##ENDIF_SUPPLEMENTAL##
requires:
- build-and-test
- production-build-and-test
context:
- ${MUNITOR_CONTEXT_REGISTRY}
filters:
+14
View File
@@ -31,6 +31,20 @@ workflows:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/docker_scan:
name: image-scan
image_name: ${MUNITOR_IMAGE_NAME}
registry: ${MUNITOR_DOCKER_REGISTRY}
##IF_SUPPLEMENTAL##
supplemental_images: '${MUNITOR_SUPPLEMENTAL_IMAGES_JSON}'
##ENDIF_SUPPLEMENTAL##
requires:
- build-and-test
filters:
branches:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/secrets_scan:
name: secrets-scan
filters:
+14
View File
@@ -31,6 +31,20 @@ workflows:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/docker_scan:
name: image-scan
image_name: ${MUNITOR_IMAGE_NAME}
registry: ${MUNITOR_DOCKER_REGISTRY}
##IF_SUPPLEMENTAL##
supplemental_images: '${MUNITOR_SUPPLEMENTAL_IMAGES_JSON}'
##ENDIF_SUPPLEMENTAL##
requires:
- build-and-test
filters:
branches:
only:
- /feature\/.*/
- /hotfix\/.*/
- munitor/secrets_scan:
name: secrets-scan
filters:
@@ -2,6 +2,8 @@
jobs:
- munitor/mvn_build_and_test:
name: build-and-test
quality_checks: true
min_instruction: "${MUNITOR_COVERAGE_MIN}"
java_version: "${MUNITOR_JAVA_VERSION}"
context:
- ${MUNITOR_CONTEXT_GITHUB}
@@ -19,23 +21,6 @@
filters:
branches:
only: __BRANCH_FILTER__
- munitor/mvn_code_quality:
name: code-quality
java_version: "${MUNITOR_JAVA_VERSION}"
requires:
- build-and-test
filters:
branches:
only: __BRANCH_FILTER__
- munitor/mvn_coverage:
name: coverage
java_version: "${MUNITOR_JAVA_VERSION}"
min_instruction: "${MUNITOR_COVERAGE_MIN}"
requires:
- build-and-test
filters:
branches:
only: __BRANCH_FILTER__
- munitor/security_scan:
name: security-scan
java_version: "${MUNITOR_JAVA_VERSION}"
@@ -68,7 +53,6 @@
sonar_project_key: ${MUNITOR_SONAR_PROJECT_KEY}
requires:
- build-and-test
- coverage
context:
- ${MUNITOR_CONTEXT_SONAR}
filters:
@@ -86,8 +70,7 @@
supplemental_images: '${MUNITOR_SUPPLEMENTAL_IMAGES_JSON}'
##ENDIF_SUPPLEMENTAL##
requires:
- code-quality
- coverage
- build-and-test
- secrets-scan
- sast-scan
- security-scan
@@ -109,8 +92,7 @@
context:
- ${MUNITOR_CONTEXT_GITHUB}
requires:
- code-quality
- coverage
- build-and-test
- secrets-scan
- sast-scan
- security-scan
+6
View File
@@ -0,0 +1,6 @@
#!/usr/bin/env bats
@test "PR and release JVM builds refresh packages without disabling Node caching or Trivy" {
run bash "${BATS_TEST_DIRNAME}/test_docker_build.sh"
[ "$status" -eq 0 ]
}
+84
View File
@@ -0,0 +1,84 @@
#!/usr/bin/env bats
setup() {
ROOT="${BATS_TEST_DIRNAME}/.."
CONFIG="${BATS_TEST_TMPDIR}/rendered.yml"
}
check_workflows() {
local expected="$1"
yq -o=json '.' "$CONFIG" | EXPECTED_COVERAGE="$expected" node -e '
const assert = require("node:assert/strict");
const config = JSON.parse(require("node:fs").readFileSync(0,"utf8"));
for (const [workflow, definition] of Object.entries(config.workflows)) {
const jobs = definition.jobs.map(entry => {
const [type, options] = Object.entries(entry)[0];
assert.ok(!["munitor/mvn_code_quality", "munitor/mvn_coverage"].includes(type));
return {type, ...options};
});
const names = jobs.map(job => job.name);
assert.equal(new Set(names).size, names.length, workflow);
const buildName = workflow === "production" ? "production-build-and-test" : "build-and-test";
const build = jobs.find(job => job.name === buildName);
assert.ok(build, workflow);
assert.equal(jobs.filter(job => job.type === "munitor/mvn_build_and_test").length, 1);
assert.ok(!names.includes(workflow === "production" ? "build-and-test" : "production-build-and-test"));
const expectedFilter = {
"pr-checks": ["/feature\\/.*/", "/hotfix\\/.*/"],
develop: "develop", staging: "staging",
"release-candidate": ["/release\\/.*/"], production: "main"
}[workflow];
assert.deepEqual(build.filters.branches.only, expectedFilter, workflow);
if (workflow === "production") {
assert.equal(build.quality_checks, undefined);
assert.equal(build.min_instruction, undefined);
} else {
assert.equal(build.quality_checks, true, workflow);
assert.equal(String(build.min_instruction), process.env.EXPECTED_COVERAGE, workflow);
for (const gate of ["secrets-scan", "sast-scan", "security-scan"]) assert.ok(names.includes(gate), workflow);
}
for (const job of jobs) {
assert.deepEqual(job.filters, build.filters, `${workflow}/${job.name} filters`);
const dependencies = job.requires || [];
assert.equal(new Set(dependencies).size, dependencies.length, job.name);
for (const name of dependencies) assert.ok(names.includes(name), `${workflow}/${job.name}: ${name}`);
if (["image-scan", "security-scan", "e2e-tests", "sonar-scan"].includes(job.name)) assert.ok(dependencies.includes(buildName));
if (["docker-build-push", "sbom"].includes(job.name)) {
assert.ok(dependencies.includes(buildName));
if (workflow !== "production") for (const gate of ["secrets-scan", "sast-scan", "security-scan", "e2e-tests", "sonar-scan"].filter(name => names.includes(name))) assert.ok(dependencies.includes(gate), `${workflow}/${job.name}/${gate}`);
}
if (["update-cd-repo", "github-release"].includes(job.name)) assert.ok(dependencies.includes("docker-build-push"));
}
if (workflow === "pr-checks") assert.ok(names.includes("image-scan"));
}
for (const workflow of ["pr-checks", "develop", "staging", "release-candidate", "production"]) assert.ok(config.workflows[workflow]);
'
}
@test "full Java workflow folds quality into build and preserves optional gates" {
MUNITOR_CONFIG="${ROOT}/tests/fixtures/java-webapp.munitor.yml" MUNITOR_OUTPUT_FILE="$CONFIG" bash "${ROOT}/src/scripts/generate_config.sh"
check_workflows 80
[ "$(yq '.workflows.develop.jobs[] | .[] | select(.name == "sonar-scan") | .name' "$CONFIG")" = "sonar-scan" ]
}
@test "minimal Java workflow retains default coverage and production behavior" {
MUNITOR_CONFIG="${ROOT}/tests/fixtures/java-webapp-minimal.munitor.yml" MUNITOR_OUTPUT_FILE="$CONFIG" bash "${ROOT}/src/scripts/generate_config.sh"
check_workflows 70
}
@test "quality steps finish before workspace persistence and forward coverage threshold" {
yq -o=json '.' "${ROOT}/src/jobs/mvn_build_and_test.yml" | node -e '
const assert = require("node:assert/strict");
const job = JSON.parse(require("node:fs").readFileSync(0,"utf8"));
assert.equal(job.parameters.quality_checks.default, false);
assert.equal(job.parameters.min_instruction.default, "70");
const index = name => job.steps.findIndex(step => step === name || typeof step === "object" && name in step);
assert.ok(index("mvn_test") < index("when"));
assert.ok(index("when") < index("persist_to_workspace"));
const block = job.steps[index("when")].when;
assert.equal(block.condition, "<< parameters.quality_checks >>");
const commands = block.steps.map(step => typeof step === "string" ? step : Object.keys(step)[0]);
assert.deepEqual(commands.slice(0,4), ["mvn_checkstyle", "mvn_spotbugs", "mvn_pmd", "mvn_jacoco"]);
assert.equal(block.steps[3].mvn_jacoco.min_instruction, "<< parameters.min_instruction >>");
'
}
+47
View File
@@ -0,0 +1,47 @@
#!/usr/bin/env bats
setup() {
SCRIPT="${BATS_TEST_DIRNAME}/../src/scripts/maven_cache_key.sh"
cd "$BATS_TEST_TMPDIR"
git init -q
printf '<project><version>1-SNAPSHOT</version></project>\n' > pom.xml
git add pom.xml
}
@test "release rewrites retain the restored cache identity" {
bash "$SCRIPT"
cp .munitor-maven-cache-key before
printf '<project><version>1.2.3</version></project>\n' > pom.xml
bash "$SCRIPT"
cmp before .munitor-maven-cache-key
}
@test "nested checked-out dependency changes invalidate the cache" {
mkdir module
printf '<project><dependencies/></project>\n' > module/pom.xml
git add module/pom.xml
bash "$SCRIPT"
cp .munitor-maven-cache-key before
printf '<project><dependencies>changed</dependencies></project>\n' > module/pom.xml
git add module/pom.xml
bash "$SCRIPT"
! cmp -s before .munitor-maven-cache-key
}
@test "source changes do not invalidate dependency cache" {
bash "$SCRIPT"
cp .munitor-maven-cache-key before
printf 'class Example {}\n' > Example.java
git add Example.java
bash "$SCRIPT"
cmp before .munitor-maven-cache-key
}
@test "restore has no broad fallback and save uses the same key" {
command_root="${BATS_TEST_DIRNAME}/../src/commands"
restore=$(sed -n '/maven-v2-/p' "$command_root/restore_maven_cache.yml")
save=$(sed -n '/maven-v2-/p' "$command_root/save_maven_cache.yml")
[[ "$restore" == *'maven-v2-{{ checksum ".munitor-maven-cache-key" }}'* ]]
[[ "$save" == *'maven-v2-{{ checksum ".munitor-maven-cache-key" }}'* ]]
! grep -q 'maven-v1\|maven-v2-$' "$command_root/restore_maven_cache.yml"
}
+42
View File
@@ -0,0 +1,42 @@
#!/usr/bin/env bats
setup() {
SCRIPT="${BATS_TEST_DIRNAME}/../src/scripts/maven_workspace.sh"
export HOME="${BATS_TEST_TMPDIR}/home"
cd "$BATS_TEST_TMPDIR"
}
@test "workspace overlays current installed artifacts and snapshot metadata without copying remote dependencies" {
repo="$HOME/.m2/repository"
mkdir -p "$repo/example/library/2-SNAPSHOT" "$repo/vendor/dependency/1"
printf 'library-2-SNAPSHOT.jar>=\r\nlibrary-2-SNAPSHOT.pom>=\r\nlibrary-2-SNAPSHOT-tests.jar>=\r\n' > "$repo/example/library/2-SNAPSHOT/_remote.repositories"
for extension in jar pom; do
printf 'current build\n' > "$repo/example/library/2-SNAPSHOT/library-2-SNAPSHOT.$extension"
done
printf 'current test classes\n' > "$repo/example/library/2-SNAPSHOT/library-2-SNAPSHOT-tests.jar"
printf 'snapshot metadata\n' > "$repo/example/library/maven-metadata-local.xml"
printf 'plugin prefix metadata\n' > "$repo/example/maven-metadata-local.xml"
printf 'dependency-1.jar>central=\n' > "$repo/vendor/dependency/1/_remote.repositories"
printf 'remote dependency\n' > "$repo/vendor/dependency/1/dependency-1.jar"
mkdir -p "$repo/vendor/mutable/2-SNAPSHOT"
printf 'mutable-2-SNAPSHOT.jar>central=\n' > "$repo/vendor/mutable/2-SNAPSHOT/_remote.repositories"
printf 'updated snapshot\n' > "$repo/vendor/mutable/2-SNAPSHOT/mutable-2-SNAPSHOT.jar"
MAVEN_WORKSPACE_ACTION=save bash "$SCRIPT"
run tar -tzf .munitor-maven-installed.tgz
[ "$status" -eq 0 ]
[[ "$output" != *vendor/dependency* ]]
export HOME="${BATS_TEST_TMPDIR}/downstream"
bash "$SCRIPT"
restored="$HOME/.m2/repository/example/library"
cmp "$repo/example/library/2-SNAPSHOT/library-2-SNAPSHOT.jar" "$restored/2-SNAPSHOT/library-2-SNAPSHOT.jar"
cmp "$repo/example/library/maven-metadata-local.xml" "$restored/maven-metadata-local.xml"
cmp "$repo/example/maven-metadata-local.xml" "$restored/../maven-metadata-local.xml"
[ -f "$restored/2-SNAPSHOT/library-2-SNAPSHOT-tests.jar" ]
[ ! -e "$HOME/.m2/repository/vendor/dependency" ]
cmp "$repo/vendor/mutable/2-SNAPSHOT/mutable-2-SNAPSHOT.jar" "$HOME/.m2/repository/vendor/mutable/2-SNAPSHOT/mutable-2-SNAPSHOT.jar"
}
@test "initial build without a workspace archive needs no overlay" {
run bash "$SCRIPT"
[ "$status" -eq 0 ]
}
+49
View File
@@ -0,0 +1,49 @@
// Public CI config only: no network, credentials or publication.
const assert = require('node:assert/strict');
const fs = require('node:fs');
const config = JSON.parse(fs.readFileSync(0, 'utf8'));
const workflows = Object.values(config.workflows);
assert.equal(workflows.length, 1, 'all publishers must share the checked pack workspace');
const jobs = workflows.flatMap(workflow => workflow.jobs).map(job => {
const [type, settings] = Object.entries(job)[0];
return { type, ...settings, name: settings.name || type };
});
assert.equal(jobs.filter(job => job.type === 'orb-tools/pack').length, 1, 'pack once per pipeline');
const gates = ['orb-tools/lint', 'orb-tools/pack', 'orb-tools/review', 'test'];
for (const gate of gates) assert.equal(jobs.filter(job => job.name === gate).length, 1, `retain required context ${gate}`);
const publishers = jobs.filter(job => job.type === 'orb-tools/publish');
assert.equal(publishers.length, 3);
for (const job of publishers) {
assert.deepEqual([...job.requires].sort(), [...gates].sort(), `${job.name} must wait for every gate`);
assert.equal(job.orb_name, 'kof22/munitor');
assert.deepEqual(job.context, ['orb-publishing']);
}
function matches(pattern, value) {
return pattern.startsWith('/') && pattern.endsWith('/')
? new RegExp(`^(?:${pattern.slice(1, -1)})$`).test(value) : pattern === value;
}
function runs(job, kind, ref) {
const filter = job.filters?.[kind];
if (!filter) return kind === 'branches'; // CircleCI ignores tags unless opted in.
const list = value => Array.isArray(value) ? value : [value];
return (!filter.only || list(filter.only).some(pattern => matches(pattern, ref)))
&& (!filter.ignore || !list(filter.ignore).some(pattern => matches(pattern, ref)));
}
const cases = [
['branches', 'develop', ['publish-develop-snapshot']],
['branches', 'feature/ci-speed', ['publish-feature-sha']],
['branches', 'main', []], ['branches', 'hotfix/1.2.3', []],
['branches', 'release/1.2.3', []], ['branches', 'fix/ci-speed', []],
['tags', 'v1.2.3', ['orb-tools/publish']],
['tags', 'v1.2.3-RC.1', []], ['tags', 'v1.2', []], ['tags', 'other', []],
];
for (const [kind, ref, expected] of cases) {
assert.deepEqual(publishers.filter(job => runs(job, kind, ref)).map(job => job.name).sort(), expected.sort(), `${kind}:${ref}`);
for (const gate of gates) assert.ok(runs(jobs.find(job => job.name === gate), kind, ref), `${gate} must run for ${ref}`);
}
assert.equal(jobs.find(job => job.name === 'publish-develop-snapshot').dev_tags, 'dev:${CIRCLE_SHA1},dev:snapshot');
assert.equal(jobs.find(job => job.name === 'publish-feature-sha').dev_tags, 'dev:${CIRCLE_SHA1}');
assert.equal(jobs.find(job => job.name === 'orb-tools/publish').pub_type, 'production');
assert.equal(jobs.find(job => job.name === 'publish-feature-sha').pub_type, 'dev');
assert.equal(jobs.find(job => job.name === 'publish-develop-snapshot').pub_type, 'dev');
console.log('PASS: one pack, unchanged required contexts, all publish gates and 10 branch/tag cases');
+45
View File
@@ -0,0 +1,45 @@
#!/usr/bin/env bats
setup() {
export JOB="${BATS_TEST_DIRNAME}/../src/jobs/security_scan.yml"
}
# Resolve only this job's compile-time OWASP condition, rejecting unknown ones.
resolved_steps() {
yq -o=json '.' "$JOB" | jq --argjson enabled "$1" '
def expand:
.[] | if type == "object" and has("when") then
if .when.condition != "<< parameters.owasp >>" then error("unknown condition")
elif $enabled then .when.steps | expand else empty end
else . end;
[.steps | expand]'
}
@test "OWASP disabled skips Java Maven and NVD but retains Trivy and its artifact" {
run resolved_steps false
[ "$status" -eq 0 ]
echo "$output" | jq -e '
map(if type == "string" then . else keys[0] end) ==
["attach_workspace", "restore_trivy_cache", "run_trivy", "save_trivy_cache", "store_artifacts"]' >/dev/null
echo "$output" | jq -e '.[-1].store_artifacts.path == "/tmp/trivy-results.json"' >/dev/null
}
@test "OWASP enabled retains ordered Java Maven NVD scan and report workflow" {
run resolved_steps true
[ "$status" -eq 0 ]
echo "$output" | jq -e '
map(if type == "string" then . else keys[0] end) ==
["attach_workspace", "install_java", "restore_maven_cache", "restore_cache",
"mvn_owasp", "save_cache", "restore_trivy_cache", "run_trivy", "save_trivy_cache",
"store_artifacts", "store_artifacts"]' >/dev/null
echo "$output" | jq -e '
.[1].install_java.version == "<< parameters.java_version >>" and
.[3].restore_cache.keys == ["nvd-cache-v1-{{ .Branch }}", "nvd-cache-v1-"] and
.[5].save_cache.paths == ["~/.dependency-check"] and
.[9].store_artifacts.path == "target/dependency-check-report.html" and
.[10].store_artifacts.path == "/tmp/trivy-results.json"' >/dev/null
}
@test "OWASP remains enabled by default" {
[ "$(yq '.parameters.owasp.default' "$JOB")" = true ]
}
+53
View File
@@ -0,0 +1,53 @@
#!/usr/bin/env bash
# Assert the actual shared PR/release build command without a Docker daemon.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PROJECT_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
WORK=$(mktemp -d)
trap 'rm -rf "${WORK}"' EXIT
mkdir -p "${WORK}/bin" "${WORK}/repo"
cat > "${WORK}/bin/docker" <<'MOCK'
#!/usr/bin/env bash
[[ "$1" == --version ]] && exit 0
printf '%s\n' "$@" > "${CALL_LOG}"
MOCK
cat > "${WORK}/bin/yq" <<'MOCK'
#!/usr/bin/env bash
[[ "$1" == --version ]] && exit 0
[[ "$1" == -r && "$2" == .pipeline ]] || exit 2
sed -n 's/^pipeline: //p' "$3"
MOCK
chmod +x "${WORK}/bin/docker" "${WORK}/bin/yq"
export PATH="${WORK}/bin:${PATH}"
export CALL_LOG="${WORK}/calls" BASH_ENV="${WORK}/env"
export IMAGE_NAME=app REGISTRY=ghcr.io/example PROJECT_VERSION=1.2.3
export CIRCLE_SHA1=abc123 DOCKER_ENV_TAG=develop
cd "${WORK}/repo"
for pipeline in java-webapp gradle-webapp node-api node-webapp; do
printf 'pipeline: %s\n' "${pipeline}" > .munitor.yml
bash "${PROJECT_DIR}/src/scripts/docker_build.sh" > "${WORK}/output"
grep -qx -- '--pull' "${CALL_LOG}"
if [[ "${pipeline}" == java-webapp || "${pipeline}" == gradle-webapp ]]; then
grep -qx -- '--no-cache' "${CALL_LOG}"
else
! grep -qx -- '--no-cache' "${CALL_LOG}"
fi
grep -qx 'org.opencontainers.image.revision=abc123' "${CALL_LOG}"
grep -qx 'ghcr.io/example/app:1.2.3' "${CALL_LOG}"
grep -qx 'ghcr.io/example/app:develop' "${CALL_LOG}"
echo "PASS: ${pipeline} cache policy, provenance and tags"
done
mv .munitor.yml 'custom config.yml'
printf 'pipeline: java-webapp\n' > 'custom config.yml'
MUNITOR_CONFIG='custom config.yml' bash "${PROJECT_DIR}/src/scripts/docker_build.sh" > "${WORK}/output"
grep -qx -- '--no-cache' "${CALL_LOG}"
echo 'PASS: alternate manifest path'
bash "${PROJECT_DIR}/src/scripts/docker_build.sh" > "${WORK}/output"
grep -qx -- '--pull' "${CALL_LOG}"
! grep -qx -- '--no-cache' "${CALL_LOG}"
echo 'PASS: standalone build without manifest'
for job in docker_scan docker_build_push; do
grep -q '^ - docker_build:' "${PROJECT_DIR}/src/jobs/${job}.yml"
grep -q '^ - run_trivy_image:' "${PROJECT_DIR}/src/jobs/${job}.yml"
echo "PASS: ${job} uses shared builder and retains Trivy"
done
+139
View File
@@ -0,0 +1,139 @@
#!/usr/bin/env bash
# Exercise image scanning with local tool stubs; no Docker daemon or network needed.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SRC_SCRIPTS="$(cd "${SCRIPT_DIR}/../src/scripts" && pwd)"
WORK=$(mktemp -d)
trap 'rm -rf "${WORK}"' EXIT
PASS=0
FAIL=0
pass() { echo "PASS"; PASS=$((PASS + 1)); }
fail() { echo "FAIL"; FAIL=$((FAIL + 1)); }
mkdir -p "${WORK}/bin" "${WORK}/home/bin" "${WORK}/repo"
cat > "${WORK}/bin/docker" <<'EOF'
#!/usr/bin/env bash
printf 'docker %s\n' "$*" >> "${CALL_LOG}"
EOF
cat > "${WORK}/home/bin/trivy" <<'EOF'
#!/usr/bin/env bash
[[ -f .trivyignore ]] || exit 99
printf 'trivy %s\n' "$*" >> "${CALL_LOG}"
# Redirect the production /tmp output path into the test workspace.
while [[ $# -gt 0 ]]; do
if [[ "$1" == --output ]]; then
printf '{"Results":[]}\n' > "${REPORT_DIR}/$(basename "$2")"
fi
shift
done
exit "${SCAN_EXIT:-0}"
EOF
chmod +x "${WORK}/bin/docker" "${WORK}/home/bin/trivy"
export PATH="${WORK}/bin:${PATH}" HOME="${WORK}/home"
export CALL_LOG="${WORK}/calls" REPORT_DIR="${WORK}"
export IMAGE_NAME=app REGISTRY=ghcr.io/example CIRCLE_SHA1=abcdef123456
export SUPPLEMENTAL_IMAGES_JSON='[{"name":"migrations","dockerfile":"Dockerfile.migrations"},{"name":"worker","base":"example/base:1"}]'
cd "${WORK}/repo"
touch .trivyignore
echo "=== PR image scan tests ==="
echo -n " TEST: feature metadata survives workspace restore and supplies Docker scan tag... "
export BASH_ENV="${WORK}/build-env" CIRCLE_BRANCH=feature/image-scan
export GITVERSION_MAJOR=1 GITVERSION_MINOR=2 GITVERSION_PATCH=3
export GITVERSION_SHA=abcdef123456
bash "${SRC_SCRIPTS}/export_version_vars.sh" > "${WORK}/version.log"
bash "${SRC_SCRIPTS}/persist_version_metadata.sh" >> "${WORK}/version.log"
export BASH_ENV="${WORK}/scan-env"
bash "${SRC_SCRIPTS}/restore_version_metadata.sh" >> "${WORK}/version.log"
bash "${SRC_SCRIPTS}/docker_build.sh" >> "${WORK}/version.log"
# shellcheck disable=SC1090
source "${BASH_ENV}"
export IMAGE_TAG='ghcr.io/example/app:${DOCKER_TAG}'
if bash "${SRC_SCRIPTS}/run_trivy_image.sh" >> "${WORK}/version.log" &&
grep -q 'trivy image .*--severity HIGH,CRITICAL --exit-code 1 ghcr.io/example/app:1.2.3-image-scan-abcdef123456-SNAPSHOT' "${CALL_LOG}"; then
pass
else
fail
fi
echo -n " TEST: push=false refreshes both Dockerfile forms and scans without publishing... "
: > "${CALL_LOG}"
echo 'export DOCKER_ENV_TAG=develop' >> "${BASH_ENV}"
if PUSH_IMAGES=false bash "${SRC_SCRIPTS}/build_push_supplemental.sh" > "${WORK}/supplemental.log" &&
[[ $(grep -c '^docker build --pull --no-cache -f ' "${CALL_LOG}") == 2 ]] &&
[[ $(grep -c '^trivy image .*--severity HIGH,CRITICAL --exit-code 1 ' "${CALL_LOG}") == 2 ]] &&
! grep -Eq '^docker (push|login|tag|buildx) ' "${CALL_LOG}" &&
[[ -s "${WORK}/migrations.json" && -s "${WORK}/worker.json" ]]; then
pass
else
fail
fi
echo -n " TEST: default push refreshes both Dockerfile forms and scans before publishing tags... "
: > "${CALL_LOG}"
unset PUSH_IMAGES
if bash "${SRC_SCRIPTS}/build_push_supplemental.sh" > "${WORK}/supplemental.log" &&
[[ $(grep -c '^docker build --pull --no-cache -f ' "${CALL_LOG}") == 2 ]] &&
[[ $(grep -c '^docker push ' "${CALL_LOG}") == 4 ]] &&
awk '
/^trivy image / { image=$NF; sub(/:[^:]+$/, "", image); scanned[image]=1 }
/^docker push / { image=$NF; sub(/:[^:]+$/, "", image); if (!scanned[image]) exit 1 }
' "${CALL_LOG}"; then
pass
else
fail
fi
# Regression (0.3.5): CircleCI renders the boolean "push" parameter as "1"/"0" in
# the step environment, and release builds silently skipped the push.
for push in 1 true TRUE; do
echo -n " TEST: push=${push} (release builds) publishes every supplemental image... "
: > "${CALL_LOG}"
if PUSH_IMAGES="${push}" bash "${SRC_SCRIPTS}/build_push_supplemental.sh" > "${WORK}/supplemental.log" &&
[[ $(grep -c '^docker push ' "${CALL_LOG}") == 4 ]] &&
grep -q '^docker push ghcr.io/example/app-migrations:' "${CALL_LOG}" &&
grep -q '^docker push ghcr.io/example/app-worker:' "${CALL_LOG}" &&
grep -q 'push=true' "${WORK}/supplemental.log"; then
pass
else
fail
fi
done
for push in 0 false ""; do
echo -n " TEST: push='${push}' (PR scans) publishes nothing... "
: > "${CALL_LOG}"
if PUSH_IMAGES="${push}" bash "${SRC_SCRIPTS}/build_push_supplemental.sh" > "${WORK}/supplemental.log" &&
! grep -Eq '^docker (push|tag) ' "${CALL_LOG}"; then
pass
else
fail
fi
done
echo -n " TEST: an unrecognised push value fails instead of guessing... "
: > "${CALL_LOG}"
if PUSH_IMAGES=maybe bash "${SRC_SCRIPTS}/build_push_supplemental.sh" > "${WORK}/supplemental.log" 2>&1; then
fail
elif ! grep -Eq '^docker (build|tag|push) ' "${CALL_LOG}" && grep -q "must be true/false" "${WORK}/supplemental.log"; then
pass
else
fail
fi
for push in false true; do
echo -n " TEST: push=${push} fails on HIGH/CRITICAL findings before any push... "
: > "${CALL_LOG}"
rm -f "${WORK}/migrations.json"
if SCAN_EXIT=1 PUSH_IMAGES="${push}" bash "${SRC_SCRIPTS}/build_push_supplemental.sh" > "${WORK}/supplemental.log" 2>&1; then
fail
elif ! grep -q '^docker push ' "${CALL_LOG}" && [[ -s "${WORK}/migrations.json" ]]; then
pass
else
fail
fi
done
echo "Results: ${PASS} passed, ${FAIL} failed"
[[ ${FAIL} -eq 0 ]]
+44 -10
View File
@@ -8,8 +8,8 @@ PASS=0
FAIL=0
GENERATE_SCRIPT="${PROJECT_DIR}/src/scripts/generate_config.sh"
GENERATED_CONFIG=$(mktemp)
trap 'rm -f "${GENERATED_CONFIG}"' EXIT
GENERATED_DIR=$(mktemp -d)
trap 'rm -rf "${GENERATED_DIR}"' EXIT
# --------------------------------------------------------------------------
# Helper: render a fixture by calling the real generate_config.sh script
@@ -17,11 +17,15 @@ trap 'rm -f "${GENERATED_CONFIG}"' EXIT
# --------------------------------------------------------------------------
render_fixture() {
local fixture="$1"
local generated_config="${GENERATED_DIR}/$(basename "${fixture}")"
# Run the real script; capture stderr for diagnostics, return generated YAML
MUNITOR_CONFIG="${fixture}" MUNITOR_OUTPUT_FILE="${GENERATED_CONFIG}" \
bash "${GENERATE_SCRIPT}" >/dev/null 2>&1
cat "${GENERATED_CONFIG}"
# Fixtures are immutable and uniquely named: render once, assert many times.
if [[ ! -f "${generated_config}" ]]; then
MUNITOR_CONFIG="${fixture}" MUNITOR_OUTPUT_FILE="${generated_config}.tmp" \
bash "${GENERATE_SCRIPT}" >/dev/null 2>&1 || return 1
mv "${generated_config}.tmp" "${generated_config}"
fi
cat "${generated_config}"
}
run_test() {
@@ -157,6 +161,36 @@ run_negative_context_test() {
echo "=== Template Rendering Tests ==="
# Container scans belong only to PR jobs; RC already builds and scans before release.
for module in java-webapp gradle-webapp node-api node-webapp; do
fixture="${FIXTURES_DIR}/${module}.munitor.yml"
run_context_test "${module}: PR builds and scans the image" \
"${fixture}" "pr-checks:" " develop:" 'munitor/docker_scan:'
run_context_test "${module}: image scan is named image-scan" \
"${fixture}" "munitor/docker_scan:" "munitor/secrets_scan:" 'name: image-scan'
run_context_test "${module}: image scan waits for build-and-test" \
"${fixture}" "munitor/docker_scan:" "munitor/secrets_scan:" '\- build-and-test'
run_negative_context_test "${module}: no duplicate scan outside PR workflow" \
"${fixture}" " develop:" "END_OF_CONFIG" 'munitor/docker_scan:'
run_context_test "${module}: RC already builds and scans" \
"${fixture}" "release-candidate:" "production:" 'munitor/docker_build_push:'
echo -n " TEST: ${module}: every image scan has exactly the PR branch filters... "
output=$(render_fixture "${fixture}")
filters=$(yq -o=json -I=0 '[.workflows[].jobs[] | select(has("munitor/docker_scan")) | ."munitor/docker_scan".filters.branches.only]' <<< "${output}")
if [[ "${filters}" == '[["/feature\\/.*/","/hotfix\\/.*/"]]' ]]; then
echo "PASS"
PASS=$((PASS + 1))
else
echo "FAIL (unexpected scan filters: ${filters})"
FAIL=$((FAIL + 1))
fi
done
run_context_test "supplemental images are wired into the PR image scan" \
"${FIXTURES_DIR}/java-webapp-supplemental.munitor.yml" \
"munitor/docker_scan:" "munitor/secrets_scan:" 'supplemental_images:.*migrations'
# Test java-webapp renders with correct values
run_test "java-webapp: renders image_name" \
"${FIXTURES_DIR}/java-webapp.munitor.yml" \
@@ -642,17 +676,17 @@ run_negative_context_test "java-webapp: production has no security-scan" \
"security_scan"
# java-webapp: release-candidate has quality gates
run_context_test "java-webapp: release-candidate has code-quality" \
run_context_test "java-webapp: release-candidate enables quality checks" \
"${FIXTURES_DIR}/java-webapp.munitor.yml" \
"release-candidate:" \
"production:" \
"code-quality"
"quality_checks: true"
run_context_test "java-webapp: release-candidate has coverage" \
run_context_test "java-webapp: release-candidate forwards coverage threshold" \
"${FIXTURES_DIR}/java-webapp.munitor.yml" \
"release-candidate:" \
"production:" \
"mvn_coverage"
'min_instruction: "80"'
run_context_test "java-webapp: release-candidate has security-scan" \
"${FIXTURES_DIR}/java-webapp.munitor.yml" \
+5
View File
@@ -0,0 +1,5 @@
#!/usr/bin/env bash
# Protect required CI contexts and publication gates when consolidating workflows.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
yq -o=json '.' "${SCRIPT_DIR}/../.circleci/config.yml" | node "${SCRIPT_DIR}/own_ci_regression.cjs"
+93
View File
@@ -0,0 +1,93 @@
#!/usr/bin/env bash
# The filesystem Trivy scan and the Semgrep scan must fail when they find
# something, and Semgrep must fail when it ran no rules. Local tool stubs; no
# network needed.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
SRC_SCRIPTS="$(cd "${SCRIPT_DIR}/../src/scripts" && pwd)"
WORK=$(mktemp -d)
trap 'rm -rf "${WORK}"' EXIT
PASS=0
FAIL=0
pass() { echo "PASS"; PASS=$((PASS + 1)); }
fail() { echo "FAIL"; FAIL=$((FAIL + 1)); }
mkdir -p "${WORK}/home/bin" "${WORK}/bin" "${WORK}/repo"
cat > "${WORK}/home/bin/trivy" <<'EOF'
#!/usr/bin/env bash
printf 'trivy %s\n' "$*" >> "${CALL_LOG}"
# Real trivy exits 0 on findings unless --exit-code is given.
for a in "$@"; do [[ "$a" == --exit-code ]] && exit "${SCAN_EXIT:-0}"; done
exit 0
EOF
cat > "${WORK}/bin/semgrep" <<'EOF'
#!/usr/bin/env bash
printf '%s\n' "${SEMGREP_FAKE_REPORT}" > "${SEMGREP_REPORT}"
exit "${SEMGREP_FAKE_EXIT:-0}"
EOF
chmod +x "${WORK}/home/bin/trivy" "${WORK}/bin/semgrep"
export PATH="${WORK}/bin:${PATH}" HOME="${WORK}/home" CALL_LOG="${WORK}/calls"
export SEMGREP_REPORT="${WORK}/semgrep.json"
cd "${WORK}/repo"
echo "=== Scan gate tests ==="
echo -n " TEST: trivy fs fails the job on HIGH/CRITICAL findings... "
if ! SCAN_EXIT=1 bash "${SRC_SCRIPTS}/run_trivy.sh" > "${WORK}/trivy.log" 2>&1 &&
grep -q '^trivy fs .*--severity HIGH,CRITICAL --exit-code 1 ' "${CALL_LOG}"; then
pass
else
fail
fi
echo -n " TEST: trivy fs passes a clean scan... "
if SCAN_EXIT=0 bash "${SRC_SCRIPTS}/run_trivy.sh" > "${WORK}/trivy.log" 2>&1; then
pass
else
fail
fi
if command -v python3 &>/dev/null; then
CLEAN='{"results":[],"errors":[],"paths":{"scanned":["a.java","b.java"]}}'
FOUND='{"results":[{"check_id":"java.lang.security.x","path":"a.java","start":{"line":3}}],"errors":[],"paths":{"scanned":["a.java"]}}'
NORULES='{"results":[],"errors":[{"type":"InvalidRuleSchemaError","message":"invalid configuration file found (1 configs were invalid)"}],"paths":{"scanned":[]}}'
echo -n " TEST: semgrep passes a clean scan... "
if SEMGREP_FAKE_REPORT="${CLEAN}" bash "${SRC_SCRIPTS}/run_semgrep.sh" > "${WORK}/sg.log" 2>&1 &&
grep -q '2 files scanned, 0 findings' "${WORK}/sg.log"; then
pass
else
fail
fi
echo -n " TEST: semgrep fails on findings and lists them... "
if ! SEMGREP_FAKE_REPORT="${FOUND}" SEMGREP_FAKE_EXIT=1 bash "${SRC_SCRIPTS}/run_semgrep.sh" > "${WORK}/sg.log" 2>&1 &&
grep -q 'java.lang.security.x a.java:3' "${WORK}/sg.log"; then
pass
else
fail
fi
echo -n " TEST: semgrep warn-only mode passes with findings... "
if SEMGREP_FAIL_ON_FINDINGS=false SEMGREP_FAKE_REPORT="${FOUND}" SEMGREP_FAKE_EXIT=1 \
bash "${SRC_SCRIPTS}/run_semgrep.sh" > "${WORK}/sg.log" 2>&1; then
pass
else
fail
fi
echo -n " TEST: semgrep fails when no rules ran, even with exit 0 and warn-only... "
if ! SEMGREP_FAIL_ON_FINDINGS=false SEMGREP_FAKE_REPORT="${NORULES}" SEMGREP_FAKE_EXIT=0 \
bash "${SRC_SCRIPTS}/run_semgrep.sh" > "${WORK}/sg.log" 2>&1 &&
grep -q 'did not run its rules' "${WORK}/sg.log"; then
pass
else
fail
fi
else
echo " SKIP: semgrep gate tests need python3"
fi
echo ""
echo "Results: ${PASS} passed, ${FAIL} failed"
[[ "${FAIL}" -eq 0 ]]
+3
View File
@@ -0,0 +1,3 @@
#!/usr/bin/env bash
set -euo pipefail
bats "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/security_scan_owasp.bats"
+11
View File
@@ -0,0 +1,11 @@
#!/usr/bin/env bats
@test "combined script and template targets run each suite once" {
mkdir -p "$BATS_TEST_TMPDIR/tests"
printf 'echo SCRIPT_EXECUTED\n' > "$BATS_TEST_TMPDIR/tests/test_example.sh"
printf 'echo TEMPLATE_EXECUTED\n' > "$BATS_TEST_TMPDIR/tests/test_generate_config.sh"
run make --no-print-directory -f "$BATS_TEST_DIRNAME/../Makefile" -C "$BATS_TEST_TMPDIR" test-scripts test-templates
[ "$status" -eq 0 ]
[ "$(echo "$output" | grep -c '^SCRIPT_EXECUTED$')" -eq 1 ]
[ "$(echo "$output" | grep -c '^TEMPLATE_EXECUTED$')" -eq 1 ]
}