Compare commits

...
Author SHA1 Message Date
james.maesandCodex 2106955040 feat: promote managed patch tracks [minor]
Co-Authored-By: Codex <noreply@openai.com>
2026-10-21 15:07:28 -05:00
james.maesandCodex 02f0b93e41 fix: promote Ubuntu-only Java bootstrap (#79)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 14:31:55 -05:00
james.maesandCodex 5e433ec629 fix: refresh only the newly added Temurin repository (#77)
Co-authored-by: Codex <noreply@openai.com>
2026-10-21 13:59:00 -05:00
james.maesandCodex cec7a76f12 feat: promote central QBit publication and workflow contracts (#74)
* ci: promote shared QBit publication and maintenance

Co-Authored-By: Codex <noreply@openai.com>

* fix: include verified Maven publication coordinates

Co-Authored-By: Codex <noreply@openai.com>

---------

Co-authored-by: Codex <noreply@openai.com>
2026-10-21 13:19:23 -05:00
22 changed files with 516 additions and 32 deletions

No files matched your search

+4
View File
@@ -31,6 +31,9 @@ jobs:
- run:
name: Verify resolved multi-module publication POMs
command: python3 tests/test_maven_publication_poms.py
- run:
name: Verify calculated versions match installed Maven coordinates
command: python3 tests/test_effective_publication_version.py
pipeline-contract-test:
resource_class: small
docker:
@@ -50,6 +53,7 @@ jobs:
python3 tests/test_maven_ci_contract.py
python3 tests/test_standardize_qbit.py
python3 tests/test_qbit_hotfix_profile.py
python3 tests/test_qbit_feature_profile.py
python3 tests/test_ci_maintenance.py
python3 tests/test_rollout_version.py
python3 tests/test_validate_standard.py
+1
View File
@@ -220,3 +220,4 @@ publish-release: check-clean check-branch lint clean dev test-all
# an isolated temporary repository. Never publishes or uses signing credentials.
test-publication-poms:
python3 tests/test_maven_publication_poms.py
python3 tests/test_effective_publication_version.py
+7
View File
@@ -4,6 +4,8 @@ CircleCI orb for QQQ framework projects.
For centralized pipeline ownership, see [standardized manifest-driven pipelines](docs/GENERATED-PIPELINES.md).
The initial QBit profile uses a small `.qqq-ci.yml` and a setup entry point; existing direct job callers remain supported.
For centrally managed patch adoption and exact-version rollback, see
[managed patch updates](docs/managed-patch-updates.md).
**For:** Teams using CircleCI to build, test, and publish QQQ applications
**Status:** Stable (v0.6.x)
@@ -87,6 +89,11 @@ real publication, not a dry run, verifies the trust relationship.
| `hotfix/1.5.1` | `1.5.1` |
| `feature/*` | `1.5.0-feature-abc1234-SNAPSHOT` |
The Maven version step verifies the effective `project.version`, not only the
`revision` property. For older POMs with a literal project version, it uses
Maven's version plugin to align project and reactor coordinates with the
calculated version. A mismatch after updating fails before publication.
### Java Version
Maven jobs run on Ubuntu machine executors with Temurin JDK. Default is Java 21. To use Java 17:
+4 -2
View File
@@ -111,8 +111,10 @@ The independent baseline fixture pins job sets, dependency edges, contexts,
coverage requirements and actual verification commands across all six workflows.
Core additionally supports `publish.feature` and `publish.hotfix` (both true by
default), preserving `publish*` tag and hotfix branch publication. QBit does not
accept these keys. Snapshot, RC and GA behavior remains distinct; RC/GA/hotfix
default), preserving `publish*` tag and hotfix branch publication. QBit supports
both keys as explicit opt-ins (false by default). QBit feature publishing requires
a `publish*` tag resolving to the checked-out commit; feature branch pushes remain
test-only. Snapshot, RC and GA behavior remains distinct; RC/GA/hotfix
publication still requires the complete source coverage gate. Cold bootstrap and
fresh consumer Maven repositories remain uncached. Shared installer policy replaces
local package preparation; preliminary compilation is disabled before clean
+11 -1
View File
@@ -37,9 +37,17 @@ fail migration rather than silently changing release behavior.
Use `tools/standardize_qbit.py --write` for the two-file migration. API callers
must pass the inferred policy to
`canonical_caller(version, manifest["release_tag_policy"])`; the legacy one-argument
`canonical_caller(version, manifest["release_tag_policy"], manifest["publish"].get("feature", False))`; the legacy one-argument
form is only for manifests without an explicit policy.
`publish.feature` is false by default. Opting in enables the audited `publish*`
tag contract, with the tag required to resolve to the checked-out commit. The
canonical setup admits those tags alongside the declared release tag policy.
Feature branch pushes remain test-only. Migration accepts Workflows' exact legacy
literal `feature/.*` branch selector (not a regular expression); Git forbids `*`
in branch names, so that selector cannot publish a branch. Unknown feature
filters require a central profile change rather than silently broadening triggers.
Focused tests cover the immutable inventory, unknown-filter rejection, canonical
caller agreement, main opt-in, broad/stable tag selection, HEAD provenance,
publication opt-out and test exclusions. The packed generator suite also exercises
@@ -51,3 +59,5 @@ CMS's original YAML is retained with its immutable source SHA: its unquoted
stable-tag regex is invalid YAML. The fixture repairs only that quoting. Both
audited spellings of the stable regex map to the same ASCII-digit tag policy;
arbitrary regexes remain rejected.
CI-only commits on publishing branches retain Maven verification without publishing when their changes are limited to the manifest, CircleCI caller, shared scanner caller, RC caller, or schema-drift caller. The commit must have one parent and a conventional ci title. Tags, application POMs, schema resources, tests, and other application inputs retain normal publication behavior. Canonical caller enforcement separately verifies the shared workflow definition and immutable worker pin.
+37
View File
@@ -0,0 +1,37 @@
# Managed patch updates
The canonical caller and manifest accept either an exact stable version such as
`0.8.0` or a quoted minor track such as `"0.8"`. CircleCI resolves a minor track
to its latest published patch. A major-only version, `latest`, `volatile`, and
wildcards are rejected. The caller import, setup parameter, and manifest must
still agree exactly.
Use a minor track after qualifying that release line. Future patch fixes then
arrive on the repository's next pipeline without a version-bump PR or an extra
fleet-wide build. Existing exact pins remain supported and are not changed by
publishing this capability. YAML minor versions must be quoted to remain strings.
The central batch tool applies the same update to all supplied checkouts:
```sh
python3 tools/rollout_version.py --orb-version 0.8 --write /path/to/repo1 /path/to/repo2
```
This changes only the two canonical CI declarations. Publication policies,
contexts, branch/tag filters, and profile options stay unchanged. Normal source
and publication checks still run when a pipeline is triggered; this setting does
not skip tests. GitHub reusable-worker commit pins are separate and unchanged.
Automatic patch adoption trades exact-version reproducibility for centrally
managed fixes. Keep patch releases backward compatible, qualify central changes
before production publication, and use an exact stable pin for a canary or
rollback. For example, the same tool can move a consumer from `0.8` back to
`0.8.0`. Historical pipeline reruns can resolve a newer patch; choose an exact
pin when reproducing an earlier run. Setup and continuation each resolve their
orb imports, so patch compatibility also matters when publication overlaps a run.
Changing minor tracks is an explicit migration. Do not use this feature to push
breaking changes into a patch release or to bypass repository checks.
Reference: [CircleCI orb semantic versioning](https://circleci.com/docs/orbs/use/orb-concepts/).
+24 -1
View File
@@ -161,7 +161,7 @@ set_version() {
##############################################
## Use Maven versions plugin to set version ##
##############################################
mvn versions:set-property -Dproperty=revision -DnewVersion="$new_version" -DgenerateBackupPoms=false
mvn -f "$POM_FILE" versions:set-property -Dproperty=revision -DnewVersion="$new_version" -DgenerateBackupPoms=false
#######################
## Verify the change ##
@@ -175,6 +175,26 @@ set_version() {
fi
}
ensure_project_version() {
local expected=$1 actual
# A revision property can exist without being used by project.version.
# Ask Maven for the effective coordinate, including inherited configuration.
actual=$(mvn -f "$POM_FILE" -q -Dstyle.color=never \
org.apache.maven.plugins:maven-help-plugin:3.5.1:evaluate \
-Dexpression=project.version -DforceStdout)
if [[ "$actual" != "$expected" ]]; then
mvn -f "$POM_FILE" -B versions:set -DnewVersion="$expected" -DgenerateBackupPoms=false
actual=$(mvn -f "$POM_FILE" -q -Dstyle.color=never \
org.apache.maven.plugins:maven-help-plugin:3.5.1:evaluate \
-Dexpression=project.version -DforceStdout)
fi
if [[ "$actual" != "$expected" ]]; then
echo "ERROR: Effective project version is $actual; expected $expected" >&2
exit 1
fi
echo "Verified effective project version: $actual"
}
#############################################
## Branch Detection and Routing Functions ##
#############################################
@@ -434,6 +454,9 @@ main() {
fi
echo ""
if [[ "$DRY_RUN" != "true" ]]; then
ensure_project_version "$NEW_VERSION"
fi
echo "=== Version calculation complete ==="
}
+11 -5
View File
@@ -89,8 +89,8 @@ def load_manifest(path, setup_version):
allowed += ("java_version",)
mapping(data, allowed, "manifest")
version = data.get("orb_version")
if not isinstance(version, str) or not re.fullmatch(r"(?:\d+\.\d+\.\d+|dev:[A-Za-z0-9_-]+)", version):
raise ManifestError("orb_version must be an explicit stable or development version")
if not isinstance(version, str) or not re.fullmatch(r"(?:\d+\.\d+(?:\.\d+)?|dev:[A-Za-z0-9_-]+)", version):
raise ManifestError("orb_version must be a quoted minor track, exact stable pin, or development version")
if version != setup_version:
raise ManifestError("Manifest orb_version must match the setup job orb_version")
default_branch = data.get("default_branch")
@@ -112,7 +112,7 @@ def load_manifest(path, setup_version):
if profile in ("qbit", "core"):
publish_keys += ("asciidoc",)
if profile == "qbit":
publish_keys += ("hotfix",)
publish_keys += ("hotfix", "feature")
if profile not in ("qbit", "cli", "all", "app-starter", "voyage", "frontend-next"):
publish_keys += ("feature", "hotfix")
publishing = mapping(data.get("publish", {}), publish_keys, "publish")
@@ -141,7 +141,7 @@ def load_manifest(path, setup_version):
"docs_only": boolean(data, "docs_only", False),
"publish_context": publish_context,
"publish": {key: boolean(publishing, key, key != "asciidoc" and profile not in ("cli", "all", "repository-policy")
and not (profile == "qbit" and key == "hotfix"))
and not (profile == "qbit" and key in ("hotfix", "feature")))
for key in publish_keys}}
@@ -191,7 +191,7 @@ def proven_ci_maintenance(branch, tag):
if not changed:
return False
allowed = {b".qqq-ci.yml", b".circleci/config.yml", b".github/workflows/codeql.yml",
b".github/workflows/release-rc.yml"}
b".github/workflows/release-rc.yml", b".github/workflows/schema-drift.yml"}
return all(path in allowed for path in changed.rstrip(b"\0").split(b"\0"))
@@ -215,6 +215,12 @@ def event_kind(branch, tag, release_on_main=False, release_tag_policy="stable"):
def qbit_template(config, branch, tag):
"""Profile-owned template; add future profiles as reviewed renderers, not raw YAML."""
kind = event_kind(branch, tag, config.get("release_on_main", False), config.get("release_tag_policy", "stable"))
if tag.startswith("publish") and config["publish"].get("feature", False):
tagged = git("rev-parse", "--verify", f"refs/tags/{tag}^{{commit}}")
if not tagged and git("fetch", "--no-tags", "origin", f"refs/tags/{tag}:refs/tags/{tag}") is not None:
tagged = git("rev-parse", "--verify", f"refs/tags/{tag}^{{commit}}")
if tagged and tagged == git("rev-parse", "HEAD"):
kind = "feature"
if not tag and re.fullmatch(r"hotfix/.+", branch) and config["publish"].get("hotfix", False):
kind = "hotfix"
if kind != "test" and not config["publish"][kind]:
+22 -4
View File
@@ -29,8 +29,21 @@ if [[ -f /etc/apt/sources.list ]]; then
sudo sed -i 's|http://security.ubuntu.com/ubuntu|https://archive.ubuntu.com/ubuntu|g' /etc/apt/sources.list
fi
# Add Eclipse Temurin repository
sudo env DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=l apt-get update -qq
# Base tools come from Ubuntu, not the unrelated PPAs bundled in machine images.
# Support both the 22.04 list and the newer deb822 Ubuntu source file. Preserve
# other indexes for later jobs without contacting their repositories here.
ubuntu_sources=/etc/apt/sources.list
if [[ ! -s "$ubuntu_sources" ]]; then
ubuntu_sources=/etc/apt/sources.list.d/ubuntu.sources
fi
if [[ ! -s "$ubuntu_sources" ]]; then
echo 'Cannot locate Ubuntu package sources' >&2
exit 1
fi
sudo env DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=l apt-get update -qq \
-o "Dir::Etc::sourcelist=$ubuntu_sources" \
-o Dir::Etc::sourceparts=- \
-o APT::Get::List-Cleanup=0
sudo env DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=l apt-get install -y -qq wget apt-transport-https gpg maven curl
# Add Temurin GPG key
@@ -39,8 +52,13 @@ wget -qO - https://packages.adoptium.net/artifactory/api/gpg/key/public | sudo g
# Add repository
echo "deb [signed-by=/usr/share/keyrings/adoptium.gpg] https://packages.adoptium.net/artifactory/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/adoptium.list
# Install Java
sudo env DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=l apt-get update -qq
# Refresh only the repository just added. Re-fetching unrelated image PPAs here
# caused otherwise successful setup to fail on Launchpad 503 responses. Keep
# Ubuntu indexes from the first refresh so Java dependencies remain resolvable.
sudo env DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=l apt-get update -qq \
-o Dir::Etc::sourcelist=/etc/apt/sources.list.d/adoptium.list \
-o Dir::Etc::sourceparts=- \
-o APT::Get::List-Cleanup=0
sudo env DEBIAN_FRONTEND=noninteractive NEEDRESTART_MODE=l apt-get install -y -qq "temurin-${JAVA_VERSION}-jdk"
# Set JAVA_HOME
+11 -5
View File
@@ -89,8 +89,8 @@ def load_manifest(path, setup_version):
allowed += ("java_version",)
mapping(data, allowed, "manifest")
version = data.get("orb_version")
if not isinstance(version, str) or not re.fullmatch(r"(?:\d+\.\d+\.\d+|dev:[A-Za-z0-9_-]+)", version):
raise ManifestError("orb_version must be an explicit stable or development version")
if not isinstance(version, str) or not re.fullmatch(r"(?:\d+\.\d+(?:\.\d+)?|dev:[A-Za-z0-9_-]+)", version):
raise ManifestError("orb_version must be a quoted minor track, exact stable pin, or development version")
if version != setup_version:
raise ManifestError("Manifest orb_version must match the setup job orb_version")
default_branch = data.get("default_branch")
@@ -112,7 +112,7 @@ def load_manifest(path, setup_version):
if profile in ("qbit", "core"):
publish_keys += ("asciidoc",)
if profile == "qbit":
publish_keys += ("hotfix",)
publish_keys += ("hotfix", "feature")
if profile not in ("qbit", "cli", "all", "app-starter", "voyage", "frontend-next"):
publish_keys += ("feature", "hotfix")
publishing = mapping(data.get("publish", {}), publish_keys, "publish")
@@ -141,7 +141,7 @@ def load_manifest(path, setup_version):
"docs_only": boolean(data, "docs_only", False),
"publish_context": publish_context,
"publish": {key: boolean(publishing, key, key != "asciidoc" and profile not in ("cli", "all", "repository-policy")
and not (profile == "qbit" and key == "hotfix"))
and not (profile == "qbit" and key in ("hotfix", "feature")))
for key in publish_keys}}
@@ -191,7 +191,7 @@ def proven_ci_maintenance(branch, tag):
if not changed:
return False
allowed = {b".qqq-ci.yml", b".circleci/config.yml", b".github/workflows/codeql.yml",
b".github/workflows/release-rc.yml"}
b".github/workflows/release-rc.yml", b".github/workflows/schema-drift.yml"}
return all(path in allowed for path in changed.rstrip(b"\0").split(b"\0"))
@@ -215,6 +215,12 @@ def event_kind(branch, tag, release_on_main=False, release_tag_policy="stable"):
def qbit_template(config, branch, tag):
"""Profile-owned template; add future profiles as reviewed renderers, not raw YAML."""
kind = event_kind(branch, tag, config.get("release_on_main", False), config.get("release_tag_policy", "stable"))
if tag.startswith("publish") and config["publish"].get("feature", False):
tagged = git("rev-parse", "--verify", f"refs/tags/{tag}^{{commit}}")
if not tagged and git("fetch", "--no-tags", "origin", f"refs/tags/{tag}:refs/tags/{tag}") is not None:
tagged = git("rev-parse", "--verify", f"refs/tags/{tag}^{{commit}}")
if tagged and tagged == git("rev-parse", "HEAD"):
kind = "feature"
if not tag and re.fullmatch(r"hotfix/.+", branch) and config["publish"].get("hotfix", False):
kind = "hotfix"
if kind != "test" and not config["publish"][kind]:
+11 -5
View File
@@ -59,7 +59,7 @@ def load_yaml(raw):
raise StandardError("Invalid YAML declaration") from error
def canonical_caller(profile, version, explicit_manifest=False, release_tag_policy=None):
def canonical_caller(profile, version, explicit_manifest=False, release_tag_policy=None, publish_feature=False):
if not isinstance(profile, str) or profile not in PROFILE_PARAMETERS:
raise StandardError("Unsupported CI profile")
options = {
@@ -73,6 +73,9 @@ def canonical_caller(profile, version, explicit_manifest=False, release_tag_poli
if release_tag_policy not in tag_filters:
raise StandardError("Unsupported QBit release tag policy")
options["filters"]["tags"] = tag_filters[release_tag_policy]
if publish_feature:
options["filters"]["tags"] = {"only": ([tag_filters[release_tag_policy]["only"]]
if release_tag_policy != "none" else []) + ["/publish.*/"]}
if explicit_manifest:
options["manifest"] = ".qqq-ci.yml"
parameters = PROFILE_PARAMETERS[profile]
@@ -106,12 +109,15 @@ def validate(config_raw, manifest_raw, setup_version):
if not isinstance(manifest, dict):
raise StandardError("Manifest must be a mapping")
version = manifest.get("orb_version")
if not isinstance(version, str) or not re.fullmatch(r"(?:\d+\.\d+\.\d+|dev:[A-Za-z0-9_-]+)", version):
raise StandardError("Manifest must pin an explicit orb version")
if not isinstance(version, str) or not re.fullmatch(r"(?:\d+\.\d+(?:\.\d+)?|dev:[A-Za-z0-9_-]+)", version):
raise StandardError("Manifest must select a quoted minor track, exact stable pin, or development version")
if version != setup_version:
raise StandardError("Manifest and running setup orb versions differ")
expected = canonical_caller(manifest.get("profile"), version, release_tag_policy=manifest.get("release_tag_policy"))
explicit = canonical_caller(manifest.get("profile"), version, explicit_manifest=True, release_tag_policy=manifest.get("release_tag_policy"))
publishing = manifest.get("publish", {})
if not isinstance(publishing, dict) or type(publishing.get("feature", False)) is not bool:
raise StandardError("Feature publication must be an explicit boolean")
expected = canonical_caller(manifest.get("profile"), version, release_tag_policy=manifest.get("release_tag_policy"), publish_feature=publishing.get("feature", False))
explicit = canonical_caller(manifest.get("profile"), version, explicit_manifest=True, release_tag_policy=manifest.get("release_tag_policy"), publish_feature=publishing.get("feature", False))
if not exact(config, expected) and not exact(config, explicit):
raise StandardError("Consumer config must be the canonical setup caller with only its profile's declared and forwarded parameters")
+90
View File
@@ -0,0 +1,90 @@
version: 2.1
orbs:
qqq-orb: kingsrook/qqq-orb@0.6.8
commands:
configure-package-install:
steps:
- run:
name: Keep package installation unattended
command: |
# These build machines are discarded after the job.
sudo install -d /etc/needrestart/conf.d
printf '%s\n' '$nrconf{restart} = "l";' | sudo tee /etc/needrestart/conf.d/99-qqq-ci.conf >/dev/null
workflows:
test_only:
jobs:
- qqq-orb/mvn_test_only:
pre-steps:
- configure-package-install
context: [qqq-maven-registry-credentials]
filters:
branches:
ignore: /(develop|main|release/.*|hotfix/.*|integration.*)/
tags:
only: []
publish_snapshot:
jobs:
- qqq-orb/mvn_publish:
pre-steps:
- configure-package-install
context: [qqq-maven-registry-credentials]
branch_type: snapshot
filters:
branches:
only:
- develop
publish_feature:
jobs:
- qqq-orb/mvn_publish:
pre-steps:
- configure-package-install
context: [qqq-maven-registry-credentials]
branch_type: feature
filters:
branches:
only:
- feature/.*
tags:
only: [/publish.*/]
publish_release_candidate:
jobs:
- qqq-orb/mvn_publish:
pre-steps:
- configure-package-install
context: [qqq-maven-registry-credentials]
branch_type: release_candidate
filters:
branches:
only:
- /release\/.*/
publish_release:
jobs:
- qqq-orb/mvn_publish:
pre-steps:
- configure-package-install
context: [qqq-maven-registry-credentials]
branch_type: release
filters:
branches:
ignore: /.*/
tags:
only: [/^v\d+\.\d+\.\d+$/]
publish_hotfix_release:
jobs:
- qqq-orb/mvn_publish:
pre-steps:
- configure-package-install
context: [qqq-maven-registry-credentials]
branch_type: hotfix
filters:
branches:
only:
- /hotfix\/.*/
+22 -1
View File
@@ -387,10 +387,31 @@ class AuditTests(unittest.TestCase):
audit.audit(other, Snapshot({}, checks=[]), VERSION, SHA, policy)
def test_invalid_managed_branches_cannot_bypass_policy(self):
for managed in [[], ['main', 'main'], ['feature'], 'main', None, [True]]:
for managed in [[], ['main', 'main'], ['feature'], 'main', None, [True],
['release/*'], ['release/1.2/extra'], ['hotfix/1..2'],
['release/1.2', 'release/1.2']]:
with self.subTest(managed=managed), self.assertRaises(audit.ReadError):
self.native(policy={'managed_branches': managed, 'required_checks': {}})
def test_release_and_hotfix_enforcement_requires_actual_checks(self):
files = {'.github/workflows/build.yml': audit.qctl.generate(SHA)}
for branch in ['release/0.1', 'release/4.1.0', 'hotfix/4.0.1']:
with self.subTest(branch=branch):
snapshot = Snapshot(files)
snapshot.branch = branch
policy = {'managed_branches': [branch],
'required_checks': {branch: ['native / test']}}
with patch.object(snapshot, 'checks', wraps=snapshot.checks) as read:
report = audit.audit(ENTRY, snapshot, VERSION, SHA, policy)
self.assertEqual('pass', report['status'])
read.assert_called_once_with(branch)
snapshot.required = []
report = audit.audit(ENTRY, snapshot, VERSION, SHA, policy)
self.assertEqual('missing', report['required_checks'][branch]['state'])
self.assertEqual('findings', report['status'])
policy['required_checks'] = {}
self.assertEqual('unknown', audit.audit(ENTRY, snapshot, VERSION, SHA, policy)['status'])
def test_gh_is_explicit_get_and_never_changes_settings(self):
with patch.object(audit, 'command', return_value='{}') as command:
audit.gh('repos/QRun-IO/example')
+8
View File
@@ -40,6 +40,14 @@ class MaintenanceTests(unittest.TestCase):
with mock.patch.object(pipeline, 'git', side_effect=self.git_result):
self.assertFalse(pipeline.proven_ci_maintenance('release/0.1', ''))
def test_schema_caller_migration_is_maintenance_but_schema_inputs_are_not(self):
self.paths = b'.github/workflows/schema-drift.yml\0'
with mock.patch.object(pipeline, 'git', side_effect=self.git_result):
self.assertTrue(pipeline.proven_ci_maintenance('release/0.1', ''))
for path in (b'pom.xml\0', b'src/test/resources/schema.yml\0', b'src/test/java/SchemaDriftTest.java\0'):
self.paths = b'.github/workflows/schema-drift.yml\0' + path
self.assertFalse(pipeline.proven_ci_maintenance('release/0.1', ''))
def test_merge_root_and_unavailable_history_do_not_suppress_release(self):
for parents in (None, b'a'*40+b'\n', self.parents.rstrip()+b' '+b'c'*40+b'\n'):
self.parents = parents
@@ -0,0 +1,69 @@
#!/usr/bin/env python3
"""Exercise the real version script and Maven install; never publish remotely."""
import os
from pathlib import Path
import shutil
import subprocess
import tempfile
import xml.etree.ElementTree as ET
ROOT = Path(__file__).resolve().parents[1]
NS = {'m': 'http://maven.apache.org/POM/4.0.0'}
BASH = os.environ.get('TEST_BASH') or shutil.which('bash')
MAVEN = shutil.which('mvn') or shutil.which('mvn.cmd')
def run(args, directory, env=None):
subprocess.run(args, cwd=directory, env=env, check=True, timeout=300)
def project(body):
return '<project xmlns="http://maven.apache.org/POM/4.0.0">\n<modelVersion>4.0.0</modelVersion>\n' + body + '\n</project>\n'
def main():
assert BASH and MAVEN, 'Real Bash, Java and Maven are required'
with tempfile.TemporaryDirectory(prefix='qqq-version-') as temporary:
base = Path(temporary)
for name, version, revision, branch, expected, reactor in (
('literal', '0.4.0-SNAPSHOT', '0.4.0-RC.1', 'release/0.4', '0.4.0-RC.2', False),
('friendly', '${revision}', '0.4.0-RC.1', 'release/0.4', '0.4.0-RC.2', False),
('reactor', '0.4.0-SNAPSHOT', '0.4.0-RC.1', 'release/0.4', '0.4.0-RC.2', True),
('unchanged', '0.3.0-SNAPSHOT', '0.4.0-SNAPSHOT', 'develop', '0.4.0-SNAPSHOT', False),
):
root = base / name
root.mkdir()
modules = '<modules><module>child</module></modules>' if reactor else ''
(root / 'pom.xml').write_text(project(f'''<groupId>example.version</groupId>
<artifactId>{name}</artifactId><version>{version}</version><packaging>pom</packaging>
<properties>
<revision>{revision}</revision>
</properties>
{modules}'''), encoding='utf-8')
if reactor:
(root / 'child').mkdir()
(root / 'child/pom.xml').write_text(project(f'''<parent><groupId>example.version</groupId>
<artifactId>{name}</artifactId><version>{version}</version></parent>
<artifactId>child</artifactId><packaging>pom</packaging>'''), encoding='utf-8')
run(['git', 'init', '-q'], root)
env = dict(os.environ, CIRCLE_BRANCH=branch, CIRCLE_TAG='')
run([BASH, str(ROOT / 'src/scripts/calculate_version.sh')], root, env)
if name == 'friendly':
assert ET.parse(root / 'pom.xml').findtext('m:version', namespaces=NS) == '${revision}'
# Install to a temporary repository so the actual artifact coordinate
# proves the fix, rather than merely checking the revision property.
repository = base / 'repository'
run([MAVEN, '-B', '-ntp', '-Dmaven.repo.local=' + str(repository),
'-Dflatten.mode=resolveCiFriendliesOnly', '-DupdatePomFile=true',
'org.codehaus.mojo:flatten-maven-plugin:1.7.3:flatten', 'install'], root)
for artifact in ([name, 'child'] if reactor else [name]):
installed = repository / 'example/version' / artifact / expected / f'{artifact}-{expected}.pom'
assert installed.exists(), installed
pom = ET.parse(installed)
actual = pom.findtext('m:version', namespaces=NS) or pom.findtext('m:parent/m:version', namespaces=NS)
assert actual == expected, (artifact, actual, expected)
print(f'PASS: {name} installed at {expected}', flush=True)
if __name__ == '__main__':
main()
+30
View File
@@ -215,6 +215,14 @@ class PipelineTests(unittest.TestCase):
self.assertEqual(options["filters"]["branches"], {"ignore": "automation/qbit-rc-pending"})
if PACKED:
self.assertEqual(result, self.render(packed=True, CIRCLE_BRANCH=branch))
schema = self.repo / ".github/workflows/schema-drift.yml"
schema.write_text("# central schema drift caller migration\n", encoding="utf-8")
self.git("add", ".github/workflows/schema-drift.yml")
self.git("-c", "commit.gpgsign=false", "commit", "-qm", "ci: centralize schema drift")
result = self.render(CIRCLE_BRANCH="release/0.1")
self.assert_full(result)
if PACKED:
self.assertEqual(result, self.render(packed=True, CIRCLE_BRANCH="release/0.1"))
(self.repo / "App.java").write_text("application change\n", encoding="utf-8")
self.git("add", "App.java")
self.git("-c", "commit.gpgsign=false", "commit", "-qm", "ci: misleading title")
@@ -233,6 +241,15 @@ class PipelineTests(unittest.TestCase):
self.assertTrue(options["check_middleware_api_versions"])
self.assertEqual(options["java_version"], "17")
def test_minor_track_reaches_continuation_without_changing_jobs(self):
expected = self.render()
expected['orbs']['qqq-orb'] = 'kingsrook/qqq-orb@0.7'
self.write_manifest(orb_version='0.7')
self.assertEqual(expected, self.render(QQQ_CI_ORB_VERSION='0.7'))
if PACKED:
self.assertEqual(expected, self.render(packed=True, QQQ_CI_ORB_VERSION='0.7'))
self.render(ok=False, QQQ_CI_ORB_VERSION='0.8')
def test_branch_matrix(self):
for branch, workflow in [("develop", "publish_snapshot"), ("release/1.0", "publish_release_candidate"),
("main", "test_only"), ("hotfix/fix", "test_only"),
@@ -260,6 +277,19 @@ class PipelineTests(unittest.TestCase):
self.write_manifest(publish={"hotfix": "true"})
self.render(packed=packed, CIRCLE_BRANCH="hotfix/1.2.3", ok=False)
def test_qbit_feature_tags_source_and_packed_require_opt_in_and_head(self):
self.git("tag", "publish-test")
for packed in ([False, True] if PACKED else [False]):
self.write_manifest()
self.assert_full(self.render(packed=packed, CIRCLE_BRANCH="", CIRCLE_TAG="publish-test"))
self.write_manifest(publish={"feature": True})
self.assert_full(self.render(packed=packed, CIRCLE_BRANCH="", CIRCLE_TAG="publish-test"), "publish_feature")
self.assert_full(self.render(packed=packed, CIRCLE_BRANCH="feature/example"))
self.assert_full(self.render(packed=packed, CIRCLE_BRANCH="", CIRCLE_TAG="publish-missing"))
self.commit()
for packed in ([False, True] if PACKED else [False]):
self.assert_full(self.render(packed=packed, CIRCLE_BRANCH="", CIRCLE_TAG="publish-test"))
def test_repository_policy_source_and_packed_never_select_publication(self):
self.write_manifest(profile="repository-policy")
for packed in ([False, True] if PACKED else [False]):
+37
View File
@@ -17,6 +17,24 @@ if [[ "$*" == *apt-get* ]]; then
[[ "$1" == env && "$2" == DEBIAN_FRONTEND=noninteractive && "$3" == NEEDRESTART_MODE=l ]] || exit 90
[[ -f "$APT_CONFIG_CAPTURE" ]] && grep -qx 'Acquire::Retries "3";' "$APT_CONFIG_CAPTURE" || exit 91
[[ ${APT_FAIL:-0} == 0 ]] || exit "$APT_FAIL"
if [[ "$*" == *'apt-get update'* ]]; then
updates=$(grep -c 'apt-get update' "$INSTALL_LOG")
if [[ $updates -eq 1 ]]; then
# An unavailable image PPA must not participate in base-tool setup.
[[ "$*" == *'Dir::Etc::sourceparts=-'* ]] || exit 100
[[ "$*" == *'Dir::Etc::sourcelist=/etc/apt/sources.list'* || "$*" == *'Dir::Etc::sourcelist=/etc/apt/sources.list.d/ubuntu.sources'* ]] || exit 93
[[ "$*" == *'APT::Get::List-Cleanup=0'* ]] || exit 92
[[ ${UBUNTU_UPDATE_FAIL:-0} == 0 ]] || exit "$UBUNTU_UPDATE_FAIL"
fi
if [[ $updates -eq 2 ]]; then
# Model a PPA outage after the initial base-package installation. Only
# the new signed repository may be contacted; existing indexes survive.
[[ "$*" == *'Dir::Etc::sourcelist=/etc/apt/sources.list.d/adoptium.list'* ]] || exit 100
[[ "$*" == *'Dir::Etc::sourceparts=-'* ]] || exit 100
[[ "$*" == *'APT::Get::List-Cleanup=0'* ]] || exit 92
[[ ${TEMURIN_UPDATE_FAIL:-0} == 0 ]] || exit "$TEMURIN_UPDATE_FAIL"
fi
fi
fi
if [[ "$1" == tee && "$2" == /etc/apt/apt.conf.d/80-qqq-download-retries ]]; then
cat > "$APT_CONFIG_CAPTURE"; exit 0
@@ -42,6 +60,25 @@ rm "$APT_CONFIG_CAPTURE"
PARAM_INSTALL_NODE=true bash "$script_dir/install_java.sh" >/dev/null 2>&1
[[ $(grep -c 'apt-get' "$INSTALL_LOG") -eq 5 ]]
grep -q 'apt-get install -y -qq nodejs' "$INSTALL_LOG"
# An outage of the required repository must still stop before Java installation.
: > "$INSTALL_LOG"
if UBUNTU_UPDATE_FAIL=100 PARAM_INSTALL_NODE=false bash "$script_dir/install_java.sh" >/dev/null 2>&1; then
echo 'required Ubuntu repository failure accepted' >&2; exit 1
else
[[ $? -eq 100 ]]
fi
if grep -q 'apt-get install' "$INSTALL_LOG"; then
echo 'package installation continued after Ubuntu repository failure' >&2; exit 1
fi
: > "$INSTALL_LOG"
if TEMURIN_UPDATE_FAIL=100 PARAM_INSTALL_NODE=false bash "$script_dir/install_java.sh" >/dev/null 2>&1; then
echo 'required Temurin repository failure accepted' >&2; exit 1
else
[[ $? -eq 100 ]]
fi
if grep -q 'apt-get install -y -qq temurin-' "$INSTALL_LOG"; then
echo 'Java installation continued after repository failure' >&2; exit 1
fi
# Execute the actual generated selector. Arguments containing spaces stay intact.
export SELECTOR_LOG="$scratch/selector.log"
cat > "$scratch/bin/mvn" <<'NATIVE'
+73
View File
@@ -0,0 +1,73 @@
"""Preserve Workflows' audited publish-tag contract without broadening branches."""
import unittest
from unittest import mock
from pathlib import Path
import yaml
from test_qbit_hotfix_profile import load, migration, pipeline, selects, ROOT
standard = load('standard', 'src/scripts/validate_standard.py')
class FeaturePublicationTests(unittest.TestCase):
def config(self, manifest):
with mock.patch.object(Path, 'read_text', return_value=yaml.safe_dump(manifest)):
return pipeline.load_manifest('unused', '0.7.16')
def test_audited_event_selection_matches_legacy(self):
legacy = yaml.safe_load((ROOT / 'tests/fixtures/qbit-feature-publication.yml').read_text())
manifest = migration.infer_manifest(legacy, '0.7.16', 'main')
self.assertTrue(manifest['publish']['feature'])
caller = migration.canonical_caller('0.7.16', manifest['release_tag_policy'], True)
standard.validate(yaml.safe_dump(caller), yaml.safe_dump(manifest), '0.7.16')
setup = caller['workflows']['setup']['jobs'][0]['qqq-orb/generate_pipeline']
cases = [(branch, '') for branch in ['main', 'develop', 'feature/example', 'release/0.3.0', 'hotfix/0.3.1', 'integration-test']]
cases += [('', tag) for tag in ['publish', 'publish-example', 'v1.2.3', 'v1.2.3-RC.1', 'other']]
for branch, tag in cases:
with self.subTest(branch=branch, tag=tag):
expected = sorted((job, args.get('branch_type')) for workflow in legacy['workflows'].values()
for entry in workflow['jobs'] for job, args in entry.items()
if selects(args.get('filters', {}), branch, tag))
with mock.patch.object(pipeline, 'git', return_value=b'verified-head\n'):
rendered = pipeline.qbit_template(self.config(manifest), branch, tag)
actual = sorted((job, args.get('branch_type')) for workflow in rendered['workflows'].values()
for entry in workflow['jobs'] for job, args in entry.items()
if selects(args.get('filters', {}), branch, tag))
if not selects(setup['filters'], branch, tag):
actual = []
self.assertEqual(expected, actual)
def test_feature_is_opt_in_and_requires_exact_tag_head(self):
manifest = {'profile': 'qbit', 'orb_version': '0.7.16', 'default_branch': 'main'}
self.assertFalse(self.config(manifest)['publish']['feature'])
manifest['publish'] = {'feature': True}
for tagged in [None, b'wrong-head\n']:
def git(*args):
return b'head\n' if args == ('rev-parse', 'HEAD') else tagged
with self.subTest(tagged=tagged), mock.patch.object(pipeline, 'git', side_effect=git):
rendered = pipeline.qbit_template(self.config(manifest), '', 'publish-test')
self.assertIn('qqq-orb/mvn_test_only', rendered['workflows']['test_only']['jobs'][0])
def test_feature_tag_recovery_checks_fetched_commit(self):
manifest = {'profile': 'qbit', 'orb_version': '0.7.16', 'default_branch': 'main', 'publish': {'feature': True}}
with mock.patch.object(pipeline, 'git', side_effect=[None, b'', b'head\n', b'head\n']):
rendered = pipeline.qbit_template(self.config(manifest), '', 'publish-test')
self.assertEqual('feature', rendered['workflows']['publish_feature']['jobs'][0]['qqq-orb/mvn_publish']['branch_type'])
def test_setup_preserves_release_policy_and_rejects_drift(self):
for policy in ['stable', 'v-prefix', 'none']:
manifest = {'profile': 'qbit', 'orb_version': '0.7.16', 'default_branch': 'main',
'release_tag_policy': policy, 'publish': {'feature': True}}
caller = migration.canonical_caller('0.7.16', policy, True)
standard.validate(yaml.safe_dump(caller), yaml.safe_dump(manifest), '0.7.16')
self.assertEqual(caller, standard.canonical_caller('qbit', '0.7.16', release_tag_policy=policy, publish_feature=True))
with self.assertRaises(standard.StandardError):
standard.validate(yaml.safe_dump(migration.canonical_caller('0.7.16', policy)), yaml.safe_dump(manifest), '0.7.16')
def test_unrecognized_feature_branch_filters_are_rejected(self):
for branches in [{'only': ['/feature/.*/']}, {'only': ['main']}, {'ignore': '/.*/'}]:
with self.subTest(branches=branches), self.assertRaises(ValueError):
migration.publication_filters('feature', {'branches': branches, 'tags': {'only': ['/publish.*/']}})
if __name__ == '__main__':
unittest.main()
+25 -1
View File
@@ -68,6 +68,30 @@ class RolloutTests(unittest.TestCase):
self.assertFalse(rollout.rollout([root], '0.7.11', write=True)[0]['changed'])
self.assertEqual(applied, self.contents(root))
def test_managed_patch_track_and_exact_pin_rollback_preserve_every_profile(self):
roots = [self.checkout(profile, profile=profile, default_branch='main')
for profile in rollout.standard.PROFILE_PARAMETERS]
originals = [yaml.safe_load((root / '.qqq-ci.yml').read_text()) for root in roots]
rollout.rollout(roots, '0.7', write=True)
for root, original in zip(roots, originals):
self.assertEqual({**original, 'orb_version': '0.7'},
yaml.safe_load((root / '.qqq-ci.yml').read_text()))
rollout.standard.validate(*(data.decode() for data in self.contents(root)), '0.7')
rollout.pipeline.load_manifest(root / '.qqq-ci.yml', '0.7')
with self.assertRaises(ValueError):
rollout.pipeline.load_manifest(root / '.qqq-ci.yml', '0.8')
self.assertTrue(all(not r['changed'] for r in rollout.rollout(roots, '0.7', write=True)))
rollout.rollout(roots, '0.7.4', write=True)
for root, original in zip(roots, originals):
self.assertEqual(original, yaml.safe_load((root / '.qqq-ci.yml').read_text()))
def test_unquoted_track_is_rejected_as_numeric_yaml(self):
root = self.checkout('numeric')
manifest = root / '.qqq-ci.yml'
manifest.write_text(manifest.read_text().replace('0.7.4', '0.7'))
with self.assertRaises(ValueError):
rollout.pipeline.load_manifest(manifest, '0.7')
def test_one_custom_caller_rejects_entire_batch_before_any_write(self):
good = self.checkout('good')
bad = self.checkout('bad')
@@ -97,7 +121,7 @@ class RolloutTests(unittest.TestCase):
def test_invalid_target_and_duplicate_checkout_are_rejected(self):
root = self.checkout('qbit')
before = self.contents(root)
for version in ['latest', 'dev:test', '0.7', '0.7.11\n']:
for version in ['latest', 'volatile', 'dev:test', '0', '0.7.*', '0.7.11\n']:
with self.assertRaises(ValueError):
rollout.rollout([root], version, write=True)
with self.assertRaises(ValueError):
+4 -2
View File
@@ -318,9 +318,11 @@ def audit(entry, snapshot, version, worker_sha, policy):
branches = policy.get('required_checks', {})
managed = policy.get('managed_branches', ['main', 'develop'])
if (not isinstance(managed, list) or not managed
or any(branch not in ('main', 'develop') for branch in managed)
or any(not isinstance(branch, str) or not re.fullmatch(
r'(?:main|develop|(?:release|hotfix)/[0-9]+\.[0-9]+(?:\.[0-9]+)?)', branch)
for branch in managed)
or len(set(managed)) != len(managed)):
raise ReadError('managed_branches must be a nonempty unique main/develop list')
raise ReadError('managed_branches must be a nonempty unique main/develop or numeric release/hotfix list')
result['managed_branches'] = managed
for branch in managed:
if branch not in branches:
+2 -2
View File
@@ -56,8 +56,8 @@ def plan(checkout, version):
def rollout(checkouts, version, write=False):
if not re.fullmatch(r'\d+\.\d+\.\d+', version):
raise ValueError('Fleet upgrades require an explicit stable orb version')
if not re.fullmatch(r'\d+\.\d+(?:\.\d+)?', version):
raise ValueError('Fleet upgrades require a stable minor track or exact patch version')
# Validate the entire batch before writing any repository.
plans = [plan(path, version) for path in checkouts]
if len({item['checkout'] for item in plans}) != len(plans):
+13 -3
View File
@@ -11,7 +11,7 @@ import re
import yaml
def canonical_caller(version, release_tag_policy=None):
def canonical_caller(version, release_tag_policy=None, publish_feature=False):
caller = {
"version": 2.1,
"setup": True,
@@ -28,6 +28,9 @@ def canonical_caller(version, release_tag_policy=None):
if release_tag_policy not in filters:
raise ValueError("Unknown tag policy")
caller["workflows"]["setup"]["jobs"][0]["qqq-orb/generate_pipeline"]["filters"]["tags"] = filters[release_tag_policy]
if publish_feature:
caller["workflows"]["setup"]["jobs"][0]["qqq-orb/generate_pipeline"]["filters"]["tags"] = {
"only": ([filters[release_tag_policy]["only"]] if release_tag_policy != "none" else []) + ["/publish.*/"]}
return caller
@@ -43,6 +46,12 @@ def publication_filters(kind, filters):
raise ValueError("Unknown publication filters")
branches = filters.get("branches")
tags = filters.get("tags")
if kind == "feature":
# Preserve the audited literal branch selector; it is not a regex and
# contains a character Git forbids, so only publish* tags can select it.
if branches != {"only": ["feature/.*"]} or tags != {"only": ["/publish.*/"]}:
raise ValueError("Unknown feature publication filters")
return False, "none"
if kind in ("snapshot", "release_candidate", "hotfix"):
expected = {"snapshot": "develop", "release_candidate": r"/release\/.*/", "hotfix": r"/hotfix\/.*/"}[kind]
if not isinstance(branches, dict) or set(branches) != {"only"} or single_filter(branches["only"]) != expected or tags not in (None, {"only": []}):
@@ -123,7 +132,7 @@ def infer_manifest(config, version, default_branch):
raise ValueError("Custom pre-steps require review")
if job.endswith("/mvn_publish"):
kind = options.get("branch_type")
if kind not in ("snapshot", "release_candidate", "release", "hotfix"):
if kind not in ("snapshot", "release_candidate", "release", "hotfix", "feature"):
raise ValueError("Custom publication contract requires review")
if kind in publications:
raise ValueError("Duplicate publication jobs require review")
@@ -148,6 +157,7 @@ def infer_manifest(config, version, default_branch):
"publish": {**{kind: kind in publications
for kind in ("snapshot", "release_candidate", "release")},
**({"hotfix": True} if "hotfix" in publications else {}),
**({"feature": True} if "feature" in publications else {}),
"asciidoc": False},
}
@@ -165,7 +175,7 @@ def main():
if manifest_path.exists():
raise ValueError("Already has a manifest; use tools/rollout_version.py")
manifest = infer_manifest(config, args.orb_version, args.default_branch)
caller = canonical_caller(args.orb_version, manifest["release_tag_policy"])
caller = canonical_caller(args.orb_version, manifest["release_tag_policy"], manifest["publish"].get("feature", False))
if args.write:
for path, data in ((config_path, caller), (manifest_path, manifest)):
path.write_bytes(yaml.safe_dump(data, sort_keys=False).encode("utf-8"))