Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
78207c16be
|
||
|
|
efd50aee2f | ||
|
|
6dd473be25
|
||
|
|
1c8a7c57e5 | ||
|
|
abaebf50ff
|
||
|
|
c56e52d4cb |
No files matched your search
@@ -57,6 +57,9 @@ jobs:
|
||||
python3 tests/test_ci_maintenance.py
|
||||
python3 tests/test_rollout_version.py
|
||||
python3 tests/test_enforce_orb_checks.py
|
||||
python3 tests/test_enroll_consumers.py
|
||||
python3 tests/test_reconcile_enrollment.py
|
||||
python3 tests/test_static_analysis_exit.py
|
||||
python3 tests/test_validate_standard.py
|
||||
python3 tests/test_voyage_profile.py
|
||||
python3 tests/test_next_native.py
|
||||
|
||||
@@ -62,6 +62,9 @@ test-pipelines: pack
|
||||
python3 tests/test_ci_maintenance.py
|
||||
python3 tests/test_rollout_version.py
|
||||
python3 tests/test_enforce_orb_checks.py
|
||||
python3 tests/test_enroll_consumers.py
|
||||
python3 tests/test_reconcile_enrollment.py
|
||||
python3 tests/test_static_analysis_exit.py
|
||||
python3 tests/test_validate_standard.py
|
||||
python3 tests/test_voyage_profile.py
|
||||
python3 tests/test_next_native.py
|
||||
|
||||
@@ -136,3 +136,8 @@ For a qualified batch of canonical orb branches, use the shared
|
||||
CircleCI setup/verify gates while preserving other protections. Source enrollment,
|
||||
scanner transitions and scheduled reconciliation remain separate requirements.
|
||||
Scheduling and authenticated onboarding still need to be configured centrally.
|
||||
|
||||
## Batch enrollment
|
||||
|
||||
Use [the shared enrollment command](consumer-enrollment.md) to convert recognized
|
||||
legacy QBits and update existing canonical consumers in one validated batch.
|
||||
@@ -0,0 +1,73 @@
|
||||
# Batch consumer enrollment
|
||||
|
||||
Use one target file for canonical consumers of every supported Circle profile and
|
||||
legacy QBits whose jobs and publication filters the shared converter recognizes:
|
||||
|
||||
```json
|
||||
[
|
||||
{"checkout": "/work/qbit-example", "default_branch": "main"},
|
||||
{"checkout": "/work/qqq", "default_branch": "develop"}
|
||||
]
|
||||
```
|
||||
|
||||
The branch values must come from repository inventory. Run on clean isolated
|
||||
checkouts of the branches being migrated.
|
||||
|
||||
```sh
|
||||
python tools/enroll_consumers.py targets.json --orb-version 0.8
|
||||
python tools/enroll_consumers.py targets.json --orb-version 0.8 --write
|
||||
```
|
||||
|
||||
The command validates the entire batch before writing. Existing manifests keep
|
||||
all publication and profile options; only the managed orb version changes.
|
||||
Recognized legacy QBits receive the canonical caller and inferred publication
|
||||
policy. An unknown job, filter, profile option, duplicate target, symlink, or
|
||||
concurrent declaration edit rejects the batch. Repeating a completed enrollment
|
||||
makes no changes. Filesystem errors during the write phase can leave a partial
|
||||
batch; rerun after correcting the error. This is not a filesystem transaction.
|
||||
|
||||
Only `.circleci/config.yml` and `.qqq-ci.yml` are changed. The tool does not delete
|
||||
other workflows, infer publication from repository names, or enroll unknown
|
||||
custom pipelines. Their behavior must first be represented by a central profile.
|
||||
After committing and qualifying the exact source, apply the shared
|
||||
[branch enforcement tool](branch-enforcement.md) to supported gate profiles.
|
||||
|
||||
This command provides batch source conversion. Use the coordinator below for remote signed submissions. Local conversion alone does not prove fleet deployment or release success.
|
||||
|
||||
## Remote signed submissions
|
||||
|
||||
Supply observed current branches, without cloning each repository yourself:
|
||||
|
||||
```json
|
||||
[
|
||||
{"repository": "QRun-IO/qbit-example", "branch": "main"},
|
||||
{"repository": "QRun-IO/qqq", "branch": "develop"}
|
||||
]
|
||||
```
|
||||
|
||||
```sh
|
||||
python tools/reconcile_enrollment.py --targets remote-targets.json --orb-version 0.8 --output plan.json
|
||||
python tools/reconcile_enrollment.py --targets remote-targets.json --orb-version 0.8 --output result.json --apply
|
||||
```
|
||||
|
||||
Use existing `gh` authentication, GitHub SSH access, Git author identity and a
|
||||
working Git signing configuration. The coordinator never installs credentials
|
||||
or changes authentication. Every target is inspected at an immutable head and
|
||||
validated before any remote mutation. Only changed targets are cloned. A moved
|
||||
target stops submission; refresh the plan instead of overwriting it.
|
||||
|
||||
An automation branch derives from the source commit and proposed declarations.
|
||||
The coordinator signs its commit, pushes without force, verifies the remote
|
||||
signature, parent, exact changed paths and declaration bytes, then opens a PR.
|
||||
An interrupted push or PR creation can be resumed with the same invocation:
|
||||
matching signed branches and PRs are reused. A mismatched branch is rejected.
|
||||
Closed existing PRs are reported as existing; they are not silently reopened.
|
||||
Reports include repository, branch, immutable head and submission URL, without
|
||||
embedding declaration contents or credentials. Keep the report if a later target
|
||||
fails; earlier valid submissions remain intact.
|
||||
|
||||
This handles explicit remote target lists and signed submissions. Organization
|
||||
discovery is available through `audit_estate.py`; automatic target selection,
|
||||
qualification-aware merging, enforcement reconciliation and scheduling are still
|
||||
separate work. A submitted PR is not a completed migration. Unknown profiles and
|
||||
native workflows require their own central conversion, not deletion by this tool.
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
set -eo pipefail
|
||||
|
||||
echo "Running PMD static analysis..."
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
set -eo pipefail
|
||||
|
||||
echo "Running SpotBugs static analysis..."
|
||||
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
"""Batch enrollment preserves policy and rejects unsafe or unknown declarations."""
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
import yaml
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'tools'))
|
||||
import enroll_consumers as enrollment
|
||||
|
||||
|
||||
class EnrollmentTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.temp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self.temp.cleanup)
|
||||
self.root = Path(self.temp.name)
|
||||
|
||||
def legacy(self, name):
|
||||
root = self.root / name
|
||||
(root / '.circleci').mkdir(parents=True)
|
||||
config = {'version': 2.1, 'orbs': {'qqq-orb': 'kingsrook/qqq-orb@0.5.0'},
|
||||
'workflows': {'verify': {'jobs': [{'qqq-orb/mvn_test_only': {}}]}}}
|
||||
(root / '.circleci/config.yml').write_text(yaml.safe_dump(config), encoding='utf-8')
|
||||
return {'checkout': str(root), 'default_branch': 'main'}
|
||||
|
||||
def canonical(self, profile):
|
||||
target = self.legacy(profile)
|
||||
root = Path(target['checkout'])
|
||||
data = {'profile': profile, 'orb_version': '0.7.4', 'default_branch': 'main'}
|
||||
(root / '.qqq-ci.yml').write_text(yaml.safe_dump(data), encoding='utf-8')
|
||||
(root / '.circleci/config.yml').write_text(yaml.safe_dump(
|
||||
enrollment.rollout.standard.canonical_caller(profile, '0.7.4')), encoding='utf-8')
|
||||
return target
|
||||
|
||||
def test_mixed_fleet_preview_write_and_idempotence(self):
|
||||
legacy = self.legacy('legacy')
|
||||
targets = [legacy] + [self.canonical(p) for p in enrollment.rollout.standard.PROFILE_PARAMETERS]
|
||||
preview = enrollment.enroll(targets, '0.8')
|
||||
self.assertTrue(all(item['changed'] for item in preview))
|
||||
self.assertFalse((Path(legacy['checkout']) / '.qqq-ci.yml').exists())
|
||||
result = enrollment.enroll(targets, '0.8', write=True)
|
||||
self.assertEqual(1, sum(item['enrollment'] for item in result))
|
||||
data = yaml.safe_load((Path(legacy['checkout']) / '.qqq-ci.yml').read_text())
|
||||
self.assertFalse(any(data['publish'].values()))
|
||||
self.assertEqual('all', data['test_branch_policy'])
|
||||
self.assertTrue(all(not item['changed'] for item in enrollment.enroll(targets, '0.8', write=True)))
|
||||
|
||||
def test_unknown_orchestration_rejects_entire_batch(self):
|
||||
good, bad = self.legacy('good'), self.legacy('bad')
|
||||
path = Path(bad['checkout']) / '.circleci/config.yml'
|
||||
path.write_text(path.read_text() + '\njobs: {custom: {steps: [checkout]}}\n')
|
||||
original = (Path(good['checkout']) / '.circleci/config.yml').read_bytes()
|
||||
with self.assertRaises(ValueError):
|
||||
enrollment.enroll([good, bad], '0.8', write=True)
|
||||
self.assertEqual(original, (Path(good['checkout']) / '.circleci/config.yml').read_bytes())
|
||||
self.assertFalse((Path(good['checkout']) / '.qqq-ci.yml').exists())
|
||||
|
||||
def test_duplicate_targets_and_wrong_default_reject_before_write(self):
|
||||
target = self.canonical('core')
|
||||
for targets in ([target, target], [{**target, 'default_branch': 'develop'}]):
|
||||
with self.assertRaises(ValueError):
|
||||
enrollment.enroll(targets, '0.8', write=True)
|
||||
self.assertEqual('0.7.4', yaml.safe_load((Path(target['checkout']) / '.qqq-ci.yml').read_text())['orb_version'])
|
||||
|
||||
def test_invalid_manifest_and_unstable_versions_rejected(self):
|
||||
target = self.canonical('qbit')
|
||||
path = Path(target['checkout']) / '.qqq-ci.yml'
|
||||
path.write_text(path.read_text() + '\nunknown_policy: true\n')
|
||||
for version in ('0.8', 'dev:snapshot', 'latest', 0.8):
|
||||
with self.assertRaises(ValueError):
|
||||
enrollment.enroll([target], version, write=True)
|
||||
|
||||
def test_drift_after_preflight_is_not_overwritten(self):
|
||||
target = self.legacy('drift')
|
||||
path = Path(target['checkout']) / '.circleci/config.yml'
|
||||
original_plan = enrollment.plan
|
||||
|
||||
def changed_plan(*args):
|
||||
result = original_plan(*args)
|
||||
path.write_bytes(b'concurrent edit\n')
|
||||
return result
|
||||
|
||||
with mock.patch.object(enrollment, 'plan', side_effect=changed_plan):
|
||||
with self.assertRaises(ValueError):
|
||||
enrollment.enroll([target], '0.8', write=True)
|
||||
self.assertEqual(b'concurrent edit\n', path.read_bytes())
|
||||
self.assertFalse((Path(target['checkout']) / '.qqq-ci.yml').exists())
|
||||
|
||||
def test_manifest_symlink_is_rejected(self):
|
||||
target = self.legacy('redirect')
|
||||
outside = self.root / 'outside.yml'
|
||||
outside.write_text('untouched\n')
|
||||
try:
|
||||
(Path(target['checkout']) / '.qqq-ci.yml').symlink_to(outside)
|
||||
except OSError as error:
|
||||
self.skipTest('Symlink creation unavailable: ' + str(error))
|
||||
with self.assertRaises(ValueError):
|
||||
enrollment.enroll([target], '0.8', write=True)
|
||||
self.assertEqual('untouched\n', outside.read_text())
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,159 @@
|
||||
"""Remote enrollment keeps immutable source checks and resumes signed submissions."""
|
||||
import base64
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
import yaml
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'tools'))
|
||||
import reconcile_enrollment as remote
|
||||
|
||||
|
||||
class ReconcileTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.head = 'a' * 40
|
||||
self.new = 'b' * 40
|
||||
self.repo = 'QRun-IO/example'
|
||||
self.target = {'repository': self.repo, 'branch': 'main'}
|
||||
self.manifest = {'profile': 'qbit', 'orb_version': '0.7.4', 'default_branch': 'main',
|
||||
'publish': {'snapshot': False, 'release_candidate': False, 'release': False}}
|
||||
self.sources = {
|
||||
'.qqq-ci.yml': yaml.safe_dump(self.manifest).encode(),
|
||||
'.circleci/config.yml': yaml.safe_dump(remote.enroll_consumers.rollout.standard.canonical_caller('qbit', '0.7.4')).encode()}
|
||||
|
||||
def request(self, path, missing=False):
|
||||
if path == 'repos/' + self.repo:
|
||||
return {'archived': False, 'default_branch': 'main'}
|
||||
if path.endswith('/branches/main'):
|
||||
return {'commit': {'sha': self.head}}
|
||||
if '/contents/' in path:
|
||||
self.assertTrue(path.endswith('?ref=' + self.head))
|
||||
name = path.split('/contents/')[1].split('?')[0]
|
||||
raw = self.sources.get(name)
|
||||
return None if raw is None else {'type': 'file', 'encoding': 'base64', 'content': base64.b64encode(raw).decode()}
|
||||
raise AssertionError(path)
|
||||
|
||||
def plan(self):
|
||||
with mock.patch.object(remote, 'api', side_effect=self.request):
|
||||
return remote.inspect(self.target, '0.8')
|
||||
|
||||
def test_immutable_plan_is_deterministic_and_preserves_policy(self):
|
||||
first, second = self.plan(), self.plan()
|
||||
self.assertEqual(first['automation_branch'], second['automation_branch'])
|
||||
self.assertEqual(self.sources, first['originals'])
|
||||
self.assertEqual({**self.manifest, 'orb_version': '0.8'},
|
||||
yaml.safe_load(first['replacements']['.qqq-ci.yml']))
|
||||
self.assertTrue(first['changed'])
|
||||
|
||||
def test_missing_manifest_uses_legacy_converter(self):
|
||||
self.sources.pop('.qqq-ci.yml')
|
||||
self.sources['.circleci/config.yml'] = yaml.safe_dump({'version': 2.1,
|
||||
'orbs': {'qqq-orb': 'kingsrook/qqq-orb@0.5.0'},
|
||||
'workflows': {'verify': {'jobs': [{'qqq-orb/mvn_test_only': {}}]}}}).encode()
|
||||
plan = self.plan()
|
||||
self.assertIsNone(plan['originals']['.qqq-ci.yml'])
|
||||
self.assertFalse(any(yaml.safe_load(plan['replacements']['.qqq-ci.yml'])['publish'].values()))
|
||||
|
||||
def submission_api(self, plan, path, missing=False):
|
||||
if '/branches/main' in path:
|
||||
return {'commit': {'sha': self.head}}
|
||||
if '/git/ref/' in path:
|
||||
return {'object': {'sha': self.new}}
|
||||
if '/commits/' in path:
|
||||
return {'commit': {'verification': {'verified': True}},
|
||||
'parents': [{'sha': self.head}], 'files': [{'filename': p} for p in remote.PATHS]}
|
||||
if '/contents/' in path:
|
||||
self.assertTrue(path.endswith('?ref=' + self.new))
|
||||
name = path.split('/contents/')[1].split('?')[0]
|
||||
return {'type': 'file', 'encoding': 'base64', 'content': base64.b64encode(plan['replacements'][name]).decode()}
|
||||
if '/pulls?' in path:
|
||||
return [{'head': {'sha': self.new}, 'html_url': 'https://github.com/QRun-IO/example/pull/1'}]
|
||||
raise AssertionError(path)
|
||||
|
||||
def test_matching_existing_submission_uses_no_git_or_pr_mutations(self):
|
||||
plan = self.plan()
|
||||
with mock.patch.object(remote, 'api', side_effect=lambda *a, **k: self.submission_api(plan, *a, **k)), mock.patch.object(remote, 'command') as command:
|
||||
result = remote.apply(plan)
|
||||
self.assertEqual('existing-pr', result['state'])
|
||||
command.assert_not_called()
|
||||
|
||||
def test_unsigned_existing_branch_is_rejected_without_mutation(self):
|
||||
plan = self.plan()
|
||||
|
||||
def request(path, missing=False):
|
||||
value = self.submission_api(plan, path, missing)
|
||||
if '/commits/' in path:
|
||||
value['commit']['verification']['verified'] = False
|
||||
return value
|
||||
|
||||
with mock.patch.object(remote, 'api', side_effect=request), mock.patch.object(remote, 'command') as command:
|
||||
with self.assertRaises(ValueError):
|
||||
remote.apply(plan)
|
||||
command.assert_not_called()
|
||||
|
||||
def test_source_drift_prevents_submission(self):
|
||||
plan = self.plan()
|
||||
with mock.patch.object(remote, 'api', return_value={'commit': {'sha': 'c' * 40}}), mock.patch.object(remote, 'command') as command:
|
||||
with self.assertRaises(ValueError):
|
||||
remote.apply(plan)
|
||||
command.assert_not_called()
|
||||
|
||||
def test_interrupted_push_creates_only_missing_pr(self):
|
||||
plan = self.plan()
|
||||
seen = []
|
||||
|
||||
def request(path, missing=False):
|
||||
return [] if '/pulls?' in path else self.submission_api(plan, path, missing)
|
||||
|
||||
def command(args, cwd=None):
|
||||
seen.append(args[:3])
|
||||
self.assertEqual(['gh', 'pr', 'create'], args[:3])
|
||||
self.assertIn(self.head, Path(args[args.index('--body-file') + 1]).read_text())
|
||||
return 'https://github.com/QRun-IO/example/pull/2'
|
||||
|
||||
with mock.patch.object(remote, 'api', side_effect=request), mock.patch.object(remote, 'command', side_effect=command):
|
||||
result = remote.apply(plan)
|
||||
self.assertEqual('submitted', result['state'])
|
||||
self.assertEqual([['gh', 'pr', 'create']], seen)
|
||||
|
||||
def test_new_submission_requires_signed_commit_and_never_force_pushes(self):
|
||||
plan = self.plan()
|
||||
committed = False
|
||||
seen = []
|
||||
|
||||
def request(path, missing=False):
|
||||
if '/git/ref/' in path:
|
||||
return None
|
||||
if '/pulls?' in path:
|
||||
return []
|
||||
return self.submission_api(plan, path, missing)
|
||||
|
||||
def command(args, cwd=None):
|
||||
nonlocal committed
|
||||
seen.append(args)
|
||||
if args[:2] == ['git', 'clone']:
|
||||
root = Path(args[-1])
|
||||
(root / '.circleci').mkdir(parents=True)
|
||||
for name, data in self.sources.items():
|
||||
(root / name).write_bytes(data)
|
||||
elif args[:2] == ['git', 'rev-parse']:
|
||||
return self.new if committed else self.head
|
||||
elif args[:2] == ['git', 'commit']:
|
||||
self.assertIn('-S', args)
|
||||
for name in remote.PATHS:
|
||||
self.assertEqual(plan['replacements'][name], (cwd / name).read_bytes())
|
||||
committed = True
|
||||
elif args[:3] == ['gh', 'pr', 'create']:
|
||||
return 'https://github.com/QRun-IO/example/pull/3'
|
||||
return ''
|
||||
|
||||
with mock.patch.object(remote, 'api', side_effect=request), mock.patch.object(remote, 'command', side_effect=command):
|
||||
self.assertEqual('submitted', remote.apply(plan)['state'])
|
||||
pushes = [args for args in seen if args[:2] == ['git', 'push']]
|
||||
self.assertEqual([['git', 'push', 'origin', 'HEAD:refs/heads/' + plan['automation_branch']]], pushes)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,51 @@
|
||||
"""Maven failures must survive tee without overriding project analysis policy."""
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SCRIPTS = [('mvn_spotbugs.sh', 'SPOTBUGS_FAIL_ON_ERROR', 'spotbugs.failOnError'),
|
||||
('mvn_pmd.sh', 'PMD_FAIL_ON_VIOLATION', 'pmd.failOnViolation')]
|
||||
|
||||
|
||||
class StaticAnalysisExitTests(unittest.TestCase):
|
||||
def invoke(self, script, variable, setting, status):
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
binary = root / 'mvn'
|
||||
binary.write_text('#!/bin/sh\nprintf "%s\\n" "$@" > "$ARGUMENT_FILE"\necho analysis-output\nexit "$FAKE_MAVEN_EXIT"\n')
|
||||
binary.chmod(0o755)
|
||||
env = dict(os.environ, PATH=str(root) + os.pathsep + os.environ['PATH'],
|
||||
ARGUMENT_FILE=str(root / 'arguments'), FAKE_MAVEN_EXIT=str(status))
|
||||
env.pop(variable, None)
|
||||
if setting is not None:
|
||||
env[variable] = setting
|
||||
result = subprocess.run(['bash', str(ROOT / 'src/scripts' / script)], cwd=root,
|
||||
env=env, text=True, capture_output=True, timeout=15)
|
||||
return result, (root / 'arguments').read_text().splitlines()
|
||||
|
||||
def test_maven_failure_is_never_hidden_by_successful_tee(self):
|
||||
for script, variable, _ in SCRIPTS:
|
||||
for setting in ('true', 'false'):
|
||||
with self.subTest(script=script, setting=setting):
|
||||
result, _ = self.invoke(script, variable, setting, 17)
|
||||
self.assertEqual(17, result.returncode)
|
||||
self.assertIn('analysis-output', result.stdout)
|
||||
|
||||
def test_success_preserves_explicit_flag_and_project_defaults(self):
|
||||
for script, variable, property_name in SCRIPTS:
|
||||
for setting in ('true', 'false', None):
|
||||
with self.subTest(script=script, setting=setting):
|
||||
result, arguments = self.invoke(script, variable, setting, 0)
|
||||
self.assertEqual(0, result.returncode, result.stderr)
|
||||
if setting == 'true':
|
||||
self.assertIn('-D' + property_name + '=true', arguments)
|
||||
else:
|
||||
self.assertFalse(any(arg.startswith('-D' + property_name + '=') for arg in arguments))
|
||||
self.assertIn('-DskipTests', arguments)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,101 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Enroll a batch of audited consumers; preview by default, write with --write.
|
||||
|
||||
Existing canonical profiles retain their manifests. Legacy QBits use the shared
|
||||
policy inference. Unknown orchestration is rejected before any checkout changes.
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import tempfile
|
||||
|
||||
import yaml
|
||||
|
||||
import rollout_version as rollout
|
||||
import standardize_qbit as qbit
|
||||
|
||||
|
||||
def declaration(root, relative):
|
||||
"""Reject redirected declarations, including symlinked parent directories."""
|
||||
path = root / relative
|
||||
current = path
|
||||
while current != root:
|
||||
if current.is_symlink():
|
||||
raise ValueError('CI declarations must not use symlinks')
|
||||
current = current.parent
|
||||
if not path.resolve().is_relative_to(root):
|
||||
raise ValueError('CI declarations must remain inside the checkout')
|
||||
return path
|
||||
|
||||
|
||||
def plan(target, version):
|
||||
"""Preserve explicit publication policy and validate generated declarations."""
|
||||
if not isinstance(target, dict) or set(target) != {'checkout', 'default_branch'}:
|
||||
raise ValueError('Each target requires checkout and observed default_branch')
|
||||
if target['default_branch'] not in ('main', 'develop'):
|
||||
raise ValueError('Unsupported default branch')
|
||||
root = Path(target['checkout']).resolve(strict=True)
|
||||
config = declaration(root, '.circleci/config.yml')
|
||||
manifest = declaration(root, '.qqq-ci.yml')
|
||||
if manifest.exists():
|
||||
data = rollout.standard.load_yaml(manifest.read_text(encoding='utf-8'))
|
||||
if not isinstance(data, dict) or data.get('default_branch') != target['default_branch']:
|
||||
raise ValueError('Observed default branch differs from manifest')
|
||||
return {**rollout.plan(root, version), 'enrollment': False}
|
||||
original = config.read_bytes()
|
||||
data = qbit.infer_manifest(rollout.standard.load_yaml(original.decode('utf-8')),
|
||||
version, target['default_branch'])
|
||||
caller = rollout.standard.canonical_caller(data['profile'], version,
|
||||
release_tag_policy=data['release_tag_policy'], publish_feature=data['publish'].get('feature', False))
|
||||
config_raw = yaml.safe_dump(caller, sort_keys=False)
|
||||
manifest_raw = yaml.safe_dump(data, sort_keys=False)
|
||||
rollout.standard.validate(config_raw, manifest_raw, version)
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
temporary_manifest = Path(directory) / 'manifest.yml'
|
||||
temporary_manifest.write_text(manifest_raw, encoding='utf-8')
|
||||
rollout.pipeline.load_manifest(temporary_manifest, version)
|
||||
return {'checkout': str(root), 'profile': data['profile'], 'from': None,
|
||||
'to': version, 'enrollment': True,
|
||||
'files': [(config, original, config_raw.encode('utf-8')),
|
||||
(manifest, None, manifest_raw.encode('utf-8'))]}
|
||||
|
||||
|
||||
def enroll(targets, version, write=False):
|
||||
"""Validate the whole batch before writing; repeat applications are no-ops."""
|
||||
if not isinstance(version, str) or not re.fullmatch(r'\d+\.\d+(?:\.\d+)?', version):
|
||||
raise ValueError('Enrollment requires a stable minor track or exact patch')
|
||||
if not isinstance(targets, list) or not targets:
|
||||
raise ValueError('A nonempty target list is required')
|
||||
plans = [plan(target, version) for target in targets]
|
||||
if len({item['checkout'] for item in plans}) != len(plans):
|
||||
raise ValueError('Each checkout may appear only once')
|
||||
if write:
|
||||
for item in plans:
|
||||
for path, original, replacement in item['files']:
|
||||
declaration(Path(item['checkout']), path.relative_to(item['checkout']))
|
||||
current = path.read_bytes() if path.exists() else None
|
||||
if current != original:
|
||||
raise ValueError('Declaration changed after preflight')
|
||||
for item in plans:
|
||||
for path, original, replacement in item['files']:
|
||||
if replacement != original:
|
||||
path.write_bytes(replacement)
|
||||
return [{**{key: value for key, value in item.items() if key != 'files'},
|
||||
'changed': any(old != new for _, old, new in item['files']),
|
||||
'written': write} for item in plans]
|
||||
|
||||
|
||||
def main():
|
||||
"""Read one fleet target file and emit an auditable result."""
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('targets', type=Path)
|
||||
parser.add_argument('--orb-version', required=True)
|
||||
parser.add_argument('--write', action='store_true')
|
||||
args = parser.parse_args()
|
||||
print(json.dumps(enroll(json.loads(args.targets.read_text(encoding='utf-8')),
|
||||
args.orb_version, args.write), indent=2))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,173 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Plan remote enrollment and submit signed, repeatable PRs with --apply."""
|
||||
import argparse
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import subprocess
|
||||
import tempfile
|
||||
|
||||
import enroll_consumers
|
||||
|
||||
PATHS = ('.circleci/config.yml', '.qqq-ci.yml')
|
||||
|
||||
|
||||
def command(args, cwd=None):
|
||||
"""Use existing credentials and signing configuration without interactive prompts."""
|
||||
env = dict(os.environ, GIT_TERMINAL_PROMPT='0')
|
||||
result = subprocess.run(args, cwd=cwd, env=env, capture_output=True,
|
||||
text=True, encoding='utf-8', timeout=120)
|
||||
if result.returncode:
|
||||
raise ValueError('Command failed: ' + args[0])
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
def api(path, missing=False):
|
||||
"""Only an explicit 404 is treated as absence; other API failures stop work."""
|
||||
result = subprocess.run(['gh', 'api', path], capture_output=True,
|
||||
text=True, encoding='utf-8', timeout=60)
|
||||
try:
|
||||
data = json.loads(result.stdout)
|
||||
except ValueError:
|
||||
raise ValueError('Invalid GitHub response') from None
|
||||
if result.returncode:
|
||||
if missing and str(data.get('status')) == '404':
|
||||
return None
|
||||
raise ValueError('GitHub read unavailable: ' + path.split('?')[0])
|
||||
return data
|
||||
|
||||
|
||||
def content(repo, path, head):
|
||||
"""Read a regular declaration at an immutable commit, never at a moving branch."""
|
||||
item = api(f'repos/{repo}/contents/{path}?ref={head}', missing=True)
|
||||
if item is None:
|
||||
return None
|
||||
if item.get('type') != 'file' or item.get('encoding') != 'base64':
|
||||
raise ValueError('Declaration is not a regular file')
|
||||
return base64.b64decode(item['content'])
|
||||
|
||||
|
||||
def inspect(target, version):
|
||||
"""Resolve observed repository identity and reuse the shared enrollment planner."""
|
||||
if not isinstance(target, dict) or set(target) != {'repository', 'branch'}:
|
||||
raise ValueError('Each target requires repository and branch')
|
||||
repo, branch = target['repository'], target['branch']
|
||||
if not isinstance(repo, str) or not re.fullmatch(r'[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+', repo):
|
||||
raise ValueError('Invalid repository')
|
||||
if branch not in ('main', 'develop'):
|
||||
raise ValueError('Enrollment targets current main/develop only')
|
||||
info = api('repos/' + repo)
|
||||
if info['archived']:
|
||||
raise ValueError('Archived repository')
|
||||
head = api(f'repos/{repo}/branches/{branch}')['commit']['sha']
|
||||
originals = {path: content(repo, path, head) for path in PATHS}
|
||||
if originals[PATHS[0]] is None:
|
||||
raise ValueError('No recognized CircleCI entrypoint')
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory)
|
||||
(root / '.circleci').mkdir()
|
||||
for path, data in originals.items():
|
||||
if data is not None:
|
||||
(root / path).write_bytes(data)
|
||||
plan = enroll_consumers.plan({'checkout': str(root), 'default_branch': info['default_branch']}, version)
|
||||
replacements = {str(path.relative_to(root)).replace('\\', '/'): new for path, _, new in plan['files']}
|
||||
digest = hashlib.sha256(head.encode() + b''.join(replacements[path] for path in PATHS)).hexdigest()[:20]
|
||||
return {**target, 'head': head, 'originals': originals, 'replacements': replacements,
|
||||
'profile': plan['profile'], 'version': version,
|
||||
'changed': any(originals[path] != replacements[path] for path in PATHS),
|
||||
'automation_branch': f'automation/qqq-enroll-{branch}-{digest}'}
|
||||
|
||||
|
||||
def verify_submission(plan, head):
|
||||
"""Accept only a signed, one-parent commit with exactly the planned changes."""
|
||||
repo = plan['repository']
|
||||
commit = api(f'repos/{repo}/commits/{head}')
|
||||
expected = {path for path in PATHS if plan['originals'][path] != plan['replacements'][path]}
|
||||
if (not commit['commit']['verification']['verified']
|
||||
or [parent['sha'] for parent in commit['parents']] != [plan['head']]
|
||||
or {item['filename'] for item in commit['files']} != expected):
|
||||
raise ValueError('Existing automation branch differs from the signed plan')
|
||||
for path in PATHS:
|
||||
if content(repo, path, head) != plan['replacements'][path]:
|
||||
raise ValueError('Submitted declaration differs from plan')
|
||||
|
||||
|
||||
def apply(plan):
|
||||
"""Never force push; reuse matching submissions after interrupted executions."""
|
||||
if not plan['changed']:
|
||||
return {'state': 'unchanged'}
|
||||
repo, branch = plan['repository'], plan['automation_branch']
|
||||
if api(f'repos/{repo}/branches/{plan["branch"]}')['commit']['sha'] != plan['head']:
|
||||
raise ValueError('Target branch changed after preflight')
|
||||
ref = api(f'repos/{repo}/git/ref/heads/{branch}', missing=True)
|
||||
if ref is None:
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
root = Path(directory) / 'checkout'
|
||||
command(['git', 'clone', '-c', 'core.autocrlf=false', '--depth', '1', '--single-branch', '--branch', plan['branch'],
|
||||
f'git@github.com:{repo}.git', str(root)])
|
||||
if command(['git', 'rev-parse', 'HEAD'], root) != plan['head']:
|
||||
raise ValueError('Target branch moved before checkout')
|
||||
for path in PATHS:
|
||||
target = enroll_consumers.declaration(root, path)
|
||||
current = target.read_bytes() if target.exists() else None
|
||||
if current != plan['originals'][path]:
|
||||
raise ValueError('Checkout differs from immutable preflight')
|
||||
target.write_bytes(plan['replacements'][path])
|
||||
command(['git', 'add', '--', *PATHS], root)
|
||||
command(['git', 'diff', '--cached', '--check'], root)
|
||||
command(['git', 'commit', '-S', '-m', 'ci: enroll in the shared QQQ orb', '-m',
|
||||
'Co-Authored-By: Codex <noreply@openai.com>'], root)
|
||||
head = command(['git', 'rev-parse', 'HEAD'], root)
|
||||
if api(f'repos/{repo}/branches/{plan["branch"]}')['commit']['sha'] != plan['head']:
|
||||
raise ValueError('Target branch moved before submission')
|
||||
command(['git', 'push', 'origin', f'HEAD:refs/heads/{branch}'], root)
|
||||
else:
|
||||
head = ref['object']['sha']
|
||||
verify_submission(plan, head)
|
||||
pulls = api(f'repos/{repo}/pulls?state=all&head={repo.split("/")[0]}:{branch}&base={plan["branch"]}')
|
||||
if pulls:
|
||||
if len(pulls) != 1 or pulls[0]['head']['sha'] != head:
|
||||
raise ValueError('Ambiguous existing enrollment PR')
|
||||
return {'state': 'existing-pr', 'head': head, 'url': pulls[0]['html_url']}
|
||||
body = ('Uses the shared orb enrollment converter and preserves existing publication policy. '
|
||||
'Only the canonical caller and manifest change. Exact-head hosted checks must pass '
|
||||
'before merge; branch enforcement follows successful qualification.\n\n'
|
||||
'Source base: ' + plan['head'])
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path = Path(directory) / 'body.md'
|
||||
path.write_text(body, encoding='utf-8')
|
||||
url = command(['gh', 'pr', 'create', '--repo', repo, '--base', plan['branch'], '--head', branch,
|
||||
'--title', 'ci: enroll in the shared QQQ orb', '--body-file', str(path)])
|
||||
return {'state': 'submitted', 'head': head, 'url': url}
|
||||
|
||||
|
||||
def main():
|
||||
"""Preflight every explicit target before mutations and save resumable results."""
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--targets', type=Path, required=True)
|
||||
parser.add_argument('--orb-version', required=True)
|
||||
parser.add_argument('--apply', action='store_true')
|
||||
parser.add_argument('--output', type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
if not re.fullmatch(r'\d+\.\d+(?:\.\d+)?', args.orb_version):
|
||||
raise ValueError('A stable minor track or exact patch is required')
|
||||
targets = json.loads(args.targets.read_text(encoding='utf-8'))
|
||||
if not isinstance(targets, list) or not targets:
|
||||
raise ValueError('A nonempty target list is required')
|
||||
plans = [inspect(target, args.orb_version) for target in targets]
|
||||
if len({(p['repository'].lower(), p['branch']) for p in plans}) != len(plans):
|
||||
raise ValueError('Duplicate target')
|
||||
report = [{key: value for key, value in p.items() if key not in ('originals', 'replacements')} for p in plans]
|
||||
args.output.write_text(json.dumps(report, indent=2), encoding='utf-8')
|
||||
if args.apply:
|
||||
for plan, row in zip(plans, report):
|
||||
row['result'] = apply(plan)
|
||||
args.output.write_text(json.dumps(report, indent=2), encoding='utf-8')
|
||||
print(json.dumps({'targets': len(plans), 'changed': sum(p['changed'] for p in plans), 'applied': args.apply}))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Reference in new issue
Block a user