ci: stop cold-compiling every Reborn lane (stable hermetic Cargo home, push-only shared caches, stable toolchain, warm in-place mutation gate) (#8050)

* ci: stop cold-compiling every Reborn lane

Three reference runs (PR #8006 at 4bcdc7678a, merge-queue runs
33642416400 and 33656141094) showed every Tests (Reborn) lane compiling
its full dependency closure from cold: 0 of 47 lanes restored a Rust
build cache, and the three heaviest crate buckets compiled the closure
twice in one job (446+452 s, 483+514 s, 381+381 s, identical unit sets).
Compilation was 65% of PR test-lane minutes and 56-62% of queue minutes.

Root cause of the double build and of the dead cache: the hermetic
wrapper minted a fresh CARGO_HOME per invocation. Cargo hashes each
registry crate's absolute source path into its fingerprint, so the second
wrapper invocation in a bucket (nextest show-config, then nextest run)
marked every dependency PathToSourceChanged, and a restored target/ could
never be fresh inside the boundary (queue 33642416424's runtimes group
restored 1451 MB and still compiled 1298 units).

- scripts/ci/run-hermetic-test-process.sh: one stable, suite-owned Cargo
  home per host (still only registry/git symlinks, never host config or
  credentials). The self-test now pins a stable CARGO_HOME across
  sequential and parallel invocations, no host Cargo state inside it, and
  that a second guarded build compiles nothing.
- rust-cache saves are push-to-main only (10 sites). A merge_group save
  lands on the entry's gh-readonly-queue ref, which no pull request and
  no later entry can restore, so those saves were write-only and their
  6-7 GB per entry evicted main's caches from the 10 GB limit. Wrapper-
  built lanes share one `reborn-hermetic` lineage (crate buckets, root,
  integration, QA, Rust Reborn E2E groups); direct-cargo lanes share
  `reborn-direct` (sandbox Docker saves, mutation gate restores).
  RUST_MIN_STACK / RUSTC_BOOTSTRAP move from job env to step env so the
  lanes hash to one key.
- Crate buckets and the integration lane build on the pinned stable
  toolchain for PR, queue and dispatch runs; `cargo check --workspace
  --all-targets` (and with --all-features) on 1.98.0 passes without the
  nightly crate attributes. The instrumented push coverage run keeps
  nightly unchanged.
- The merge-queue mutation gate restores the warm cache and runs
  cargo-mutants --in-place (new opt-in gate flag, self-tested; implies
  --jobs 1) instead of a cold closure build per invariant (14.9 min for
  one invariant, 38.8 min for all eleven).

No test, lane, partition, or selection rule is removed or narrowed.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QtLJ4UiSzbjFxnMT1Vbkpj

* ci: keep the branch-coverage RUSTC_BOOTSTRAP envelope, step-scoped

check-reborn-branch-coverage-flags.py counts four exact
`RUSTC_BOOTSTRAP: \"1\"` lines across the coverage workflows; the two
in reborn-tests.yml now live on the crate-tests and integration run
steps, where the instrumented push run executes and where rust-cache
does not hash them into the shared `reborn-hermetic` key.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QtLJ4UiSzbjFxnMT1Vbkpj

* ci: keep RUST_MIN_STACK job-level, one value across the hermetic lanes

tests/reborn_coverage_lane_stack_headroom.rs reads the job-level
RUST_MIN_STACK of every lane that runs the integration package under
llvm-cov and ignores step-level values on purpose, so the step-scoped
placement failed the root partition on the exhaustive dispatch run
(33670885071). The value is job-level again and 64 MiB everywhere the
hermetic wrapper runs (crate buckets and the integration batch up from
8 MiB), because rust-cache hashes job-level RUST* variables into its key
and a per-lane value would split the shared reborn-hermetic lineage; a
larger minimum is a per-thread virtual reservation, not committed
memory. RUSTC_BOOTSTRAP stays step-scoped: the wrapper drops it either
way and the branch-coverage checker only counts the line.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QtLJ4UiSzbjFxnMT1Vbkpj

* ci: harden the stable hermetic Cargo home and single-source its cache

Review follow-ups on #8050:

- Compare canonical paths in the host-Cargo-home guard, so a symlink or a
  `..` spelling of the host home is refused like the literal path, and
  refuse an ancestor of the host home (the scrub below would otherwise
  delete host files). Regression: three equivalent spellings must exit 2
  and leave the host home untouched.
- Scrub Cargo config and credential files from the stable Cargo home on
  every entry: the home now outlives an invocation, so state one guarded
  command left there must not reach the next. Regression: files planted
  by one invocation are absent in the next.
- Only the push run's root partitions and QA replay save the
  `reborn-hermetic` cache (same complete integration-test closure); the
  Rust Reborn E2E groups restore only, so a small group finishing first
  cannot pin a partial dependency set under the once-per-lockfile key.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QtLJ4UiSzbjFxnMT1Vbkpj

* ci: pass no --jobs to cargo-mutants in place

cargo-mutants 27.1 rejects --jobs next to --in-place even with a value
of 1 (the merge-queue gate for #8050 failed with "the argument
'--in-place' cannot be used with '--jobs <JOBS>'"). In-place runs now
pass no --jobs at all; the self-test stub enforces the real mutual
exclusion so the pair cannot be emitted again, and asserts --jobs is
absent in both in-place forms.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QtLJ4UiSzbjFxnMT1Vbkpj

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Benjamin Kurrek
2026-09-02 20:30:00 +00:00
committed by GitHub
parent 51582d3930
commit f3610f504a
11 changed files with 449 additions and 74 deletions

View File

@@ -10,6 +10,26 @@ a check belongs to exactly one tier on purpose.
| Post-merge confirm | `push` to `main` | Runs exhaustive coverage, warms shared caches, and feeds Codecov/canaries. A main-only deterministic failure means the affected-area classifier missed an impact and must be widened. |
| Deep / scheduled | `schedule` (nightly) | Exhaustive suites too slow for the queue: legacy v1 matrix, full browser E2E, stress scans. |
## Build caches
Rust build caches (Swatinem/rust-cache) are saved only by pushes to `main`
(the stress lane's by its nightly scan) and restored by every other event.
GitHub lets a run restore caches from its own ref, its base branch, and the
default branch, so a save from a `merge_group` run lands on the queue entry's
`gh-readonly-queue/...` ref where no pull request and no later entry can read
it; those saves were write-only and evicted `main`'s entries from the 10 GB
repository limit. Two plain lineages exist. `reborn-hermetic` is every lane
that builds inside `scripts/ci/run-hermetic-test-process.sh` (crate buckets,
root partitions, the integration batch, QA replay, the Rust Reborn E2E
groups): pinned stable toolchain, mold, and a stable hermetic Cargo home, so
one entry is fresh for all of them. A shared key is written once per lockfile
by the first job to finish, so only the push run's root partitions and QA
replay, which build the same complete integration-test closure, save it;
every other job restores. `reborn-direct` is the lanes that run
cargo against the host Cargo home (the sandbox Docker tests, which save it,
and the merge-queue mutation gate, which restores it). The instrumented
coverage lanes on `main` keep their separate `-cov-llvm21` lineages.
## The invariant
**No attributable deterministic failure may be main-only.** The merge queue

View File

@@ -432,9 +432,9 @@ jobs:
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
with:
shared-key: clippy
# Keep saves to protected-branch and merge_group runs so the ~10 GB
# repo cache LRU is seeded by shared states, not arbitrary PR branches.
save-if: ${{ matrix.name == 'all-features' && ((github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group') }}
# Saved from pushes to main only (merge_group saves land on a queue
# ref nothing else can restore; see reborn-tests.yml).
save-if: ${{ matrix.name == 'all-features' && github.event_name == 'push' && github.ref == 'refs/heads/main' }}
- name: Setup OVH sccache
uses: ./.github/actions/setup-sccache-dist
with:
@@ -579,9 +579,9 @@ jobs:
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
with:
shared-key: reborn-cli-smoke
# Keep saves to protected-branch and merge_group runs so the ~10 GB
# repo cache LRU is seeded by shared states, not arbitrary PR branches.
save-if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group' }}
# Saved from pushes to main only (merge_group saves land on a queue
# ref nothing else can restore; see reborn-tests.yml).
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
- name: Setup OVH sccache
uses: ./.github/actions/setup-sccache-dist
with:

View File

@@ -66,7 +66,9 @@ jobs:
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
with:
key: ironclaw-stress-libsql
save-if: ${{ github.event_name == 'schedule' || github.event_name == 'merge_group' }}
# Saved by the nightly scan only (this workflow has no push trigger);
# merge_group saves land on a queue ref nothing else can restore.
save-if: ${{ github.event_name == 'schedule' }}
- name: Run fast libsql bottleneck suite
if: github.event_name == 'pull_request' || github.event_name == 'merge_group'

View File

@@ -176,9 +176,9 @@ jobs:
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
with:
key: hooks-parity
# Keep saves to protected-branch and merge_group runs so the ~10 GB
# repo cache LRU is seeded by shared states, not arbitrary PR branches.
save-if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group' }}
# Saved from pushes to main only (merge_group saves land on a queue
# ref nothing else can restore; see reborn-tests.yml).
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
# Full parity matrix across all three backends. Backend implementations
# always compile; --features integration adds the multi-host adversarial
# suite, and IRONCLAW_REQUIRE_POSTGRES=1 (set above) makes the Postgres
@@ -280,9 +280,9 @@ jobs:
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
with:
key: wasm-extensions
# Keep saves to protected-branch and merge_group runs so the ~10 GB
# repo cache LRU is seeded by shared states, not arbitrary PR branches.
save-if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group' }}
# Saved from pushes to main only (merge_group saves land on a queue
# ref nothing else can restore; see reborn-tests.yml).
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
- name: Install cargo-component
uses: ./.github/actions/install-cargo-component
- name: Build all WASM extensions against current WIT

View File

@@ -133,7 +133,9 @@ jobs:
timeout-minutes: 35
env:
# Full-runtime debug futures exceed libtest's default thread stack in
# merged-state runs. Match the root Reborn lane's measured headroom.
# merged-state runs. Match the root Reborn lane's measured headroom;
# job-level and identical across the hermetic lanes so this job shares
# reborn-tests.yml's `reborn-hermetic` cache key (see its crate-tests).
RUST_MIN_STACK: "67108864"
strategy:
fail-fast: false
@@ -158,8 +160,16 @@ jobs:
- name: Restore Rust cache
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
with:
key: reborn-e2e-${{ matrix.group }}
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
# The same hermetic stable+mold build as every guarded Tests (Reborn)
# lane, so the four groups share that workflow's `reborn-hermetic`
# lineage instead of minting four 1.3-1.5 GB per-group entries.
# Restore-only: a shared key is written once per lockfile by the
# first job to finish, so only the jobs that build the complete
# integration-test closure (root partitions and QA replay in
# reborn-tests.yml) may save it; a small group finishing first would
# otherwise pin a partial dependency set.
shared-key: reborn-hermetic
save-if: false
- name: Run deterministic Reborn Rust gate
env:

View File

@@ -230,16 +230,21 @@ jobs:
LLM_USE_CODEX_AUTH: "false"
OLLAMA_BASE_URL: ""
OPENAI_API_KEY: ""
RUSTC_BOOTSTRAP: "1"
# The deep composed-runtime futures (turn -> capability -> skills script
# mount, and the channel admission chain) overflow the 2 MiB default
# test-thread stack on this lane's Linux codegen — first observed as a
# SIGABRT in composition-core's
# `the_model_runs_a_skills_script_from_the_workdir_the_body_advertises`.
# 8 MiB matches the integration-lane headroom below; the deepest
# subtrees stay Box::pin'd at their seams — this is headroom, not a
# substitute.
RUST_MIN_STACK: "8388608"
# `the_model_runs_a_skills_script_from_the_workdir_the_body_advertises`;
# 8 MiB was the measured headroom, and the deepest subtrees stay
# Box::pin'd at their seams — this is headroom, not a substitute.
# Job-level on purpose, and the same 64 MiB in every hermetic lane:
# rust-cache hashes job-level RUST* variables into its cache key, so a
# per-lane value would split the shared `reborn-hermetic` lineage into
# keys no other lane can restore, and
# tests/reborn_coverage_lane_stack_headroom.rs reads the job-level value.
# A larger minimum than a lane strictly needs is a virtual reservation
# per thread, not committed memory.
RUST_MIN_STACK: "67108864"
strategy:
fail-fast: false
max-parallel: ${{ github.event_name == 'pull_request' && 3 || 14 }}
@@ -263,7 +268,20 @@ jobs:
docker system prune -af || true
df -h /
# Plain (pull-request, merge-queue, dispatch) runs build on the pinned
# stable toolchain with the same mold RUSTFLAGS as the root, QA and
# integration lanes, so all of them share one build-cache lineage and
# one artifact graph. `cargo check --workspace --all-targets` on 1.98.0
# passes without the nightly crate attributes below, which only the
# instrumented coverage build on main still compiles with.
- name: Install Rust
if: needs.changes.outputs.run_coverage != 'true'
uses: ./.github/actions/setup-rust
with:
mold: true
- name: Install Rust (coverage toolchain)
if: needs.changes.outputs.run_coverage == 'true'
uses: ./.github/actions/setup-rust
with:
toolchain: nightly-2025-11-01
@@ -377,11 +395,25 @@ jobs:
# shared key keeps the registry + shared-dep build in one small,
# always-resident entry, so the registry is downloaded once, not ~60x.
#
# `-cov` suffix: instrumented llvm-cov build is a different artifact than plain `reborn-tests-crates`; separate lineage avoids corrupting whichever restores second (same as `reborn-integration-cov` below).
shared-key: ${{ needs.changes.outputs.run_coverage == 'true' && 'reborn-tests-crates-cov-llvm21' || 'reborn-tests-crates' }}
# Keep saves to protected-branch and merge_group runs so the ~10 GB
# repo cache LRU is seeded by shared states, not arbitrary PR branches.
save-if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group' }}
# `-cov` suffix: instrumented llvm-cov build is a different artifact than a plain build; separate lineage avoids corrupting whichever restores second (same as `reborn-integration-cov` below).
#
# `reborn-hermetic` is the one plain lineage for every lane that
# builds inside scripts/ci/run-hermetic-test-process.sh (crate
# buckets, root partitions, integration batch, QA replay, the Rust
# Reborn E2E groups): same toolchain, same RUSTFLAGS, same stable
# hermetic Cargo home, so one entry serves all of them. Only the
# push run's root partitions and QA replay save it: a shared key is
# written once per lockfile by the first job to finish, and those
# jobs build the same complete integration-test closure, so whichever
# wins saves the same content. Everything else restores.
shared-key: ${{ needs.changes.outputs.run_coverage == 'true' && 'reborn-tests-crates-cov-llvm21' || 'reborn-hermetic' }}
# Saved from pushes to main only; pull-request and merge-queue runs
# restore. A merge_group save lands on the queue entry's own
# gh-readonly-queue ref, which no pull request and no later queue
# entry can restore (GitHub scopes a run to its own ref, its base
# and the default branch), so those saves were write-only and their
# 6-7 GB per entry evicted main's caches from the 10 GB repo limit.
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
- name: Setup OVH sccache
uses: ./.github/actions/setup-sccache-dist
@@ -414,6 +446,11 @@ jobs:
CARGO_PACKAGES: ${{ toJSON(matrix.bucket.cargo_packages || fromJSON('[]')) }}
EXACT_TARGETS: ${{ toJSON(matrix.bucket.exact_targets || fromJSON('[]')) }}
COLLECT_COVERAGE: ${{ needs.changes.outputs.run_coverage }}
# Branch-coverage compatibility envelope for the instrumented push
# run (scripts/ci/check-reborn-branch-coverage-flags.py counts it).
# Step-scoped: the hermetic wrapper drops it either way, and a
# job-level RUST* key would enter the shared rust-cache key.
RUSTC_BOOTSTRAP: "1"
run: |
echo "Running Reborn crate bucket: ${BUCKET_NAME}"
printf '%s\n' "${BUCKET_PACKAGES}" | jq -r '.[] | "- " + .'
@@ -576,6 +613,7 @@ jobs:
# frames of that pipeline overflow the 8 MiB default (measured need:
# ~10 MiB for reborn_qa_smoke_scenarios_e2e). Same pathology and value
# as the documented RUST_MIN_STACK in reborn_qa_recorded_behavior.rs.
# Job-level and identical across the hermetic lanes: see crate-tests.
RUST_MIN_STACK: "67108864"
strategy:
fail-fast: false
@@ -606,14 +644,18 @@ jobs:
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
with:
# All 4 root partitions compile the *same* root-package build (they
# only run different test partitions), so one shared cache is strictly
# better than 4 per-partition copies (~1 GB each) — the cached target
# is fully valid for every partition, and it stops the 4 copies from
# crowding the shared LRU.
shared-key: reborn-tests-root
# Keep saves to protected-branch and merge_group runs so the ~10 GB
# repo cache LRU is seeded by shared states, not arbitrary PR branches.
save-if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group' }}
# only run different test partitions), and that build is the same
# hermetic stable+mold build every other guarded lane makes, so they
# share the one `reborn-hermetic` lineage (see crate-tests above).
# These push-run jobs (and QA replay) are the lineage's producers.
shared-key: reborn-hermetic
# Saved from pushes to main only; pull-request and merge-queue runs
# restore. A merge_group save lands on the queue entry's own
# gh-readonly-queue ref, which no pull request and no later queue
# entry can restore (GitHub scopes a run to its own ref, its base
# and the default branch), so those saves were write-only and their
# 6-7 GB per entry evicted main's caches from the 10 GB repo limit.
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
- name: Setup OVH sccache cache
uses: ./.github/actions/setup-sccache-dist
@@ -659,8 +701,17 @@ jobs:
- name: Restore Rust cache
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
with:
shared-key: reborn-sandbox-docker-tests
save-if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group' }}
# `reborn-direct` is the plain lineage for lanes that run cargo
# directly (no hermetic wrapper, so the host Cargo home): this job
# saves it on push, the merge-queue mutation gate restores it.
shared-key: reborn-direct
# Saved from pushes to main only; pull-request and merge-queue runs
# restore. A merge_group save lands on the queue entry's own
# gh-readonly-queue ref, which no pull request and no later queue
# entry can restore (GitHub scopes a run to its own ref, its base
# and the default branch), so those saves were write-only and their
# 6-7 GB per entry evicted main's caches from the 10 GB repo limit.
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
- name: Prepare sandbox worker and proxy images
run: |
@@ -694,13 +745,13 @@ jobs:
OLLAMA_BASE_URL: ""
OPENAI_API_KEY: ""
CARGO_INCREMENTAL: "0"
RUSTC_BOOTSTRAP: "1"
# llvm-cov instrumentation inflates stack frames; the deep channel
# admission futures (webhook -> pairing gate -> workflow) overflow the
# default 2 MiB test-thread stack under instrumentation only. The
# deepest subtrees are Box::pin'd at their seams; this is headroom for
# the instrumented lane, not a substitute.
RUST_MIN_STACK: "8388608"
# deepest subtrees are Box::pin'd at their seams; 8 MiB was the measured
# headroom for the instrumented lane, not a substitute.
# Job-level and identical across the hermetic lanes: see crate-tests.
RUST_MIN_STACK: "67108864"
strategy:
fail-fast: false
matrix:
@@ -726,7 +777,16 @@ jobs:
- name: Pre-pull Postgres test image
run: docker pull postgres:16-alpine
# Stable for plain runs, nightly only for the instrumented push run —
# the same split as crate-tests above, for the same shared lineage.
- name: Install Rust
if: needs.changes.outputs.run_coverage != 'true'
uses: ./.github/actions/setup-rust
with:
mold: true
- name: Install Rust (coverage toolchain)
if: needs.changes.outputs.run_coverage == 'true'
uses: ./.github/actions/setup-rust
with:
toolchain: nightly-2025-11-01
@@ -785,10 +845,15 @@ jobs:
# cache LRU. A dedicated key (never the plain `reborn-tests-*` keys
# above) keeps an instrumented (llvm-cov) build from ever evicting
# or being evicted by a plain one — the artifacts are incompatible.
shared-key: ${{ needs.changes.outputs.run_coverage == 'true' && 'reborn-integration-cov-llvm21' || 'reborn-integration-tests' }}
# Keep saves to protected-branch and merge_group runs so the ~10 GB
# repo cache LRU is seeded by shared states, not arbitrary PR branches.
save-if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group' }}
# Plain runs share the `reborn-hermetic` lineage (see crate-tests).
shared-key: ${{ needs.changes.outputs.run_coverage == 'true' && 'reborn-integration-cov-llvm21' || 'reborn-hermetic' }}
# Saved from pushes to main only; pull-request and merge-queue runs
# restore. A merge_group save lands on the queue entry's own
# gh-readonly-queue ref, which no pull request and no later queue
# entry can restore (GitHub scopes a run to its own ref, its base
# and the default branch), so those saves were write-only and their
# 6-7 GB per entry evicted main's caches from the 10 GB repo limit.
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
- name: Install cargo-llvm-cov
if: needs.changes.outputs.run_coverage == 'true'
@@ -800,6 +865,11 @@ jobs:
REBORN_COV_LANES_JSON: ${{ toJSON(matrix.batch.lanes) }}
REBORN_COV_LANE_TEST_TIMEOUT: 45m
REBORN_COV_COLLECT: ${{ needs.changes.outputs.run_coverage }}
# Branch-coverage compatibility envelope for the instrumented push
# run (scripts/ci/check-reborn-branch-coverage-flags.py counts it).
# Step-scoped: the hermetic wrapper drops it either way, and a
# job-level RUST* key would enter the shared rust-cache key.
RUSTC_BOOTSTRAP: "1"
run: |
if [[ "${REBORN_COV_COLLECT}" == "true" ]]; then
# A shared instrumented cache can contain profraw from another
@@ -1001,18 +1071,34 @@ jobs:
with:
mold: true
# Restore the direct-cargo lineage the sandbox Docker job saves on push,
# so the baseline build of each named package starts from warm
# dependencies instead of a cold closure per invariant (the cold shape
# set a 40-minute merge-queue wall clock when the gate ran everything).
- name: Restore Rust cache
if: github.event_name == 'merge_group' && steps.scope.outputs.should_run == 'true'
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
with:
shared-key: reborn-direct
save-if: false
- name: Install cargo-mutants
if: github.event_name == 'merge_group' && steps.scope.outputs.should_run == 'true'
uses: taiki-e/install-action@62b0f2dec647a8e604c6a0fda0e38530180dce20 # v2
with:
tool: cargo-mutants@27.1.0
# `--in-place` mutates this disposable checkout and builds in its own
# `target/`, which is what lets the restored cache count; cargo-mutants
# cannot run in place with parallel jobs, so the gate runs one at a
# time against a warm tree instead of three at a time against cold copies.
- name: Run affected named critical invariants
if: github.event_name == 'merge_group' && steps.scope.outputs.should_run == 'true'
run: |
python3 scripts/ci/critical_mutation_gate.py \
--manifest tests/integration/critical-mutation-functions.toml \
--changed-files changed-files.txt
--changed-files changed-files.txt \
--in-place
qa-recorded-fixtures:
name: Reborn QA recorded fixtures
@@ -1035,6 +1121,7 @@ jobs:
# the boundary itself, so any layout shift decides it. The root-tests and
# group-suite jobs already set this for the same pathology; this job was
# the one that ran the documented binary without it.
# Job-level and identical across the hermetic lanes: see crate-tests.
RUST_MIN_STACK: "67108864"
steps:
- name: Checkout repository
@@ -1086,8 +1173,16 @@ jobs:
- name: Restore Rust cache
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
with:
key: reborn-qa-recorded-fixtures
save-if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group' }}
# Same hermetic stable+mold build as the root partitions: one
# shared lineage (see crate-tests), saved by this push-run job.
shared-key: reborn-hermetic
# Saved from pushes to main only; pull-request and merge-queue runs
# restore. A merge_group save lands on the queue entry's own
# gh-readonly-queue ref, which no pull request and no later queue
# entry can restore (GitHub scopes a run to its own ref, its base
# and the default branch), so those saves were write-only and their
# 6-7 GB per entry evicted main's caches from the 10 GB repo limit.
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
- name: Setup OVH sccache cache
uses: ./.github/actions/setup-sccache-dist

View File

@@ -33,12 +33,23 @@ as a network exception.
Every stage runs through `scripts/ci/run-hermetic-test-process.sh`. The boundary
uses a default-deny environment allowlist: only required compiler/tool paths and
named deterministic test controls survive. Cargo receives a temporary home that
links only the existing offline `registry` and `git` caches, never host Cargo
credentials or configuration; the resolved Rust sysroot is placed directly on
named deterministic test controls survive. Cargo receives a suite-owned home
(`ironclaw-hermetic-cargo-home` under `RUNNER_TEMP`, else the temp directory;
`IRONCLAW_HERMETIC_CARGO_HOME` relocates it) that links only the existing
offline `registry` and `git` caches, never host Cargo credentials or
configuration. That home is one stable path per host rather than a fresh
directory per invocation: cargo hashes each registry crate's absolute source
path into its fingerprint, so a per-invocation home invalidated the whole
dependency closure between one guarded command and the next and left every
restored CI build cache stale. Because that home outlives an invocation, the
runner removes any Cargo configuration or credential file from it on entry
(only the two cache links and Cargo's own lock and cache-tracker files may
persist) and refuses an `IRONCLAW_HERMETIC_CARGO_HOME` that resolves to, or
contains, the host Cargo home. The resolved Rust sysroot is placed directly on
`PATH` without exporting the host Rustup home. Compiler output remains in the
repository's explicit `target/` build directory so prebuilt E2E binaries and
incremental CI artifacts keep their documented paths.
incremental CI artifacts keep their documented paths, and so a `target/`
restored from the `reborn-hermetic` Actions cache is fresh inside the boundary.
That boundary:

View File

@@ -84,11 +84,17 @@ def is_inside(path: str, root: str) -> bool:
)
def run(command: list[str], cwd: pathlib.Path) -> subprocess.CompletedProcess[str]:
def run(
command: list[str], cwd: pathlib.Path, *, in_place: bool = False
) -> subprocess.CompletedProcess[str]:
environment = os.environ.copy()
# cargo-mutants creates isolated copies of the source tree. A shared target
# directory can leak incompatible build artifacts into those copies.
environment.pop("CARGO_TARGET_DIR", None)
if not in_place:
# cargo-mutants creates isolated copies of the source tree. A shared
# target directory can leak incompatible build artifacts into those
# copies. In-place runs build in the checkout's own target directory
# on purpose (a restored CI cache lives there), so they keep the
# caller's environment.
environment.pop("CARGO_TARGET_DIR", None)
return subprocess.run(
command,
cwd=cwd,
@@ -223,13 +229,36 @@ def main() -> int:
parser.add_argument("--changed-files", type=pathlib.Path)
parser.add_argument("--all", action="store_true")
parser.add_argument("--selection-only", action="store_true")
parser.add_argument("--jobs", type=int, default=3)
parser.add_argument(
"--jobs",
type=int,
help="parallel cargo-mutants jobs (default 3; not passed with --in-place, "
"which cargo-mutants runs as one job)",
)
parser.add_argument("--timeout", type=int, default=300)
parser.add_argument(
"--in-place",
action="store_true",
help=(
"mutate the checkout in place (cargo-mutants --in-place) so the "
"baseline and every mutant build reuse its warm target directory; "
"CI-only, for a disposable checkout; cargo-mutants runs in-place "
"work as one job and rejects --jobs alongside --in-place"
),
)
args = parser.parse_args()
try:
if args.all and (args.base or args.head or args.changed_files):
raise GateError("--all cannot be combined with diff-selection arguments")
if args.jobs < 1 or args.timeout < 1:
if args.in_place and args.jobs not in (None, 1):
raise GateError(
"--in-place runs one cargo-mutants job at a time (parallel jobs "
"need separate tree copies); drop --jobs or pass --jobs 1"
)
# cargo-mutants 27.1 refuses `--jobs` next to `--in-place` even for a
# value of 1, so in-place runs pass no `--jobs` at all.
jobs = None if args.in_place else (args.jobs if args.jobs is not None else 3)
if (jobs is not None and jobs < 1) or args.timeout < 1:
raise GateError("--jobs and --timeout must be positive integers")
repo_root = args.repo_root.resolve()
entries = load_manifest(args.manifest, repo_root)
@@ -348,8 +377,7 @@ def main() -> int:
pattern,
"--timeout",
str(args.timeout),
"--jobs",
str(args.jobs),
*(("--jobs", str(jobs)) if jobs is not None else ("--in-place",)),
"--output",
temp,
"--annotations",
@@ -361,6 +389,7 @@ def main() -> int:
),
],
repo_root,
in_place=args.in_place,
)
# 0 = all caught, 2 = survivors, and 3 = timeouts. The latter
# two are verdicts that must be read from mutants.out.

View File

@@ -74,12 +74,56 @@ fi
original_home="${HOME:-}"
original_cargo_home="${CARGO_HOME:-${original_home:+${original_home}/.cargo}}"
sanitized_cargo_home="${hermetic_root}/cargo-home"
# The sanitized Cargo home is one stable path per host, deliberately outside
# the per-invocation hermetic root. Cargo hashes the absolute source path of
# every registry crate into that crate's fingerprint, so a Cargo home whose
# path changed on every invocation marked the whole dependency closure
# `PathToSourceChanged` on the next one: the crate buckets compiled their
# closure twice per job (nextest show-config, then nextest run), and a
# restored CI cache could never be fresh inside the boundary. The directory
# still links only the offline `registry` and `git` caches — never host Cargo
# configuration or credentials — and `scripts/ci/test-hermetic-test-process.sh`
# pins both halves. Override with IRONCLAW_HERMETIC_CARGO_HOME for a
# differently placed (never the host's own) Cargo home.
sanitized_cargo_home="${IRONCLAW_HERMETIC_CARGO_HOME:-${temp_parent%/}/ironclaw-hermetic-cargo-home}"
mkdir -p "${sanitized_cargo_home}"
for cargo_cache in registry git; do
if [[ -n "${original_cargo_home}" && -d "${original_cargo_home}/${cargo_cache}" ]]; then
ln -s "${original_cargo_home}/${cargo_cache}" "${sanitized_cargo_home}/${cargo_cache}"
# Compare canonical paths: a symlink or a `..` spelling of the host Cargo home
# must be rejected just like the literal path, or the guarded command would
# read host Cargo configuration and credentials through it. The sanitized
# home may not be an ancestor of the host home either (the scrub below would
# then delete host files).
canonical_dir() {
(cd "$1" 2>/dev/null && pwd -P) || printf '%s\n' "$1"
}
sanitized_cargo_home_real="$(canonical_dir "${sanitized_cargo_home}")"
if [[ -n "${original_cargo_home}" ]]; then
original_cargo_home_real="$(canonical_dir "${original_cargo_home}")"
if [[ "${sanitized_cargo_home_real}" == "${original_cargo_home_real}" \
|| "${original_cargo_home_real}" == "${sanitized_cargo_home_real}"/* ]]; then
echo "IRONCLAW_HERMETIC_CARGO_HOME must not be (or contain) the host Cargo home: ${sanitized_cargo_home} resolves to ${sanitized_cargo_home_real}" >&2
exit 2
fi
fi
# The stable home is shared by consecutive invocations, so anything a guarded
# command could leave behind that Cargo would read next time is removed up
# front: configuration and credential files never exist here. Only the two
# cache links below and Cargo's own lock/cache-tracker files may persist.
for host_cargo_state in config config.toml credentials credentials.toml; do
rm -f "${sanitized_cargo_home_real}/${host_cargo_state}"
done
for cargo_cache in registry git; do
source_cache="${original_cargo_home:+${original_cargo_home}/${cargo_cache}}"
cache_link="${sanitized_cargo_home}/${cargo_cache}"
if [[ -z "${source_cache}" || ! -d "${source_cache}" ]]; then
continue
fi
if [[ -L "${cache_link}" && "$(readlink "${cache_link}")" == "${source_cache}" ]]; then
continue
fi
# Concurrent invocations race to create the same link; whichever wins, the
# link must point at the host cache afterwards.
ln -sfn "${source_cache}" "${cache_link}" 2>/dev/null \
|| [[ -L "${cache_link}" && "$(readlink "${cache_link}")" == "${source_cache}" ]]
done
tool_path="${PATH:-/usr/bin:/bin}"

View File

@@ -87,6 +87,15 @@ if [[ " $* " != *" --cargo-test-arg --lib --cargo-test-arg authorize_contract "*
echo "scoped cargo test args were not forwarded" >&2
exit 8
fi
if [ -n "${STUB_ARGV_FILE:-}" ]; then
printf '%s\n' "$*" >"${STUB_ARGV_FILE}"
fi
# cargo-mutants 27.1's real argument contract: `--in-place` and `--jobs` are
# mutually exclusive (verified with the binary; the merge queue failed on it).
if [[ " $* " == *" --in-place "* && " $* " == *" --jobs "* ]]; then
echo "error: the argument '--in-place' cannot be used with '--jobs <JOBS>'" >&2
exit 1
fi
out=""
pattern=""
while [ "$#" -gt 0 ]; do
@@ -137,16 +146,37 @@ passes=0
failures=0
capture_at() {
local root="$1" mode="$2" manifest="$3" changed="$4"
shift 4
set +e
CAP_OUT="$(PATH="${work}/bin:${PATH}" STUB_MODE="${mode}" python3 "${gate}" \
CAP_OUT="$(PATH="${work}/bin:${PATH}" STUB_MODE="${mode}" \
STUB_ARGV_FILE="${work}/argv.txt" python3 "${gate}" \
--manifest "${manifest}" \
--repo-root "${root}" \
--changed-files "${changed}" 2>&1)"
--changed-files "${changed}" "$@" 2>&1)"
CAP_RC=$?
set -e
}
capture() {
capture_at "${case_root}" "${1}" "${work}/manifest.toml" "${work}/changed.txt"
local mode="$1"
shift
capture_at "${case_root}" "${mode}" "${work}/manifest.toml" "${work}/changed.txt" "$@"
}
check_argv() {
local label="$1" needle="$2" expect="$3"
local argv=""
if [ -f "${work}/argv.txt" ]; then
argv=" $(cat "${work}/argv.txt") "
fi
if [ "${expect}" = present ] && [[ "${argv}" == *" ${needle} "* ]]; then
echo " ok ${label}"
passes=$((passes + 1))
elif [ "${expect}" = absent ] && [[ "${argv}" != *" ${needle} "* ]]; then
echo " ok ${label}"
passes=$((passes + 1))
else
echo " FAIL ${label}: cargo-mutants argv was: ${argv}" >&2
failures=$((failures + 1))
fi
}
check_rc() {
local label="$1" expected="$2"
@@ -175,6 +205,22 @@ echo "▶ named critical gate happy path"
capture caught
check_rc "all named mutants caught passes" 0
check_text "pass summary refuses a score" "0 survived; 0 timed out"
check_argv "copy mode keeps three parallel tree copies" "--jobs 3" present
check_argv "copy mode does not mutate the checkout" "--in-place" absent
echo "▶ in-place mode reuses the checkout's warm target directory"
rm -f "${work}/argv.txt"
capture caught --in-place
check_rc "in-place run passes" 0
check_argv "in-place is forwarded to cargo-mutants" "--in-place" present
check_argv "in-place passes no --jobs (cargo-mutants rejects the pair)" "--jobs" absent
capture caught --in-place --jobs 3
check_rc "in-place with parallel jobs is refused" 1
check_text "the refusal explains the tree-copy constraint" "one cargo-mutants job at a time"
rm -f "${work}/argv.txt"
capture caught --in-place --jobs 1
check_rc "in-place with an explicit single job passes" 0
check_argv "an explicit single job is still not forwarded in place" "--jobs" absent
echo "▶ survivor and timeout sabotage"
capture missed

View File

@@ -37,6 +37,7 @@ run_probe() {
REBORN_COV_COLLECT="false" \
IRONCLAW_E2E_EMULATE_SLACK_CHANNEL_BEARER="emulate-slack-channel-token" \
IRONCLAW_HERMETIC_SABOTAGE="${sabotage}" \
RUNNER_TEMP="${probe_dir}" \
"${runner}" -- bash -c '
set -euo pipefail
for key in \
@@ -59,13 +60,24 @@ run_probe() {
fi
done
# The Cargo home must be sanitized (no host config or credentials) but
# must NOT live under the per-invocation root: cargo hashes the
# absolute source path of every registry crate into its fingerprint, so
# a per-invocation Cargo home marks the whole dependency closure
# PathToSourceChanged on the next invocation and a cached target
# directory can never be fresh. Stability is asserted outside.
case "${CARGO_HOME}" in
"${IRONCLAW_HERMETIC_ROOT}"/*) ;;
*)
echo "CARGO_HOME is outside the hermetic root: ${CARGO_HOME}" >&2
"${IRONCLAW_HERMETIC_ROOT}"/*)
echo "CARGO_HOME lives under the per-invocation hermetic root: ${CARGO_HOME}" >&2
exit 38
;;
esac
for host_cargo_file in config config.toml credentials credentials.toml; do
if [[ -e "${CARGO_HOME}/${host_cargo_file}" ]]; then
echo "hermetic CARGO_HOME exposes host Cargo state: ${CARGO_HOME}/${host_cargo_file}" >&2
exit 43
fi
done
if [[ -n "${RUSTUP_HOME+x}" ]]; then
echo "ambient RUSTUP_HOME leaked into the hermetic process" >&2
exit 39
@@ -129,16 +141,47 @@ run_probe() {
exit 34
fi
printf "%s\n" "${IRONCLAW_HERMETIC_ROOT}"
printf "%s\t%s\n" "${IRONCLAW_HERMETIC_ROOT}" "${CARGO_HOME}"
'
}
first_root="$(run_probe)"
second_root="$(run_probe)"
first_probe="$(run_probe)"
second_probe="$(run_probe)"
first_root="${first_probe%%$'\t'*}"
second_root="${second_probe%%$'\t'*}"
first_cargo_home="${first_probe#*$'\t'}"
second_cargo_home="${second_probe#*$'\t'}"
if [[ "${first_root}" == "${second_root}" ]]; then
echo "hermetic invocations reused mutable state root: ${first_root}" >&2
exit 1
fi
# Regression guard for the double-compile: the sanitized Cargo home is one
# stable path per host, so cargo fingerprints (which hash each registry
# crate's absolute source path) survive from one invocation to the next and
# from a restored CI cache into the job that restored it.
if [[ "${first_cargo_home}" != "${second_cargo_home}" ]]; then
echo "hermetic CARGO_HOME changed between invocations (every registry crate would be PathToSourceChanged): ${first_cargo_home} vs ${second_cargo_home}" >&2
exit 1
fi
expected_cargo_home="${probe_dir}/ironclaw-hermetic-cargo-home"
if [[ "${first_cargo_home}" != "${expected_cargo_home}" ]]; then
echo "hermetic CARGO_HOME is not the suite-owned path under RUNNER_TEMP: ${first_cargo_home} (expected ${expected_cargo_home})" >&2
exit 1
fi
host_cargo_home="${CARGO_HOME:-${HOME}/.cargo}"
if [[ "${first_cargo_home}" == "${host_cargo_home}" ]]; then
echo "hermetic CARGO_HOME is the host Cargo home: ${first_cargo_home}" >&2
exit 1
fi
for cargo_cache in registry git; do
if [[ -d "${host_cargo_home}/${cargo_cache}" ]]; then
if [[ ! -L "${first_cargo_home}/${cargo_cache}" ]] \
|| [[ "$(readlink "${first_cargo_home}/${cargo_cache}")" != "${host_cargo_home}/${cargo_cache}" ]]; then
echo "hermetic CARGO_HOME does not link the offline host ${cargo_cache} cache" >&2
exit 1
fi
fi
done
parallel_dir="${probe_dir}/parallel-roots"
mkdir -p "${parallel_dir}"
@@ -150,12 +193,87 @@ done
for pid in "${parallel_pids[@]}"; do
wait "${pid}"
done
parallel_root_count="$(LC_ALL=C sort -u "${parallel_dir}"/* | wc -l | tr -d '[:space:]')"
parallel_root_count="$(cut -f1 "${parallel_dir}"/* | LC_ALL=C sort -u | wc -l | tr -d '[:space:]')"
if [[ "${parallel_root_count}" != "4" ]]; then
echo "parallel hermetic invocations did not receive four isolated roots" >&2
LC_ALL=C sort "${parallel_dir}"/* >&2
exit 1
fi
parallel_cargo_home_count="$(cut -f2 "${parallel_dir}"/* | LC_ALL=C sort -u | wc -l | tr -d '[:space:]')"
if [[ "${parallel_cargo_home_count}" != "1" ]]; then
echo "parallel hermetic invocations did not share one stable Cargo home" >&2
LC_ALL=C sort "${parallel_dir}"/* >&2
exit 1
fi
# The stable Cargo home is shared by consecutive invocations, so state one
# guarded command leaves there must not reach the next: Cargo configuration
# and credential files are scrubbed on entry.
RUNNER_TEMP="${probe_dir}" "${runner}" -- bash -c '
set -euo pipefail
printf "[net]\nretry = 99\n" > "${CARGO_HOME}/config.toml"
printf "[registry]\ntoken = \"poison\"\n" > "${CARGO_HOME}/credentials.toml"
: > "${CARGO_HOME}/config"
: > "${CARGO_HOME}/credentials"
'
poison_output="$(
RUNNER_TEMP="${probe_dir}" "${runner}" -- bash -c '
for host_cargo_file in config config.toml credentials credentials.toml; do
if [[ -e "${CARGO_HOME}/${host_cargo_file}" ]]; then
echo "poisoned:${host_cargo_file}"
fi
done
echo "checked"
'
)"
if [[ "${poison_output}" != "checked" ]]; then
echo "Cargo state written by one hermetic invocation reached the next: ${poison_output}" >&2
exit 1
fi
# The host-home guard compares canonical paths: a symlink to the host Cargo
# home, a `..` spelling of it, and an ancestor of it must all be refused.
fake_host_cargo_home="${probe_dir}/host-cargo-home"
mkdir -p "${fake_host_cargo_home}/registry" "${fake_host_cargo_home}/git" "${fake_host_cargo_home}/nested"
ln -s "${fake_host_cargo_home}" "${probe_dir}/host-cargo-home-link"
for bad_cargo_home in \
"${probe_dir}/host-cargo-home-link" \
"${fake_host_cargo_home}/nested/../../host-cargo-home" \
"${probe_dir}"
do
set +e
guard_output="$(
CARGO_HOME="${fake_host_cargo_home}" \
IRONCLAW_HERMETIC_CARGO_HOME="${bad_cargo_home}" \
"${runner}" -- true 2>&1
)"
guard_status=$?
set -e
if [[ "${guard_status}" -ne 2 || "${guard_output}" != *"must not be (or contain) the host Cargo home"* ]]; then
echo "hermetic Cargo home guard accepted a path equivalent to the host Cargo home: ${bad_cargo_home} (status ${guard_status})" >&2
printf '%s\n' "${guard_output}" >&2
exit 1
fi
done
if [[ -e "${fake_host_cargo_home}/config.toml" || ! -d "${fake_host_cargo_home}/registry" ]]; then
echo "the rejected host Cargo home was modified" >&2
exit 1
fi
# The behavioral form of the same guard: a second invocation must compile
# nothing. Builds the smallest workspace crate, so a cold run costs seconds.
# The pre-build outside the boundary fetches its handful of dependencies;
# the guarded runs are offline by design.
if [[ -z "${IRONCLAW_HERMETIC_SELF_TEST_SKIP_CARGO:-}" ]] && command -v cargo >/dev/null 2>&1; then
(cd "${repo_root}" && cargo build -p ironclaw_prompt_envelope --quiet)
RUNNER_TEMP="${probe_dir}" "${runner}" -- cargo build -p ironclaw_prompt_envelope --quiet
second_build="$(RUNNER_TEMP="${probe_dir}" "${runner}" -- cargo build -p ironclaw_prompt_envelope 2>&1)"
if [[ "${second_build}" == *"Compiling"* ]]; then
echo "second hermetic invocation recompiled dependencies; the Cargo home is not stable across invocations:" >&2
printf '%s\n' "${second_build}" >&2
exit 1
fi
fi
no_prepare_bin="${probe_dir}/no-prepare-bin"
mkdir -p "${no_prepare_bin}"