mirror of
https://github.com/nearai/ironclaw.git
synced 2026-09-03 08:06:01 +08:00
ci: stop cold-compiling every Reborn lane (stable hermetic Cargo home, push-only shared caches, stable toolchain, warm in-place mutation gate) (#8050)
* ci: stop cold-compiling every Reborn lane
Three reference runs (PR #8006 at 4bcdc7678a, merge-queue runs
33642416400 and 33656141094) showed every Tests (Reborn) lane compiling
its full dependency closure from cold: 0 of 47 lanes restored a Rust
build cache, and the three heaviest crate buckets compiled the closure
twice in one job (446+452 s, 483+514 s, 381+381 s, identical unit sets).
Compilation was 65% of PR test-lane minutes and 56-62% of queue minutes.
Root cause of the double build and of the dead cache: the hermetic
wrapper minted a fresh CARGO_HOME per invocation. Cargo hashes each
registry crate's absolute source path into its fingerprint, so the second
wrapper invocation in a bucket (nextest show-config, then nextest run)
marked every dependency PathToSourceChanged, and a restored target/ could
never be fresh inside the boundary (queue 33642416424's runtimes group
restored 1451 MB and still compiled 1298 units).
- scripts/ci/run-hermetic-test-process.sh: one stable, suite-owned Cargo
home per host (still only registry/git symlinks, never host config or
credentials). The self-test now pins a stable CARGO_HOME across
sequential and parallel invocations, no host Cargo state inside it, and
that a second guarded build compiles nothing.
- rust-cache saves are push-to-main only (10 sites). A merge_group save
lands on the entry's gh-readonly-queue ref, which no pull request and
no later entry can restore, so those saves were write-only and their
6-7 GB per entry evicted main's caches from the 10 GB limit. Wrapper-
built lanes share one `reborn-hermetic` lineage (crate buckets, root,
integration, QA, Rust Reborn E2E groups); direct-cargo lanes share
`reborn-direct` (sandbox Docker saves, mutation gate restores).
RUST_MIN_STACK / RUSTC_BOOTSTRAP move from job env to step env so the
lanes hash to one key.
- Crate buckets and the integration lane build on the pinned stable
toolchain for PR, queue and dispatch runs; `cargo check --workspace
--all-targets` (and with --all-features) on 1.98.0 passes without the
nightly crate attributes. The instrumented push coverage run keeps
nightly unchanged.
- The merge-queue mutation gate restores the warm cache and runs
cargo-mutants --in-place (new opt-in gate flag, self-tested; implies
--jobs 1) instead of a cold closure build per invariant (14.9 min for
one invariant, 38.8 min for all eleven).
No test, lane, partition, or selection rule is removed or narrowed.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QtLJ4UiSzbjFxnMT1Vbkpj
* ci: keep the branch-coverage RUSTC_BOOTSTRAP envelope, step-scoped
check-reborn-branch-coverage-flags.py counts four exact
`RUSTC_BOOTSTRAP: \"1\"` lines across the coverage workflows; the two
in reborn-tests.yml now live on the crate-tests and integration run
steps, where the instrumented push run executes and where rust-cache
does not hash them into the shared `reborn-hermetic` key.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QtLJ4UiSzbjFxnMT1Vbkpj
* ci: keep RUST_MIN_STACK job-level, one value across the hermetic lanes
tests/reborn_coverage_lane_stack_headroom.rs reads the job-level
RUST_MIN_STACK of every lane that runs the integration package under
llvm-cov and ignores step-level values on purpose, so the step-scoped
placement failed the root partition on the exhaustive dispatch run
(33670885071). The value is job-level again and 64 MiB everywhere the
hermetic wrapper runs (crate buckets and the integration batch up from
8 MiB), because rust-cache hashes job-level RUST* variables into its key
and a per-lane value would split the shared reborn-hermetic lineage; a
larger minimum is a per-thread virtual reservation, not committed
memory. RUSTC_BOOTSTRAP stays step-scoped: the wrapper drops it either
way and the branch-coverage checker only counts the line.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QtLJ4UiSzbjFxnMT1Vbkpj
* ci: harden the stable hermetic Cargo home and single-source its cache
Review follow-ups on #8050:
- Compare canonical paths in the host-Cargo-home guard, so a symlink or a
`..` spelling of the host home is refused like the literal path, and
refuse an ancestor of the host home (the scrub below would otherwise
delete host files). Regression: three equivalent spellings must exit 2
and leave the host home untouched.
- Scrub Cargo config and credential files from the stable Cargo home on
every entry: the home now outlives an invocation, so state one guarded
command left there must not reach the next. Regression: files planted
by one invocation are absent in the next.
- Only the push run's root partitions and QA replay save the
`reborn-hermetic` cache (same complete integration-test closure); the
Rust Reborn E2E groups restore only, so a small group finishing first
cannot pin a partial dependency set under the once-per-lockfile key.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QtLJ4UiSzbjFxnMT1Vbkpj
* ci: pass no --jobs to cargo-mutants in place
cargo-mutants 27.1 rejects --jobs next to --in-place even with a value
of 1 (the merge-queue gate for #8050 failed with "the argument
'--in-place' cannot be used with '--jobs <JOBS>'"). In-place runs now
pass no --jobs at all; the self-test stub enforces the real mutual
exclusion so the pair cannot be emitted again, and asserts --jobs is
absent in both in-place forms.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QtLJ4UiSzbjFxnMT1Vbkpj
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
20
.github/workflows/README.md
vendored
20
.github/workflows/README.md
vendored
@@ -10,6 +10,26 @@ a check belongs to exactly one tier on purpose.
|
||||
| Post-merge confirm | `push` to `main` | Runs exhaustive coverage, warms shared caches, and feeds Codecov/canaries. A main-only deterministic failure means the affected-area classifier missed an impact and must be widened. |
|
||||
| Deep / scheduled | `schedule` (nightly) | Exhaustive suites too slow for the queue: legacy v1 matrix, full browser E2E, stress scans. |
|
||||
|
||||
## Build caches
|
||||
|
||||
Rust build caches (Swatinem/rust-cache) are saved only by pushes to `main`
|
||||
(the stress lane's by its nightly scan) and restored by every other event.
|
||||
GitHub lets a run restore caches from its own ref, its base branch, and the
|
||||
default branch, so a save from a `merge_group` run lands on the queue entry's
|
||||
`gh-readonly-queue/...` ref where no pull request and no later entry can read
|
||||
it; those saves were write-only and evicted `main`'s entries from the 10 GB
|
||||
repository limit. Two plain lineages exist. `reborn-hermetic` is every lane
|
||||
that builds inside `scripts/ci/run-hermetic-test-process.sh` (crate buckets,
|
||||
root partitions, the integration batch, QA replay, the Rust Reborn E2E
|
||||
groups): pinned stable toolchain, mold, and a stable hermetic Cargo home, so
|
||||
one entry is fresh for all of them. A shared key is written once per lockfile
|
||||
by the first job to finish, so only the push run's root partitions and QA
|
||||
replay, which build the same complete integration-test closure, save it;
|
||||
every other job restores. `reborn-direct` is the lanes that run
|
||||
cargo against the host Cargo home (the sandbox Docker tests, which save it,
|
||||
and the merge-queue mutation gate, which restores it). The instrumented
|
||||
coverage lanes on `main` keep their separate `-cov-llvm21` lineages.
|
||||
|
||||
## The invariant
|
||||
|
||||
**No attributable deterministic failure may be main-only.** The merge queue
|
||||
|
||||
12
.github/workflows/code_style.yml
vendored
12
.github/workflows/code_style.yml
vendored
@@ -432,9 +432,9 @@ jobs:
|
||||
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
|
||||
with:
|
||||
shared-key: clippy
|
||||
# Keep saves to protected-branch and merge_group runs so the ~10 GB
|
||||
# repo cache LRU is seeded by shared states, not arbitrary PR branches.
|
||||
save-if: ${{ matrix.name == 'all-features' && ((github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group') }}
|
||||
# Saved from pushes to main only (merge_group saves land on a queue
|
||||
# ref nothing else can restore; see reborn-tests.yml).
|
||||
save-if: ${{ matrix.name == 'all-features' && github.event_name == 'push' && github.ref == 'refs/heads/main' }}
|
||||
- name: Setup OVH sccache
|
||||
uses: ./.github/actions/setup-sccache-dist
|
||||
with:
|
||||
@@ -579,9 +579,9 @@ jobs:
|
||||
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
|
||||
with:
|
||||
shared-key: reborn-cli-smoke
|
||||
# Keep saves to protected-branch and merge_group runs so the ~10 GB
|
||||
# repo cache LRU is seeded by shared states, not arbitrary PR branches.
|
||||
save-if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group' }}
|
||||
# Saved from pushes to main only (merge_group saves land on a queue
|
||||
# ref nothing else can restore; see reborn-tests.yml).
|
||||
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
|
||||
- name: Setup OVH sccache
|
||||
uses: ./.github/actions/setup-sccache-dist
|
||||
with:
|
||||
|
||||
4
.github/workflows/ironclaw-stress.yml
vendored
4
.github/workflows/ironclaw-stress.yml
vendored
@@ -66,7 +66,9 @@ jobs:
|
||||
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
|
||||
with:
|
||||
key: ironclaw-stress-libsql
|
||||
save-if: ${{ github.event_name == 'schedule' || github.event_name == 'merge_group' }}
|
||||
# Saved by the nightly scan only (this workflow has no push trigger);
|
||||
# merge_group saves land on a queue ref nothing else can restore.
|
||||
save-if: ${{ github.event_name == 'schedule' }}
|
||||
|
||||
- name: Run fast libsql bottleneck suite
|
||||
if: github.event_name == 'pull_request' || github.event_name == 'merge_group'
|
||||
|
||||
12
.github/workflows/platform-and-compat.yml
vendored
12
.github/workflows/platform-and-compat.yml
vendored
@@ -176,9 +176,9 @@ jobs:
|
||||
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
|
||||
with:
|
||||
key: hooks-parity
|
||||
# Keep saves to protected-branch and merge_group runs so the ~10 GB
|
||||
# repo cache LRU is seeded by shared states, not arbitrary PR branches.
|
||||
save-if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group' }}
|
||||
# Saved from pushes to main only (merge_group saves land on a queue
|
||||
# ref nothing else can restore; see reborn-tests.yml).
|
||||
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
|
||||
# Full parity matrix across all three backends. Backend implementations
|
||||
# always compile; --features integration adds the multi-host adversarial
|
||||
# suite, and IRONCLAW_REQUIRE_POSTGRES=1 (set above) makes the Postgres
|
||||
@@ -280,9 +280,9 @@ jobs:
|
||||
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
|
||||
with:
|
||||
key: wasm-extensions
|
||||
# Keep saves to protected-branch and merge_group runs so the ~10 GB
|
||||
# repo cache LRU is seeded by shared states, not arbitrary PR branches.
|
||||
save-if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group' }}
|
||||
# Saved from pushes to main only (merge_group saves land on a queue
|
||||
# ref nothing else can restore; see reborn-tests.yml).
|
||||
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
|
||||
- name: Install cargo-component
|
||||
uses: ./.github/actions/install-cargo-component
|
||||
- name: Build all WASM extensions against current WIT
|
||||
|
||||
16
.github/workflows/reborn-e2e.yml
vendored
16
.github/workflows/reborn-e2e.yml
vendored
@@ -133,7 +133,9 @@ jobs:
|
||||
timeout-minutes: 35
|
||||
env:
|
||||
# Full-runtime debug futures exceed libtest's default thread stack in
|
||||
# merged-state runs. Match the root Reborn lane's measured headroom.
|
||||
# merged-state runs. Match the root Reborn lane's measured headroom;
|
||||
# job-level and identical across the hermetic lanes so this job shares
|
||||
# reborn-tests.yml's `reborn-hermetic` cache key (see its crate-tests).
|
||||
RUST_MIN_STACK: "67108864"
|
||||
strategy:
|
||||
fail-fast: false
|
||||
@@ -158,8 +160,16 @@ jobs:
|
||||
- name: Restore Rust cache
|
||||
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
|
||||
with:
|
||||
key: reborn-e2e-${{ matrix.group }}
|
||||
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
|
||||
# The same hermetic stable+mold build as every guarded Tests (Reborn)
|
||||
# lane, so the four groups share that workflow's `reborn-hermetic`
|
||||
# lineage instead of minting four 1.3-1.5 GB per-group entries.
|
||||
# Restore-only: a shared key is written once per lockfile by the
|
||||
# first job to finish, so only the jobs that build the complete
|
||||
# integration-test closure (root partitions and QA replay in
|
||||
# reborn-tests.yml) may save it; a small group finishing first would
|
||||
# otherwise pin a partial dependency set.
|
||||
shared-key: reborn-hermetic
|
||||
save-if: false
|
||||
|
||||
- name: Run deterministic Reborn Rust gate
|
||||
env:
|
||||
|
||||
159
.github/workflows/reborn-tests.yml
vendored
159
.github/workflows/reborn-tests.yml
vendored
@@ -230,16 +230,21 @@ jobs:
|
||||
LLM_USE_CODEX_AUTH: "false"
|
||||
OLLAMA_BASE_URL: ""
|
||||
OPENAI_API_KEY: ""
|
||||
RUSTC_BOOTSTRAP: "1"
|
||||
# The deep composed-runtime futures (turn -> capability -> skills script
|
||||
# mount, and the channel admission chain) overflow the 2 MiB default
|
||||
# test-thread stack on this lane's Linux codegen — first observed as a
|
||||
# SIGABRT in composition-core's
|
||||
# `the_model_runs_a_skills_script_from_the_workdir_the_body_advertises`.
|
||||
# 8 MiB matches the integration-lane headroom below; the deepest
|
||||
# subtrees stay Box::pin'd at their seams — this is headroom, not a
|
||||
# substitute.
|
||||
RUST_MIN_STACK: "8388608"
|
||||
# `the_model_runs_a_skills_script_from_the_workdir_the_body_advertises`;
|
||||
# 8 MiB was the measured headroom, and the deepest subtrees stay
|
||||
# Box::pin'd at their seams — this is headroom, not a substitute.
|
||||
# Job-level on purpose, and the same 64 MiB in every hermetic lane:
|
||||
# rust-cache hashes job-level RUST* variables into its cache key, so a
|
||||
# per-lane value would split the shared `reborn-hermetic` lineage into
|
||||
# keys no other lane can restore, and
|
||||
# tests/reborn_coverage_lane_stack_headroom.rs reads the job-level value.
|
||||
# A larger minimum than a lane strictly needs is a virtual reservation
|
||||
# per thread, not committed memory.
|
||||
RUST_MIN_STACK: "67108864"
|
||||
strategy:
|
||||
fail-fast: false
|
||||
max-parallel: ${{ github.event_name == 'pull_request' && 3 || 14 }}
|
||||
@@ -263,7 +268,20 @@ jobs:
|
||||
docker system prune -af || true
|
||||
df -h /
|
||||
|
||||
# Plain (pull-request, merge-queue, dispatch) runs build on the pinned
|
||||
# stable toolchain with the same mold RUSTFLAGS as the root, QA and
|
||||
# integration lanes, so all of them share one build-cache lineage and
|
||||
# one artifact graph. `cargo check --workspace --all-targets` on 1.98.0
|
||||
# passes without the nightly crate attributes below, which only the
|
||||
# instrumented coverage build on main still compiles with.
|
||||
- name: Install Rust
|
||||
if: needs.changes.outputs.run_coverage != 'true'
|
||||
uses: ./.github/actions/setup-rust
|
||||
with:
|
||||
mold: true
|
||||
|
||||
- name: Install Rust (coverage toolchain)
|
||||
if: needs.changes.outputs.run_coverage == 'true'
|
||||
uses: ./.github/actions/setup-rust
|
||||
with:
|
||||
toolchain: nightly-2025-11-01
|
||||
@@ -377,11 +395,25 @@ jobs:
|
||||
# shared key keeps the registry + shared-dep build in one small,
|
||||
# always-resident entry, so the registry is downloaded once, not ~60x.
|
||||
#
|
||||
# `-cov` suffix: instrumented llvm-cov build is a different artifact than plain `reborn-tests-crates`; separate lineage avoids corrupting whichever restores second (same as `reborn-integration-cov` below).
|
||||
shared-key: ${{ needs.changes.outputs.run_coverage == 'true' && 'reborn-tests-crates-cov-llvm21' || 'reborn-tests-crates' }}
|
||||
# Keep saves to protected-branch and merge_group runs so the ~10 GB
|
||||
# repo cache LRU is seeded by shared states, not arbitrary PR branches.
|
||||
save-if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group' }}
|
||||
# `-cov` suffix: instrumented llvm-cov build is a different artifact than a plain build; separate lineage avoids corrupting whichever restores second (same as `reborn-integration-cov` below).
|
||||
#
|
||||
# `reborn-hermetic` is the one plain lineage for every lane that
|
||||
# builds inside scripts/ci/run-hermetic-test-process.sh (crate
|
||||
# buckets, root partitions, integration batch, QA replay, the Rust
|
||||
# Reborn E2E groups): same toolchain, same RUSTFLAGS, same stable
|
||||
# hermetic Cargo home, so one entry serves all of them. Only the
|
||||
# push run's root partitions and QA replay save it: a shared key is
|
||||
# written once per lockfile by the first job to finish, and those
|
||||
# jobs build the same complete integration-test closure, so whichever
|
||||
# wins saves the same content. Everything else restores.
|
||||
shared-key: ${{ needs.changes.outputs.run_coverage == 'true' && 'reborn-tests-crates-cov-llvm21' || 'reborn-hermetic' }}
|
||||
# Saved from pushes to main only; pull-request and merge-queue runs
|
||||
# restore. A merge_group save lands on the queue entry's own
|
||||
# gh-readonly-queue ref, which no pull request and no later queue
|
||||
# entry can restore (GitHub scopes a run to its own ref, its base
|
||||
# and the default branch), so those saves were write-only and their
|
||||
# 6-7 GB per entry evicted main's caches from the 10 GB repo limit.
|
||||
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
|
||||
|
||||
- name: Setup OVH sccache
|
||||
uses: ./.github/actions/setup-sccache-dist
|
||||
@@ -414,6 +446,11 @@ jobs:
|
||||
CARGO_PACKAGES: ${{ toJSON(matrix.bucket.cargo_packages || fromJSON('[]')) }}
|
||||
EXACT_TARGETS: ${{ toJSON(matrix.bucket.exact_targets || fromJSON('[]')) }}
|
||||
COLLECT_COVERAGE: ${{ needs.changes.outputs.run_coverage }}
|
||||
# Branch-coverage compatibility envelope for the instrumented push
|
||||
# run (scripts/ci/check-reborn-branch-coverage-flags.py counts it).
|
||||
# Step-scoped: the hermetic wrapper drops it either way, and a
|
||||
# job-level RUST* key would enter the shared rust-cache key.
|
||||
RUSTC_BOOTSTRAP: "1"
|
||||
run: |
|
||||
echo "Running Reborn crate bucket: ${BUCKET_NAME}"
|
||||
printf '%s\n' "${BUCKET_PACKAGES}" | jq -r '.[] | "- " + .'
|
||||
@@ -576,6 +613,7 @@ jobs:
|
||||
# frames of that pipeline overflow the 8 MiB default (measured need:
|
||||
# ~10 MiB for reborn_qa_smoke_scenarios_e2e). Same pathology and value
|
||||
# as the documented RUST_MIN_STACK in reborn_qa_recorded_behavior.rs.
|
||||
# Job-level and identical across the hermetic lanes: see crate-tests.
|
||||
RUST_MIN_STACK: "67108864"
|
||||
strategy:
|
||||
fail-fast: false
|
||||
@@ -606,14 +644,18 @@ jobs:
|
||||
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
|
||||
with:
|
||||
# All 4 root partitions compile the *same* root-package build (they
|
||||
# only run different test partitions), so one shared cache is strictly
|
||||
# better than 4 per-partition copies (~1 GB each) — the cached target
|
||||
# is fully valid for every partition, and it stops the 4 copies from
|
||||
# crowding the shared LRU.
|
||||
shared-key: reborn-tests-root
|
||||
# Keep saves to protected-branch and merge_group runs so the ~10 GB
|
||||
# repo cache LRU is seeded by shared states, not arbitrary PR branches.
|
||||
save-if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group' }}
|
||||
# only run different test partitions), and that build is the same
|
||||
# hermetic stable+mold build every other guarded lane makes, so they
|
||||
# share the one `reborn-hermetic` lineage (see crate-tests above).
|
||||
# These push-run jobs (and QA replay) are the lineage's producers.
|
||||
shared-key: reborn-hermetic
|
||||
# Saved from pushes to main only; pull-request and merge-queue runs
|
||||
# restore. A merge_group save lands on the queue entry's own
|
||||
# gh-readonly-queue ref, which no pull request and no later queue
|
||||
# entry can restore (GitHub scopes a run to its own ref, its base
|
||||
# and the default branch), so those saves were write-only and their
|
||||
# 6-7 GB per entry evicted main's caches from the 10 GB repo limit.
|
||||
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
|
||||
|
||||
- name: Setup OVH sccache cache
|
||||
uses: ./.github/actions/setup-sccache-dist
|
||||
@@ -659,8 +701,17 @@ jobs:
|
||||
- name: Restore Rust cache
|
||||
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
|
||||
with:
|
||||
shared-key: reborn-sandbox-docker-tests
|
||||
save-if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group' }}
|
||||
# `reborn-direct` is the plain lineage for lanes that run cargo
|
||||
# directly (no hermetic wrapper, so the host Cargo home): this job
|
||||
# saves it on push, the merge-queue mutation gate restores it.
|
||||
shared-key: reborn-direct
|
||||
# Saved from pushes to main only; pull-request and merge-queue runs
|
||||
# restore. A merge_group save lands on the queue entry's own
|
||||
# gh-readonly-queue ref, which no pull request and no later queue
|
||||
# entry can restore (GitHub scopes a run to its own ref, its base
|
||||
# and the default branch), so those saves were write-only and their
|
||||
# 6-7 GB per entry evicted main's caches from the 10 GB repo limit.
|
||||
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
|
||||
|
||||
- name: Prepare sandbox worker and proxy images
|
||||
run: |
|
||||
@@ -694,13 +745,13 @@ jobs:
|
||||
OLLAMA_BASE_URL: ""
|
||||
OPENAI_API_KEY: ""
|
||||
CARGO_INCREMENTAL: "0"
|
||||
RUSTC_BOOTSTRAP: "1"
|
||||
# llvm-cov instrumentation inflates stack frames; the deep channel
|
||||
# admission futures (webhook -> pairing gate -> workflow) overflow the
|
||||
# default 2 MiB test-thread stack under instrumentation only. The
|
||||
# deepest subtrees are Box::pin'd at their seams; this is headroom for
|
||||
# the instrumented lane, not a substitute.
|
||||
RUST_MIN_STACK: "8388608"
|
||||
# deepest subtrees are Box::pin'd at their seams; 8 MiB was the measured
|
||||
# headroom for the instrumented lane, not a substitute.
|
||||
# Job-level and identical across the hermetic lanes: see crate-tests.
|
||||
RUST_MIN_STACK: "67108864"
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
@@ -726,7 +777,16 @@ jobs:
|
||||
- name: Pre-pull Postgres test image
|
||||
run: docker pull postgres:16-alpine
|
||||
|
||||
# Stable for plain runs, nightly only for the instrumented push run —
|
||||
# the same split as crate-tests above, for the same shared lineage.
|
||||
- name: Install Rust
|
||||
if: needs.changes.outputs.run_coverage != 'true'
|
||||
uses: ./.github/actions/setup-rust
|
||||
with:
|
||||
mold: true
|
||||
|
||||
- name: Install Rust (coverage toolchain)
|
||||
if: needs.changes.outputs.run_coverage == 'true'
|
||||
uses: ./.github/actions/setup-rust
|
||||
with:
|
||||
toolchain: nightly-2025-11-01
|
||||
@@ -785,10 +845,15 @@ jobs:
|
||||
# cache LRU. A dedicated key (never the plain `reborn-tests-*` keys
|
||||
# above) keeps an instrumented (llvm-cov) build from ever evicting
|
||||
# or being evicted by a plain one — the artifacts are incompatible.
|
||||
shared-key: ${{ needs.changes.outputs.run_coverage == 'true' && 'reborn-integration-cov-llvm21' || 'reborn-integration-tests' }}
|
||||
# Keep saves to protected-branch and merge_group runs so the ~10 GB
|
||||
# repo cache LRU is seeded by shared states, not arbitrary PR branches.
|
||||
save-if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group' }}
|
||||
# Plain runs share the `reborn-hermetic` lineage (see crate-tests).
|
||||
shared-key: ${{ needs.changes.outputs.run_coverage == 'true' && 'reborn-integration-cov-llvm21' || 'reborn-hermetic' }}
|
||||
# Saved from pushes to main only; pull-request and merge-queue runs
|
||||
# restore. A merge_group save lands on the queue entry's own
|
||||
# gh-readonly-queue ref, which no pull request and no later queue
|
||||
# entry can restore (GitHub scopes a run to its own ref, its base
|
||||
# and the default branch), so those saves were write-only and their
|
||||
# 6-7 GB per entry evicted main's caches from the 10 GB repo limit.
|
||||
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
|
||||
|
||||
- name: Install cargo-llvm-cov
|
||||
if: needs.changes.outputs.run_coverage == 'true'
|
||||
@@ -800,6 +865,11 @@ jobs:
|
||||
REBORN_COV_LANES_JSON: ${{ toJSON(matrix.batch.lanes) }}
|
||||
REBORN_COV_LANE_TEST_TIMEOUT: 45m
|
||||
REBORN_COV_COLLECT: ${{ needs.changes.outputs.run_coverage }}
|
||||
# Branch-coverage compatibility envelope for the instrumented push
|
||||
# run (scripts/ci/check-reborn-branch-coverage-flags.py counts it).
|
||||
# Step-scoped: the hermetic wrapper drops it either way, and a
|
||||
# job-level RUST* key would enter the shared rust-cache key.
|
||||
RUSTC_BOOTSTRAP: "1"
|
||||
run: |
|
||||
if [[ "${REBORN_COV_COLLECT}" == "true" ]]; then
|
||||
# A shared instrumented cache can contain profraw from another
|
||||
@@ -1001,18 +1071,34 @@ jobs:
|
||||
with:
|
||||
mold: true
|
||||
|
||||
# Restore the direct-cargo lineage the sandbox Docker job saves on push,
|
||||
# so the baseline build of each named package starts from warm
|
||||
# dependencies instead of a cold closure per invariant (the cold shape
|
||||
# set a 40-minute merge-queue wall clock when the gate ran everything).
|
||||
- name: Restore Rust cache
|
||||
if: github.event_name == 'merge_group' && steps.scope.outputs.should_run == 'true'
|
||||
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
|
||||
with:
|
||||
shared-key: reborn-direct
|
||||
save-if: false
|
||||
|
||||
- name: Install cargo-mutants
|
||||
if: github.event_name == 'merge_group' && steps.scope.outputs.should_run == 'true'
|
||||
uses: taiki-e/install-action@62b0f2dec647a8e604c6a0fda0e38530180dce20 # v2
|
||||
with:
|
||||
tool: cargo-mutants@27.1.0
|
||||
|
||||
# `--in-place` mutates this disposable checkout and builds in its own
|
||||
# `target/`, which is what lets the restored cache count; cargo-mutants
|
||||
# cannot run in place with parallel jobs, so the gate runs one at a
|
||||
# time against a warm tree instead of three at a time against cold copies.
|
||||
- name: Run affected named critical invariants
|
||||
if: github.event_name == 'merge_group' && steps.scope.outputs.should_run == 'true'
|
||||
run: |
|
||||
python3 scripts/ci/critical_mutation_gate.py \
|
||||
--manifest tests/integration/critical-mutation-functions.toml \
|
||||
--changed-files changed-files.txt
|
||||
--changed-files changed-files.txt \
|
||||
--in-place
|
||||
|
||||
qa-recorded-fixtures:
|
||||
name: Reborn QA recorded fixtures
|
||||
@@ -1035,6 +1121,7 @@ jobs:
|
||||
# the boundary itself, so any layout shift decides it. The root-tests and
|
||||
# group-suite jobs already set this for the same pathology; this job was
|
||||
# the one that ran the documented binary without it.
|
||||
# Job-level and identical across the hermetic lanes: see crate-tests.
|
||||
RUST_MIN_STACK: "67108864"
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
@@ -1086,8 +1173,16 @@ jobs:
|
||||
- name: Restore Rust cache
|
||||
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
|
||||
with:
|
||||
key: reborn-qa-recorded-fixtures
|
||||
save-if: ${{ (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'merge_group' }}
|
||||
# Same hermetic stable+mold build as the root partitions: one
|
||||
# shared lineage (see crate-tests), saved by this push-run job.
|
||||
shared-key: reborn-hermetic
|
||||
# Saved from pushes to main only; pull-request and merge-queue runs
|
||||
# restore. A merge_group save lands on the queue entry's own
|
||||
# gh-readonly-queue ref, which no pull request and no later queue
|
||||
# entry can restore (GitHub scopes a run to its own ref, its base
|
||||
# and the default branch), so those saves were write-only and their
|
||||
# 6-7 GB per entry evicted main's caches from the 10 GB repo limit.
|
||||
save-if: ${{ github.event_name == 'push' && github.ref == 'refs/heads/main' }}
|
||||
|
||||
- name: Setup OVH sccache cache
|
||||
uses: ./.github/actions/setup-sccache-dist
|
||||
|
||||
@@ -33,12 +33,23 @@ as a network exception.
|
||||
|
||||
Every stage runs through `scripts/ci/run-hermetic-test-process.sh`. The boundary
|
||||
uses a default-deny environment allowlist: only required compiler/tool paths and
|
||||
named deterministic test controls survive. Cargo receives a temporary home that
|
||||
links only the existing offline `registry` and `git` caches, never host Cargo
|
||||
credentials or configuration; the resolved Rust sysroot is placed directly on
|
||||
named deterministic test controls survive. Cargo receives a suite-owned home
|
||||
(`ironclaw-hermetic-cargo-home` under `RUNNER_TEMP`, else the temp directory;
|
||||
`IRONCLAW_HERMETIC_CARGO_HOME` relocates it) that links only the existing
|
||||
offline `registry` and `git` caches, never host Cargo credentials or
|
||||
configuration. That home is one stable path per host rather than a fresh
|
||||
directory per invocation: cargo hashes each registry crate's absolute source
|
||||
path into its fingerprint, so a per-invocation home invalidated the whole
|
||||
dependency closure between one guarded command and the next and left every
|
||||
restored CI build cache stale. Because that home outlives an invocation, the
|
||||
runner removes any Cargo configuration or credential file from it on entry
|
||||
(only the two cache links and Cargo's own lock and cache-tracker files may
|
||||
persist) and refuses an `IRONCLAW_HERMETIC_CARGO_HOME` that resolves to, or
|
||||
contains, the host Cargo home. The resolved Rust sysroot is placed directly on
|
||||
`PATH` without exporting the host Rustup home. Compiler output remains in the
|
||||
repository's explicit `target/` build directory so prebuilt E2E binaries and
|
||||
incremental CI artifacts keep their documented paths.
|
||||
incremental CI artifacts keep their documented paths, and so a `target/`
|
||||
restored from the `reborn-hermetic` Actions cache is fresh inside the boundary.
|
||||
|
||||
That boundary:
|
||||
|
||||
|
||||
@@ -84,11 +84,17 @@ def is_inside(path: str, root: str) -> bool:
|
||||
)
|
||||
|
||||
|
||||
def run(command: list[str], cwd: pathlib.Path) -> subprocess.CompletedProcess[str]:
|
||||
def run(
|
||||
command: list[str], cwd: pathlib.Path, *, in_place: bool = False
|
||||
) -> subprocess.CompletedProcess[str]:
|
||||
environment = os.environ.copy()
|
||||
# cargo-mutants creates isolated copies of the source tree. A shared target
|
||||
# directory can leak incompatible build artifacts into those copies.
|
||||
environment.pop("CARGO_TARGET_DIR", None)
|
||||
if not in_place:
|
||||
# cargo-mutants creates isolated copies of the source tree. A shared
|
||||
# target directory can leak incompatible build artifacts into those
|
||||
# copies. In-place runs build in the checkout's own target directory
|
||||
# on purpose (a restored CI cache lives there), so they keep the
|
||||
# caller's environment.
|
||||
environment.pop("CARGO_TARGET_DIR", None)
|
||||
return subprocess.run(
|
||||
command,
|
||||
cwd=cwd,
|
||||
@@ -223,13 +229,36 @@ def main() -> int:
|
||||
parser.add_argument("--changed-files", type=pathlib.Path)
|
||||
parser.add_argument("--all", action="store_true")
|
||||
parser.add_argument("--selection-only", action="store_true")
|
||||
parser.add_argument("--jobs", type=int, default=3)
|
||||
parser.add_argument(
|
||||
"--jobs",
|
||||
type=int,
|
||||
help="parallel cargo-mutants jobs (default 3; not passed with --in-place, "
|
||||
"which cargo-mutants runs as one job)",
|
||||
)
|
||||
parser.add_argument("--timeout", type=int, default=300)
|
||||
parser.add_argument(
|
||||
"--in-place",
|
||||
action="store_true",
|
||||
help=(
|
||||
"mutate the checkout in place (cargo-mutants --in-place) so the "
|
||||
"baseline and every mutant build reuse its warm target directory; "
|
||||
"CI-only, for a disposable checkout; cargo-mutants runs in-place "
|
||||
"work as one job and rejects --jobs alongside --in-place"
|
||||
),
|
||||
)
|
||||
args = parser.parse_args()
|
||||
try:
|
||||
if args.all and (args.base or args.head or args.changed_files):
|
||||
raise GateError("--all cannot be combined with diff-selection arguments")
|
||||
if args.jobs < 1 or args.timeout < 1:
|
||||
if args.in_place and args.jobs not in (None, 1):
|
||||
raise GateError(
|
||||
"--in-place runs one cargo-mutants job at a time (parallel jobs "
|
||||
"need separate tree copies); drop --jobs or pass --jobs 1"
|
||||
)
|
||||
# cargo-mutants 27.1 refuses `--jobs` next to `--in-place` even for a
|
||||
# value of 1, so in-place runs pass no `--jobs` at all.
|
||||
jobs = None if args.in_place else (args.jobs if args.jobs is not None else 3)
|
||||
if (jobs is not None and jobs < 1) or args.timeout < 1:
|
||||
raise GateError("--jobs and --timeout must be positive integers")
|
||||
repo_root = args.repo_root.resolve()
|
||||
entries = load_manifest(args.manifest, repo_root)
|
||||
@@ -348,8 +377,7 @@ def main() -> int:
|
||||
pattern,
|
||||
"--timeout",
|
||||
str(args.timeout),
|
||||
"--jobs",
|
||||
str(args.jobs),
|
||||
*(("--jobs", str(jobs)) if jobs is not None else ("--in-place",)),
|
||||
"--output",
|
||||
temp,
|
||||
"--annotations",
|
||||
@@ -361,6 +389,7 @@ def main() -> int:
|
||||
),
|
||||
],
|
||||
repo_root,
|
||||
in_place=args.in_place,
|
||||
)
|
||||
# 0 = all caught, 2 = survivors, and 3 = timeouts. The latter
|
||||
# two are verdicts that must be read from mutants.out.
|
||||
|
||||
@@ -74,12 +74,56 @@ fi
|
||||
|
||||
original_home="${HOME:-}"
|
||||
original_cargo_home="${CARGO_HOME:-${original_home:+${original_home}/.cargo}}"
|
||||
sanitized_cargo_home="${hermetic_root}/cargo-home"
|
||||
# The sanitized Cargo home is one stable path per host, deliberately outside
|
||||
# the per-invocation hermetic root. Cargo hashes the absolute source path of
|
||||
# every registry crate into that crate's fingerprint, so a Cargo home whose
|
||||
# path changed on every invocation marked the whole dependency closure
|
||||
# `PathToSourceChanged` on the next one: the crate buckets compiled their
|
||||
# closure twice per job (nextest show-config, then nextest run), and a
|
||||
# restored CI cache could never be fresh inside the boundary. The directory
|
||||
# still links only the offline `registry` and `git` caches — never host Cargo
|
||||
# configuration or credentials — and `scripts/ci/test-hermetic-test-process.sh`
|
||||
# pins both halves. Override with IRONCLAW_HERMETIC_CARGO_HOME for a
|
||||
# differently placed (never the host's own) Cargo home.
|
||||
sanitized_cargo_home="${IRONCLAW_HERMETIC_CARGO_HOME:-${temp_parent%/}/ironclaw-hermetic-cargo-home}"
|
||||
mkdir -p "${sanitized_cargo_home}"
|
||||
for cargo_cache in registry git; do
|
||||
if [[ -n "${original_cargo_home}" && -d "${original_cargo_home}/${cargo_cache}" ]]; then
|
||||
ln -s "${original_cargo_home}/${cargo_cache}" "${sanitized_cargo_home}/${cargo_cache}"
|
||||
# Compare canonical paths: a symlink or a `..` spelling of the host Cargo home
|
||||
# must be rejected just like the literal path, or the guarded command would
|
||||
# read host Cargo configuration and credentials through it. The sanitized
|
||||
# home may not be an ancestor of the host home either (the scrub below would
|
||||
# then delete host files).
|
||||
canonical_dir() {
|
||||
(cd "$1" 2>/dev/null && pwd -P) || printf '%s\n' "$1"
|
||||
}
|
||||
sanitized_cargo_home_real="$(canonical_dir "${sanitized_cargo_home}")"
|
||||
if [[ -n "${original_cargo_home}" ]]; then
|
||||
original_cargo_home_real="$(canonical_dir "${original_cargo_home}")"
|
||||
if [[ "${sanitized_cargo_home_real}" == "${original_cargo_home_real}" \
|
||||
|| "${original_cargo_home_real}" == "${sanitized_cargo_home_real}"/* ]]; then
|
||||
echo "IRONCLAW_HERMETIC_CARGO_HOME must not be (or contain) the host Cargo home: ${sanitized_cargo_home} resolves to ${sanitized_cargo_home_real}" >&2
|
||||
exit 2
|
||||
fi
|
||||
fi
|
||||
# The stable home is shared by consecutive invocations, so anything a guarded
|
||||
# command could leave behind that Cargo would read next time is removed up
|
||||
# front: configuration and credential files never exist here. Only the two
|
||||
# cache links below and Cargo's own lock/cache-tracker files may persist.
|
||||
for host_cargo_state in config config.toml credentials credentials.toml; do
|
||||
rm -f "${sanitized_cargo_home_real}/${host_cargo_state}"
|
||||
done
|
||||
for cargo_cache in registry git; do
|
||||
source_cache="${original_cargo_home:+${original_cargo_home}/${cargo_cache}}"
|
||||
cache_link="${sanitized_cargo_home}/${cargo_cache}"
|
||||
if [[ -z "${source_cache}" || ! -d "${source_cache}" ]]; then
|
||||
continue
|
||||
fi
|
||||
if [[ -L "${cache_link}" && "$(readlink "${cache_link}")" == "${source_cache}" ]]; then
|
||||
continue
|
||||
fi
|
||||
# Concurrent invocations race to create the same link; whichever wins, the
|
||||
# link must point at the host cache afterwards.
|
||||
ln -sfn "${source_cache}" "${cache_link}" 2>/dev/null \
|
||||
|| [[ -L "${cache_link}" && "$(readlink "${cache_link}")" == "${source_cache}" ]]
|
||||
done
|
||||
|
||||
tool_path="${PATH:-/usr/bin:/bin}"
|
||||
|
||||
@@ -87,6 +87,15 @@ if [[ " $* " != *" --cargo-test-arg --lib --cargo-test-arg authorize_contract "*
|
||||
echo "scoped cargo test args were not forwarded" >&2
|
||||
exit 8
|
||||
fi
|
||||
if [ -n "${STUB_ARGV_FILE:-}" ]; then
|
||||
printf '%s\n' "$*" >"${STUB_ARGV_FILE}"
|
||||
fi
|
||||
# cargo-mutants 27.1's real argument contract: `--in-place` and `--jobs` are
|
||||
# mutually exclusive (verified with the binary; the merge queue failed on it).
|
||||
if [[ " $* " == *" --in-place "* && " $* " == *" --jobs "* ]]; then
|
||||
echo "error: the argument '--in-place' cannot be used with '--jobs <JOBS>'" >&2
|
||||
exit 1
|
||||
fi
|
||||
out=""
|
||||
pattern=""
|
||||
while [ "$#" -gt 0 ]; do
|
||||
@@ -137,16 +146,37 @@ passes=0
|
||||
failures=0
|
||||
capture_at() {
|
||||
local root="$1" mode="$2" manifest="$3" changed="$4"
|
||||
shift 4
|
||||
set +e
|
||||
CAP_OUT="$(PATH="${work}/bin:${PATH}" STUB_MODE="${mode}" python3 "${gate}" \
|
||||
CAP_OUT="$(PATH="${work}/bin:${PATH}" STUB_MODE="${mode}" \
|
||||
STUB_ARGV_FILE="${work}/argv.txt" python3 "${gate}" \
|
||||
--manifest "${manifest}" \
|
||||
--repo-root "${root}" \
|
||||
--changed-files "${changed}" 2>&1)"
|
||||
--changed-files "${changed}" "$@" 2>&1)"
|
||||
CAP_RC=$?
|
||||
set -e
|
||||
}
|
||||
capture() {
|
||||
capture_at "${case_root}" "${1}" "${work}/manifest.toml" "${work}/changed.txt"
|
||||
local mode="$1"
|
||||
shift
|
||||
capture_at "${case_root}" "${mode}" "${work}/manifest.toml" "${work}/changed.txt" "$@"
|
||||
}
|
||||
check_argv() {
|
||||
local label="$1" needle="$2" expect="$3"
|
||||
local argv=""
|
||||
if [ -f "${work}/argv.txt" ]; then
|
||||
argv=" $(cat "${work}/argv.txt") "
|
||||
fi
|
||||
if [ "${expect}" = present ] && [[ "${argv}" == *" ${needle} "* ]]; then
|
||||
echo " ok ${label}"
|
||||
passes=$((passes + 1))
|
||||
elif [ "${expect}" = absent ] && [[ "${argv}" != *" ${needle} "* ]]; then
|
||||
echo " ok ${label}"
|
||||
passes=$((passes + 1))
|
||||
else
|
||||
echo " FAIL ${label}: cargo-mutants argv was: ${argv}" >&2
|
||||
failures=$((failures + 1))
|
||||
fi
|
||||
}
|
||||
check_rc() {
|
||||
local label="$1" expected="$2"
|
||||
@@ -175,6 +205,22 @@ echo "▶ named critical gate happy path"
|
||||
capture caught
|
||||
check_rc "all named mutants caught passes" 0
|
||||
check_text "pass summary refuses a score" "0 survived; 0 timed out"
|
||||
check_argv "copy mode keeps three parallel tree copies" "--jobs 3" present
|
||||
check_argv "copy mode does not mutate the checkout" "--in-place" absent
|
||||
|
||||
echo "▶ in-place mode reuses the checkout's warm target directory"
|
||||
rm -f "${work}/argv.txt"
|
||||
capture caught --in-place
|
||||
check_rc "in-place run passes" 0
|
||||
check_argv "in-place is forwarded to cargo-mutants" "--in-place" present
|
||||
check_argv "in-place passes no --jobs (cargo-mutants rejects the pair)" "--jobs" absent
|
||||
capture caught --in-place --jobs 3
|
||||
check_rc "in-place with parallel jobs is refused" 1
|
||||
check_text "the refusal explains the tree-copy constraint" "one cargo-mutants job at a time"
|
||||
rm -f "${work}/argv.txt"
|
||||
capture caught --in-place --jobs 1
|
||||
check_rc "in-place with an explicit single job passes" 0
|
||||
check_argv "an explicit single job is still not forwarded in place" "--jobs" absent
|
||||
|
||||
echo "▶ survivor and timeout sabotage"
|
||||
capture missed
|
||||
|
||||
@@ -37,6 +37,7 @@ run_probe() {
|
||||
REBORN_COV_COLLECT="false" \
|
||||
IRONCLAW_E2E_EMULATE_SLACK_CHANNEL_BEARER="emulate-slack-channel-token" \
|
||||
IRONCLAW_HERMETIC_SABOTAGE="${sabotage}" \
|
||||
RUNNER_TEMP="${probe_dir}" \
|
||||
"${runner}" -- bash -c '
|
||||
set -euo pipefail
|
||||
for key in \
|
||||
@@ -59,13 +60,24 @@ run_probe() {
|
||||
fi
|
||||
done
|
||||
|
||||
# The Cargo home must be sanitized (no host config or credentials) but
|
||||
# must NOT live under the per-invocation root: cargo hashes the
|
||||
# absolute source path of every registry crate into its fingerprint, so
|
||||
# a per-invocation Cargo home marks the whole dependency closure
|
||||
# PathToSourceChanged on the next invocation and a cached target
|
||||
# directory can never be fresh. Stability is asserted outside.
|
||||
case "${CARGO_HOME}" in
|
||||
"${IRONCLAW_HERMETIC_ROOT}"/*) ;;
|
||||
*)
|
||||
echo "CARGO_HOME is outside the hermetic root: ${CARGO_HOME}" >&2
|
||||
"${IRONCLAW_HERMETIC_ROOT}"/*)
|
||||
echo "CARGO_HOME lives under the per-invocation hermetic root: ${CARGO_HOME}" >&2
|
||||
exit 38
|
||||
;;
|
||||
esac
|
||||
for host_cargo_file in config config.toml credentials credentials.toml; do
|
||||
if [[ -e "${CARGO_HOME}/${host_cargo_file}" ]]; then
|
||||
echo "hermetic CARGO_HOME exposes host Cargo state: ${CARGO_HOME}/${host_cargo_file}" >&2
|
||||
exit 43
|
||||
fi
|
||||
done
|
||||
if [[ -n "${RUSTUP_HOME+x}" ]]; then
|
||||
echo "ambient RUSTUP_HOME leaked into the hermetic process" >&2
|
||||
exit 39
|
||||
@@ -129,16 +141,47 @@ run_probe() {
|
||||
exit 34
|
||||
fi
|
||||
|
||||
printf "%s\n" "${IRONCLAW_HERMETIC_ROOT}"
|
||||
printf "%s\t%s\n" "${IRONCLAW_HERMETIC_ROOT}" "${CARGO_HOME}"
|
||||
'
|
||||
}
|
||||
|
||||
first_root="$(run_probe)"
|
||||
second_root="$(run_probe)"
|
||||
first_probe="$(run_probe)"
|
||||
second_probe="$(run_probe)"
|
||||
first_root="${first_probe%%$'\t'*}"
|
||||
second_root="${second_probe%%$'\t'*}"
|
||||
first_cargo_home="${first_probe#*$'\t'}"
|
||||
second_cargo_home="${second_probe#*$'\t'}"
|
||||
if [[ "${first_root}" == "${second_root}" ]]; then
|
||||
echo "hermetic invocations reused mutable state root: ${first_root}" >&2
|
||||
exit 1
|
||||
fi
|
||||
# Regression guard for the double-compile: the sanitized Cargo home is one
|
||||
# stable path per host, so cargo fingerprints (which hash each registry
|
||||
# crate's absolute source path) survive from one invocation to the next and
|
||||
# from a restored CI cache into the job that restored it.
|
||||
if [[ "${first_cargo_home}" != "${second_cargo_home}" ]]; then
|
||||
echo "hermetic CARGO_HOME changed between invocations (every registry crate would be PathToSourceChanged): ${first_cargo_home} vs ${second_cargo_home}" >&2
|
||||
exit 1
|
||||
fi
|
||||
expected_cargo_home="${probe_dir}/ironclaw-hermetic-cargo-home"
|
||||
if [[ "${first_cargo_home}" != "${expected_cargo_home}" ]]; then
|
||||
echo "hermetic CARGO_HOME is not the suite-owned path under RUNNER_TEMP: ${first_cargo_home} (expected ${expected_cargo_home})" >&2
|
||||
exit 1
|
||||
fi
|
||||
host_cargo_home="${CARGO_HOME:-${HOME}/.cargo}"
|
||||
if [[ "${first_cargo_home}" == "${host_cargo_home}" ]]; then
|
||||
echo "hermetic CARGO_HOME is the host Cargo home: ${first_cargo_home}" >&2
|
||||
exit 1
|
||||
fi
|
||||
for cargo_cache in registry git; do
|
||||
if [[ -d "${host_cargo_home}/${cargo_cache}" ]]; then
|
||||
if [[ ! -L "${first_cargo_home}/${cargo_cache}" ]] \
|
||||
|| [[ "$(readlink "${first_cargo_home}/${cargo_cache}")" != "${host_cargo_home}/${cargo_cache}" ]]; then
|
||||
echo "hermetic CARGO_HOME does not link the offline host ${cargo_cache} cache" >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
parallel_dir="${probe_dir}/parallel-roots"
|
||||
mkdir -p "${parallel_dir}"
|
||||
@@ -150,12 +193,87 @@ done
|
||||
for pid in "${parallel_pids[@]}"; do
|
||||
wait "${pid}"
|
||||
done
|
||||
parallel_root_count="$(LC_ALL=C sort -u "${parallel_dir}"/* | wc -l | tr -d '[:space:]')"
|
||||
parallel_root_count="$(cut -f1 "${parallel_dir}"/* | LC_ALL=C sort -u | wc -l | tr -d '[:space:]')"
|
||||
if [[ "${parallel_root_count}" != "4" ]]; then
|
||||
echo "parallel hermetic invocations did not receive four isolated roots" >&2
|
||||
LC_ALL=C sort "${parallel_dir}"/* >&2
|
||||
exit 1
|
||||
fi
|
||||
parallel_cargo_home_count="$(cut -f2 "${parallel_dir}"/* | LC_ALL=C sort -u | wc -l | tr -d '[:space:]')"
|
||||
if [[ "${parallel_cargo_home_count}" != "1" ]]; then
|
||||
echo "parallel hermetic invocations did not share one stable Cargo home" >&2
|
||||
LC_ALL=C sort "${parallel_dir}"/* >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# The stable Cargo home is shared by consecutive invocations, so state one
|
||||
# guarded command leaves there must not reach the next: Cargo configuration
|
||||
# and credential files are scrubbed on entry.
|
||||
RUNNER_TEMP="${probe_dir}" "${runner}" -- bash -c '
|
||||
set -euo pipefail
|
||||
printf "[net]\nretry = 99\n" > "${CARGO_HOME}/config.toml"
|
||||
printf "[registry]\ntoken = \"poison\"\n" > "${CARGO_HOME}/credentials.toml"
|
||||
: > "${CARGO_HOME}/config"
|
||||
: > "${CARGO_HOME}/credentials"
|
||||
'
|
||||
poison_output="$(
|
||||
RUNNER_TEMP="${probe_dir}" "${runner}" -- bash -c '
|
||||
for host_cargo_file in config config.toml credentials credentials.toml; do
|
||||
if [[ -e "${CARGO_HOME}/${host_cargo_file}" ]]; then
|
||||
echo "poisoned:${host_cargo_file}"
|
||||
fi
|
||||
done
|
||||
echo "checked"
|
||||
'
|
||||
)"
|
||||
if [[ "${poison_output}" != "checked" ]]; then
|
||||
echo "Cargo state written by one hermetic invocation reached the next: ${poison_output}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# The host-home guard compares canonical paths: a symlink to the host Cargo
|
||||
# home, a `..` spelling of it, and an ancestor of it must all be refused.
|
||||
fake_host_cargo_home="${probe_dir}/host-cargo-home"
|
||||
mkdir -p "${fake_host_cargo_home}/registry" "${fake_host_cargo_home}/git" "${fake_host_cargo_home}/nested"
|
||||
ln -s "${fake_host_cargo_home}" "${probe_dir}/host-cargo-home-link"
|
||||
for bad_cargo_home in \
|
||||
"${probe_dir}/host-cargo-home-link" \
|
||||
"${fake_host_cargo_home}/nested/../../host-cargo-home" \
|
||||
"${probe_dir}"
|
||||
do
|
||||
set +e
|
||||
guard_output="$(
|
||||
CARGO_HOME="${fake_host_cargo_home}" \
|
||||
IRONCLAW_HERMETIC_CARGO_HOME="${bad_cargo_home}" \
|
||||
"${runner}" -- true 2>&1
|
||||
)"
|
||||
guard_status=$?
|
||||
set -e
|
||||
if [[ "${guard_status}" -ne 2 || "${guard_output}" != *"must not be (or contain) the host Cargo home"* ]]; then
|
||||
echo "hermetic Cargo home guard accepted a path equivalent to the host Cargo home: ${bad_cargo_home} (status ${guard_status})" >&2
|
||||
printf '%s\n' "${guard_output}" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
if [[ -e "${fake_host_cargo_home}/config.toml" || ! -d "${fake_host_cargo_home}/registry" ]]; then
|
||||
echo "the rejected host Cargo home was modified" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# The behavioral form of the same guard: a second invocation must compile
|
||||
# nothing. Builds the smallest workspace crate, so a cold run costs seconds.
|
||||
# The pre-build outside the boundary fetches its handful of dependencies;
|
||||
# the guarded runs are offline by design.
|
||||
if [[ -z "${IRONCLAW_HERMETIC_SELF_TEST_SKIP_CARGO:-}" ]] && command -v cargo >/dev/null 2>&1; then
|
||||
(cd "${repo_root}" && cargo build -p ironclaw_prompt_envelope --quiet)
|
||||
RUNNER_TEMP="${probe_dir}" "${runner}" -- cargo build -p ironclaw_prompt_envelope --quiet
|
||||
second_build="$(RUNNER_TEMP="${probe_dir}" "${runner}" -- cargo build -p ironclaw_prompt_envelope 2>&1)"
|
||||
if [[ "${second_build}" == *"Compiling"* ]]; then
|
||||
echo "second hermetic invocation recompiled dependencies; the Cargo home is not stable across invocations:" >&2
|
||||
printf '%s\n' "${second_build}" >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
no_prepare_bin="${probe_dir}/no-prepare-bin"
|
||||
mkdir -p "${no_prepare_bin}"
|
||||
|
||||
Reference in New Issue
Block a user