4a63e53161
* test(validation): add pinned auto-testing manual lane * Install pinned ILM client for canonical validation lane * Prepare Python client for pool validation cases * Install validation client for replication cases * test(validation): provide the Python client for Heal cases * test(validation): build the candidate-bound Heal inspector artifact * test(validation): expose tool artifacts through the existing manual lane * test(validation): verify pinned Heal tools before fleet execution * test(validation): verify Heal tools against released shard captures * ci: audit retained validation environments before finalization * test(validation): add optional native Heal regression tooling * test(validation): route native tools through the existing driver * Run validation tool resolution on hosted Linux without acquiring the fleet lock * test(validation): bind C07 to pinned native authority evidence * test(validation): build optional C10 fixture codec on hosted Linux * test(ci): bind C10 execution to the verified shared codec * feat(ci): expose read-only validation fleet inventory * docs(ci): describe supported validation recovery * test(ci): add isolated manager native validation builds * test(ci): expose manager regressions through tools-only dispatch * ci(validation): consume pinned manager native prerequisites * fix(heal): preserve typed bucket errors for bounded retries * fix(heal): preserve typed errors and SSE-C response headers * ci(validation): publish isolated immutable candidate packages * test(validation): enforce private candidate manifest schema * test(validation): verify isolated package publication failures * ci(validation): dispatch isolated packages through the driver * fix(heal): deeply verify read repair payload failures * test(ecstore): verify incarnation fence at repair commit * ci(validation): verify C11 native storage evidence * fix(ci): isolate validation package publication credentials * test(ci): archive scoped F07 directory fsync evidence * fix(ecstore): retain typed quorum errors for missing parts * chore(ecstore): shrink typed-error ratchet baseline * fix(ecstore): retain below-quorum data during no-remove heal * fix(ci): allow recovery of completed cancelled validation runs * fix(heal): retain exhausted admin batch repair receipts * fix(observability): retain typed object error diagnostics * fix(s3): map typed RPC unavailability to service refusal * fix(filemeta): classify supported inline framing damage as corrupt * test(ecstore): exercise heal during committed overwrite cleanup * ci(validation): admit source-bound G01 native evidence before fleet execution * test(ecstore): fence version races and bind cleanup slots * ci(validation): require complete race native runtime preflight * test(ecstore): fence marker and batch deletes during heal * ci(validation): require G04 native and runtime admission * test(ecstore): fence multipart continue complete and abort during heal * test(ecstore): cover default heal owners and copy metadata races * ci(validation): admit source-bound owner and copy race proofs * test(scanner): assert typed corruption after metadata truncation * fix(ci): require complete native admission for multipart races * test(validation): wire I05 queued-successor native prerequisites * fix(scanner): bind object heals to the scanning disk scope * test(scanner): reuse storage boundary for scope regression * style: format RPC digest precedence assertion * fix(heal): query automatic tasks on their owning node * test(admin): reuse error facade in heal query regression * fix(heal): retain cancellation after successful bucket storage I/O * fix(ci): retain validation evidence outside runner temp * fix(heal): retain resolved object identity on terminal failures * test(scanner): adapt scope regression to admission receipts * ci(validation): inspect shared runner without fleet access * ci(validation): prepare Python before creating client environments * ci(validation): isolate each client virtual environment * ci(validation): retain interrupted evidence without fleet recovery * ci(validation): bind recovery to retained runner evidence * ci(validation): wire persistence and lock source plans * fix(validation): align read-only dispatch defaults --------- Co-authored-by: Chris <anzhengchao@gmail.com>
RustFS Testing
Use this when: you need to pick a test layer for a change, name a test so a gate keeps selecting it, understand why #[serial] does nothing under nextest, or handle a flaky test.
Source of truth: .config/nextest.toml (profiles, test-groups, quarantine), .config/make/tests.mak (make test), .github/workflows/*.yml (what runs when; matrix in ci-gates.md).
Test taxonomy
Pick the lowest layer that can prove the change; add a higher-layer test only when the behaviour is not observable below it.
| Layer | What it covers | Entry command | When it runs (details: ci-gates.md) |
|---|---|---|---|
| Unit & crate integration | Per-crate logic and in-process integration tests | cargo nextest run --all --exclude e2e_test (or -p <crate>); make test wraps it |
Every PR, required (Test and Lint, ci profile) |
| ecstore black-box | Erasure-coded read/write/recovery validation; profiles quick / full / destructive / fuzz |
scripts/run_ecstore_validation_suite.sh --profile quick |
Local and release validation only; not wired into any workflow. Contract: ecstore-validation-suite-design.md |
e2e (e2e_test crate) |
A real rustfs binary per test, driven over the S3, admin, and protocol APIs |
python3 scripts/e2e_binary.py build --features e2e-test-hooks, then python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke -p e2e_test |
PR: e2e-smoke (report-only); merge queue / main push: e2e-full; nightly: e2e-repl-nightly, e2e-nightly, e2e-protocols, e2e-distributed. Guide: crates/e2e_test/README.md; 4-node 4-disk map: distributed-e2e.md |
| Outbound target matrix | Replication of every object shape (empty, plain, retention, legal hold, multipart, checksummed, SSE-S3) against every remote-target failure mode the fake target models; an explicit expectation table pins known-red cells to an open issue | python3 scripts/e2e_binary.py build, then python3 scripts/e2e_binary.py run -- cargo nextest run -p e2e_test -E 'test(/^replication_target_matrix_test::/)' |
With e2e-repl-nightly; required locally for any change to outbound client defaults (SOP: docs/postmortems/2026-09-03-replication-checksum-default-regression.md) |
| s3s-e2e conformance | External S3 conformance tool against a live server | ./scripts/e2e-run.sh ./target/debug/rustfs <data-dir> |
PR, report-only (second half of the End-to-End Tests job) |
| S3 compatibility | ceph/s3-tests (boto3; allow-list scripts/s3-tests/implemented_tests.txt) and MinIO mint |
scripts/s3-tests/run.sh; mint via .github/workflows/mint.yml |
s3-tests: PR report-only plus a weekly full sweep; mint: weekly, report-only |
| Chaos / fault-injection | Single-node disk fault injection (crates/e2e_test/src/chaos.rs, crates/e2e_test/src/fault_proxy.rs) plus the 4-node kill/fresh-drive/blackhole cases in crates/e2e_test/src/distributed/chaos_test.rs |
Part of the e2e crate (e2e-reliability and e2e-distributed) |
Reliability cases with e2e-full; 4-node chaos on storage-sensitive PRs and nightly via e2e-distributed |
| Fuzz | cargo-fuzz targets over untrusted parsing surfaces; isolated sub-workspace under fuzz/ |
./scripts/fuzz/run.sh (see fuzz/README.md) |
PR smoke on the paths listed in .github/workflows/fuzz.yml, plus nightly corpus |
| Benchmarks | Criterion benches under each crate's benches/ |
cargo bench -p <crate> |
On demand; never a gate |
| Conditional reads | Paired ordinary/hit/miss GET latency, throughput and read-lock measurements | scripts/run_conditional_read_abba.py |
Local conditional-read acceptance; guide and scope |
Every script named above is indexed with status and wiring in scripts/README.md. Fixed GHSA advisories map to named regression tests in security-regressions.md.
The ECStore core invariant gates map required native commit, rollback, lock, read, multipart and metadata regressions to the existing CI run and distinguish them from process and network fault evidence.
The scanner checkpoint fixture diagnoses retained subtree coverage across budget interruption, persistence, reload, and plan invalidation.
The scanner cache cost profile separates clone, subtree copy, encoding, and counted save costs without changing production cache behavior.
The scanner memory acceptance runner verifies owned S3 datasets, scanner restart progress, and Linux process/cgroup memory evidence under a fixed container limit.
The Pool layout compatibility reference defines the topology and EC regression matrix for single-drive, single-node multi-drive, and multi-node expansion pools.
The Scanner/Heal topology protection gates map G-EXP, G-REB and G-DEC and their fault windows to the scanner_heal_topology_ cases, their recorded inventory, and the basis of their timeouts.
The immutable validation candidate reference describes isolated package builds and exact candidate resolution for the current auto-testing fleet.
The distributed multipart RPC delay gates define authenticated method-entry barriers, latency and peer-loss oracles, and the boundary between hook evidence and production incident attribution.
Naming conventions
Reserved test-name substrings (migration gate)
scripts/check_migration_gate_count.sh (runs in Test and Lint) selects migration-critical tests by name substring and fails when the count drops below .config/migration-gate-floor.txt. A rename that drops a substring silently thins the gate, so these substrings are reserved:
| Substring | Guards |
|---|---|
data_movement |
Cross-pool / cross-set object data-movement proofs |
rebalance |
Pool rebalance correctness |
decommission |
Pool decommission correctness |
source_cleanup |
Post-migration source cleanup |
delete_marker |
Delete-marker handling across migration |
A deliberate reduction lowers the floor in the same PR. The list above mirrors the script; change both together.
General naming
- Name a regression test after what it pins (issue or advisory number, or the invariant) so
rgfinds the guard for a past bug. - e2e lane membership is selected by test-name patterns in
.config/nextest.toml(for example_real_dual_node/_real_single_noderoute replication tests to the nightly lane). Follow the existing marker of the suite you extend. - Symbol naming follows the Rust API Guidelines (see
AGENTS.md).
nextest and #[serial]
cargo-nextest is the runner: make test requires it and CI installs it. nextest runs every test in its own process, so serial_test's in-process #[serial] mutex does not serialize tests against each other; it only affects the plain cargo test fallback. Cross-test serialization under nextest comes from a [test-groups] entry with max-threads = 1 in .config/nextest.toml (for example ecstore-serial-flaky, e2e-reliability) or from a -j 1 lane. Prefer making tests self-isolating (per-test instance context, random port, own temp dir) over adding serialization. RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 make test runs plain cargo test; its results are not authoritative because [test-groups] do not apply.
Package-scoped ECStore runs do not inherit workspace feature unification. The native decommission lock, capacity-lease and restart fixtures in core::pools_test::decommission_lock_order_tests require test-util. Select that module separately and fail an empty run, so unrelated tests matching another branch of a combined filter cannot hide its absence:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features test-util \
--no-tests fail -E 'test(/^core::pools_test::decommission_lock_order_tests::/)'
Time-driven tests use paused tokio time (start_paused plus tokio::time::advance) or explicit synchronization instead of fixed sleep windows.
Profiles
All profiles are defined in .config/nextest.toml; its block comments hold the filters and rationale.
| Profile | Role |
|---|---|
default |
Local runs; never retries |
ci |
PR gate for everything except e2e_test; global retries = 0 plus the quarantine list |
e2e-smoke |
PR subset of e2e_test |
e2e-full |
Merge-queue / main-push single-node e2e lane |
e2e-repl-nightly |
Nightly slow / cross-process replication lane |
e2e-nightly |
Nightly serial multi-process cluster fault lane |
e2e-distributed |
Storage-sensitive PR and nightly 4-node 4-disk S3 / lock / versioning / replication / quota / expand / decommission / rebalance / site-replication / chaos / upgrade (history + IAM AK/SK) lane |
e2e-protocols |
Nightly fixed-port FTPS/SFTP/WebDAV lane, run with -j 1 |
Membership of each e2e profile is pinned by a digest in .config/e2e-<profile>-selection.txt and checked by scripts/check_test_wiring.py --check-profile <profile> before the lane runs. To list what a profile selects on your platform (the result is platform-dependent because some modules are linux-only):
cargo nextest list -p e2e_test --profile e2e-smoke --message-format json \
| jq -r '.["rust-suites"][].testcases | to_entries[] | select(.value["filter-match"].status == "matches") | .key | split("::")[0]' \
| sort | uniq -c
Flake policy
A flaky test fails non-deterministically without a corresponding code change. Retry semantics live in .config/nextest.toml: default never retries, ci has global retries = 0, and only quarantined tests get retries = 2 under ci. A quarantined test that passes on retry is marked flaky in target/nextest/ci/junit.xml (uploaded as a CI artifact); that marker, not a green check, is how a live flake stays visible.
- Discover — a non-deterministic failure (CI or local) or a
flakyJUnit marker. - Open an issue within 24h describing symptom, suspected cause, and affected suite. No silent re-runs.
- Quarantine — add a
[[profile.ci.overrides]]entry withretries = 2and a comment linking exactly one OPEN issue. The current quarantine list is the[[profile.ci.overrides]]block in.config/nextest.toml. - Fix or delete within 30 days — make the test robust and remove the entry, or delete the test. An entry without a live OPEN issue link is a policy violation.
Coverage
.github/workflows/coverage.ymlmeasures workspace line coverage on its schedule and on manual dispatch:cargo llvm-cov nextest --workspace --exclude e2e_testunder theciprofile, the same scope as the PR gate. The per-crate table lands in the job summary; lcov and JSON exports are uploaded as an artifact (retention set in the workflow).- PRs touching the paths listed in
coverage.ymlalso run a report-only comparison against.config/coverage-baselines.tomlviascripts/check_security_coverage.py: a regression is recorded in the summary without failing the job; missing or malformed coverage evidence fails closed. make coverage(.config/make/coverage.mak) is the local equivalent; it writestarget/llvm-cov/lcov.infoandcoverage.jsonand prints the same table viascripts/coverage_per_crate.py.- Not measured: doctests (
ci.ymlruns them uninstrumented) and thee2e_testcrate. - A baseline change needs a linked coverage run and a reviewed explanation in the PR.
- For requested coverage assessments, reuse a report only when its revision, scope, and format match. If measurement is necessary, check disk/tool availability and use package/test-scoped
cargo llvm-covwith the repository nextest configuration; whole-workspace measurement requires that requested scope. - Missing branch/changed-line metrics are unknown, not zero. If measurement cannot run, mark coverage unverified and propose tests from inspected code without invented percentages.
- Rank evidenced gaps by changed behavior, data loss/security/compatibility risk, and uncovered branches. Pair each gap with a focused test, failure criterion, and scoped verification; do not introduce a threshold unless required.