b06923beaf
* test(e2e): add scanner/heal topology protection matrix Add the S00 protection baseline for the Scanner/Heal program (rustfs/backlog#2873, program rustfs/backlog#2872): seven scanner_heal_topology_ cases in the distributed e2e lane that pin pool expansion, rebalance and decommission behavior before the runtime changes. The fixture seeds a deterministic inventory (empty, inline, external, multipart, special and nested keys; a versioned bucket with an explicit null version, three history versions and a delete marker per key; an Object Lock bucket with legal hold and COMPLIANCE retention), records every version with its requested and listed version id, length and SHA-256, verifies the manifest through every node, reads physical placement from xl.meta on the pool drives, and reads the strict durability mode back from the admin API and the node logs. Fault cases hold the mover at proven windows through file barriers that compile only with the e2e-test-hooks feature: before the multipart target commit and before source cleanup (shared by decommission and rebalance), plus the existing pool-metadata before_publish cutpoint. An unreached window fails the case. DistCluster gains capture_logs_under so a case can read node startup events without a global environment variable. docs/testing/scanner-heal-topology-gates.md maps the gates to the tests and records the timeout basis. * test(e2e): keep the barrier failure message static The error(format!) ratchet rejects new formatted io::Error::other call sites in ecstore. The reached observation already records the stage and object, so the injected failure needs no per-call detail. * test(e2e): require e2e-test-hooks for topology fault cases The four scanner_heal_topology_ fault cases inject at server barriers that exist only with the e2e-test-hooks feature. Against a default server they waited for the barrier timeout and reported it as not reached. Check the verified binary's features first so the missing prerequisite fails immediately and is named. * ci(e2e): build the distributed lane with e2e-test-hooks Every other e2e lane builds and runs the fault-test hooks server. The distributed lane alone used the default server, so the topology fault cases could never reach their barriers there. Build and run with the same feature so the receipt matches, and update the documented local commands. The upgrade cases spawn the pinned previous release directly and do not compare feature sets. * test(e2e): keep topology artifacts under RUSTFS_E2E_LOG_DIR The topology fixture captured node logs, barrier observations and evidence in a temp dir removed on drop, so the distributed workflow uploaded nothing for these cases. Fall back to RUSTFS_E2E_LOG_DIR, which the workflow uploads, when no explicit evidence directory is set. * test(e2e): remove retired pools before restart; race the pre-ACK start The queue case restarted with decommissioned pools 0 and 1 still on the command line, which the server rightly refuses. Add remove_single_node_pools and restart without them, then expect the remaining pools renumbered to 0 and 1. The pre-ACK case now races the start request against the barrier. A start that returns first, or stays in flight past the documented 60 s budget, is reported as such instead of as an unreached barrier, and the evidence records the reach latency. capture_logs_under now overrides a log path assigned from RUSTFS_E2E_LOG_DIR, so the seed node logs into the case directory too. * test(e2e): bound fault-window waits by the movement budget On Linux CI a busy target capacity gate delayed the first fault object by about two minutes, past the 120 s wait borrowed from the server's release poll. Wait for a movement window within the 240 s movement budget instead, and fail at once when the decommission or rebalance ends without entering the window, so an uninjected fault is never counted. * test(e2e): bound status probes while waiting for a fault window The admin decommission status request can wait on a mover held at a test barrier until the barrier's own 120 s release timeout. The window wait issued that request each loop, so a reached barrier went unnoticed until the server timed it out and failed the decommission, which surfaced as a false resume failure. Bound each status probe to 5 s and treat a slow probe as still running. * test(e2e): retry a busy decommission start A decommission start that meets a busy movement gate can answer a retryable 503 instead of waiting. The harness treated any non-proof failure as fatal. Share one start helper that retries 503 within the case's movement budget and the fleet-proof 500 for 45 s, and keeps every other status fatal. The pre-ACK case retries 503 within its window budget.
11 KiB
RustFS Testing
Use this when: you need to pick a test layer for a change, name a test so a gate keeps selecting it, understand why #[serial] does nothing under nextest, or handle a flaky test.
Source of truth: .config/nextest.toml (profiles, test-groups, quarantine), .config/make/tests.mak (make test), .github/workflows/*.yml (what runs when; matrix in ci-gates.md).
Test taxonomy
Pick the lowest layer that can prove the change; add a higher-layer test only when the behaviour is not observable below it.
| Layer | What it covers | Entry command | When it runs (details: ci-gates.md) |
|---|---|---|---|
| Unit & crate integration | Per-crate logic and in-process integration tests | cargo nextest run --all --exclude e2e_test (or -p <crate>); make test wraps it |
Every PR, required (Test and Lint, ci profile) |
| ecstore black-box | Erasure-coded read/write/recovery validation; profiles quick / full / destructive / fuzz |
scripts/run_ecstore_validation_suite.sh --profile quick |
Local and release validation only; not wired into any workflow. Contract: ecstore-validation-suite-design.md |
e2e (e2e_test crate) |
A real rustfs binary per test, driven over the S3, admin, and protocol APIs |
python3 scripts/e2e_binary.py build --features e2e-test-hooks, then python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke -p e2e_test |
PR: e2e-smoke (report-only); merge queue / main push: e2e-full; nightly: e2e-repl-nightly, e2e-nightly, e2e-protocols, e2e-distributed. Guide: crates/e2e_test/README.md; 4-node 4-disk map: distributed-e2e.md |
| Outbound target matrix | Replication of every object shape (empty, plain, retention, legal hold, multipart, checksummed, SSE-S3) against every remote-target failure mode the fake target models; an explicit expectation table pins known-red cells to an open issue | python3 scripts/e2e_binary.py build, then python3 scripts/e2e_binary.py run -- cargo nextest run -p e2e_test -E 'test(/^replication_target_matrix_test::/)' |
With e2e-repl-nightly; required locally for any change to outbound client defaults (SOP: docs/postmortems/2026-09-03-replication-checksum-default-regression.md) |
| s3s-e2e conformance | External S3 conformance tool against a live server | ./scripts/e2e-run.sh ./target/debug/rustfs <data-dir> |
PR, report-only (second half of the End-to-End Tests job) |
| S3 compatibility | ceph/s3-tests (boto3; allow-list scripts/s3-tests/implemented_tests.txt) and MinIO mint |
scripts/s3-tests/run.sh; mint via .github/workflows/mint.yml |
s3-tests: PR report-only plus a weekly full sweep; mint: weekly, report-only |
| Chaos / fault-injection | Single-node disk fault injection (crates/e2e_test/src/chaos.rs, crates/e2e_test/src/fault_proxy.rs) plus the 4-node kill/fresh-drive/blackhole cases in crates/e2e_test/src/distributed/chaos_test.rs |
Part of the e2e crate (e2e-reliability and e2e-distributed) |
Reliability cases with e2e-full; 4-node chaos on storage-sensitive PRs and nightly via e2e-distributed |
| Fuzz | cargo-fuzz targets over untrusted parsing surfaces; isolated sub-workspace under fuzz/ |
./scripts/fuzz/run.sh (see fuzz/README.md) |
PR smoke on the paths listed in .github/workflows/fuzz.yml, plus nightly corpus |
| Benchmarks | Criterion benches under each crate's benches/ |
cargo bench -p <crate> |
On demand; never a gate |
| Conditional reads | Paired ordinary/hit/miss GET latency, throughput and read-lock measurements | scripts/run_conditional_read_abba.py |
Local conditional-read acceptance; guide and scope |
Every script named above is indexed with status and wiring in scripts/README.md. Fixed GHSA advisories map to named regression tests in security-regressions.md.
The ECStore core invariant gates map required native commit, rollback, lock, read, multipart and metadata regressions to the existing CI run and distinguish them from process and network fault evidence.
The scanner checkpoint fixture diagnoses retained subtree coverage across budget interruption, persistence, reload, and plan invalidation.
The scanner cache cost profile separates clone, subtree copy, encoding, and counted save costs without changing production cache behavior.
The Pool layout compatibility reference defines the topology and EC regression matrix for single-drive, single-node multi-drive, and multi-node expansion pools.
The Scanner/Heal topology protection gates map G-EXP, G-REB and G-DEC and their fault windows to the scanner_heal_topology_ cases, their recorded inventory, and the basis of their timeouts.
Naming conventions
Reserved test-name substrings (migration gate)
scripts/check_migration_gate_count.sh (runs in Test and Lint) selects migration-critical tests by name substring and fails when the count drops below .config/migration-gate-floor.txt. A rename that drops a substring silently thins the gate, so these substrings are reserved:
| Substring | Guards |
|---|---|
data_movement |
Cross-pool / cross-set object data-movement proofs |
rebalance |
Pool rebalance correctness |
decommission |
Pool decommission correctness |
source_cleanup |
Post-migration source cleanup |
delete_marker |
Delete-marker handling across migration |
A deliberate reduction lowers the floor in the same PR. The list above mirrors the script; change both together.
General naming
- Name a regression test after what it pins (issue or advisory number, or the invariant) so
rgfinds the guard for a past bug. - e2e lane membership is selected by test-name patterns in
.config/nextest.toml(for example_real_dual_node/_real_single_noderoute replication tests to the nightly lane). Follow the existing marker of the suite you extend. - Symbol naming follows the Rust API Guidelines (see
AGENTS.md).
nextest and #[serial]
cargo-nextest is the runner: make test requires it and CI installs it. nextest runs every test in its own process, so serial_test's in-process #[serial] mutex does not serialize tests against each other; it only affects the plain cargo test fallback. Cross-test serialization under nextest comes from a [test-groups] entry with max-threads = 1 in .config/nextest.toml (for example ecstore-serial-flaky, e2e-reliability) or from a -j 1 lane. Prefer making tests self-isolating (per-test instance context, random port, own temp dir) over adding serialization. RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 make test runs plain cargo test; its results are not authoritative because [test-groups] do not apply.
Time-driven tests use paused tokio time (start_paused plus tokio::time::advance) or explicit synchronization instead of fixed sleep windows.
Profiles
All profiles are defined in .config/nextest.toml; its block comments hold the filters and rationale.
| Profile | Role |
|---|---|
default |
Local runs; never retries |
ci |
PR gate for everything except e2e_test; global retries = 0 plus the quarantine list |
e2e-smoke |
PR subset of e2e_test |
e2e-full |
Merge-queue / main-push single-node e2e lane |
e2e-repl-nightly |
Nightly slow / cross-process replication lane |
e2e-nightly |
Nightly serial multi-process cluster fault lane |
e2e-distributed |
Storage-sensitive PR and nightly 4-node 4-disk S3 / lock / versioning / replication / quota / expand / decommission / rebalance / site-replication / chaos / upgrade (history + IAM AK/SK) lane |
e2e-protocols |
Nightly fixed-port FTPS/SFTP/WebDAV lane, run with -j 1 |
Membership of each e2e profile is pinned by a digest in .config/e2e-<profile>-selection.txt and checked by scripts/check_test_wiring.py --check-profile <profile> before the lane runs. To list what a profile selects on your platform (the result is platform-dependent because some modules are linux-only):
cargo nextest list -p e2e_test --profile e2e-smoke --message-format json \
| jq -r '.["rust-suites"][].testcases | to_entries[] | select(.value["filter-match"].status == "matches") | .key | split("::")[0]' \
| sort | uniq -c
Flake policy
A flaky test fails non-deterministically without a corresponding code change. Retry semantics live in .config/nextest.toml: default never retries, ci has global retries = 0, and only quarantined tests get retries = 2 under ci. A quarantined test that passes on retry is marked flaky in target/nextest/ci/junit.xml (uploaded as a CI artifact); that marker, not a green check, is how a live flake stays visible.
- Discover — a non-deterministic failure (CI or local) or a
flakyJUnit marker. - Open an issue within 24h describing symptom, suspected cause, and affected suite. No silent re-runs.
- Quarantine — add a
[[profile.ci.overrides]]entry withretries = 2and a comment linking exactly one OPEN issue. The current quarantine list is the[[profile.ci.overrides]]block in.config/nextest.toml. - Fix or delete within 30 days — make the test robust and remove the entry, or delete the test. An entry without a live OPEN issue link is a policy violation.
Coverage
.github/workflows/coverage.ymlmeasures workspace line coverage on its schedule and on manual dispatch:cargo llvm-cov nextest --workspace --exclude e2e_testunder theciprofile, the same scope as the PR gate. The per-crate table lands in the job summary; lcov and JSON exports are uploaded as an artifact (retention set in the workflow).- PRs touching the paths listed in
coverage.ymlalso run a report-only comparison against.config/coverage-baselines.tomlviascripts/check_security_coverage.py: a regression is recorded in the summary without failing the job; missing or malformed coverage evidence fails closed. make coverage(.config/make/coverage.mak) is the local equivalent; it writestarget/llvm-cov/lcov.infoandcoverage.jsonand prints the same table viascripts/coverage_per_crate.py.- Not measured: doctests (
ci.ymlruns them uninstrumented) and thee2e_testcrate. - A baseline change needs a linked coverage run and a reviewed explanation in the PR.
- For requested coverage assessments, reuse a report only when its revision, scope, and format match. If measurement is necessary, check disk/tool availability and use package/test-scoped
cargo llvm-covwith the repository nextest configuration; whole-workspace measurement requires that requested scope. - Missing branch/changed-line metrics are unknown, not zero. If measurement cannot run, mark coverage unverified and propose tests from inspected code without invented percentages.
- Rank evidenced gaps by changed behavior, data loss/security/compatibility risk, and uncovered branches. Pair each gap with a focused test, failure criterion, and scoped verification; do not introduce a threshold unless required.