mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-26 01:42:27 +08:00
d2eda195e6
* fix(decommission): defer source cleanup quorum failures A quorum failure while deleting the source of an already copied entry was fatal and marked the whole pool failed, although the copied source is still in place and an uncommitted delete transaction is settled by delete recovery. Carry the cleanup error typed and resolve a quorum failure as Deferred: the entry is replayed at the end of the round, and a deferral that persists takes the existing capacity-blocked pause path instead of a terminal failure. Other cleanup errors and quorum errors from other stages stay terminal. Add set-level regressions for parent and child keys (`a/b`, `a/b/c`): parent cleanup leaves every child file untouched, and parent cleanup reaches quorum while the child is cleaned up or rewritten concurrently. * test(ecstore): keep source cleanup errors typed in prefix cleanup tests Return SourceCleanupError from the test helper instead of formatting it into Error::other, and shrink the error(format!) ratchet baseline for core/pools.rs, which lost one call site in the cleanup failure change. * fix(decommission): retry cleanup quorum failures and count replays once Retry a source cleanup quorum failure in place, up to three attempts with the existing source-cleanup backoff, before deferring the entry. Carry the versions an entry already counted on its deferral, so the end-of-round replay does not add a fully counted entry to items_decommissioned and bytes_done again. A deferral that persists pauses the pool through the existing deferred drain path. When source cleanup caused it, persist a reason led by "source cleanup blocked" without the carrier error's free-drive wording, and report waitingReason "source_cleanup" instead of "capacity". Add a source_cleanup test fault stage and a routine test that drives a cleanup quorum fault through deferral and replay to completion. * fix(decommission): wait seconds between cleanup quorum retries Give the in-place source cleanup quorum retry a fixed floor of one and then two seconds, with the existing jitter on top, so it can ride out a drive reconnect or health-check blip instead of retrying within a few hundred milliseconds. The wait stays bounded and cancel-safe. Find a source cleanup pause reason through stage wrappers, as the capacity-blocked check does, and add a routine test where cleanup keeps losing quorum: the decommission pauses with a source cleanup reason, the bucket stays queued and the source stays in place. Document the additive waitingReason value source_cleanup and its compatibility with older nodes reading the same pool.bin. * fix(decommission): never retry a lost source cleanup fence The cleanup retry classifier used is_quorum_error(), which also matches NamespaceLockQuorumUnavailable. The delete commit fence reports a lost namespace lock or mutation fence with that variant, so a lost fence was retried in place and only stopped later at the pre-cleanup fence check. A lost fence means this worker no longer owns the key; it must stop without retrying. Retry and defer only drive quorum failures: ErasureReadQuorum, ErasureWriteQuorum, InsufficientReadQuorum and InsufficientWriteQuorum. --------- Co-authored-by: Hauser <housemecn@gmail.com>
72 lines
3.4 KiB
Plaintext
72 lines
3.4 KiB
Plaintext
# `::other(format!` ratchet baseline for crates/ecstore/src (backlog#1845 PR2).
|
|
#
|
|
# SHRINK-ONLY: entries are `count|file`. A PR may lower a count or drop a file
|
|
# (after replacing the call sites with typed variants) by re-running
|
|
# scripts/check_error_other_format_ratchet.sh --update-baseline. A PR that
|
|
# raises a count or adds a file is introducing a new quorum-bucketing hazard
|
|
# and must carry an explicit exemption rationale in its description.
|
|
4|crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs
|
|
3|crates/ecstore/src/bucket/lifecycle/durable_namespace.rs
|
|
2|crates/ecstore/src/bucket/lifecycle/metadata_boundary.rs
|
|
4|crates/ecstore/src/bucket/lifecycle/tier_delete_journal.rs
|
|
1|crates/ecstore/src/bucket/lifecycle/transition_transaction.rs
|
|
27|crates/ecstore/src/bucket/metadata.rs
|
|
22|crates/ecstore/src/bucket/metadata_sys.rs
|
|
3|crates/ecstore/src/bucket/msgp_decode.rs
|
|
1|crates/ecstore/src/bucket/object_lock/objectlock_sys.rs
|
|
1|crates/ecstore/src/bucket/replication/replication_object_config.rs
|
|
2|crates/ecstore/src/bucket/replication/replication_pool.rs
|
|
2|crates/ecstore/src/bucket/replication/replication_target_boundary.rs
|
|
3|crates/ecstore/src/cluster/rpc/http_auth.rs
|
|
2|crates/ecstore/src/cluster/rpc/internode_data_transport.rs
|
|
10|crates/ecstore/src/cluster/rpc/peer_rest_client.rs
|
|
10|crates/ecstore/src/cluster/rpc/peer_s3_client.rs
|
|
10|crates/ecstore/src/cluster/rpc/remote_disk.rs
|
|
6|crates/ecstore/src/config/com.rs
|
|
14|crates/ecstore/src/config/storageclass.rs
|
|
175|crates/ecstore/src/core/pools.rs
|
|
6|crates/ecstore/src/data_movement/mod.rs
|
|
2|crates/ecstore/src/data_usage/local_snapshot.rs
|
|
12|crates/ecstore/src/data_usage/mod.rs
|
|
5|crates/ecstore/src/disk/local.rs
|
|
1|crates/ecstore/src/disk/mod.rs
|
|
5|crates/ecstore/src/erasure/codec/bridge.rs
|
|
1|crates/ecstore/src/erasure/coding/decode_reader.rs
|
|
8|crates/ecstore/src/erasure/coding/encode.rs
|
|
25|crates/ecstore/src/erasure/coding/erasure.rs
|
|
3|crates/ecstore/src/layout/disks_layout.rs
|
|
2|crates/ecstore/src/layout/endpoint.rs
|
|
17|crates/ecstore/src/layout/endpoints.rs
|
|
1|crates/ecstore/src/layout/format.rs
|
|
1|crates/ecstore/src/layout/pool_space.rs
|
|
1|crates/ecstore/src/layout/set_layout.rs
|
|
19|crates/ecstore/src/object_api/readers.rs
|
|
1|crates/ecstore/src/object_api/types.rs
|
|
3|crates/ecstore/src/runtime/sources.rs
|
|
4|crates/ecstore/src/services/batch_processor.rs
|
|
12|crates/ecstore/src/services/notification_sys.rs
|
|
16|crates/ecstore/src/services/rebalance/control.rs
|
|
1|crates/ecstore/src/services/rebalance/entry.rs
|
|
8|crates/ecstore/src/services/rebalance/meta.rs
|
|
8|crates/ecstore/src/services/rebalance/runtime.rs
|
|
17|crates/ecstore/src/services/rebalance/worker.rs
|
|
33|crates/ecstore/src/services/tier/tier.rs
|
|
1|crates/ecstore/src/services/tier/tier_config.rs
|
|
1|crates/ecstore/src/services/tier/warm_backend.rs
|
|
2|crates/ecstore/src/services/tier/warm_backend_gcs.rs
|
|
1|crates/ecstore/src/services/tier/warm_backend_s3.rs
|
|
1|crates/ecstore/src/services/tier/warm_backend_wasabi.rs
|
|
7|crates/ecstore/src/set_disk/core/io_primitives.rs
|
|
1|crates/ecstore/src/set_disk/mod.rs
|
|
3|crates/ecstore/src/set_disk/ops/bitrot_self_verify.rs
|
|
1|crates/ecstore/src/set_disk/ops/heal.rs
|
|
8|crates/ecstore/src/set_disk/ops/multipart.rs
|
|
2|crates/ecstore/src/set_disk/ops/object.rs
|
|
3|crates/ecstore/src/set_disk/read.rs
|
|
5|crates/ecstore/src/store/bucket.rs
|
|
1|crates/ecstore/src/store/heal_walk.rs
|
|
10|crates/ecstore/src/store/init.rs
|
|
3|crates/ecstore/src/store/multipart.rs
|
|
6|crates/ecstore/src/store/object.rs
|
|
3|crates/ecstore/src/store/rebalance/support.rs
|