* fix(quota): retire degraded floors from durable usage reports
* docs(quota): clarify retained usage cache refresh behavior
* fix(admin): report unknown bucket usage as null in GetBucketQuota
GetBucketQuota serialized missing usage as size 0, which contradicted the
D3 rule that unknown usage is never reported as zero and, after retiring
degraded floors, made a bucket near its limit look empty. Make size
optional and stop collapsing a missing or unreadable usage snapshot to 0.
* fix(scanner): recover dual corruption with retained epoch fencing
* fix(scanner): preserve recovery markers and legacy quota floors
* test(scanner): expect retained usage recovery marker state
* docs(scanner): register dual-corruption recovery evidence objects
Add the .bloomcycle.bin.dual-recovery/ evidence family to the persisted
objects table and describe its owner, write-once lifecycle and recovery
rule in Deletion And Recovery Rules.
* fix(decommission): move the free version of an expired tier version
Decommission only queued the lifecycle expiry of a transitioned version, so a source cleanup that ran first deleted the version itself and appended its tier free version after the entry's last pass. The post-check then failed the decommission until an operator re-ran it.
Apply the expiry of a transitioned version synchronously, as rebalance does. Its free version is on the source before the cleanup preflight, which reports the entry changed, and the retried entry moves the record to a target in the same pass. An expiry that cannot be applied leaves the version live, so it moves like any other version.
Refs rustfs/backlog#2873
* fix(decommission): move a delete-all expired tier version live
Applying a delete-all expiry inline made its prefix delete wait for the bucket lifecycle write lock behind the entry's own read lock until the lock timeout, blocking every write in the bucket meanwhile. Move such a transitioned version live and leave the delete-all to lifecycle on the target.
Cover a genuinely unapplied expiry and a free-version worker that consumes the source record before the entry moves it.
Refs rustfs/backlog#2873
* fix(heal): explain unverified legacy heal outcomes
Preserve storage diagnostics in canonical unknown outcomes and explain
legacy payload identity limits through the shared local/peer status path.
Borrow expected receipt identities while preserving all proof checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): keep diagnostics from pausing merged MRF repairs
Require the actual MRF executor before publishing legacy pause events.
An ordinary queued owner can retain a merged durable MRF notice without
having performed the specialized deep check. Preserve its diagnostic
without granting it authority to park the repair responsibility.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): type MRF outcomes and defer status diagnostics
Carry storage-owned legacy observations in process and return a typed
terminal result only after the qualified MRF Deep check. Publish pause
events from that result and exact identities instead of source or text.
Keep object names out of retry classification and compute only the
status diagnostic selected by the canonical execution state.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(decommission): defer source cleanup quorum failures
A quorum failure while deleting the source of an already copied entry
was fatal and marked the whole pool failed, although the copied source
is still in place and an uncommitted delete transaction is settled by
delete recovery. Carry the cleanup error typed and resolve a quorum
failure as Deferred: the entry is replayed at the end of the round, and
a deferral that persists takes the existing capacity-blocked pause path
instead of a terminal failure. Other cleanup errors and quorum errors
from other stages stay terminal.
Add set-level regressions for parent and child keys (`a/b`, `a/b/c`):
parent cleanup leaves every child file untouched, and parent cleanup
reaches quorum while the child is cleaned up or rewritten concurrently.
* test(ecstore): keep source cleanup errors typed in prefix cleanup tests
Return SourceCleanupError from the test helper instead of formatting it
into Error::other, and shrink the error(format!) ratchet baseline for
core/pools.rs, which lost one call site in the cleanup failure change.
* fix(decommission): retry cleanup quorum failures and count replays once
Retry a source cleanup quorum failure in place, up to three attempts
with the existing source-cleanup backoff, before deferring the entry.
Carry the versions an entry already counted on its deferral, so the
end-of-round replay does not add a fully counted entry to
items_decommissioned and bytes_done again.
A deferral that persists pauses the pool through the existing deferred
drain path. When source cleanup caused it, persist a reason led by
"source cleanup blocked" without the carrier error's free-drive wording,
and report waitingReason "source_cleanup" instead of "capacity".
Add a source_cleanup test fault stage and a routine test that drives a
cleanup quorum fault through deferral and replay to completion.
* fix(decommission): wait seconds between cleanup quorum retries
Give the in-place source cleanup quorum retry a fixed floor of one and
then two seconds, with the existing jitter on top, so it can ride out a
drive reconnect or health-check blip instead of retrying within a few
hundred milliseconds. The wait stays bounded and cancel-safe.
Find a source cleanup pause reason through stage wrappers, as the
capacity-blocked check does, and add a routine test where cleanup keeps
losing quorum: the decommission pauses with a source cleanup reason, the
bucket stays queued and the source stays in place.
Document the additive waitingReason value source_cleanup and its
compatibility with older nodes reading the same pool.bin.
* fix(decommission): never retry a lost source cleanup fence
The cleanup retry classifier used is_quorum_error(), which also matches
NamespaceLockQuorumUnavailable. The delete commit fence reports a lost
namespace lock or mutation fence with that variant, so a lost fence was
retried in place and only stopped later at the pre-cleanup fence check.
A lost fence means this worker no longer owns the key; it must stop
without retrying. Retry and defer only drive quorum failures:
ErasureReadQuorum, ErasureWriteQuorum, InsufficientReadQuorum and
InsufficientWriteQuorum.
---------
Co-authored-by: Hauser <housemecn@gmail.com>
* fix(decommission): let a failed start roll back its own activation
A fresh decommission start whose peer reload failed could not roll back:
the pool metadata stale-write guard from #6604 refuses any save that
clears a persisted active or queued decommission, so the pool stayed
active with no worker until the watchdog or a leader restart picked it
up.
The rollback save now carries the activation that start committed. The
guard exempts a pool only while its persisted record still equals that
activation field for field, so a worker checkpoint or a later start
keeps the rejection. A canceled record is not an active decommission
and is cleared by the rollback as well. Every other save path passes no
activation and is guarded as before.
A peer may already have reloaded the activation and started a worker
before the rollback. Saves are now fenced by decommission generation: a
write carrying a start_time is rejected when the persisted record
belongs to a later start, or has no decommission state and was updated
after that start_time, so such a worker cannot make the pool active
again. Its queued promotion and progress checkpoint also adopt the
persisted record and stop the worker.
Refs rustfs/backlog#2873
* fix(decommission): fence terminal writes of a withdrawn decommission
Terminal records carry no start_time, so the generation fence did not
cover them: after a start rollback, a peer worker of the rolled back
generation that failed (for example on a bucket completion) could still
persist a failed record over the restored pool. The merge now rejects a
failed, canceled or complete record over a persisted record without
decommission state; every attempt persists decommission state first, so
no legitimate path writes a terminal record there.
Peers also drop a withdrawn decommission sooner. A pool metadata reload
replaces a running decommission with a newer record that withdrew it,
even under an active worker; bucket completion adopts such a record like
promotion and progress checkpoints do; and a withdrawn progress
checkpoint stops the worker instead of logging a failed save and moving
more entries.
Tests cover the peer failure mark, a rollback retry that sees a worker
checkpoint land between attempts, and the reload withdrawal. A heal
fixture that wrote complete records over a fresh pool.bin now persists
the active attempt first.
Refs rustfs/backlog#2873
* fix(decommission): fence peer writes after a retry start rollback
A retry start replaces a failed or canceled record that keeps its
unresolved ledger, so its rollback restores that terminal record rather
than clearing the pool. The withdrawal checks only recognized a record
without decommission state, and the generation fence ignored a persisted
record without a start_time. A peer worker of the rolled back retry
could therefore checkpoint the pool back to active.
The generation fence now compares the persisted last_update whenever
the persisted record has no start_time of its own (cleared, terminal or
queued). The progress checkpoint also checks the generation against the
record read under the pool metadata lock, as bucket completion already
does, and the save-path withdrawal check treats a terminal record newer
than the running view as a withdrawal, so the peer adopts it and stops.
A reload still never lets a newer terminal record replace an active
worker's view, so after a retry's rollback a peer worker stops at its
next bucket completion or progress checkpoint rather than on reload.
Tests cover the retry rollback against a peer checkpoint and direct
write, and a peer bucket completion after a fresh start's rollback.
Refs rustfs/backlog#2873
* fix(decommission): keep a retry rollback's restored record intact
After a retry start rolls back, the restored record is failed, so a
peer worker's own failure mark passed every merge guard and replaced
it. A failure mark now saves through save_current_pool_meta_checked,
which runs the withdrawal check against the record read under the pool
metadata lock and adopts that record instead of writing.
The withdrawal check compared the restored record with the worker's
in-memory last_update, which a rejected save had already raised, so the
peer never adopted the rollback and the watchdog kept resuming it. It
now compares against the running generation (start_time), and a
rejected unresolved-entry save undoes its in-memory change.
save_current_pool_meta_for_decommission_start is also compiled for the
test-util feature again; a test-util helper still calls it, which broke
the workspace clippy and Connect builds.
Refs rustfs/backlog#2873
* fix(decommission): keep an unresolved entry when its own save fails
The rollback of a rejected unresolved-entry save also dropped the entry
on transient failures such as a quorum or lock-fence error, so the
following failure mark persisted a failed record without it and an
operator could clear a decommission that skipped an unreadable object.
Keep the entry and undo only the last_update bump, which is all a
withdrawn worker needs to still see the withdrawal.
Refs rustfs/backlog#2873
Under sustained writes nearly every cycle fell back to a full walk: scope
admission required the global dirty generation to stay unchanged across the
bucket listing, and after a Complete cycle the dirty-usage wake ran the next
full walk immediately, back to back.
- Track dirty-map removals with a process-local removal generation and admit
a scoped walk when no entry was removed or replayed during the listing
(`covers_listed_pending`). Writes to listed buckets in the window join the
start snapshot; unlisted buckets dirtied after the cutoff stay pending and
are never cleared by compare-and-remove certification. The strict
`covers_all_pending` still gates segment-reuse evidence.
- Classify scope fallbacks (removal during listing, generation exhausted,
dirty-bucket overflow, no listed dirty bucket, baseline unavailable, peer
scope unavailable), log `scoped_scan_fallback`, and report them in
usage_freshness as additive serde(default) fields.
- Pace the cycle after a Complete fallback while dirty buckets remain on the
capped retry schedule clamped to the cycle interval, with the dirty wake
disabled; the paced timer wake still requests a scope.
- Update the per-bucket convergence design doc.
No persisted format or internode protocol change.
Co-authored-by: Chris <anzhengchao@gmail.com>
* fix(ecstore): leave no source free version for a moved tier version
Data-movement source cleanup deletes each source version separately and
never set skip_tier_free_version, so every transitioned version that
decommission or rebalance had already copied to a target pool left a new
tier free version on the source. Decommission then failed its post-check
for any pool with tiered objects; rebalance left the record behind. The
record claims the remote object the target copy still reads and stays
blocked behind that live reference.
Decommission and rebalance now pass the identities of transitioned
versions whose copy committed on a target, and the cleanup skips the
free version for exactly those. A transitioned version that did not
move, such as one lifecycle expired during the move, still gets its free
version because that record is its only remote-delete owner.
Refs rustfs/backlog#2873, rustfs/backlog#2872
* fix(ecstore): keep the free version of a superseded moved source
Data movement treats a source null version as complete when a newer
unversioned object already sits on the target, without copying it.
That outcome returned the same Ok as a committed copy, so a restored
transitioned null version superseded this way was recorded as
transferred and lost its source free version, leaving its remote object
with no owner.
The copy paths now return DataMovementCopy::Committed or ::Superseded,
and decommission and rebalance record only committed copies of
transitioned versions. A superseded source keeps its free version.
Tests cover the superseded restored null version for rebalance and
decommission, a restored and a null transitioned version moving by
rebalance, an entry with one lifecycle-expired and one moved version,
the per-version skip with two versions of one key, and a directory
object. The tiered stale-target routine test from #8724 now expects the
decommission to complete. The ILM ownership contract and the
decommission notes describe the moved, superseded, expired and legacy
cases.
Refs rustfs/backlog#2873, rustfs/backlog#2872
* test(ecstore): build the decommission ILM fixtures without s3s
The superseded and expiry-mix decommission tests named s3s request and
lifecycle types directly, which added pools_test.rs to the ecstore
files that reference s3s and failed the footprint ratchet. Set the
restore request through ObjectOptions, and store the lifecycle rule as
bucket metadata XML and load it with get_expiry_configs, as the
decommission routine does.
* test(ecstore): pin restored decommission copies and multipart supersession
Decommission recorded a committed data copy of a restored transitioned
version without any test failing if the record were dropped; add a
routine-level decommission of a restored version that checks the post-
check passes, the source keeps nothing, and the target keeps the full
remote tuple. Add the multipart variant of the superseded restored null
version (rebalance), and a unit test that separates committed,
superseded and mismatched targets in the overwrite resume adjudication.
* fix(decommission): run the leader-side start to completion
A forwarded decommission start awaits `ECStore::decommission` inline in
the RPC handler. When the forwarding node's RPC times out, the stream is
reset and the handler future is dropped wherever it is. Dropped after
`pool.bin` records the pool as active but before the peers reload and
the worker is spawned, the start left the pool active with no worker and
peers on the old metadata, and a retry was rejected as already active.
An admin client that disconnects can do the same.
Run the start in its own task and let the caller await only its result.
The start then always reaches its end: it fails before persisting
anything, or persists the pool, has the peers reload and spawns the
local workers, rolling back if the reload fails. A retry still cannot
start twice: it waits on the start gate held by the running start and
either reports the busy gate or is rejected because the pool is active.
`decommission` now takes `self: &Arc<Self>` and spawns workers on that
store instead of looking up the global object store, which is the same
store in production and makes the path testable; the unused
`require_decommission_store` helper is removed.
Refs rustfs/backlog#2873
* fix(decommission): detach the leader-side start from its caller
9ac4c10de carried only the reproduction (the testable entry point, the
peer-reload pause point and the proof test). This adds the fix it
describes: `ECStore::decommission` spawns the start as its own task and
awaits only its result, so a caller that stops waiting, such as a
forwarded start whose internode RPC timed out, can no longer drop the
start between persisting the pool as active and having the peers reload
and the local worker start.
Also prove a retry while the first start runs cannot start the pool
twice, update the operator and architecture docs for the new behavior,
and shrink the error(format!) ratchet baseline for the removed helper.
Refs rustfs/backlog#2873
* fix(decommission): single-flight start, recheck pool state under gate
Review of 3a5276c5b:
- A start re-checks the pool state under the start gate before it
cancels routines, so a start racing another one can no longer cancel
the worker the other start just spawned.
- Only one start runs per instance at a time: another request, including
the retry of an operator whose request timed out, is rejected as
already running instead of stacking a second detached start and its
preflight heals. The start runs through `run_detached_mutation`, and a
failure nobody waits for any more is logged.
- Tests: the proof test pins its drop after the save and requires a
worker that persists a cancellation; the retry test really drops the
first caller; new tests cover the under-gate recheck and a peer reload
failure after the caller is gone. That last one shows the start
rollback cannot clear a fresh start because the pool metadata
stale-write guard refuses it; the gap is asserted and documented and
tracked in rustfs/backlog#2873.
- The start tests run on their own large-stack thread: the io_uring lane
(`uring_`, which also selects `during_`) runs them with libtest's 2 MiB
default stack, where the proof test overflowed.
- Docs: single-flight starts, status and cancel during preflight, the
rollback gap and its recovery, and the release requirement.
Refs rustfs/backlog#2873
* fix(decommission): turn away local start retries at the start slot
Second review of 29253430a:
- The leader-local admin pre-check refreshed rebalance metadata under
the start gate before the start slot was checked, so a local retry
waited out its 20 s gate budget behind a running start (which holds
the gate through its peer reload) and then got a 503. The pre-check
now checks the slot first and returns `DecommissionAlreadyRunning`,
which the handler reports as a 400 that keeps the reason.
- Prove the reload-failure path really ran in the detached start (the
injection is consumed), add a test that a fresh start's failed
rollback reports both the reload error and the stale-write refusal,
and assert an abandoned start stays tracked for shutdown.
Refs rustfs/backlog#2873
The data-movement operation gate is a write-preferring RwLock. A scanner
publication lease stores a read guard on the leasing node, and the
coordinator's lease-scoped rename, delete and validation on that node
queued a second read. A movement writer queued in between (a decommission
start, a status refresh, a peer reload) then waited for the lease while
the coordinator's second read waited for the writer, so both stalled
until the 60 s lease TTL. Admin status reads also took the movement
writer, so every status call waited for the slowest in-flight copy or
cleanup and stalled new movement reads while it queued.
- Lease-scoped target admission shares the read admission the lease
already holds instead of queueing a new read.
- Pool and rebalance status reads publish under the gates only when they
are free; otherwise they merge the persisted metadata into a private
copy and publish nothing.
- A decommission start bounds its start, activation and movement gate
waits with one 30 s budget and fails with a retryable lock timeout
before persisting anything; the admin handler maps it to 503 with
Retry-After.
Refs rustfs/backlog#2873
* test(heal): add failing regressions for a source-independent absent verdict
Add the regression tests for backlog#2900 that fail on the current code,
where the same object-level not-found is treated as deleted on the plain
heal path but as recreate_failed on the recreate path, and where the
erasure-set healer, the task predicates and the error helpers disagree on
wrapped not-found and transient inputs.
Also add the per-object disposition cache module with its unit tests, the
manager admission tests for it, the compat register entry for the display
boundary, and the admission-invariant notes. The module and the manager
tests are not declared yet, so nothing is wired into the build.
* fix(heal): decide object absence from storage evidence, not request source
Replace the five heal error classifiers (erasure-set healer outcome
mapping, task-level not-found and lock predicates, error.rs recovery
helpers and the storage existence-check matcher) with one typed
classify_heal_error. An object-level not-found from the locked storage
read is Absent for every source and every recreate_missing value; the
recreate path, the usage-observation and scanner object-dir whitelists
are gone, and display-text matching survives only in the marked
classify_error_display boundary.
Add a bounded, expiring per-object disposition memory consulted before
queue admission so a best-effort producer cannot requeue an absent or
terminally failed object on every tick; durable MRF, admin, forced,
high-priority and dry-run requests bypass it. The read-repair producer no
longer forces recreate_missing.
* fix(heal): keep rendered grace windows out of task retries
A grace window rendered into a task message belongs to an attempt that
already finished; its owner schedules the next one. Only typed grace
errors replay the task, as before the classifier merge. Update the
ecstore read-repair black-box test: the producer no longer forces
recreate_missing.
* fix(heal): classify ambiguous legacy delete residue as permanent explicitly
* fix(heal): scope disposition memory and bound absence parsing
A restarted node published FullReady from its own view while survivors still counted its drives unavailable (stale DNS for the old pod IP, then the Offline->Returning->Online hysteresis). A rolling update could then stop the next pod and cost a set its read quorum (#8669).
Add a read-only PeerDriveAdmission internode RPC that reports how a peer sees another host's drives, and gate the startup FullReady transition until a strict majority of each erasure set's peers count this node's drives online. The gate shares the startup budget, releases after 15s of unreachable peers, skips immediately when older peers cannot answer, and can be disabled with RUSTFS_STARTUP_PEER_READMISSION_ENABLE=false.
Hard quota admission is owned by the durable reservation ledger (decision
D3, rustfs/backlog#2872), but the request-path quota check still read
scanner usage on every write and mapped `UsageUnavailable` to a retryable
503 even for buckets the ledger admits. For a durable quota the checker now
treats scanner usage as reporting input: an unavailable lookup yields
`current_usage = None` and `allowed = true`, counted and logged at debug
level, and admin quota stats report the usage as null instead of zero.
Buckets without a reservation protocol keep the fail-closed snapshot path.
Quotas saved before the reservation protocols are re-saved under protocol
v2 with the same limit, type and created_at. The write goes through a new
guarded save that keeps the admin save's lifecycle/transaction guards,
incarnation check and fleet capability proof and additionally refuses when
the quota revision moved since the cached read, so a concurrent quota edit
wins. It is triggered lazily from the quota read path (background, one
attempt per bucket with a retry interval, never blocking the read) and in
bulk by `POST /rustfs/admin/v3/quota-migration`, which reports every bucket.
The ADR's Consequences section inventories the remaining scanner-usage
"authority" consumers with their meaning after D3, retirement condition and
compatibility window; the operations guide documents the migration and the
report-only usage semantics.
Refs rustfs/backlog#2904, rustfs/backlog#2872, rustfs/rustfs#8678,
rustfs/rustfs#5716, rustfs/rustfs#7350
Co-authored-by: Hauser <housemecn@gmail.com>
* fix(scanner): stop treating an ordinary PUT as data movement in cycle termination
A continuously written deployment never reached a `Complete` scanner cycle
(#8494, #8305, #7108). Three places in the termination path vetoed the cycle
for an ordinary PUT; each now lets the write shade its bucket, the path
per-bucket certification (#8542) built, and keeps the data-movement fences.
1. Usage publication is fenced by data movement only.
`scanner_data_usage_publication_blocked` no longer includes
`namespace_commits_pending()`, so the post-walk probe, the local defer
decision, the post-lease recheck and the usage-store route probes stop
superseding or deferring a cycle for a pending rename fanout. The scanner
publication leases (acquire, validate, target guard) drop the
pending-commit rejection and the namespace-generation staleness check and
keep the movement generation, the operation gate, generation exhaustion
and the purpose/target restriction. A test-util probe lets fixtures wait
for tails directly.
2. The plan identity is split into a coverage identity and a reuse identity.
`scanner_activity_coverage_digest` (peer set, protocol, topology, movement,
and a remote peer's generations) is what a bucket checkpoint is started
and resumed under, through `ScannerBucketScanPlan::checkpoint_coverage_digest`,
`DataUsageCacheReuseOptions::checkpoint_plan_digest` and
`prepare_bucket_checkpoint_with_coverage`. The reuse identity
(`scanner_activity_structural_digest`) keeps `instance_id` and
`maintenance_generation`, so a restart or a maintenance change still
invalidates the baseline and the set caches; a completed sweep is stored
under it. `publication_blocked` leaves every digest and stays a guardrail
veto. A remote worker still resumes under the single digest on the wire.
3. A bucket walk that stops early no longer makes the cycle `Incomplete`
when the bucket has a certified value. The set scan stands in with the
bucket's flattened value from its previous complete set cache
(`carry_partial_bucket_values`), the cycle reports it pending
(`ScannerBucketScanStatus::PartialCarried`), keeps it in the dirty map,
and the next scoped cycle walks it because the baseline's pending list
joins the dirty buckets. A bucket with no certified value keeps the cycle
`Incomplete`; failed buckets are unchanged.
No persisted file, generation, receipt or environment variable is added.
Refs #8494, #8305, #7108, rustfs/backlog#2903.
* chore(ecstore): gate the namespace commit probe to test builds
The scanner publication barrier no longer consults
InstanceContext::namespace_commits_pending, so it has no production
caller left; keep it for tests and the test-util facade only.
* test(rustfs): observe namespace commit ownership directly in node_service tests
The scanner usage publication barrier is movement-only now, so it no
longer reports a pending rename. The node_service tests that assert a
cancelled handler keeps its physical owner, or that wait for rename
tails to drain, read the namespace commit state through a test-only
ECStore probe instead.
* fix(scanner): preserve checkpoints on identity read failures
Distinguish confirmed missing incarnation sidecars from lookup errors before
cache preparation. Preserve durable checkpoint progress when identity reads
are unavailable and report post-scan uncertainty separately from replacement.
Reuse the transaction-fenced reader for optional and required lookups while
retaining existing missing, malformed, system-bucket and lock-loss behavior.
Verify preservation and recovery through a real coordinator with the identity
sidecar write-locked, including unchanged cache revisions and dirty work.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(ecstore): share required incarnation conversion
Keep the missing-incarnation error contract in one conversion used by both
required lookup entry points. This removes a duplicate formatted error site
and satisfies the existing error-format ratchet without changing its baseline.
Pin the full missing-identity error text in the real SetDisks regression.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(heal): bound cross-disk MRF checkpoint I/O
Run independent journal disks with bounded concurrency while retaining each
replica's payload, companion and manifest ordering and the mirror phases.
Share encoded bodies, deduplicate local roots and collect every result
without changing receipt deadlines or persistence failure semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): pin Normal mode when no bitrot start is recorded
Regression for rustfs/backlog#2905: three consecutive cycles that find no
recorded bitrot start currently run Deep through the selection window, and a
Normal cycle never records the cadence start.
* fix(scanner): anchor the bitrot cadence and budget per-disk I/O
Decide the scan mode from a recorded Deep window start only. A cycle that
finds no recorded start runs Normal and anchors the bitrot cadence in
process memory instead of opening the selection window; the Deep cycle that
follows the elapsed cadence records the window start exactly as before. A
missing record used to be read as a window that started at cycle 0, which
ran Deep for deep_window_cycles cycles whenever the start could not be read
back (rustfs/rustfs#8015). The recorded start is also remembered across a
skipped or failed save, and /v3/scanner/status reports the cadence state
under a new bitrot object.
Add a per-local-disk background I/O budget (operations and bytes per second,
defaults by detected media, auto|off|<rate>) that DynamicSleeper charges
before directory listings and object metadata reads as a hard ceiling
layered on the foreground-priority disk admission gate, with runtime config
keys, per-disk status, cycle metrics, pacing pressure, usage_freshness,
Prometheus series, Helm values, admin config help, and operator docs.
Refs rustfs/backlog#2905, rustfs/rustfs#8015, rustfs/rustfs#8377.
* test(scanner): expect Normal retries on erasure sets without a bitrot start
The pending-PUT retry fixture asserted that a 4-drive set revisits every
bucket because the cycle ran Deep. Without a recorded Deep window the cycle
now runs Normal on every layout and reuses the cold inventory.
* fix(scanner): give startup gates automatic exits instead of a pause
Every fail-closed scanner gate now has a bounded automatic exit without
deleting quarantine evidence or relaxing the leader, CAS, movement or
usage-floor fences (rustfs/backlog#2901).
- Recovery retries back off instead of pausing: after the short budget the
observation reads `backoff`, stays retryable, names the next attempt and is
capped at ten minutes; one success resets the count and a 24 h window
restarts the budget (rustfs/rustfs#8297, #8670). The usage-floor failure no
longer carries the retry count across attempts.
- A corrupt `.bloomcycle.bin` is rebuilt on the iteration after its quarantine
marker is durable: the fenced leadership claim CAS-replaces the quarantined
revision under a higher epoch and the marker stays as evidence.
`future_schema` and `cleanup-pending` remain blocked.
- An unclaimable pause ledger runs the leader degraded: cycles run untracked
behind the data-movement fence, `persistence_state` reports `degraded`, and
the claim is retried at most every five minutes (rustfs/rustfs#8251).
- The usage-state full-rebuild reset starts from the empty baseline when every
usage slot is corrupt and no cycle state exists (rustfs/rustfs#8670, #6951).
- A cycle-state save reconciles non-conflict failures against durable state
and retries under the same epoch; an exhausted save reloads durable state in
place. Only a newer owner or an unreadable primary ends the iteration, so
peers see a changed fence only when ownership changed (rustfs/rustfs#8141).
* test(scanner): align usage-state reset tests with the empty-baseline rebuild
The full-rebuild reset no longer refuses corrupt or identity-less usage slots
when no cycle state exists (rustfs/rustfs#8670). The two tests that encoded
that refusal now check the new contract: the reset starts from the empty
baseline with a bootstrap-pending marker, never an authoritative zero floor,
and a valid backup still supplies the recovery floor.
---------
Co-authored-by: cxymds <cxymds@gmail.com>
Supersede the 2026-09-03 decision: hard quota admission is owned by the
durable reservation ledger, and scanner data usage is best-effort reporting
input. Record that a pending ordinary PUT commit is a write-barrier event
rather than a usage-publication veto, and split the scanner plan identity
into a coverage identity and a reuse identity.
Refs rustfs/backlog#2872, rustfs/backlog#2903, rustfs/backlog#2904.
Reject single, batch, and exact-version deletes before staging mutations.
Transfer real lifecycle ownership through detached deletion and reuse the
cleaner's existing owner to avoid a queued-writer deadlock.
Correct the physical-tail cancellation observer and cover disconnected PUT
publication plus managed delete refusal with four-node tests.
