* feat(scanner): expose startup phases and attempt deadlines
Share one node-local startup snapshot between native status and metrics.
Track actual waits and first attempts without changing admission timing,
coverage acknowledgement, or publication semantics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): observe startup recovery before initialization
Create the observation owner at enabled startup entry and move it through
the existing initializer, preserving admission timing and disabled cleanup.
Cover blocked recovery discovery and cancellation through the public entry.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): pin historical compression index reads
Exercise fixed pre-varint index bytes through the production reader,
including full and ranged reads, index fallback, and body CRC failure.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): use typed compression plan offsets
Use the existing usize storage offsets directly in the compatibility fixtures, fixing the same-type conversion errors reported by Clippy. Keep the planned-offset and decoded-byte assertions unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Workers::give() decrements in_use to 0 before calling semaphore.add_permits(1), while a queued Workers::take() task only increments in_use after its woken future is scheduled and polled, allowing Workers::wait() to return prematurely while a woken worker is still in flight.
Root Cause:
In crates/concurrency/src/workers.rs, Workers::give() decrements in_use inside self.in_use.send_modify() before calling self.semaphore.add_permits(1), while Workers::take() only increments in_use after self.semaphore.acquire().await returns and the woken task is scheduled and polled. During the handoff window when the last active worker calls give() and transfers its semaphore permit to a queued take() waiter (so semaphore.available_permits() < limit), in_use is 0, causing Workers::wait() (rx.wait_for(|&held| held == 0)) to complete prematurely while work is still in flight and Workers::available() to over-report available capacity.
Fix:
Require both held == 0 and self.semaphore.available_permits() == self.limit in Workers::wait(), notify self.in_use watchers after self.semaphore.add_permits(1) in Workers::give() and via an AcquireCancelGuard in Workers::take(), and bound Workers::available() by self.semaphore.available_permits().
Signed-off-by: czcbangkai <czcbangkai@google.com>
Co-authored-by: Hauser <housemecn@gmail.com>
* perf(ecstore): unify per-disk blocking workers with spin keep-alive and coalesce DELETE hops
Unify per-disk blocking I/O and fsync workers under a 3-lane DiskBlockingQueue
(Foreground, Isolated, Fsync) with MAX_NAMESPACE_WORKERS_PER_DISK = 1 and a
bounded user-space spin/yield keep-alive (WORKER_IDLE_SPIN_ITERS = 256), and
coalesce redundant blocking dispatches along the DELETE hot path:
- Expand DiskBlockingQueue to 3 lanes (Foreground, Isolated, Fsync) with 1
warm worker per disk/lane and a 256-iteration spin/yield keep-alive before
exiting spawn_blocking, preventing Tokio BlockingPool shared.idle futex
wake/wait thrashing between back-to-back stages of PUT and DELETE.
- Route fsync_dir, fsync_dir_with_owner, and namespace file-sync barriers
through the per-disk Fsync lane on FSYNC_RUNTIME, and replace LocalDisk's
duplicate MetadataReadQueue with LocalDisk::run_metadata_read backed by
self.blocking_queue.
- Route write_all_internal(Owned), open_file(O_TRUNC), remove_file_with_owner,
and remove_dir_with_owner through DiskBlockingQueue with pre-execution
cancellation checks.
- Coalesce DELETE directory scans in commit_reserved_version_delete and
finish_version_delete into single blocking passes with periodic cancellation
polling, batch rollback file writes in write_delete_rollback_files, batch
rollback unlinks + directory fsync in remove_files_and_fsync_with_owner, and
coalesce post-rename directory fsyncs in activate_delete_preparation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
* perf(ecstore): isolate durable work and reduce idle queue locking
Route coalesced writes and delete settlement through the fsync lane while preserving namespace leases, cancellation ownership, and durability ordering. Keep relaxed work on the foreground lane and replace idle mutex polling with an advisory atomic hint; dequeue and worker retirement remain mutex-protected.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): yield disk workers across shared blocking pools
Limit each draining task to 64 completed jobs and reuse the worker guard to reschedule remaining work behind other disks. Preserve per-disk FIFO, worker limits, and per-operation ownership. Add a deterministic two-disk regression with one blocking thread.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): release queued jobs when workers cannot start
Construct the worker guard before submission and track whether execution began. Release pending jobs outside the lane mutex when the last worker is rejected, so waiters, resource units, and self-owning namespace leases cannot remain stranded. Cover closed runtimes and shutdown during batch handoff.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(connect): synchronize restart with observed inventory
Wait for the received POST instead of treating transport backoff as proof of delivery. Inject a pre-HTTP connection drop and use a verified loopback IP fixture without changing timeout or replay assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(ecstore): unify disk worker lifecycle and drain loop
Represent submitted, running, and retired workers explicitly. Share dequeue, execution, idle recheck, and retirement in one loop while preserving bounded handoff and rejected-task ownership cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(storage): separate real I/O from paused-clock assertions
Complete durable fixtures before pausing virtual time, drive the immediate scanner tick before freezing time during filesystem waits, and keep checkpoint-backed retry accounting on a real clock. Preserve default fsync isolation, disk deadlines, retry counts, and all outcome assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
A delete-all lifecycle expiry is a prefix delete that takes the bucket
lifecycle write lock, while the rebalance entry holds that lock for read
through the inline apply and its bucket incarnation fence. The write
waited out the lock timeout, stalled every lifecycle read in the bucket,
and failed the entry on every retry.
Like decommission (#8777), rebalance now moves such a version live and
lifecycle expires it on the target.
Refs rustfs/backlog#2873
The listing pipeline signalled "this page exhausted the namespace" by
setting `MetaCacheEntriesSortedResult.err = Some(Error::Unexpected)` and
had every consumer filter that variant, so any future `Unexpected` on the
producer path would have been read as a complete listing. A second truth
source, the shared `producer_limit_reached` atomic, was last-writer-wins
across the disks of a page and was never serialized to remote peers.
Collectors now return `ListPage { entries, exhausted }`, errors surface as
errors, and the early empty-listing exits return an exhausted empty page.
The producer's stop reason travels in the metacache stream itself:
`MetacacheWriter::close_with_limit_reached` appends it after the `false`
terminator as repeated terminator markers (the only trailing bytes a
deployed reader tolerates, because `list_path_raw_inner` keeps polling
readers after EOF), `MetacacheReader::limit_reached` decodes it, and
`list_path_raw_inner` folds every reader's trailer into a
`ListPathRawOutcome` that reaches the collector. Remote disks therefore
report their per-disk limit stops as well, and the `producer_limit_reached`
fields on `WalkDirOptions`, `ListPathRawOptions` and `ListPathOptions` are
removed.
* test(e2e): tighten scoped multipart fault acceptance
Target Abort root deletion instead of child transaction cleanup, require
actual metadata peer hits, and keep faulted APIs single-attempt. Validate
bounded ordinary recovery and safe quorum refusal for partial mutations,
with the recovery enhancement tracked in backlog#2940.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): drive concurrent TLS reads independently
Give each coordinator its own supervised task and cooperative polling
budget while retaining the existing workload, deadline and byte/hash
assertions. Preserve node logs after fixture teardown and report the
operation, response phase and received bytes when a transfer times out.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(listing): honor persisted erasure quorum
Separate metadata admission and primary-reader sizing from each version's
persisted erasure requirements. Propagate agreed-entry uncertainty and
report LIST read-quorum failures as storage unavailability.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(listing): borrow metadata during quorum resolution
Keep validated metadata inspection and agreed-version selection in
filemeta. Resolve independent reader votes through borrowed slots and
retain owned buffers until a filtered result is required.
Preserve fixed-baseline legacy comparisons, cached latest fast paths,
quorum policy, and disk fan-out. Add ownership regressions and paired
release benchmarks for cold and cached version histories.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): remove redundant fallback clone
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scripts): collect scanner host telemetry on macOS
Use platform-supported iostat arguments and expose optional collector
failures without interrupting scanner-status collection.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): reuse validated metadata read path buffers
Share the checked bucket/object tuple across metadata read consumers and borrow clean normalization inputs without skipping filesystem validation. Keep Windows reconstruction and the second bucket check intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scripts): finalize scanner telemetry for failed runs
Initialize host telemetry artifacts before external captures so reused output directories cannot retain prior successes. Share normal and error-path finalization, reap collectors on request failure, and preserve the primary exit status even if status recording fails.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): distinguish unreadable recovery inventories
Treat only the exact system-prefix read-quorum result as incomplete during bounded polling. Preserve strict final reads and the shutdown/drain assertions, record failed observations, and cover interrupted empty streaks and unrelated errors.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(rio): ZigZag-encode signed delta varints and widen scratch buffer in compress_index
Index::append_to passes signed delta residuals (c_off and u_off, which are negative whenever a compressed block is smaller than est_block_uncomp / 2) to write_varint(buf, v: i64), which checks while v >= 0x80 without ZigZag encoding and truncates every negative delta to a single byte, while tmp = [0u8; 8] panics out-of-bounds on 9-10 byte varints.
Root Cause:
In crates/rio/src/compress_index.rs, Index::append_to delta-encodes u_off and c_off (c_off -= prev.compressed_offset + c_predict, where c_predict starts at est_block_uncomp / 2) as signed i64 residuals and passes them to write_varint(buf, v: i64). However, write_varint checks while v >= 0x80 on the signed i64 without ZigZag encoding (unlike Go binary.PutVarint / binary.Varint in the S2 index format): for any negative v, v >= 0x80 is false, so write_varint writes a single truncated byte v as u8 without continuation bits, either silently corrupting compressed_offset on Index::load (when (v as u8) < 0x80) or desynchronizing the varint stream (when (v as u8) >= 0x80). Additionally, tmp = [0u8; 8] at line 223 panics with an out-of-bounds index when encoding 9-10 byte varints, and read_varint lacks an overflow check on 10-byte inputs.
Fix:
Use ZigZag encoding and decoding in write_varint and read_varint, reject 64-bit varint overflows in read_varint, and widen the scratch buffer tmp in Index::append_to from [0u8; 8] to [0u8; 10].
Signed-off-by: czcbangkai <czcbangkai@google.com>
* test(rio): use est_block_uncomp field and compare IndexInfo fields in compress_index test
---------
Signed-off-by: czcbangkai <czcbangkai@google.com>
`LocalDisk::walk_dir` stored its own `limit_reached` into the
`producer_limit_reached` flag that every producer of a page shares, so a
walk reaching EOF after a sibling stopped at its per-disk limit cleared the
signal and the collector reported a complete page. Use `fetch_or` so the
flag answers "did any bounded walk stop early", whatever the finish order.
`MetaCacheEntry::is_in_dir` stripped the directory prefix repeatedly, so
`a/a/b` counted as a direct child of `a/`. `MetaCacheEntriesSorted::
forward_past` kept every entry when none sorted after the marker and split
on an index computed over the flattened entries. Both are masked today by
the collector's pre-filtering; fix them and pin the contracts with tests.
