Add `tee_reader` / `tee_reader_with_options` in `rustfs-rio`: a
`TeePrimary` that drives the source and a `TeeSecondary` that observes
an identical copy of every chunk through a byte-bounded queue. The
primary returns `Pending` when the queue is full, so both sides advance
at the pace of the slowest consumer; it is meant for small objects only.
Termination: source EOF and errors propagate to the secondary with the
same `io::ErrorKind`; dropping the secondary turns the primary into a
pass-through; dropping the primary early fails the secondary with
`BrokenPipe` by default, or hands the remaining source to a background
drain task bounded by `max_drain_bytes` when
`TeeOptions::drain_on_primary_drop` is set. `TeeSecondary::into_stream`
exposes the queued `Bytes` chunks without an extra copy.
Includes a proptest equivalence test, backpressure, error, drop,
drain-limit and cancel-safety tests, and a criterion bench comparing
tee throughput against a direct read (64 MiB in 1 MiB chunks).
* test(e2e): extend fake S3 target as an on-demand migration source
Add ListObjectsV2 paging, Range GET/HEAD, unversioned buckets, standard
and user metadata replay, ResponseStatus/TruncateBodyAt/Stall fault
actions, Range/User-Agent/prefix/continuation-token journal fields,
count_requests, direct seeding, and a configurable object cap to the
programmable fake S3 target, and add the on_demand_migration e2e
harness (OdmTestEnv, admin wrappers, source seeding, local-state
assertions, second RustFS source) with its self-test.
* test(ci): refresh darwin e2e-full selection for ODM harness
* refactor(ecstore): extract shared remote S3 client builder
Move the aws_sdk_s3 client construction out of bucket_target_sys into
bucket/remote_s3_client.rs: endpoint assembly, credential provider,
path-style selection, custom CA / skip-TLS transports and the outbound
SSRF gate now build from a neutral RemoteS3EndpointSpec so replication
targets and the upcoming on-demand migration source client share one
policy. Replication builds its client through From<&BucketTarget>; the
gate keeps its relaxed semantics (private allowed, loopback only behind
RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also
gains optional connect/read timeouts and a User-Agent suffix
interceptor, both unset for replication.
Refs rustfs/backlog#2149
* feat(ecstore): add on-demand migration SourceClient
Add bucket/on_demand_migration/source_client.rs on top of the shared
remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with
source-prefix mapping, GetObjectTagging and an admin probe. Every request
carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and
a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source
objects are rejected as unsupported. SourceError classifies SDK failures
(not found, access denied, throttled, timeout, connect, server error)
with retryability and a stable metrics label. Debug output redacts
credentials.
Refs rustfs/backlog#2149
* docs(operations): point outbound policy at shared remote S3 client builder
* fix(ilm): enqueue committed tier free versions
* fix(ilm): stabilize causal cleanup CI coverage
* test(ilm): make expire GET race deterministic
* test(ilm): synchronize expiry with active GET
A 6 GiB object uploaded to the source as a 768-part multipart upload was
replicated to a generic S3 target with a single PutObject, and the target
rejected the body with EntityTooLarge. No CreateMultipartUpload was ever
issued, so the multipart replication transport never ran for the object
it exists for.
`replication_put_object_options` seeded the transport from
`object_info.is_multipart()` and then overwrote it with the second
return value of `decrypt_checksums`. Those two booleans do not mean the
same thing: the first is the object's storage shape, read from the ETag,
while the second reports whether the stored *checksum record* carries
per-part data. A full-object checksum -- what `aws s3 cp` writes by
default for a CRC algorithm -- is serialized with no MULTIPART flag even
on a multipart upload, so the record reports false and the object was
routed as a single PUT. `decrypt_checksums` documents this in
object_api/types.rs: callers that need routing must consult
`is_multipart()`. Replication did the opposite.
Route on the object's own shape, and let the checksum record only add
multipart-ness, never take it away. Objects already stored with such a
record are fixed too: the ETag was always right.
This also repairs the diagnosis of rustfs#6825, where the single-PUT
5 GiB guard fired against an object that was multipart all along and
told the operator to re-upload it as multipart.
Tests cover the three shapes the router has to separate: a multipart
object with a full-object checksum record (the regression, which fails
without this change), a multipart object with a composite record, and a
single-part object that must not be promoted onto multipart.
DeleteBucket answers from a raw per-disk residue scan rather than from a
listing, so it can refuse for a reason no S3 request can observe: the
client drains every version the API will show, DeleteBucket still returns
BucketNotEmpty, and the client-visible message is the generic "The bucket
you tried to delete is not empty" for every blocker kind.
The server does know which residue blocked it, and where — that is what
`bucket_delete_blocked` carries. But it was emitted at `debug`, below
both the `error` DEFAULT_LOG_LEVEL and the `info` the CI s3-tests lane
runs at, so it was never actually written down. An intermittent
BucketNotEmpty in that lane leaves a server log with no trace of the
refusal at all, which is not a diagnosable state: confirmed against the
artifact log of a failing run, where the rejected bucket appears only in
span-close lines and the blocker event is absent entirely.
Split the blocker kinds by whether the client can still reach the
residue. A visible version or a tier free-version is an ordinary 409 —
the bucket really is not empty and the caller can list and delete what is
left — so that stays at `warn`. UnknownXlMeta, OrphanDirectory, and
DiagnosticBudgetExceeded are on-disk state no S3 request can remove; that
is a server-side integrity problem and is now reported at `error`, with
the blocker kind, the residue counts, and the sample path.
This does not change what DeleteBucket accepts or rejects, and does not
retry or suppress anything — it makes the existing diagnosis reachable.
Refs #7005, #7010
fix(docs): update stale docs/README.md references to docs/architecture/README.md
The file docs/README.md was removed in a previous commit but references
in AGENTS.md, ARCHITECTURE.md, and CLAUDE.md were not updated. The
expanded check_doc_paths.sh now catches these stale references.
Update the workspace s3s pin and refresh the lockfile with cargo update/upgrade.
Remove the unused lifecycle url dependency reported by cargo shear.
Tighten the s3s footprint ratchet to the current observed baseline.
* fix(http): drain request bodies after early responses
* fix(http): log early response body drain failures
---------
Co-authored-by: houseme <housemecn@gmail.com>
PR #7023 rewrote the handoff retry scripts with shell parameter
expansions collapsed into Actions expression syntax: ${GH_TOKEN:-},
${{attempt}}, ${{DISPATCHED:-0}}, ${{TITLE}} etc. GitHub parses
${{...}} as workflow expressions, and bare identifiers are invalid
there, so all seven shared-VM suite workflows (upgrade, s3-compat, kms,
tier, storage, heal, pool-expand) were rejected as invalid workflow
files on main.
Symptoms since 2026-09-01 23:11 +0800 (bba9347):
- every push to any branch produced 'failure' runs with no jobs
('This run likely failed because of a workflow file issue')
- the nightly functional chain dispatched rustfs-chain-upgrade at
17:08Z but the event was silently dropped: zero repository_dispatch
runs for all eight shared-VM suites overnight (only performance,
whose file was untouched, ran)
- the workflows API listed them by path instead of name
Fix: restore the shell expansions (${VAR}, ${VAR:-default}); quote the
expected-event name without legacy backticks; render the markdown fence
via printf so shellcheck can parse the block. actionlint and YAML
validation now pass clean on all eleven rustfs-*.yml workflows.
Relocate issue-triage and pr-review from the tool-specific .mimocode
directory into the shared .agents/skills tree that every agent already
reads (AGENTS.md, .claude/skills symlink), and ignore .mimocode/ so a
local copy never gets recommitted.
- New RustFS Replication Test workflow (rustfs-replication-test.yml):
standalone workflow_dispatch (suite selector bucket/site/all) and
repository_dispatch rustfs-chain-replication; runs on the shared
smoke-testing runner under the shared functional concurrency group.
- Suite never fails the workflow (continue-on-error): failures are filed
as redacted issues in rustfs/backlog (deduped per run) and the report is
uploaded to rustfs/dashboard functional-reports/replication/<date>.md.
- Security now hands off to Replication, making it the tenth and final
link: upgrade -> s3 -> kms -> tier -> storage -> heal -> pool ->
security -> replication (performance stays parallel on pf-testing).
- Depends on rustfs/auto-testing#27 (rustfs-replication-test.sh).
Co-authored-by: houseme <housemecn@gmail.com>
Replication could fail an object with nothing in the server log an
operator could act on. Every failure branch in the resyncer is quieter
than `error` on purpose — most sit on the hot path and fire once per
object per ARN — but `DEFAULT_LOG_LEVEL` is `error`, so on a stock
deployment a failed object produced no line at all. Raising those
branches to `warn` (#6840) did not close this: the default filter still
dropped them.
Report the terminal outcome instead of the branches. `replicate_object_
with_outcome` and `replicate_delete_with_outcome` now emit one `error`
per failed (object, target) once the per-target results are merged,
carrying the object key, version id, target ARN and endpoint, and the
target's own error, redacted through `sanitize_resync_error_detail` so
an echoed credential cannot reach the log. Volume is bounded by objects
that actually fail rather than by attempts inside a transfer.
Also state the single-PutObject size limit instead of discovering it at
the target. Replication picks its transport from the source object's
storage shape, not its size, so an object written with one PutObject
replicates with one PutObject however large it is — and S3 caps that at
5 GiB. Such an object could never reach a generic S3 target, and only
found out after streaming the whole body. `replication_single_put_size_
error` fails it up front with a message naming the size, the limit, and
the remedy.
Version-identity drift moves to `error` on a 10-minute per-ARN throttle.
It was `warn` deduped once per ARN per process, so the one line
explaining why a purged version is still on the target was both filtered
out by default and gone for good after it first fired.
Fixes#6825
Refs #6822
`build_metrics_summary` emitted a single metric entry for the local
deployment with `online` hardcoded to `true` and `last_online` stamped
with the current time, so `mc admin replicate status` reported "I am
online" rather than whether the remote site was reachable. A peer could
be down for minutes with replication failing while the status page
stayed green, leaving operators with no signal that the link had
dropped.
Emit an entry for every peer instead, deriving `online` from the
`reachable_peers` set the handler already computes by probing each peer,
and take `total_downtime`/`last_online` from the replication heartbeat's
existing `EpHealth` tracking. Node-local replication counters stay on
the local entry so a two-site cluster does not double-count its own
traffic.
The new `BucketTargetSys::endpoint_health` accessor deliberately does not
call `init_hc`: unlike `is_offline` it must not create health entries as
a side effect, or merely rendering the status page would mark an unknown
peer online.
Failure counters (`Errors`) are unchanged and still read zero; that is a
separate defect in the bucket-level statistics path and is not addressed
here.
The repository_dispatch handoff step was continue-on-error with a single
attempt: if the call failed (token lacking contents:write, transient API
error), the chain stalled silently while every job stayed green.
Each handoff now retries 3x and, if all attempts fail, files an alert
issue in rustfs/backlog with the exact recovery command before exiting 1
(still continue-on-error, so suite workflows themselves never fail).
`local_idp_settings` stamped the site region into the reported OpenID
settings whenever the federated identity service was published, which it
is even with OpenID disabled and no provider configured. The add
preflight compares those settings verbatim, so two sites in different
regions could never be paired: `replicate add` failed with `IDP settings
mismatch` while both sites reported an identical, empty `identity_openid`
config.
Report the empty OpenID settings when no provider is configured, so the
region only qualifies real provider identities, and name the diverging
field in the rejection instead of emitting a bare mismatch. Scalar values
are echoed; nested objects and credential-derived leaves are reported by
presence only.
Fixes#7003
Problem: the nightly functional chain has not completed end-to-end.
Evidence from recent runs:
- workflow_run events are fire-and-forget: after KMS finished at 17:09Z
on 8/31 no tier run was created; rustfs-storage-test.yml has never run.
- 'if: conclusion == success' gates skip downstream suites on any
failure (security was skipped after pool failed on 9/1 01:48Z).
- rustfs-pool-expand-test.yml embedded a heal pass without
continue-on-error, so a heal failure failed the whole workflow.
Fixes:
- Add rustfs-functional-chain.yml: entry point that dispatches the first
suite via repository_dispatch; each suite hands off to the next with an
explicit, re-drivable API call instead of workflow_run triggers.
- Split heal out of the pool workflow (renamed to RustFS Pool Expansion
Test): heal now runs exactly once per chain, in rustfs-heal-test.yml
(storage -> heal -> pool).
- Every suite job gets continue-on-error so a failing test never fails
the workflow; failures are filed as issues in rustfs/backlog (report
+ redacted log tail) and the chain moves on.
- Clone rustfs/auto-testing with the PF token via 'gh repo clone' plus a
5-attempt retry loop (transient clone failures aborted whole suites).
- Stop rewriting functional/index.html from every suite (divergent
copies raced each other with stale SHAs); the canonical index now
lives in the dashboard repo.
- Standalone workflow_dispatch runs are unchanged and never forward the
chain; performance runs on its own runner, dispatched in parallel.
Six set_disk::ops tests failed non-deterministically only under
concurrent full-suite load, rotating between runs while each passed in
isolation. All six share one root cause: a lock-owning put_object
quorum-acks once the rename fanout reaches write quorum and lets a
detached tail task finish the lagging disks, so a fixture that inspects
per-disk state immediately after PUT can observe a disk the tail has not
reached yet.
The two heal report fixtures, the inline-commit fixture, and the
transaction-fencing fixture read or delete physical shards right after
PUT, and hit FileNotFound on a lagging disk. The two metadata-cache
fixtures prime the cache after PUT, and the read fanout refuses to publish
a cache entry while any disk still reports an error, so the priming read
observably published nothing.
Keep every affected setup PUT on the full-fanout commit path with
no_lock: true, following the existing precedent in this module, so PUT
returns only after every disk has committed. The option only governs lock
acquisition, so it does not weaken what any of these fixtures assert; the
transaction-fencing gate in particular is driven by the fleet proof and
env vars, never by the lock option. Where a fixture also depends on cache
publication, re-prime until the current generation is observably cached
instead of asserting on a single read that a loaded host can stall past
the cache TTL. The heal race fixture's shard damage injection is
best-effort by construction, so it now skips injection when the previous
round's tail still lags rather than unwrapping a read that may
legitimately race.
No production code changes, and no retries or sleeps added.
* test(ecstore): retain final decommission capacity snapshot override
take_decommission_capacity_info_override_for_test used to pop the queue
to exhaustion, after which get_decommission_all_pool_capacity_infos
silently fell back to the host's real statfs numbers. Any new sampling
point added to the decommission start paths re-introduced that host
dependency and broke tests on some dev machines (#6989 patched one
instance by topping up snapshot counts, but the coupling remained).
Keep the final queued snapshot and replay it for every subsequent
sample so tests always observe injected capacity once an override is
installed. All existing injection patterns (single snapshot, repeated
identical snapshots, decreasing sequences ending at the post-operation
state) keep their semantics.
* test(ci): serialize load-sensitive heal and cache-generation tests
Under a heavily parallel nextest run (~792 ecstore tests), two tests of
set_disk::ops::heal::heal_result_report_tests failed nondeterministically
per round (different members each time; all 29 pass standalone). Every
test in the module builds a TempDir-backed 4-disk hermetic erasure set
and drives MiB-scale writes plus deep-scan heal: under load a single
disk's IO can fail while write quorum still holds, flipping per-disk
readback and aggregate-outcome assertions. The module's #[serial]
markers do not serialize across nextest's process-per-test boundary.
Verification also caught complete_multipart_generation_retires_cached_snapshot
failing once under the same load; it and its object.rs sibling carry
#[serial(metadata_cache_invalidation_probe)] and assert
get_object_metadata_cache generation semantics - the same shape that
forced the transition matrix tests into the serial group.
Add both families to the ecstore-serial-flaky test-group in the default
and ci profiles. Preventive serialization only, no retries. Three full
parallel rounds after the change: 792/792 passed each round.
A completed multipart upload's staging cleanup prunes empty parent
directories up to the volume root, which removes shared prefixes such as
`data-movement/` and the per-object `<sha>/` while a concurrent
new_multipart_upload builds its destination chain below them. The writer
holds a descriptor to the pruned component, so its next handle-relative
mkdirat fails NotFound. Because rename never retried NotFound, the cleanup
fan-out failed several disks in the same window and broke write quorum.
Give rename preparation its own retry rule: a NotFound is retried once per
component below the base directory, so a rebuilt walk outlasts a pruning
walk, which removes ancestors monotonically upward and stops at the base.
A destination whose parent is the base keeps NotFound terminal, so
speculative cleanup renames still fail fast, and the base is only ever
opened, never created, so a genuinely missing base still fails. The rename
itself keeps its own budget and its unchanged NotFound-is-terminal rule.
scripts/check_test_wiring.py and scripts/check_security_coverage.py import
tomllib, which landed in Python 3.11. macOS ships /usr/bin/python3 at 3.9, so
`make pre-commit` failed on a clean machine with `ModuleNotFoundError: No
module named 'tomllib'` in test-wiring-check, even though the checkers
themselves are fine.
Add scripts/python_bin.sh, which resolves an interpreter (explicit
RUSTFS_PYTHON, then python3.14..3.11/python3/python on PATH, then a
`uv run --python 3.12 --no-project` fallback) and execs it, failing with the
concrete remediation when nothing usable exists. Route the Make call sites
through RUSTFS_PYTHON_BIN. CI workflows keep calling python3 directly because
their runners already provide 3.11+.
The mixed-version rolling upgrade suite asserted a single list_objects_v2
snapshot seconds after restarting a node. Peers keep a restarted node's
drive in Suspect/Returning for ~probe_interval(2s) x success_threshold(3),
and while one drive is excluded the strict listing quorum (write quorum,
3 of 4) drops objects that were themselves legally written at 3/4 during
an earlier node's identical post-restart window, under-counting the
listing (observed as 254 vs 258 in CI) even though every object still
GETs correctly. Replace the snapshot asserts with a bounded convergence
poll; a real upgrade data-loss regression still fails after the deadline.
* fix(release): normalize development package versions
* ci: build only the rustfs release binary
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
test(ecstore): assert inline fanout gate on deterministic scheduled metric
non_inline_data_read_early_stop_does_not_add_inline_fanout_on_unequal_layout
compared disk_call_counters::KIND_READ_VERSION totals between the two-phase
read-plan gate being off and on. That counter records inside each spawned
fanout task, so the single-pending inline hedge read races the early-stop
abort_all(): whether the hedge task gets its first poll before cancellation
decides a 4-vs-5 count per read. Under concurrent nextest load the two reads
can disagree (reproduced locally at ~5% when run beside one other test,
matching the CI failure on PR #6961).
Assert on the rustfs_io_get_object_metadata_fanout_scheduled histogram
instead, which records the scheduling decision synchronously in the fanout
loop and is deterministic, using the CapturingRecorder + current-thread
runtime pattern already used by the neighboring tests in this module.
