Commit Graph

1003 Commits

Author SHA1 Message Date
overtrue
9fbf5d0553 feat(storage): retain managed artifact versions for database backups 2026-10-20 23:51:32 +09:00
overtrue
e7bebb0fc5 chore: sync main through 5313b72ba into integration 2026-10-20 20:06:41 +09:00
overtrue
cacdaa65a0 feat(storage): add fenced exact artifact version deletion 2026-10-20 20:05:56 +09:00
Chris
5313b72baa fix(decommission): run the leader-side start to completion (#8723)
* fix(decommission): run the leader-side start to completion

A forwarded decommission start awaits `ECStore::decommission` inline in
the RPC handler. When the forwarding node's RPC times out, the stream is
reset and the handler future is dropped wherever it is. Dropped after
`pool.bin` records the pool as active but before the peers reload and
the worker is spawned, the start left the pool active with no worker and
peers on the old metadata, and a retry was rejected as already active.
An admin client that disconnects can do the same.

Run the start in its own task and let the caller await only its result.
The start then always reaches its end: it fails before persisting
anything, or persists the pool, has the peers reload and spawns the
local workers, rolling back if the reload fails. A retry still cannot
start twice: it waits on the start gate held by the running start and
either reports the busy gate or is rejected because the pool is active.

`decommission` now takes `self: &Arc<Self>` and spawns workers on that
store instead of looking up the global object store, which is the same
store in production and makes the path testable; the unused
`require_decommission_store` helper is removed.

Refs rustfs/backlog#2873

* fix(decommission): detach the leader-side start from its caller

9ac4c10de carried only the reproduction (the testable entry point, the
peer-reload pause point and the proof test). This adds the fix it
describes: `ECStore::decommission` spawns the start as its own task and
awaits only its result, so a caller that stops waiting, such as a
forwarded start whose internode RPC timed out, can no longer drop the
start between persisting the pool as active and having the peers reload
and the local worker start.

Also prove a retry while the first start runs cannot start the pool
twice, update the operator and architecture docs for the new behavior,
and shrink the error(format!) ratchet baseline for the removed helper.

Refs rustfs/backlog#2873

* fix(decommission): single-flight start, recheck pool state under gate

Review of 3a5276c5b:

- A start re-checks the pool state under the start gate before it
  cancels routines, so a start racing another one can no longer cancel
  the worker the other start just spawned.
- Only one start runs per instance at a time: another request, including
  the retry of an operator whose request timed out, is rejected as
  already running instead of stacking a second detached start and its
  preflight heals. The start runs through `run_detached_mutation`, and a
  failure nobody waits for any more is logged.
- Tests: the proof test pins its drop after the save and requires a
  worker that persists a cancellation; the retry test really drops the
  first caller; new tests cover the under-gate recheck and a peer reload
  failure after the caller is gone. That last one shows the start
  rollback cannot clear a fresh start because the pool metadata
  stale-write guard refuses it; the gap is asserted and documented and
  tracked in rustfs/backlog#2873.
- The start tests run on their own large-stack thread: the io_uring lane
  (`uring_`, which also selects `during_`) runs them with libtest's 2 MiB
  default stack, where the proof test overflowed.
- Docs: single-flight starts, status and cancel during preflight, the
  rollback gap and its recovery, and the release requirement.

Refs rustfs/backlog#2873

* fix(decommission): turn away local start retries at the start slot

Second review of 29253430a:

- The leader-local admin pre-check refreshed rebalance metadata under
  the start gate before the start slot was checked, so a local retry
  waited out its 20 s gate budget behind a running start (which holds
  the gate through its peer reload) and then got a 503. The pre-check
  now checks the slot first and returns `DecommissionAlreadyRunning`,
  which the handler reports as a 400 that keeps the reason.
- Prove the reload-failure path really ran in the detached start (the
  injection is consumed), add a test that a fresh start's failed
  rollback reports both the reload error and the stale-write refusal,
  and assert an abandoned start stays tracked for shutdown.

Refs rustfs/backlog#2873
2026-10-20 18:20:13 +09:00
overtrue
f320f791a4 test(decommission): prove the shared gate budget and each bounded wait
Second review of 652c56875..0ef02a485:

- The operator doc and the budget comment claimed a forwarded start that
  outlives the internode RPC timeout still completes on the leader. The
  leader drops it wherever it is, possibly after pool.bin records the
  pool as active but before its worker starts or the peers reload.
  Document that state, the watchdog that resumes the worker, and the
  checks to run; the cancel-safety fix is tracked in rustfs/backlog#2873.
- Prove the budget is shared across waits (paused clock: a 60 ms wait
  leaves 40 ms for the next gate).
- Prove the rebalance-refresh start gate and activation gate waits and
  the decommission() entry are bounded and named.

Refs rustfs/backlog#2873
2026-10-20 13:29:14 +09:00
overtrue
4d2e362e23 fix(decommission): count only gate waiting in the start budget
Review of 652c56875 found that the 30 s start budget was a wall-clock
deadline fixed when the start began, so slow preflight work (bucket heals,
capacity scans) could spend it before the activation gates were reached.
A gate held for a few milliseconds then failed the start at once, and a
retry repeated the preflight.

- Replace the deadline with a budget that only counts time spent actually
  waiting for a gate; a free gate is always taken, and the lock timeout
  reports the time waited.
- Lower the budget to 20 s, below the default 30 s internode RPC timeout,
  so a forwarded start gets the leader's answer instead of an RPC timeout.
- Name each bounded wait (rebalance metadata refresh or decommission
  activation, and which gate), and route every start-gate rebalance load
  through one load_rebalance_meta_within.
- Move the admin pre-check error mapping into a function and test it.
- Add proof tests: a slow preflight followed by a brief gate wait still
  starts; the activation wait and the admin pre-check are bounded and
  named; the rebalance status read does not wait for an in-flight
  rebalance entry.
- Correct the operator doc: preflight bucket heals are outside the budget,
  and forwarded starts report a generic internal error.

Refs rustfs/backlog#2873
2026-10-20 13:29:14 +09:00
overtrue
a55f2f671b fix(decommission): keep status reads and starts off the movement gate
The data-movement operation gate is a write-preferring RwLock. A scanner
publication lease stores a read guard on the leasing node, and the
coordinator's lease-scoped rename, delete and validation on that node
queued a second read. A movement writer queued in between (a decommission
start, a status refresh, a peer reload) then waited for the lease while
the coordinator's second read waited for the writer, so both stalled
until the 60 s lease TTL. Admin status reads also took the movement
writer, so every status call waited for the slowest in-flight copy or
cleanup and stalled new movement reads while it queued.

- Lease-scoped target admission shares the read admission the lease
  already holds instead of queueing a new read.
- Pool and rebalance status reads publish under the gates only when they
  are free; otherwise they merge the persisted metadata into a private
  copy and publish nothing.
- A decommission start bounds its start, activation and movement gate
  waits with one 30 s budget and fails with a retryable lock timeout
  before persisting anything; the admin handler maps it to 503 with
  Retry-After.

Refs rustfs/backlog#2873
2026-10-20 13:29:14 +09:00
overtrue
efe2255ccf chore: sync main through 6f582b054 into integration 2026-10-20 04:51:06 +09:00
Chris
6f582b0548 fix(heal): decide object absence from storage evidence, not source (#8697)
* test(heal): add failing regressions for a source-independent absent verdict

Add the regression tests for backlog#2900 that fail on the current code,
where the same object-level not-found is treated as deleted on the plain
heal path but as recreate_failed on the recreate path, and where the
erasure-set healer, the task predicates and the error helpers disagree on
wrapped not-found and transient inputs.

Also add the per-object disposition cache module with its unit tests, the
manager admission tests for it, the compat register entry for the display
boundary, and the admission-invariant notes. The module and the manager
tests are not declared yet, so nothing is wired into the build.

* fix(heal): decide object absence from storage evidence, not request source

Replace the five heal error classifiers (erasure-set healer outcome
mapping, task-level not-found and lock predicates, error.rs recovery
helpers and the storage existence-check matcher) with one typed
classify_heal_error. An object-level not-found from the locked storage
read is Absent for every source and every recreate_missing value; the
recreate path, the usage-observation and scanner object-dir whitelists
are gone, and display-text matching survives only in the marked
classify_error_display boundary.

Add a bounded, expiring per-object disposition memory consulted before
queue admission so a best-effort producer cannot requeue an absent or
terminally failed object on every tick; durable MRF, admin, forced,
high-priority and dry-run requests bypass it. The read-repair producer no
longer forces recreate_missing.

* fix(heal): keep rendered grace windows out of task retries

A grace window rendered into a task message belongs to an attempt that
already finished; its owner schedules the next one. Only typed grace
errors replay the task, as before the classifier merge. Update the
ecstore read-repair black-box test: the producer no longer forces
recreate_missing.

* fix(heal): classify ambiguous legacy delete residue as permanent explicitly

* fix(heal): scope disposition memory and bound absence parsing
2026-10-20 00:11:49 +08:00
Chris
f063a8e862 fix(readiness): hold startup readiness until peers re-admit the node (#8704)
A restarted node published FullReady from its own view while survivors still counted its drives unavailable (stale DNS for the old pod IP, then the Offline->Returning->Online hysteresis). A rolling update could then stop the next pod and cost a set its read quorum (#8669).

Add a read-only PeerDriveAdmission internode RPC that reports how a peer sees another host's drives, and gate the startup FullReady transition until a strict majority of each erasure set's peers count this node's drives online. The gate shares the startup budget, releases after 15s of unreachable peers, skips immediately when older peers cannot answer, and can be disabled with RUSTFS_STARTUP_PEER_READMISSION_ENABLE=false.
2026-10-20 00:28:48 +09:00
Chris
473cddd7dc ci: gate pull requests on Clippy and warm its check artifacts (#8708)
Clippy ran only after merge, so lint regressions merged freely and then
failed every main run until fixed (10 of 12 main runs on 2026-10-17/18).
Run the workspace Clippy step on every event, warm its check-mode
artifacts in cache-warm with the identical invocation so pull requests
re-check only what they touch, and pin the unconditional step in the
ci_gate contract and its self-test.
2026-10-20 00:27:40 +09:00
Chris
9a1f3d2c4c fix(quota): keep ledger-admitted writes independent of scanner usage (#8699)
Hard quota admission is owned by the durable reservation ledger (decision
D3, rustfs/backlog#2872), but the request-path quota check still read
scanner usage on every write and mapped `UsageUnavailable` to a retryable
503 even for buckets the ledger admits. For a durable quota the checker now
treats scanner usage as reporting input: an unavailable lookup yields
`current_usage = None` and `allowed = true`, counted and logged at debug
level, and admin quota stats report the usage as null instead of zero.
Buckets without a reservation protocol keep the fail-closed snapshot path.

Quotas saved before the reservation protocols are re-saved under protocol
v2 with the same limit, type and created_at. The write goes through a new
guarded save that keeps the admin save's lifecycle/transaction guards,
incarnation check and fleet capability proof and additionally refuses when
the quota revision moved since the cached read, so a concurrent quota edit
wins. It is triggered lazily from the quota read path (background, one
attempt per bucket with a retry interval, never blocking the read) and in
bulk by `POST /rustfs/admin/v3/quota-migration`, which reports every bucket.

The ADR's Consequences section inventories the remaining scanner-usage
"authority" consumers with their meaning after D3, retirement condition and
compatibility window; the operations guide documents the migration and the
report-only usage semantics.

Refs rustfs/backlog#2904, rustfs/backlog#2872, rustfs/rustfs#8678,
rustfs/rustfs#5716, rustfs/rustfs#7350

Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-19 22:45:08 +09:00
Chris
7c75f319bc fix(scanner): stop treating an ordinary PUT as data movement in cycle termination (#8696)
* fix(scanner): stop treating an ordinary PUT as data movement in cycle termination

A continuously written deployment never reached a `Complete` scanner cycle
(#8494, #8305, #7108). Three places in the termination path vetoed the cycle
for an ordinary PUT; each now lets the write shade its bucket, the path
per-bucket certification (#8542) built, and keeps the data-movement fences.

1. Usage publication is fenced by data movement only.
   `scanner_data_usage_publication_blocked` no longer includes
   `namespace_commits_pending()`, so the post-walk probe, the local defer
   decision, the post-lease recheck and the usage-store route probes stop
   superseding or deferring a cycle for a pending rename fanout. The scanner
   publication leases (acquire, validate, target guard) drop the
   pending-commit rejection and the namespace-generation staleness check and
   keep the movement generation, the operation gate, generation exhaustion
   and the purpose/target restriction. A test-util probe lets fixtures wait
   for tails directly.

2. The plan identity is split into a coverage identity and a reuse identity.
   `scanner_activity_coverage_digest` (peer set, protocol, topology, movement,
   and a remote peer's generations) is what a bucket checkpoint is started
   and resumed under, through `ScannerBucketScanPlan::checkpoint_coverage_digest`,
   `DataUsageCacheReuseOptions::checkpoint_plan_digest` and
   `prepare_bucket_checkpoint_with_coverage`. The reuse identity
   (`scanner_activity_structural_digest`) keeps `instance_id` and
   `maintenance_generation`, so a restart or a maintenance change still
   invalidates the baseline and the set caches; a completed sweep is stored
   under it. `publication_blocked` leaves every digest and stays a guardrail
   veto. A remote worker still resumes under the single digest on the wire.

3. A bucket walk that stops early no longer makes the cycle `Incomplete`
   when the bucket has a certified value. The set scan stands in with the
   bucket's flattened value from its previous complete set cache
   (`carry_partial_bucket_values`), the cycle reports it pending
   (`ScannerBucketScanStatus::PartialCarried`), keeps it in the dirty map,
   and the next scoped cycle walks it because the baseline's pending list
   joins the dirty buckets. A bucket with no certified value keeps the cycle
   `Incomplete`; failed buckets are unchanged.

No persisted file, generation, receipt or environment variable is added.
Refs #8494, #8305, #7108, rustfs/backlog#2903.

* chore(ecstore): gate the namespace commit probe to test builds

The scanner publication barrier no longer consults
InstanceContext::namespace_commits_pending, so it has no production
caller left; keep it for tests and the test-util facade only.

* test(rustfs): observe namespace commit ownership directly in node_service tests

The scanner usage publication barrier is movement-only now, so it no
longer reports a pending rename. The node_service tests that assert a
cancelled handler keeps its physical owner, or that wait for rename
tails to drain, read the namespace commit state through a test-only
ECStore probe instead.
2026-10-19 22:23:59 +09:00
overtrue
f4b694ed19 chore: sync main through 05b483b31 into integration 2026-10-19 19:38:08 +09:00
Hauser
05b483b31a fix(storage): preserve scanner checkpoints and bound MRF journal I/O (#8692)
* fix(scanner): preserve checkpoints on identity read failures

Distinguish confirmed missing incarnation sidecars from lookup errors before
cache preparation. Preserve durable checkpoint progress when identity reads
are unavailable and report post-scan uncertainty separately from replacement.

Reuse the transaction-fenced reader for optional and required lookups while
retaining existing missing, malformed, system-bucket and lock-loss behavior.
Verify preservation and recovery through a real coordinator with the identity
sidecar write-locked, including unchanged cache revisions and dirty work.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): share required incarnation conversion

Keep the missing-incarnation error contract in one conversion used by both
required lookup entry points. This removes a duplicate formatted error site
and satisfies the existing error-format ratchet without changing its baseline.

Pin the full missing-identity error text in the real SetDisks regression.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): bound cross-disk MRF checkpoint I/O

Run independent journal disks with bounded concurrency while retaining each
replica's payload, companion and manifest ordering and the mirror phases.
Share encoded bodies, deduplicate local roots and collect every result
without changing receipt deadlines or persistence failure semantics.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-19 18:16:49 +08:00
Chris
f3cef98afc fix(scanner): anchor the bitrot cadence and budget per-disk I/O (#8701)
* test(scanner): pin Normal mode when no bitrot start is recorded

Regression for rustfs/backlog#2905: three consecutive cycles that find no
recorded bitrot start currently run Deep through the selection window, and a
Normal cycle never records the cadence start.

* fix(scanner): anchor the bitrot cadence and budget per-disk I/O

Decide the scan mode from a recorded Deep window start only. A cycle that
finds no recorded start runs Normal and anchors the bitrot cadence in
process memory instead of opening the selection window; the Deep cycle that
follows the elapsed cadence records the window start exactly as before. A
missing record used to be read as a window that started at cycle 0, which
ran Deep for deep_window_cycles cycles whenever the start could not be read
back (rustfs/rustfs#8015). The recorded start is also remembered across a
skipped or failed save, and /v3/scanner/status reports the cadence state
under a new bitrot object.

Add a per-local-disk background I/O budget (operations and bytes per second,
defaults by detected media, auto|off|<rate>) that DynamicSleeper charges
before directory listings and object metadata reads as a hard ceiling
layered on the foreground-priority disk admission gate, with runtime config
keys, per-disk status, cycle metrics, pacing pressure, usage_freshness,
Prometheus series, Helm values, admin config help, and operator docs.

Refs rustfs/backlog#2905, rustfs/rustfs#8015, rustfs/rustfs#8377.

* test(scanner): expect Normal retries on erasure sets without a bitrot start

The pending-PUT retry fixture asserted that a 4-drive set revisits every
bucket because the cycle ran Deep. Without a recorded Deep window the cycle
now runs Normal on every layout and reuses the cold inventory.
2026-10-19 18:15:03 +08:00
Chris
2b57fa8758 fix(heal): block replacement and root records per entry (#8700)
Count only post-format rebuild failures against the replacement retry
budget; handoffs, format fences, outages, grace waits and conflicts keep
their durable reason without spending an attempt. Retired generations
(committed handoff or approved legacy migration) no longer own their set,
so fresh admission for other slots proceeds, and a committed predecessor
becomes collectible once its successor is verified.

Block discovery conflicts per generation instead of per set, re-evaluate
them on every pass, and expose the blocked entries with reasons. Report
root heal inventory conflicts per record and keep replaying the others.
Revalidate blocked checkpoints hourly with the current reader and clear
markers the reader accepts. Return a dedicated error for ambiguous legacy
delete residue instead of the retryable SlowDown.

Implements rustfs/backlog#2902. Fixes rustfs/rustfs#8664. Addresses
rustfs/rustfs#8366, rustfs/rustfs#8565, rustfs/rustfs#8645 and part of
rustfs/rustfs#8608.

Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-19 18:12:43 +08:00
Chris
2534b6175b fix(scanner): startup gates back off, rebuild and run degraded (#8695)
* fix(scanner): give startup gates automatic exits instead of a pause

Every fail-closed scanner gate now has a bounded automatic exit without
deleting quarantine evidence or relaxing the leader, CAS, movement or
usage-floor fences (rustfs/backlog#2901).

- Recovery retries back off instead of pausing: after the short budget the
  observation reads `backoff`, stays retryable, names the next attempt and is
  capped at ten minutes; one success resets the count and a 24 h window
  restarts the budget (rustfs/rustfs#8297, #8670). The usage-floor failure no
  longer carries the retry count across attempts.
- A corrupt `.bloomcycle.bin` is rebuilt on the iteration after its quarantine
  marker is durable: the fenced leadership claim CAS-replaces the quarantined
  revision under a higher epoch and the marker stays as evidence.
  `future_schema` and `cleanup-pending` remain blocked.
- An unclaimable pause ledger runs the leader degraded: cycles run untracked
  behind the data-movement fence, `persistence_state` reports `degraded`, and
  the claim is retried at most every five minutes (rustfs/rustfs#8251).
- The usage-state full-rebuild reset starts from the empty baseline when every
  usage slot is corrupt and no cycle state exists (rustfs/rustfs#8670, #6951).
- A cycle-state save reconciles non-conflict failures against durable state
  and retries under the same epoch; an exhausted save reloads durable state in
  place. Only a newer owner or an unreadable primary ends the iteration, so
  peers see a changed fence only when ownership changed (rustfs/rustfs#8141).

* test(scanner): align usage-state reset tests with the empty-baseline rebuild

The full-rebuild reset no longer refuses corrupt or identity-less usage slots
when no cycle state exists (rustfs/rustfs#8670). The two tests that encoded
that refusal now check the new contract: the reset starts from the empty
baseline with a bootstrap-pending marker, never an authoritative zero floor,
and a valid backup still supplies the recovery floor.

---------

Co-authored-by: cxymds <cxymds@gmail.com>
2026-10-19 17:49:00 +08:00
overtrue
3f3e892d86 feat(admin): stream sealed artifact inventories 2026-10-19 16:39:50 +09:00
overtrue
874d2babcc chore: sync main CI scheduling into plugin integration 2026-10-19 16:27:42 +09:00
Chris
90749fbb06 ci: run feature lanes nightly and warm caches on a schedule (#8683) 2026-10-19 16:18:22 +09:00
overtrue
7c85783a08 chore: merge main into plugin runner integration 2026-10-19 15:28:53 +09:00
Chris
1ec620ac9c fix(ci): restore the release-profile short-credential guard (#8691)
#8682 replaced the cold `cargo check -p rustfs --release` with a
`cargo rustc ... -C debug-assertions=off` on the rustfs lib alone. That
leaves the dependencies on the dev-profile cfg, so the lib also fails on
items gated on one side of cfg(debug_assertions) only (E0599 on
Filenames::Embedded), and the step's "fails solely at the compile_error"
assertion can never hold; the lane has been red on main and every pull
request since. Restore the whole-graph release check, drop the two-job
limit that only a linking step needed, and record why the single-crate
variant is unsound.
2026-10-19 14:03:57 +08:00
overtrue
541fbbe74a feat(storage): authenticate all-member artifact inventory streams 2026-10-19 14:46:05 +09:00
overtrue
08ba7a7976 feat(storage): inventory sealed artifact versions on local disks 2026-10-19 13:19:30 +09:00
overtrue
81f214c99a merge: sync upstream deletion recovery and rollout fixes 2026-10-19 12:14:05 +09:00
Hauser
d156640451 fix(ecstore): recover deletes beside stale data backups
Treat valid foreign transaction markers as inert data-directory leaves during journal-free inspection. Permit fresh authorized missing-key deletion to reclaim safe legacy residue alongside certified absence receipts while preserving automatic ambiguity and generation fences.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-10-19 11:40:51 +09:00
Chris
c72ed5e909 fix(helm): hold rolling updates until survivors re-admit a replaced pod (#8686) 2026-10-19 11:40:11 +09:00
Chris
392ccadb78 ci: build development packages nightly instead of on every main push (#8684) 2026-10-19 11:39:38 +09:00
overtrue
dff05785de merge: sync upstream auth and deployment fixes 2026-10-19 11:12:56 +09:00
overtrue
cf2e3af4d9 merge: sync main into plugin runner integration 2026-10-19 10:29:29 +09:00
Chris
4d7cd23f8e ci: reuse the dev cache in Connect boundary lanes and raise build jobs (#8682)
Raise CARGO_BUILD_JOBS to 3 on every event in the workspace test lane and
record the closed backlog#1601 experiment: the test-profile compile took
28m21s at 2 on a pull request and 16-23 min at 3 across three successful
main runs.

Merge the two Connect boundary jobs into connect-feature-boundaries. The
offline enrollment gate no longer exports a fresh CARGO_TARGET_DIR, so it
rebuilds only the rustfs crate on top of the restored ci-dev cache; the E2E
root is isolated by the feature gate and the dedicated binary name, not by
the target directory. The short-credential release guard is now
`cargo rustc -p rustfs --lib --features connect-e2e-short-credentials --
-C debug-assertions=off`, which trips the same cfg(not(debug_assertions))
compile_error against cached dev-profile dependencies instead of a cold
release check of the whole graph. Every assertion of both lanes is kept.

Update ci_gate.py and the ci-gates.md PR matrix for the merged job.
2026-10-19 09:16:12 +09:00
Chris
ff70c06cef fix(auth): restrict root-derived accounts and credential logs (#8674) 2026-10-19 08:57:31 +09:00
Chris
fca83c72e4 ci: move compile-free and prebuilt-binary lanes to hosted runners (#8679)
* ci: move compile-free and prebuilt-binary lanes to hosted runners

Standard GitHub-hosted runners are free for this public repository, while
every sm-standard minute is paid self-hosted capacity. Move the jobs that
never compile (quick checks, typos, scope selection, the required-checks
aggregate, audit, closed-PR cancellation) and the two s3-tests lanes that
only run the prebuilt debug binary to ubuntu-latest, drop the wget install
the custom image needed for typos, and move the upgrade-compatibility
workflow, whose build shape already runs on hosted runners in
e2e-distributed, with renamed cache keys so the hosted and self-hosted
images never share a rust-cache entry. Gate the package summary job on
resolve having run, so development builds stop starting a pod to report a
skipped job.

* docs(ci): document runner billing and placement rules

Record in .github/workflows/README.md which runner labels GitHub bills
(none for this public repository), which ones RustFS pays for, the rule for
when a job may leave ubuntu-latest, the rust-cache single-writer and
same-image constraints, the patterns that have leaked self-hosted minutes,
and a checklist for new or moved jobs. Point to it from .github/AGENTS.md,
docs/testing/ci-gates.md and the ci.yml placement comment so the rule has a
single home.
2026-10-19 08:55:29 +09:00
Chris
22740876b3 docs(architecture): record scanner usage authority and PUT-tail decisions (#8678)
Supersede the 2026-09-03 decision: hard quota admission is owned by the
durable reservation ledger, and scanner data usage is best-effort reporting
input. Record that a pending ordinary PUT commit is a write-barrier event
rather than a usage-publication veto, and split the scanner plan identity
into a coverage identity and a reuse identity.

Refs rustfs/backlog#2872, rustfs/backlog#2903, rustfs/backlog#2904.
2026-10-19 08:33:28 +09:00
overtrue
e0f0628bc1 feat(storage): seal artifact writes across one disk inventory 2026-10-19 08:15:20 +09:00
Chris
ab7001a714 docs: unify agent instructions across tools (#8675) 2026-10-19 07:52:51 +09:00
overtrue
6de3034e36 fix(storage): reject ordinary deletes from managed artifact buckets
Reject single, batch, and exact-version deletes before staging mutations.
Transfer real lifecycle ownership through detached deletion and reuse the
cleaner's existing owner to avoid a queued-writer deadlock.

Correct the physical-tail cancellation observer and cover disconnected PUT
publication plus managed delete refusal with four-node tests.
2026-10-19 06:57:35 +09:00
InitCool
e2583ddcd6 feat(oidc): make the Console session duration configurable (#8568)
* feat(oidc): make the Console session duration configurable

The OIDC callback hard-coded the Console credential lifetime at 3600
seconds, so a Console session always ended after about an hour.
RUSTFS_CONSOLE_AUTH_TIMEOUT, the documented knob next to this behaviour,
is the Console HTTP request timeout and cannot extend a session.

Resolve the lifetime from the process-level
RUSTFS_CONSOLE_OIDC_SESSION_DURATION instead and forward it to
complete_authorization_code(). Unset, unparseable, zero, and negative
values keep DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS (3600), so
existing deployments keep the previous behaviour and a misconfigured
variable can never issue an already expired credential. Values above
CONSOLE_OIDC_SESSION_DURATION_MAX_SECS (7 days) are clamped to that
maximum with a warning, which also caps how far an operator can stretch a
credential by accident.

Keep the Console logout token for that same lifetime. OidcStateStore
pinned the logout cache to a hard-coded hour, so a session configured to
last longer than that dropped its logout token while the credential was
still valid: RP-initiated logout could no longer resolve, and the browser
was sent back to the Console login page with the IdP session still
active. The logout cache now derives its TTL from the same
console_session_duration_secs(), which applies the same clamp, and the
callback reads its lifetime from that one function instead of a local
copy, so the credential lifetime and the logout-token retention cannot
drift apart. Add tests covering the default hour, an extended session,
the maximum, and out-of-range values.

The variable is deliberately not part of ENV_IDENTITY_OPENID_KEYS: that
list is suffix-scanned per provider, and this setting applies to every
provider. It has no identity_openid KVS equivalent and needs a restart,
like the other process-level settings in the same document.

Document the setting in the login flow, the process-level table, and the
troubleshooting table, and clarify that RUSTFS_CONSOLE_AUTH_TIMEOUT is a
request timeout rather than a session lifetime.

* test(ecstore): retain offline marker fixtures across reconnect

Keep simulated drive handles attached and reject I/O until explicit rejoin.
Attempt reconnect in the current-marker case so the former CI race is
deterministic while retaining physical cleanup and historical-data checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(oidc): align session diagnostics and lifetime regression tests

Use structured IAM diagnostics for clamping and move configuration tests
to their owning crate. Cover duration boundaries, logout retention and
signed credential expiration through the federation binding.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-19 06:52:05 +09:00
overtrue
ae259e127d feat(storage): seal distributed artifact repair writes 2026-10-19 05:49:40 +09:00
overtrue
8233e3fe4c feat(storage): enable distributed artifact PUT sealing 2026-10-19 02:38:26 +09:00
overtrue
0307c9a7e2 fix(storage): retain admission through HTTP stream writes 2026-10-19 01:44:38 +09:00
overtrue
6a8d0aba58 fix(storage): retain admission through deletion cleanup 2026-10-19 01:06:40 +09:00
overtrue
ee6c2afe43 fix(storage): serialize multipart publication and rollback 2026-10-19 00:09:40 +09:00
overtrue
3678703a98 fix(storage): retain dual-volume admission through file rename 2026-10-18 23:11:43 +09:00
overtrue
fe798c13e7 fix(rpc): admit grouped version deletion before classification 2026-10-18 22:35:09 +09:00
overtrue
8c55396bfe fix(rpc): admit staged single-object version deletion 2026-10-18 22:04:43 +09:00
overtrue
46daaadbb3 fix(rpc): admit legacy single-object delete rollback 2026-10-18 21:40:48 +09:00
overtrue
7183849227 fix(storage): admit ordinary version deletion before classification 2026-10-18 18:41:13 +08:00
Hauser
6681dd12e3 fix(scanner): expose mixed sweep checkpoint transitions (#8657)
* fix(scanner): expose mixed sweep checkpoint transitions

Correlate checkpoint admission, incomplete sweeps, and successful terminal
Partial persistence by bucket/source and cycle/leader epoch. Share the
traversal-position predicate and local/remote save diagnostic without
changing certification, acknowledgement, or persistence contracts.

Extend real-disk cache regressions across cycles and leader changes and
verify that failed terminal saves retain an older periodic frontier until
a successful terminal save permits fresh certification.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): report the aggregate incomplete sweep gate

Expose the existing unresolved_objects decision in the incomplete-sweep
event. Nested cache-entry failures can keep a sweep partial while both
the root counter and the capped diagnostic failure map are empty.

Extend the real-disk regression with diagnostics disabled and distinguish
this case from mixed-only coverage without adding another cache walk.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-18 16:50:42 +08:00