Commit Graph

262 Commits

Author SHA1 Message Date
houseme
efaf07d323 feat: preserve request ids across async recovery logs (#3451)
* feat(obs): promote request ids in structured logs

* refactor(tracing): propagate spans into request tasks

* test(ecstore): baseline recovery monitor log chains

* fix(replication): reduce startup resync log noise

* chore(docs): stop tracking local recovery baseline

* chore(obs): polish request id logging cleanup
2026-06-14 23:16:04 +08:00
Henry Guo
9372ee7032 feat(table-catalog): bridge table data-plane policy (#3436)
* feat(table-catalog): bridge table data-plane policy

* test(table-catalog): harden vended credential smoke

* test(table-catalog): cover data-plane policy denials

* fix(table-catalog): protect relocated warehouse scope

* fix(table-catalog): skip invalid warehouse entries

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-14 18:13:02 +08:00
Henry Guo
fc17e75fb2 test(table-catalog): verify vended credential data-plane scope (#3429) 2026-06-14 16:35:54 +08:00
houseme
e8012bd1ba refactor(logging): normalize admin telemetry and error messages (#3430) 2026-06-14 13:27:10 +08:00
Henry Guo
dc82efbab4 test(scanner): add validation harness (#3428)
* test(scanner): add validation harness

* fix(scanner): harden validation harness

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-14 12:40:05 +08:00
安正超
bed875d3e0 fix(logging): remove guardrail script trailing whitespace (#3424) 2026-06-14 08:35:59 +08:00
houseme
efa89a98ed refactor(logging): standardize protocol and observability events (#3419)
* refactor(logging): standardize object capacity events

* refactor(logging): standardize protocol server events

* refactor(logging): standardize swift protocol events

* refactor(logging): standardize observability events

* refactor(logging): move masking helper and extend guardrails
2026-06-14 07:14:45 +08:00
Henry Guo
357287d626 feat(table-catalog): issue scoped table credentials (#3413)
* feat(table-catalog): issue scoped table credentials

* fix(table-catalog): harden credential vending defaults

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-14 02:20:28 +08:00
houseme
7da10db852 refactor(logging): standardize heal and scanner events (#3414)
* refactor(logging): standardize heal and scanner events

* chore(git): untrack local logging governance note

* chore(git): ignore local logging governance note
2026-06-14 01:47:39 +08:00
houseme
9059a9c68d refactor(logging): standardize concurrency and trusted proxy events (#3417)
* refactor(logging): standardize concurrency and proxy events

* chore(logging): extend guardrails for concurrency and proxies

* feat(skill): add rustfs logging governance skill
2026-06-14 01:00:26 +08:00
Henry Guo
49a893f010 feat(table-catalog): add credential endpoint boundary (#3407)
* feat(table-catalog): add credential scope boundary

* feat(table-catalog): add credential endpoint boundary

* fix(table-catalog): remove redundant scope clone

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-13 21:26:59 +08:00
Henry Guo
74604201bb test(table-catalog): add client conformance baseline (#3400)
* test(table-catalog): add client conformance baseline

* docs(table-catalog): remove internal roadmap labels

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-06-13 13:54:07 +08:00
houseme
0d68851f7a refactor(logging): standardize rustfs runtime events (#3396)
* fix(rpc): adjust log levels and reduce noise in storage RPC layer

Issue #682: pool rebalance normal flow logs were using warn! instead
of info!. Additionally, several high-frequency RPC entry points (ping,
write_stream, read_at, walk_dir) were logging at info! level, causing
unnecessary log noise in production.

Changes:
- load_rebalance_meta: warn! → info! for normal flow (3 occurrences)
- ping body decode: info! → debug! (fires on every health check)
- write_stream/read_at/walk_dir entry: info! → debug!
- metrics.rs: lowercase error messages, add structured error field
- http_service.rs: add structured error field to walk_dir failure
- Add tracing::instrument fields for start_rebalance context

* refactor(logging): standardize rustfs runtime events

* build(deps): bump workspace dependency versions

* build(deps): pin time to 0.3.47

* build(deps): update postgres client versions
2026-06-13 08:39:47 +08:00
escapecode
7a8514bdfa feat(sftp): add macOS and Windows platform support (#3372)
The session watchdog now selects its detection method per
platform. It previously probed kernel TCP state through a
Linux-only procfs path, so on macOS every healthy idle session
was killed within a minute, and on Windows the watchdog never
spawned at all, leaving wedged sessions with no cleanup. Linux
keeps its fast-kill watchdog unchanged. Other platforms get a
silence-only backstop that kills a session only at the
documented 30-minute idle ceiling.

The host-key loader now has a Windows arm. It loads OpenSSH
format host keys from the configured directory and logs a
one-time warning to restrict NTFS ACLs on the key directory,
the same operator-managed approach FTPS, WebDAV, KMS, and IAM
already use on Windows. Startup previously aborted with
UnsupportedPlatform because the Unix mode-bit permission check
has no Windows equivalent. tokio's io-uring feature is now
enabled only in Linux builds. io-uring is a Linux kernel
interface and enabling it unconditionally broke the Windows
build.

Co-authored-by: houseme <housemecn@gmail.com>
2026-06-12 18:08:23 +08:00
安正超
7d6d56a547 feat(extension): add schema contracts (#3384) 2026-06-12 15:07:17 +08:00
安正超
8df38ea12c ci(docker): add release image scan report (#3382) 2026-06-12 13:11:59 +08:00
安正超
212a0913be ci(release): emit sbom and provenance assets (#3381) 2026-06-12 12:56:02 +08:00
安正超
99e68f82a2 ci(audit): report unpinned workflow actions (#3379) 2026-06-12 11:33:23 +08:00
houseme
82af181dcf refactor(logging): unify governance runtime events (#3367) 2026-06-11 22:26:02 +08:00
GatewayJ
b5676dcc8e fix(table-catalog): support PyIceberg REST commits (#3342)
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-11 22:22:13 +08:00
houseme
0a987d870b refactor(logging): reduce runtime noise (#3363) 2026-06-11 19:49:01 +08:00
Henry Guo
7191a3abae docs(scanner): document runtime scanner controls (#3339)
* docs(scanner): document runtime scanner controls

* docs(scanner): split English and Chinese README

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
2026-06-11 12:04:34 +08:00
唐小鸭
f7724d223b feat(rio): rio_v2 is compatible with minio for storing data. (#3115)
* Set up a compatibility layer for replacing old Rio components with new ones.

* fix(rio). compress range

* feat(rio). Add the experimental feature rio_v2 to support minio data at the binary level.

* feat(rio_v2): add sse-c test

* test compression component

* simple fix

* fix minlz encode

* fix metadata

* fix kms key cache error

* Update launch.json

* ci: set nix crate download user agent

* fix: gate obs pyroscope backend

* ignore minio test

* fix encrypt check

* fix

* fix

* fix

* Update object_usecase.rs

* Update ci.yml

* fix

* ci add rio-v2 test

* fix

* ci fix

* fix

* Reconstructed into a more reasonable compatibility mode

* fix

* fix

---------

Signed-off-by: houseme <housemecn@gmail.com>
Signed-off-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-06-08 11:59:14 +00:00
安正超
dee550a831 ci: add architecture migration rule checks (#3264)
Co-authored-by: houseme <housemecn@gmail.com>
2026-06-07 16:23:38 +00:00
安正超
ae9d25879d ci: stabilize architecture layer guard (#3255) 2026-06-07 19:26:35 +08:00
安正超
7cc730d9c0 ci: isolate s3 test runner state (#3180) 2026-06-02 22:40:37 +08:00
CptOfEvilMinions
e91e513ab3 feat: Helm chart support extra volumes (#2982) 2026-06-01 16:52:35 +08:00
安正超
ac97ceb744 fix(config): restore default credential startup (#3114)
* fix(config): restore default credential startup

* fix: align e2e credentials with server env

* fix(config): restore default credential consistency

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-05-29 11:52:46 +00:00
houseme
2b82432f9e fix(ecstore): send valid ping body in remote locker (#3112)
* fix(ecstore): send valid ping body in remote locker

Build ping requests with a flatbuffer payload so health checks remain compatible with the ping response parser after restart.

* fix(bench): use multi-host warp target during failover

Normalize comma-separated warp host lists in run_object_batch_bench and let four-node failover bench pass BENCH_WARP_HOSTS so rolling restart does not pin load to a single restarting node.

* feat(health): add compat health probes with busy/KMS checks

  - Add /health/live liveness probe endpoint
  - Add busy protection (429) for readiness probes, gated by RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE
  - Add KMS readiness check for /health/ready, gated by RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE
  - Add lock quorum status caching with TTL to reduce RPC pressure
  - Consolidate health response building into build_health_response_parts
  - Register /health/live in console router and readiness gate
  - Remove MinIO references from newly added health code

* fix(health): decouple kms readiness from lock quorum
2026-05-29 08:02:50 +00:00
houseme
088c4bda43 fix(ecstore): harden issue3031 multipart validation path (#3106)
* fix(ecstore): harden issue3031 multipart validation path

- clear stale multipart part destinations before rename fan-out
- add repeated part overwrite regression coverage
- reduce remote disk startup false-fault escalation to suspect-first
- refine remote locker diagnostics and lower scanner leader-lock log noise
- add a dedicated 4-node issue3031 docker validation script

* refactor(admin): inline console version json macro

- drop the unused serde_json::json import in admin console
- call serde_json::json! inline in version_handler
- keep the console version response behavior unchanged

* fix(remote-disk): recover suspect health on probe success

- record probe success during remote disk health checks so suspect drives recover
- use async_with_vars for the remote disk health probe test
- make the missing-listener test assert the state transition more robustly
2026-05-28 14:26:31 +00:00
安正超
02a7d3c228 fix: derive run script CORS console port (#3068) 2026-05-23 04:52:46 +00:00
GatewayJ
c9f0f25f55 fix: bind run script to localhost (#3063) 2026-05-22 14:28:17 +00:00
Henry Guo
0985f0b37b feat(internode): label transport operation metrics (#3045)
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-21 15:20:39 +00:00
Henry Guo
69dcf9e6cb fix(tooling): harden internode transport benchmark setup (#3037)
* refactor(config): centralize internode transport constants

* fix(bench): guard all ripgrep calls behind dry-run check

Move require_cmd rg and metrics collection inside the non-dry-run
path so that --dry-run works on hosts without rg installed.

* feat(tooling): cross-platform protoc setup for Linux and macOS

Make install-protoc.sh support Linux (x86_64, aarch64) alongside
macOS, and bump CI protoc from 29.3 to 33.1 to match the version
required by the gproto build script.

* fix(bench): record internode baseline error counts

* fix(skill): correct YAML frontmatter formatting for release-version-bump

* chore(ci): bump protoc version to 34.1

* fix(tooling): bump protoc 33.1 to 34.1 in install script, restore SKILL.md description

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-05-21 05:47:32 +00:00
houseme
dcbffb084f chore(deps): refresh workspace deps and linux fs_type gating (#3030)
* chore(deps): refresh workspace deps and linux fs_type gating

- refresh workspace dependency pins and lockfile updates

- remove now-unused crate dependency entries in multiple Cargo.toml files

- enable profiling export defaults in config and scripts/run.sh

- gate os::fs_type module/function/tests to Linux to avoid non-Linux dead_code warnings

* fix(utils): simplify fs_type linux gating

- keep fs_type module-level linux cfg in os::mod

- remove redundant linux cfg on get_fs_type and test module

* chore(deps): bump s3s git revision

- update workspace s3s dependency to rev 507e1312b211c3ddc214b03875d6fabd15d22ed5

- refresh Cargo.lock source entry for s3s

* chore(dev): allow mysql_async git source and env overrides

- allow mysql_async git source in deny.toml allow-git list

- make scripts/run.sh core env vars overrideable via existing shell env

* fix(utils): import get_fs_type in fs_type tests

- add explicit super::get_fs_type import in fs_type test module

- fix Linux E0425 unresolved function errors in unit tests

* chore(dev): tune run script observability defaults

- make profiling export env overrideable in scripts/run.sh

- set RUSTFS_OBS_SAMPLE_RATIO default from 2.0 to 1.0

- update allow-git review window comments in deny.toml

* test(obs): stabilize profiling env alias tests
2026-05-20 15:03:07 +00:00
Henry Guo
19b69abe5c feat(internode): harden p0 transport boundary and baseline tooling (#3017)
* feat(internode): p0 transport baseline and ci hardening

* fix(internode): avoid double wrapping transport errors

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-20 11:06:26 +00:00
Henry Guo
e929814edc feat(ecstore): add internode transport boundary and TCP baseline runner (#3010)
* docs: add internode data transport RFC

* feat: add internode operation metrics

* fix feedback

* fix(ci): fallback protoc token to github.token

* feat(ecstore): add internode transport boundary and baseline runner

* feat(internode): harden data transport baseline

* Revert "feat(internode): harden data transport baseline"

This reverts commit 5b8d6b8aa4.

* fix(internode): address baseline review comments

* fix(ci): pin setup-protoc to stable release

* fix(ci): install protoc via apt on linux

* fix(ci): restore protoc install for macos and windows

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-19 22:41:34 +00:00
houseme
25c6bdf490 perf(filemeta): phase-1~3 rename_data metadata optimization (#3011)
* chore(perf): harden amd64 profiling benchmark flow

* fix(profiling): isolate bench buckets and map protobuf conflict

* perf: avoid blocking owned local writes

* style: format profile admin handler

* docs: clarify observability trace validation

* perf: reduce mkdir overhead on local writes

* perf: add rename_data meta microbenchmark

* perf(filemeta): fast-path data_dir decode in version meta

* perf(filemeta): collapse data-dir lookup into one scan

* perf(filemeta): reduce scan allocs and refresh meta bench

* perf(ecstore): skip mkdir path on read-only open

* perf(filemeta): single-pass unshared data-dir scan

* perf(filemeta): add two-key inline remove fast path

* perf(filemeta): compare remove-two keys by bytes first

* bench(ecstore): add remove_two-only micro benchmark

* bench(ecstore): stabilize rename_data meta benchmark timing

* bench(ecstore): align rename_data path with remove_two

* perf(filemeta): avoid uuid string alloc in remove_two

* perf(filemeta): add fast-path for empty inline data

* perf(filemeta): streamline add_version match branch

* perf(filemeta): fast-return remove_key on miss

* perf(filemeta): speed up add_version insertion lookup

* style(ecstore): normalize formatting in perf-tuning files

* refactor(filemeta): unify inline data removal paths
2026-05-19 10:20:24 +00:00
houseme
6e12289339 fix(runtime): finalize issue 2941 profiling cleanup (#2983)
* perf(runtime): narrow profiling support and upgrade starshard

* style(notify): normalize starshard imports

* perf(ecstore): reduce list_path_raw coordination overhead

* docs(scripts): add issue 2941 perf capture workflow

* fix(runtime): finalize issue 2941 profiling cleanup

* build(deps): bump quick-xml to 0.40.0

* chore(scripts): untrack local perf capture guide

* fix(scripts): honor label in perf capture output
2026-05-16 11:09:04 +00:00
安正超
6898e720dd fix(security): harden proxy auth and default credentials (#2981)
* fix(security): harden proxy auth and default credentials

* fix(security): address proxy and credential feedback
2026-05-16 04:01:50 +00:00
yihong
824c4f7673 docs: fix some dead links (#2975)
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-05-16 10:06:53 +08:00
安正超
f84d8983ed test(s3): promote passing SSE multipart cases (#2900) 2026-05-10 13:16:23 +00:00
安正超
6896b38dc2 test(s3): promote lifecycle expiration header tests (#2858) 2026-05-07 13:08:42 +00:00
安正超
abc07a9dc4 fix(build): quote build script features argument (#2841) 2026-05-07 02:47:53 +00:00
安正超
96d41b6349 test(helm): cover standalone scale-to-zero rendering (#2831) 2026-05-06 21:13:40 +00:00
安正超
68fcbffcb6 test(build): cover build script feature flags (#2837) 2026-05-07 05:18:51 +08:00
安正超
4728abcff1 fix(security): document unsafe and TLS overrides (#2835) 2026-05-06 15:09:02 +00:00
安正超
b10db403b6 test(s3): promote passing copy metadata case (#2832) 2026-05-06 13:58:32 +00:00
Michael Graff
3898d524fe security: same-origin console CORS, fail-closed helm creds, deny.toml, sample-config hardening (#2769)
Signed-off-by: Michael Graff <explorer@flame.org>
Signed-off-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
2026-05-06 07:34:44 +00:00
houseme
3dd0692917 refactor: unify credential env constants and deploy env usage (#2821)
Co-authored-by: Henry Guo <marshawcoco@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
2026-05-06 04:30:29 +00:00