Both nightly runs of the warp A/B rig failed at server bring-up:
`endpoint http://127.0.0.1:9000/health did not become healthy` — exactly 60s
after start. The server binds its public listener only after startup converges
(erasure-format load + IAM); its own budget
(RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS) defaults to 120s, so the rig's 60s
health poll gave up before a slow cold start on the shared sm-standard-2 runner
finished. The rustfs.log lived under /tmp (not the uploaded target/hotpath-ab/),
so the root cause was invisible in the artifact.
Root-cause + diagnosability (scripts/run_hotpath_warp_ab.sh):
- Health poll is configurable via --health-timeout, default 180s (> the 120s
server startup budget). Fails fast if the local server process exits before
becoming healthy instead of polling out the full window.
- Server logs + a startup-env file now write under OUT_DIR/server-logs/ so they
ride along in the CI artifact. On a health failure the rig dumps the last 50
log lines to the job log.
Workflow (.github/workflows/performance-ab.yml):
- On every run, write gate.md (or, on a pre-gate failure, the failing phase's
server-log tails) into $GITHUB_STEP_SUMMARY; short artifact retention.
- Short measurement matrix (--duration/--rounds/--cooldown) so the expanded
matrix fits; timeout-minutes 90 -> 120 as a Phase-0 stopgap until perf-3
caches the baseline binary (the ~65min double build dominates).
- TODO(ci-8/perf-2) hook comment for the failure-alert composite action.
Workload cells (absorbed perf-4): add put-4kib/get-4kib (the #4221 fsync
regression size, ~-10% @4KiB, previously invisible) and get-10mib (historical
large-GET EOF size) to both the PR-labeled and nightly matrices — 6 workloads
x 2 phases x 2 drive-sync = 24 cells. A 1KiB cell is deferred to protect the
budget until perf-3.
Refs rustfs/backlog#1152 (perf-1, absorbed perf-4), rustfs/backlog#1155.
fd18b1df49