rustfs/scripts/s3-tests

S3 Compatibility Tests

This directory contains scripts for running S3 compatibility tests against RustFS.

Quick Start

Run the local S3 compatibility test script:

./scripts/s3-tests/run.sh

The script will automatically:

1. Check prerequisites: Verify port availability (unless in existing mode)
2. Build/Start RustFS:
   • Build mode (default): Compile with cargo build --release (skips if binary is recent < 30 min)
   • Binary mode: Use specified or default binary path
   • Docker mode: Build Docker image and start container
   • Existing mode: Skip startup, connect to running service
3. Wait for readiness: Multi-step health check (process/container → port → log → S3 API)
4. Prepare environment:
   • Generate s3tests configuration from template
   • Provision alt user via admin API
   • Fetch s3-tests at the pinned revision (S3TESTS_REV)
   • Install missing dependencies (awscurl, tox, gettext)
5. Run tests: Execute ceph s3-tests via tox with configured filters
6. Collect results: Save logs and test results in artifacts/s3tests-${TEST_MODE}/

Deployment Modes

The script supports four deployment modes, controlled via the DEPLOY_MODE environment variable:

1. Build Mode (Default)

Compile with cargo build --release and run:

DEPLOY_MODE=build ./scripts/s3-tests/run.sh
# Or simply (build is the default)
./scripts/s3-tests/run.sh

# Force rebuild even if binary exists and is recent
./scripts/s3-tests/run.sh --no-cache

Behavior:

• Automatically compiles RustFS binary if it doesn't exist
• If binary exists and was compiled less than 30 minutes ago, compilation is skipped (unless --no-cache is specified)
• Automatically starts the service after compilation
• Automatically fetches s3-tests at the pinned revision if missing
• Automatically installs missing dependencies (awscurl, tox, gettext)
• Automatic Cleanup: Process is automatically stopped when script exits

2. Binary File Mode

Use pre-compiled binary file:

# Use default path (./target/release/rustfs)
DEPLOY_MODE=binary ./scripts/s3-tests/run.sh

# Specify custom binary path
DEPLOY_MODE=binary RUSTFS_BINARY=./target/release/rustfs ./scripts/s3-tests/run.sh

Behavior:

• Uses existing binary file (must exist, script will not compile)
• Automatically starts the service using the specified binary
• Automatically fetches s3-tests at the pinned revision if missing
• Automatically installs missing dependencies (awscurl, tox, gettext)
• Automatic Cleanup: Process is automatically stopped when script exits

3. Docker Mode

Build Docker image and run in container:

DEPLOY_MODE=docker ./scripts/s3-tests/run.sh

Behavior:

• Automatically builds Docker image using Dockerfile.source
• Creates Docker network (rustfs-net) if it doesn't exist
• Automatically fetches s3-tests at the pinned revision if missing
• Automatically installs missing dependencies (awscurl, tox, gettext)
• Automatic Cleanup: Container and network are automatically removed when script exits

4. Existing Service Mode

Connect to an already running RustFS service:

DEPLOY_MODE=existing S3_HOST=127.0.0.1 S3_PORT=9000 ./scripts/s3-tests/run.sh

# Connect to remote service
DEPLOY_MODE=existing S3_HOST=192.168.1.100 S3_PORT=9000 ./scripts/s3-tests/run.sh

Behavior:

• Skips service startup and port availability checks
• Connects directly to the specified service endpoint
• Automatically fetches s3-tests at the pinned revision if missing
• Automatically installs missing dependencies (awscurl, tox, gettext)
• Note: The service must already have the alt user (rustfsalt) provisioned, or the script will provision it automatically

Automatic Cleanup: The script uses trap handlers to automatically clean up resources when it exits (success or failure):

• Stops RustFS process (build/binary mode)
• Stops and removes Docker container (docker mode)
• Removes Docker network (docker mode)

Configuration Options

Command Line Options

• -h, --help: Show help message
• --no-cache: Force rebuild even if binary exists and is recent (for build mode)

Deployment Configuration

• DEPLOY_MODE: Deployment mode, options:
  • build: Compile with cargo build --release and run (default)
  • binary: Use pre-compiled binary file
  • docker: Build Docker image and run in container
  • existing: Use already running service
• RUSTFS_BINARY: Path to binary file (for binary mode, default: ./target/release/rustfs)
• DATA_ROOT: Root directory for test data storage (default: target)
  • Final path: ${DATA_ROOT}/test-data/${CONTAINER_NAME}
  • Example: DATA_ROOT=/tmp stores data in /tmp/test-data/rustfs-single/

Service Configuration

• S3_ACCESS_KEY: Main user access key (default: rustfsadmin)
• S3_SECRET_KEY: Main user secret key (default: rustfsadmin)
• S3_ALT_ACCESS_KEY: Alt user access key (default: rustfsalt)
• S3_ALT_SECRET_KEY: Alt user secret key (default: rustfsalt)
• S3_REGION: S3 region (default: us-east-1)
• S3_HOST: S3 service host (default: 127.0.0.1)
• S3_PORT: S3 service port (default: 9000)

Test Parameters

• TEST_MODE: Test mode (default: single)
• TEST_SCOPE: Test scope (default: implemented)
  • implemented: run only the implemented_tests.txt whitelist (the PR gate)
  • all: run the entire upstream suite (used by the weekly full sweep)
• MAXFAIL: Stop after N failures, 0 = never stop (default: 1)
• XDIST: Enable parallel execution with N workers (default: 0, disabled)
• S3TESTS_REPO: s3-tests repository URL (default: https://github.com/ceph/s3-tests.git)
• S3TESTS_REV: Pinned s3-tests commit for reproducible runs
  • Bump deliberately: upstream changes can rename tests or change assertions, so a bump usually requires reclassifying the test list files
• MARKEXPR: pytest marker expression for filtering tests
  • Default: no marker filtering; file-based test lists control the selected tests
  • Can be customized to test specific marker groups
• TESTEXPR: optional pytest -k expression for custom runs
  • Default: exact pytest node ids loaded from implemented_tests.txt
  • Setting TESTEXPR overrides the implemented test list

Configuration Files

• S3TESTS_CONF_TEMPLATE: Path to s3tests config template (default: .github/s3tests/s3tests.conf)
  • Relative to project root
  • Uses envsubst to substitute variables (e.g., ${S3_HOST})
• S3TESTS_CONF: Path to generated s3tests config (default: s3tests.conf)
  • Relative to project root
  • This file is generated from the template before running tests

Examples

Build Mode (Default)

# Basic usage - compiles and runs automatically
# Skips compilation if binary exists and is less than 30 minutes old
./scripts/s3-tests/run.sh

# Force rebuild (skip cache check, always compile)
./scripts/s3-tests/run.sh --no-cache

# Run all tests, stop after 50 failures
MAXFAIL=50 ./scripts/s3-tests/run.sh

# Enable parallel execution (4 worker processes)
# Automatically installs pytest-xdist if needed
XDIST=4 ./scripts/s3-tests/run.sh

# Use custom data storage location
# Data will be stored in /tmp/test-data/rustfs-single/
DATA_ROOT=/tmp ./scripts/s3-tests/run.sh

# Run specific test markers (e.g., test multipart uploads only)
MARKEXPR="multipart" ./scripts/s3-tests/run.sh

Binary File Mode

# First compile the binary
cargo build --release

# Run with default path
DEPLOY_MODE=binary ./scripts/s3-tests/run.sh

# Specify custom path
DEPLOY_MODE=binary RUSTFS_BINARY=/path/to/rustfs ./scripts/s3-tests/run.sh

# Use binary with parallel tests
DEPLOY_MODE=binary XDIST=4 ./scripts/s3-tests/run.sh

Docker Mode

# Build Docker image and run in container
DEPLOY_MODE=docker ./scripts/s3-tests/run.sh

# Run with parallel tests
DEPLOY_MODE=docker XDIST=4 ./scripts/s3-tests/run.sh

Existing Service Mode

# Connect to locally running service (default: 127.0.0.1:9000)
DEPLOY_MODE=existing ./scripts/s3-tests/run.sh

# Connect to remote service
DEPLOY_MODE=existing S3_HOST=192.168.1.100 S3_PORT=9000 ./scripts/s3-tests/run.sh

# Test specific features (custom marker expression)
DEPLOY_MODE=existing MARKEXPR="not lifecycle and not versioning" ./scripts/s3-tests/run.sh

# Use custom credentials
DEPLOY_MODE=existing \
  S3_ACCESS_KEY=myaccesskey \
  S3_SECRET_KEY=mysecretkey \
  ./scripts/s3-tests/run.sh

KMS Test Fixtures

Managed deployments enable local KMS and provision two distinct public test keys. S3_KMS_KEY_ID selects the primary key (default rustfs-s3tests-default-key), and S3_KMS_SECONDARY_KEY_ID selects the second key (default rustfs-s3tests-secondary-key). Their IDs and key material differ so encrypted copy tests exercise a change of key. The generated config sets kms_keyid and kms_keyid2 explicitly.

S3_KMS_DEFAULT_KEY_ID defaults to the primary key for managed local KMS. Set it to an empty value to start without a default key. The generated kms_default_keyid setting fixes the expectation for an SSE-KMS PUT without a key ID: with a configured default, the patched upstream test requires a successful PUT, HEAD and GET with that key ID and matching object contents; without one, the request must fail. The test remains in the implemented gate.

For DEPLOY_MODE=existing, the harness does not provision keys or infer the server's default. Set both key IDs to keys that already exist, and set S3_KMS_DEFAULT_KEY_ID only when the target has a default. Custom config templates should include kms_keyid, kms_keyid2, and kms_default_keyid, or the commented placeholders from the bundled template.

Conditional write contract

The conditional-write patch keeps the upstream tests in the gate while aligning their requests with the S3 write contract: If-None-Match accepts *, and an existing current object returns 412. RustFS returns 501 NotImplemented for unsupported ETag-valued If-None-Match; the S3 documentation does not specify that error code for PutObject or CompleteMultipartUpload. The patched tests assert rejection without replacing object contents, successful creation after a delete marker, and the existing If-Match checks. No conditional-write tests are excluded.

Custom Configuration Files

# Use custom config template and output path
S3TESTS_CONF_TEMPLATE=my-configs/s3tests.conf.template \
S3TESTS_CONF=my-s3tests.conf \
./scripts/s3-tests/run.sh

Test Results

Test results are saved in the artifacts/s3tests-${TEST_MODE}/ directory (default: artifacts/s3tests-single/):

• junit.xml: Test results in JUnit format (compatible with CI/CD systems)
• pytest.log: Detailed pytest logs with full test output
• all-collected-nodeids.txt: Exact node IDs in the pinned upstream suite
• selected-nodeids.txt: Exact node IDs expected in this run
• unsharded-selected-nodeids.txt: Exact node IDs before deterministic sharding
• compat-report.md: Classification report generated by report_compat.py — regressions against implemented_tests.txt, promotion candidates (tests that pass but are still listed as unimplemented/excluded), and tests missing from every list
• rustfs-${TEST_MODE}/rustfs.log: RustFS service logs
• rustfs-${TEST_MODE}/inspect.json: Service metadata (PID, binary path, mode, etc.)

View results:

# Check test summary
cat artifacts/s3tests-single/junit.xml | grep -E "testsuite|testcase"

# View test logs
less artifacts/s3tests-single/pytest.log

# View service logs
less artifacts/s3tests-single/rustfs-single/rustfs.log

Prerequisites

Required System Dependencies

The following dependencies must be installed manually on your system:

All Deployment Modes

• Python 3: Required for running s3-tests

  • Check: python3 --version
  • Install:
    • macOS: Usually pre-installed, or brew install python3
    • Linux: apt-get install python3 or yum install python3

• Git: Required for cloning s3-tests repository

  • Check: git --version
  • Install:
    • macOS: Usually pre-installed, or brew install git
    • Linux: apt-get install git or yum install git

• Port checking tools: One of the following for port availability checks

  • nc (netcat): apt-get install netcat or brew install netcat
  • OR timeout command: Usually pre-installed on Linux
  • OR bash built-in TCP redirection support

Docker Mode Only

• Docker: Required only when using DEPLOY_MODE=docker
  • Check: docker --version
  • Install: Docker Installation Guide

Build Mode Only

• Rust toolchain: Required when using DEPLOY_MODE=build (default)
  • Check: rustc --version and cargo --version
  • Install: Rust Installation Guide

Auto-installed Dependencies

The script will automatically install the following dependencies if missing (no manual action required):

• awscurl: For S3 API calls and user provisioning

  • Installed via: python3 -m pip install --user --upgrade pip awscurl
  • Location: $HOME/.local/bin/awscurl

• tox: For running s3-tests in isolated Python environment

  • Installed via: python3 -m pip install --user --upgrade pip tox
  • Location: $HOME/.local/bin/tox

• gettext-base: For envsubst command (config file generation)

  • macOS: Automatically installs via brew install gettext
  • Linux: Automatically installs via sudo apt-get install gettext-base
  • Note: macOS installation may require manual intervention if brew fails

• s3-tests repository: Automatically fetched if not present

  • Source: https://github.com/ceph/s3-tests.git, pinned to the commit in S3TESTS_REV (see run.sh) for reproducible runs
  • Location: ${PROJECT_ROOT}/s3-tests

Note: The script adds $HOME/.local/bin to PATH automatically, so auto-installed Python tools are accessible.

Proxy Configuration

The script automatically disables proxy for localhost requests to avoid interference. All proxy environment variables (http_proxy, https_proxy, HTTP_PROXY, HTTPS_PROXY) are unset at script startup. The NO_PROXY variable is set to 127.0.0.1,localhost,::1.

Cleanup

The test script automatically cleans up processes and containers when it exits. However, if you need to manually clean up:

Using the Cleanup Script

A dedicated cleanup script is available to clean up test resources:

# Report port 9000 usage and clean the test data directory
./scripts/s3-tests/cleanup.sh

# Use custom port
S3_PORT=9001 ./scripts/s3-tests/cleanup.sh

The cleanup script will:

• Report any process using the specified port (default: 9000)
• Clean test data directory at target/test-data/rustfs-single/
• Leave unrelated processes running

Manual Cleanup

If the cleanup script doesn't work or you need more control:

# Kill process on port 9000
lsof -ti:9000 | xargs kill -9

# Or use netstat/ss
kill -9 $(netstat -tuln | grep :9000 | awk '{print $7}' | cut -d'/' -f1)

# Remove test data
rm -rf target/test-data/rustfs-single/

# Stop Docker container (if using docker mode)
docker rm -f rustfs-single

# Remove Docker network (if using docker mode)
docker network rm rustfs-net

Troubleshooting

Port Already in Use

If port 9000 is already in use, change the port:

S3_PORT=9001 ./scripts/s3-tests/run.sh

Note: The script automatically checks if the port is available before starting (except in existing mode). If the port is in use, the script will exit with an error message.

Container Start Failure

Check Docker logs:

docker logs rustfs-single

Binary Not Found

For binary mode, ensure the binary is compiled:

cargo build --release

Or specify the correct path:

DEPLOY_MODE=binary RUSTFS_BINARY=/path/to/rustfs ./scripts/s3-tests/run.sh

Test Timeout

Increase wait time or check service status:

curl http://127.0.0.1:9000/health

Existing Service Not Accessible

For existing mode, ensure the service is running and accessible:

# Check if service is reachable
curl http://192.168.1.100:9000/health

# Verify S3 API is responding
awscurl --service s3 --region us-east-1 \
  --access_key rustfsadmin \
  --secret_key rustfsadmin \
  -X GET "http://192.168.1.100:9000/"

Workflow Integration

This script is the single source of truth for running ceph s3-tests against RustFS. Two GitHub Actions workflows delegate to it:

• PR gate (.github/workflows/ci.yml, job s3-implemented-tests): runs the implemented_tests.txt whitelist against a single-node debug binary on every pull request (DEPLOY_MODE=binary, MAXFAIL=0, XDIST=4). Any failure blocks the PR.
• Full sweep (.github/workflows/e2e-s3tests.yml): weekly scheduled (and manually dispatchable) run of the ENTIRE upstream suite (TEST_SCOPE=all) against a Docker deployment — single node or a real 4-node distributed cluster behind HAProxy. Regressions, unclassified tests, incomplete execution, and infrastructure errors fail the sweep; classified unsupported behavior remains informational. Scheduled topology runs are split into four deterministic exact-node-ID shards, and every case has a five-minute timeout, so one stalled case cannot erase the entire sweep's evidence.

Keeping both workflows on this script means local runs, the PR gate, and the scheduled sweep always execute tests the same way (same pinned s3-tests revision, same config template, same user provisioning).

A third workflow, mint (.github/workflows/mint.yml), complements ceph s3-tests with MinIO Mint: the functional suites of many real client SDKs and tools (awscli, mc, aws-sdk-java/go/php/ruby, minio-go/java/js/py, s3cmd, ...). It runs weekly, is report-only for test failures, and publishes a per-suite pass/fail table in the job summary.

Companion Tools

• report_compat.py — diffs a junit.xml result against the classification lists and the exact pytest collection; run before execution to reject stale or missing classifications, then after execution to detect regressions and incomplete parameterized cases.

• api_coverage.py — quantifies S3 API surface coverage by comparing the s3s S3 trait (at the revision pinned in Cargo.toml) against the methods RustFS overrides in impl S3 for FS:

  python3 scripts/s3-tests/api_coverage.py            # summary + missing ops
  python3 scripts/s3-tests/api_coverage.py --output api-coverage.md
  

• compare_dual_targets.py — sends one signed S3 request to two endpoints (e.g. RustFS and MinIO) and diffs status, headers, and body; useful when triaging a behavioral difference found by the suites above.

See Also

• GitHub Actions Workflow
• S3 Tests Configuration
• Ceph S3 Tests Repository