rustfs/docs/operations/kms-backend-security.md

# KMS backend security properties

**Use this when:** choosing a KMS backend, scheduling or debugging master key rotation, planning a rolling upgrade of a cluster with KMS enabled, or auditing where master key material lives and who can read it.
**Source of truth:** `crates/kms/src/config.rs` (`KmsBackend`, `ENV_KMS_*` constants), `crates/kms/src/backends/{local,vault,vault_transit,aws}.rs`, `crates/kms/src/encryption/dek.rs` (`DataKeyEnvelope`), `rustfs/src/admin/route_policy.rs` (KMS route actions).

RustFS ships several KMS backends. They differ not only in deployment effort but in **where master key material lives and who can read it**. Pick a backend based on the confidentiality boundary you need, not on the name alone. Related: [Vault KMS authentication runbook](vault-kms-authentication.md) (credential sources, refresh, fail-closed window), [Cryptographic compliance positioning](kms-cryptographic-compliance.md), [Per-key KMS authorization](kms-per-key-authorization.md), [KMS admin API contract](kms-admin-contract.md), [KMS observability runbook](kms-observability-runbook.md).

## Backend comparison

| Backend | Config tag | Master key material location | At-rest protection of key material | Durability | Rotation | Intended use |
| --- | --- | --- | --- | --- | --- | --- |
| Local | `Local` | Files under `key_dir`, encrypted with the configured local master key | Local master key (AES-GCM) + file permissions | Crash-durable commits on local filesystems only; see [Local backend durability and deployment support matrix](#local-backend-durability-and-deployment-support-matrix) | Rejected by design (single material) | Development, testing and demos only; not supported for production |
| Static | `Static` | Provided out-of-band via environment/file; never persisted by RustFS | Operator-managed secret distribution | No state persisted by RustFS | Rejected (read-only backend) | Development and testing with an externally supplied key; not supported for production |
| Vault KV2 | `VaultKV2` (legacy alias `Vault`) | Stored **directly** in Vault KV v2 (Base64-encoded plaintext) | Vault ACLs + KV v2 at-rest encryption + TLS only | Delegated to Vault storage | Versioned retention (immutable per-version records + current pointer) | Deployments that accept Vault KV ACLs as the sole confidentiality boundary |
| Vault Transit | `VaultTransit` | Key-encryption keys never leave Vault; only Transit ciphertext is visible outside | Vault Transit engine (cryptographic isolation) | Delegated to Vault storage | Via Vault Transit key versioning | Deployments that need key material to be unreadable through storage APIs |
| AWS KMS | `AWS` (alias `AwsKms`) | Key material never leaves AWS KMS; RustFS mirrors no key state | AWS KMS (cryptographic isolation) + IAM | Delegated to AWS | On-demand `RotateKeyOnDemand`; prior backing keys stay usable for decryption | Deployments rooted in AWS IAM — read [AWS KMS: deviations from the shared backend contract](#aws-kms-deviations-from-the-shared-backend-contract) first |

## Migrating from MinIO: encrypted objects do not carry over

> **Warning: default RustFS builds fail closed on objects that MinIO encrypted.** This applies to SSE-S3, SSE-KMS, and SSE-C, whichever KMS backend you configure; configuring `Static` with MinIO's key material does not make them readable. Such objects list and HEAD normally (their `xl.meta` parses), and only the payload read fails — with S3 `InvalidObjectState`, never plaintext. Read a sample of encrypted objects, not just their listings, before decommissioning the MinIO deployment.

The read path exists behind the `rio-v2` feature as a migration-only build, the reverse direction (RustFS-written SSE objects read by MinIO) is unsupported, and the migration options are enumerated in [MinIO file-format interoperability, Part C](../architecture/minio-file-format-compat.md#part-c--server-side-encryption-sse). The AWS KMS and MinIO KES wire protocols are non-targets of that document and of this one.

## Vault KV2: what the backend does and does not do

The Vault KV2 backend uses Vault purely as a **secure storage** service:

- Master key material is generated by RustFS and written to KV v2 as a Base64-encoded value (`encrypted_key_material` is an encoding, not a ciphertext).
- The backend never calls the Vault Transit engine. The `mount_path` configuration field and the `RUSTFS_KMS_VAULT_MOUNT_PATH` environment variable are deprecated leftovers: accepted for compatibility and ignored.
- Data-encryption keys (DEKs) handed to the object-encryption path are still wrapped with AES-256-GCM under the master key; the statement above concerns the master key's storage in Vault, not the DEK envelope.
- No runtime interface reports this boundary. `GET /rustfs/admin/v3/kms/status` names the active backend (`backend_type: vault-kv2`) and returns a `capabilities` matrix, but that matrix enumerates only supported operations. Determining the confidentiality boundary in force means reading `backend_type` and applying the comparison table above.
- The `at_rest_protection: storage-only` field carried by a KMS backup manifest declares the protection state of key material inside a backup bundle, not a property the running backend reports about itself.
- Key rotation retains every historical master key version as an immutable record under `{prefix}/{key_id}/versions/{N}` and only then moves the current-version pointer; see [Master key rotation](#master-key-rotation-retention-destruction-and-upgrade-ordering).

> **Warning: KV read access is equivalent to holding the master keys.** Any Vault identity (token, AppRole, or policy) that can `read` the RustFS key path in KV v2 can recover the plaintext master key material and decrypt every object protected by those keys. If this is not acceptable, use the Vault Transit backend instead.

## Minimal Vault policy for the KV2 backend

Scope the RustFS token/AppRole to exactly the KV v2 mount and key prefix it is configured with (defaults shown: mount `secret`, prefix `rustfs/kms/keys`), and grant no other identity read access to that subtree:

```hcl
# RustFS KMS (Vault KV2 backend) — key storage only, no Transit access needed.
path "secret/data/rustfs/kms/keys/*" {
  capabilities = ["create", "read", "update"]
}

path "secret/metadata/rustfs/kms/keys/*" {
  capabilities = ["list", "read", "delete"]
}
```

- The trailing wildcards also cover the per-version material records under `.../keys/{key_id}/versions/{N}`; no extra policy paths are needed.
- `delete` on the metadata path is required only for permanent key deletion (`force_immediate`); drop it if you never hard-delete keys. RustFS refuses `force_immediate` unless the server sets `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION=true`, so leaving that gate off keeps the capability unreachable whatever the Vault policy allows.
- Do not attach `sudo`, wildcard mounts, or Transit paths; the KV2 backend does not use them.
- Audit KV reads on the key prefix: every read event is a potential master-key disclosure.

## Master key rotation: retention, destruction, and upgrade ordering

Rotation is reachable through `POST /rustfs/admin/v3/kms/keys/rotate` (`kms:RotateKey`, high risk); it is not exposed on the S3 surface. Local and Static advertise no `rotate` capability (`capabilities.rotate` is false in the `kms/status` response) and reject rotation with `UnsupportedCapability`. Vault Transit delegates rotation to the Transit engine's own versioning (ciphertext is version-prefixed, e.g. `vault:v1:...`). Vault KV2 rotates by retaining every historical version, as described below.

### Rotation drivers and scheduling, per backend

The rotate endpoint is one API over three different mechanisms, and which component performs the rotation decides how periodic rotation must be scheduled.

| Backend | Can rotate | Who performs the rotation | How to schedule periodic rotation | Wrap ceiling |
| --- | --- | --- | --- | --- |
| Local | No | Nobody — the rotate endpoint is refused with `UnsupportedCapability` | Cannot be scheduled; migrating to a rotating backend is the only path | Unmitigable |
| Static | No | Nobody — same refusal; the material is supplied out-of-band and read-only | Cannot be scheduled; migrate | Unmitigable |
| Vault KV2 | Yes | **RustFS** owns the protocol: freeze the outgoing material as an immutable version record, persist the new material, move the current pointer with a check-and-set write | Exactly **one external scheduler** (cron, Kubernetes CronJob) calling the rotate endpoint with credentials scoped to `kms:RotateKey` | Reset by each rotation |
| Vault Transit | Yes | **Vault's Transit engine**; RustFS forwards the call and records the version bump | Vault's native `auto_rotate_period` on the Transit key. Do **not** also drive the RustFS endpoint: two owners of the version cadence means neither configured period holds. The reported key version advances only through RustFS, so on an auto-rotating key treat it as a floor | Not applicable (wraps inside Vault) |
| AWS KMS | Yes | **AWS** — the endpoint maps to `RotateKeyOnDemand` | AWS's native automatic rotation. Do **not** drive periodic rotation through the RustFS endpoint: AWS caps lifetime on-demand rotations, so a scheduler exhausts the quota and then fails forever. Keep the endpoint for incidents. RustFS neither enables nor observes AWS automatic rotation and records no rotation timestamp, so its rotation-age signals measure key age on this backend | Not applicable (wraps inside AWS) |

**Wrap ceiling.** Where RustFS wraps DEKs locally (Local, Static, Vault KV2) every DEK is wrapped with AES-256-GCM under the master key using a random 96-bit nonce, and NIST SP 800-38D caps AES-GCM at 2^32 invocations per key under random nonces. Each encrypted-object write is one wrap, so the count tracks lifetime encrypted writes. Rotation installs fresh material and restarts the count; on Local and Static there is no rotation, so the ceiling can only be escaped by migrating.

**Why there is no built-in rotation timer (KV2).** A timer inside the server cannot verify the [upgrade-before-first-rotation constraint](#upgrade-before-first-rotation-hard-constraint), and rotation is not idempotent: without leader election, N nodes on the same schedule would advance the key version N times per period. Verify that your scheduler keeps up with the [rotation readiness fields](#rotation-readiness-reported-never-acted-on) and the `KmsKeyRotationOverdue` alert in the [KMS observability runbook](kms-observability-runbook.md#kmskeyrotationoverdue).

**Pre-rotation checklist** (before the first rotation of any key, and before enabling any schedule):

1. Every node runs a build that understands the `master_key_version` envelope field — the [hard constraint](#upgrade-before-first-rotation-hard-constraint) below. A timer cannot check this; you must.
2. No rolling upgrade is in progress — see [Do not do these during a mixed-version window](#do-not-do-these-during-a-mixed-version-window).
3. The [retention and destruction preconditions](#retention-and-destruction-preconditions) are understood: every version record a stored DEK envelope references must remain readable, and no retention tooling prunes the version subtree.
4. For KV2, exactly one scheduler exists.
5. `RUSTFS_KMS_ROTATION_MAX_AGE_SECS` is set to the rotation period your policy requires, so the per-key `rotation_due` verdict and the rotation-age alert verify the schedule instead of assuming it.

### Rotation readiness: reported, never acted on

RustFS reports which keys have outlived a period you configure; nothing consults the verdict before encrypting or decrypting, and it has no effect on readiness or liveness.

| Setting | Meaning | Unset or unparsable | Floor |
| --- | --- | --- | --- |
| `RUSTFS_KMS_ROTATION_MAX_AGE_SECS` | Rotation period in whole seconds; keys older than this report `rotation_due` with reason `age` or `never_rotated` | No age verdict is reported (a warning is logged for an unparsable value) — how often keys must rotate is a compliance decision, not a built-in default | 1 hour |
| `RUSTFS_KMS_ROTATION_MAX_WRAPS` | Data keys one key's material may wrap before `rotation_due` with reason `wraps` | No wrap verdict is reported | 1,000,000 (wraps are accounted in reserved blocks of that size) |

`GET /rustfs/admin/v3/kms/keys` carries `rotation_due` and `rotation_due_reason` per key: `age` (rotated, but longer ago than the period), `never_rotated` (in use longer than the period, never rotated), `wraps` (wrap budget exceeded; wins over `age` when both hold, because the AES-GCM ceiling is not negotiable), or `unsupported` (the backend cannot rotate). Only backends where RustFS wraps locally and can rotate count wraps (Vault KV2); Transit and AWS report no count. `GET /rustfs/admin/v3/kms/keys/{key_id}` does **not** carry these fields — its response records a creation date but no rotation timestamp, so read the verdict from the listing.

### Vault KV2 versioned retention model

Each rotation writes the new version's material to `{prefix}/{key_id}/versions/{N}` as an immutable, create-only record, and only after that material is durably persisted does a check-and-set write move the top-level record (the current-version pointer, which also mirrors the current material as a fast path). The first rotation additionally freezes the pre-rotation material as a version record and pins it as the key's `baseline_version`; DEK envelopes written before versioning existed (no `master_key_version` field) always resolve to that baseline, never to whatever version is current.

Decryption loads exactly the version recorded in the envelope and fails closed with a typed `KeyVersionNotFound` error when that version's record is missing. There is deliberately no fallback to the current material: falling back would silently feed the wrong key to AEAD and mask tampered envelopes.

### Retention and destruction preconditions

- Every version record that any stored DEK envelope references must remain readable. The bulk rekey sweep (`POST /rustfs/admin/v3/kms/keys/rekey`, `kms:Rekey`) rewraps stored envelopes onto the current version; until a sweep has completed with zero failures after the last rotation, assume **every** version of a rotated key is referenced. A completed sweep is evidence, not authority — the deletion gate stays the decision point. Replication strips encryption metadata in transit, so each replica site runs its own sweep.
- Version records are ordinary KV v2 secrets under the key subtree. Never run `kv metadata delete` or `kv destroy` against `{prefix}/{key_id}/versions/*`, and do not apply `delete-version-after` or retention tooling to that subtree. Each version record has a single KV revision, so KV `max-versions` settings neither protect nor endanger history — but metadata deletion removes a record entirely.
- Permanent key deletion through RustFS (`force_immediate` after `PendingDeletion`) purges the key's version records together with the key record; that is the only supported way to remove them. It requires `RUSTFS_KMS_ALLOW_IMMEDIATE_DELETION=true` on the server and a `DELETE` with a JSON body that sets `force_immediate` and echoes the key id as `confirm_key_id`; the query-parameter form is refused outright. Leave the gate off except while actively destroying keys — the pending-deletion window plus `CancelKeyDeletion` is the only recovery path for objects encrypted under the key.
- `force_immediate` is refused with `409 Conflict` while any bucket's default encryption configuration names the key or the key is the KMS service default key. A scheduled deletion is not refused for that reason: it destroys nothing and stays cancellable, and the background sweep re-checks the same references before destroying material.
- For Vault Transit, retention is governed by the Transit key's `min_decryption_version`: never raise it above the oldest version that may still protect live ciphertext.

### Reading the `impact` section

`DeleteKey` responses always carry an `impact` section listing the configuration that points at the key (buckets whose default encryption names it, and whether it is the service default key). `DescribeKey` (`GET /rustfs/admin/v3/kms/keys/{key_id}`) returns the same section only when asked with `impact=true`, because collecting it lists every bucket; a value other than `true`/`false` is rejected with `400`. **An absent section means "not collected", never "nothing references this key".**

`coverage.scanned` names the sources that were read and `coverage.not_scanned` the ones that were not — which currently includes every object encrypted under the key. `completeness` is `exact` only over the scanned sources and `unavailable` when a source could not be read; both an unreadable source and an outstanding reference stop the sweep from destroying material. **An empty `references` list does not mean the key is unused:** no object metadata is consulted, so a key with no configuration references can still protect live data.

### Upgrade before first rotation (hard constraint)

Do not rotate any key until **every** RustFS node runs a build that understands the `master_key_version` envelope field. Older binaries ignore the field and always decrypt with the current material: harmless while nothing has been rotated, but after a rotation they fail to decrypt every object wrapped by an earlier key version. Complete the rolling upgrade of the entire cluster first, then rotate. The rest of this constraint class is collected in [Mixed-version clusters during a rolling upgrade](#mixed-version-clusters-during-a-rolling-upgrade).

## Mixed-version clusters during a rolling upgrade

During a rolling upgrade KMS state is shared three ways: **Vault** holds key records and Transit metadata, **cluster storage** holds the persisted KMS configuration, and **each node's process memory** holds caches and the live backend instance. Nodes on different builds agree on the first, may disagree on the third, and can disagree on configuration for as long as the operator leaves them running, because the reload broadcast that converges configuration is one of the things an older build rejects. This section is written for the KV2 and Transit backends; the Local backend is unsupported for multi-node deployments regardless of version (see the [deployment support matrix](#deployment-support-matrix)).

### Persisted formats are backward compatible in both directions

No coordinated format cutover is required; the compatibility is deliberate and covered by decode tests.

| Record | Compatibility mechanism | Caveat |
| --- | --- | --- |
| DEK envelopes | `DataKeyEnvelope::master_key_version` is optional and omitted when absent, so envelopes from non-rotating backends stay byte-identical to the historical seven-field JSON. An upgraded node resolves a pre-versioning envelope to the key's `baseline_version`, or to the current version for a never-rotated key. Unknown fields are skipped; a bounded field-name sample is logged at a rate-limited `warn` and counted by `rustfs_kms_persisted_unknown_fields_total{record_kind="data-key-envelope"}` | An older binary reading a *new* envelope ignores the version field and decrypts with the current material — the rotation constraint above |
| Local key records | Each `<key_id>.key` carries `format_version: 1`; records without it default to 1. A reader accepts a version at most the one it understands and rejects newer with `UnsupportedFormatVersion` before decrypting. Unknown fields are accepted and counted by `rustfs_kms_persisted_unknown_fields_total{record_kind="local-key-record"}` | Once a future version greater than 1 has written a record, do not roll back to a build predating the marker |
| KV2 key records | `baseline_version` is read with a serde default; `None` means "never rotated" | An old build drops the field on write-back (see below) |
| Transit metadata records | Decode on either build | — |

### DEK envelope context binding (`RUSTFS_KMS_ENVELOPE_AAD`)

Historically the KV2 and Local backends sealed only the DEK plaintext; the `encryption_context` rode in the envelope unauthenticated and was checked by field comparison alone, so a party able to rewrite the stored envelope could rewrite the context. With `RUSTFS_KMS_ENVELOPE_AAD=true`, newly wrapped envelopes bind the canonical context bytes as AES-GCM additional data and carry `context_binding: 1`; rewriting the stored context, or stripping the flag, then fails authentication. Static, Vault Transit and AWS already bound the context through their own mechanisms and are unaffected.

Rollout constraint: reading bound envelopes needs no switch, but **a node that predates the field cannot open them** — its unwrap runs without the additional data and fails authentication. The switch defaults off (`ENV_KMS_ENVELOPE_AAD` in `crates/kms/src/config.rs`); enable it only after every node runs a release that understands `context_binding`, mirroring the `RUSTFS_ENCRYPTION_FRAME_V2` rollout. With the switch on, a rewrap sweep upgrades unbound envelopes to the bound format (converging to zero writes on re-run); a bound envelope never regresses to the unbound shape, and an envelope carrying an unrecognized `context_binding` value is refused rather than decrypted without its binding.

### Guarantees that hold only once every node is upgraded

These are properties of builds from `1.0.0-rc.1` onward; a single older node removes them for the whole cluster.

| Guarantee | Upgraded behaviour | What an old node does |
| --- | --- | --- |
| Check-and-set lifecycle writes | Every KV2 lifecycle mutation (create, enable, disable, tag, schedule/cancel deletion) and every Transit metadata write is a versioned read followed by a check-and-set write, retried on conflict | Writes blind, so it can overwrite a check-and-set commit without any conflict being reported — the lost update the change eliminated |
| `baseline_version` survives write-back | Preserved | Reads the record without error and drops `baseline_version` on any write-back; the key then resolves pre-versioning envelopes to the current version, which after a rotation is the wrong material |
| `wrap_budget_reserved` only overestimates | The KV2 record's wrap counter (behind `rustfs_kms_max_key_wrap_operations`) is reserved in blocks and never understates | Drops the field on write-back, regressing the count toward zero. Nothing breaks — the counter is advisory and the next reservation re-establishes a floor — but do not trust a *low* reading taken during or shortly after a mixed-version window |
| Version-record awareness | Version records under `{prefix}/{key_id}/versions/{N}` are create-only (check-and-set of 0), so two nodes racing a version number produce exactly one creator | Never reads or writes the sub-path, and its key listing reports the KV2 directory entry (`my-key/`) as though it were a key |

### Windows in which nodes can legitimately disagree

Even with every node on the same build, some state is process-local. These windows are bounded by design, except the last one.

| What can diverge | Bound | Mechanism |
| --- | --- | --- |
| Transit key lifecycle state used by the `encrypt` and `generate_data_key` gates | ≤ `METADATA_CACHE_TTL` (300 s, not tunable — this cache gates cryptographic operations) | Per-node in-process Transit metadata cache, TTL- and capacity-bounded, with targeted invalidation when a data-path call reports the key gone server-side. Builds older than `1.0.0-rc.1` held this cache with no TTL and no capacity bound: on such a node the window is "until the process restarts" |
| `describe_key` output | One metadata cache TTL: `cache_ttl_seconds` from the KMS configure request, 300 s when omitted, clamped down to 24 h at use (clamped rather than rejected; zero is refused while caching is enabled) | Manager-level key metadata cache; a reporting cache the KV2 state gates never read. `kms service-status` and the configuration endpoint report the effective post-clamp value. Builds older than `1.0.0-rc.1` ignored `cache_ttl_seconds` and ran 300 s while persisting 3600 s as the default, so a cluster configured before then widens to the stored 3600 s on upgrade — read the reported value back |
| KV2 key lifecycle state | None | The KV2 backend re-reads the key record from Vault for every lifecycle and data-key operation |
| Active KMS configuration | One best-effort reload broadcast; unbounded for any peer that did not apply it | See below |

### Configuration changes converge through a best-effort peer reload

`POST /rustfs/admin/v3/kms/configure` and `/kms/reconfigure` persist the new configuration to cluster storage at `config/kms_config.json`, switch the KMS service **on the node that handled the request**, and broadcast a reload signal once to every peer. A peer that accepts the signal re-reads the persisted configuration and reconfigures itself; a peer already running that configuration treats it as a no-op. Convergence is best effort and the request never fails on account of a peer:

- There is no background retry. A peer that is unreachable, whose build predates the KMS subsystem, or whose reload fails keeps its previous configuration until a later `reconfigure` reaches it or it restarts. For those peers the split is unbounded.
- The admin response reports success either way, but its message names every peer that did not converge, and the server logs one `kms_peer_config_reload_failed` warning per peer.
- While a split lasts, both configurations are live: if the change switched backends, or changed the Vault mount or key prefix, nodes write new key material to different places and a key created through one node is invisible to the others.

`GET /rustfs/admin/v3/kms/service-status` returns a `cluster_config` object holding one redacted configuration fingerprint per node plus a `consistent` flag, true only when every node answered with the same fingerprint (an unreachable peer, a build reporting no fingerprint, and an unconfigured node each read as divergent). Secrets are substituted out before fingerprinting, so the field detects a configuration split, not a credential split. Treat a `configure` or `reconfigure` whose response names unconverged peers as unfinished: re-issue it once those peers are reachable, or restart them.

### Recommended rolling upgrade order

Follow the node-at-a-time procedure in the [multi-node restart runbook](rolling-restart.md); this adds the KMS-specific sequencing.

1. **Freeze KMS administrative traffic** for the duration: no key creation, enable, disable, tagging, schedule-deletion, cancel-deletion, rotation, or reconfiguration. Object read and write traffic continues normally.
2. **Upgrade one node at a time**, waiting for each to report ready before starting the next.
3. **Verify no node is left behind** before unfreezing. A single old node reintroduces blind writes and strips `baseline_version` on its next lifecycle write.
4. **Resume administrative traffic.**
5. **Only then perform the first rotation of any key.**
6. **If the KMS configuration was changed at any point**, confirm `cluster_config.consistent` is true in the `service-status` response, and re-issue the change — or restart the node — for every peer still reporting a different fingerprint.

### Do not do these during a mixed-version window

- **Rotate any key.** Unrecoverable for objects an old node must read.
- **Issue any KV2 lifecycle write to an old node.** Its blind write can clobber a concurrent check-and-set commit and drops `baseline_version`.
- **Create the same key ID from two nodes.** The create path is create-only on upgraded builds, but an old node's blind write does not honor that: the later writer's material wins and every DEK wrapped with the earlier material becomes permanently unwrappable.
- **Assume a disable or schedule-deletion took effect cluster-wide.** Old Transit nodes cache lifecycle state without expiry; confirm per node, or restart the old nodes.
- **Reconfigure the KMS backend and consider it done.** The reload broadcast is exactly what an old build rejects; check the response message and `cluster_config.consistent`.
- **Delete or prune version records** under `{prefix}/{key_id}/versions/*` for any reason. This is never safe, mixed-version or not; see [Retention and destruction preconditions](#retention-and-destruction-preconditions).

## Choosing between Vault KV2 and Vault Transit

Use **Vault Transit** (`VaultTransit`) when key material must be cryptographically isolated from anyone holding storage-level read access: Transit keeps key-encryption keys inside Vault, only ever returns ciphertext, and supports server-side key versioning and rotation. Use **Vault KV2** only when you accept that the Vault ACL on the key path *is* the confidentiality boundary and want the operational simplicity of a single KV mount.

## AWS KMS: deviations from the shared backend contract

Select it with `RUSTFS_KMS_BACKEND=aws`. Credentials and region resolution are delegated entirely to the standard `aws-config` provider chain (environment, shared profile, container/IMDS role), so RustFS never stores, persists, or redacts AWS credential material. Only two non-credential settings are read: `RUSTFS_KMS_AWS_REGION` and `RUSTFS_KMS_AWS_ENDPOINT_URL`. A plaintext (`http://`) endpoint override would expose every KMS request including plaintext data keys, so it is refused unless the development opt-in is set.

AWS owns key state, backing-key rotation, and the deletion window, and this backend mirrors none of it locally. Four behaviours therefore differ from every RustFS-managed backend:

| Behaviour | RustFS-managed backends | AWS KMS backend |
| --- | --- | --- |
| Decryption with a `Disabled` or `PendingDeletion` key | Kept working, so disabling a key never breaks reads of objects already encrypted under it | **Refused by AWS.** Objects encrypted under a key that is later disabled become unreadable until it is re-enabled |
| Key deletion | Physical deletion available | **No physical delete.** `ScheduleKeyDeletion` is the only removal path; AWS destroys the material when the 7-30 day window elapses. `force_immediate` is refused |
| Cancelling a scheduled deletion | Key returns to `Enabled` | Key is left **`Disabled`**; enable it explicitly |
| Creating a key under a caller-chosen name | The requested name becomes the key id | **Refused.** AWS assigns identifiers and this backend does not manage aliases |

Consequences of the last row: **SSE-S3 key auto-creation and the synthetic KMS probe are unavailable on this backend**, because both address a key by a name they choose. Pre-create keys in AWS and reference them by AWS key id or ARN.

The AWS backend is exempt from `backends::contract_tests::assert_state_machine_contract`, whose assumptions (disabled keys still decrypt, cancel returns to `Enabled`, caller-assigned names) AWS violates. The exemption is pinned by the offline `aws_backend_shared_contract_exemption_is_pinned` test in `crates/kms/src/backends/aws.rs`; if AWS changes any of these semantics, integrate the backend into the shared driver and remove the exemption rather than weakening the shared assertions.

Key versions are opaque: RustFS reports `key_version` as 1 and cannot enumerate versions. Rotation uses `RotateKeyOnDemand`, which retains prior backing keys for decryption; AWS's automatic yearly rotation is neither enabled nor reported on by RustFS.

The KMS admin API accepts the backend as `"backend_type": "AWS"` (aliases `aws`, `aws-kms`, `aws_kms`, `AwsKms`) on `/v3/kms/configure` and `/v3/kms/reconfigure`. The body carries `region` (**required**), and optionally `endpoint_url`, `default_key_id`, and the shared timeout/retry/cache settings; credential fields are rejected as unknown, because every node resolves credentials through its own provider chain. `region` is mandatory even though `RUSTFS_KMS_AWS_REGION` is optional at startup: the admin configuration is replayed on every node, and a request that left the region to each node's ambient chain would let nodes address different regions — and therefore different keys — under an identical configuration. `default_key_id` must be an AWS key id or ARN that already exists.

## Vault TLS: custom CA and mutual TLS

Both Vault backends (KV2 and Transit) support a private certificate authority and client-certificate (mTLS) authentication at the connection layer:

| Setting | Environment variable | Admin configure field | Meaning |
| --- | --- | --- | --- |
| CA bundle | `RUSTFS_KMS_VAULT_CA_CERT` | `ca_cert_path` | Path to a PEM CA bundle; when set, only this bundle is trusted |
| Client certificate | `RUSTFS_KMS_VAULT_CLIENT_CERT` | `client_cert_path` | Path to a PEM client certificate presented to Vault; requires the client key |
| Client key | `RUSTFS_KMS_VAULT_CLIENT_KEY` | `client_key_path` | Path to the PEM private key matching the client certificate |
| Skip verification | `RUSTFS_KMS_VAULT_SKIP_TLS_VERIFY` | `skip_tls_verify` | Disables server certificate verification; gated on the insecure development defaults opt-in |

Paths are read on the node applying the configuration, so the files must exist at the same path on every node. Certificate and key must be configured together; the files are read and parsed when the backend starts, so a bad path or malformed PEM fails the configuration rather than a later request. The `kms/status` backend summary reports `has_custom_ca` and `has_client_identity` booleans, never file contents. RustFS always sets the trust roots and identity explicitly — to the configured values or to empty — so the `VAULT_CACERT`, `VAULT_CAPATH`, `VAULT_CLIENT_CERT` and `VAULT_CLIENT_KEY` process environment variables cannot splice TLS material into the connection behind the KMS configuration.

## Local backend durability and deployment support matrix

The Local backend stores one JSON record per key (`<key_id>.key`) plus an Argon2id salt file (`.master-key.salt`) inside the configured `key_dir`. For where the key material lives and who can read it, see the [backend comparison](#backend-comparison).

### Positioning

- `Local` is the default backend (`kms_backend` defaults to `local`) and is a development, testing and demo backend; it is not supported for production. Activating a backend whose capabilities report `production_supported: false` logs a `kms_backend_positioning` warning on every start, restart and reconfigure, and the `kms/status` capability matrix carries the same flag. The positioning is a warning, not a gate.
- Configuration validation enforces stricter rules outside explicit development mode: a master key is required and `key_dir` must not live under the process temp directory.
- The RustFS Kubernetes operator places the key directory on a PersistentVolumeClaim, so keys survive pod rescheduling.
- Production multi-node deployments should use the Vault Transit backend.

### Deployment support matrix

| Deployment | Supported | Notes |
| --- | --- | --- |
| Local filesystem (ext4, XFS, APFS, ...) | Yes | The commit protocol relies on POSIX `rename`/`hard_link` atomicity and `fsync` durability |
| Kubernetes PVC | Yes | Only when the PersistentVolume is backed by a local or block filesystem; this is how the RustFS operator provisions the key directory |
| NFS or other shared/network filesystems | No | Network filesystems do not reliably provide the atomicity and fsync semantics the protocol depends on; an NFS-backed PersistentVolume is this case |
| Multiple RustFS processes sharing one `key_dir` | No | Concurrent key **creation** is linearized (`hard_link` refuses to clobber), but every other write is a read-modify-write with no cross-process lock, so concurrent writers can silently lose updates |

Within a single process, per-key write locks serialize read-modify-write updates.

### Crash recovery behavior

Every mutation of the key directory uses a durable commit protocol:

1. A temp file (`<name>.tmp-<uuid>`) is created exclusively in `key_dir`.
2. The content is written and fsynced (`sync_all`).
3. The file is published atomically: `rename` to replace an existing file, `hard_link` to create a new one without clobbering.
4. The parent directory is fsynced so the new directory entry is durable.

Deletion mirrors the tail of the protocol (`remove_file` followed by a parent directory fsync). A crash at any step leaves either the complete old state or the complete new state, plus at most an unpublished temp file. On startup the backend:

- **Removes orphaned commit temp files.** The matcher is strict (`<prefix>.tmp-<uuid>`, never anything ending in `.key`), so published key files are never touched.
- **Validates every published `.key` file.** A record that fails to decode fails startup rather than being silently skipped.
- **Guards the salt file.** If `.master-key.salt` is missing but the directory contains keys marked `encrypted-master-key`, initialization fails closed naming the salt path. A regenerated salt derives a different master key and can never decrypt those keys, so the recovery is to **restore the salt file (or the whole directory) from backup**, never to let a fresh salt be generated. The guard is equally strict about a record it cannot read or interpret (for example one written by a newer RustFS naming an at-rest protection this build does not implement): no replacement salt is generated for such a directory either. An empty directory, or a legacy directory predating the salt file, initializes normally.

### Filesystem permissions and the boundaries the protocol assumes

- The key directory is held at `0o700` and every file published into it — key records, the salt, restore staging — is written owner-only. The requested mode is applied and re-read on the open file before the content becomes durable, so the process umask cannot widen it.
- A directory wider than `0o700` is **narrowed on every start** (and re-read to confirm), not refused: kubelet creates `emptyDir` at `0o777`, several PVC provisioners `mkdir -m 0777`, and a `--tmpfs` mount lands at `1777`. Only a directory this process cannot secure is fatal. Narrowing is logged with the previous mode whenever it was reachable beyond the owner.
- Publishing never writes through a symlink: `hard_link` refuses any existing destination (including a dangling symlink) and `rename` replaces the link itself. Startup removes anything wearing a commit-temp name that is not a directory, symlinks included; the protocol only ever creates temps with `create_new`, so such an entry is either its own leftover or something planted.

Two boundaries are **not** verified, and deployments should not assume them:

- **Cross-device operations.** The temp file is always created in the destination's own directory, so `rename` and `hard_link` never cross a filesystem; that invariant is tested, a real cross-device attempt is not. The restore staging directory is always `.restore-staging` inside `key_dir` — do not bind-mount it onto another filesystem.
- **The key directory being replaced mid-commit.** Paths are re-resolved from the directory name rather than held as a directory descriptor. If `key_dir` is swapped between the `rename` and the parent `fsync`, the fsync lands on the replacement and the call still reports success. Reaching this requires write access to the key directory's **parent**, which nothing checks — keep the parent owner-writable too. Closing it properly means moving to `renameat`/`linkat` against a held descriptor; it is a recorded gap.

### Backing up the key directory

Back up `key_dir` as a whole, including the hidden `.master-key.salt` file. A key file on its own is not restorable: decrypting it requires the master key derived from the configured `master_key` **and** the persisted salt. Restoring a partial directory leaves the backend unable to decrypt, and the salt guard will (correctly) refuse to start. Losing the salt file with no backup means every key encrypted under it is unrecoverable. The rehearsal procedure is in the [KMS disaster-recovery drill](kms-disaster-recovery-drill.md).