mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-03 06:37:34 +08:00
922552083f
A 6 GiB object uploaded to the source as a 768-part multipart upload was replicated to a generic S3 target with a single PutObject, and the target rejected the body with EntityTooLarge. No CreateMultipartUpload was ever issued, so the multipart replication transport never ran for the object it exists for. `replication_put_object_options` seeded the transport from `object_info.is_multipart()` and then overwrote it with the second return value of `decrypt_checksums`. Those two booleans do not mean the same thing: the first is the object's storage shape, read from the ETag, while the second reports whether the stored *checksum record* carries per-part data. A full-object checksum -- what `aws s3 cp` writes by default for a CRC algorithm -- is serialized with no MULTIPART flag even on a multipart upload, so the record reports false and the object was routed as a single PUT. `decrypt_checksums` documents this in object_api/types.rs: callers that need routing must consult `is_multipart()`. Replication did the opposite. Route on the object's own shape, and let the checksum record only add multipart-ness, never take it away. Objects already stored with such a record are fixed too: the ETag was always right. This also repairs the diagnosis of rustfs#6825, where the single-PUT 5 GiB guard fired against an object that was multipart all along and told the operator to re-upload it as multipart. Tests cover the three shapes the router has to separate: a multipart object with a full-object checksum record (the regression, which fails without this change), a multipart object with a composite record, and a single-part object that must not be promoted onto multipart.
109 lines
5.2 KiB
Markdown
109 lines
5.2 KiB
Markdown
# Replication object size and shape limits (generic S3 targets)
|
|
|
|
**Use this when:** an object fails to replicate to an S3-compatible target with `EntityTooLarge`/`EntityTooSmall`, or you need to know whether a large or oddly-chunked object is replicable before relying on it.
|
|
**Source of truth:** `crates/ecstore/src/bucket/replication/` (transport selection and part replay), `crates/replication/` (target client), `crates/config/src/constants/` (`RUSTFS_OBS_LOGGER_LEVEL`).
|
|
|
|
What RustFS can and cannot replicate to a generic S3 target (AWS S3, Wasabi,
|
|
MinIO, or any other S3-compatible endpoint configured as a bucket replication
|
|
target), and how a rejected object shows up in the log.
|
|
|
|
## The route is chosen by the source object's shape, not its size
|
|
|
|
RustFS mirrors how the object was written on the source:
|
|
|
|
| Source object was written as | Replication transport |
|
|
| --- | --- |
|
|
| a single `PutObject` | a single `PutObject` on the target |
|
|
| a multipart upload | a multipart upload replaying **the source's own part layout** |
|
|
|
|
RustFS does not re-chunk on the replication side. A single-`PutObject` object is
|
|
never converted into a multipart upload for the target, and a multipart object's
|
|
parts are never merged or re-split. The target's part layout is the source's,
|
|
because heal and delete convergence address the replica by that identity.
|
|
|
|
This is why object size alone does not tell you whether an object is
|
|
replicable — how it was uploaded does.
|
|
|
|
The shape is read from the object's own ETag: a multipart ETag carries a
|
|
`-<part count>` suffix. Nothing else selects the transport — in particular the
|
|
checksum algorithm and checksum type (`COMPOSITE` or `FULL_OBJECT`) an object
|
|
was uploaded with have no bearing on it.
|
|
|
|
## Limits
|
|
|
|
### Single-`PutObject` objects: 5 GiB
|
|
|
|
S3 caps `PutObject` at **5 GiB**. This is an S3 API limit that every target
|
|
enforces, not a RustFS tunable.
|
|
|
|
An object larger than 5 GiB that was written to the source with a single
|
|
`PutObject` therefore **cannot be replicated to a generic S3 target**. RustFS
|
|
detects this before streaming the body and fails the object immediately, rather
|
|
than uploading gigabytes only to collect an `EntityTooLarge` from the remote.
|
|
|
|
**Remedy:** re-upload the object using multipart. Most S3 clients do this
|
|
automatically above a threshold (the AWS CLI defaults to 8 MiB); a client
|
|
configured with a very high multipart threshold, or one that streams a single
|
|
`PutObject`, is the usual way an object ends up on the wrong side of this limit.
|
|
|
|
### Multipart objects: the target's multipart limits, applied to the source's layout
|
|
|
|
Because the source's part layout is replayed verbatim, the target's own
|
|
multipart constraints apply to that layout:
|
|
|
|
| Constraint | Target rejects with |
|
|
| --- | --- |
|
|
| every part except the last must be ≥ 5 MiB | `EntityTooSmall` |
|
|
| no part may exceed 5 GiB | `EntityTooLarge` |
|
|
| at most 10,000 parts | failure at `CompleteMultipartUpload` |
|
|
|
|
A source object whose parts satisfy these is replicable up to the S3 multipart
|
|
maximum of 5 TiB.
|
|
|
|
## Reliability characteristics for large objects
|
|
|
|
Worth knowing before replicating multi-gigabyte objects:
|
|
|
|
- Parts are transferred **sequentially**.
|
|
- There is **no part-level retry**. A failure on any single part fails the whole
|
|
object; the target-side multipart upload is then aborted so no incomplete
|
|
upload is left behind.
|
|
- Retry happens at the object level (MRF replay / heal scanner), so a failure
|
|
late in a large transfer re-sends the object from the beginning.
|
|
|
|
For a 6 GiB object this means one long all-or-nothing transfer window. Part-level
|
|
retry and resumable transfer are tracked as a separate improvement.
|
|
|
|
## What a failed object looks like in the log
|
|
|
|
A replication attempt that ends in a terminal `FAILED` state emits one `error`
|
|
line per failed target. It is at `error` deliberately: the default log level
|
|
(`RUSTFS_OBS_LOGGER_LEVEL`, default `error`) must not hide an object that never
|
|
reached its target.
|
|
|
|
```
|
|
ERROR ... event=replication_object_failed bucket=photos object=backups/vm-image.qcow2
|
|
version_id=... arn=arn:replication::wasabi endpoint=s3.wasabisys.com
|
|
op_type=OBJECT size=6442450944 replication_status=FAILED
|
|
error="object of 6442450944 bytes was not written as multipart on the source and
|
|
exceeds the 5368709120 byte single-PutObject limit of an S3 target;
|
|
re-upload it with multipart to make it replicable"
|
|
Replication failed for object
|
|
```
|
|
|
|
The `error` field carries the target's own error code and message where the
|
|
target produced one, so a remote rejection is diagnosable without lowering the
|
|
log level and reproducing. It is passed through the same redaction as the
|
|
persisted resync detail, so an error echoing a credential or signed URL is
|
|
replaced with `[redacted sensitive resync error detail]`.
|
|
|
|
Raise `RUSTFS_OBS_LOGGER_LEVEL` to `warn` to additionally see the per-attempt
|
|
failure branches (target offline, HEAD failures, per-part errors) that sit
|
|
underneath this summary.
|
|
|
|
## Related
|
|
|
|
- [Replication target check](replication-check.md) — validate a target's
|
|
configuration, versioning, and version fidelity before relying on it.
|
|
- [Presigned size limits](presigned-size-limits.md) — per-request and per-upload caps a backend can put on presigned uploads.
|