bug:502-/data/multipart/complete

[bwalsh]

Bug report: Complete multipart upload fails after 5GB transfer

--
 

Summary

When a user initiates a 5GB file transfer, data-client uploads all parts using a MinChunkSize of 5MB (1024 parts). The multipart upload fails at completion: the Fence endpoint returns NoSuchUpload, and MinIO logs a s3.CompleteMultipart 404, client receives a 502 and retries (getting same result each time)
 

Steps to reproduce

1. Start a 5GB file transfer in data-client.
2. Ensure data-client uses MinChunkSize=5MB and uploads 1024 parts.
3. Observe that all parts are sent successfully.
4. Trigger the multipart completion request.
    

Expected behavior

The multipart completion succeeds and the 5GB object is finalized.
 

Actual behavior

The complete multipart request fails:

• Fence logs a NoSuchUpload error for CompleteMultipartUpload.
• MinIO reports s3.CompleteMultipart 404. See https://aced-console-minio1.ohsu.edu/tools/trace
   

Logs / error messages

Fence

WARNING:retry.api:An error occurred (NoSuchUpload) when calling the CompleteMultipartUpload operation: The specif
10.244.0.71 - - [19/Jan/2026:04:59:06 +0000] "POST /data/multipart/complete HTTP/1.0" 499 0 "-" "Go-http-client/1

@lbeckman314 @matthewpeterkort Can we check minio server logs?

Environment

• data-client
• MinIO (S3-compatible backend)

Workaround

Change data-client to use a MinChunkSize of 10MB (512 parts). Multipart works.

hypothesis

Does fence send entire payload sent to minio? 1024 parts is a moderately large message

[bwalsh]

When to use single-part vs multipart upload in Amazon S3

TL;DR (practical defaults)

• < 100 MB → Single-part

• 100 MB – 500 MB → Either (lean single-part unless you need resume/retry)

• ≥ 500 MB → Multipart

• > 5 GB → Multipart is required

---

Single-part upload

What it is: One HTTP PUT sends the whole object.

Use when

• Files are small (tens of MBs).

• You want simplicity and low overhead.

• Network is stable (few retries expected).

Pros

• Minimal API calls (one request).

• Lowest client/server overhead.

• Simple error handling.

Cons

• If it fails, you restart from zero.

• No parallelism → slower for large files.

• Not allowed above 5 GB.

---

Multipart upload

What it is: Split the object into parts (≥ 5 MB each), upload parts independently, then CompleteMultipartUpload.

Use when

• Files are large (hundreds of MBs to TBs).

• You want parallel uploads for speed.

• You need resumability on flaky networks.

• You expect partial retries instead of full restarts.

Pros

• Parallelism → higher throughput.

• Retry only failed parts.

• Required for objects > 5 GB.

Cons

• More API calls (init + N parts + complete).

• Overhead can dominate for small files.

• Must abort failed uploads to avoid orphaned parts (which can incur storage costs).

---

Performance & cost nuances (often overlooked)

• API call overhead: Multipart adds extra requests; for small files this can slow things down and increase request charges.

• Rate limits / gateways: Many tiny multipart uploads can amplify load (init/complete calls) and trigger 429/5xx from proxies or S3-compatible backends.

• Part size matters: Too small → too many parts; too large → less parallelism.

  • Common sweet spots: 8–64 MB per part.

• Backend compatibility: Some S3-compatible systems (e.g., Ceph RGW) have higher per-request overhead; multipart for small objects can be counterproductive.

---

Recommended decision table

File size | Recommended | Why -- | -- | -- < 10 MB | Single-part | Lowest overhead 10–100 MB | Single-part | Multipart overhead not worth it 100–500 MB | Depends | Use multipart if you need resume/retry 500 MB–5 GB | Multipart | Faster + safer > 5 GB | Multipart (required) | S3 limit

---

Heuristic you can encode in clients

if size > 5GB:
  multipart
elif size >= 500MB:
  multipart
elif size >= 100MB and (need_resume or unstable_network):
  multipart
else:
  single_part

[bwalsh]

Optimal chunk size

https://share.google/aimode/kUxbjYNTiA7Qy4rCf

[bwalsh]

Recommended decision table

File size	Recommended	Why
< 10 MB	Single-part	Lowest overhead
10–100 MB	Single-part	Multipart overhead not worth it
100–500 MB	Depends	Use multipart if you need resume/retry
500 MB–5 GB	Multipart	Faster + safer
> 5 GB	Multipart (required)	S3 limit