Skip to content

Add S3-style multi-part upload mechanism for large file parallel upload #1801

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mishushakov wants to merge 62 commits into from
Closed

Add S3-style multi-part upload mechanism for large file parallel upload #1801

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
43605ed
feat(envd): add multipart file upload API
mishushakov Jan 28, 2026
cd4dbee
perf(envd): skip fsync and async cleanup in multipart upload
mishushakov Jan 28, 2026
e94a0e9
fix(envd): address multipart upload code review issues
mishushakov Jan 28, 2026
38a70d6
chore: auto-commit generated changes
github-actions[bot] Jan 28, 2026
eb1dada
fix(envd): address additional multipart upload security and robustnes…
mishushakov Jan 28, 2026
1f97459
refactor(envd): remove unused ETag computation from multipart upload
mishushakov Jan 28, 2026
c028cb0
perf(envd): rewrite multipart upload with direct file writes
mishushakov Jan 28, 2026
e3e771d
chore: auto-commit generated changes
github-actions[bot] Jan 28, 2026
c382cf9
style(envd): fix linter issues in multipart upload code
mishushakov Jan 28, 2026
8bf6c2d
fix(envd): use valid UUID in non-existent session test
mishushakov Jan 28, 2026
f972ced
fix(envd): use exact matching for /files auth bypass
mishushakov Jan 28, 2026
13f9684
style(envd): use slices.Contains for exact path matching
mishushakov Jan 28, 2026
1df46a3
security(envd): add filepath.Clean to sanitize user-provided paths
mishushakov Jan 28, 2026
3da52cd
security(envd): add sanitizePath function with explicit validation
mishushakov Jan 28, 2026
cb264fc
revert(envd): undo path.go changes
mishushakov Jan 28, 2026
a549c14
Merge branch 'main' into mishushakov/multipart-upload
mishushakov Jan 28, 2026
764436a
perf(envd): reduce mutex contention in multipart upload
mishushakov Jan 28, 2026
212e881
feat(envd): support gzip Content-Encoding on file upload APIs
mishushakov Jan 29, 2026
5b155b0
revert(envd): remove gzip encoding from simple file upload
mishushakov Jan 29, 2026
a934990
perf(envd): check session limit before file allocation
mishushakov Jan 30, 2026
c7b1e30
revert(envd): remove gzip Content-Encoding from multipart upload
mishushakov Jan 30, 2026
2a91f48
refactor(envd): remove redundant session limit check
mishushakov Jan 30, 2026
35d1347
Merge branch 'main' into mishushakov/multipart-upload
mishushakov Jan 30, 2026
8183127
fix(envd): enforce exact part size and reject empty file uploads
mishushakov Jan 30, 2026
1be7c61
Merge branch 'main' into mishushakov/multipart-upload
mishushakov Feb 24, 2026
761bf2d
chore: auto-commit generated changes
github-actions[bot] Feb 24, 2026
816fb6e
fix(envd): harden multipart upload against race conditions, auth bypa…
mishushakov Feb 24, 2026
8c8a28f
refactor(envd): clean up multipart upload code
mishushakov Feb 24, 2026
fa2cc97
spec(envd): add minimum: 0 constraint to part query parameter
mishushakov Feb 24, 2026
de2d2b3
refactor(envd): drop handler-level negative part number check
mishushakov Feb 24, 2026
a30b15b
refactor(envd): drop handler-level totalSize/partSize range checks
mishushakov Feb 24, 2026
d249ab7
docs(envd): clarify that validateSigning already covers access token …
mishushakov Feb 24, 2026
ac5210c
refactor(envd): replace validateSigning with validateAccessToken on u…
mishushakov Feb 24, 2026
2198c96
chore: auto-commit generated changes
github-actions[bot] Feb 24, 2026
e8fb58b
fix(envd): remove upload paths from auth bypass list
mishushakov Feb 24, 2026
d40de31
refactor(envd): reset auth.go and rely on WithAuthorization middleware
mishushakov Feb 24, 2026
f8d86ad
fix(envd): improve multipart upload error messages with contextual de…
mishushakov Feb 24, 2026
07626ae
fix(envd): harden multipart upload against DoS and edge cases
mishushakov Feb 24, 2026
167fbcf
fix(envd): pass context parameter to newTestAPI to satisfy contextche…
mishushakov Feb 24, 2026
8a33941
fix(envd): add negative totalSize guard, concurrent part protection, …
mishushakov Feb 24, 2026
110b6ab
chore: auto-commit generated changes
github-actions[bot] Feb 24, 2026
40d37e0
fix(envd): simplify missing parts error to just show count
mishushakov Feb 24, 2026
a6f257f
refactor(envd): clean up multipart upload PR
mishushakov Feb 24, 2026
31c37cd
fix(envd): fix race conditions and validation in multipart upload
mishushakov Feb 24, 2026
0ee7005
refactor(envd): use int instead of uint for part numbers and counts
mishushakov Feb 24, 2026
ba06568
refactor(envd): remove unnecessary partNumber alias
mishushakov Feb 24, 2026
6344ba8
fix(envd): clean up remaining issues in multipart upload
mishushakov Feb 24, 2026
3ebee33
refactor(envd): unexport package-internal multipart upload types
mishushakov Feb 24, 2026
4cdcb0f
fix(envd): enforce single active session per path and fix completed/w…
mishushakov Feb 24, 2026
7f6b5cf
refactor(envd): remove TTL cleanup and reject completed-part re-uploads
mishushakov Feb 24, 2026
52a9b7a
fix(envd): add missing response codes to OpenAPI spec
mishushakov Feb 24, 2026
47c8737
fix(envd): write to temp file during multipart upload to preserve ori…
mishushakov Feb 24, 2026
39cbe4d
fix(envd): move filesystem I/O out of uploadsLock in delete handler
mishushakov Feb 24, 2026
9948c93
fix(envd): move filesystem I/O out of uploadsLock in init handler
mishushakov Feb 24, 2026
64491e5
fix(envd): defer map deletion until after finalization in complete/de…
mishushakov Feb 24, 2026
e30d702
fix(envd): treat short part reads as 400 instead of 500
mishushakov Feb 24, 2026
d0c6ecb
fix(envd): address multipart upload resource leak and correctness issues
mishushakov Feb 24, 2026
4f1d02e
fix(envd): close race conditions in multipart upload init and complet…
mishushakov Feb 24, 2026
5d77cde
fix(envd): move EnsureDirs after authoritative session-limit check
mishushakov Feb 25, 2026
1005e97
fix(envd): return 200 for part writes that succeed during concurrent …
mishushakov Feb 26, 2026
ebc550c
fix(envd): prevent int overflow in numParts on 32-bit systems
mishushakov Feb 26, 2026
5c9d823
fix(envd): limit part upload body size with MaxBytesReader
mishushakov Feb 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
252 changes: 252 additions & 0 deletions packages/envd/internal/api/api.gen.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading