storage 4.0: streams-only I/O, one-word upload verbs, copy replaces transfer#84
Merged
Conversation
Contributor
Greptile SummaryThis PR moves storage operations to stream-only I/O and updates the public API for storage 4.0. The main changes are:
Confidence Score: 5/5The latest fixes look safe to merge.
Important Files Changed
Reviews (4): Last reviewed commit: "Address review: full-write loops, zero-l..." | Re-trigger Greptile |
File contents now move as PSR-7 streams end to end, so memory stays bounded regardless of file size — the primary driver of worker OOMs under concurrent large-file traffic. - read() returns StreamInterface; S3 reads stream into a temporary stream via the client's StreamingClientInterface sink - write() and uploads take StreamInterface; S3 hashes the stream for SigV4 (hash pass, rewind, send — the AWS SDK approach), so seekable streams are required - uploadData()/uploadChunk() merge into one public upload() verb; prepareUpload()/finalizeUpload() become prepare()/finalize(); the per-chunk hook is a protected abstract - transfer() is replaced by copy(source, target, ?to, chunkSize) with the multipart abort-on-failure hoisted into the base class; move() is copy + delete everywhere, with Local keeping its rename override - TRANSFER_CHUNK_SIZE renamed to COPY_CHUNK_SIZE - README gains an "Upgrading from 3.x" section Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Give the S3 device optional bucket knowledge (subclasses pass theirs automatically) and override copy(): a same-device copy with a known bucket runs entirely server side — CopyObject up to 5 GB, chunked UploadPartCopy beyond, with abort on failure — so no bytes move through PHP. move() inherits the fast path as copy + delete. Without a bucket, or across devices, copy() falls back to the streamed base implementation. Closes CLO-4268. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…onsumption - Local writeFile and the S3 read sink loop until every byte of a chunk is written, so a partial fwrite can no longer truncate data silently - S3 read(length: 0) returns an empty stream instead of building an invalid reversed range header - Streams are consumed from the beginning on every adapter: Local now rewinds seekable streams like the transport does for S3, and the Device::write contract documents it (hashing from the caller's position would mismatch what cURL actually sends after its rewind) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
loks0n
force-pushed
the
feat/storage-streaming
branch
from
July 23, 2026 18:00
1f3181b to
09ba850
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the agreed direction for CLO-4257: streaming is the only I/O path — no parallel string/stream API families. File contents move as PSR-7 streams end to end, so PHP memory stays bounded regardless of file size (the primary driver of the worker OOM incidents).
API (breaking — next release is
storage/4.0.0)read()returnsStreamInterface. Getting bytes into memory is now explicit ((string)/getContents()) at the call site.write()and uploads takeStreamInterface. Strings wrap for free (new Stream($data)); file handles wrap without buffering (Stream::fromResource($handle)).uploadData()+uploadChunk()collapse into one publicupload()(whole file = single-chunk default);prepareUpload()/finalizeUpload()becomeprepare()/finalize(). Public chunked-upload surface is four one-word verbs:prepare/upload/finalize/abort. The per-chunk write is aprotectedhook, mirroring how the S3 wire verbs are already non-public.transfer()is replaced bycopy($source, $target, ?Device $to = null, $chunkSize)— one concrete base implementation (the two near-identical per-device copies are deleted), with the multipart abort-on-failure logic hoisted along with it.move()=copy()+delete()on every adapter;Localkeeps its same-devicerename()override, and a future S3 server-sideCopyObject(CLO-4268) slots in as acopy()override with no caller changes.TRANSFER_CHUNK_SIZE→COPY_CHUNK_SIZE.Transport
StreamingClientInterface::stream()into a temporary stream (php://temp, spills to disk past 2 MB). A capped 16 KB head buffer keeps error XML parseable without buffering success bodies.SignatureV4::getPayloadrequires seekable and hashes with position restore;UNSIGNED-PAYLOADis not their default either). Seekable streams are required; the curl adapter already streams request bodies viaCURLOPT_UPLOAD/CURLOPT_READFUNCTION.Retrydecorator correctly refuses to duplicate delivered bytes).Docs
README examples updated and an "Upgrading from 3.x" section added mapping every removed API to its replacement.
Test plan
copyover the wire.bin/monorepo validatepasses.🤖 Generated with Claude Code