Skip to content

storage 4.0: streams-only I/O, one-word upload verbs, copy replaces transfer#84

Merged
loks0n merged 4 commits into
mainfrom
feat/storage-streaming
Jul 23, 2026
Merged

storage 4.0: streams-only I/O, one-word upload verbs, copy replaces transfer#84
loks0n merged 4 commits into
mainfrom
feat/storage-streaming

Conversation

@loks0n

@loks0n loks0n commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements the agreed direction for CLO-4257: streaming is the only I/O path — no parallel string/stream API families. File contents move as PSR-7 streams end to end, so PHP memory stays bounded regardless of file size (the primary driver of the worker OOM incidents).

API (breaking — next release is storage/4.0.0)

  • read() returns StreamInterface. Getting bytes into memory is now explicit ((string) / getContents()) at the call site.
  • write() and uploads take StreamInterface. Strings wrap for free (new Stream($data)); file handles wrap without buffering (Stream::fromResource($handle)).
  • uploadData() + uploadChunk() collapse into one public upload() (whole file = single-chunk default); prepareUpload()/finalizeUpload() become prepare()/finalize(). Public chunked-upload surface is four one-word verbs: prepare / upload / finalize / abort. The per-chunk write is a protected hook, mirroring how the S3 wire verbs are already non-public.
  • transfer() is replaced by copy($source, $target, ?Device $to = null, $chunkSize) — one concrete base implementation (the two near-identical per-device copies are deleted), with the multipart abort-on-failure logic hoisted along with it. move() = copy() + delete() on every adapter; Local keeps its same-device rename() override, and a future S3 server-side CopyObject (CLO-4268) slots in as a copy() override with no caller changes.
  • TRANSFER_CHUNK_SIZECOPY_CHUNK_SIZE.

Transport

  • S3 downloads stream through StreamingClientInterface::stream() into a temporary stream (php://temp, spills to disk past 2 MB). A capped 16 KB head buffer keeps error XML parseable without buffering success bodies.
  • S3 uploads hash the stream for SigV4 (incremental MD5+SHA-256, rewind, send) — validated against the AWS SDK's approach (SignatureV4::getPayload requires seekable and hashes with position restore; UNSIGNED-PAYLOAD is not their default either). Seekable streams are required; the curl adapter already streams request bodies via CURLOPT_UPLOAD/CURLOPT_READFUNCTION.
  • Known tradeoff: streamed GETs don't retry after bytes reach the sink (the Retry decorator correctly refuses to duplicate delivered bytes).

Docs

README examples updated and an "Upgrading from 3.x" section added mapping every removed API to its replacement.

Test plan

  • Full suite green: 75 unit + 26 e2e against MinIO — e2e now exercises real streamed uploads (including multipart and out-of-order parts), ranged streamed reads, and cross-device copy over the wire.
  • PHPStan level max clean, Pint clean, Vale clean, bin/monorepo validate passes.

🤖 Generated with Claude Code

@greptile-apps

greptile-apps Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR moves storage operations to stream-only I/O and updates the public API for storage 4.0. The main changes are:

  • PSR-7 streams for reads, writes, and uploads.
  • Unified multipart upload verbs.
  • Shared copy and move behavior across adapters.
  • Streamed S3 downloads, signed uploads, and server-side copies.
  • Updated tests and migration documentation.

Confidence Score: 5/5

The latest fixes look safe to merge.

  • Zero-length S3 reads now return an empty stream without an invalid request.
  • Local writes and S3 download buffering now handle partial writes.
  • Seekable streams are rewound consistently before upload.

Important Files Changed

Filename Overview
packages/storage/src/Storage/Device.php Defines the stream-only API and shared copy and move behavior.
packages/storage/src/Storage/Device/Local.php Adds streamed reads and writes, seekable-stream rewinding, and checked partial-write handling.
packages/storage/src/Storage/Device/S3.php Adds streamed downloads, stream hashing, zero-length range handling, and server-side copy support.
packages/storage/src/Storage/Device/Telemetry.php Updates telemetry delegation for the renamed stream-based API.

Reviews (4): Last reviewed commit: "Address review: full-write loops, zero-l..." | Re-trigger Greptile

Comment thread packages/storage/src/Storage/Device/Local.php Outdated
Comment thread packages/storage/src/Storage/Device/S3.php
Comment thread packages/storage/src/Storage/Device/S3.php
Comment thread packages/storage/src/Storage/Device/S3.php
loks0n and others added 4 commits July 23, 2026 18:58
File contents now move as PSR-7 streams end to end, so memory stays
bounded regardless of file size — the primary driver of worker OOMs
under concurrent large-file traffic.

- read() returns StreamInterface; S3 reads stream into a temporary
  stream via the client's StreamingClientInterface sink
- write() and uploads take StreamInterface; S3 hashes the stream for
  SigV4 (hash pass, rewind, send — the AWS SDK approach), so seekable
  streams are required
- uploadData()/uploadChunk() merge into one public upload() verb;
  prepareUpload()/finalizeUpload() become prepare()/finalize(); the
  per-chunk hook is a protected abstract
- transfer() is replaced by copy(source, target, ?to, chunkSize) with
  the multipart abort-on-failure hoisted into the base class; move()
  is copy + delete everywhere, with Local keeping its rename override
- TRANSFER_CHUNK_SIZE renamed to COPY_CHUNK_SIZE
- README gains an "Upgrading from 3.x" section

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Give the S3 device optional bucket knowledge (subclasses pass theirs
automatically) and override copy(): a same-device copy with a known
bucket runs entirely server side — CopyObject up to 5 GB, chunked
UploadPartCopy beyond, with abort on failure — so no bytes move
through PHP. move() inherits the fast path as copy + delete.
Without a bucket, or across devices, copy() falls back to the
streamed base implementation.

Closes CLO-4268.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…onsumption

- Local writeFile and the S3 read sink loop until every byte of a chunk
  is written, so a partial fwrite can no longer truncate data silently
- S3 read(length: 0) returns an empty stream instead of building an
  invalid reversed range header
- Streams are consumed from the beginning on every adapter: Local now
  rewinds seekable streams like the transport does for S3, and the
  Device::write contract documents it (hashing from the caller's
  position would mismatch what cURL actually sends after its rewind)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@loks0n
loks0n force-pushed the feat/storage-streaming branch from 1f3181b to 09ba850 Compare July 23, 2026 18:00
@loks0n
loks0n merged commit c73877c into main Jul 23, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant