diff --git a/.claude/skills/package-design/SKILL.md b/.claude/skills/package-design/SKILL.md new file mode 100644 index 000000000..e8d96ac3f --- /dev/null +++ b/.claude/skills/package-design/SKILL.md @@ -0,0 +1,49 @@ +--- +name: package-design +description: Design principles and a working checklist for creating or modernizing a utopia-php package (major version redesigns, new libraries, API reviews). Distilled from the storage 3.0 rewrite. +--- + +# Package design + +How to shape a utopia-php package. Apply when designing a new package, preparing a major version, or reviewing an API. The goal is always the same: a small, honest, immutable surface — aim for a net-negative line count on redesigns. + +## State and construction + +- All configuration enters through the constructor as `readonly` promoted properties. No `setX()` methods, no static mutable state, no global registries. If a value must vary per operation, make it a method argument (`transfer(..., int $chunkSize)`), not instance state. +- Guarantee coroutine safety: an instance must hold zero per-request state. Build request-scoped data (headers, buffers) as locals and pass them through; two concurrent calls on one instance must never race. Grep for instance properties written outside the constructor — each one is a suspect. +- Mark secrets `#[\SensitiveParameter]`. Optional dependencies default in the signature (`?ClientInterface $client = null`) and resolve in the constructor body. +- Prefer removals that fail loudly. Deleting a method breaks callers at call time; changing the *meaning* of a same-shaped signature (e.g. a `string` parameter switching from file path to raw data) corrupts silently. When semantics must change, change the method name too, or delete the old name. + +## Surface area + +- An abstract base class may only demand what every implementation can honestly provide. A method one adapter stubs with `-1`/no-op belongs on the concrete class, not the contract. Same test for return shapes: if two adapters return incompatible structures from the "same" method, it is not one abstraction — drop it from the base. +- One flow per concern. Two parallel method families differing only in input form (file path vs data) should collapse to one primitive; hoist the shared composition (`prepare → chunk → finalize`) into the base as a concrete method. +- Cross-cutting behavior composes via decorators and injected dependencies, never baked into adapters: telemetry is a decorator, retries are a client decorator with a domain `Strategy`, transport is an injected PSR-18 client. Never ship two mechanisms for the same concern — pick the composable one and delete the other. +- Enums over string constants for closed, library-controlled sets (device types, ACLs). Keep plain strings/constants for open sets that outpace releases (provider regions) or accept custom values. +- Delete metadata that code doesn't consume: human-prose `getName()`/`getDescription()` accessors, dead parameters, unused constants. Reuse platform types (`Utopia\Psr7\Method`) instead of redeclaring them. +- Internal wire formats deserve small `final readonly` value objects with honest narrowing, not `stdClass` juggling. + +## Dependencies + +- Audit `composer.json` against actual usage (`grep` each dependency and each `ext-*`) — absorbed packages carry fossils. +- Depend on PSR interfaces at the seam; default to the utopia implementation (`utopia-php/client`, `utopia-php/psr7`) in the constructor body. Match transport defaults to the old behavior deliberately (e.g. no request timeout for unbounded uploads) — a client's own defaults are tuned for RPC, not file transfer. +- Sibling dependencies use Packagist constraints, never path repositories. Run `bin/monorepo validate` and regenerate the root dependency graph (`bin/monorepo graph`) after touching a manifest. + +## Static analysis and guards + +- New or redesigned packages pin `phpstan.neon` at `level: max` (include the root baseline) and extend the root `rector.php` with the stable prepared sets (`typeDeclarationDocblocks`, `privatization`, `instanceOf`, `rectorPreset`). Skip `naming`/`codingStyle` — they fight pint. +- Fix analysis errors by improving the design (typed response objects, shaped `@phpstan-type` aliases for by-ref arrays, runtime narrowing), never with `@phpstan-ignore` or silencing casts. +- A runtime guard beats a docblock type: if invalid input can corrupt data (`$chunkSize <= 0`), throw at runtime and drop the `positive-int` annotation — PHPStan flags guard-plus-annotation as dead code, and the guard is the contract callers actually get. The throw doubles as type narrowing downstream. +- Rector caches aggressively: after config changes, run `vendor/bin/rector process --clear-cache` locally before trusting `bin/monorepo check` — CI runs cold and will find what your cache hides. + +## Tests + +- Keep the tier contract: `composer test` = unit on bare host; `composer test:e2e` = against the package's compose services. Unit-test transport seams with scripted stubs that implement the *full* adapter interface, so decorators (retry, telemetry) are exercised for real rather than simulated. +- When a decorator or strategy reads a stream, verify re-readability in a test — a consumed body is a silent correctness bug. +- PHPUnit 12 assertions (`assertIsArray`, `fail(): never`) narrow types natively under PHPStan; prefer them over hand-rolled `if (!is_array(...))` blocks in tests. + +## Releasing a major + +- Releases are tags shaped `/`; check existing tags before naming the version — branch names lie. +- README carries an "Upgrading from N.x" section mapping every removed API to its replacement, one bullet per break. Docs are Vale-linted (`vale README.md docs`): sentence-case headings, no weasel words, backtick code identifiers. +- Verify against live services before shipping protocol changes (the e2e MinIO suite caught nothing precisely because the SigV4 rewrite signed exactly what it sent — that is the standard: sign/send/store exactly one representation). diff --git a/.vale/styles/config/vocabularies/Utopia/accept.txt b/.vale/styles/config/vocabularies/Utopia/accept.txt index c15fcff27..8c7a27545 100644 --- a/.vale/styles/config/vocabularies/Utopia/accept.txt +++ b/.vale/styles/config/vocabularies/Utopia/accept.txt @@ -145,6 +145,10 @@ backoff backpressure allowlist accessor(?:s)? +ACLs? +composable +docblocks? +redeclaring lookups loopback lockdown diff --git a/README.md b/README.md index f01b4971c..1982f838e 100644 --- a/README.md +++ b/README.md @@ -82,9 +82,10 @@ graph TD queue --> validators servers --> di servers --> validators - storage --> system storage --> telemetry storage --> validators + storage --> client + storage --> psr7 auth circuit-breaker nats diff --git a/packages/storage/README.md b/packages/storage/README.md index 2a0157f42..cc749fbc0 100644 --- a/packages/storage/README.md +++ b/packages/storage/README.md @@ -3,7 +3,6 @@ > [!IMPORTANT] > This repository is a read-only mirror of the [utopia-php monorepo](https://github.com/utopia-php/monorepo). Development happens in [`packages/storage`](https://github.com/utopia-php/monorepo/tree/main/packages/storage) — please open issues and pull requests there. -[![Build Status](https://travis-ci.org/utopia-php/storage.svg?branch=master)](https://travis-ci.com/utopia-php/storage) ![Total Downloads](https://img.shields.io/packagist/dt/utopia-php/storage.svg) [![Discord](https://img.shields.io/discord/564160730845151244?label=discord)](https://appwrite.io/discord) @@ -20,27 +19,19 @@ composer require utopia-php/storage ### Basic usage +Devices are immutable value objects: construct one with its configuration and use it anywhere, including across coroutines. + ```php upload('/local/path/to/file.png', 'destination/path/file.png'); +$device->uploadData(file_get_contents('/local/path/to/file.png'), 'destination/path/file.png', 'image/png'); // Check if file exists $exists = $device->exists('destination/path/file.png'); @@ -59,11 +50,9 @@ $device->delete('destination/path/file.png'); Use the local filesystem for storing files. ```php -use Utopia\Storage\Storage; use Utopia\Storage\Device\Local; -// Initialize local storage -Storage::setDevice('files', new Local('/path/to/storage')); +$device = new Local('/path/to/storage'); ``` ### AWS S3 @@ -71,27 +60,37 @@ Storage::setDevice('files', new Local('/path/to/storage')); Store files in Amazon S3 or compatible services. ```php -use Utopia\Storage\Storage; +use Utopia\Storage\Acl; use Utopia\Storage\Device\S3; -// Initialize S3 storage -Storage::setDevice('files', new S3( +$device = new S3( 'root', // Root path in bucket 'YOUR_ACCESS_KEY', 'YOUR_SECRET_KEY', - 'YOUR_BUCKET_NAME', - S3::US_EAST_1, // Region (default: us-east-1) - S3::ACL_PRIVATE // Access control (default: private) -)); + 'YOUR_BUCKET_NAME.s3.us-east-1.amazonaws.com', // Host + 'us-east-1', // Region + Acl::Private // Access control (default: private) +); +``` -// Available regions -// S3::US_EAST_1, S3::US_EAST_2, S3::US_WEST_1, S3::US_WEST_2, S3::AP_SOUTH_1, -// S3::AP_NORTHEAST_1, S3::AP_NORTHEAST_2, S3::AP_NORTHEAST_3, S3::AP_SOUTHEAST_1, -// S3::AP_SOUTHEAST_2, S3::EU_CENTRAL_1, S3::EU_WEST_1, S3::EU_WEST_2, S3::EU_WEST_3, -// And more - check the S3 class for all available regions +The provider-specific adapters below build the host for you from a bucket and region. Every S3-family adapter also accepts optional named constructor arguments: + +```php +use Utopia\Storage\Acl; +use Utopia\Storage\Device\AWS; + +$device = new AWS( + 'root', + 'YOUR_ACCESS_KEY', + 'YOUR_SECRET_KEY', + 'YOUR_BUCKET_NAME', + AWS::US_EAST_1, + Acl::Private, + client: $psrClient, // any PSR-18 client (default: utopia-php/client with retries, see below) +); // Available ACL options -// S3::ACL_PRIVATE, S3::ACL_PUBLIC_READ, S3::ACL_PUBLIC_READ_WRITE, S3::ACL_AUTHENTICATED_READ +// Acl::Private, Acl::PublicRead, Acl::PublicReadWrite, Acl::AuthenticatedRead ``` ### DigitalOcean Spaces @@ -99,18 +98,17 @@ Storage::setDevice('files', new S3( Store files in DigitalOcean Spaces. ```php -use Utopia\Storage\Storage; +use Utopia\Storage\Acl; use Utopia\Storage\Device\DOSpaces; -// Initialize DO Spaces storage -Storage::setDevice('files', new DOSpaces( +$device = new DOSpaces( 'root', // Root path in bucket 'YOUR_ACCESS_KEY', 'YOUR_SECRET_KEY', 'YOUR_BUCKET_NAME', DOSpaces::NYC3, // Region (default: nyc3) - DOSpaces::ACL_PRIVATE // Access control (default: private) -)); + Acl::Private // Access control (default: private) +); // Available regions // DOSpaces::NYC3, DOSpaces::SGP1, DOSpaces::FRA1, DOSpaces::SFO2, DOSpaces::SFO3, DOSpaces::AMS3 @@ -121,21 +119,20 @@ Storage::setDevice('files', new DOSpaces( Store files in Backblaze B2 Cloud Storage. ```php -use Utopia\Storage\Storage; +use Utopia\Storage\Acl; use Utopia\Storage\Device\Backblaze; -// Initialize Backblaze storage -Storage::setDevice('files', new Backblaze( +$device = new Backblaze( 'root', // Root path in bucket 'YOUR_ACCESS_KEY', 'YOUR_SECRET_KEY', 'YOUR_BUCKET_NAME', Backblaze::US_WEST_004, // Region (default: us-west-004) - Backblaze::ACL_PRIVATE // Access control (default: private) -)); + Acl::Private // Access control (default: private) +); // Available regions (clusters) -// Backblaze::US_WEST_000, Backblaze::US_WEST_001, Backblaze::US_WEST_002, +// Backblaze::US_WEST_000, Backblaze::US_WEST_001, Backblaze::US_WEST_002, // Backblaze::US_WEST_004, Backblaze::EU_CENTRAL_003 ``` @@ -144,18 +141,17 @@ Storage::setDevice('files', new Backblaze( Store files in Linode Object Storage. ```php -use Utopia\Storage\Storage; +use Utopia\Storage\Acl; use Utopia\Storage\Device\Linode; -// Initialize Linode storage -Storage::setDevice('files', new Linode( +$device = new Linode( 'root', // Root path in bucket 'YOUR_ACCESS_KEY', 'YOUR_SECRET_KEY', 'YOUR_BUCKET_NAME', Linode::EU_CENTRAL_1, // Region (default: eu-central-1) - Linode::ACL_PRIVATE // Access control (default: private) -)); + Acl::Private // Access control (default: private) +); // Available regions // Linode::EU_CENTRAL_1, Linode::US_SOUTHEAST_1, Linode::US_EAST_1, Linode::AP_SOUTH_1 @@ -166,18 +162,17 @@ Storage::setDevice('files', new Linode( Store files in Wasabi Cloud Storage. ```php -use Utopia\Storage\Storage; +use Utopia\Storage\Acl; use Utopia\Storage\Device\Wasabi; -// Initialize Wasabi storage -Storage::setDevice('files', new Wasabi( +$device = new Wasabi( 'root', // Root path in bucket 'YOUR_ACCESS_KEY', 'YOUR_SECRET_KEY', 'YOUR_BUCKET_NAME', Wasabi::EU_CENTRAL_1, // Region (default: eu-central-1) - Wasabi::ACL_PRIVATE // Access control (default: private) -)); + Acl::Private // Access control (default: private) +); // Available regions // Wasabi::US_EAST_1, Wasabi::US_EAST_2, Wasabi::US_WEST_1, Wasabi::US_CENTRAL_1, @@ -190,11 +185,8 @@ Storage::setDevice('files', new Wasabi( All storage adapters provide a consistent API for working with files: ```php -// Get storage device -$device = Storage::getDevice('files'); - -// Upload a file -$device->upload('/path/to/local/file.jpg', 'remote/path/file.jpg'); +// Upload file contents +$device->uploadData(file_get_contents('/path/to/local/file.jpg'), 'remote/path/file.jpg', 'image/jpeg'); // Check if file exists $exists = $device->exists('remote/path/file.jpg'); @@ -215,34 +207,121 @@ $contents = $device->read('remote/path/file.jpg'); $chunk = $device->read('remote/path/file.jpg', 0, 1024); // Read first 1KB // Multipart/chunked uploads -$device->upload('/local/file.mp4', 'remote/video.mp4', 1, 3); // Part 1 of 3 - -// Create directory -$device->createDirectory('remote/new-directory'); - -// List files in directory -$files = $device->listFiles('remote/directory'); +$metadata = []; +$device->uploadData($firstChunk, 'remote/video.mp4', 'video/mp4', 1, 3, $metadata); // Part 1 of 3 + +// Resumable uploads: prepare, upload chunks in any order, finalize +$device->prepareUpload('remote/video.mp4', 'video/mp4', 3, $metadata); +$device->uploadChunk($secondChunk, 'remote/video.mp4', 2, 3, $metadata); +$device->finalizeUpload('remote/video.mp4', 3, $metadata); + +// List files under a prefix, one page at a time +$list = $device->listFiles('remote/directory', 100); +foreach ($list->files as $file) { + echo $file->path . ' (' . $file->size . " bytes)\n"; +} +if ($list->cursor !== null) { + $list = $device->listFiles('remote/directory', 100, $list->cursor); // Next page +} // Delete file $device->delete('remote/path/file.jpg'); // Delete directory -$device->deleteDirectory('remote/directory'); +$device->deletePath('remote/directory'); // Transfer files between storage devices -$sourceDevice = Storage::getDevice('source'); -$targetDevice = Storage::getDevice('target'); - $sourceDevice->transfer('source/path.jpg', 'target/path.jpg', $targetDevice); + +// Transfer with a custom chunk size (default: 20 MB) +$sourceDevice->transfer('source/path.jpg', 'target/path.jpg', $targetDevice, 10000000); +``` + +## Custom HTTP client + +The S3-family adapters send requests through any [PSR-18](https://www.php-fig.org/psr/psr-18/) client. By default they use [utopia-php/client](https://github.com/utopia-php/client) with the cURL adapter, no request timeout, and the `Retry` decorator configured with `S3\RetryStrategy` — it retries transient S3 rate-limiting errors (`SlowDown`, `ServiceUnavailable`, `Throttling`, `RequestThrottled`, and plain 429/503 responses) three times with a 500 ms delay. + +Inject your own client to change the transport or the retry policy — for example the Swoole coroutine adapter with more aggressive retries: + +```php +use Utopia\Client; +use Utopia\Client\Adapter\SwooleCoroutine\Client as SwooleAdapter; +use Utopia\Client\Decorator\Retry; +use Utopia\Storage\Device\S3; +use Utopia\Storage\Device\S3\RetryStrategy; + +$client = new Retry( + new Client(new SwooleAdapter())->withTimeout(60), + new RetryStrategy(retries: 5, delay: 1.0), +); + +$device = new S3('root', 'ACCESS_KEY', 'SECRET_KEY', 'HOST', 'us-east-1', client: $client); +``` + +Omit the `Retry` decorator to disable retries entirely, or pass any `Utopia\Client\Decorator\Retry\Strategy` of your own. + +## Error handling + +Every runtime failure throws a subclass of `Utopia\Storage\Exception\StorageException`, so one catch covers any storage problem. Match more precisely when you need to branch: + +```php +use Utopia\Storage\Exception\NotFoundException; +use Utopia\Storage\Exception\RemoteException; +use Utopia\Storage\Exception\StorageException; +use Utopia\Storage\Exception\TransportException; +use Utopia\Storage\Exception\UploadException; + +try { + $contents = $device->read('remote/path/file.jpg'); +} catch (NotFoundException) { + // The file does not exist +} catch (TransportException $e) { + // The request never reached the service; $e->getPrevious() is the PSR-18 exception +} catch (RemoteException $e) { + // The service answered with an error: $e->getCode() is the HTTP status, + // $e->errorCode the service's own error identifier (for example `SlowDown`) +} catch (UploadException) { + // A chunked upload is in an invalid state (missing chunk, never prepared) +} catch (StorageException $e) { + // Anything else, such as a local filesystem failure +} +``` + +Invalid arguments (a non-positive chunk size, a page size above the adapter limit) throw SPL `\InvalidArgumentException` — these are programmer errors, not storage failures. + +## Telemetry + +Wrap any device with the `Telemetry` decorator to record a `storage.operation` histogram for every call through a [utopia-php/telemetry](https://github.com/utopia-php/telemetry) adapter: + +```php +use Utopia\Storage\Device\Local; +use Utopia\Storage\Device\Telemetry; + +$device = new Telemetry($telemetryAdapter, new Local('/path/to/storage')); ``` +## Upgrading from 2.x + +Version 3.0 makes every device immutable and safe to share across coroutines, and removes all global state: + +- The `Storage` class is gone entirely: replace `Storage::setDevice('files', $device)` and `Storage::getDevice('files')` with your own wiring (a container, or passing the device instance directly), and inline `Storage::human()` if you used it. +- `setTelemetry()` is gone: wrap the device in the `Utopia\Storage\Device\Telemetry` decorator instead. `setHttpVersion()` is gone: transport options now belong to the injected PSR-18 client. The static `S3::setRetryAttempts()`/`S3::setRetryDelay()` moved into the `S3\RetryStrategy` used by the default client's `Retry` decorator — inject your own client to tune or disable retries. +- `setTransferChunkSize()`/`getTransferChunkSize()` became a per-call argument: `transfer($path, $destination, $device, $chunkSize)`. +- String constants became enums: the `Storage::DEVICE_*` constants are now the `Utopia\Storage\DeviceType` enum (`getType()` returns it), and the `S3::ACL_*` constants are now the `Utopia\Storage\Acl` enum. +- The S3 adapter no longer stores request headers on the instance, so one device can serve concurrent requests (for example Swoole coroutines) without data races. +- Requests go through a PSR-18 client instead of raw cURL calls. The default is [utopia-php/client](https://github.com/utopia-php/client) with the cURL adapter; pass the `client` constructor argument to swap the transport. +- Uploads are data-based: `upload($sourcePath, ...)` was removed — read the file yourself and call `uploadData($data, ...)` — and `uploadChunk()` now takes the chunk contents instead of a source file path. +- Filesystem-only methods (`createDirectory()`, `getDirectorySize()`, `getPartitionFreeSpace()`, `getPartitionTotalSpace()`) left the base `Device` contract and remain on `Local`; wrap devices in the `Telemetry` decorator and use its `getDevice()` accessor when you need them. `getFiles()` — which returned path strings on `Local` but a raw `ListObjectsV2` array on `S3` — is replaced by `listFiles(prefix, max, cursor)` returning typed `FileList`/`FileInfo` value objects consistently on every adapter. +- `getName()` and `getDescription()` were removed; use `getType()` to identify an adapter. +- Exceptions are typed: every runtime failure extends `Utopia\Storage\Exception\StorageException` (`NotFoundException`, `TransportException`, `RemoteException`, `UploadException`) instead of bare `\Exception`, and missing files throw `NotFoundException` consistently on every adapter and method — including S3 HEAD responses, which previously surfaced as a generic error with an empty message. Existing `catch (\Exception)` blocks keep working. + ## Adding new adapters For information on adding new storage adapters, see the [Adding New Storage Adapter](https://github.com/utopia-php/storage/blob/master/docs/adding-new-storage-adapter.md) guide. ## System requirements -Utopia Storage requires PHP 7.4 or later. We recommend using the latest PHP version whenever possible. +Utopia Storage requires PHP 8.5 or later. We recommend using the latest PHP version whenever possible. ## Contributing diff --git a/packages/storage/composer.json b/packages/storage/composer.json index 1b40306f5..cf019f1b4 100644 --- a/packages/storage/composer.json +++ b/packages/storage/composer.json @@ -22,18 +22,21 @@ }, "autoload-dev": { "psr-4": { - "Utopia\\Tests\\Storage\\": "tests/Storage" + "Utopia\\Tests\\Storage\\": "tests/Storage", + "Utopia\\Tests\\Storage\\E2E\\": "tests/E2E" } }, "require": { "php": ">=8.5", "ext-fileinfo": "*", - "ext-zlib": "*", "ext-curl": "*", "ext-simplexml": "*", - "utopia-php/system": "0.*.*", "utopia-php/telemetry": "^0.4", - "utopia-php/validators": "0.3.*" + "utopia-php/validators": "0.3.*", + "psr/http-client": "^1.0", + "psr/http-message": "^1.1 || ^2.0", + "utopia-php/client": "0.2.*", + "utopia-php/psr7": "0.2.*" }, "config": { "allow-plugins": { diff --git a/packages/storage/composer.lock b/packages/storage/composer.lock index 6374adbf4..530dbff26 100644 --- a/packages/storage/composer.lock +++ b/packages/storage/composer.lock @@ -4,7 +4,7 @@ "Read more about it at https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies", "This file is @generated automatically" ], - "content-hash": "fc0d06f2f6e7ba294835c5862488cbef", + "content-hash": "903e32d83a34d3421251f162b603aff9", "packages": [ { "name": "brick/math", @@ -144,23 +144,23 @@ }, { "name": "google/protobuf", - "version": "v4.33.6", + "version": "v5.35.1", "source": { "type": "git", "url": "https://github.com/protocolbuffers/protobuf-php.git", - "reference": "84b008c23915ed94536737eae46f41ba3bccfe67" + "reference": "55bb4a7d6739b5af0927b96213c1371a3afb7cfb" }, "dist": { "type": "zip", - "url": "https://api.github.com/repos/protocolbuffers/protobuf-php/zipball/84b008c23915ed94536737eae46f41ba3bccfe67", - "reference": "84b008c23915ed94536737eae46f41ba3bccfe67", + "url": "https://api.github.com/repos/protocolbuffers/protobuf-php/zipball/55bb4a7d6739b5af0927b96213c1371a3afb7cfb", + "reference": "55bb4a7d6739b5af0927b96213c1371a3afb7cfb", "shasum": "" }, "require": { - "php": ">=8.1.0" + "php": ">=8.2.0" }, "require-dev": { - "phpunit/phpunit": ">=10.5.62 <11.0.0" + "phpunit/phpunit": ">=11.5.0 <12.0.0" }, "suggest": { "ext-bcmath": "Need to support JSON deserialization" @@ -182,9 +182,9 @@ "proto" ], "support": { - "source": "https://github.com/protocolbuffers/protobuf-php/tree/v4.33.6" + "source": "https://github.com/protocolbuffers/protobuf-php/tree/v5.35.1" }, - "time": "2026-03-18T17:32:05+00:00" + "time": "2026-06-11T21:19:23+00:00" }, { "name": "nyholm/psr7", @@ -332,16 +332,16 @@ }, { "name": "open-telemetry/api", - "version": "1.9.0", + "version": "1.10.0", "source": { "type": "git", "url": "https://github.com/opentelemetry-php/api.git", - "reference": "6f8d237ce2c304ca85f31970f788e7f074d147be" + "reference": "7c029c4a6fd457094a20569bf98f93d95e9a7559" }, "dist": { "type": "zip", - "url": "https://api.github.com/repos/opentelemetry-php/api/zipball/6f8d237ce2c304ca85f31970f788e7f074d147be", - "reference": "6f8d237ce2c304ca85f31970f788e7f074d147be", + "url": "https://api.github.com/repos/opentelemetry-php/api/zipball/7c029c4a6fd457094a20569bf98f93d95e9a7559", + "reference": "7c029c4a6fd457094a20569bf98f93d95e9a7559", "shasum": "" }, "require": { @@ -398,7 +398,7 @@ "issues": "https://github.com/open-telemetry/opentelemetry-php/issues", "source": "https://github.com/open-telemetry/opentelemetry-php" }, - "time": "2026-02-25T13:24:05+00:00" + "time": "2026-07-06T12:28:04+00:00" }, { "name": "open-telemetry/context", @@ -525,20 +525,20 @@ }, { "name": "open-telemetry/gen-otlp-protobuf", - "version": "1.9.0", + "version": "1.10.0", "source": { "type": "git", "url": "https://github.com/opentelemetry-php/gen-otlp-protobuf.git", - "reference": "a229cf161d42001d64c8f21e8f678581fe1c66b9" + "reference": "66f04d0e448ad333033bfc7baae1aa56330be088" }, "dist": { "type": "zip", - "url": "https://api.github.com/repos/opentelemetry-php/gen-otlp-protobuf/zipball/a229cf161d42001d64c8f21e8f678581fe1c66b9", - "reference": "a229cf161d42001d64c8f21e8f678581fe1c66b9", + "url": "https://api.github.com/repos/opentelemetry-php/gen-otlp-protobuf/zipball/66f04d0e448ad333033bfc7baae1aa56330be088", + "reference": "66f04d0e448ad333033bfc7baae1aa56330be088", "shasum": "" }, "require": { - "google/protobuf": "^3.22 || ^4.0", + "google/protobuf": "^3.22 || ^4.0 || ^5.0", "php": "^8.0" }, "suggest": { @@ -584,20 +584,20 @@ "issues": "https://github.com/open-telemetry/opentelemetry-php/issues", "source": "https://github.com/open-telemetry/opentelemetry-php" }, - "time": "2025-10-19T06:44:33+00:00" + "time": "2026-06-17T12:06:32+00:00" }, { "name": "open-telemetry/sdk", - "version": "1.14.0", + "version": "1.15.0", "source": { "type": "git", "url": "https://github.com/opentelemetry-php/sdk.git", - "reference": "6e3d0ce93e76555dd5e2f1d19443ff45b990e410" + "reference": "77e1aa73850154abb86937d52a70883edc3b4547" }, "dist": { "type": "zip", - "url": "https://api.github.com/repos/opentelemetry-php/sdk/zipball/6e3d0ce93e76555dd5e2f1d19443ff45b990e410", - "reference": "6e3d0ce93e76555dd5e2f1d19443ff45b990e410", + "url": "https://api.github.com/repos/opentelemetry-php/sdk/zipball/77e1aa73850154abb86937d52a70883edc3b4547", + "reference": "77e1aa73850154abb86937d52a70883edc3b4547", "shasum": "" }, "require": { @@ -605,7 +605,7 @@ "nyholm/psr7-server": "^1.1", "open-telemetry/api": "^1.8", "open-telemetry/context": "^1.4", - "open-telemetry/sem-conv": "^1.0", + "open-telemetry/sem-conv": "^1.38.0", "php": "^8.1", "php-http/discovery": "^1.14", "psr/http-client": "^1.0", @@ -628,7 +628,10 @@ "spi": { "OpenTelemetry\\API\\Configuration\\ConfigEnv\\EnvComponentLoader": [ "OpenTelemetry\\API\\Instrumentation\\Configuration\\General\\ConfigEnv\\EnvComponentLoaderHttpConfig", - "OpenTelemetry\\API\\Instrumentation\\Configuration\\General\\ConfigEnv\\EnvComponentLoaderPeerConfig" + "OpenTelemetry\\API\\Instrumentation\\Configuration\\General\\ConfigEnv\\EnvComponentLoaderPeerConfig", + "OpenTelemetry\\SDK\\ConfigEnv\\Trace\\SpanSuppressionStrategySemConv", + "OpenTelemetry\\SDK\\ConfigEnv\\Trace\\SpanSuppressionStrategySpanKind", + "OpenTelemetry\\SDK\\ConfigEnv\\Distribution\\DistributionConfigurationSdk" ], "OpenTelemetry\\SDK\\Common\\Configuration\\Resolver\\ResolverInterface": [ "OpenTelemetry\\SDK\\Common\\Configuration\\Resolver\\SdkConfigurationResolver" @@ -638,7 +641,7 @@ ] }, "branch-alias": { - "dev-main": "1.12.x-dev" + "dev-main": "1.14.x-dev" } }, "autoload": { @@ -681,7 +684,7 @@ "issues": "https://github.com/open-telemetry/opentelemetry-php/issues", "source": "https://github.com/open-telemetry/opentelemetry-php" }, - "time": "2026-03-21T11:50:01+00:00" + "time": "2026-07-14T13:09:54+00:00" }, { "name": "open-telemetry/sem-conv", @@ -1875,26 +1878,93 @@ "time": "2025-06-29T15:42:06+00:00" }, { - "name": "utopia-php/system", - "version": "0.10.5", + "name": "utopia-php/client", + "version": "0.2.3", "source": { "type": "git", - "url": "https://github.com/utopia-php/system.git", - "reference": "57a2ab17f5346f171c796e29a6c2202bfaa21303" + "url": "https://github.com/utopia-php/client.git", + "reference": "377dc1f79441ac1584f25dba57904ee1cf0f626b" }, "dist": { "type": "zip", - "url": "https://api.github.com/repos/utopia-php/system/zipball/57a2ab17f5346f171c796e29a6c2202bfaa21303", - "reference": "57a2ab17f5346f171c796e29a6c2202bfaa21303", + "url": "https://api.github.com/repos/utopia-php/client/zipball/377dc1f79441ac1584f25dba57904ee1cf0f626b", + "reference": "377dc1f79441ac1584f25dba57904ee1cf0f626b", "shasum": "" }, "require": { - "php": ">=8.0.0" + "php": ">=8.5", + "psr/http-client": "^1.0", + "psr/http-factory": "^1.0", + "psr/http-message": "^1.1 || ^2.0", + "utopia-php/pools": "^1.0", + "utopia-php/psr7": "^0.2", + "utopia-php/span": "^1.1 || ^3.0 || ^4.0" + }, + "require-dev": { + "swoole/ide-helper": "^6.0" + }, + "suggest": { + "ext-curl": "Required to use the cURL HTTP client adapter.", + "ext-simplexml": "Required to decode XML responses with Response::xml().", + "ext-swoole": "Required to use the Swoole coroutine HTTP client adapter." + }, + "type": "library", + "autoload": { + "psr-4": { + "Utopia\\": "src/" + } + }, + "notification-url": "https://packagist.org/downloads/", + "license": [ + "MIT" + ], + "description": "A lightweight PSR-18 HTTP client with cURL and Swoole coroutine backends", + "keywords": [ + "client", + "curl", + "http", + "php", + "psr-18", + "swoole", + "utopia" + ], + "support": { + "issues": "https://github.com/utopia-php/client/issues", + "source": "https://github.com/utopia-php/client/tree/0.2.3" + }, + "time": "2026-07-13T15:29:09+00:00" + }, + { + "name": "utopia-php/pools", + "version": "1.1.0", + "source": { + "type": "git", + "url": "https://github.com/utopia-php/pools.git", + "reference": "fd5aa7e89685ffc74f9dff3d23c270ef30c7bbf3" + }, + "dist": { + "type": "zip", + "url": "https://api.github.com/repos/utopia-php/pools/zipball/fd5aa7e89685ffc74f9dff3d23c270ef30c7bbf3", + "reference": "fd5aa7e89685ffc74f9dff3d23c270ef30c7bbf3", + "shasum": "" + }, + "require": { + "php": ">=8.4", + "utopia-php/telemetry": "^0.4" + }, + "require-dev": { + "swoole/ide-helper": "6.*" + }, + "suggest": { + "ext-mongodb": "Needed to support MongoDB database pools", + "ext-pdo": "Needed to support MariaDB, MySQL or SQLite database pools", + "ext-redis": "Needed to support Redis cache pools", + "ext-swoole": "Needed to support Swoole based pool adapter" }, "type": "library", "autoload": { "psr-4": { - "Utopia\\System\\": "src/System" + "Utopia\\Pools\\": "src/Pools" } }, "notification-url": "https://packagist.org/downloads/", @@ -1903,27 +1973,108 @@ ], "authors": [ { - "name": "Eldad Fux", - "email": "eldad@appwrite.io" - }, - { - "name": "Torsten Dittmann", - "email": "torsten@appwrite.io" + "name": "Team Appwrite", + "email": "team@appwrite.io" } ], - "description": "A simple library for obtaining information about the host's system.", + "description": "A simple library to manage connection pools", "keywords": [ "framework", "php", - "system", - "upf", + "pools", + "utopia" + ], + "support": { + "issues": "https://github.com/utopia-php/pools/issues", + "source": "https://github.com/utopia-php/pools/tree/1.1.0" + }, + "time": "2026-07-17T10:19:09+00:00" + }, + { + "name": "utopia-php/psr7", + "version": "0.2.0", + "source": { + "type": "git", + "url": "https://github.com/utopia-php/psr7.git", + "reference": "115753c36194d53abe5587d383810724ac3a782d" + }, + "dist": { + "type": "zip", + "url": "https://api.github.com/repos/utopia-php/psr7/zipball/115753c36194d53abe5587d383810724ac3a782d", + "reference": "115753c36194d53abe5587d383810724ac3a782d", + "shasum": "" + }, + "require": { + "php": ">=8.4", + "psr/http-factory": "^1.0", + "psr/http-message": "^1.1 || ^2.0" + }, + "suggest": { + "ext-simplexml": "Required to decode XML responses with Response::xml()." + }, + "type": "library", + "autoload": { + "psr-4": { + "Utopia\\Psr7\\": "src/Psr7/" + } + }, + "notification-url": "https://packagist.org/downloads/", + "license": [ + "MIT" + ], + "description": "PSR-7 HTTP message implementations and PSR-17 factories for Utopia", + "keywords": [ + "http", + "php", + "psr-17", + "psr-7", "utopia" ], "support": { - "issues": "https://github.com/utopia-php/system/issues", - "source": "https://github.com/utopia-php/system/tree/0.10.5" + "issues": "https://github.com/utopia-php/psr7/issues", + "source": "https://github.com/utopia-php/psr7/tree/0.2.0" + }, + "time": "2026-07-06T12:40:23+00:00" + }, + { + "name": "utopia-php/span", + "version": "4.0.1", + "source": { + "type": "git", + "url": "https://github.com/utopia-php/span.git", + "reference": "d11f2714324cb22b286b2afbf3ea9de32c68de83" + }, + "dist": { + "type": "zip", + "url": "https://api.github.com/repos/utopia-php/span/zipball/d11f2714324cb22b286b2afbf3ea9de32c68de83", + "reference": "d11f2714324cb22b286b2afbf3ea9de32c68de83", + "shasum": "" + }, + "require": { + "php": ">=8.2" + }, + "require-dev": { + "swoole/ide-helper": "^5.0" + }, + "suggest": { + "ext-swoole": "Required for coroutine-based storage" + }, + "type": "library", + "autoload": { + "psr-4": { + "Utopia\\Span\\": "src/Span/" + } + }, + "notification-url": "https://packagist.org/downloads/", + "license": [ + "MIT" + ], + "description": "Simple span tracing library for PHP with coroutine support", + "support": { + "issues": "https://github.com/utopia-php/span/issues", + "source": "https://github.com/utopia-php/span/tree/4.0.1" }, - "time": "2026-06-26T10:48:18+00:00" + "time": "2026-06-20T09:45:06+00:00" }, { "name": "utopia-php/telemetry", @@ -1978,16 +2129,16 @@ }, { "name": "utopia-php/validators", - "version": "0.3.0", + "version": "0.3.1", "source": { "type": "git", "url": "https://github.com/utopia-php/validators.git", - "reference": "5a3ef67ce17c2e148da7895e8de699614f0afdfa" + "reference": "d9c2269ebd2596a09681ccd2fd133eeedfcdf9ba" }, "dist": { "type": "zip", - "url": "https://api.github.com/repos/utopia-php/validators/zipball/5a3ef67ce17c2e148da7895e8de699614f0afdfa", - "reference": "5a3ef67ce17c2e148da7895e8de699614f0afdfa", + "url": "https://api.github.com/repos/utopia-php/validators/zipball/d9c2269ebd2596a09681ccd2fd133eeedfcdf9ba", + "reference": "d9c2269ebd2596a09681ccd2fd133eeedfcdf9ba", "shasum": "" }, "require": { @@ -2012,9 +2163,9 @@ ], "support": { "issues": "https://github.com/utopia-php/validators/issues", - "source": "https://github.com/utopia-php/validators/tree/0.3.0" + "source": "https://github.com/utopia-php/validators/tree/0.3.1" }, - "time": "2026-07-13T09:46:33+00:00" + "time": "2026-07-14T11:55:48+00:00" } ], "packages-dev": [], @@ -2026,7 +2177,6 @@ "platform": { "php": ">=8.5", "ext-fileinfo": "*", - "ext-zlib": "*", "ext-curl": "*", "ext-simplexml": "*" }, diff --git a/packages/storage/docs/adding-new-storage-adapter.md b/packages/storage/docs/adding-new-storage-adapter.md index 9ffabd092..9b6d2935a 100644 --- a/packages/storage/docs/adding-new-storage-adapter.md +++ b/packages/storage/docs/adding-new-storage-adapter.md @@ -18,8 +18,8 @@ In order to start implementing new storage adapter, add new file inside `package Always use properly named environment variables if any credentials are required. -### 2.2. Introduce new device constant -Introduce newly added device constant in `src/Storage/Storage.php` alongside existing device constants. The device constant should start with `const DEVICE_` as the existing ones. +### 2.2. Introduce new device type +Add a case for the new device to the `DeviceType` enum in `src/Storage/DeviceType.php` alongside the existing cases, and return it from your adapter's `getType()` method. ## 3. Test your adapter diff --git a/packages/storage/phpstan.neon b/packages/storage/phpstan.neon new file mode 100644 index 000000000..5d9dff0bd --- /dev/null +++ b/packages/storage/phpstan.neon @@ -0,0 +1,5 @@ +includes: + - ../../phpstan.neon + +parameters: + level: max diff --git a/packages/storage/phpunit.xml b/packages/storage/phpunit.xml index 1de1f16f1..48e0eb4d9 100644 --- a/packages/storage/phpunit.xml +++ b/packages/storage/phpunit.xml @@ -6,13 +6,10 @@ cacheDirectory=".phpunit.cache"> - tests/Storage/Device/LocalTest.php - tests/Storage/Device/S3SlowDownTest.php - tests/Storage/StorageTest.php - tests/Storage/Validator + tests/Storage - tests/Storage/Device/S3Test.php + tests/E2E diff --git a/packages/storage/rector.php b/packages/storage/rector.php new file mode 100644 index 000000000..35a483abe --- /dev/null +++ b/packages/storage/rector.php @@ -0,0 +1,11 @@ +withPreparedSets( + typeDeclarationDocblocks: true, + privatization: true, + instanceOf: true, + rectorPreset: true, +); diff --git a/packages/storage/src/Storage/Acl.php b/packages/storage/src/Storage/Acl.php new file mode 100644 index 000000000..3c866b046 --- /dev/null +++ b/packages/storage/src/Storage/Acl.php @@ -0,0 +1,19 @@ +, chunks?: int, content_type?: string, uploadId?: string} + */ abstract class Device { /** - * Max chunk size while transferring file from one device to another - */ - protected int $transferChunkSize = 20000000; // 20 MB - - /** - * Sets the maximum number of keys returned to the response. By default, the action returns up to 1,000 key names. - */ - protected const MAX_PAGE_SIZE = PHP_INT_MAX; - - protected Histogram $storageOperationTelemetry; - - public function __construct(Telemetry $telemetry = new NoTelemetry()) - { - $this->setTelemetry($telemetry); - } - - /** - * Set Transfer Chunk Size - */ - public function setTransferChunkSize(int $chunkSize): void - { - $this->transferChunkSize = $chunkSize; - } - - /** - * Get Transfer Chunk Size - */ - public function getTransferChunkSize(): int - { - return $this->transferChunkSize; - } - - /** - * Get Name. - * - * Get storage device name + * Default max chunk size while transferring file from one device to another */ - abstract public function getName(): string; + public const TRANSFER_CHUNK_SIZE = 20000000; // 20 MB /** * Get Type. * * Get storage device type */ - abstract public function getType(): string; - - /** - * Get Description. - * - * Get storage device description and purpose. - */ - abstract public function getDescription(): string; - - public function setTelemetry(Telemetry $telemetry): void - { - $this->storageOperationTelemetry = Histogram::lazy( - telemetry: $telemetry, - name: 'storage.operation', - unit: 's', - advisory: ['ExplicitBucketBoundaries' => [0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10]], - ); - } + abstract public function getType(): DeviceType; /** * Get Root. @@ -85,43 +36,39 @@ abstract public function getRoot(): string; * * Each device hold a complex directory structure that is being build in this method. */ - abstract public function getPath(string $filename, ?string $prefix = null): string; - - /** - * Upload. - * - * Upload a file to desired destination in the selected disk - * return number of chunks uploaded or 0 if it fails. - * - * - * @throws Exception - */ - abstract public function upload(string $source, string $path, int $chunk = 1, int $chunks = 1, array &$metadata = []): int; + abstract public function getPath(string $filename): string; /** * Prepare Upload. * * Initialize adapter-specific upload state without transferring a chunk body. * - * @throws Exception + * @param UploadMetadata $metadata + * + * @throws StorageException */ abstract public function prepareUpload(string $path, string $contentType, int $chunks = 1, array &$metadata = []): void; /** * Upload Chunk. * - * Upload exactly one chunk without finalizing the full upload. + * Upload exactly one chunk of file contents without finalizing the full upload. + * Returns the number of chunks received so far. * - * @throws Exception + * @param UploadMetadata $metadata + * + * @throws StorageException */ - abstract public function uploadChunk(string $source, string $path, int $chunk = 1, int $chunks = 1, array &$metadata = []): int; + abstract public function uploadChunk(string $data, string $path, int $chunk = 1, int $chunks = 1, array &$metadata = []): int; /** * Finalize Upload. * * Complete a prepared upload once all chunks are known to be present. * - * @throws Exception + * @param UploadMetadata $metadata + * + * @throws StorageException */ abstract public function finalizeUpload(string $path, int $chunks = 1, array &$metadata = []): bool; @@ -129,28 +76,43 @@ abstract public function finalizeUpload(string $path, int $chunks = 1, array &$m * Upload Data. * * Upload file contents to desired destination in the selected disk. - * return number of chunks uploaded or 0 if it fails. + * Returns the number of chunks received so far. * + * @param UploadMetadata $metadata * - * @throws Exception + * @throws StorageException */ - abstract public function uploadData(string $data, string $path, string $contentType, int $chunk = 1, int $chunks = 1, array &$metadata = []): int; + public function uploadData(string $data, string $path, string $contentType, int $chunk = 1, int $chunks = 1, array &$metadata = []): int + { + $this->prepareUpload($path, $contentType, $chunks, $metadata); + $chunksReceived = $this->uploadChunk($data, $path, $chunk, $chunks, $metadata); + + if ($chunks > 1 && $chunks === $chunksReceived && ! $this->finalizeUpload($path, $chunks, $metadata)) { + throw new UploadException('Failed to finalize upload ' . $path); + } + + return $chunksReceived; + } /** * Abort Chunked Upload */ - abstract public function abort(string $path, string $extra = ''): bool; + abstract public function abort(string $path, string $uploadId = ''): bool; /** * Read file by given path. + * + * @param int<0, max> $offset + * @param int<0, max>|null $length */ abstract public function read(string $path, int $offset = 0, ?int $length = null): string; /** * Transfer * Transfer a file from current device to destination device. + * */ - abstract public function transfer(string $path, string $destination, Device $device): bool; + abstract public function transfer(string $path, string $destination, Device $device, int $chunkSize = self::TRANSFER_CHUNK_SIZE): bool; /** * Write file by given path. @@ -159,8 +121,6 @@ abstract public function write(string $path, string $data, string $contentType): /** * Move file from given source to given path, return true on success and false on failure. - * - * @see http://php.net/manual/en/function.filesize.php */ public function move(string $source, string $target): bool { @@ -177,8 +137,6 @@ public function move(string $source, string $target): bool /** * Delete file in given path return true on success and false on failure. - * - * @see http://php.net/manual/en/function.filesize.php */ abstract public function delete(string $path, bool $recursive = false): bool; @@ -193,64 +151,27 @@ abstract public function deletePath(string $path): bool; abstract public function exists(string $path): bool; /** - * Returns given file path its size. + * List files under the given prefix, one page at a time. * - * @see http://php.net/manual/en/function.filesize.php + * @param int<1, max> $max + */ + abstract public function listFiles(string $prefix = '', int $max = 1000, ?string $cursor = null): FileList; + + /** + * Returns given file path its size. */ abstract public function getFileSize(string $path): int; /** * Returns given file path its mime type. - * - * @see http://php.net/manual/en/function.mime-content-type.php */ abstract public function getFileMimeType(string $path): string; /** * Returns given file path its MD5 hash value. - * - * @see http://php.net/manual/en/function.md5-file.php */ abstract public function getFileHash(string $path): string; - /** - * Create a directory at the specified path. - * - * Returns true on success or if the directory already exists and false on error - */ - abstract public function createDirectory(string $path): bool; - - /** - * Get directory size in bytes. - * - * Return -1 on error - * - * Based on http://www.jonasjohn.de/snippets/php/dir-size.htm - */ - abstract public function getDirectorySize(string $path): int; - - /** - * Get Partition Free Space. - * - * disk_free_space — Returns available space on filesystem or disk partition - */ - abstract public function getPartitionFreeSpace(): float; - - /** - * Get Partition Total Space. - * - * disk_total_space — Returns the total size of a filesystem or disk partition - */ - abstract public function getPartitionTotalSpace(): float; - - /** - * Get all files and directories inside a directory. - * - * @param string $dir Directory to scan - * @return array - */ - abstract public function getFiles(string $dir, int $max = self::MAX_PAGE_SIZE, string $continuationToken = ''): array; - /** * Get the absolute path by resolving strings like ../, .., //, /\ and so on. * diff --git a/packages/storage/src/Storage/Device/AWS.php b/packages/storage/src/Storage/Device/AWS.php index 2ea561b1b..5bd82acb7 100644 --- a/packages/storage/src/Storage/Device/AWS.php +++ b/packages/storage/src/Storage/Device/AWS.php @@ -4,7 +4,9 @@ namespace Utopia\Storage\Device; -use Utopia\Storage\Storage; +use Psr\Http\Client\ClientInterface; +use Utopia\Storage\Acl; +use Utopia\Storage\DeviceType; class AWS extends S3 { @@ -64,32 +66,29 @@ class AWS extends S3 public const US_GOV_WEST_1 = 'us-gov-west-1'; /** - * S3 Constructor + * AWS Constructor */ - public function __construct(string $root, string $accessKey, string $secretKey, string $bucket, string $region = self::US_EAST_1, string $acl = self::ACL_PRIVATE) - { + public function __construct( + string $root, + string $accessKey, + #[\SensitiveParameter] + string $secretKey, + string $bucket, + string $region = self::US_EAST_1, + Acl $acl = Acl::Private, + ?ClientInterface $client = null, + ) { $host = match ($region) { self::CN_NORTH_1, self::CN_NORTH_4, self::CN_NORTHWEST_1 => $bucket . '.s3.' . $region . '.amazonaws.cn', default => $bucket . '.s3.' . $region . '.amazonaws.com' }; - parent::__construct($root, $accessKey, $secretKey, $host, $region, $acl); - } - - #[\Override] - public function getName(): string - { - return 'AWS S3 Storage'; + parent::__construct($root, $accessKey, $secretKey, $host, $region, $acl, $client); } #[\Override] - public function getType(): string + public function getType(): DeviceType { - return Storage::DEVICE_AWS_S3; + return DeviceType::AwsS3; } - #[\Override] - public function getDescription(): string - { - return 'S3 Bucket Storage drive for AWS'; - } } diff --git a/packages/storage/src/Storage/Device/Backblaze.php b/packages/storage/src/Storage/Device/Backblaze.php index c4dbacc9c..49f1372fa 100644 --- a/packages/storage/src/Storage/Device/Backblaze.php +++ b/packages/storage/src/Storage/Device/Backblaze.php @@ -4,7 +4,9 @@ namespace Utopia\Storage\Device; -use Utopia\Storage\Storage; +use Psr\Http\Client\ClientInterface; +use Utopia\Storage\Acl; +use Utopia\Storage\DeviceType; class Backblaze extends S3 { @@ -27,27 +29,23 @@ class Backblaze extends S3 /** * Backblaze Constructor */ - public function __construct(string $root, string $accessKey, string $secretKey, string $bucket, string $region = self::US_WEST_004, string $acl = self::ACL_PRIVATE) - { + public function __construct( + string $root, + string $accessKey, + #[\SensitiveParameter] + string $secretKey, + string $bucket, + string $region = self::US_WEST_004, + Acl $acl = Acl::Private, + ?ClientInterface $client = null, + ) { $host = $bucket . '.' . 's3' . '.' . $region . '.backblazeb2.com'; - parent::__construct($root, $accessKey, $secretKey, $host, $region, $acl); - } - - #[\Override] - public function getName(): string - { - return 'Backblaze B2 Storage'; - } - - #[\Override] - public function getDescription(): string - { - return 'Backblaze B2 Storage'; + parent::__construct($root, $accessKey, $secretKey, $host, $region, $acl, $client); } #[\Override] - public function getType(): string + public function getType(): DeviceType { - return Storage::DEVICE_BACKBLAZE; + return DeviceType::Backblaze; } } diff --git a/packages/storage/src/Storage/Device/DOSpaces.php b/packages/storage/src/Storage/Device/DOSpaces.php index 4c4beacc5..a6aeea01a 100644 --- a/packages/storage/src/Storage/Device/DOSpaces.php +++ b/packages/storage/src/Storage/Device/DOSpaces.php @@ -4,7 +4,9 @@ namespace Utopia\Storage\Device; -use Utopia\Storage\Storage; +use Psr\Http\Client\ClientInterface; +use Utopia\Storage\Acl; +use Utopia\Storage\DeviceType; class DOSpaces extends S3 { @@ -26,27 +28,23 @@ class DOSpaces extends S3 /** * DOSpaces Constructor */ - public function __construct(string $root, string $accessKey, string $secretKey, string $bucket, string $region = self::NYC3, string $acl = self::ACL_PRIVATE) - { + public function __construct( + string $root, + string $accessKey, + #[\SensitiveParameter] + string $secretKey, + string $bucket, + string $region = self::NYC3, + Acl $acl = Acl::Private, + ?ClientInterface $client = null, + ) { $host = $bucket . '.' . $region . '.digitaloceanspaces.com'; - parent::__construct($root, $accessKey, $secretKey, $host, $region, $acl); - } - - #[\Override] - public function getName(): string - { - return 'Digitalocean Spaces Storage'; - } - - #[\Override] - public function getDescription(): string - { - return 'Digitalocean Spaces Storage'; + parent::__construct($root, $accessKey, $secretKey, $host, $region, $acl, $client); } #[\Override] - public function getType(): string + public function getType(): DeviceType { - return Storage::DEVICE_DO_SPACES; + return DeviceType::DoSpaces; } } diff --git a/packages/storage/src/Storage/Device/Linode.php b/packages/storage/src/Storage/Device/Linode.php index 55e1aff12..74519be0f 100644 --- a/packages/storage/src/Storage/Device/Linode.php +++ b/packages/storage/src/Storage/Device/Linode.php @@ -4,7 +4,9 @@ namespace Utopia\Storage\Device; -use Utopia\Storage\Storage; +use Psr\Http\Client\ClientInterface; +use Utopia\Storage\Acl; +use Utopia\Storage\DeviceType; class Linode extends S3 { @@ -22,27 +24,23 @@ class Linode extends S3 /** * Object Storage Constructor */ - public function __construct(string $root, string $accessKey, string $secretKey, string $bucket, string $region = self::EU_CENTRAL_1, string $acl = self::ACL_PRIVATE) - { + public function __construct( + string $root, + string $accessKey, + #[\SensitiveParameter] + string $secretKey, + string $bucket, + string $region = self::EU_CENTRAL_1, + Acl $acl = Acl::Private, + ?ClientInterface $client = null, + ) { $host = $bucket . '.' . $region . '.' . 'linodeobjects.com'; - parent::__construct($root, $accessKey, $secretKey, $host, $region, $acl); - } - - #[\Override] - public function getName(): string - { - return 'Linode Object Storage'; - } - - #[\Override] - public function getDescription(): string - { - return 'Linode Object Storage'; + parent::__construct($root, $accessKey, $secretKey, $host, $region, $acl, $client); } #[\Override] - public function getType(): string + public function getType(): DeviceType { - return Storage::DEVICE_LINODE; + return DeviceType::Linode; } } diff --git a/packages/storage/src/Storage/Device/Local.php b/packages/storage/src/Storage/Device/Local.php index 7e9566d9b..de22aa87e 100644 --- a/packages/storage/src/Storage/Device/Local.php +++ b/packages/storage/src/Storage/Device/Local.php @@ -1,35 +1,32 @@ root; } - public function getPath(string $filename, ?string $prefix = null): string + public function getPath(string $filename): string { return $this->getAbsolutePath($this->getRoot() . DIRECTORY_SEPARATOR . $filename); } /** - * Upload. - * - * Upload a file to desired destination in the selected disk. - * return number of chunks uploaded or 0 if it fails. - * - * - * @throws Exception + * @param UploadMetadata $metadata */ - public function upload(string $source, string $path, int $chunk = 1, int $chunks = 1, array &$metadata = []): int - { - $this->prepareUpload($path, '', $chunks, $metadata); - $chunksReceived = $this->uploadChunk($source, $path, $chunk, $chunks, $metadata); - - if ($chunks > 1 && $chunks === $chunksReceived && ! $this->finalizeUpload($path, $chunks, $metadata)) { - throw new Exception('Failed to finalize upload ' . $path); - } - - return $chunksReceived; - } - public function prepareUpload(string $path, string $contentType, int $chunks = 1, array &$metadata = []): void { $this->createDirectory(\dirname($path)); @@ -70,15 +49,15 @@ public function prepareUpload(string $path, string $contentType, int $chunks = 1 $metadata['chunks'] ??= 0; } - public function uploadChunk(string $source, string $path, int $chunk = 1, int $chunks = 1, array &$metadata = []): int + public function uploadChunk(string $data, string $path, int $chunk = 1, int $chunks = 1, array &$metadata = []): int { $this->createDirectory(\dirname($path)); $metadata['parts'] ??= []; $metadata['chunks'] ??= 0; if ($chunks === 1) { - if (! move_uploaded_file($source, $path) && ! rename($source, $path)) { - throw new Exception('Can\'t upload file ' . $path); + if (file_put_contents($path, $data) === false) { + throw new StorageException('Can\'t write file ' . $path); } $metadata['parts'][$chunk] = true; @@ -93,12 +72,8 @@ public function uploadChunk(string $source, string $path, int $chunk = 1, int $c $chunkFilePath = $tmp . DIRECTORY_SEPARATOR . pathinfo($path, PATHINFO_FILENAME) . '.part.' . $chunk; // skip writing chunk if the chunk was re-uploaded - if (! file_exists($chunkFilePath)) { - if (! rename($source, $chunkFilePath)) { - throw new Exception('Failed to write chunk ' . $chunk); - } - } elseif (file_exists($source)) { - unlink($source); + if (!file_exists($chunkFilePath) && file_put_contents($chunkFilePath, $data) === false) { + throw new StorageException('Failed to write chunk ' . $chunk); } $chunksReceived = $this->countChunks($tmp, $path); @@ -119,10 +94,10 @@ public function finalizeUpload(string $path, int $chunks = 1, array &$metadata = } $tmp = \dirname($path) . DIRECTORY_SEPARATOR . 'tmp_' . basename($path); - for ($i = 1; $i <= $chunks; $i++) { + for ($i = 1; $i <= $chunks; ++$i) { $part = $tmp . DIRECTORY_SEPARATOR . pathinfo($path, PATHINFO_FILENAME) . '.part.' . $i; if (! file_exists($part)) { - throw new Exception('Missing chunk ' . $i); + throw new UploadException('Missing chunk ' . $i); } } @@ -131,48 +106,6 @@ public function finalizeUpload(string $path, int $chunks = 1, array &$metadata = return true; } - /** - * Upload Data. - * - * Upload file contents to desired destination in the selected disk. - * return number of chunks uploaded or 0 if it fails. - * - * - * @throws Exception - */ - public function uploadData(string $data, string $path, string $contentType, int $chunk = 1, int $chunks = 1, array &$metadata = []): int - { - $this->prepareUpload($path, $contentType, $chunks, $metadata); - - if ($chunks === 1) { - if (! file_put_contents($path, $data)) { - throw new Exception('Can\'t write file ' . $path); - } - - return $chunks; - } - - $tmp = \dirname($path) . DIRECTORY_SEPARATOR . 'tmp_' . basename($path); - $this->createDirectory($tmp); - - $chunkFilePath = $tmp . DIRECTORY_SEPARATOR . pathinfo($path, PATHINFO_FILENAME) . '.part.' . $chunk; - - // skip writing chunk if the chunk was re-uploaded - if (!file_exists($chunkFilePath) && ! file_put_contents($chunkFilePath, $data)) { - throw new Exception('Failed to write chunk ' . $chunk); - } - - $chunksReceived = $this->countChunks($tmp, $path); - $metadata['parts'][$chunk] = true; - $metadata['chunks'] = $chunksReceived; - - if ($chunks > 1 && $chunks === $chunksReceived && ! $this->finalizeUpload($path, $chunks, $metadata)) { - throw new Exception('Failed to finalize upload ' . $path); - } - - return $chunksReceived; - } - private function countChunks(string $tmp, string $path): int { $escaped = (fn(string $literal): string => str_replace(['\\', '*', '?', '[', ']', '{', '}'], ['\\\\', '\\*', '\\?', '\\[', '\\]', '\\{', '\\}'], $literal)); @@ -185,7 +118,7 @@ private function countChunks(string $tmp, string $path): int $count = 0; foreach ($files as $file) { if (preg_match('/\.part\.\d+$/', $file)) { - $count++; + ++$count; } } @@ -203,24 +136,24 @@ private function joinChunks(string $path, int $chunks): void $dest = fopen($tmpAssemble, 'wb'); if ($dest === false) { - throw new Exception('Failed to open temporary assembly file ' . $tmpAssemble); + throw new StorageException('Failed to open temporary assembly file ' . $tmpAssemble); } $partsToUnlink = []; - for ($i = 1; $i <= $chunks; $i++) { + for ($i = 1; $i <= $chunks; ++$i) { $part = $tmp . DIRECTORY_SEPARATOR . pathinfo($path, PATHINFO_FILENAME) . '.part.' . $i; $src = @fopen($part, 'rb'); if ($src === false) { fclose($dest); unlink($tmpAssemble); - throw new Exception('Failed to open chunk ' . $part); + throw new StorageException('Failed to open chunk ' . $part); } if (stream_copy_to_stream($src, $dest) === false) { fclose($src); fclose($dest); unlink($tmpAssemble); - throw new Exception('Failed to copy chunk ' . $part); + throw new StorageException('Failed to copy chunk ' . $part); } fclose($src); $partsToUnlink[] = $part; @@ -235,7 +168,7 @@ private function joinChunks(string $path, int $chunks): void return; } unlink($tmpAssemble); - throw new Exception('Failed to finalize assembled file ' . $path); + throw new StorageException('Failed to finalize assembled file ' . $path); } foreach ($partsToUnlink as $part) { @@ -252,25 +185,29 @@ private function joinChunks(string $path, int $chunks): void /** * Transfer */ - public function transfer(string $path, string $destination, Device $device): bool + public function transfer(string $path, string $destination, Device $device, int $chunkSize = self::TRANSFER_CHUNK_SIZE): bool { + if ($chunkSize <= 0) { + throw new \InvalidArgumentException('Chunk size must be greater than zero'); + } + if (! $this->exists($path)) { - throw new Exception('File Not Found'); + throw new NotFoundException('File not found'); } $size = $this->getFileSize($path); $contentType = $this->getFileMimeType($path); - if ($size <= $this->transferChunkSize) { + if ($size <= $chunkSize) { $source = $this->read($path); return $device->write($destination, $source, $contentType); } - $totalChunks = (int) ceil($size / $this->transferChunkSize); + $totalChunks = (int) ceil($size / $chunkSize); $metadata = ['content_type' => $contentType]; - for ($counter = 0; $counter < $totalChunks; $counter++) { - $start = $counter * $this->transferChunkSize; - $data = $this->read($path, $start, $this->transferChunkSize); + for ($counter = 0; $counter < $totalChunks; ++$counter) { + $start = $counter * $chunkSize; + $data = $this->read($path, $start, $chunkSize); $device->uploadData($data, $destination, $contentType, $counter + 1, $totalChunks, $metadata); } @@ -280,7 +217,7 @@ public function transfer(string $path, string $destination, Device $device): boo /** * Abort Chunked Upload */ - public function abort(string $path, string $extra = ''): bool + public function abort(string $path, string $uploadId = ''): bool { if (file_exists($path)) { unlink($path); @@ -289,9 +226,9 @@ public function abort(string $path, string $extra = ''): bool $tmp = \dirname($path) . DIRECTORY_SEPARATOR . 'tmp_' . basename($path) . DIRECTORY_SEPARATOR; if (! file_exists(\dirname($tmp))) { // Checks if directory path to file exists - throw new Exception('File doesn\'t exist: ' . \dirname($path)); + throw new NotFoundException('File doesn\'t exist: ' . \dirname($path)); } - $files = $this->getFiles($tmp); + $files = $this->scanDirectory($tmp); foreach ($files as $file) { $this->delete($file, true); @@ -304,7 +241,7 @@ public function abort(string $path, string $extra = ''): bool * Read file by given path. * * - * @throws Exception + * @throws StorageException */ public function read(string $path, int $offset = 0, ?int $length = null): string { @@ -312,7 +249,12 @@ public function read(string $path, int $offset = 0, ?int $length = null): string throw new NotFoundException('File not found'); } - return file_get_contents($path, use_include_path: false, offset: $offset, length: $length); + $contents = file_get_contents($path, use_include_path: false, offset: $offset, length: $length); + if ($contents === false) { + throw new StorageException('Failed to read file ' . $path); + } + + return $contents; } /** @@ -322,7 +264,7 @@ public function write(string $path, string $data, string $contentType = ''): boo { // Checks if directory path to file exists if (!file_exists(\dirname($path)) && ! @mkdir(\dirname($path), 0755, true)) { - throw new Exception('Can\'t create directory ' . \dirname($path)); + throw new StorageException('Can\'t create directory ' . \dirname($path)); } return (bool) file_put_contents($path, $data); @@ -333,7 +275,7 @@ public function write(string $path, string $data, string $contentType = ''): boo * * @see http://php.net/manual/en/function.filesize.php * - * @throws Exception + * @throws StorageException */ #[\Override] public function move(string $source, string $target): bool @@ -344,7 +286,7 @@ public function move(string $source, string $target): bool // Checks if directory path to file exists if (!file_exists(\dirname($target)) && ! @mkdir(\dirname($target), 0755, true)) { - throw new Exception('Can\'t create directory ' . \dirname($target)); + throw new StorageException('Can\'t create directory ' . \dirname($target)); } return rename($source, $target); } @@ -392,11 +334,11 @@ public function deletePath(string $path): bool { $path = realpath($this->getRoot() . DIRECTORY_SEPARATOR . $path); - if (! file_exists($path) || ! is_dir($path)) { + if ($path === false || ! is_dir($path)) { return false; } - $files = $this->getFiles($path); + $files = $this->scanDirectory($path); foreach ($files as $file) { if (is_dir($file)) { @@ -424,7 +366,12 @@ public function exists(string $path): bool */ public function getFileSize(string $path): int { - return filesize($path); + $size = $this->exists($path) ? filesize($path) : false; + if ($size === false) { + throw $this->exists($path) ? new StorageException('Failed to get size of file ' . $path) : new NotFoundException('File not found: ' . $path); + } + + return $size; } /** @@ -434,7 +381,12 @@ public function getFileSize(string $path): int */ public function getFileMimeType(string $path): string { - return mime_content_type($path); + $mimeType = $this->exists($path) ? mime_content_type($path) : false; + if ($mimeType === false) { + throw $this->exists($path) ? new StorageException('Failed to get mime type of file ' . $path) : new NotFoundException('File not found: ' . $path); + } + + return $mimeType; } /** @@ -444,7 +396,12 @@ public function getFileMimeType(string $path): string */ public function getFileHash(string $path): string { - return md5_file($path); + $hash = $this->exists($path) ? md5_file($path) : false; + if ($hash === false) { + throw $this->exists($path) ? new StorageException('Failed to hash file ' . $path) : new NotFoundException('File not found: ' . $path); + } + + return $hash; } /** @@ -504,7 +461,7 @@ public function getDirectorySize(string $path): int */ public function getPartitionFreeSpace(): float { - return disk_free_space($this->getRoot()); + return disk_free_space($this->getRoot()) ?: 0.0; } /** @@ -514,27 +471,63 @@ public function getPartitionFreeSpace(): float */ public function getPartitionTotalSpace(): float { - return disk_total_space($this->getRoot()); + return disk_total_space($this->getRoot()) ?: 0.0; } /** - * Get all files and directories inside a directory. + * List all files under the given directory, recursively, sorted by path. * - * @return string[] + * The cursor is a numeric offset into the sorted listing. */ - public function getFiles(string $dir, int $max = self::MAX_PAGE_SIZE, string $continuationToken = ''): array + public function listFiles(string $prefix = '', int $max = 1000, ?string $cursor = null): FileList { - $dir = rtrim($dir, DIRECTORY_SEPARATOR); + $paths = []; + $pending = [rtrim($prefix, DIRECTORY_SEPARATOR)]; + while ($pending !== []) { + $directory = array_pop($pending); + foreach ($this->scanDirectory($directory) as $entry) { + if (is_dir($entry)) { + $pending[] = $entry; + } else { + $paths[] = $entry; + } + } + } + sort($paths); + + $offset = is_numeric($cursor) ? (int) $cursor : 0; + $page = \array_slice($paths, $offset, $max); + $files = []; + foreach ($page as $path) { + $modified = filemtime($path); + $files[] = new FileInfo( + path: $path, + size: filesize($path) ?: 0, + modifiedAt: $modified === false ? null : new \DateTimeImmutable('@' . $modified), + ); + } + + return new FileList( + files: $files, + cursor: $offset + \count($page) < \count($paths) ? (string) ($offset + \count($page)) : null, + ); + } - foreach (glob($dir . DIRECTORY_SEPARATOR . '*') as $file) { - $files[] = $file; - } + /** + * Get all files and directories directly inside a directory, hidden entries included. + * + * @return string[] + */ + private function scanDirectory(string $dir): array + { + $dir = rtrim($dir, DIRECTORY_SEPARATOR); + $files = glob($dir . DIRECTORY_SEPARATOR . '*') ?: []; /** * Hidden files */ - foreach (glob($dir . DIRECTORY_SEPARATOR . '.[!.]*') as $file) { + foreach (glob($dir . DIRECTORY_SEPARATOR . '.[!.]*') ?: [] as $file) { $files[] = $file; } diff --git a/packages/storage/src/Storage/Device/S3.php b/packages/storage/src/Storage/Device/S3.php index 966731968..3c17e46ae 100644 --- a/packages/storage/src/Storage/Device/S3.php +++ b/packages/storage/src/Storage/Device/S3.php @@ -1,102 +1,76 @@ '', - 'date' => '', - 'content-md5' => '', - 'content-type' => '', - ]; - - protected string $fqdn; - - protected array $amzHeaders = []; - - /** - * Http version - */ - protected ?int $curlHttpVersion = null; + private readonly ClientInterface $client; /** * S3 Constructor - */ - public function __construct(protected string $root, protected string $accessKey, protected string $secretKey, string $host, protected string $region, protected string $acl = self::ACL_PRIVATE) - { - parent::__construct(); - + * + * @param ClientInterface|null $client PSR-18 client used for every request; defaults to `utopia-php/client` with the cURL adapter, no request timeout, and transient-error retries via `S3\RetryStrategy` + */ + public function __construct( + protected readonly string $root, + private readonly string $accessKey, + #[\SensitiveParameter] + private readonly string $secretKey, + string $host, + protected readonly string $region, + protected readonly Acl $acl = Acl::Private, + ?ClientInterface $client = null, + ) { if (str_starts_with($host, 'http://') || str_starts_with($host, 'https://')) { $this->fqdn = $host; - $this->headers['host'] = str_replace(['http://', 'https://'], '', $host); + $this->host = str_replace(['http://', 'https://'], '', $host); } else { $this->fqdn = 'https://' . $host; - $this->headers['host'] = $host; + $this->host = $host; } - } - public function getName(): string - { - return 'S3 Storage'; + $this->client = $client ?? new Retry( + new HttpClient(new CurlAdapter())->withTimeout(0.0), + new S3\RetryStrategy(), + ); } - public function getType(): string + public function getType(): DeviceType { - return Storage::DEVICE_S3; - } - - public function getDescription(): string - { - return 'S3 Storage drive for generic S3-compatible provider'; + return DeviceType::S3; } public function getRoot(): string @@ -104,82 +78,27 @@ public function getRoot(): string return $this->root; } - public function getPath(string $filename, ?string $prefix = null): string + public function getPath(string $filename): string { return $this->getRoot() . DIRECTORY_SEPARATOR . $filename; } /** - * Set http version - */ - public function setHttpVersion(?int $httpVersion): self - { - $this->curlHttpVersion = $httpVersion; - - return $this; - } - - /** - * Set retry attempts + * @param UploadMetadata $metadata */ - public static function setRetryAttempts(int $attempts): void - { - self::$retryAttempts = $attempts; - } - - /** - * Set retry delay in milliseconds - */ - public static function setRetryDelay(int $delay): void - { - self::$retryDelay = $delay; - } - - /** - * Upload. - * - * Upload a file to desired destination in the selected disk. - * return number of chunks uploaded or 0 if it fails. - * - * - * @throws Exception - */ - public function upload(string $source, string $path, int $chunk = 1, int $chunks = 1, array &$metadata = []): int - { - $contentType = mime_content_type($source) ?: ''; - $this->prepareUpload($path, $contentType, $chunks, $metadata); - $chunksReceived = $this->uploadChunk($source, $path, $chunk, $chunks, $metadata); - - if ($chunks > 1 && $chunks === $chunksReceived && ! $this->finalizeUpload($path, $chunks, $metadata)) { - throw new Exception('Failed to finalize upload ' . $path); - } - - return $chunksReceived; - } - public function prepareUpload(string $path, string $contentType, int $chunks = 1, array &$metadata = []): void { $metadata['parts'] ??= []; $metadata['chunks'] ??= 0; $metadata['content_type'] ??= $contentType; - if ($chunks === 1 || ! empty($metadata['uploadId'])) { + if ($chunks === 1 || isset($metadata['uploadId']) && !\in_array($metadata['uploadId'], ['', '0', [], 0], true)) { return; } $metadata['uploadId'] = $this->createMultipartUpload($path, $contentType); } - public function uploadChunk(string $source, string $path, int $chunk = 1, int $chunks = 1, array &$metadata = []): int - { - $data = file_get_contents($source); - if ($data === false) { - throw new Exception('Can\'t read file ' . $source); - } - - return $this->uploadChunkData($data, $path, $metadata['content_type'] ?? (mime_content_type($source) ?: ''), $chunk, $chunks, $metadata); - } - public function finalizeUpload(string $path, int $chunks = 1, array &$metadata = []): bool { if ($this->exists($path)) { @@ -191,13 +110,13 @@ public function finalizeUpload(string $path, int $chunks = 1, array &$metadata = } if (empty($metadata['uploadId'])) { - throw new Exception('Missing multipart upload ID'); + throw new UploadException('Missing multipart upload ID'); } $metadata['parts'] ??= []; - for ($i = 1; $i <= $chunks; $i++) { + for ($i = 1; $i <= $chunks; ++$i) { if (! \array_key_exists($i, $metadata['parts'])) { - throw new Exception('Missing chunk ' . $i); + throw new UploadException('Missing chunk ' . $i); } } @@ -207,28 +126,12 @@ public function finalizeUpload(string $path, int $chunks = 1, array &$metadata = } /** - * Upload Data. - * - * Upload file contents to desired destination in the selected disk. - * return number of chunks uploaded or 0 if it fails. - * - * - * @throws Exception + * @param UploadMetadata $metadata */ - public function uploadData(string $data, string $path, string $contentType, int $chunk = 1, int $chunks = 1, array &$metadata = []): int + public function uploadChunk(string $data, string $path, int $chunk = 1, int $chunks = 1, array &$metadata = []): int { - $this->prepareUpload($path, $contentType, $chunks, $metadata); - $chunksReceived = $this->uploadChunkData($data, $path, $contentType, $chunk, $chunks, $metadata); + $contentType = $metadata['content_type'] ?? ''; - if ($chunks > 1 && $chunks === $chunksReceived && ! $this->finalizeUpload($path, $chunks, $metadata)) { - throw new Exception('Failed to finalize upload ' . $path); - } - - return $chunksReceived; - } - - private function uploadChunkData(string $data, string $path, string $contentType, int $chunk = 1, int $chunks = 1, array &$metadata = []): int - { if ($chunk === 1 && $chunks === 1) { $this->write($path, $data, $contentType); $metadata['parts'][$chunk] = true; @@ -238,7 +141,7 @@ private function uploadChunkData(string $data, string $path, string $contentType } if (empty($metadata['uploadId'])) { - throw new Exception('Missing multipart upload ID'); + throw new UploadException('Missing multipart upload ID'); } $metadata['parts'] ??= []; @@ -247,7 +150,7 @@ private function uploadChunkData(string $data, string $path, string $contentType $etag = $this->uploadPart($data, $path, $contentType, $chunk, $metadata['uploadId']); // skip incrementing if the chunk was re-uploaded if (! \array_key_exists($chunk, $metadata['parts'])) { - $metadata['chunks']++; + ++$metadata['chunks']; } $metadata['parts'][$chunk] = $etag; @@ -257,8 +160,12 @@ private function uploadChunkData(string $data, string $path, string $contentType /** * Transfer */ - public function transfer(string $path, string $destination, Device $device): bool + public function transfer(string $path, string $destination, Device $device, int $chunkSize = self::TRANSFER_CHUNK_SIZE): bool { + if ($chunkSize <= 0) { + throw new \InvalidArgumentException('Chunk size must be greater than zero'); + } + $response = []; try { $response = $this->getInfo($path); @@ -268,17 +175,17 @@ public function transfer(string $path, string $destination, Device $device): boo $size = (int) ($response['content-length'] ?? 0); $contentType = $response['content-type'] ?? ''; - if ($size <= $this->transferChunkSize) { + if ($size <= $chunkSize) { $source = $this->read($path); return $device->write($destination, $source, $contentType); } - $totalChunks = (int) ceil($size / $this->transferChunkSize); + $totalChunks = (int) ceil($size / $chunkSize); $metadata = ['content_type' => $contentType]; - for ($counter = 0; $counter < $totalChunks; $counter++) { - $start = $counter * $this->transferChunkSize; - $data = $this->read($path, $start, $this->transferChunkSize); + for ($counter = 0; $counter < $totalChunks; ++$counter) { + $start = $counter * $chunkSize; + $data = $this->read($path, $start, $chunkSize); $device->uploadData($data, $destination, $contentType, $counter + 1, $totalChunks, $metadata); } @@ -291,49 +198,60 @@ public function transfer(string $path, string $destination, Device $device): boo * Initiate a multipart upload and return an upload ID. * * - * @throws Exception + * @throws StorageException */ protected function createMultipartUpload(string $path, string $contentType): string { $uri = $path !== '' ? '/' . str_replace(['%2F', '%3F'], ['/', '?'], rawurlencode($path)) : '/'; - $this->headers['content-md5'] = base64_encode(md5('', true)); - unset($this->amzHeaders['x-amz-content-sha256']); - $this->headers['content-type'] = $contentType; - $this->amzHeaders['x-amz-acl'] = $this->acl; - $response = $this->call('s3:createMultipartUpload', self::METHOD_POST, $uri, '', ['uploads' => '']); + $response = $this->call( + Method::POST, + $uri, + '', + ['uploads' => ''], + headers: ['content-type' => $contentType], + amzHeaders: ['x-amz-acl' => $this->acl->value], + ); - return $response->body['UploadId']; + $uploadId = \is_array($response->body) ? ($response->body['UploadId'] ?? null) : null; + if (! \is_string($uploadId)) { + throw new RemoteException('Missing upload ID in S3 response'); + } + + return $uploadId; } /** * Upload Part * * - * @throws Exception + * @throws StorageException */ protected function uploadPart(string $data, string $path, string $contentType, int $chunk, string $uploadId): string { $uri = $path !== '' ? '/' . str_replace(['%2F', '%3F'], ['/', '?'], rawurlencode($path)) : '/'; - $this->headers['content-type'] = $contentType; - $this->headers['content-md5'] = base64_encode(md5($data, true)); - $this->amzHeaders['x-amz-content-sha256'] = hash('sha256', $data); - unset($this->amzHeaders['x-amz-acl']); // ACL header is not allowed in parts, only createMultipartUpload accepts this header. - - $response = $this->call('s3:uploadPart', self::METHOD_PUT, $uri, $data, [ - 'partNumber' => $chunk, - 'uploadId' => $uploadId, - ]); + // ACL header is not allowed in parts, only createMultipartUpload accepts this header. + $response = $this->call( + Method::PUT, + $uri, + $data, + [ + 'partNumber' => $chunk, + 'uploadId' => $uploadId, + ], + headers: ['content-type' => $contentType], + ); - return $response->headers['etag']; + return $response->headers['etag'] ?? throw new RemoteException('Missing ETag in S3 response'); } /** * Complete Multipart Upload * + * @param array $parts * - * @throws Exception + * @throws StorageException */ protected function completeMultipartUpload(string $path, string $uploadId, array $parts): bool { @@ -343,14 +261,20 @@ protected function completeMultipartUpload(string $path, string $uploadId, array $body = ''; foreach ($parts as $key => $etag) { + if (! \is_string($etag)) { + throw new UploadException('Missing ETag for part ' . $key); + } $body .= "{$etag}{$key}"; } $body .= ''; - $this->headers['content-type'] = 'application/xml'; - $this->amzHeaders['x-amz-content-sha256'] = hash('sha256', $body); - $this->headers['content-md5'] = base64_encode(md5($body, true)); - $this->call('s3:completeMultipartUpload', self::METHOD_POST, $uri, $body, ['uploadId' => $uploadId]); + $this->call( + Method::POST, + $uri, + $body, + ['uploadId' => $uploadId], + headers: ['content-type' => 'application/xml'], + ); return true; } @@ -359,14 +283,12 @@ protected function completeMultipartUpload(string $path, string $uploadId, array * Abort Chunked Upload * * - * @throws Exception + * @throws StorageException */ - public function abort(string $path, string $extra = ''): bool + public function abort(string $path, string $uploadId = ''): bool { $uri = $path !== '' ? '/' . str_replace(['%2F', '%3F'], ['/', '?'], rawurlencode($path)) : '/'; - unset($this->headers['content-type']); - $this->headers['content-md5'] = base64_encode(md5('', true)); - $this->call('s3:abort', self::METHOD_DELETE, $uri, '', ['uploadId' => $extra]); + $this->call(Method::DELETE, $uri, '', ['uploadId' => $uploadId]); return true; } @@ -375,20 +297,22 @@ public function abort(string $path, string $extra = ''): bool * Read file or part of file by given path, offset and length. * * - * @throws Exception + * @throws StorageException */ public function read(string $path, int $offset = 0, ?int $length = null): string { - unset($this->amzHeaders['x-amz-acl']); - unset($this->amzHeaders['x-amz-content-sha256']); - unset($this->headers['content-type']); - $this->headers['content-md5'] = base64_encode(md5('', true)); $uri = ($path !== '') ? '/' . str_replace('%2F', '/', rawurlencode($path)) : '/'; + + $headers = []; if ($length !== null) { $end = $offset + $length - 1; - $this->headers['range'] = "bytes=$offset-$end"; + $headers['range'] = "bytes=$offset-$end"; + } + $response = $this->call(Method::GET, $uri, headers: $headers, decode: false); + + if (! \is_string($response->body)) { + throw new RemoteException('Unexpected S3 read response'); } - $response = $this->call('s3:read', self::METHOD_GET, $uri, decode: false); return $response->body; } @@ -397,18 +321,19 @@ public function read(string $path, int $offset = 0, ?int $length = null): string * Write file by given path. * * - * @throws Exception + * @throws StorageException */ public function write(string $path, string $data, string $contentType = ''): bool { $uri = $path !== '' ? '/' . str_replace(['%2F', '%3F'], ['/', '?'], rawurlencode($path)) : '/'; - $this->headers['content-type'] = $contentType; - $this->headers['content-md5'] = base64_encode(md5($data, true)); // TODO whould this work well with big file? can we skip it? - $this->amzHeaders['x-amz-content-sha256'] = hash('sha256', $data); - $this->amzHeaders['x-amz-acl'] = $this->acl; - - $this->call('s3:write', self::METHOD_PUT, $uri, $data); + $this->call( + Method::PUT, + $uri, + $data, + headers: ['content-type' => $contentType], + amzHeaders: ['x-amz-acl' => $this->acl->value], + ); return true; } @@ -418,17 +343,13 @@ public function write(string $path, string $data, string $contentType = ''): boo * * @see http://php.net/manual/en/function.filesize.php * - * @throws Exception + * @throws StorageException */ public function delete(string $path, bool $recursive = false): bool { $uri = ($path !== '') ? '/' . str_replace('%2F', '/', rawurlencode($path)) : '/'; - unset($this->headers['content-type']); - unset($this->amzHeaders['x-amz-acl']); - unset($this->amzHeaders['x-amz-content-sha256']); - $this->headers['content-md5'] = base64_encode(md5('', true)); - $this->call('s3:delete', self::METHOD_DELETE, $uri); + $this->call(Method::DELETE, $uri); return true; } @@ -436,22 +357,18 @@ public function delete(string $path, bool $recursive = false): bool /** * Get list of objects in the given path. * + * @return array * - * @throws Exception + * @throws StorageException */ protected function listObjects(string $prefix = '', int $maxKeys = self::MAX_PAGE_SIZE, string $continuationToken = ''): array { if ($maxKeys > self::MAX_PAGE_SIZE) { - throw new Exception('Cannot list more than ' . self::MAX_PAGE_SIZE . ' objects'); + throw new \InvalidArgumentException('Cannot list more than ' . self::MAX_PAGE_SIZE . ' objects'); } $uri = '/'; $prefix = ltrim($prefix, '/'); /** S3 specific requirement that prefix should never contain a leading slash */ - $this->headers['content-type'] = 'text/plain'; - $this->headers['content-md5'] = base64_encode(md5('', true)); - - unset($this->amzHeaders['x-amz-content-sha256']); - unset($this->amzHeaders['x-amz-acl']); $parameters = [ 'list-type' => 2, @@ -463,7 +380,11 @@ protected function listObjects(string $prefix = '', int $maxKeys = self::MAX_PAG $parameters['continuation-token'] = $continuationToken; } - $response = $this->call('s3:list', self::METHOD_GET, $uri, '', $parameters); + $response = $this->call(Method::GET, $uri, '', $parameters, headers: ['content-type' => 'text/plain']); + + if (! \is_array($response->body)) { + throw new RemoteException('Unexpected S3 list response'); + } return $response->body; } @@ -472,7 +393,7 @@ protected function listObjects(string $prefix = '', int $maxKeys = self::MAX_PAG * Delete files in given path, path must be a directory. Return true on success and false on failure. * * - * @throws Exception + * @throws StorageException */ public function deletePath(string $path): bool { @@ -482,25 +403,33 @@ public function deletePath(string $path): bool $continuationToken = ''; do { $objects = $this->listObjects($path, continuationToken: $continuationToken); - $count = (int) ($objects['KeyCount'] ?? 1); - if ($count < 1) { + $token = $objects['NextContinuationToken'] ?? ''; + $continuationToken = \is_string($token) ? $token : ''; + + // A single object is returned as one associative entry, multiple objects as a list of them. + $contents = $objects['Contents'] ?? []; + $entries = \is_array($contents) ? (isset($contents['Key']) ? [$contents] : $contents) : []; + + $keys = []; + foreach ($entries as $object) { + $key = \is_array($object) ? ($object['Key'] ?? null) : null; + if (\is_string($key)) { + $keys[] = $key; + } + } + + if ($keys === []) { break; } - $continuationToken = $objects['NextContinuationToken'] ?? ''; + $body = ''; - if ($count > 1) { - foreach ($objects['Contents'] as $object) { - $body .= "{$object['Key']}"; - } - } else { - $body .= "{$objects['Contents']['Key']}"; + foreach ($keys as $key) { + $body .= "{$key}"; } $body .= 'true'; $body .= ''; - $this->amzHeaders['x-amz-content-sha256'] = hash('sha256', $body); - $this->headers['content-md5'] = base64_encode(md5($body, true)); - $this->call('s3:deletePath', self::METHOD_POST, $uri, $body, ['delete' => '']); - } while (! empty($continuationToken)); + $this->call(Method::POST, $uri, $body, ['delete' => ''], headers: ['content-type' => 'application/xml']); + } while ($continuationToken !== ''); return true; } @@ -552,90 +481,62 @@ public function getFileHash(string $path): string { $etag = $this->getInfo($path)['etag'] ?? ''; - return (empty($etag)) ? $etag : substr((string) $etag, 1, -1); + return $etag === '' ? $etag : substr($etag, 1, -1); } /** - * Create a directory at the specified path. + * List objects under the given prefix, one page at a time. * - * Returns true on success or if the directory already exists and false on error - */ - public function createDirectory(string $path): bool - { - /* S3 is an object store and does not have the concept of directories */ - return true; - } - - /** - * Get directory size in bytes. + * The cursor is the S3 continuation token. * - * Return -1 on error - * - * Based on http://www.jonasjohn.de/snippets/php/dir-size.htm + * @throws StorageException */ - public function getDirectorySize(string $path): int + public function listFiles(string $prefix = '', int $max = self::MAX_PAGE_SIZE, ?string $cursor = null): FileList { - return -1; - } + $data = $this->listObjects($prefix, $max, $cursor ?? ''); - /** - * Get Partition Free Space. - * - * disk_free_space — Returns available space on filesystem or disk partition - */ - public function getPartitionFreeSpace(): float - { - return -1; - } - - /** - * Get Partition Total Space. - * - * disk_total_space — Returns the total size of a filesystem or disk partition - */ - public function getPartitionTotalSpace(): float - { - return -1; - } - - /** - * Get all files and directories inside a directory. - * - * @param string $dir Directory to scan - * @return array - * - * @throws Exception - */ - public function getFiles(string $dir, int $max = self::MAX_PAGE_SIZE, string $continuationToken = ''): array - { - $data = $this->listObjects($dir, $max, $continuationToken); + // A single object is returned as one associative entry, multiple objects as a list of them. + $contents = $data['Contents'] ?? []; + $entries = \is_array($contents) ? (isset($contents['Key']) ? [$contents] : $contents) : []; - // Set to false if all the results were returned. Set to true if more keys are available to return. - $data['IsTruncated'] = $data['IsTruncated'] === 'true'; - - // KeyCount is the number of keys returned with this request. - $data['KeyCount'] = \intval($data['KeyCount']); + $files = []; + foreach ($entries as $object) { + if (! \is_array($object)) { + continue; + } + if (! \is_string($object['Key'] ?? null)) { + continue; + } + $size = $object['Size'] ?? null; + $modified = $object['LastModified'] ?? null; + $etag = $object['ETag'] ?? null; + $files[] = new FileInfo( + path: $object['Key'], + size: is_numeric($size) ? (int) $size : 0, + modifiedAt: \is_string($modified) ? new \DateTimeImmutable($modified) : null, + etag: \is_string($etag) ? trim($etag, '"') : null, + ); + } - // Sets the maximum number of keys returned to the response. By default, the action returns up to 1,000 key names. - $data['MaxKeys'] = \intval($data['MaxKeys']); + $token = $data['NextContinuationToken'] ?? null; - return $data; + return new FileList( + files: $files, + cursor: ($data['IsTruncated'] ?? null) === 'true' && \is_string($token) ? $token : null, + ); } /** * Get file info * + * @return array * - * @throws Exception + * @throws StorageException */ private function getInfo(string $path): array { - unset($this->headers['content-type']); - unset($this->amzHeaders['x-amz-acl']); - unset($this->amzHeaders['x-amz-content-sha256']); - $this->headers['content-md5'] = base64_encode(md5('', true)); $uri = $path !== '' ? '/' . str_replace('%2F', '/', rawurlencode($path)) : '/'; - $response = $this->call('s3:info', self::METHOD_HEAD, $uri); + $response = $this->call(Method::HEAD, $uri); return $response->headers; } @@ -643,8 +544,12 @@ private function getInfo(string $path): array /** * Generate the headers for AWS Signature V4 * + * @param non-empty-string $method + * @param array $parameters + * @param array $headers + * @param array $amzHeaders */ - private function getSignatureV4(string $method, string $uri, array $parameters = []): string + private function getSignatureV4(string $method, string $uri, array $parameters, array $headers, array $amzHeaders): string { $service = 's3'; $region = $this->region; @@ -652,15 +557,15 @@ private function getSignatureV4(string $method, string $uri, array $parameters = $algorithm = 'AWS4-HMAC-SHA256'; $combinedHeaders = []; - $amzDateStamp = substr((string) $this->amzHeaders['x-amz-date'], 0, 8); + $amzDateStamp = substr($amzHeaders['x-amz-date'] ?? '', 0, 8); // CanonicalHeaders - foreach ($this->headers as $k => $v) { - $combinedHeaders[strtolower((string) $k)] = trim((string) $v); + foreach ($headers as $k => $v) { + $combinedHeaders[strtolower($k)] = trim($v); } - foreach ($this->amzHeaders as $k => $v) { - $combinedHeaders[strtolower((string) $k)] = trim((string) $v); + foreach ($amzHeaders as $k => $v) { + $combinedHeaders[strtolower($k)] = trim($v); } uksort($combinedHeaders, $this->sortMetaHeadersCmp(...)); @@ -683,7 +588,7 @@ private function getSignatureV4(string $method, string $uri, array $parameters = $amzPayload[] = ''; // add a blank entry so we end up with an extra line break $amzPayload[] = implode(';', array_keys($combinedHeaders)); // SignedHeaders - $amzPayload[] = $this->amzHeaders['x-amz-content-sha256']; // payload hash + $amzPayload[] = $amzHeaders['x-amz-content-sha256'] ?? ''; // payload hash $amzPayloadStr = implode("\n", $amzPayload); // request as string @@ -693,7 +598,7 @@ private function getSignatureV4(string $method, string $uri, array $parameters = // stringToSign $stringToSignStr = implode("\n", [ $algorithm, - $this->amzHeaders['x-amz-date'], + $amzHeaders['x-amz-date'] ?? '', implode('/', $credentialScope), hash('sha256', $amzPayloadStr), ]); @@ -715,194 +620,114 @@ private function getSignatureV4(string $method, string $uri, array $parameters = } /** - * Get the S3 response + * Execute a signed S3 request and return its response. * + * Headers are built per call — the instance holds no request state, so a + * single device is safe to share across concurrent coroutines. * - * @throws Exception + * @param non-empty-string $method + * @param array $parameters + * @param array $headers + * @param array $amzHeaders + * + * @throws StorageException */ - protected function call(string $operation, string $method, string $uri, string $data = '', array $parameters = [], bool $decode = true): \stdClass + protected function call(string $method, string $uri, string $data = '', array $parameters = [], array $headers = [], array $amzHeaders = [], bool $decode = true): S3\Response { - $startTime = microtime(true); - $uri = $this->getAbsolutePath($uri); $url = $this->fqdn . $uri . '?' . http_build_query($parameters, '', '&', PHP_QUERY_RFC3986); - $response = new \stdClass(); - $response->body = ''; - $response->headers = []; - // Basic setup - $curl = curl_init(); - curl_setopt($curl, CURLOPT_USERAGENT, 'utopia-php/storage'); - curl_setopt($curl, CURLOPT_URL, $url); + $headers = array_filter($headers, fn(string $value): bool => $value !== ''); + $headers['host'] = $this->host; + $headers['date'] = gmdate('D, d M Y H:i:s T'); + $headers['content-md5'] = base64_encode(md5($data, true)); - // Headers - $httpHeaders = []; - $this->amzHeaders['x-amz-date'] = gmdate('Ymd\THis\Z'); + $amzHeaders = array_filter($amzHeaders, fn(string $value): bool => $value !== ''); + $amzHeaders['x-amz-date'] = gmdate('Ymd\THis\Z'); + $amzHeaders['x-amz-content-sha256'] = hash('sha256', $data); - if (! isset($this->amzHeaders['x-amz-content-sha256'])) { - $this->amzHeaders['x-amz-content-sha256'] = hash('sha256', $data); + $request = new Request($method, Uri::parse($url), new Stream($data)); + foreach ([...$amzHeaders, ...$headers] as $header => $value) { + $request = $request->withHeader($header, $value); } + $request = $request + ->withHeader('authorization', $this->getSignatureV4($method, $uri, $parameters, $headers, $amzHeaders)) + ->withHeader('user-agent', 'utopia-php/storage'); - foreach ($this->amzHeaders as $header => $value) { - if ((string) $value !== '') { - $httpHeaders[] = $header . ': ' . $value; - } + try { + $response = $this->client->sendRequest($request); + } catch (ClientExceptionInterface $e) { + throw new TransportException($e->getMessage(), $e->getCode(), $e); } - $this->headers['date'] = gmdate('D, d M Y H:i:s T'); + $code = $response->getStatusCode(); + $responseBody = (string) $response->getBody(); - foreach ($this->headers as $header => $value) { - if ((string) $value !== '') { - $httpHeaders[] = $header . ': ' . $value; - } + if ($code >= 400) { + $this->parseAndThrowS3Error($responseBody, $code); } - $httpHeaders[] = 'Authorization: ' . $this->getSignatureV4($method, $uri, $parameters); - - curl_setopt($curl, CURLOPT_HTTPHEADER, $httpHeaders); - curl_setopt($curl, CURLOPT_HEADER, false); - curl_setopt($curl, CURLOPT_RETURNTRANSFER, false); - - if ($this->curlHttpVersion != null) { - curl_setopt($curl, CURLOPT_HTTP_VERSION, $this->curlHttpVersion); + $responseHeaders = []; + foreach ($response->getHeaders() as $name => $values) { + $responseHeaders[strtolower((string) $name)] = implode(', ', $values); } - curl_setopt($curl, CURLOPT_WRITEFUNCTION, function ($curl, string $data) use ($response): int { - $response->body .= $data; - - return \strlen($data); - }); - curl_setopt($curl, CURLOPT_HEADERFUNCTION, function ($curl, string $header) use (&$response): int { - $len = \strlen($header); - $header = explode(':', $header, 2); - - if (\count($header) < 2) { // ignore invalid headers - return $len; - } - - $response->headers[strtolower(trim($header[0]))] = trim($header[1]); + $contentType = $responseHeaders['content-type'] ?? ''; + $isXml = $contentType === 'application/xml' || (str_starts_with($responseBody, 'code = curl_getinfo($curl, CURLINFO_HTTP_CODE); - - $attempt = 0; - while ( - $attempt < self::$retryAttempts - && $this->isTransientError($response->code, $response->body) - ) { - usleep(self::$retryDelay * 1000); - $attempt++; - $result = curl_exec($curl); - $response->code = curl_getinfo($curl, CURLINFO_HTTP_CODE); - } - - try { - if (! $result) { - throw new Exception(curl_error($curl)); - } - - if ($response->code >= 400) { - $this->parseAndThrowS3Error($response->body, $response->code); - } - - // Parse body into XML - if ($decode && ((isset($response->headers['content-type']) && $response->headers['content-type'] == 'application/xml') || (str_starts_with((string) $response->body, 'headers['content-type'] ?? '') !== 'image/svg+xml'))) { - $response->body = simplexml_load_string((string) $response->body); - $response->body = json_decode(json_encode($response->body), true); - } - - return $response; - } finally { - - $this->storageOperationTelemetry->record( - microtime(true) - $startTime, - [ - 'storage' => $this->getType(), - 'operation' => $operation, - 'attempts' => $attempt, - ], - ); - } + return new S3\Response( + code: $code, + headers: $responseHeaders, + body: $decode && $isXml ? $this->decodeXml($responseBody) : $responseBody, + ); } /** - * Parse S3 XML error response and throw appropriate exception + * Decode an XML response body into an associative array. * - * @param string $errorBody The error response body - * @param int $statusCode The HTTP status code + * @return array * - * @throws NotFoundException When the error is NoSuchKey - * @throws Exception For other S3 errors + * @throws StorageException */ - private function parseAndThrowS3Error(string $errorBody, int $statusCode): void + private function decodeXml(string $body): array { - if (str_starts_with($errorBody, 'Code ?? ''); - $errorMessage = (string) ($xml->Message ?? ''); - - if ($errorCode === 'NoSuchKey') { - throw new NotFoundException($errorMessage ?: 'File not found', $statusCode); - } - } catch (NotFoundException $e) { - throw $e; - } catch (\Throwable) { - // If XML parsing fails, fall through to original error - } + $xml = simplexml_load_string($body); + $encoded = $xml === false ? false : json_encode($xml); + $decoded = $encoded === false ? null : json_decode($encoded, true); + if (! \is_array($decoded)) { + throw new RemoteException('Failed to decode S3 XML response'); } - throw new Exception($errorBody, $statusCode); + return $decoded; } /** - * Determine whether an S3 response indicates a transient rate-limiting error - * (e.g. SlowDown, ServiceUnavailable) that should be retried with exponential backoff. + * Parse an S3 error response and throw the matching exception. * - * The XML body is parsed first so that specific S3 error codes are detected regardless - * of HTTP status. A 503/429 with a parseable but non-transient error code is NOT retried. - * Unparseable 429/503 responses fall back to status-code detection. + * @param string $errorBody The error response body + * @param int $statusCode The HTTP status code * - * @param int $statusCode HTTP response status code - * @param string $body Response body + * @throws NotFoundException When the object does not exist (404, or a NoSuchKey error code) + * @throws RemoteException For every other error response */ - protected function isTransientError(int $statusCode, string $body): bool + private function parseAndThrowS3Error(string $errorBody, int $statusCode): never { - $trimmed = ltrim($body); - if (str_starts_with($trimmed, 'Code ?? ''); - if (\in_array($code, ['SlowDown', 'ServiceUnavailable', 'Throttling', 'RequestThrottled'], true)) { - return true; - } - // Successfully parsed XML with a non-transient error code — do not retry. - if ($code !== '') { - return false; - } + $errorCode = (string) ($xml->Code ?? '') ?: null; + $errorMessage = (string) ($xml->Message ?? '') ?: null; } } - // Fall back to HTTP status code for responses that cannot be parsed as XML. - return $statusCode === 429 || $statusCode === 503; + // HEAD error responses carry no body, so the status code is the only signal. + if ($statusCode === 404 || $errorCode === 'NoSuchKey') { + throw new NotFoundException($errorMessage ?? 'File not found', $statusCode); + } + + throw new RemoteException($errorMessage ?? ($errorBody !== '' ? $errorBody : 'S3 request failed'), $statusCode, $errorCode); } /** diff --git a/packages/storage/src/Storage/Device/S3/Response.php b/packages/storage/src/Storage/Device/S3/Response.php new file mode 100644 index 000000000..07383d1b8 --- /dev/null +++ b/packages/storage/src/Storage/Device/S3/Response.php @@ -0,0 +1,18 @@ + $headers + * @param string|array $body Raw body, or the decoded XML document + */ + public function __construct( + public int $code, + public array $headers, + public string|array $body, + ) {} +} diff --git a/packages/storage/src/Storage/Device/S3/RetryStrategy.php b/packages/storage/src/Storage/Device/S3/RetryStrategy.php new file mode 100644 index 000000000..172668893 --- /dev/null +++ b/packages/storage/src/Storage/Device/S3/RetryStrategy.php @@ -0,0 +1,74 @@ + + */ + private const array TRANSIENT_ERROR_CODES = ['SlowDown', 'ServiceUnavailable', 'Throttling', 'RequestThrottled']; + + /** + * @var array + */ + private const array TRANSIENT_STATUS_CODES = [429, 503]; + + /** + * @param int $retries Retries after the initial attempt + * @param float $delay Delay between attempts in seconds + */ + public function __construct( + private int $retries = 3, + private float $delay = 0.5, + ) {} + + public function delay(RequestInterface $request, int $attempt, ?ResponseInterface $response, ?ClientExceptionInterface $error): ?float + { + if ($attempt > $this->retries || ! $response instanceof ResponseInterface) { + return null; + } + + return $this->isTransient($response) ? $this->delay : null; + } + + private function isTransient(ResponseInterface $response): bool + { + $body = (string) $response->getBody(); + + $trimmed = ltrim($body); + if (str_starts_with($trimmed, 'Code ?? ''); + if (\in_array($code, self::TRANSIENT_ERROR_CODES, true)) { + return true; + } + // Successfully parsed XML with a non-transient error code — do not retry. + if ($code !== '') { + return false; + } + } + } + + // Fall back to HTTP status code for responses that cannot be parsed as XML. + return \in_array($response->getStatusCode(), self::TRANSIENT_STATUS_CODES, true); + } +} diff --git a/packages/storage/src/Storage/Device/Telemetry.php b/packages/storage/src/Storage/Device/Telemetry.php index 8e7270f46..343ce40e1 100644 --- a/packages/storage/src/Storage/Device/Telemetry.php +++ b/packages/storage/src/Storage/Device/Telemetry.php @@ -1,162 +1,171 @@ telemetry = Histogram::lazy( + telemetry: $telemetry, + name: 'storage.operation', + unit: 's', + advisory: ['ExplicitBucketBoundaries' => [0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10]], + ); } - #[\Override] - public function setTelemetry(Adapter $telemetry): void + /** + * The decorated device, for access to adapter-specific methods that are + * not part of the base contract (for example `Local` partition metrics or + * the `S3` object listing). + */ + public function getDevice(): Device { - parent::setTelemetry($telemetry); - $this->underlying->setTelemetry($telemetry); + return $this->device; } - private function measure(string $method, &...$args): mixed + /** + * @template T + * + * @param callable(): T $operation + * @return T + */ + private function measure(string $method, callable $operation): mixed { $start = microtime(true); try { - return $this->underlying->{$method}(...$args); + return $operation(); } finally { - $this->storageOperationTelemetry->record( + $this->telemetry->record( microtime(true) - $start, [ - 'storage' => $this->underlying->getType(), + 'storage' => $this->device->getType()->value, 'operation' => "device:$method", ], ); } } - public function getName(): string - { - return $this->underlying->getName(); - } - - public function getType(): string - { - return $this->underlying->getType(); - } - - public function getDescription(): string + public function getType(): DeviceType { - return $this->underlying->getDescription(); + return $this->device->getType(); } public function getRoot(): string { - return $this->underlying->getRoot(); + return $this->device->getRoot(); } - public function getPath(string $filename, ?string $prefix = null): string + public function getPath(string $filename): string { - return $this->measure(__FUNCTION__, $filename, $prefix); - } - - public function upload(string $source, string $path, int $chunk = 1, int $chunks = 1, array &$metadata = []): int - { - return $this->measure(__FUNCTION__, $source, $path, $chunk, $chunks, $metadata); + return $this->measure(__FUNCTION__, fn(): string => $this->device->getPath($filename)); } + /** + * @param UploadMetadata $metadata + */ public function prepareUpload(string $path, string $contentType, int $chunks = 1, array &$metadata = []): void { - $this->measure(__FUNCTION__, $path, $contentType, $chunks, $metadata); + $this->measure(__FUNCTION__, function () use ($path, $contentType, $chunks, &$metadata): void { + $this->device->prepareUpload($path, $contentType, $chunks, $metadata); + }); } - public function uploadChunk(string $source, string $path, int $chunk = 1, int $chunks = 1, array &$metadata = []): int + /** + * @param UploadMetadata $metadata + */ + public function uploadChunk(string $data, string $path, int $chunk = 1, int $chunks = 1, array &$metadata = []): int { - return $this->measure(__FUNCTION__, $source, $path, $chunk, $chunks, $metadata); + return $this->measure(__FUNCTION__, function () use ($data, $path, $chunk, $chunks, &$metadata): int { + return $this->device->uploadChunk($data, $path, $chunk, $chunks, $metadata); + }); } + /** + * @param UploadMetadata $metadata + */ public function finalizeUpload(string $path, int $chunks = 1, array &$metadata = []): bool { - return $this->measure(__FUNCTION__, $path, $chunks, $metadata); - } - - public function uploadData(string $data, string $path, string $contentType, int $chunk = 1, int $chunks = 1, array &$metadata = []): int - { - return $this->measure(__FUNCTION__, $data, $path, $contentType, $chunk, $chunks, $metadata); + return $this->measure(__FUNCTION__, function () use ($path, $chunks, &$metadata): bool { + return $this->device->finalizeUpload($path, $chunks, $metadata); + }); } - public function abort(string $path, string $extra = ''): bool + public function abort(string $path, string $uploadId = ''): bool { - return $this->measure(__FUNCTION__, $path, $extra); + return $this->measure(__FUNCTION__, fn(): bool => $this->device->abort($path, $uploadId)); } + /** + * @param int<0, max> $offset + * @param int<0, max>|null $length + */ public function read(string $path, int $offset = 0, ?int $length = null): string { - return $this->measure(__FUNCTION__, $path, $offset, $length); + return $this->measure(__FUNCTION__, fn(): string => $this->device->read($path, $offset, $length)); } - public function transfer(string $path, string $destination, Device $device): bool + public function transfer(string $path, string $destination, Device $device, int $chunkSize = self::TRANSFER_CHUNK_SIZE): bool { - return $this->measure(__FUNCTION__, $path, $destination, $device); + return $this->measure(__FUNCTION__, fn(): bool => $this->device->transfer($path, $destination, $device, $chunkSize)); } public function write(string $path, string $data, string $contentType): bool { - return $this->measure(__FUNCTION__, $path, $data, $contentType); + return $this->measure(__FUNCTION__, fn(): bool => $this->device->write($path, $data, $contentType)); } public function delete(string $path, bool $recursive = false): bool { - return $this->measure(__FUNCTION__, $path, $recursive); + return $this->measure(__FUNCTION__, fn(): bool => $this->device->delete($path, $recursive)); } public function deletePath(string $path): bool { - return $this->measure(__FUNCTION__, $path); + return $this->measure(__FUNCTION__, fn(): bool => $this->device->deletePath($path)); } public function exists(string $path): bool { - return $this->measure(__FUNCTION__, $path); + return $this->measure(__FUNCTION__, fn(): bool => $this->device->exists($path)); } - public function getFileSize(string $path): int + /** + * @param int<1, max> $max + */ + public function listFiles(string $prefix = '', int $max = 1000, ?string $cursor = null): FileList { - return $this->measure(__FUNCTION__, $path); + return $this->measure(__FUNCTION__, fn(): FileList => $this->device->listFiles($prefix, $max, $cursor)); } - public function getFileMimeType(string $path): string - { - return $this->measure(__FUNCTION__, $path); - } - - public function getFileHash(string $path): string - { - return $this->measure(__FUNCTION__, $path); - } - - public function createDirectory(string $path): bool - { - return $this->measure(__FUNCTION__, $path); - } - - public function getDirectorySize(string $path): int - { - return $this->measure(__FUNCTION__, $path); - } - - public function getPartitionFreeSpace(): float + public function getFileSize(string $path): int { - return $this->measure(__FUNCTION__); + return $this->measure(__FUNCTION__, fn(): int => $this->device->getFileSize($path)); } - public function getPartitionTotalSpace(): float + public function getFileMimeType(string $path): string { - return $this->measure(__FUNCTION__); + return $this->measure(__FUNCTION__, fn(): string => $this->device->getFileMimeType($path)); } - public function getFiles(string $dir, int $max = self::MAX_PAGE_SIZE, string $continuationToken = ''): array + public function getFileHash(string $path): string { - return $this->measure(__FUNCTION__, $dir, $max, $continuationToken); + return $this->measure(__FUNCTION__, fn(): string => $this->device->getFileHash($path)); } } diff --git a/packages/storage/src/Storage/Device/Wasabi.php b/packages/storage/src/Storage/Device/Wasabi.php index eebefe29b..cb3bfede5 100644 --- a/packages/storage/src/Storage/Device/Wasabi.php +++ b/packages/storage/src/Storage/Device/Wasabi.php @@ -4,7 +4,9 @@ namespace Utopia\Storage\Device; -use Utopia\Storage\Storage; +use Psr\Http\Client\ClientInterface; +use Utopia\Storage\Acl; +use Utopia\Storage\DeviceType; class Wasabi extends S3 { @@ -34,27 +36,23 @@ class Wasabi extends S3 /** * Wasabi Constructor */ - public function __construct(string $root, string $accessKey, string $secretKey, string $bucket, string $region = self::EU_CENTRAL_1, string $acl = self::ACL_PRIVATE) - { + public function __construct( + string $root, + string $accessKey, + #[\SensitiveParameter] + string $secretKey, + string $bucket, + string $region = self::EU_CENTRAL_1, + Acl $acl = Acl::Private, + ?ClientInterface $client = null, + ) { $host = $bucket . '.' . 's3' . '.' . $region . '.' . 'wasabisys' . '.' . 'com'; - parent::__construct($root, $accessKey, $secretKey, $host, $region, $acl); - } - - #[\Override] - public function getName(): string - { - return 'Wasabi Storage'; - } - - #[\Override] - public function getDescription(): string - { - return 'Wasabi Storage'; + parent::__construct($root, $accessKey, $secretKey, $host, $region, $acl, $client); } #[\Override] - public function getType(): string + public function getType(): DeviceType { - return Storage::DEVICE_WASABI; + return DeviceType::Wasabi; } } diff --git a/packages/storage/src/Storage/DeviceType.php b/packages/storage/src/Storage/DeviceType.php new file mode 100644 index 000000000..c28bb3240 --- /dev/null +++ b/packages/storage/src/Storage/DeviceType.php @@ -0,0 +1,22 @@ + $files + */ + public function __construct( + public array $files, + public ?string $cursor = null, + ) {} +} diff --git a/packages/storage/src/Storage/Storage.php b/packages/storage/src/Storage/Storage.php deleted file mode 100644 index f085d06fc..000000000 --- a/packages/storage/src/Storage/Storage.php +++ /dev/null @@ -1,121 +0,0 @@ - [ - 'B', - 'KiB', - 'MiB', - 'GiB', - 'TiB', - 'PiB', - 'EiB', - 'ZiB', - 'YiB', - ], - 'metric' => [ - 'B', - 'kB', - 'MB', - 'GB', - 'TB', - 'PB', - 'EB', - 'ZB', - 'YB', - ], - ]; - - $factor = (int) floor((\strlen((string) $bytes) - 1) / 3); - - return \sprintf("%.{$decimals}f%s", $bytes / $mod ** $factor, $units[$system][$factor]); - } -} diff --git a/packages/storage/src/Storage/Validator/File.php b/packages/storage/src/Storage/Validator/File.php deleted file mode 100644 index 0cb3c3650..000000000 --- a/packages/storage/src/Storage/Validator/File.php +++ /dev/null @@ -1,47 +0,0 @@ - $allowed + */ public function __construct(protected array $allowed) {} /** @@ -35,8 +43,11 @@ public function getDescription(): string */ public function isValid($filename): bool { - $ext = pathinfo((string) $filename, PATHINFO_EXTENSION); - $ext = strtolower($ext); + if (! \is_string($filename)) { + return false; + } + $ext = strtolower(pathinfo($filename, PATHINFO_EXTENSION)); + return \in_array($ext, $this->allowed); } diff --git a/packages/storage/src/Storage/Validator/FileName.php b/packages/storage/src/Storage/Validator/FileName.php index f84fa26e0..2e2e7caf8 100644 --- a/packages/storage/src/Storage/Validator/FileName.php +++ b/packages/storage/src/Storage/Validator/FileName.php @@ -6,6 +6,9 @@ use Utopia\Validator; +/** + * @see \Utopia\Tests\Storage\Validator\FileNameTest + */ class FileName extends Validator { /** diff --git a/packages/storage/src/Storage/Validator/FileSize.php b/packages/storage/src/Storage/Validator/FileSize.php index 6c8bb2b99..5235d7e99 100644 --- a/packages/storage/src/Storage/Validator/FileSize.php +++ b/packages/storage/src/Storage/Validator/FileSize.php @@ -6,6 +6,9 @@ use Utopia\Validator; +/** + * @see \Utopia\Tests\Storage\Validator\FileSizeTest + */ class FileSize extends Validator { /** diff --git a/packages/storage/src/Storage/Validator/FileType.php b/packages/storage/src/Storage/Validator/FileType.php index 4fa422304..5d472f228 100644 --- a/packages/storage/src/Storage/Validator/FileType.php +++ b/packages/storage/src/Storage/Validator/FileType.php @@ -1,10 +1,15 @@ */ protected $types = [ self::FILE_TYPE_JPEG => "\xFF\xD8\xFF", @@ -30,16 +35,21 @@ class FileType extends Validator self::FILE_TYPE_GZIP => 'application/x-gzip', ]; + /** + * @var array + */ protected array $allowed; /** + * @param array $allowed + * * @throws Exception */ public function __construct(array $allowed) { foreach ($allowed as $key) { if (! isset($this->types[$key])) { - throw new Exception('Unknown file mime type'); + throw new \InvalidArgumentException('Unknown file mime type'); } } @@ -65,7 +75,7 @@ public function getDescription(): string */ public function isValid($path): bool { - if (! is_readable($path)) { + if (! \is_string($path) || ! is_readable($path)) { return false; } @@ -76,9 +86,14 @@ public function isValid($path): bool } $bytes = fgets($handle, 8); + if ($bytes === false) { + fclose($handle); + + return false; + } foreach ($this->allowed as $key) { - if (str_starts_with($bytes, (string) $this->types[$key])) { + if (str_starts_with($bytes, $this->types[$key])) { fclose($handle); return true; diff --git a/packages/storage/src/Storage/Validator/Upload.php b/packages/storage/src/Storage/Validator/Upload.php index a47f27ec6..0d8f3541b 100644 --- a/packages/storage/src/Storage/Validator/Upload.php +++ b/packages/storage/src/Storage/Validator/Upload.php @@ -6,6 +6,9 @@ use Utopia\Validator; +/** + * @see \Utopia\Tests\Storage\Validator\UploadTest + */ class Upload extends Validator { /** diff --git a/packages/storage/tests/Storage/S3Base.php b/packages/storage/tests/E2E/S3Base.php similarity index 71% rename from packages/storage/tests/Storage/S3Base.php rename to packages/storage/tests/E2E/S3Base.php index 5259ad9b3..62b8ab876 100644 --- a/packages/storage/tests/Storage/S3Base.php +++ b/packages/storage/tests/E2E/S3Base.php @@ -2,7 +2,7 @@ declare(strict_types=1); -namespace Utopia\Tests\Storage; +namespace Utopia\Tests\Storage\E2E; use PHPUnit\Framework\TestCase; use Utopia\Storage\Device\Local; @@ -11,13 +11,36 @@ abstract class S3Base extends TestCase { - abstract protected function init(): void; + /** + * @return resource + */ + private function openStream(string $path) + { + $handle = fopen($path, 'rb'); + if ($handle === false) { + throw new \RuntimeException('Failed to open ' . $path); + } - abstract protected function getAdapterName(): string; + return $handle; + } - abstract protected function getAdapterType(): string; + /** + * @param resource $handle + * @param int<1, max> $length + */ + private function readBytes($handle, int $length): string + { + $contents = fread($handle, $length); + if ($contents === false) { + throw new \RuntimeException('Failed to read stream'); + } - abstract protected function getAdapterDescription(): string; + return $contents; + } + + abstract protected function init(): void; + + abstract protected function getAdapterType(): \Utopia\Storage\DeviceType; /** * @var S3 @@ -35,12 +58,21 @@ protected function setUp(): void $this->uploadTestFiles(); } + private function uploadFile(string $source, string $path): void + { + $data = file_get_contents($source); + if ($data === false) { + throw new \RuntimeException('Failed to read ' . $source); + } + $this->object->uploadData($data, $path, mime_content_type($source) ?: ''); + } + private function uploadTestFiles(): void { - $this->object->upload(__DIR__ . '/../resources/disk-a/kitten-1.jpg', $this->object->getPath('testing/kitten-1.jpg')); - $this->object->upload(__DIR__ . '/../resources/disk-a/kitten-2.jpg', $this->object->getPath('testing/kitten-2.jpg')); - $this->object->upload(__DIR__ . '/../resources/disk-b/kitten-1.png', $this->object->getPath('testing/kitten-1.png')); - $this->object->upload(__DIR__ . '/../resources/disk-b/kitten-2.png', $this->object->getPath('testing/kitten-2.png')); + $this->uploadFile(__DIR__ . '/../resources/disk-a/kitten-1.jpg', $this->object->getPath('testing/kitten-1.jpg')); + $this->uploadFile(__DIR__ . '/../resources/disk-a/kitten-2.jpg', $this->object->getPath('testing/kitten-2.jpg')); + $this->uploadFile(__DIR__ . '/../resources/disk-b/kitten-1.png', $this->object->getPath('testing/kitten-1.png')); + $this->uploadFile(__DIR__ . '/../resources/disk-b/kitten-2.png', $this->object->getPath('testing/kitten-2.png')); } private function removeTestFiles(): void @@ -56,45 +88,45 @@ protected function tearDown(): void $this->removeTestFiles(); } - public function testGetFiles(): void + public function testListFiles(): void { $path = $this->object->getPath('testing/'); - $files = $this->object->getFiles($path); - $this->assertEquals(4, $files['KeyCount']); - $this->assertEquals(false, $files['IsTruncated']); - $this->assertIsArray($files['Contents']); + $list = $this->object->listFiles($path); - $file = $files['Contents'][0]; + $this->assertCount(4, $list->files); + $this->assertNull($list->cursor); - $this->assertArrayHasKey('Key', $file); - $this->assertArrayHasKey('LastModified', $file); - $this->assertArrayHasKey('ETag', $file); - $this->assertArrayHasKey('StorageClass', $file); - $this->assertArrayHasKey('Size', $file); + $file = $list->files[0]; + $this->assertStringContainsString('testing/kitten', $file->path); + $this->assertGreaterThan(0, $file->size); + $this->assertInstanceOf(\DateTimeImmutable::class, $file->modifiedAt); + $this->assertNotNull($file->etag); } - public function testGetFilesPagination(): void + public function testListFilesPagination(): void { $path = $this->object->getPath('testing/'); - $files = $this->object->getFiles($path, 3); - $this->assertEquals(3, $files['KeyCount']); - $this->assertEquals(3, $files['MaxKeys']); - $this->assertEquals(true, $files['IsTruncated']); - $this->assertIsArray($files['Contents']); - $this->assertArrayHasKey('NextContinuationToken', $files); + $list = $this->object->listFiles($path, 3); + $this->assertCount(3, $list->files); + $this->assertNotNull($list->cursor); - $files = $this->object->getFiles($path, 1000, $files['NextContinuationToken']); - $this->assertEquals(1, $files['KeyCount']); - $this->assertEquals(1000, $files['MaxKeys']); - $this->assertEquals(false, $files['IsTruncated']); - $this->assertIsArray($files['Contents']); - $this->assertArrayNotHasKey('NextContinuationToken', $files); + $list = $this->object->listFiles($path, 1000, $list->cursor); + $this->assertCount(1, $list->files); + $this->assertNull($list->cursor); } - public function testName(): void + public function testListFilesSingleObject(): void { - $this->assertEquals($this->getAdapterName(), $this->object->getName()); + $path = $this->object->getPath('single/'); + $this->object->write($path . 'only.txt', 'one', 'text/plain'); + + $list = $this->object->listFiles($path); + $this->assertCount(1, $list->files); + $this->assertSame(ltrim($path . 'only.txt', '/'), $list->files[0]->path); + $this->assertNull($list->cursor); + + $this->object->delete($path . 'only.txt'); } public function testType(): void @@ -102,11 +134,6 @@ public function testType(): void $this->assertEquals($this->getAdapterType(), $this->object->getType()); } - public function testDescription(): void - { - $this->assertEquals($this->getAdapterDescription(), $this->object->getDescription()); - } - public function testRoot(): void { $this->assertEquals($this->root, $this->object->getRoot()); @@ -170,7 +197,7 @@ public function testDelete(): void public function testSVGUpload(): void { - $this->assertEquals(true, $this->object->upload(__DIR__ . '/../resources/disk-b/appwrite.svg', $this->object->getPath('testing/appwrite.svg'))); + $this->uploadFile(__DIR__ . '/../resources/disk-b/appwrite.svg', $this->object->getPath('testing/appwrite.svg')); $this->assertEquals(file_get_contents(__DIR__ . '/../resources/disk-b/appwrite.svg'), $this->object->read($this->object->getPath('testing/appwrite.svg'))); $this->assertEquals(true, $this->object->exists($this->object->getPath('testing/appwrite.svg'))); $this->assertEquals(true, $this->object->delete($this->object->getPath('testing/appwrite.svg'))); @@ -178,7 +205,7 @@ public function testSVGUpload(): void public function testXMLUpload(): void { - $this->assertEquals(true, $this->object->upload(__DIR__ . '/../resources/disk-a/config.xml', $this->object->getPath('testing/config.xml'))); + $this->uploadFile(__DIR__ . '/../resources/disk-a/config.xml', $this->object->getPath('testing/config.xml')); $this->assertEquals(file_get_contents(__DIR__ . '/../resources/disk-a/config.xml'), $this->object->read($this->object->getPath('testing/config.xml'))); $this->assertEquals(true, $this->object->exists($this->object->getPath('testing/config.xml'))); $this->assertEquals(true, $this->object->delete($this->object->getPath('testing/config.xml'))); @@ -232,27 +259,7 @@ public function testFileHash(): void $this->assertEquals('8a9ed992b77e4b62b10e3a5c8ed72062', $this->object->getFileHash($this->object->getPath('testing/kitten-2.png'))); } - public function testDirectoryCreate(): void - { - $this->assertTrue($this->object->createDirectory('temp')); - } - - public function testDirectorySize(): void - { - $this->assertEquals(-1, $this->object->getDirectorySize('resources/disk-a/')); - } - - public function testPartitionFreeSpace(): void - { - $this->assertEquals(-1, $this->object->getPartitionFreeSpace()); - } - - public function testPartitionTotalSpace(): void - { - $this->assertEquals(-1, $this->object->getPartitionTotalSpace()); - } - - public function testPartUpload() + public function testPartUpload(): string { $source = __DIR__ . '/../resources/disk-a/large_file.mp4'; $dest = $this->object->getPath('uploaded.mp4'); @@ -268,24 +275,17 @@ public function testPartUpload() $metadata = [ 'parts' => [], 'chunks' => 0, - 'uploadId' => null, - 'content_type' => mime_content_type($source), + 'content_type' => mime_content_type($source) ?: '', ]; - $handle = @fopen($source, 'rb'); - $op = __DIR__ . '/chunk.part'; + $handle = $this->openStream($source); while ($start < $totalSize) { - $contents = fread($handle, $chunkSize); - $op = __DIR__ . '/chunk.part'; - $cc = fopen($op, 'wb'); - fwrite($cc, $contents); - fclose($cc); - $this->object->upload($op, $dest, $chunk, $chunks, $metadata); + $contents = $this->readBytes($handle, $chunkSize); + $this->object->uploadData($contents, $dest, $metadata['content_type'] ?? '', $chunk, $chunks, $metadata); $start += \strlen($contents); - $chunk++; + ++$chunk; fseek($handle, $start); } - @fclose($handle); - unlink($op); + fclose($handle); $this->assertEquals(filesize($source), $this->object->getFileSize($dest)); @@ -299,7 +299,7 @@ public function testPartUpload() return $dest; } - public function testPartUploadRetry() + public function testPartUploadRetry(): string { $source = __DIR__ . '/../resources/disk-a/large_file.mp4'; $dest = $this->object->getPath('uploaded.mp4'); @@ -315,43 +315,30 @@ public function testPartUploadRetry() $metadata = [ 'parts' => [], 'chunks' => 0, - 'uploadId' => null, - 'content_type' => mime_content_type($source), + 'content_type' => mime_content_type($source) ?: '', ]; - $handle = @fopen($source, 'rb'); - $op = __DIR__ . '/chunk.part'; + $handle = $this->openStream($source); while ($start < $totalSize) { - $contents = fread($handle, $chunkSize); - $op = __DIR__ . '/chunk.part'; - $cc = fopen($op, 'wb'); - fwrite($cc, $contents); - fclose($cc); - $this->object->upload($op, $dest, $chunk, $chunks, $metadata); + $contents = $this->readBytes($handle, $chunkSize); + $this->object->uploadData($contents, $dest, $metadata['content_type'] ?? '', $chunk, $chunks, $metadata); $start += \strlen($contents); - $chunk++; + ++$chunk; break; } - @fclose($handle); - unlink($op); + fclose($handle); $chunk = 1; $start = 0; // retry from first to make sure duplicate chunk re-upload works without issue - $handle = @fopen($source, 'rb'); - $op = __DIR__ . '/chunk.part'; + $handle = $this->openStream($source); while ($start < $totalSize) { - $contents = fread($handle, $chunkSize); - $op = __DIR__ . '/chunk.part'; - $cc = fopen($op, 'wb'); - fwrite($cc, $contents); - fclose($cc); - $this->object->upload($op, $dest, $chunk, $chunks, $metadata); + $contents = $this->readBytes($handle, $chunkSize); + $this->object->uploadData($contents, $dest, $metadata['content_type'] ?? '', $chunk, $chunks, $metadata); $start += \strlen($contents); - $chunk++; + ++$chunk; fseek($handle, $start); } - @fclose($handle); - unlink($op); + fclose($handle); $this->assertEquals(filesize($source), $this->object->getFileSize($dest)); @@ -365,7 +352,7 @@ public function testPartUploadRetry() return $dest; } - public function testOutOfOrderPartUpload() + public function testOutOfOrderPartUpload(): string { $source = __DIR__ . '/../resources/disk-a/large_file.mp4'; $dest = $this->object->getPath('uploaded-out-of-order.mp4'); @@ -377,30 +364,23 @@ public function testOutOfOrderPartUpload() // Read all chunk contents into memory so we can upload out of order $parts = []; - $handle = @fopen($source, 'rb'); + $handle = $this->openStream($source); $chunkNum = 1; while ($chunkNum <= $chunks) { - $contents = fread($handle, $chunkSize); - $parts[$chunkNum] = $contents; - $chunkNum++; + $parts[$chunkNum] = $this->readBytes($handle, $chunkSize); + ++$chunkNum; } - @fclose($handle); + fclose($handle); $metadata = [ 'parts' => [], 'chunks' => 0, - 'uploadId' => null, - 'content_type' => mime_content_type($source), + 'content_type' => mime_content_type($source) ?: '', ]; // Upload chunks in reverse order - for ($i = $chunks; $i >= 1; $i--) { - $op = __DIR__ . '/chunk.part'; - $cc = fopen($op, 'wb'); - fwrite($cc, $parts[$i]); - fclose($cc); - $this->object->upload($op, $dest, $i, $chunks, $metadata); - unlink($op); + for ($i = $chunks; $i >= 1; --$i) { + $this->object->uploadData($parts[$i], $dest, $metadata['content_type'] ?? '', $i, $chunks, $metadata); } $this->assertEquals(filesize($source), $this->object->getFileSize($dest)); @@ -425,12 +405,10 @@ public function testPartRead(string $path): void public function testTransferLarge(string $path): void { // chunked file - $this->object->setTransferChunkSize(10000000); // 10 mb - $device = new Local(__DIR__ . '/../resources/disk-a'); $destination = $device->getPath('largefile.mp4'); - $this->assertTrue($this->object->transfer($path, $destination, $device)); + $this->assertTrue($this->object->transfer($path, $destination, $device, 10000000)); // 10 mb chunks $this->assertTrue($device->exists($destination)); $this->assertSame('video/mp4', $device->getFileMimeType($destination)); @@ -440,15 +418,13 @@ public function testTransferLarge(string $path): void public function testTransferSmall(): void { - $this->object->setTransferChunkSize(10000000); // 10 mb - $device = new Local(__DIR__ . '/../resources/disk-a'); $path = $this->object->getPath('text-for-read.txt'); $this->object->write($path, 'Hello World', 'text/plain'); $destination = $device->getPath('hello.txt'); - $this->assertTrue($this->object->transfer($path, $destination, $device)); + $this->assertTrue($this->object->transfer($path, $destination, $device, 10000000)); // 10 mb chunks $this->assertTrue($device->exists($destination)); $this->assertSame('Hello World', $device->read($destination)); diff --git a/packages/storage/tests/E2E/S3Test.php b/packages/storage/tests/E2E/S3Test.php new file mode 100644 index 000000000..38a3ec52b --- /dev/null +++ b/packages/storage/tests/E2E/S3Test.php @@ -0,0 +1,35 @@ +root = '/root'; + $host = $this->env('S3_HOST', 'http://utopia-storage-test.localhost:9805'); + $key = $this->env('S3_ACCESS_KEY', 'minioadmin'); + $secret = $this->env('S3_SECRET', 'minioadmin'); + + $this->object = new S3($this->root, $key, $secret, $host, 'us-east-1', Acl::Private); + } + + protected function getAdapterType(): DeviceType + { + return $this->object->getType(); + } + +} diff --git a/packages/storage/tests/Storage/Device/LocalTest.php b/packages/storage/tests/Storage/Device/LocalTest.php index 3980a76cf..4d700426e 100644 --- a/packages/storage/tests/Storage/Device/LocalTest.php +++ b/packages/storage/tests/Storage/Device/LocalTest.php @@ -7,55 +7,70 @@ use PHPUnit\Framework\TestCase; use Utopia\Storage\Device\Local; use Utopia\Storage\Exception\NotFoundException; +use Utopia\Storage\Exception\UploadException; final class LocalTest extends TestCase { /** - * @var Local + * @return resource */ - protected $object; - - protected function setUp(): void + private function openStream(string $path) { - $this->object = new Local(realpath(__DIR__ . '/../../resources/disk-a')); - } + $handle = fopen($path, 'rb'); + if ($handle === false) { + throw new \RuntimeException('Failed to open ' . $path); + } - protected function tearDown(): void {} + return $handle; + } - public function testPaths(): void + /** + * @param resource $handle + * @param int<1, max> $length + */ + private function readBytes($handle, int $length): string { - $this->assertEquals('/storage/functions', $this->object->getAbsolutePath('////storage/functions')); - $this->assertEquals('/storage/functions', $this->object->getAbsolutePath('storage/functions')); - $this->assertEquals('/storage/functions', $this->object->getAbsolutePath('/storage/functions')); - $this->assertEquals('/storage/functions', $this->object->getAbsolutePath('//storage///functions//')); - $this->assertEquals('/storage/functions', $this->object->getAbsolutePath('\\\\\storage\functions')); - $this->assertEquals('/storage/functions', $this->object->getAbsolutePath('..\\\\\//storage\\//functions')); - $this->assertEquals('/storage/functions', $this->object->getAbsolutePath('./..\\\\\//storage\\//functions')); + $contents = fread($handle, $length); + if ($contents === false) { + throw new \RuntimeException('Failed to read stream'); + } + + return $contents; } - public function testName(): void + private \Utopia\Storage\Device\Local $object; + + protected function setUp(): void { - $this->assertEquals('Local Storage', $this->object->getName()); + $this->object = new Local(realpath(__DIR__ . '/../../resources/disk-a') ?: ''); } - public function testType(): void + protected function tearDown(): void {} + + public function testPaths(): void { - $this->assertEquals('local', $this->object->getType()); + $this->assertSame('/storage/functions', $this->object->getAbsolutePath('////storage/functions')); + $this->assertSame('/storage/functions', $this->object->getAbsolutePath('storage/functions')); + $this->assertSame('/storage/functions', $this->object->getAbsolutePath('/storage/functions')); + $this->assertSame('/storage/functions', $this->object->getAbsolutePath('//storage///functions//')); + $this->assertSame('/storage/functions', $this->object->getAbsolutePath('\\\\\storage\functions')); + $this->assertSame('/storage/functions', $this->object->getAbsolutePath('..\\\\\//storage\\//functions')); + $this->assertSame('/storage/functions', $this->object->getAbsolutePath('./..\\\\\//storage\\//functions')); } - public function testDescription(): void + public function testType(): void { - $this->assertEquals('Adapter for Local storage that is in the physical or virtual machine or mounted to it.', $this->object->getDescription()); + $this->assertSame(\Utopia\Storage\DeviceType::Local, $this->object->getType()); } public function testRoot(): void { - $this->assertEquals($this->object->getRoot(), $this->object->getAbsolutePath(__DIR__ . '/../../resources/disk-a')); + $this->assertSame($this->object->getRoot(), $this->object->getAbsolutePath(__DIR__ . '/../../resources/disk-a')); } public function testPath(): void { - $this->assertEquals($this->object->getPath('image.png'), $this->object->getAbsolutePath(__DIR__ . '/../../resources/disk-a') . '/image.png'); + $this->assertSame($this->object->getPath('image.png'), $this->object->getAbsolutePath(__DIR__ . '/../../resources/disk-a') . '/image.png'); } public function testWrite(): void @@ -70,7 +85,7 @@ public function testWrite(): void public function testRead(): void { $this->assertEquals(true, $this->object->write($this->object->getPath('text-for-read.txt'), 'Hello World')); - $this->assertEquals('Hello World', $this->object->read($this->object->getPath('text-for-read.txt'))); + $this->assertSame('Hello World', $this->object->read($this->object->getPath('text-for-read.txt'))); $this->object->delete($this->object->getPath('text-for-read.txt')); } @@ -93,9 +108,9 @@ public function testFileExists(): void public function testMove(): void { $this->assertEquals(true, $this->object->write($this->object->getPath('text-for-move.txt'), 'Hello World')); - $this->assertEquals('Hello World', $this->object->read($this->object->getPath('text-for-move.txt'))); + $this->assertSame('Hello World', $this->object->read($this->object->getPath('text-for-move.txt'))); $this->assertEquals(true, $this->object->move($this->object->getPath('text-for-move.txt'), $this->object->getPath('text-for-move-new.txt'))); - $this->assertEquals('Hello World', $this->object->read($this->object->getPath('text-for-move-new.txt'))); + $this->assertSame('Hello World', $this->object->read($this->object->getPath('text-for-move-new.txt'))); $this->assertFileDoesNotExist($this->object->getPath('text-for-move.txt')); $this->assertIsNotReadable($this->object->getPath('text-for-move.txt')); $this->assertFileExists($this->object->getPath('text-for-move-new.txt')); @@ -104,10 +119,54 @@ public function testMove(): void $this->object->delete($this->object->getPath('text-for-move-new.txt')); } + public function testTransferRejectsInvalidChunkSize(): void + { + $device = new Local(realpath(__DIR__ . '/../../resources/disk-b') ?: ''); + + $this->expectExceptionMessage('Chunk size must be greater than zero'); + $this->object->transfer($this->object->getPath('kitten-1.jpg'), $device->getPath('kitten-1.jpg'), $device, 0); + } + + public function testListFiles(): void + { + $directory = $this->object->getPath('list-files'); + $this->object->write($directory . '/a.txt', 'aa'); + $this->object->write($directory . '/nested/b.txt', 'bb'); + $this->object->write($directory . '/.hidden', 'hh'); + + $list = $this->object->listFiles($directory, 2); + $this->assertCount(2, $list->files); + $this->assertNotNull($list->cursor); + + $rest = $this->object->listFiles($directory, 10, $list->cursor); + $this->assertCount(1, $rest->files); + $this->assertNull($rest->cursor); + + $paths = array_map(fn(\Utopia\Storage\FileInfo $file): string => $file->path, array_merge($list->files, $rest->files)); + $this->assertContains($directory . '/a.txt', $paths); + $this->assertContains($directory . '/nested/b.txt', $paths); + $this->assertContains($directory . '/.hidden', $paths); + $this->assertSame(2, $list->files[1]->size); + + $this->object->delete($directory, true); + } + + public function testStatOnMissingFileThrowsNotFound(): void + { + $this->expectException(NotFoundException::class); + $this->object->getFileSize($this->object->getPath('does-not-exist.txt')); + } + + public function testMoveIdenticalName(): void + { + $file = '/kitten-1.jpg'; + $this->assertFalse($this->object->move($file, $file)); + } + public function testDelete(): void { $this->assertEquals(true, $this->object->write($this->object->getPath('text-for-delete.txt'), 'Hello World')); - $this->assertEquals('Hello World', $this->object->read($this->object->getPath('text-for-delete.txt'))); + $this->assertSame('Hello World', $this->object->read($this->object->getPath('text-for-delete.txt'))); $this->assertEquals(true, $this->object->delete($this->object->getPath('text-for-delete.txt'))); $this->assertFileDoesNotExist($this->object->getPath('text-for-delete.txt')); $this->assertIsNotReadable($this->object->getPath('text-for-delete.txt')); @@ -127,24 +186,24 @@ public function testRecursiveDeleteRemovesHiddenFiles(): void public function testFileSize(): void { - $this->assertEquals(599639, $this->object->getFileSize(__DIR__ . '/../../resources/disk-a/kitten-1.jpg')); - $this->assertEquals(131958, $this->object->getFileSize(__DIR__ . '/../../resources/disk-a/kitten-2.jpg')); + $this->assertSame(599639, $this->object->getFileSize(__DIR__ . '/../../resources/disk-a/kitten-1.jpg')); + $this->assertSame(131958, $this->object->getFileSize(__DIR__ . '/../../resources/disk-a/kitten-2.jpg')); } public function testFileMimeType(): void { - $this->assertEquals('image/jpeg', $this->object->getFileMimeType(__DIR__ . '/../../resources/disk-a/kitten-1.jpg')); - $this->assertEquals('image/jpeg', $this->object->getFileMimeType(__DIR__ . '/../../resources/disk-a/kitten-2.jpg')); - $this->assertEquals('image/png', $this->object->getFileMimeType(__DIR__ . '/../../resources/disk-b/kitten-1.png')); - $this->assertEquals('image/png', $this->object->getFileMimeType(__DIR__ . '/../../resources/disk-b/kitten-2.png')); + $this->assertSame('image/jpeg', $this->object->getFileMimeType(__DIR__ . '/../../resources/disk-a/kitten-1.jpg')); + $this->assertSame('image/jpeg', $this->object->getFileMimeType(__DIR__ . '/../../resources/disk-a/kitten-2.jpg')); + $this->assertSame('image/png', $this->object->getFileMimeType(__DIR__ . '/../../resources/disk-b/kitten-1.png')); + $this->assertSame('image/png', $this->object->getFileMimeType(__DIR__ . '/../../resources/disk-b/kitten-2.png')); } public function testFileHash(): void { - $this->assertEquals('7551f343143d2e24ab4aaf4624996b6a', $this->object->getFileHash(__DIR__ . '/../../resources/disk-a/kitten-1.jpg')); - $this->assertEquals('81702fdeef2e55b1a22617bce4951cb5', $this->object->getFileHash(__DIR__ . '/../../resources/disk-a/kitten-2.jpg')); - $this->assertEquals('03010f4f02980521a8fd6213b52ec313', $this->object->getFileHash(__DIR__ . '/../../resources/disk-b/kitten-1.png')); - $this->assertEquals('8a9ed992b77e4b62b10e3a5c8ed72062', $this->object->getFileHash(__DIR__ . '/../../resources/disk-b/kitten-2.png')); + $this->assertSame('7551f343143d2e24ab4aaf4624996b6a', $this->object->getFileHash(__DIR__ . '/../../resources/disk-a/kitten-1.jpg')); + $this->assertSame('81702fdeef2e55b1a22617bce4951cb5', $this->object->getFileHash(__DIR__ . '/../../resources/disk-a/kitten-2.jpg')); + $this->assertSame('03010f4f02980521a8fd6213b52ec313', $this->object->getFileHash(__DIR__ . '/../../resources/disk-b/kitten-1.png')); + $this->assertSame('8a9ed992b77e4b62b10e3a5c8ed72062', $this->object->getFileHash(__DIR__ . '/../../resources/disk-b/kitten-2.png')); } public function testDirectoryCreate(): void @@ -160,7 +219,7 @@ public function testDirectorySize(): void $this->assertGreaterThan(0, $this->object->getDirectorySize(__DIR__ . '/../../resources/disk-b/')); } - public function testPartUpload() + public function testPartUpload(): string { $source = __DIR__ . '/../../resources/disk-a/large_file.mp4'; $dest = $this->object->getPath('uploaded.mp4'); @@ -172,19 +231,15 @@ public function testPartUpload() $chunk = 1; $start = 0; - $handle = @fopen($source, 'rb'); + $handle = $this->openStream($source); while ($start < $totalSize) { - $contents = fread($handle, $chunkSize); - $op = __DIR__ . '/chunk.part'; - $cc = fopen($op, 'wb'); - fwrite($cc, $contents); - fclose($cc); - $this->object->upload($op, $dest, $chunk, $chunks); + $contents = $this->readBytes($handle, $chunkSize); + $this->object->uploadData($contents, $dest, '', $chunk, $chunks); $start += \strlen($contents); - $chunk++; + ++$chunk; fseek($handle, $start); } - @fclose($handle); + fclose($handle); $this->assertEquals(filesize($source), $this->object->getFileSize($dest)); $this->assertEquals(md5_file($source), $this->object->getFileHash($dest)); @@ -202,14 +257,11 @@ public function testUploadChunkDoesNotFinalizeUntilFinalizeUpload(): void ]; foreach ($parts as $chunk => $data) { - $source = __DIR__ . '/chunk-' . $chunk . '.part'; - file_put_contents($source, $data); - - $this->object->uploadChunk($source, $dest, $chunk, 3, $metadata); + $this->object->uploadChunk($data, $dest, $chunk, 3, $metadata); $this->assertFalse($this->object->exists($dest)); } - $this->assertSame(3, $metadata['chunks']); + $this->assertSame(3, $metadata['chunks'] ?? null); $this->assertTrue($this->object->finalizeUpload($dest, 3, $metadata)); $this->assertSame('aaabbbccc', $this->object->read($dest)); $this->assertTrue($this->object->finalizeUpload($dest, 3, $metadata)); @@ -221,22 +273,20 @@ public function testFinalizeUploadRequiresAllLocalChunks(): void { $dest = $this->object->getPath('chunked-phase-missing.txt'); $metadata = []; - $source = __DIR__ . '/chunk-missing.part'; - file_put_contents($source, 'aaa'); - $this->object->uploadChunk($source, $dest, 1, 2, $metadata); + $this->object->uploadChunk('aaa', $dest, 1, 2, $metadata); try { $this->object->finalizeUpload($dest, 2, $metadata); $this->fail('Expected missing chunk exception'); - } catch (\Exception $e) { + } catch (UploadException $e) { $this->assertSame('Missing chunk 2', $e->getMessage()); } finally { $this->object->abort($dest); } } - public function testPartUploadRetry() + public function testPartUploadRetry(): string { $source = __DIR__ . '/../../resources/disk-a/large_file.mp4'; $dest = $this->object->getPath('uploaded2.mp4'); @@ -248,38 +298,28 @@ public function testPartUploadRetry() $chunk = 1; $start = 0; - $handle = @fopen($source, 'rb'); - $op = __DIR__ . '/chunkx.part'; + $handle = $this->openStream($source); while ($start < $totalSize) { - $contents = fread($handle, $chunkSize); - $op = __DIR__ . '/chunkx.part'; - $cc = fopen($op, 'wb'); - fwrite($cc, $contents); - fclose($cc); - $this->object->upload($op, $dest, $chunk, $chunks); + $contents = $this->readBytes($handle, $chunkSize); + $this->object->uploadData($contents, $dest, '', $chunk, $chunks); $start += \strlen($contents); - $chunk++; + ++$chunk; break; } - @fclose($handle); + fclose($handle); $chunk = 1; $start = 0; // retry from first to make sure duplicate chunk re-upload works without issue - $handle = @fopen($source, 'rb'); - $op = __DIR__ . '/chunkx.part'; + $handle = $this->openStream($source); while ($start < $totalSize) { - $contents = fread($handle, $chunkSize); - $op = __DIR__ . '/chunkx.part'; - $cc = fopen($op, 'wb'); - fwrite($cc, $contents); - fclose($cc); - $this->object->upload($op, $dest, $chunk, $chunks); + $contents = $this->readBytes($handle, $chunkSize); + $this->object->uploadData($contents, $dest, '', $chunk, $chunks); $start += \strlen($contents); - $chunk++; + ++$chunk; fseek($handle, $start); } - @fclose($handle); + fclose($handle); $this->assertEquals(filesize($source), $this->object->getFileSize($dest)); $this->assertEquals(md5_file($source), $this->object->getFileHash($dest)); @@ -298,19 +338,15 @@ public function testAbort(): void $chunk = 1; $start = 0; - $handle = @fopen($source, 'rb'); + $handle = $this->openStream($source); while ($chunk < 3) { // only upload two chunks - $contents = fread($handle, $chunkSize); - $op = __DIR__ . '/chunk.part'; - $cc = fopen($op, 'wb'); - fwrite($cc, $contents); - fclose($cc); - $this->object->upload($op, $dest, $chunk, $chunks); + $contents = $this->readBytes($handle, $chunkSize); + $this->object->uploadData($contents, $dest, '', $chunk, $chunks); $start += \strlen($contents); - $chunk++; + ++$chunk; fseek($handle, $start); } - @fclose($handle); + fclose($handle); // using file name with same first four chars $source = __DIR__ . '/../../resources/disk-a/large_file.mp4'; @@ -322,19 +358,15 @@ public function testAbort(): void $chunk = 1; $start = 0; - $handle = @fopen($source, 'rb'); + $handle = $this->openStream($source); while ($chunk < 3) { // only upload two chunks - $contents = fread($handle, $chunkSize); - $op = __DIR__ . '/chunk.part'; - $cc = fopen($op, 'wb'); - fwrite($cc, $contents); - fclose($cc); - $this->object->upload($op, $dest1, $chunk, $chunks); + $contents = $this->readBytes($handle, $chunkSize); + $this->object->uploadData($contents, $dest1, '', $chunk, $chunks); $start += \strlen($contents); - $chunk++; + ++$chunk; fseek($handle, $start); } - @fclose($handle); + fclose($handle); $this->assertTrue($this->object->abort($dest)); $this->assertTrue($this->object->abort($dest1)); @@ -363,12 +395,10 @@ public function testPartitionTotalSpace(): void public function testTransferLarge(string $path): void { // chunked file - $this->object->setTransferChunkSize(10000000); // 10 mb - - $device = new Local(realpath(__DIR__ . '/../../resources/disk-b')); + $device = new Local(realpath(__DIR__ . '/../../resources/disk-b') ?: ''); $destination = $device->getPath('largefile.mp4'); - $this->assertTrue($this->object->transfer($path, $destination, $device)); + $this->assertTrue($this->object->transfer($path, $destination, $device, 10000000)); // 10 mb chunks $this->assertTrue($device->exists($destination)); $this->assertSame('video/mp4', $device->getFileMimeType($destination)); @@ -378,15 +408,13 @@ public function testTransferLarge(string $path): void public function testTransferSmall(): void { - $this->object->setTransferChunkSize(10000000); // 10 mb - - $device = new Local(realpath(__DIR__ . '/../../resources/disk-b')); + $device = new Local(realpath(__DIR__ . '/../../resources/disk-b') ?: ''); $path = $this->object->getPath('text-for-read.txt'); $this->object->write($path, 'Hello World'); $destination = $device->getPath('hello.txt'); - $this->assertTrue($this->object->transfer($path, $destination, $device)); + $this->assertTrue($this->object->transfer($path, $destination, $device, 10000000)); // 10 mb chunks $this->assertTrue($device->exists($destination)); $this->assertSame('Hello World', $device->read($destination)); @@ -426,20 +454,18 @@ public function testDeletePath(): void $this->assertEquals(false, $this->object->exists($path3)); } - public function testGetFiles(): void + public function testListFilesEmptyAndGrowingDirectory(): void { $dir = $this->object->getPath('get-files-test'); $this->assertTrue($this->object->createDirectory($dir)); - $files = $this->object->getFiles($dir); - $this->assertCount(0, $files); + $this->assertCount(0, $this->object->listFiles($dir)->files); $this->object->write($dir . DIRECTORY_SEPARATOR . 'new-file.txt', 'Hello World'); $this->object->write($dir . DIRECTORY_SEPARATOR . 'new-file-two.txt', 'Hello World'); - $files = $this->object->getFiles($dir); - $this->assertCount(2, $files); + $this->assertCount(2, $this->object->listFiles($dir)->files); $this->assertTrue($this->object->deletePath('get-files-test')); } @@ -505,7 +531,7 @@ public function testJoinChunksCleansUpTempFilesOnSuccess(): void $this->assertDirectoryDoesNotExist($tmpDir, 'Temp chunk directory should be removed after assembly'); $this->assertFileDoesNotExist($tmpAssemble, 'Temp assembly file should be removed after rename'); - for ($i = 1; $i <= 3; $i++) { + for ($i = 1; $i <= 3; ++$i) { $this->assertFileDoesNotExist( $tmpDir . DIRECTORY_SEPARATOR . 'test.part.' . $i, "Part file $i should be removed after assembly", diff --git a/packages/storage/tests/Storage/Device/S3/RetryStrategyTest.php b/packages/storage/tests/Storage/Device/S3/RetryStrategyTest.php new file mode 100644 index 000000000..60899a4f7 --- /dev/null +++ b/packages/storage/tests/Storage/Device/S3/RetryStrategyTest.php @@ -0,0 +1,83 @@ +SlowDownPlease reduce your request rate.'; + $strategy = new RetryStrategy(delay: 0.5); + + $this->assertEqualsWithDelta(0.5, $strategy->delay($this->request(), 1, $this->response(503, $body), null), PHP_FLOAT_EPSILON); + } + + public function testNonTransientXmlErrorIsNotRetried(): void + { + $body = 'NoSuchKeyThe specified key does not exist.'; + + $this->assertNull(new RetryStrategy()->delay($this->request(), 1, $this->response(404, $body), null)); + } + + public function testStatusFallbackIsTransient(): void + { + $strategy = new RetryStrategy(); + + $this->assertNotNull($strategy->delay($this->request(), 1, $this->response(429), null)); + $this->assertNotNull($strategy->delay($this->request(), 1, $this->response(503), null)); + } + + /** XML error code takes precedence over HTTP status — 503 with non-transient XML must not be retried. */ + public function test503WithNonTransientXmlIsNotRetried(): void + { + $body = 'InternalErrorInternal server error.'; + + $this->assertNull(new RetryStrategy()->delay($this->request(), 1, $this->response(503, $body), null)); + } + + public function testRetriesAreCapped(): void + { + $strategy = new RetryStrategy(retries: 2); + $response = $this->response(503); + + $this->assertNotNull($strategy->delay($this->request(), 1, $response, null)); + $this->assertNotNull($strategy->delay($this->request(), 2, $response, null)); + $this->assertNull($strategy->delay($this->request(), 3, $response, null)); + } + + public function testTransportErrorsAreNotRetried(): void + { + $error = new NetworkException($this->request(), 'Connection reset'); + + $this->assertNull(new RetryStrategy()->delay($this->request(), 1, null, $error)); + } + + public function testReadingTheBodyLeavesItReadable(): void + { + $body = 'SlowDown'; + $response = $this->response(503, $body); + + $this->assertNotNull(new RetryStrategy()->delay($this->request(), 1, $response, null)); + $this->assertSame($body, (string) $response->getBody()); + } +} diff --git a/packages/storage/tests/Storage/Device/S3SlowDownTest.php b/packages/storage/tests/Storage/Device/S3SlowDownTest.php deleted file mode 100644 index d8240a232..000000000 --- a/packages/storage/tests/Storage/Device/S3SlowDownTest.php +++ /dev/null @@ -1,206 +0,0 @@ -isTransientError($statusCode, $body); - } - - #[\Override] - protected function call(string $operation, string $method, string $uri, string $data = '', array $parameters = [], bool $decode = true): \stdClass - { - $this->calls[] = $operation; - $this->headersByOperation[$operation] = $this->headers; - - if ($operation === 's3:info') { - if (! $this->objectExists) { - throw new \Exception('Not found'); - } - - return (object) ['headers' => ['content-length' => 1], 'body' => '']; - } - - if ($operation === 's3:createMultipartUpload') { - return (object) ['headers' => [], 'body' => ['UploadId' => 'upload-123']]; - } - - if ($operation === 's3:uploadPart') { - return (object) ['headers' => ['etag' => 'etag-' . $parameters['partNumber']], 'body' => '']; - } - - if ($operation === 's3:completeMultipartUpload') { - $this->completedBody = $data; - $this->objectExists = true; - - return (object) ['headers' => [], 'body' => '']; - } - - return (object) ['headers' => [], 'body' => '']; - } -} - -final class S3SlowDownTest extends TestCase -{ - private TestableS3 $s3; - - protected function setUp(): void - { - $this->s3 = new TestableS3( - root: '/root', - accessKey: 'test-key', - secretKey: 'test-secret', - host: 'https://s3.example.com', - region: 'us-east-1', - ); - } - - protected function tearDown(): void - { - S3::setRetryAttempts(3); - S3::setRetryDelay(500); - } - - public function testTransientXmlErrorIsRetried(): void - { - $body = 'SlowDownPlease reduce your request rate.'; - $this->assertTrue($this->s3->exposedIsTransientError(503, $body)); - } - - public function testNonTransientXmlErrorIsNotRetried(): void - { - $body = 'NoSuchKeyThe specified key does not exist.'; - $this->assertFalse($this->s3->exposedIsTransientError(404, $body)); - } - - public function testStatusFallbackIsTransient(): void - { - $this->assertTrue($this->s3->exposedIsTransientError(429, '')); - $this->assertTrue($this->s3->exposedIsTransientError(503, '')); - } - - /** XML error code takes precedence over HTTP status — 503 with non-transient XML must not be retried. */ - public function test503WithNonTransientXmlIsNotRetried(): void - { - $body = 'InternalErrorInternal server error.'; - $this->assertFalse($this->s3->exposedIsTransientError(503, $body)); - } - - public function testDefaultRetrySettings(): void - { - $prop = fn(string $name): mixed => new \ReflectionProperty(S3::class, $name)->getValue(); - $this->assertSame(3, $prop('retryAttempts')); - $this->assertSame(500, $prop('retryDelay')); - } - - public function testPrepareUploadCreatesMultipartMetadata(): void - { - $metadata = []; - - $this->s3->prepareUpload('/root/file.txt', 'text/plain', 2, $metadata); - - $this->assertSame('upload-123', $metadata['uploadId']); - $this->assertSame([], $metadata['parts']); - $this->assertSame(0, $metadata['chunks']); - $this->assertSame(['s3:createMultipartUpload'], $this->s3->calls); - } - - public function testUploadChunkRecordsPartWithoutCompleting(): void - { - $metadata = []; - $source = __DIR__ . '/s3-chunk.part'; - file_put_contents($source, 'aaa'); - - $this->s3->prepareUpload('/root/file.txt', 'text/plain', 2, $metadata); - $chunks = $this->s3->uploadChunk($source, '/root/file.txt', 1, 2, $metadata); - - $this->assertSame(1, $chunks); - $this->assertSame('etag-1', $metadata['parts'][1]); - $this->assertNotContains('s3:completeMultipartUpload', $this->s3->calls); - - unlink($source); - } - - public function testSingleChunkUploadDataDoesNotFinalizeOrCheckExists(): void - { - $metadata = []; - - $this->assertSame(1, $this->s3->uploadData('aaa', '/root/file.txt', 'text/plain', 1, 1, $metadata)); - $this->assertSame(['s3:write'], $this->s3->calls); - $this->assertSame([1 => true], $metadata['parts']); - $this->assertSame(1, $metadata['chunks']); - } - - public function testFinalizeUploadRequiresAllS3Parts(): void - { - $metadata = [ - 'uploadId' => 'upload-123', - 'parts' => [1 => 'etag-1'], - 'chunks' => 1, - ]; - - $this->expectException(\Exception::class); - $this->expectExceptionMessage('Missing chunk 2'); - $this->s3->finalizeUpload('/root/file.txt', 2, $metadata); - } - - public function testFinalizeUploadCompletesS3PartsInNumericOrder(): void - { - $metadata = [ - 'uploadId' => 'upload-123', - 'parts' => [ - 10 => 'etag-10', - 9 => 'etag-9', - 8 => 'etag-8', - 7 => 'etag-7', - 6 => 'etag-6', - 5 => 'etag-5', - 4 => 'etag-4', - 3 => 'etag-3', - 2 => 'etag-2', - 1 => 'etag-1', - ], - 'chunks' => 10, - ]; - - $this->assertTrue($this->s3->finalizeUpload('/root/file.txt', 10, $metadata)); - - $part1 = strpos($this->s3->completedBody, '1'); - $part2 = strpos($this->s3->completedBody, '2'); - $part10 = strpos($this->s3->completedBody, '10'); - - $this->assertNotFalse($part1); - $this->assertNotFalse($part2); - $this->assertNotFalse($part10); - $this->assertLessThan($part2, $part1); - $this->assertLessThan($part10, $part2); - } - - public function testFinalizeUploadSendsCompleteBodyAsXml(): void - { - $metadata = [ - 'uploadId' => 'upload-123', - 'parts' => [1 => 'etag-1', 2 => 'etag-2'], - 'chunks' => 2, - ]; - - $this->assertTrue($this->s3->finalizeUpload('/root/file.txt', 2, $metadata)); - $this->assertSame('application/xml', $this->s3->headersByOperation['s3:completeMultipartUpload']['content-type']); - } -} diff --git a/packages/storage/tests/Storage/Device/S3Test.php b/packages/storage/tests/Storage/Device/S3Test.php index 1a0a8e1e2..76c408087 100644 --- a/packages/storage/tests/Storage/Device/S3Test.php +++ b/packages/storage/tests/Storage/Device/S3Test.php @@ -4,33 +4,390 @@ namespace Utopia\Tests\Storage\Device; +use PHPUnit\Framework\TestCase; +use Psr\Http\Message\RequestInterface; +use Psr\Http\Message\ResponseInterface; +use Utopia\Client\Adapter; +use Utopia\Client\Decorator\Retry; +use Utopia\Client\Exception\NetworkException; +use Utopia\Client\Tls; +use Utopia\Psr7\Request; +use Utopia\Psr7\Response; +use Utopia\Psr7\Stream; +use Utopia\Psr7\Uri; use Utopia\Storage\Device\S3; -use Utopia\Tests\Storage\S3Base; +use Utopia\Storage\Device\S3\Response as S3Response; +use Utopia\Storage\Device\S3\RetryStrategy; +use Utopia\Storage\Exception\NotFoundException; +use Utopia\Storage\Exception\RemoteException; +use Utopia\Storage\Exception\StorageException; +use Utopia\Storage\Exception\TransportException; +use Utopia\Storage\Exception\UploadException; -final class S3Test extends S3Base +/** + * Testable S3 subclass that exposes protected helpers. + */ +class TestableS3 extends S3 { - protected function init(): void + /** + * @var array + */ + public array $calls = []; + + public string $completedBody = ''; + + /** + * @var array> + */ + public array $headersByOperation = []; + + private bool $objectExists = false; + + #[\Override] + protected function call(string $method, string $uri, string $data = '', array $parameters = [], array $headers = [], array $amzHeaders = [], bool $decode = true): S3Response + { + $operation = match (true) { + $method === 'HEAD' => 's3:info', + $method === 'POST' && \array_key_exists('uploads', $parameters) => 's3:createMultipartUpload', + $method === 'PUT' && isset($parameters['partNumber']) => 's3:uploadPart', + $method === 'POST' && isset($parameters['uploadId']) => 's3:completeMultipartUpload', + $method === 'PUT' => 's3:write', + default => 's3:' . strtolower($method), + }; + $this->calls[] = $operation; + $this->headersByOperation[$operation] = $headers; + + if ($operation === 's3:info') { + if (! $this->objectExists) { + throw new \Exception('Not found'); + } + + return new S3Response(code: 200, headers: ['content-length' => '1'], body: ''); + } + + if ($operation === 's3:createMultipartUpload') { + return new S3Response(code: 200, headers: [], body: ['UploadId' => 'upload-123']); + } + + if ($operation === 's3:uploadPart') { + return new S3Response(code: 200, headers: ['etag' => 'etag-' . $parameters['partNumber']], body: ''); + } + + if ($operation === 's3:completeMultipartUpload') { + $this->completedBody = $data; + $this->objectExists = true; + + return new S3Response(code: 200, headers: [], body: ''); + } + + return new S3Response(code: 200, headers: [], body: ''); + } +} + +/** + * Client stub that replays scripted responses and records every request. + * Implements the full utopia-php/client Adapter so the Retry decorator can wrap it. + */ +final class ScriptedClient implements Adapter +{ + /** + * @var array + */ + public array $requests = []; + + /** + * @param array $responses + */ + public function __construct(private array $responses) {} + + public function sendRequest(RequestInterface $request): ResponseInterface + { + $this->requests[] = $request; + $response = array_shift($this->responses); + if ($response instanceof \Psr\Http\Client\ClientExceptionInterface) { + throw $response; + } + if (! $response instanceof ResponseInterface) { + throw new \RuntimeException('No scripted response left for ' . $request->getMethod() . ' ' . $request->getUri()); + } + + return $response; + } + + public function stream(RequestInterface $request, callable $sink): ResponseInterface + { + $response = $this->sendRequest($request); + $sink((string) $response->getBody()); + + return $response; + } + + public function withTimeout(float $seconds): static + { + return $this; + } + + public function withConnectTimeout(float $seconds): static + { + return $this; + } + + public function withSslVerification(bool $enabled = true): static + { + return $this; + } + + public function withCustomCA(string $path): static + { + return $this; + } + + public function withCertificate(string $certPath, string $keyPath, ?string $passphrase = null): static + { + return $this; + } + + public function withMinTlsVersion(Tls $version): static + { + return $this; + } + + public function withConnectionReuse(bool $enabled = true): static + { + return $this; + } +} + +final class S3Test extends TestCase +{ + private TestableS3 $s3; + + protected function setUp(): void + { + $this->s3 = new TestableS3( + root: '/root', + accessKey: 'test-key', + secretKey: 'test-secret', + host: 'https://s3.example.com', + region: 'us-east-1', + ); + } + + public function testPrepareUploadCreatesMultipartMetadata(): void + { + $metadata = []; + + $this->s3->prepareUpload('/root/file.txt', 'text/plain', 2, $metadata); + + $this->assertSame('upload-123', $metadata['uploadId'] ?? null); + $this->assertSame([], $metadata['parts'] ?? null); + $this->assertSame(0, $metadata['chunks'] ?? null); + $this->assertSame(['s3:createMultipartUpload'], $this->s3->calls); + } + + public function testUploadChunkRecordsPartWithoutCompleting(): void + { + $metadata = []; + + $this->s3->prepareUpload('/root/file.txt', 'text/plain', 2, $metadata); + $chunks = $this->s3->uploadChunk('aaa', '/root/file.txt', 1, 2, $metadata); + + $this->assertSame(1, $chunks); + $this->assertSame('etag-1', ($metadata['parts'] ?? [])[1] ?? null); + $this->assertNotContains('s3:completeMultipartUpload', $this->s3->calls); + } + + public function testSingleChunkUploadDataDoesNotFinalizeOrCheckExists(): void + { + $metadata = []; + + $this->assertSame(1, $this->s3->uploadData('aaa', '/root/file.txt', 'text/plain', 1, 1, $metadata)); + $this->assertSame(['s3:write'], $this->s3->calls); + $this->assertSame([1 => true], $metadata['parts'] ?? null); + $this->assertSame(1, $metadata['chunks'] ?? null); + } + + public function testFinalizeUploadRequiresAllS3Parts(): void + { + $metadata = [ + 'uploadId' => 'upload-123', + 'parts' => [1 => 'etag-1'], + 'chunks' => 1, + ]; + + $this->expectException(UploadException::class); + $this->expectExceptionMessage('Missing chunk 2'); + $this->s3->finalizeUpload('/root/file.txt', 2, $metadata); + } + + public function testFinalizeUploadCompletesS3PartsInNumericOrder(): void + { + $metadata = [ + 'uploadId' => 'upload-123', + 'parts' => [ + 10 => 'etag-10', + 9 => 'etag-9', + 8 => 'etag-8', + 7 => 'etag-7', + 6 => 'etag-6', + 5 => 'etag-5', + 4 => 'etag-4', + 3 => 'etag-3', + 2 => 'etag-2', + 1 => 'etag-1', + ], + 'chunks' => 10, + ]; + + $this->assertTrue($this->s3->finalizeUpload('/root/file.txt', 10, $metadata)); + + $part1 = strpos($this->s3->completedBody, '1'); + $part2 = strpos($this->s3->completedBody, '2'); + $part10 = strpos($this->s3->completedBody, '10'); + + $this->assertNotFalse($part1); + $this->assertNotFalse($part2); + $this->assertNotFalse($part10); + $this->assertLessThan($part2, $part1); + $this->assertLessThan($part10, $part2); + } + + public function testFinalizeUploadSendsCompleteBodyAsXml(): void { - $this->root = '/root'; - $host = $_SERVER['S3_HOST'] ?? 'http://utopia-storage-test.localhost:9805'; - $key = $_SERVER['S3_ACCESS_KEY'] ?? 'minioadmin'; - $secret = $_SERVER['S3_SECRET'] ?? 'minioadmin'; + $metadata = [ + 'uploadId' => 'upload-123', + 'parts' => [1 => 'etag-1', 2 => 'etag-2'], + 'chunks' => 2, + ]; - $this->object = new S3($this->root, $key, $secret, $host, 'us-east-1', S3::ACL_PRIVATE); + $this->assertTrue($this->s3->finalizeUpload('/root/file.txt', 2, $metadata)); + $this->assertSame('application/xml', $this->s3->headersByOperation['s3:completeMultipartUpload']['content-type']); } - protected function getAdapterName(): string + private function device(ScriptedClient $client): S3 { - return 'S3 Storage'; + return new S3( + root: '/root', + accessKey: 'test-key', + secretKey: 'test-secret', + host: 'https://s3.example.com', + region: 'us-east-1', + client: new Retry($client, new RetryStrategy(delay: 0.0)), + ); } - protected function getAdapterType(): string + private function slowDown(): Response { - return $this->object->getType(); + $body = 'SlowDownPlease reduce your request rate.'; + + return new Response(503, body: new Stream($body))->withHeader('content-type', 'application/xml'); + } + + public function testWriteSendsSignedRequest(): void + { + $client = new ScriptedClient([new Response(200)]); + + $this->assertTrue($this->device($client)->write('/root/file.txt', 'Hello World', 'text/plain')); + $this->assertCount(1, $client->requests); + + $request = $client->requests[0]; + $this->assertSame('PUT', $request->getMethod()); + $this->assertSame('s3.example.com', $request->getUri()->getHost()); + $this->assertSame('/root/file.txt', $request->getUri()->getPath()); + $this->assertSame('Hello World', (string) $request->getBody()); + $this->assertSame('text/plain', $request->getHeaderLine('content-type')); + $this->assertSame('private', $request->getHeaderLine('x-amz-acl')); + $this->assertSame(hash('sha256', 'Hello World'), $request->getHeaderLine('x-amz-content-sha256')); + $this->assertSame(base64_encode(md5('Hello World', true)), $request->getHeaderLine('content-md5')); + $this->assertStringStartsWith('AWS4-HMAC-SHA256 Credential=test-key/', $request->getHeaderLine('authorization')); + $this->assertSame('utopia-php/storage', $request->getHeaderLine('user-agent')); } - protected function getAdapterDescription(): string + public function testTransientErrorIsRetriedUntilSuccess(): void { - return 'S3 Storage drive for generic S3-compatible provider'; + $client = new ScriptedClient([$this->slowDown(), $this->slowDown(), new Response(200)]); + + $this->assertTrue($this->device($client)->write('/root/file.txt', 'Hello World', 'text/plain')); + $this->assertCount(3, $client->requests); + } + + public function testTransientErrorRetriesAreExhausted(): void + { + $client = new ScriptedClient([$this->slowDown(), $this->slowDown(), $this->slowDown(), $this->slowDown()]); + + try { + $this->device($client)->write('/root/file.txt', 'Hello World', 'text/plain'); + self::fail('Expected exception after exhausting retries'); + } catch (RemoteException $e) { + $this->assertSame(503, $e->getCode()); + $this->assertSame('SlowDown', $e->errorCode); + } + + // Initial attempt plus the default three retries. + $this->assertCount(4, $client->requests); + } + + public function testNoSuchKeyBecomesNotFoundException(): void + { + $body = 'NoSuchKeyThe specified key does not exist.'; + $client = new ScriptedClient([new Response(404, body: new Stream($body))]); + + $this->expectException(NotFoundException::class); + $this->device($client)->read('/root/missing.txt'); + } + + public function testHeadNotFoundHasNoBodyButThrowsNotFound(): void + { + // HEAD error responses carry no body, so the 404 status is the only signal. + $client = new ScriptedClient([new Response(404)]); + + $this->expectException(NotFoundException::class); + $this->device($client)->getFileSize('/root/missing.txt'); + } + + public function testTransportFailureIsWrapped(): void + { + $psrError = new NetworkException(new Request('PUT', Uri::parse('https://s3.example.com/root')), 'Connection reset'); + $client = new ScriptedClient([$psrError]); + + try { + $this->device($client)->write('/root/file.txt', 'Hello World', 'text/plain'); + self::fail('Expected transport exception'); + } catch (TransportException $e) { + $this->assertSame($psrError, $e->getPrevious()); + } + } + + public function testEveryFailureIsCatchableAsStorageException(): void + { + $body = 'AccessDeniedAccess denied.'; + $client = new ScriptedClient([new Response(403, body: new Stream($body))]); + + try { + $this->device($client)->read('/root/file.txt'); + self::fail('Expected storage exception'); + } catch (StorageException $e) { + $this->assertInstanceOf(RemoteException::class, $e); + $this->assertSame('Access denied.', $e->getMessage()); + $this->assertSame(403, $e->getCode()); + } + } + + public function testXmlListingIsDecodedIntoTypedFiles(): void + { + $body = '2true1000next-token' + . 'root/a.txt112026-01-02T03:04:05.000Z"abc123"' + . 'root/b.txt222026-01-02T03:04:06.000Z"def456"' + . ''; + $client = new ScriptedClient([new Response(200, body: new Stream($body))->withHeader('content-type', 'application/xml')]); + + $list = $this->device($client)->listFiles('/root/testing'); + + $this->assertCount(2, $list->files); + $this->assertSame('root/a.txt', $list->files[0]->path); + $this->assertSame(11, $list->files[0]->size); + $this->assertSame('abc123', $list->files[0]->etag); + $this->assertSame('2026-01-02', $list->files[0]->modifiedAt?->format('Y-m-d')); + $this->assertSame('next-token', $list->cursor); } } diff --git a/packages/storage/tests/Storage/Device/TelemetryTest.php b/packages/storage/tests/Storage/Device/TelemetryTest.php new file mode 100644 index 000000000..7a00ce5b2 --- /dev/null +++ b/packages/storage/tests/Storage/Device/TelemetryTest.php @@ -0,0 +1,35 @@ +getPath('lorem.txt'); + + $this->assertArrayNotHasKey('storage.operation', $telemetry->histograms); + + $device->exists($path); + + $this->assertArrayHasKey('storage.operation', $telemetry->histograms); + } + + public function testDecoratedDeviceIsAccessible(): void + { + $underlying = new Local(__DIR__ . '/../../resources/disk-a'); + $device = new Telemetry(new TestTelemetry(), $underlying); + + $this->assertSame($underlying, $device->getDevice()); + } +} diff --git a/packages/storage/tests/Storage/StorageTest.php b/packages/storage/tests/Storage/StorageTest.php deleted file mode 100644 index b50f63ba0..000000000 --- a/packages/storage/tests/Storage/StorageTest.php +++ /dev/null @@ -1,63 +0,0 @@ -assertInstanceOf(\Utopia\Storage\Device\Local::class, Storage::getDevice('disk-a')); - $this->assertInstanceOf(\Utopia\Storage\Device\Local::class, Storage::getDevice('disk-b')); - - try { - Storage::getDevice('disk-c'); - $this->fail('Expected exception not thrown'); - } catch (Exception $e) { - $this->assertSame('The device "disk-c" is not listed', $e->getMessage()); - } - } - - public function testExists(): void - { - $this->assertTrue(Storage::exists('disk-a')); - $this->assertTrue(Storage::exists('disk-b')); - $this->assertFalse(Storage::exists('disk-c')); - } - - public function testMoveIdenticalName(): void - { - $file = '/kitten-1.jpg'; - $device = Storage::getDevice('disk-a'); - $this->assertFalse($device->move($file, $file)); - } - - public function testStorageOperationTelemetryIsCreatedOnFirstRecord(): void - { - $telemetry = new TestTelemetry(); - $underlying = new Local(__DIR__ . '/../resources/disk-a'); - $device = new Telemetry($telemetry, $underlying); - $path = $underlying->getPath('lorem.txt'); - - $this->assertArrayNotHasKey('storage.operation', $telemetry->histograms); - - $device->exists($path); - - $this->assertArrayHasKey('storage.operation', $telemetry->histograms); - } -} diff --git a/packages/storage/tests/Storage/Validator/FileExtTest.php b/packages/storage/tests/Storage/Validator/FileExtTest.php index 454b48562..48db187f3 100644 --- a/packages/storage/tests/Storage/Validator/FileExtTest.php +++ b/packages/storage/tests/Storage/Validator/FileExtTest.php @@ -9,10 +9,7 @@ final class FileExtTest extends TestCase { - /** - * @var FileExt - */ - protected $object; + private \Utopia\Storage\Validator\FileExt $object; protected function setUp(): void { diff --git a/packages/storage/tests/Storage/Validator/FileNameTest.php b/packages/storage/tests/Storage/Validator/FileNameTest.php index b64e4f5c5..0a38abf3b 100644 --- a/packages/storage/tests/Storage/Validator/FileNameTest.php +++ b/packages/storage/tests/Storage/Validator/FileNameTest.php @@ -9,10 +9,7 @@ final class FileNameTest extends TestCase { - /** - * @var FileName - */ - protected $object; + private \Utopia\Storage\Validator\FileName $object; protected function setUp(): void { diff --git a/packages/storage/tests/Storage/Validator/FileSizeTest.php b/packages/storage/tests/Storage/Validator/FileSizeTest.php index d8ebc13b8..c2b86f3e4 100644 --- a/packages/storage/tests/Storage/Validator/FileSizeTest.php +++ b/packages/storage/tests/Storage/Validator/FileSizeTest.php @@ -9,10 +9,7 @@ final class FileSizeTest extends TestCase { - /** - * @var FileSize - */ - protected $object; + private \Utopia\Storage\Validator\FileSize $object; protected function setUp(): void { diff --git a/packages/storage/tests/Storage/Validator/FileTypeTest.php b/packages/storage/tests/Storage/Validator/FileTypeTest.php index e0346ed37..3dfef1cfd 100644 --- a/packages/storage/tests/Storage/Validator/FileTypeTest.php +++ b/packages/storage/tests/Storage/Validator/FileTypeTest.php @@ -9,10 +9,7 @@ final class FileTypeTest extends TestCase { - /** - * @var FileType - */ - protected $object; + private \Utopia\Storage\Validator\FileType $object; protected function setUp(): void { diff --git a/packages/storage/tests/Storage/Validator/UploadTest.php b/packages/storage/tests/Storage/Validator/UploadTest.php index 5b1e397ee..d6730c03e 100644 --- a/packages/storage/tests/Storage/Validator/UploadTest.php +++ b/packages/storage/tests/Storage/Validator/UploadTest.php @@ -9,10 +9,7 @@ final class UploadTest extends TestCase { - /** - * @var Upload - */ - protected $object; + private \Utopia\Storage\Validator\Upload $object; protected function setUp(): void {