feat(aws/sns): add SNS service + S3 ObjectCreated notifications#170
Open
andthezhang wants to merge 2 commits into
Open
feat(aws/sns): add SNS service + S3 ObjectCreated notifications#170andthezhang wants to merge 2 commits into
andthezhang wants to merge 2 commits into
Conversation
PutObject read the request body with `.text()` (and the presigned-POST path with `file.text()`), which decodes bytes as UTF-8 and replaces every non-UTF-8 byte with U+FFFD (EF BF BD). Any binary object — audio, images, gzip, protobuf — came back corrupted on GetObject, and Content-Length was inflated because each replacement char re-encodes to 3 bytes. Read the raw bytes via `arrayBuffer()` instead and persist them base64. ETag is now md5 of the raw bytes, Content-Length is the true byte count, and GetObject decodes the base64 back to a byte-exact Uint8Array. Copy already round-trips the stored body verbatim, so it inherits the fix. `md5()` now accepts Buffer/Uint8Array so it can hash raw bytes directly. Test: round-trips a binary object (NUL, 0x80/0xff/0xfe, PNG magic) byte-for-byte and asserts Content-Length is the raw byte count.
Adds an SNS emulator and wires S3 event notifications to it, so the common "upload to S3 -> fan out a notification -> deliver a request" flow works locally. SNS (POST /sns/, query-style): - CreateTopic, ListTopics, DeleteTopic - Subscribe, Unsubscribe (sqs, http, https protocols) - Publish (SNS envelope JSON, or raw message when RawMessageDelivery=true) - Get/SetTopicAttributes, Get/SetSubscriptionAttributes - HTTP/HTTPS subscriptions receive an SNS-shaped JSON body; a subscription RedrivePolicy with deadLetterTargetArn delivers failed HTTP notifications to the target SQS queue (no AWS-style retry backoff). S3 -> SNS: - PUT/GET /:bucket?notification to configure TopicConfiguration - s3:ObjectCreated:Put events fan out to matching topic configurations, with simple prefix/suffix filtering. Known limitation (intentional, matching LocalStack/Moto): HTTP/HTTPS notification bodies carry fake signature fields; the emulator does not implement AWS-real SNS signature verification. Tests: real @aws-sdk/client-sns E2E coverage for topic lifecycle, SQS and HTTP subscription delivery (incl. DLQ), and S3 ObjectCreated fan-out, plus seedFromConfig topic seeding.
Contributor
|
@andthezhang is attempting to deploy a commit to the Vercel Labs Team on Vercel. A member of the Team first needs to authorize it. |
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an SNS emulator and wires S3 event notifications into it, so the common flow — upload an object to S3 → S3 fires an
ObjectCreatedevent → SNS fans it out → a subscriber receives a request — works locally.Important
Stacked on #168. This branch is built on top of
fix/s3-binary-safe-objects(#168) because S3→SNS fan-out depends on objects round-tripping correctly. Please merge #168 first — until then, this PR's diff includes #168's changes (helpers.ts, the binary-safes3.tsPUT/GET, and the binary round-trip test). After #168 merges, this diff will show SNS only.What's added
SNS (
POST /sns/, query-style):CreateTopic,ListTopics,DeleteTopicSubscribe,Unsubscribe(sqs,http,httpsprotocols)Publish— SNS envelope JSON, or the raw message whenRawMessageDelivery=trueGet/SetTopicAttributes,Get/SetSubscriptionAttributesRedrivePolicywithdeadLetterTargetArndelivers failed HTTP notifications to the target SQS queue (no AWS-style retry backoff).S3 → SNS:
PUT/GET /:bucket?notificationto configure aTopicConfigurations3:ObjectCreated:Putevents fan out to matching topic configurations, with simple prefix/suffix filtering.Known limitation (intentional)
Consistent with LocalStack / Moto: HTTP/HTTPS notification bodies carry fake signature fields — the emulator does not implement AWS-real SNS signature verification.
Tests
Real
@aws-sdk/client-snsE2E coverage: topic lifecycle, SQS + HTTP subscription delivery (including DLQ), and S3ObjectCreatedfan-out, plusseedFromConfigtopic seeding. Adds@aws-sdk/client-snsas a dev dependency (lockfile change is purely additive).pnpm --filter @emulators/aws test→ all green;type-checkclean.