Multi-Storage Presigner

A Node.js library for generating presigned URLs for multiple object storage providers (AWS S3, Cloudflare R2) with automatic load balancing.

Features

• Support for multiple storage providers (AWS S3 and Cloudflare R2)
• Automatic load balancing between providers using various strategies
• Live configuration updates from file or remote URL
• Rate limiting and error handling
• RESTful API endpoints for generating presigned URLs
• Monitoring and statistics

Installation

npm install multibucket

TypeScript

This package includes a minimal index.d.ts so TypeScript projects can import it with type information. Example:

import MultiBucket = require('multibucket');
// or (with `esModuleInterop` enabled in tsconfig):
// import MultiBucket from 'multibucket';

const mb = new MultiBucket({ providers: [] });

If your TypeScript project still complains about missing types, you can either enable esModuleInterop in tsconfig.json, or, as a temporary workaround, add a declaration stub in the consuming project like declare module 'multibucket';.

Dependencies

This library requires the following dependencies:

npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner express axios chokidar

Basic Usage

const MultiBucket = require('multibucket');

// Create an instance with initial providers
const storagePresigner = new MultiBucket({
  providers: [
    {
      id: 's3-main',
      type: 's3',
      bucket: 'my-main-bucket',
      region: 'us-east-1',
      accessKeyId: 'YOUR_AWS_ACCESS_KEY',
      secretAccessKey: 'YOUR_AWS_SECRET_KEY',
      weight: 3,
      rateLimit: 100
    },
    {
      id: 'r2-cloudflare',
      type: 'r2',
      bucket: 'my-r2-bucket',
      endpoint: 'https://account-id.r2.cloudflarestorage.com',
      accessKeyId: 'YOUR_R2_ACCESS_KEY',
      secretAccessKey: 'YOUR_R2_SECRET_KEY',
      publicUrlBase: 'https://cdn.example.com'
    }
  ],
  loadBalanceStrategy: 'round-robin',
  defaultExpiry: 3600
});

// Start the API server
storagePresigner.createServer(3000);

Configuration Options

Constructor Options

• providers: Array of storage provider configurations
• configSource: Path or URL to a config file (optional)
• loadBalanceStrategy: Strategy for load balancing (default: 'round-robin')
• defaultExpiry: Default expiry time for presigned URLs in seconds (default: 3600)

Provider Configuration

Each provider object requires the following properties:

Common Properties:

• id: Unique identifier for the provider
• type: Provider type ('s3' or 'r2')
• bucket: Bucket name
• accessKeyId: Access key ID
• secretAccessKey: Secret access key
• weight (optional): Weight for weighted-random load balancing
• rateLimit (optional): Maximum requests per second
• publicUrlBase (optional): Base URL for public access

S3-specific Properties:

• region: AWS region
• endpoint (optional): Custom endpoint for S3-compatible services
• forcePathStyle (optional): Use path-style addressing

R2-specific Properties:

• endpoint: R2 endpoint URL

Load Balancing Strategies

• round-robin: Cycle through providers sequentially
• least-used: Select the provider with the fewest requests
• least-errors: Select the provider with the lowest error rate
• weighted-random: Select providers randomly based on their weight

External Configuration

You can provide a path to a JSON file or a URL in the configSource option:

const storagePresigner = new MultiBucket({
  configSource: './storage-config.json'
});

The library will watch for changes to the file or poll the URL to update the configuration dynamically.

API Endpoints

When you start the server with createServer(), the following endpoints are available:

Generate Upload URL

POST /generate-upload-url

Request body:

{
  "filename": "example.jpg",
  "contentType": "image/jpeg",
  "path": "uploads/images",
  "expiry": 1800,
  "providerId": "s3-main"
}

Response:

{
  "uploadUrl": "https://my-main-bucket.s3.us-east-1.amazonaws.com/uploads/images/uuid-example.jpg?...",
  "publicUrl": "https://my-main-bucket.s3.us-east-1.amazonaws.com/uploads/images/uuid-example.jpg",
  "key": "uploads/images/uuid-example.jpg",
  "bucket": "my-main-bucket",
  "provider": "s3-main",
  "expires": "2023-06-01T12:30:00.000Z"
}

Generate Read URL

POST /generate-read-url

Request body:

{
  "key": "uploads/images/uuid-example.jpg",
  "providerId": "s3-main",
  "expiry": 3600
}

Response:

{
  "readUrl": "https://my-main-bucket.s3.us-east-1.amazonaws.com/uploads/images/uuid-example.jpg?...",
  "key": "uploads/images/uuid-example.jpg",
  "bucket": "my-main-bucket",
  "provider": "s3-main",
  "expires": "2023-06-01T13:00:00.000Z"
}

Get Stats

GET /stats

Response:

{
  "providerCount": 2,
  "totalRequests": 150,
  "providerStats": [
    {
      "id": "s3-main",
      "type": "s3",
      "requestCount": 100,
      "errorCount": 2,
      "errorRate": "0.0200"
    },
    {
      "id": "r2-cloudflare",
      "type": "r2",
      "requestCount": 50,
      "errorCount": 0,
      "errorRate": "0.0000"
    }
  ]
}

Health Check

GET /health

Response:

{
  "status": "ok",
  "providers": 2,
  "timestamp": "2023-06-01T11:00:00.000Z"
}

Programmatic Usage

You can also generate presigned URLs programmatically:

// Generate upload URL
const uploadUrlInfo = await storagePresigner.generateUploadUrl({
  filename: 'example.jpg',
  contentType: 'image/jpeg',
  path: 'uploads/images',
  expiry: 1800
});

// Generate read URL
const readUrlInfo = await storagePresigner.generateReadUrl({
  key: 'uploads/images/uuid-example.jpg',
  providerId: 's3-main',
  expiry: 3600
});