A TypeScript SDK for uploading large files (10GB+) to a chunked upload server. Supports parallel uploads, resume capability, and phase-aware progress tracking.
- Large File Support: Upload files of any size (10GB+)
- Automatic Chunking: Files split into 50MB chunks (Cloudflare compatible)
- Parallel Uploads: Configurable concurrency for faster uploads
- Resumable: Continue interrupted uploads from where they left off
- Phase Progress Tracking:
uploading -> finalizing -> complete - Retry Logic: Automatic retry for failed chunks
- TypeScript: Full type definitions included
- Isomorphic: Works in both browser and Node.js
npm install chunked-uploader-sdkPOST /upload/{id}/completeis now async (202while finalizing,200when complete).getStatus()now returns phase and split progress:phaseupload_progress_percentfinalizing_progress_percentfinalization_error
onProgressnow includes phase-aware fields:phase,phaseProgress,uploadProgress,finalizingProgress
import { ChunkedUploader } from 'chunked-uploader-sdk';
const uploader = new ChunkedUploader({
baseUrl: 'https://upload.example.com',
apiKey: 'your-api-key',
});
// Upload a file with progress tracking
const result = await uploader.uploadFile(file, {
onProgress: (event) => {
console.log(`Progress: ${event.overallProgress.toFixed(1)}%`);
},
});
if (result.success) {
console.log('Upload complete:', result.finalPath);
} else {
console.error('Upload failed:', result.error);
}interface ChunkedUploaderConfig {
/** Base URL of the chunked upload server */
baseUrl: string;
/** API key for management endpoints */
apiKey: string;
/** Request timeout in milliseconds (default: 30000) */
timeout?: number;
/** Number of concurrent chunk uploads (default: 3) */
concurrency?: number;
/** Retry attempts for failed chunks (default: 3) */
retryAttempts?: number;
/** Delay between retries in milliseconds (default: 1000) */
retryDelay?: number;
/** Poll interval while waiting for finalization (default: 2000) */
finalizePollIntervalMs?: number;
/** Finalization timeout in milliseconds (default: 7200000 / 2h) */
finalizeTimeoutMs?: number;
/** Custom fetch implementation */
fetch?: typeof fetch;
}Upload a file with automatic chunking and parallel multi-part uploads.
The SDK sends multiple HTTP requests concurrently for different parts, maximizing upload throughput. Chunks are read on-demand (streaming) to avoid loading the entire file into memory.
const result = await uploader.uploadFile(file, {
webhookUrl: 'https://your-server.com/webhook',
concurrency: 5, // Upload 5 parts simultaneously (default: 3)
onProgress: (event) => {
console.log(`Part ${event.currentPart}/${event.totalParts}`);
console.log(`Overall: ${event.overallProgress}%`);
},
onPartComplete: (result) => {
console.log(`Part ${result.partNumber} ${result.success ? 'done' : 'failed'}`);
},
signal: abortController.signal, // For cancellation
});Parallel Upload Features:
- Concurrent requests: Multiple parts upload simultaneously (configurable)
- Streaming reads: Chunks read on-demand, not loaded into memory
- Independent retries: Failed parts retry without blocking others
- Progress tracking: Real-time progress for each part and overall
Resume an interrupted upload.
// Store part tokens from initial upload
const tokenMap = new Map<number, string>();
initResponse.parts.forEach(p => tokenMap.set(p.part, p.token));
// Later, resume the upload
const result = await uploader.resumeUpload(uploadId, file, {
partTokens: tokenMap,
onProgress: (event) => console.log(`${event.overallProgress}%`),
});Initialize an upload session manually.
The filename parameter can include a path (e.g., "videos/2024/december/large-video.mp4"). When a path is included, the server will store the file at that path.
// Simple filename (stored at default location)
const response = await uploader.initUpload('large-video.mp4', fileSize);
console.log(`Upload ID: ${response.file_id}`);
console.log(`Parts: ${response.total_parts}`);
// With custom path (stored at videos/2024/december/)
const response = await uploader.initUpload('videos/2024/december/large-video.mp4', fileSize);
console.log(`File will be stored at: ${response.file_id}`);Upload a single chunk.
const result = await uploader.uploadPart(
uploadId,
0,
partToken,
chunkData
);Get upload progress and status.
const status = await uploader.getStatus(uploadId, { includeParts: true });
console.log(`Phase: ${status.phase}`);
console.log(`Upload: ${status.upload_progress_percent}%`);
console.log(`Finalizing: ${status.finalizing_progress_percent}%`);
console.log(`Uploaded: ${status.uploaded_parts}/${status.total_parts}`);
// Find pending parts
const pending = (status.parts ?? []).filter(p => p.status === 'pending');Trigger finalization and wait until the upload reaches a terminal state.
The SDK handles async server completion (202 Accepted) by polling GET /upload/{id}/status.
const result = await uploader.completeUpload(uploadId);
console.log(`Status: ${result.status}`);
console.log(`Finalizing: ${result.finalizing_progress_percent}%`);
console.log(`File path: ${result.final_path}`);Cancel an upload and cleanup.
await uploader.cancelUpload(uploadId);Check server health.
const health = await uploader.healthCheck();
console.log(`Server status: ${health.status}`);import { ChunkedUploader } from 'chunked-uploader-sdk';
const uploader = new ChunkedUploader({
baseUrl: 'http://localhost:3000',
apiKey: 'your-api-key',
});
// Browser: File input
const input = document.querySelector('input[type="file"]') as HTMLInputElement;
input.addEventListener('change', async () => {
const file = input.files?.[0];
if (!file) return;
const result = await uploader.uploadFile(file);
console.log(result);
});For full control over the storage path, use initUpload with a path-prefixed filename:
async function uploadToPath(file: File, targetPath: string) {
const uploader = new ChunkedUploader({
baseUrl: 'http://localhost:3000',
apiKey: 'your-api-key',
});
// Include path in filename (e.g., "videos/2024/december/my-video.mp4")
const remoteFilename = `${targetPath}/${file.name}`;
// Initialize with custom path
const init = await uploader.initUpload(remoteFilename, file.size);
// Upload using resumeUpload (handles parallel uploads)
const tokenMap = new Map(init.parts.map(p => [p.part, p.token]));
const result = await uploader.resumeUpload(init.file_id, file, {
partTokens: tokenMap,
onProgress: (event) => {
console.log(`Progress: ${event.overallProgress.toFixed(1)}%`);
},
});
// Result: file stored at videos/2024/december/my-video.mp4
console.log('Final path:', result.finalPath);
}
// Usage
uploadToPath(file, 'videos/2024/december');const progressBar = document.querySelector('.progress-bar') as HTMLElement;
const statusText = document.querySelector('.status') as HTMLElement;
const result = await uploader.uploadFile(file, {
onProgress: (event) => {
progressBar.style.width = `${event.phaseProgress}%`;
statusText.textContent =
event.phase === 'uploading'
? `Uploading ${event.uploadProgress.toFixed(0)}%`
: event.phase === 'finalizing'
? `Finalizing ${event.finalizingProgress.toFixed(0)}%`
: 'Complete';
},
onPartComplete: (result) => {
if (!result.success) {
console.error(`Part ${result.partNumber} failed:`, result.error);
}
},
});const STORAGE_KEY = 'pending_upload';
interface StoredUpload {
uploadId: string;
tokens: [number, string][];
}
async function uploadWithResume(file: File) {
// Check for existing upload
const stored = localStorage.getItem(STORAGE_KEY);
if (stored) {
const data: StoredUpload = JSON.parse(stored);
const tokenMap = new Map(data.tokens);
try {
const result = await uploader.resumeUpload(data.uploadId, file, {
partTokens: tokenMap,
});
if (result.success) {
localStorage.removeItem(STORAGE_KEY);
return result;
}
} catch (error) {
console.log('Resume failed, starting fresh');
}
}
// Start new upload
const initResponse = await uploader.initUpload(file.name, file.size);
// Store tokens for resume
const toStore: StoredUpload = {
uploadId: initResponse.file_id,
tokens: initResponse.parts.map(p => [p.part, p.token]),
};
localStorage.setItem(STORAGE_KEY, JSON.stringify(toStore));
// Upload file
const tokenMap = new Map(initResponse.parts.map(p => [p.part, p.token]));
// Manual upload with stored tokens
const result = await uploader.resumeUpload(initResponse.file_id, file, {
partTokens: tokenMap,
});
if (result.success) {
localStorage.removeItem(STORAGE_KEY);
}
return result;
}const abortController = new AbortController();
// Cancel button
cancelButton.addEventListener('click', () => {
abortController.abort();
});
const result = await uploader.uploadFile(file, {
signal: abortController.signal,
onProgress: (event) => {
console.log(`${event.overallProgress}%`);
},
});
if (!result.success && result.error?.message === 'Upload aborted') {
console.log('Upload was cancelled');
}For maximum throughput on fast connections:
const uploader = new ChunkedUploader({
baseUrl: 'http://localhost:3000',
apiKey: 'your-api-key',
concurrency: 10, // Default concurrency
});
// Override per-upload for even more parallelism
const result = await uploader.uploadFile(largeFile, {
concurrency: 20, // 20 simultaneous part uploads!
onProgress: (event) => {
const mbUploaded = (event.uploadedParts * 50).toFixed(0);
console.log(`${mbUploaded}MB uploaded (${event.overallProgress.toFixed(1)}%)`);
},
});import { ChunkedUploader } from 'chunked-uploader-sdk';
import { readFile } from 'fs/promises';
const uploader = new ChunkedUploader({
baseUrl: 'http://localhost:3000',
apiKey: 'your-api-key',
concurrency: 5, // More concurrent uploads for server-side
});
async function uploadFromDisk(filePath: string) {
const buffer = await readFile(filePath);
const result = await uploader.uploadFile(buffer, {
onProgress: (event) => {
process.stdout.write(`\rProgress: ${event.overallProgress.toFixed(1)}%`);
},
});
console.log('\nUpload complete:', result);
}For full control over the upload process:
async function manualUpload(file: File) {
// 1. Initialize
const init = await uploader.initUpload(file.name, file.size);
const chunkSize = init.chunk_size;
// 2. Upload parts manually
for (const part of init.parts) {
const start = part.part * chunkSize;
const end = Math.min(start + chunkSize, file.size);
const chunk = file.slice(start, end);
const result = await uploader.uploadPart(
init.file_id,
part.part,
part.token,
chunk
);
console.log(`Part ${result.part_number} uploaded`);
}
// 3. Complete
const complete = await uploader.completeUpload(init.file_id);
console.log('File path:', complete.final_path);
}The SDK throws ChunkedUploaderError for API errors:
import { ChunkedUploaderError } from 'chunked-uploader-sdk';
try {
await uploader.uploadFile(file);
} catch (error) {
if (error instanceof ChunkedUploaderError) {
console.error('Upload error:', error.message);
console.error('Status code:', error.statusCode);
console.error('Error code:', error.code);
}
}This SDK is designed to work with the Chunked Upload Server.
Server API Endpoints:
POST /upload/init- Initialize upload (API Key auth)PUT /upload/{id}/part/{n}- Upload chunk (JWT auth per part)GET /upload/{id}/status- Get upload/finalization phase progress (API Key auth)POST /upload/{id}/complete- Start or check completion (API Key auth)DELETE /upload/{id}- Cancel upload (API Key auth)GET /health- Health check
- Server: chunked-uploader - A production-ready Rust HTTP server for resumable chunked uploads (10GB+) with support for local filesystem, SMB/NAS, and S3-compatible storage backends.
MIT