Conversation
commit 047758c Merge: 6290f5c e3b0828 Author: Lev Brouk <lev@e2b.dev> Date: Thu Feb 19 10:19:44 2026 -0800 Merge branch 'main' of github.com:e2b-dev/infra into lev-compression-readonly-v2 commit 6290f5c Author: Lev Brouk <lev@e2b.dev> Date: Thu Feb 19 10:16:09 2026 -0800 refactor: simplify fetchSession to use terminated() pattern from StreamingChunker Replace fetchState enum with terminated() method (fetchErr != nil || bytesReady == chunkLen), use close(ch) for signaling, and defer fetchMap cleanup in runFetch. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 2257f00 Merge: a04c603 45bad1f Author: Lev Brouk <lev@e2b.dev> Date: Thu Feb 19 01:36:54 2026 -0800 Merge branch 'lev-compression-readonly-v2' of github.com:levb/infra into lev-compression-readonly-v2 # Conflicts: # packages/orchestrator/internal/sandbox/block/chunk.go # packages/orchestrator/internal/sandbox/block/chunker_concurrency_test.go # packages/orchestrator/internal/sandbox/block/device.go # packages/orchestrator/internal/sandbox/block/fetch_session.go # packages/shared/pkg/storage/frame_table.go commit a04c603 Author: Lev Brouk <lev@e2b.dev> Date: Thu Feb 19 01:31:15 2026 -0800 refactor: unify chunker fetch paths via GetFrame Both compressed and uncompressed chunker paths now use GetFrame with an onRead callback, eliminating the separate OpenSeekable/OpenRangeReader dependency. This removes ChunkerStorage interface, objectType field, and the duplicate getOrCreate/runFetch method pairs. Added readSize parameter to FrameGetter/StorageProvider, enabling configurable progressive read granularity (from FF minReadBatchSizeKB) for both paths. Uncompressed GetFrame now supports onRead callbacks via readProgressive. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit d6d38a1 Author: Lev Brouk <lev@e2b.dev> Date: Thu Feb 19 00:46:51 2026 -0800 fix: V4 header serialize/deserialize symmetry and LZ4 error handling Fix serializer writing StartAt bytes for FrameTable with CompressionNone + 0 frames (packed value 0), while deserializer skips them — corrupting subsequent mappings. Also make CompressLZ4 return an error on incompressible data instead of raw bytes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 379d792 Author: Lev Brouk <lev@e2b.dev> Date: Thu Feb 19 00:02:56 2026 -0800 feat: wire chunker-config feature flag for useCompressedAssets and minReadBatchSizeKB Replace the hardcoded `var useCompressedAssets = true` global with the `chunker-config` JSON feature flag from LaunchDarkly. The FF client is threaded through Cache → storageTemplate → NewStorage → File → StorageDiff → Chunker. useCompressedAssets is read at template-fetch time; minReadBatchSizeKB is read JIT in each uncompressed fetch. Removes the dead `useStreaming` flag field. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit c437636 Author: Lev Brouk <lev@e2b.dev> Date: Wed Feb 18 23:38:08 2026 -0800 docs: clarify OpenRangeReader is uncompressed-only Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit b8bd92d Author: Lev Brouk <lev@e2b.dev> Date: Wed Feb 18 23:36:07 2026 -0800 style: auto-fix formatting from golangci-lint Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 14a77e4 Author: Lev Brouk <lev@e2b.dev> Date: Wed Feb 18 23:35:29 2026 -0800 fix: resolve all remaining lint issues - Use require.NoError instead of assert.NoError in cache_test.go (testifylint) - Remove error return from propagateDependencyFrames since it always returns nil (unparam) - Remove unused fetchAllTemplates function (unused) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 7f1a333 Author: Lev Brouk <lev@e2b.dev> Date: Wed Feb 18 23:33:12 2026 -0800 fix: nilerr lint in probeAssets, export MetadataVersionCompressed const - Add //nolint:nilerr directives for intentional error swallowing in probeAssets - Export MetadataVersionCompressed and use it in compress-build instead of hardcoded 4 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 9cba1be Author: Lev Brouk <lev@e2b.dev> Date: Wed Feb 18 23:30:48 2026 -0800 refactor: address PR review — cleanups, renames, move compression constants - Remove dev comment from orchestrator/main.go - Restore test-integration target to match main - Remove dev-only compress-build make target - Remove trailing comment from Diff interface - Rename loadV4orV3Header → loadHeaderPreferV4 - Inline loadHeader into loadV3Header (now takes path directly) - Move compression constants/types/options to compressed_upload.go Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 6b4b528 Author: Lev Brouk <lev@e2b.dev> Date: Wed Feb 18 23:18:16 2026 -0800 refactor: use io interfaces in device.go, errgroup in probeAssets Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 494b915 Merge: 8d4ec3e a7dfa0a Author: Lev Brouk <lev@e2b.dev> Date: Wed Feb 18 13:56:27 2026 -0800 Merge branch 'main' of github.com:e2b-dev/infra into lev-compression-readonly-v2 # Conflicts: # packages/orchestrator/internal/sandbox/block/chunk.go # packages/orchestrator/internal/sandbox/build/storage_diff.go # packages/shared/go.mod # packages/shared/pkg/storage/storage_cache_seekable.go commit 45bad1f Author: Lev Brouk <lev@e2b.dev> Date: Wed Feb 18 13:50:44 2026 -0800 tidy comments: remove restated-code comments, keep invariants and intent Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 8d4ec3e Author: Lev Brouk <lev@e2b.dev> Date: Wed Feb 18 11:27:13 2026 -0800 refactor: rename block.FramedReader → Reader, Slice/ReadAt → GetBlock/ReadBlock Address PR review comments: - Rename FramedReader → Reader, Slice → GetBlock, ReadAt → ReadBlock for the 4-arg compression-aware interface (block.Slicer unchanged) - Restore original variable names in newStorageDiff (storagePath, storageObjectType) - Remove dead Size() methods on StorageDiff and localDiff Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit e002bf3 Merge: d4c7a1f 17bc657 Author: Lev Brouk <lev@e2b.dev> Date: Wed Feb 18 11:00:12 2026 -0800 Merge remote-tracking branch 'levb/lev-compression-readonly-v2' into lev-compression-readonly-v2 # Conflicts: # packages/orchestrator/cmd/compress-build/main.go # packages/orchestrator/internal/sandbox/block/chunk_decompress.go commit d4c7a1f Author: Lev Brouk <lev@e2b.dev> Date: Wed Feb 18 10:55:52 2026 -0800 refactor: reduce diff footprint — restore device.go, consolidate chunker, simplify Size() - Restore device.go with BytesNotAvailableError, Slicer, ReadonlyDevice, Device - Add FramedReader interface for compression-aware read methods - Remove Chunker interface (single impl), rename DecompressMMapChunker → Chunker - Merge chunk_decompress.go back into chunk.go - Diff interface embeds block.FramedReader instead of listing methods - Revert SeekableReader.Size() from (int64, int64, error) to (int64, error) - Simplify AssetInfo: LZ4Size/ZstSize int64 → HasLZ4/HasZst bool - Remove debug-only ChunkerCreations metric - Add compression timing output to compress-build (per-frame + summary) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 17bc657 Author: Lev Brouk <lev@e2b.dev> Date: Wed Feb 18 10:38:46 2026 -0800 refactor: v4-prefixed asset naming, color-coded compression output, cleanup unused - Rename compressed assets: memfile.lz4 → v4.memfile.lz4, memfile.compressed.header.lz4 → v4.memfile.header.lz4 - Add color-coded compression ratio matrix to inspect-build (red/yellow/default/green/cyan/blue by compressibility) - Color-code per-frame ratios in compress-build verbose output - Add V4DataName/V4HeaderName/V4DataPath helpers, remove unused Path(), CompressedPath(), V4HeaderPath() methods - Privatize v4Prefix, v4HeaderSuffix, splitPath (internal only) - Gitignore prebuilt compress-build/inspect-build binaries Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 2b61a6e Author: Lev Brouk <lev@e2b.dev> Date: Wed Feb 18 09:48:48 2026 -0800 feat: compressed-only read path — Size() returns (uncompressed, compressed), metadata-driven asset probing Extend the Size() interface from (int64, error) to (uncompressed, compressed int64, err error) across all storage backends (GCS, AWS, FS, cache, mocks). Compressed objects store an "uncompressed-size" custom metadata key so the orchestrator can derive the mmap allocation size without requiring the uncompressed object to exist in storage. Key changes: - Size() returns both uncompressed and compressed sizes from GCS/S3 metadata - probeAssets() handles compressed-only assets via HasUncompressed field on AssetInfo - NFS cache size.txt stores "uncompressed compressed" pair for round-tripping - Header version reverted to V3 for uncompressed builds, V4 for compressed - Frame size cap at 16 MiB uncompressed (DefaultMaxFrameUncompressedSize) - compress-build: verbose flag, hex formatting, max-frame-u flag, GCS metadata uploads - inspect-build: hex formatting, uncompressed-size metadata display Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 7444f22 Author: Lev Brouk <lev@e2b.dev> Date: Wed Feb 18 06:02:16 2026 -0800 refactor: add onProgress callback, explicit Init(), and probeAssets error handling - GetFrame supports onProgress for pipelined fetch+decompress - StorageDiff.Init() explicit initialization pattern - probeAssets properly handles Size() errors - Remove unused device.go - Add DiffStore cache tests and Init support Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 4ff3865 Author: Lev Brouk <lev@e2b.dev> Date: Sun Feb 15 18:55:17 2026 -0800 feat: add compress-build command and compressed upload utilities Add compress-build CLI tool for compressing build artifacts, along with supporting LZ4 compression and compressed upload helpers in the storage package. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 144577f Merge: a242336 1712a15 Author: Lev Brouk <lev@e2b.dev> Date: Sun Feb 15 18:54:07 2026 -0800 refactor: unify chunkers into single dual-mode DecompressMMapChunker Remove CompressedFileLRUChunker, FullFetchChunker, and FrameCache in favor of a single DecompressMMapChunker that serves both compressed and uncompressed callers from a shared mmap cache. The routing decision is made per-request based on FrameTable presence and asset availability. At init time, three asset variants (uncompressed, .lz4, .zst) are probed in parallel via Size() calls. The AssetInfo struct (BasePath + sizes) replaces the old feature-flag-driven chunker selection. Compressed paths are derived from BasePath + compression suffix when needed. Key fixes discovered during testing: - Stale err variable in Slice causing spurious BytesNotAvailableError on the uncompressed path - fetchMap key collision between compressed (frame offset) and uncompressed (chunk offset) sessions, fixed with composite fetchKey - Stale session reuse race: removeFetchSession now called before complete()/fail() to prevent woken waiters from finding old sessions Also removes the ChunkerMode feature flag and golang-lru/v2 dependency. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit a242336 Author: Lev Brouk <lev@e2b.dev> Date: Sun Feb 15 06:34:04 2026 -0800 refactor: parallelize disk persist and decompression on first compressed frame fetch On first fetch, the disk write and decompression both only read from the GCS buffer. Run them concurrently via errgroup instead of sequentially. Extract FrameCache.Persist for the atomic disk write, consolidate fetchDecompressAndCache into getOrFetchFrame, and remove the now-redundant WaitMap from FrameCache (deduplication handled by fetchMap). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 3c33f28 Author: Lev Brouk <lev@e2b.dev> Date: Sun Feb 15 05:49:00 2026 -0800 chore: rename chunk_compress.go to chunk_compressed.go Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit cf67689 Author: Lev Brouk <lev@e2b.dev> Date: Sun Feb 15 05:46:34 2026 -0800 refactor: restore legacy ReadAt fetch path in UncompressedMMapChunker Use SeekableReader.ReadAt instead of FrameGetter.GetFrame, matching main's Chunker. Restores RemoteReadsTimerFactory timing and readBytes length check. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 82ccb3c Author: Lev Brouk <lev@e2b.dev> Date: Sun Feb 15 05:42:07 2026 -0800 refactor: rename chunkers, simplify FrameCache, reduce test parallelism - Rename CompressMMapLRUChunker -> CompressedFileLRUChunker (no mmap) - Rename FrameFileCache -> FrameCache, give it storage ownership - Rename virtSize -> size in DecompressMMapChunker - Simplify CompressedFileLRUChunker.Slice error handling (no fast/slow) - Replace FrameCache.FileSize dir scan with atomic sizeOnDisk counter - Fix FrameTable.Size() named returns (U,C -> uncompressed,compressed) - Reduce integration test parallelism from 4 to 2 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 4178cad Author: Lev Brouk <lev@e2b.dev> Date: Sun Feb 15 03:58:13 2026 -0800 refactor: replace singleflight with WaitMap, inline LRU, add FrameFileCache, consolidate decompression - Upgrade golang-lru to v2 generics, inline LRU directly - Replace singleflight.Group with utils.WaitMap in all chunkers - Replace MMapFrameCache with per-frame FrameFileCache - Add DecompressReader/DecompressFrame helpers in storage - Simplify DecompressMMapChunker to reject cross-frame reads - Switch both chunker types to decompressMMapChunker Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit cd7592e Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 17:57:35 2026 -0800 fix: add missing error check in DeserializeBytes for non-EOF errors When binary.Read returns io.ErrUnexpectedEOF (not enough bytes for a full v3SerializableBuildMap record), the error was silently swallowed and garbage partial data was appended to mappings. This caused panics in NewHeader when astronomically large offsets were passed to bitset. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 5dae56f Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 15:51:06 2026 -0800 refactor: move chunker type controls to consumers Move ChunkerType enum and UseCompressedAssets flag out of shared storage package into the orchestrator packages where they're actually used. These will be replaced with feature flags. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit f311985 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 15:42:58 2026 -0800 refactor: replace TotalUncompressedSize/TotalCompressedSize with Size() (U, C) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit f728045 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 15:42:08 2026 -0800 refactor: move frame table types from storage.go to frame_table.go Renamed compress_frame_table.go → frame_table.go and moved all frame-related types (CompressionType, FrameOffset, FrameSize, Range, FrameTable, ChunkerType, FrameGetter, IsCompressed) from storage.go into it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 5559bd7 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 15:40:12 2026 -0800 test: trim redundant V4_NoCompression test, compact V3 assertions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit d4cfb20 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 15:36:33 2026 -0800 typo commit 7636704 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 15:34:08 2026 -0800 white space commit 936aa20 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 15:33:24 2026 -0800 fix: set metadataVersion back to 3 for new headers V4 read path is ready but we only create/store V3 headers for now. V4 serialization code retained for upcoming CLI tooling. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 8850c02 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 14:33:29 2026 -0800 refactor: add DeserializeV4, move LZ4 into header package - DeserializeBytes reverted to v0-v3 only (matches main) - DeserializeV4 handles LZ4 decompression + v4 frame table parsing - Renamed .compressed.header.lz4 → .header.v4 - Removed storage/lz4.go (DecompressLZ4/MaxCompressedHeaderSize unused) - template/storage.go no longer imports compression utilities Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit ebd2ba3 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 14:19:50 2026 -0800 refactor: rename header loaders to loadV3Header and loadV4orV3Header Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 6034e73 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 14:06:06 2026 -0800 refactor: extract loadHeaderDefault and loadHeaderWithCompressed Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 1df233b Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 14:05:20 2026 -0800 refactor: extract loadHeader and loadCompressedHeader helpers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit d8f28d1 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 13:57:41 2026 -0800 fix: align struct literal whitespace in storage_diff.go Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 5e022c2 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 13:56:11 2026 -0800 refactor: add storage.LoadBlob helper for OpenBlob+GetBlob pattern Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 4b173b6 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 13:31:51 2026 -0800 refactor: revert cosmetic renames to reduce diff from main - objectPath → storagePath (storage_diff.go field, matches main) - in → reader (serialization.go local var, matches main) - storage → base (chunk.go field, matches main) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit bebc088 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 12:44:10 2026 -0800 refactor: rename buildId to buildID in getBuild for consistency with main Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit eddc011 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 12:37:07 2026 -0800 revert: Chunker.Chunk back to Chunker.Slice Revert the rename to minimize diff from main. Naming cleanup deferred to a separate PR. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit c571b69 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 12:33:49 2026 -0800 refactor: rename Chunker.Chunk length param to blockSize The Chunk method is always called with exactly one block (page size), never an arbitrary range. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit b3d222e Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 12:31:39 2026 -0800 docs: trim Chunker interface comments Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit b57ff42 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 12:28:04 2026 -0800 refactor: rename Chunker.Slice to Chunker.Chunk Rename to better communicate the contract: callers must request block-aligned ranges within a single storage chunk, never arbitrary byte ranges. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 396cb63 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 12:19:28 2026 -0800 refactor: remove stats counters from chunkers Remove atomic stats counters (slices, fetchBytes, decompressions, etc.) from all three chunker implementations. Instrumentation will be added separately via the existing OTEL metrics infrastructure. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 085e582 Author: Lev Brouk <lev@e2b.dev> Date: Sat Feb 14 12:15:39 2026 -0800 feat: compressed asset read path Add support for reading compressed (LZ4/zstd) template assets with frame-based decompression. Key changes: - Add GetFrame to StorageProvider for range-based frame reads with optional decompression - Add DeserializeBytes for direct []byte header deserialization; keep Deserialize(ctx, Blob) for existing callers - Implement v4 header serialization with FrameTable (compression type, frame offsets/sizes per mapping) - Add three chunker implementations: UncompressedMMapChunker, DecompressMMapChunker, and CompressMMapLRUChunker with two-level caching - Lazy chunker initialization in StorageDiff via sync.Once - GetShiftedMapping returns *BuildMap carrying FrameTable through read path - Compressed header loading in template storage with LZ4 decompression - Feature-flagged via UseCompressedAssets (default off) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 1712a15 Author: Lev Brouk <lev@e2b.dev> Date: Fri Feb 13 14:20:47 2026 -0800 PR feedback commit 6a3f58d Author: Lev Brouk <lev@e2b.dev> Date: Fri Feb 13 13:40:08 2026 -0800 missed lint commit 1133ce1 Author: Lev Brouk <lev@e2b.dev> Date: Fri Feb 13 13:34:56 2026 -0800 PR feedback: added a fetch session test commit 5df2962 Merge: cd8f8c8 364f8c6 Author: Lev Brouk <lev@e2b.dev> Date: Fri Feb 13 13:02:56 2026 -0800 Merge branch 'main' of github.com:e2b-dev/infra into lev-slice-return-faster commit cd8f8c8 Merge: 80eaa83 b4538be Author: Lev Brouk <lev@e2b.dev> Date: Wed Feb 11 06:41:29 2026 -0800 Merge branch 'main' of github.com:e2b-dev/infra into lev-slice-return-faster commit 80eaa83 Merge: ec913e3 48e117c Author: Lev Brouk <lev@e2b.dev> Date: Tue Feb 10 14:58:22 2026 -0800 Merge branch 'main' of github.com:e2b-dev/infra into lev-slice-return-faster # Conflicts: # packages/shared/pkg/feature-flags/flags.go commit ec913e3 Merge: f4b9770 204c3e0 Author: Lev Brouk <lev@e2b.dev> Date: Tue Feb 10 13:35:59 2026 -0800 Merge branch 'main' of github.com:e2b-dev/infra into lev-slice-return-faster commit f4b9770 Merge: 1073c6d 9dca469 Author: Lev Brouk <lev@e2b.dev> Date: Mon Feb 9 14:31:30 2026 -0800 Merge branch 'main' of github.com:e2b-dev/infra into lev-slice-return-faster commit 1073c6d Author: Lev Brouk <lev@e2b.dev> Date: Mon Feb 9 14:30:36 2026 -0800 feat: add use-streaming-chunker feature flag for safe rollout Extract Chunker interface from the concrete FullFetchChunker (renamed from Chunker) so StorageDiff can switch implementations at runtime. When the use-streaming-chunker flag is enabled, StreamingChunker is used; otherwise the original FullFetchChunker is used (safe default). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit c56eb0e Author: Lev Brouk <lev@e2b.dev> Date: Mon Feb 9 14:15:11 2026 -0800 revert: remove temporary fmt.Printf instrumentation from Slice() Benchmarking complete — StreamingChunker shows ~20% p50 improvement on sandbox Create times vs baseline Chunker. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit afc0707 Author: Lev Brouk <lev@e2b.dev> Date: Mon Feb 9 13:39:57 2026 -0800 temp: add sampled fmt.Printf instrumentation to Slice() for perf comparison Logs every 100th Slice() call (hit/miss) and all failures. Temporary — will be removed after cluster benchmarking. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 3c06282 Author: Lev Brouk <lev@e2b.dev> Date: Mon Feb 9 08:44:37 2026 -0800 fix: address PR review feedback - Remove ReadAt fallback in cachedSeekable.OpenRangeReader: Seekable embeds StreamingReader, so the type assertion was dead code - Reuse buffer in StreamingChunker.WriteTo instead of allocating 4MB per iteration Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit dd6d9e6 Author: Lev Brouk <lev@e2b.dev> Date: Mon Feb 9 08:36:05 2026 -0800 feat: switch StorageDiff to StreamingChunker Wire StreamingChunker into the single production call site (StorageDiff.Init). All UFFD page faults, NBD reads, and prefetch operations now use progressive chunk fetching. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 987a593 Author: Lev Brouk <lev@e2b.dev> Date: Mon Feb 9 08:13:25 2026 -0800 test: replace analytical benchmark with wall-clock measurements The previous benchmark used a zero-latency upstream and reported analytically computed latency predictions. Replace with a bandwidth-limited upstream (real time.Sleep) tuned to observed production latencies: GCS: 20ms TTFB + 100 MB/s → ~62ms per 4MB chunk (observed ~60ms) NFS: 1ms TTFB + 500 MB/s → ~9ms per 4MB chunk (observed ~9-10ms) Results confirm the analytical predictions and additionally surface a ~5ms constant overhead in the old Chunker's post-transfer path (batch setIsCached + WaitMap signaling) that the analytical model could not capture. Also reduces concurrent callers from 20 to 3 to match realistic UFFD/NBD concurrency (1-2 vCPU faults + 1 NBD request), and pre-generates a shared offset sequence across all sub-benchmarks for direct comparability. Results (300 iterations × 3 counts, 3 concurrent callers): GCS/StreamingChunker 42790 avg-us/caller 63260 worst-us/caller GCS/StreamingChunker 42776 avg-us/caller 63221 worst-us/caller GCS/StreamingChunker 42811 avg-us/caller 63400 worst-us/caller GCS/Chunker 66204 avg-us/caller 68199 worst-us/caller GCS/Chunker 66351 avg-us/caller 68814 worst-us/caller GCS/Chunker 66261 avg-us/caller 67971 worst-us/caller NFS/StreamingChunker 6038 avg-us/caller 10387 worst-us/caller NFS/StreamingChunker 6020 avg-us/caller 10392 worst-us/caller NFS/StreamingChunker 6041 avg-us/caller 10434 worst-us/caller NFS/Chunker 13812 avg-us/caller 15299 worst-us/caller NFS/Chunker 13839 avg-us/caller 15880 worst-us/caller NFS/Chunker 13773 avg-us/caller 15514 worst-us/caller Backend Chunker Avg Worst Avg speedup GCS StreamingChunker 42.8ms 63.3ms 1.55x GCS Chunker (baseline) 66.3ms 68.3ms — NFS StreamingChunker 6.0ms 10.4ms 2.3x NFS Chunker (baseline) 13.8ms 15.5ms — The ~5ms worst-case gap (constant across both backends) is fixed overhead in the old Chunker: burst of 1024 sync.Map.Store calls after ReadAt + WaitMap channel signaling. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit c2f0081 Author: Lev Brouk <lev@e2b.dev> Date: Mon Feb 9 07:22:39 2026 -0800 fix: harden StreamingChunker and keep it out of production paths Revert storage_diff.go to use the existing Chunker so the StreamingChunker is not yet on any production code path. StreamingChunker improvements: - Sort waiters by endByte for O(satisfied) notification instead of O(all) - Add 60s fetch timeout and thread context through runFetch/progressiveRead - Add panic recovery in runFetch so waiters never hang on panics - Add cancelOnCloseReader to GCS OpenRangeReader for proper context cleanup - Add nolint:containedctx to cacheWriteThroughReader.ctx - Add TestStreamingChunker_PanicRecovery test Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> commit 3ba0a05 Author: Lev Brouk <lev@e2b.dev> Date: Fri Feb 6 16:17:13 2026 -0800 feat: add StreamingChunker for progressive 4MB chunk fetching Replace the bulk Chunker with a StreamingChunker that reads data progressively from upstream (GCS/NFS), notifying blocked callers as soon as their specific byte range arrives instead of waiting for the entire 4MB chunk. Benchmarks show ~2.3x lower average caller latency under random access patterns (~4.5ms vs ~10.2ms simulated GCS). - Add StreamingReader interface (OpenRangeReader) to storage package, implemented on all backends (GCS, FS, AWS, NFS cache) - Add read-through caching on cachedSeekable.OpenRangeReader - Embed StreamingReader in Seekable interface - Switch StorageDiff from Chunker to StreamingChunker Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Match zstd window size (2 MiB) between raw and framed encoders
- Set concurrency=1 on all encode/decode paths
- Pre-allocate shared decode buffer to eliminate allocation skew
- Use per-codec level sets: LZ4 {0, 9}, zstd {1-4} (library constants)
- Add Raw Size, Frm Size, Size OH columns for compressed size comparison
- Add Dec/Frm column showing average decompression time per frame
- Stream results row-by-row as each codec/level completes
- Fix SuppressNoisyLogs silencing log.Fatalf error messages
- Skip zstd level 0 (invalid in klauspost/compress)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements dual-write (uncompressed + compressed) uploads during template builds, gated by the compress-config feature flag. Key changes: - MultipartUploader implements PartUploader for CompressStream integration - StoreFileCompressed added to StorageProvider with GCP, FS, cache, AWS (stub) - NFS cache write-through for compressed frames via OnFrameReady callback - NFS cache read-through for compressed GetFrame (cache compressed, decompress on read) - Two-phase upload in LayerExecutor: data files → header finalization → cache index - PendingFrameTables collects frame tables across layers for header serialization - compress-config JSON flag consolidates all compression options (type, level, sizes) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… up FF naming - Remove ChunkSize from FramedUploadOptions and feature flag; define FrameAlignmentSize as a package-level const (= MemoryChunkSize) that guarantees no chunker request (UFFD, NBD, prefetch) crosses a frame boundary. - Rename FF fields: frameTargetMB, frameMaxUncompressedMB, uploadPartTargetMB. - Replace manual readChunk loop with io.ReadFull. - Log chunker-config and compress-config FF values on orchestrator start. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…lean up - Remove unused objectPath param from rangeReadFunc; providers pass openRangeReader directly instead of wrapping in a closure - Change openRangeReader to take length int (callers always cast) - Add GCS GetFrame telemetry via googleReadTimerFactory - Rename cache op constants to match method names: cacheOpWrite→cacheOpPut, cacheOpWriteFromFileSystem→cacheOpStoreFile - Delete CompressConfig struct; GetUploadOptions reads the FF flag directly and returns nil for disabled compression or invalid CompressionType - Push feature-flags client into TemplateBuild so callers don't thread compressOpts through method signatures - Simplify multi-layer upload pipeline: remove redundant dataFileWg barrier from UploadTracker; waitForPreviousUploads already serializes headers - Rename .zst→.zstd suffixes and spell out abbreviated variable names - Remove all [TIMING] fmt.Printf debug prints - Remove unused limitedFileReader type, pre-built inspect-build binary Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
…lap prevention Replace separate fetchMap (session deduplication) and regionLock (overlap prevention) with a single slot-based regionLock that stores *fetchSession pointers in MemoryChunkSize-aligned slots. Single-slot requests join existing sessions; multi-slot requests wait for all occupied slots to clear. Also rename StorageDiff.flags to featureFlags for clarity. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rename TemplateBuild.Upload → UploadAll, UploadData → UploadExceptV4Headers - Rename Snapshot.UploadDataFiles → UploadLayerData (multi-layer intent) - Delete DataUploadResult and UploadDataFilesResult; return (memFT, rootFT, err) directly - Add applyFrameTablesForBuild + uploadCompressedHeadersForBuild for single-layer path — applies FTs directly by buildID, no PendingFrameTables - Extract serializeAndUploadHeader shared by both upload paths - Inline uploadLayerAsync into PauseAndUpload Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ables TemplateBuild now takes *Snapshot directly (extracting paths from Diffs), owns the PendingFrameTables lifecycle, and exposes a simplified API: UploadAll (synchronous), UploadExceptV4Headers, UploadV4Header. Deletes Snapshot.Upload/UploadLayerExceptV4Headers and collapses 5 V4 header upload variants into one code path. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Restore validateGetFrameParams (from lev-zstd-compression) at the GetFrame entry point and use makeFrameFilename for the inline Sprintf in getFrameCompressed. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…a into lev-compression-readonly-v3
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Range() was zero-initializing currentOffset instead of starting from ft.StartAt, causing wrong frame selection on subset FrameTables. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add sections F (failure modes + unresolved questions) and G (cost/benefit from staging bucket sample: memfile 4.0x, storage/CPU/memory/net analysis). Fix orphaned "The runtime stack is:" sentence. Reorder NFS cache TODO to #1. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Uncompressed reads (ft=nil) previously skipped alignment/bounds validation entirely because checks were gated behind `if decompress`. Now validateGetFrameParams checks IsCompressed(frameTable) instead, so uncompressed reads always enforce chunk alignment and buffer size. Also adds a Debug log on uncompressed NFS cache miss for parity with the compressed path, and updates compression-architecture.md TODOs (removes resolved NFS cache passthrough item, adds write-through TODO). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add three new items to the compression architecture doc: - Compressed-only write mode (skip uncompressed uploads) - Purity enforcement (no mixed compressed/uncompressed layer stacks) - Sync vs async layer compression timing and purity implications Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Inline testFrameSize constant directly in the function body to fix unparam lint error. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…sessions Eliminate two major sources of contention on the GetBlock hot path: the sync.Map block-presence tracker (70ms in pprof) and the slot-based regionLock, replacing them with simpler, faster primitives. --- Block presence tracking (cache.go) --- Replace dual-map tracking (sync.Map `dirty` + []atomic.Bool `cached`) with a single pre-allocated `dirty []atomic.Bool`. Each block transitions false→true exactly once — lock-free O(1) read and write, zero contention. No Cache instance has blocks that are "present but not dirty": - Overlay cache (WriteAt): every present block is dirty - Chunker cache (fetches): ExportToDiff is never called - copyProcessMemory cache: all blocks from initial snapshot Changes: - Remove `dirty sync.Map` field, rename `cached` → `dirty` - Merge `setIsCachedDirty()` into `setIsCached()` - Rewrite `dirtySortedKeys()` as linear scan (sorted by construction) - Update callers in WriteAtWithoutLock and copyProcessMemory --- Session management (chunk.go, fetch_session.go, region_lock.go) --- Replace regionLock (slot-based, MemoryChunkSize-aligned, with spin-wait) with a simple session list behind a mutex. At most ~4-8 sessions are active at a time, so a linear scan is sufficient. This removes 135 lines of complex slot coordination code. Replace per-waiter channels (sorted insertion, binary search, individual notification) with a single broadcast channel per session. On advance(), close the current channel and allocate a fresh one — all waiters wake and re-check bytesReady atomically. Simpler, fewer allocations under fan-out. --- Metric attributes (chunk.go, meters.go) --- Restructure precomputed OTEL attributes into a `precomputedAttrs` struct with named fields (successFromCache, successFromRemote, failCacheRead, failRemoteFetch, failLocalReadAgain) instead of positional variables. Add `Stopwatch.Record()` as a unified method for precomputed attributes (replaces SuccessPrecomputed). Export `telemetry.Success` / `telemetry.Failure` attribute constants. Decouple readSize from blockSize to avoid broadcast-wake storms when blockSize is small. Raise defaultMinReadBatchSize from 16KB to 256KB. --- Test simplification (chunker_test.go, chunk_bench_test.go) --- Unify testFrameGetter, testUncompressedStorage, and slowFramedFile into a single `slowFrameGetter` that serves raw data with optional ttfb/bandwidth simulation. Chunker always passes decompress=true, so real LZ4 round-tripping in test fakes was unnecessary. Add `storage.CompressBytes()` convenience wrapper and `memPartUploader` to shared/pkg/storage, replacing duplicated bufferPartUploader in bench. Simplify makeCompressedTestData to build synthetic FrameTables without real compression. Drop Zstd level 2 from benchmarks (levels 1 and 3 suffice). Remove redundant comments and section dividers. 8 files changed, 386 insertions(+), 725 deletions(-) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Port main's fullFetchChunker (WaitMap-based, no progressive delivery) as an unexported benchmark-only type. Rename chunk.go → chunk_framed.go to make room. Refactor all benchmarks into shared runners (runColdLeaf, runCacheHitLeaf) so Legacy, Uncompressed, and compressed codecs use identical benchmark loops. Hierarchy: profile → frame → block → mode. Fix benchmark data generator: use 1–16 byte runs instead of 1–4096 so compression ratio stays under ~4x and frame counts scale properly with TargetFrameSize. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…lient mock, fix chunker_test - Add BlocksTimerFactory to metrics.Metrics, alias SlicesTimerFactory to it - Generate MockFlagsClient for block package benchmarks - Fix truncated NewChunker call in TestChunker_BackgroundFetch - Fix missing eg.Wait() in TestChunker_ConcurrentDifferentOffsets - Widen newTestMetrics to accept testing.TB for bench compatibility - Re-apply bench metrics: U-MB/op, U-MB/s, C-MB/op, fetches/op Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP, distilled from and supersedes #1627