S3 remote build cache with per-project content digests#545
Open
oyvindberg wants to merge 8 commits intofeature-build-invalidatedfrom
Open
S3 remote build cache with per-project content digests#545oyvindberg wants to merge 8 commits intofeature-build-invalidatedfrom
oyvindberg wants to merge 8 commits intofeature-build-invalidatedfrom
Conversation
…in config Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Computes digest from project config + source/resource content + transitive dep digests. Uses git ls-tree for clean directories (fast, no file I/O), falls back to filesystem for dirty/generated files. Golden digest tests ensure stability across platforms and versions. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use git-compatible blob hashing (SHA-1 with blob header) for filesystem path, matching git ls-tree output. Relativize git ls-tree paths to source dir. Add test that creates a temp git repo, verifies uncommitted filesystem digest matches committed git ls-tree digest. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
S3Client: manual AWS Signature V4 signing, zero dependencies. TarGz: pure JDK tar.gz pack/unpack (no commons-compress needed). RemoteCache: pull pre-compiled classes from S3, push after building. Schema, docs, and sidebar updated. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use zinc's built-in RootPaths + RelativeReadMapper/RelativeWriteMapper
to relativize absolute paths in analysis files. Three roots: build dir
(sources + products), coursier cache (library JARs).
Analysis files now contain relative paths like ${0}/src/main/scala/Foo.scala
instead of /home/user/project/src/main/scala/Foo.scala, making them
portable across machines for remote cache.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
bleep config remote-cache setup — interactive prompt for AWS key/secret bleep config remote-cache clear — remove stored credentials Also supports AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY env vars (no setup needed). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
S3 remote build cache with per-project content digests
Depends on #544 (
bleep build invalidated).The CI dream
What's included
Model layer:
remote-cacheconfig inbleep.yaml(URI + region)remoteCacheCredentialsin~/.config/bleep/config.yamlBLEEP_REMOTE_CACHE_S3_ACCESS_KEY_ID/BLEEP_REMOTE_CACHE_S3_SECRET_ACCESS_KEYenv varsPer-project content digests (
ProjectDigest):git ls-treefor clean directories (precomputed hashes, no file I/O) — fast on large reposSHA-1("blob <size>\0" + content)) so both paths produce identical digestsS3 client:
Remote cache commands:
bleep remote-cache pull— download + extract tar.gz archives per projectbleep remote-cache push— tar.gz classes + zinc analysis, uploadPortable zinc analysis:
ZincBridgenow usesReadWriteMapperswithRootPaths(build dir, coursier cache, products dir)${0}/src/main/scala/Foo.scala) instead of absolute pathsCredential management:
bleep config remote-cache setup— interactive TUI promptbleep config remote-cache clear— remove stored credentialsDocumentation:
usage/remote-cache.mdxwith setup, usage, GitHub Actions, GitLab CI, MinIO/R2 examplesConfig examples
bleep.yaml:
~/.config/bleep/config.yaml:
Or env vars:
Test plan
bleep build invalidatedcommand for CI-optimized rebuilds #544)bleep compileclean build succeeds🤖 Generated with Claude Code