Skip to content

S3 remote build cache with per-project content digests#545

Open
oyvindberg wants to merge 8 commits intofeature-build-invalidatedfrom
feature-remote-cache
Open

S3 remote build cache with per-project content digests#545
oyvindberg wants to merge 8 commits intofeature-build-invalidatedfrom
feature-remote-cache

Conversation

@oyvindberg
Copy link
Copy Markdown
Owner

@oyvindberg oyvindberg commented Apr 4, 2026

S3 remote build cache with per-project content digests

Depends on #544 (bleep build invalidated).

The CI dream

bleep remote-cache pull                                        # restore cached classes
bleep build invalidated --base origin/main | xargs bleep compile  # only compile what changed
bleep remote-cache push                                        # cache results for next run

What's included

Model layer:

  • remote-cache config in bleep.yaml (URI + region)
  • remoteCacheCredentials in ~/.config/bleep/config.yaml
  • BLEEP_REMOTE_CACHE_S3_ACCESS_KEY_ID / BLEEP_REMOTE_CACHE_S3_SECRET_ACCESS_KEY env vars
  • JSON schema updated

Per-project content digests (ProjectDigest):

  • SHA-256 from project config + source content + resource content + transitive dep digests
  • Uses git ls-tree for clean directories (precomputed hashes, no file I/O) — fast on large repos
  • Falls back to filesystem hashing for dirty/generated files
  • Filesystem hashing uses git-compatible blob hashes (SHA-1("blob <size>\0" + content)) so both paths produce identical digests
  • Golden digest tests ensure stability across platforms and versions

S3 client:

  • Manual AWS Signature V4 signing — zero new dependencies
  • Works with any S3-compatible service (AWS S3, MinIO, Cloudflare R2)

Remote cache commands:

  • bleep remote-cache pull — download + extract tar.gz archives per project
  • bleep remote-cache push — tar.gz classes + zinc analysis, upload
  • Resources affect the digest but are NOT cached (matches sbt's design)

Portable zinc analysis:

  • ZincBridge now uses ReadWriteMappers with RootPaths (build dir, coursier cache, products dir)
  • Analysis files contain relative paths (${0}/src/main/scala/Foo.scala) instead of absolute paths
  • Portable across machines — no post-processing needed on pull

Credential management:

  • bleep config remote-cache setup — interactive TUI prompt
  • bleep config remote-cache clear — remove stored credentials
  • Env vars work without any setup

Documentation:

  • usage/remote-cache.mdx with setup, usage, GitHub Actions, GitLab CI, MinIO/R2 examples
  • Sidebar updated

Config examples

bleep.yaml:

remote-cache:
  uri: s3://my-bleep-cache/builds
  region: eu-north-1

~/.config/bleep/config.yaml:

remoteCacheCredentials:
  accessKeyId: AKIA...
  secretAccessKey: wJal...

Or env vars:

export BLEEP_REMOTE_CACHE_S3_ACCESS_KEY_ID=AKIA...
export BLEEP_REMOTE_CACHE_S3_SECRET_ACCESS_KEY=wJal...

Test plan

🤖 Generated with Claude Code

oyvindberg and others added 8 commits April 4, 2026 14:24
…in config

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Computes digest from project config + source/resource content + transitive
dep digests. Uses git ls-tree for clean directories (fast, no file I/O),
falls back to filesystem for dirty/generated files. Golden digest tests
ensure stability across platforms and versions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use git-compatible blob hashing (SHA-1 with blob header) for filesystem
path, matching git ls-tree output. Relativize git ls-tree paths to source
dir. Add test that creates a temp git repo, verifies uncommitted filesystem
digest matches committed git ls-tree digest.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
S3Client: manual AWS Signature V4 signing, zero dependencies.
TarGz: pure JDK tar.gz pack/unpack (no commons-compress needed).
RemoteCache: pull pre-compiled classes from S3, push after building.
Schema, docs, and sidebar updated.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use zinc's built-in RootPaths + RelativeReadMapper/RelativeWriteMapper
to relativize absolute paths in analysis files. Three roots: build dir
(sources + products), coursier cache (library JARs).

Analysis files now contain relative paths like ${0}/src/main/scala/Foo.scala
instead of /home/user/project/src/main/scala/Foo.scala, making them
portable across machines for remote cache.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
bleep config remote-cache setup  — interactive prompt for AWS key/secret
bleep config remote-cache clear  — remove stored credentials

Also supports AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY env vars (no setup needed).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@oyvindberg oyvindberg marked this pull request as ready for review April 4, 2026 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant