Skip to content

Images stored against content hashes#783

Open
gilescope wants to merge 27 commits intomainfrom
giles-content-hashes
Open

Images stored against content hashes#783
gilescope wants to merge 27 commits intomainfrom
giles-content-hashes

Conversation

@gilescope
Copy link
Contributor

@gilescope gilescope commented Feb 25, 2026

Overview

Replace 8-char git commit hashes with 12-char tree content hashes (git rev-parse HEAD^{tree}) in Docker image tags. Identical source trees now produce the same tag regardless of commit metadata, allowing CI to skip redundant builds entirely.

Key changes:

  • Image tags use {VERSION}-{12-char-tree-hash}-{ARCH} format instead of {VERSION}-{8-char-commit-hash}-{ARCH}
  • CI checks GHCR for existing images before building — if both node and toolkit images exist, the build is skipped
  • force_rebuild workflow_dispatch input provides an escape hatch
  • Format/lint and feature unification checks also skip when images already exist
  • Downstream jobs (E2E tests, genesis rebuild) reference image tags from the build job outputs instead of recomputing them
  • ARM64 builds no longer gated behind ci:arm64 label
  • Feature unification check split into its own job
  • Earthfile image targets use CONTENT_HASH LET instead of EARTHLY_GIT_SHORT_HASH
  • Toolkit reuses pre-built compactc from +prep-no-copy instead of rebuilding from +node-ci-image-single-platform
  • GenerateGenesis output downgraded from println! to log::debug!
  • ADR 0004 documents the decision and rationale

🗹 TODO before merging

  • Ready

📌 Submission Checklist

  • Changes are backward-compatible (or flagged if breaking)
  • Pull request description explains why the change is needed
  • Self-reviewed the diff
  • I have included a change file, or skipped for this reason:
  • If the changes introduce a new feature, I have bumped the node minor version
  • Update documentation (if relevant)
  • Updated AGENTS.md if build commands, architecture, or workflows changed
  • No new todos introduced

🧪 Testing Evidence

CI runs on this PR demonstrate the content hash tagging and skip logic in action.

  • Additional tests are provided (if possible)

🔱 Fork Strategy

  • Node Runtime Update
  • Node Client Update
  • Other: CI workflow and Earthfile changes only — no runtime or client changes
  • N/A

Links

  • ADR: docs/decisions/0004-tree-content-hash-image-tags.md

gilescope and others added 3 commits February 25, 2026 08:22
@github-actions
Copy link
Contributor

github-actions bot commented Feb 25, 2026

kics-logo

KICS version: v2.1.19

Category Results
CRITICAL CRITICAL 0
HIGH HIGH 2
MEDIUM MEDIUM 52
LOW LOW 3
INFO INFO 64
TRACE TRACE 0
TOTAL TOTAL 121
Metric Values
Files scanned placeholder 27
Files parsed placeholder 27
Files failed to scan placeholder 0
Total executed queries placeholder 73
Queries failed to execute placeholder 0
Execution time placeholder 12

@gilescope gilescope closed this Mar 7, 2026
@gilescope gilescope reopened this Mar 7, 2026
@gilescope gilescope closed this Mar 7, 2026
@gilescope gilescope reopened this Mar 7, 2026
Signed-off-by: Giles Cope <gilescope@gmail.com>
@gilescope gilescope changed the title WIP: content hashes Images stored against content hashes Mar 7, 2026
Signed-off-by: Giles Cope <gilescope@gmail.com>
@gilescope gilescope closed this Mar 7, 2026
@gilescope gilescope reopened this Mar 7, 2026
Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>
@gilescope gilescope closed this Mar 7, 2026
@gilescope gilescope reopened this Mar 7, 2026
Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>
@gilescope gilescope marked this pull request as ready for review March 9, 2026 08:36
@gilescope gilescope requested a review from a team as a code owner March 9, 2026 08:36
Copy link
Contributor

@ozgb ozgb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great addition - a couple of suggestions

@NachoPal
Copy link
Contributor

Ok if it saves unnecessary extra CI time. On the other hand, not being able to pin an image to a specific commit can be troublesome for debugging and auditability. As @ozgb suggested this could be addressed with Docker annotations or embedding the commit hash as an env var in the image.

The tree hash granularity could also be improved by hashing only the directories that affect the binary (node, runtime, pallets, etc.) rather than the entire repo tree, to avoid unnecessary rebuilds from docs-only or CI-only changes.

@gilescope
Copy link
Contributor Author

gilescope commented Mar 14, 2026

I don't want to put a particular commit into the image as that would be inappropriate, but images are now tagged with the git commit(s) that the image was as well as the one content hash.

gilescope and others added 3 commits March 16, 2026 11:56
@gilescope gilescope enabled auto-merge March 17, 2026 20:18
@gilescope gilescope requested a review from ozgb March 17, 2026 20:19
@gilescope gilescope added this pull request to the merge queue Mar 18, 2026
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to a conflict with the base branch Mar 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants