feat(source-files): wire archive overlays into prep-sources with post-overlay hash tracking#276
feat(source-files): wire archive overlays into prep-sources with post-overlay hash tracking#276Tonisal-byte wants to merge 5 commits into
Conversation
c8b6390 to
0a209cc
Compare
bae831d to
8635511
Compare
There was a problem hiding this comment.
Pull request overview
This PR wires archive-scoped overlays into the prep-sources pipeline: archives modified by overlays are deterministically repacked, their corresponding sources entries are re-hashed, and configuration can record/validate expected post-overlay hashes via a new origin.type = "overlay" source-file origin.
Changes:
- Add new
OriginTypeOverlay/ schema enum value"overlay"and associated config validation rules for overlay-originsource-filesentries. - Apply archive overlays during
prep-sources, repack modified archives, and rehash only the affectedsourcesentries; additionally validate configured post-overlay hashes during full prep. - Adjust Fedora lookaside extraction to avoid skipping files that are not actually fetched by
FetchFiles(e.g., overlay-origin entries).
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| schemas/azldev.schema.json | Adds "overlay" to the origin.type enum in the JSON schema. |
| scenario/snapshots/TestSnapshotsContainer_config_generate-schema_stdout_1.snap | Updates schema snapshot output for the new enum value. |
| scenario/snapshots/TestSnapshots_config_generate-schema_stdout_1.snap | Updates schema snapshot output for the new enum value. |
| internal/providers/sourceproviders/sourcemanager.go | Skips downloads for overlay-origin refs; refactors provider attempts into helper; adds internal guard for overlay origin in download path. |
| internal/providers/sourceproviders/fedorasourceprovider.go | Avoids skipping upstream lookaside downloads for non-fetched origin types (e.g., overlay). |
| internal/projectconfig/fingerprint_test.go | Improves fingerprint tag parsing/validation to account for hashstructure tag options. |
| internal/projectconfig/configfile.go | Adds overlay-origin validation rules and calls archive-overlay/origin validation during config validation. |
| internal/projectconfig/configfile_test.go | Adds validation coverage for archive overlay requiring an overlay-origin entry. |
| internal/projectconfig/component.go | Introduces OriginTypeOverlay, OriginType.IsFetched, and ValidateArchiveOverlayOrigins. |
| internal/app/azldev/core/sources/sourceprep.go | Applies archive overlays first, tracks repacked archives, rehashes corresponding sources entries, and validates configured post-overlay hashes. |
| internal/app/azldev/core/sources/sourceprep_test.go | Adds end-to-end tests around archive overlay repacking + sources rehashing behavior. |
| internal/app/azldev/core/sources/archiveoverlays.go | Implements batching of archive overlays per archive (single extract/modify/repack cycle per archive). |
| internal/app/azldev/core/sources/archiveoverlays_test.go | Adds coverage for archive overlay glob semantics and end-to-end behavior via PrepareSources. |
| internal/app/azldev/core/sources/archiveoverlays_internal_test.go | Adds direct unit coverage for archive overlay grouping behavior. |
| internal/app/azldev/core/components/resolver.go | Validates archive overlay/origin invariants after config resolution/overlay-file expansion. |
| docs/user/reference/config/overlays.md | Documents drift-protection requirement for archive overlays via overlay-origin source-files. |
| docs/user/reference/config/components.md | Documents the new "overlay" origin type and the workflow for recording post-overlay hashes. |
Add the archive extract/apply/repack engine plus the ComponentOverlay 'archive' field, validation, and fingerprint handling. Extends internal/utils/archive with compression sniffing, ExtractAuto, and strict unsupported-entry handling. Pipeline wiring and hash tracking follow in a subsequent change.
…-overlay hash tracking Apply archive overlays in the source-prep pipeline, rehash repacked archives in the 'sources' file, and add the 'overlay' source-file origin type that records and validates the expected post-overlay hash.
175703d to
08ddb93
Compare
| assert.NotEqual(t, originalHash, entry.Hash, "'sources' entry hash must have been updated") | ||
| } | ||
|
|
||
| func TestPrepareSources_ArchiveOverlayRequiresOverlayOrigin(t *testing.T) { |
There was a problem hiding this comment.
Should have a matrix of tests (and maybe a full e2e scenario test?)
- overlay-origin without archive overlay fails
- archive overlay without origin fails
- matching custom/download origin for an archive fails
- overlay-origin w/ configured result matches
- stale hash fails
- algorithms are stable (ie SHA512 -> SHA512, not SHA512 -> SHA256)
- local lookaside and fedora paths both keep the original archive when overlay-origin ref is used
- --skip-sources still ensures hashes are valid
|
|
||
| // In full prep-sources mode, cross-check any 'overlay'-origin source-file entries against | ||
| // the hashes that were just computed so stale stated hashes are caught immediately. | ||
| if !p.skipLookaside { |
There was a problem hiding this comment.
Render always sets skipLookaside, my feeling is we either need to track the hashes of every input (including scripts etc.), or re-run this during render. The issue is sources is created by render, and afaik that is the definitive truth used by koji to get source files. We need a way to ensure that a PR has the correct values for the sources file.
Wires archive-scoped overlays into the source preparation pipeline.
This applies archive overlays during prep-sources, repacks modified archives, refreshes their sources file hashes, and adds origin.type = "overlay" for recording expected post-overlay archive hashes. It also ensures overlay-origin source files are not downloaded separately and validates stale configured hashes during full source prep.
Example: