feat(source-files): wire archive overlays into prep-sources with post-overlay hash tracking

.github/instructions/docs.instructions.md

@@ -25,3 +25,13 @@ Files under `docs/user/reference/cli/` are generated by `azldev docs markdown` (
 3. Commit the regenerated files
 
 **Never hand-edit files under `docs/user/reference/cli/`** — they will be overwritten on the next `mage docs`.
+
+## Archive overlay documentation
+
+When documenting archive overlays:
+
+- Distinguish the upstream archive hash from the configured post-overlay hash. The upstream algorithm verifies the download; the TOML `hash-type` verifies and records the repacked result, and they may differ.
+- State that repacking preserves the archive's content-detected compression, including intentionally or accidentally misleading filename extensions.
+- Describe ordering accurately: archive overlays are batched per archive and run before spec and loose-file overlays; declaration order is preserved within a scope. Loose-file overlays may not also target an archive modified by archive overlays.
+- Document the bidirectional association: every modified archive needs an overlay-origin entry, and every overlay-origin entry needs a matching archive overlay. `--allow-no-hashes` may temporarily relax only the missing-origin/hash bootstrap case, never orphaned origins.
+- Do not present `render --check-only` download/repack shortcuts as behavioral guarantees. They are implementation/UX optimizations and must not weaken validation that rendered specs match TOML configuration.

.github/instructions/go.instructions.md

@@ -137,3 +137,13 @@ Components can override the project-default distro via `Spec.UpstreamDistro`. Th
 - **`env.Distro()`** — safe when all components share the same distro (e.g., iterating over results in `saveComponentLocks`). Breaks if components override the distro.
 - **`sourceproviders.ResolveDistro(env, comp)`** — use when you need the full distro context for a specific component (snapshot time, dist-git branch, lookaside URI). This is what `resolveOneSourceIdentity` uses to create the source manager.
 - **Per-component release version** — when computing fingerprints per-component, resolve the distro per-component to get the correct `ReleaseVer`. Using the project-default release version is wrong when component-level distro overrides exist.
+
+## Archive Overlay Invariants
+
+When working on archive overlays or `origin.type = "overlay"`, preserve these rules:
+
+- Overlay-origin entries are declarations of a post-overlay hash, not separately fetched files. `SourceManager.FetchFiles` must skip them, while component lookaside extraction must still fetch the original upstream archive. Build lookaside skip lists with `OriginType.IsFetched()`.
+- Validate associations in both directions: each archive modified by an archive-scoped overlay needs one matching overlay-origin `source-files` entry, and each overlay-origin entry must target an archive that is actually modified. During `--allow-no-hashes` bootstrapping, a modified archive may temporarily omit its origin entry, but an orphaned origin entry is never valid.
+- Use the upstream `sources` hash algorithm only to fetch and verify the original archive. After repacking, compute and validate the digest with the TOML entry's configured `hash-type` (SHA-256 or SHA-512); the two algorithms need not match.
+- Preserve the archive's actual compression detected from its contents, even when its filename extension claims a different format. Do not “fix” a mislabeled archive during repacking.
+- Archive overlays are batched per archive and run before spec and loose-file overlays. Preserve declaration order within each scope. Reject a loose-file overlay that can modify, remove, replace, or rename an archive also targeted by an archive-scoped overlay; otherwise it can silently discard the repacked result.

.github/instructions/testing.instructions.md

@@ -230,3 +230,13 @@ New component subcommands (`internal/app/azldev/cmds/component/`) require:
 - **Bypassing `CmdFactory`** with raw `exec.Command` → test spawns real process
 - **Updating snapshots blindly** to make a test pass → masks real bugs
 - **Non-deterministic output** in snapshots (timestamps, abs paths) → normalize first
+
+## Archive Overlay Regression Coverage
+
+Changes to archive overlay preparation should retain tests for these independent edge cases:
+
+- An upstream entry using a different or legacy hash algorithm (for example MD5) is downloaded with that algorithm, then the repacked result is hashed with the TOML-configured SHA-256/SHA-512 algorithm.
+- An archive whose extension does not match its actual compression is repacked using its content-detected compression.
+- Overlay-origin files are not included in lookaside skip lists, while fetched custom/download origins are.
+- Archive-overlay and overlay-origin associations are rejected when missing in either direction (except the documented `--allow-no-hashes` bootstrap relaxation).
+- Loose-file overlays cannot remove, replace, rename onto, or glob-match an archive that is also modified by an archive-scoped overlay.

docs/user/reference/config/components.md

@@ -315,7 +315,7 @@ The `[[components.<name>.source-files]]` array defines additional source files t
 
 ### Origin
 
-Two origin types are supported.
+Three origin types are supported.
 
 #### `"download"` — fetch from a URI
 
@@ -356,7 +356,39 @@ origin.mock-packages = ["cmake"]                 # omit if not needed
 origin.inputs        = ["yara-4.5.4.tar.gz"]     # available to the script as ./yara-4.5.4.tar.gz
 ```
 
-> **Note:** Upstream source tarballs are automatically available as inputs before running custom generation scripts. There is no need to re-declare an upstream file in `source-files` to use it as an input.
+**Note:** Upstream source tarballs are automatically available as inputs before running custom generation scripts. There is no need to re-declare an upstream file in `source-files` to use it as an input.
+
+#### `"overlay"` — record a post-overlay hash
+
+- **`hash` and `hash-type` are required** for normal use. To bootstrap a new archive overlay, omit both and run `prep-sources --allow-no-hashes` once; it computes the post-overlay hash for you.
+- **`replace-upstream = true` is required** — the archive already exists in the upstream `sources` file, and this entry replaces its hash with the post-overlay value.
+- During `prep-sources` (full run), azldev verifies that the hash it computed after repacking the archive matches the stated `hash`. A mismatch means the config is stale and must be updated.
+- The post-overlay archive is hashed using the configured `hash-type`, regardless of the algorithm used by the upstream `sources` entry. The archive's actual compression format is preserved when repacking, even when it does not match the filename extension.
+
+See [Recording the post-overlay hash for archive overlays](#recording-the-post-overlay-hash-for-archive-overlays) below for the full workflow.
+
+### Recording the post-overlay hash for archive overlays
+
+When you apply archive overlays (e.g. removing vendored files from a tarball) using `file-remove` or `file-search-replace` with an archive-scoped path, the repacked archive has a different hash than the original. Use a `source-files` entry with `origin.type = "overlay"` to record the expected post-overlay hash:
+
+```toml
+[[components.apache-commons-compress.source-files]]
+filename = "commons-compress-1.27.1-src.tar.gz"
+hash = "c7a2cef26959e687ad19b96b5ba8393d7514095e13bf0f29bd41e6b3c3cb2260d8ff23283ff3d5fd137b2522b843e7f0f50ab46bcf0f66df5383674f35f223ab"
+hash-type = "SHA512"
+origin = { type = "overlay" }
+replace-upstream = true
+replace-reason = "Upstream source tarball contains test fixtures flagged as malware by the AZL RPM signing pipeline. These files are not needed at runtime and are removed to allow SRPM publication."
+```
+
+**Workflow:**
+
+1. Add the archive overlay(s) in the component's `[[overlays]]` array.
+2. Run `prep-sources --allow-no-hashes` once — this repacks the archive and writes the computed hash to the output `sources` file.
+3. Paste the computed `hash` and `hash-type` into the `source-files` entry above.
+4. Run `prep-sources` again to confirm the hash matches, then commit.
+
+`replace-upstream = true` and `replace-reason` are required because the archive is already in the upstream `sources` file. The entry replaces its hash with the post-overlay value, regardless of how many overlays target that archive.
 
 ### Replacing an upstream `sources` entry
 

docs/user/reference/config/overlays.md

@@ -2,7 +2,7 @@
 
 Overlays are semantic patches that modify RPM spec files and other source files during component processing. They allow you to make targeted changes to upstream specs without maintaining full forks.
 
-Overlays are defined within a component's configuration in your TOML config file and are applied in the order they appear. Each overlay specifies a type and the parameters needed to perform its modification.
+Overlays are defined within a component's configuration in your TOML config file. Overlays targeting the same scope are applied in declaration order. Archive overlays are batched and applied before spec and loose-file overlays; because these scopes normally modify different files, this does not change their result.
 
 > **Note:** Overlays are applied in sequence and modifications are non-atomic. If an overlay fails mid-way, previously applied changes remain. Work on copies if atomicity is required.
 
@@ -64,7 +64,16 @@ file = "vendor/**"                   # files inside the archive
 
 > **Note:** Archive overlays are batched per archive — all overlays targeting the same archive
 > share a single extract/modify/repack cycle. When wired into the source-preparation pipeline, the `sources` file
-> should be rehashed afterward to reflect the repacked archive; they are processed independently of spec and loose-file overlays.
+> is rehashed afterward to reflect the repacked archive. A loose-file overlay cannot target an archive that is also modified by an archive-scoped overlay; azldev rejects that ambiguous combination rather than allowing a later loose-file operation to discard or corrupt the repacked archive.
+
+> **Required drift protection:** Every archive targeted by an archive overlay must have one matching
+> `source-files` entry with `origin.type = "overlay"`. The entry records the expected post-overlay
+> hash; one entry can cover multiple overlays for the same archive. Conversely, every
+> `origin.type = "overlay"` entry must have a matching archive overlay; stale entries are rejected.
+> While bootstrapping a new overlay, `prep-sources --allow-no-hashes` permits the archive overlay
+> to temporarily omit its origin entry, or permits the matching origin entry to omit its hash.
+> An origin entry without a matching archive overlay is always rejected. See
+> [Components](components.md#recording-the-post-overlay-hash-for-archive-overlays).
 
 > **Extraction root:** The inner path is interpreted relative to the archive's extraction root: if the archive unpacks to a single top-level directory (the conventional `%{name}-%{version}` layout) that directory is the root; otherwise the archive root is used.
 

internal/app/azldev/core/sources/archiveoverlays.go

@@ -15,6 +15,88 @@ import (
 	"github.com/microsoft/azure-linux-dev-tools/internal/utils/rootfs"
 )
 
+// applyArchiveOverlays groups archive overlays by target archive and processes
+// them in order. Multiple overlays targeting the same archive are batched into
+// a single extract/modify/repack cycle. File removals inside the archive reuse
+// the same machinery as loose-file overlays ([applyNonSpecOverlay]).
+//
+// It returns the names of the archives that were actually repacked. In dry-run
+// mode no archive is repacked, so the returned slice is empty even when archive
+// overlays were present.
+func applyArchiveOverlays(
+	dryRunnable opctx.DryRunnable,
+	eventListener opctx.EventListener,
+	sourcesDirPath string,
+	overlays []projectconfig.ComponentOverlay,
+) ([]string, error) {
+	groups := groupOverlaysByArchive(overlays)
+
+	if len(groups) == 0 {
+		return nil, nil
+	}
+
+	operationCount := 0
+	for _, group := range groups {
+		operationCount += len(group.overlays)
+	}
+
+	event := eventListener.StartEvent("Applying archive overlays",
+		"archives", len(groups),
+		"operations", operationCount,
+	)
+	defer event.End()
+
+	var repacked []string
+
+	for _, group := range groups {
+		didRepack, err := processArchive(dryRunnable, sourcesDirPath, group.archive, group.overlays)
+		if err != nil {
+			return nil, fmt.Errorf("archive overlay failed for %#q:\n%w", group.archive, err)
+		}
+
+		if didRepack {
+			repacked = append(repacked, group.archive)
+		}
+	}
+
+	return repacked, nil
+}
+
+// archiveGroup holds overlays targeting the same archive, preserving order.
+type archiveGroup struct {
+	archive  string
+	overlays []projectconfig.ComponentOverlay
+}
+
+// groupOverlaysByArchive groups archive overlays by [projectconfig.ComponentOverlay.Archive],
+// preserving insertion order within each group and across groups.
+// Non-archive overlays are silently skipped.
+func groupOverlaysByArchive(overlays []projectconfig.ComponentOverlay) []archiveGroup {
+	orderMap := make(map[string]int)
+
+	var groups []archiveGroup
+
+	for _, overlay := range overlays {
+		if !overlay.ModifiesArchive() {
+			continue
+		}
+
+		archiveName := overlay.Archive
+
+		idx, exists := orderMap[archiveName]
+		if !exists {
+			idx = len(groups)
+			orderMap[archiveName] = idx
+
+			groups = append(groups, archiveGroup{archive: archiveName})
+		}
+
+		groups[idx].overlays = append(groups[idx].overlays, overlay)
+	}
+
+	return groups
+}
+
 // processArchive extracts an archive to a temp directory, applies all overlays,
 // and deterministically repacks it with the original compression, atomically
 // replacing the original via a temp file + rename. It returns true when the

internal/app/azldev/core/sources/archiveoverlays_internal_test.go

@@ -19,6 +19,60 @@ import (
 	"github.com/stretchr/testify/require"
 )
 
+func TestGroupOverlaysByArchive(t *testing.T) {
+	t.Run("groups overlays by archive name preserving order", func(t *testing.T) {
+		overlays := []projectconfig.ComponentOverlay{
+			{
+				Type:     projectconfig.ComponentOverlayRemoveFile,
+				Archive:  "pkg-1.0.tar.gz",
+				Filename: "unwanted.conf",
+			},
+			{
+				Type:     projectconfig.ComponentOverlayRemoveFile,
+				Archive:  "pkg-1.0.tar.gz",
+				Filename: "config.h",
+			},
+			{
+				Type:     projectconfig.ComponentOverlayRemoveFile,
+				Archive:  "other-2.0.tar.xz",
+				Filename: "docs/*.md",
+			},
+		}
+
+		groups := groupOverlaysByArchive(overlays)
+
+		require.Len(t, groups, 2)
+
+		assert.Equal(t, "pkg-1.0.tar.gz", groups[0].archive)
+		require.Len(t, groups[0].overlays, 2)
+		assert.Equal(t, "unwanted.conf", groups[0].overlays[0].Filename)
+		assert.Equal(t, "config.h", groups[0].overlays[1].Filename)
+
+		assert.Equal(t, "other-2.0.tar.xz", groups[1].archive)
+		require.Len(t, groups[1].overlays, 1)
+		assert.Equal(t, "docs/*.md", groups[1].overlays[0].Filename)
+	})
+
+	t.Run("skips overlays that are not archive-scoped", func(t *testing.T) {
+		overlays := []projectconfig.ComponentOverlay{
+			{Type: projectconfig.ComponentOverlaySetSpecTag, Tag: "Version", Value: "1.0"},
+			{Type: projectconfig.ComponentOverlayRemoveFile, Archive: "pkg.tar.gz", Filename: "f"},
+			// Plain (non-archive) file overlay: no archive field, so it must be skipped.
+			{Type: projectconfig.ComponentOverlayRemoveFile, Filename: "loose.txt"},
+			// Bare archive name with no archive field: a loose removal of the archive itself.
+			{Type: projectconfig.ComponentOverlayRemoveFile, Filename: "drop-me.tar.gz"},
+			{Type: projectconfig.ComponentOverlayAddFile, Filename: "new.txt", Source: "src"},
+		}
+
+		groups := groupOverlaysByArchive(overlays)
+
+		require.Len(t, groups, 1)
+		assert.Equal(t, "pkg.tar.gz", groups[0].archive)
+		require.Len(t, groups[0].overlays, 1)
+		assert.Equal(t, "f", groups[0].overlays[0].Filename)
+	})
+}
+
 // TestProcessArchive_DryRunDoesNotModifyArchive verifies that, in dry-run mode,
 // processArchive skips the extract/repack cycle entirely and leaves the original
 // archive on disk byte-for-byte unchanged (repacking would otherwise rewrite it).