build: split package compilation stages

[zhouguangyuan0718]

Summary

• introduce immutable package build specifications and explicit package build results
• split serial package processing into preflight, backend execution, and publication stages
• make fingerprints use effective dependencies from both the selected package and enabled alternate package
• preserve the existing non-runtime/runtime second phase and deterministic package order
• archive LLVM-produced package objects directly from MemoryBuffer together with file-backed cgo/Plan9 members
• fail closed on LLVM archive errors; LLVM package objects never spill to temporary .o files
• keep backend execution serial in this layer; no worker pool or LLVM concurrency is enabled yet

Dependency and review boundary

Based on current main, which includes #2212, #2221, and #1736. In-memory archive writing additionally depends on xgo-dev/llvm#47. The temporary module replacement will be removed after that binding PR merges.

The intended diff for this PR is:

• base: 2310ff87 (current main with compiler: isolate per-invocation frontend state #2221)
• head: dcf4cca4
• 12 files
• +1032 / -131
• 8 commits

Please review only 2310ff87..dcf4cca4.

Tests

• go test ./internal/build ./cl ./ssa
• go test -race ./internal/build -run TestNormalizeToArchive -count=1
• ordinary, Thin LTO, and Full LTO clean builds and runtime checks
• cgo + Plan9 mixed-member clean build and runtime check
• verified LLVM package publication starts no external ar and creates no temporary LLVM package .o
• git diff --check

[fennoai]

Review: build — model package build units explicitly

Overall a clean, well-documented refactor that stages the build pipeline for bounded parallel compilation. The register*/build* split is sensible, the buildSSAPkgs worker pool is correctly bounded (min(parallelism(), len(unique))), dedupes by *ssa.Package, and runs fixSSAOrder serially afterward, and Package.Build() is documented as thread-safe — so the concurrency introduced here is race-free. Behavior parity with the old buildOne switch looks preserved. Findings below are correctness/maintainability nits; none are blocking.

Inline findings

• internal/goflags/flagfile.go:30 — adding -p to argumentListFlagNames also changes flag-file line parsing (wholeLineValueFlag), a likely regression.
• internal/build/collect.go:214 — llvmVersion is only eagerly initialized when cache is enabled, leaving the lazy first-use write the doc comment claims to remove.
• internal/build/build.go:926 — inconsistent second packageBuildResultFor(spec) snapshot.

Additional (non-inline) notes

• internal/build/package_build.go:60-66: packageBuildResult.spec, cacheHit, and archiveFile are populated but never read by production code (only by the test). They read as dead fields; consider trimming until the scheduler PR needs them, or add a note that they are intentional forward-looking scaffolding.
• internal/build/build.go:2040 (jobs := make(chan ssaBuildEntry)): the dispatch channel is unbuffered, so the producer hands off one entry at a time. Since unique is fully materialized before dispatch, a buffered channel (make(chan ssaBuildEntry, len(unique))) would let the producer enqueue without blocking and reduce handoff overhead. Minor.
• internal/build/collect.go:241-246 (dependencyFingerprint): for a dep with no module version that is not in pkgByID, a throwaway temp aPackage is fingerprinted with no memoization, so the same dep is recomputed once per parent. Worth a context-level fingerprint cache keyed by dep.ID if fan-in is high. (Largely pre-existing; noting because effectiveDependencies now feeds this path.)

A -race CI run over a multi-package build would be worthwhile to lock in the new concurrency guarantee.

[fennoai]

Adding "p" to argumentListFlagNames affects two consumers with different semantics. normalizeBuildFlags correctly turns -p 4 into -p=4, but wholeLineValueFlag (this same file) treats an argument-list flag as consuming the entire remainder of the line unquoted. -ldflags/-gcflags want that (space-separated list values), but -p takes a single integer.

A flags.txt line like -p=4 -v will now be captured whole as the flag -p=4 -v instead of split into -p=4 and -v. That reaches parseBuildParallel, where strconv.Atoi("4 -v") fails and ApplyBuildFlags returns an error, and -v is lost. Consider keeping -p out of the wholeLineValueFlag set (e.g. split the list into "normalize-only" vs "whole-line-value" names), and adding a test for -p alongside other flags on one flag-file line.

[fennoai]

initializePackageBuildState documents that it "avoids lazy first-use writes becoming data races later," but it only initializes llvmVersion/llvmVersionReady when cacheEnabled() is true (this early return). When the cache is disabled, getLLVMVersion still performs a lazy first-use write. Since getLLVMVersion is used for fingerprinting regardless of cache state, this is exactly the latent race the comment sets out to prevent once package building is parallelized. Consider initializing llvmVersion/llvmVersionReady unconditionally, before the cacheEnabled() check.

[fennoai]

result was already captured at line 913 (and returned for the cache-hit path at 915), but the success path recomputes packageBuildResultFor(spec) here instead of returning that value. Because normalizeToArchive mutates aPkg.ArchiveFile, the two snapshots differ in archiveFile. Harmless today only because archiveFile is unused, but it's a latent inconsistency if that field becomes load-bearing. Recompute once after normalizeToArchive/appendExternalLinkArgs and return that single value.

[codecov]

Codecov Report

❌ Patch coverage is 85.25896% with 37 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
internal/build/build.go	81.01%	15 Missing and 15 partials ⚠️
internal/build/package_archive.go	83.33%	4 Missing and 3 partials ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

LLGo baseline benchmarks

e98e124c378a | workflow run | long-term charts

Program measurements

Platform	Workload	File size	vs main	Build	vs main	Run	vs main
Linux	cprintf	18544 B	new	311.039 ms	new	1.263 ms	new
Linux	fmtprintf	2217608 B	new	3.218 s	new	2.433 ms	new
Linux	println	71504 B	new	308.992 ms	new	1.704 ms	new
macOS	cprintf	84672 B	new	450.680 ms	new	2.515 ms	new
macOS	fmtprintf	2361520 B	new	3.458 s	new	23.047 ms	new
macOS	println	125712 B	new	395.757 ms	new	4.055 ms	new

Core language and compiler benchmarks

Platform	Benchmark	ns/op	vs main
Linux	BenchmarkLookupPCRandom	13.320 ns/op	new
Linux	BenchmarkMergeCompilerFlags	150.800 ns/op	new
Linux	BenchmarkMergeLinkerFlags	94.410 ns/op	new
Linux	BenchmarkChannelBuffered	34.110 ns/op	new
Linux	BenchmarkChannelHandoff	28398 ns/op	new
Linux	BenchmarkDefer	44.280 ns/op	new
Linux	BenchmarkDirectCall	1.563 ns/op	new
Linux	BenchmarkGlobalRead	1.868 ns/op	new
Linux	BenchmarkGlobalWrite	2.487 ns/op	new
Linux	BenchmarkGoroutine	32281 ns/op	new
Linux	BenchmarkInterfaceCall	8.107 ns/op	new
Linux	BenchmarkRuntimeGetG	1.868 ns/op	new
macOS	BenchmarkLookupPCRandom	13.370 ns/op	new
macOS	BenchmarkMergeCompilerFlags	144.800 ns/op	new
macOS	BenchmarkMergeLinkerFlags	86.770 ns/op	new
macOS	BenchmarkChannelBuffered	32.400 ns/op	new
macOS	BenchmarkChannelHandoff	5912 ns/op	new
macOS	BenchmarkDefer	58.180 ns/op	new
macOS	BenchmarkDirectCall	1.903 ns/op	new
macOS	BenchmarkGlobalRead	1.346 ns/op	new
macOS	BenchmarkGlobalWrite	2.782 ns/op	new
macOS	BenchmarkGoroutine	25773 ns/op	new
macOS	BenchmarkInterfaceCall	10.060 ns/op	new
macOS	BenchmarkRuntimeGetG	3.510 ns/op	new

No main baseline exists yet; all metrics are marked new.

Warning

• The rendered benchmark data could not be pushed.