hook: introduce post-linuxkit initramfs compressor/optimizer/reporter

[rpardini]

hook: introduce post-linuxkit initramfs compressor/optimizer/reporter

• lk-containers: implement wrappers around linuxkit/* images
  • for each linuxkit/ image used:
    • auto-generate Dockerfile under images/hook-linuxkit-xyz
    • build it as regular lk_containers for the arch
    • replace usage in hook.template.yaml with HOOK_CONTAINER_LINUXKIT_XYZ_IMAGE
  • this way we capitalize on the caching mechanisms already in place
    Signed-off-by: Ricardo Pardini ricardo@pardini.net
• kernel: armbian: do not remove modules from kernel
  • to make the (rather large) Armbian kernel's modules fit in 2gb RAM
    machines, we used to (quite bluntly) remove modules that weren't
    strictly needed for Hook:
    • drivers/net/wireless
    • kernel/sound
    • drivers/media
    • drivers/infiniband
  • with the initramfs optimization work, we can stop removing those
    • this fixes certain video/DRM problems
    • opens up the wireless path for the future
      Signed-off-by: Ricardo Pardini ricardo@pardini.net
• bash: bump shellcheck 0.10.0 -> 0.11.0; shellfmt 3.10.0 -> 3.12.0
  • squash out some instances of
    SC2268 (style): Avoid x-prefix in comparisons
    • I'm too old and it shows, I wrote all of them
  • squash one unused var and wrong comment ref
    Signed-off-by: Ricardo Pardini ricardo@pardini.net
• bash: common: use cat if bat is missing in log_file_bat()
  • otherwise contents wouldn't be shown at all
  • maybe it's time to add bat to downloaded dependencies (like linuxkit)?
    Signed-off-by: Ricardo Pardini ricardo@pardini.net
• kernel: config: update hook-{default|latest-lts}-defconfigs, no changes
  • this is done via
    • ./build.sh kernel-config hook-latest-lts-amd64
    • ./build.sh kernel-config hook-default-amd64
    • ./build.sh kernel-config hook-default-arm64
    • ./build.sh kernel-config hook-latest-lts-arm64
  • and then making no changes and exiting
  • here, ARM_SDE_INTERFACE=y is being removed as it is implied
    by having ACPI_APEI_GHES=y, ACPI=y, ACPI_APEI=y and ARM64=y
    Signed-off-by: Ricardo Pardini ricardo@pardini.net
• hook: introduce post-linuxkit initramfs compressor/optimizer/reporter
  • turns out LinuxKit ends up producing initramfs's that are very large
    • multiple copies of exact same files in different fs paths
    • gzip compression
    • multiple cpio layers causing multiple instances of same filepath stored
  • add a Docker-based postprocess step that does multiple tricks to fix it
    • extract/repack cpio to flatten it
    • rdfind to replace duplicates with hardlinks
    • zstd (-9, multithread) compression
    • reports usage and large duplicate files in different paths
      • so we can optimize our lk containers for better dedupe
  • gains are at
    • 25% for zstd compression (meson64)
    • 10Mb for rdfind, without any lk optimization yet
      Signed-off-by: Ricardo Pardini ricardo@pardini.net
• initramfs: unpack and repack cpio's like the kernel does
  • unpack and repack cpio's like the kernel does; order vs timestamps
    matter, since the kernel overwrites without mtime comparision
    Signed-off-by: Ricardo Pardini ricardo@pardini.net
• bash: inventory: drop old linuxkit version from 'peg'
  • 'peg' has served its purpose, should we remove it?
    Signed-off-by: Ricardo Pardini ricardo@pardini.net
• hook: lk-containers: bump linuxkit images to latest matching versions
  • this way we drastically reduce the number of different binaries
    found in the final initramfs, allowing the initramfs compressor
    to deduplicate them into hardlinks
    • libssl / libcrypto
    • busybox
    • musl
      Signed-off-by: Ricardo Pardini ricardo@pardini.net
• linuxkit: bump LinuxKit 1.6.0 -> 1.8.2
  Signed-off-by: Ricardo Pardini ricardo@pardini.net
• hook-lk-containers.sh: fix for TARGETARCH
  Signed-off-by: Ricardo Pardini ricardo@pardini.net
• initramfs: compressor: report gains for both cpio and compressed initramfs
  Signed-off-by: Ricardo Pardini ricardo@pardini.net

[rpardini]

Done a week of tests, this seems to be working fine. Will force-push with Sign-off's and undraft.