Skip to content

u-boot CI: replace vendored tools/hi_gzip/ with python3 zlib one-liner #2079

@widgetii

Description

@widgetii

Background

`OpenIPC/u-boot-hi3516ev200` and `OpenIPC/u-boot-gk7205v200` both vendor an 11 MB `tools/hi_gzip/` directory — gzip-1.8 source plus a HiSi patch whose only meaningful change is one line:

```diff

  • #define WSIZE 0x8000 /* 32 KiB */
  • #define WSIZE 0x2000 /* 8 KiB */
    ```

The patch limits deflate's sliding window to 8 KiB. Real V4 HW gzip decompressors only have memory for an 8 KiB window — anything bigger trips `Uncompress Fail!` on real silicon (the bug behind the gk7205v300 hardware regression that `OpenIPC/u-boot-gk7205v200#14` fixed).

System `gzip` has `WSIZE` baked in as a compile-time constant — no runtime flag changes it. That's the entire reason hi_gzip is vendored.

Cost of the current approach across the fleet:

  • ~11 MB per repo (gzip-1.8.tar.gz ships a full source tree per fork).
  • ~1 min host build per CI run on cache miss.
  • HiSi's Makefile uses bash builtins (`popd`), so CI must override `SHELL=/bin/bash`.
  • Same vendored tree duplicated across multiple HiSi/Goke u-boot forks.

Proposal

`zlib`'s `deflateInit2` exposes `windowBits` as a runtime parameter. Anything that calls into zlib can emit gzip-format output with the 8 KiB window the HW decompressor wants. Concretely:

```sh
python3 -c '
import zlib, sys

wbits = 16 + windowBits (16 = gzip format, 13 = 8 KiB window)

c = zlib.compressobj(7, zlib.DEFLATED, 16 + 13)
sys.stdout.buffer.write(c.compress(sys.stdin.buffer.read()) + c.flush())
' < u-boot.bin > image_data.gzip
```

Verified equivalent at the silicon level — both encoders produce gzip streams with bounded back-distances <= 8 KiB; the HW decompressor parses both. Bytes differ (different encoder choices), but the functional contract is identical.

What changes per repo

  • Delete `tools/hi_gzip/` (~11 MB).
  • Update `arch/arm/cpu/armv7//{hw_compressed/,}Makefile`'s `image_data.gzip` rule to pipe through `python3` instead of `./gzip`.
  • Drop the `make -C tools/hi_gzip` and `cp tools/hi_gzip/bin/gzip ...` steps from `.github/workflows/build.yml`.

Affects `u-boot-hi3516ev200` (3 SoCs) and `u-boot-gk7205v200` (4 SoCs). The other u-boot repos in the fleet don't ship hi_gzip; they're either pre-Kbuild (mini-boot only) or use a different wrapper path.

Hard prerequisite

`widgetii/qemu-hisilicon#85` — tighten the QEMU HW-gzip model to actually enforce the 8 KiB window. Without it, swapping the encoder upstream is risky because the qemu_smoke gate accepts WSIZE>8 KiB streams (which is what let the original gk regression slip through). With #85 landed, qemu_smoke independently verifies the constraint, making encoder swaps safe.

Tradeoffs

  • Reproducibility vs. HiSi reference: hi_gzip produces byte-identical output to HiSi's reference toolchain. Swapping to zlib means our `u-boot-z.bin` differs at the byte level from any HiSi reference build. Both work on real silicon; only binary-diff / supply-chain audits would care.
  • Not blocking: this is pure cleanup. The current vendored approach works correctly and is now well-tested on real hardware. Nothing breaks if we never do this.

Suggested order

  1. Land HW gzip model too permissive: accepts WSIZE>8 KiB streams real silicon rejects widgetii/qemu-hisilicon#85 (QEMU rejects WSIZE>8 KiB streams).
  2. Sibling PRs in `u-boot-hi3516ev200` and `u-boot-gk7205v200` to swap encoder.
  3. Verify qemu_smoke now exercises the constraint and downstream artifacts still boot on real silicon (one camera per family).

Out of scope

  • Whether to merge the two affected u-boot repos' `tools/hi_gzip/` into a shared OpenIPC subrepo — that's a much bigger refactor and only worthwhile if more hi_gzip-using forks land in the future.
  • Pre-Kbuild u-boot repos (`hi3519v101`, `hi3516cv200`, etc.) that ship raw u-boot.bin or use a separate wrapper path — they don't have hi_gzip in the loop.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions