Build failing with Docker BuildKit enabled

[andreyzher]

Description

When attempting to build the ubi8/s2i-core image with Docker's new build engine (BuildKit), it fails to tag the resulting image.

Build log :
ubi8-s2i-core.txt

Normally I don't build with BuildKit enabled, but the latest version on Mac seems to have defaulted to this, which is how this was noticed.

Is it a known issue?

Thanks

Versions

Docker:

$ docker --version
Docker version 20.10.5, build 55c4c88

Build host:

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)

and

macOS Big Sur 11.2.3

Steps to reproduce

1. Set DOCKER_BUILDKIT=1
2. Build RHEL8 Core image

   make build TARGET=rhel8 VERSIONS=core VERBOSE=1
   

[phracek]

Just for understanding about the testing scenario.
Your notebook is MacBook or MacOS based device.
You are building s2i-base-container on host RHEL-7.9 and trying to build s2i-base-container for RHEL8, right?

[andreyzher]

Re-reading my own description, I realise I didn't make it particularly clear.

I built s2i-base-container on multiple different environments to check:

• MacBook Pro running macOS 11.2.3
• RHEL 7.9
• RHEL 8.3

The container I was trying to build was RHEL8, yes.

In all 3 cases, the build fails when running with BuildKit enabled:

 => => exporting layers                                                                                                                                                                                                                0.7s
 => => writing image sha256:9e10721411b643beca1c52363ff9aff30a2122914341eefdef6ab052a6176990                                                                                                                                           0.0s
Error response from daemon: conflict: unable to delete 069d0214cf24 (must be forced) - image is referenced in multiple repositories
Error response from daemon: conflict: unable to delete 069d0214cf24 (must be forced) - image is referenced in multiple repositories
VERSIONS="core" SKIP_SQUASH=1 UPDATE_BASE= OS=rhel8 CLEAN_AFTER= DOCKER_BUILD_CONTEXT=. OPENSHIFT_NAMESPACES="" CUSTOM_REPO="" REGISTRY="""" /usr/bin/env bash common/tag.sh
Error: No such object:
make[1]: *** [core] Error 1
make[1]: Leaving directory `/home/developer/s2i-base-container'
make: *** [build-serial] Error

When BuildKit is disabled (using DOCKER_BUILDKIT=0 or the corresponding daemon flag), then the image is built and tagged successfully.

[frenzymadness]

I'm able to reproduce the same issue when building Python container with docker and DOCKER_BUILDKIT=1. The problem is that after the build the file .image-id exists but it's empty which confuses our tag.sh script where IMAGE_ID variable is empty.

The real problem is actually here: https://github.com/sclorg/container-common-scripts/blob/25fa46839e12aaa42d0545db8d93c0572a1f2f69/build.sh#L178

Because the output of the docker builds with buildkit looks differently, we have to find a way how to parse the image id from it and add a regex for it there. Example output:

-> building using docker build  --label io.openshift.builder-version="e42e04e" -f "$dockerfile" "${DOCKER_BUILD_CONTEXT}"
[+] Building 0.1s (11/11) FINISHED                                                            
 => [internal] load .dockerignore                                                        0.0s
 => => transferring context: 2B                                                          0.0s
 => [internal] load build definition from Dockerfile.fedora                              0.0s
 => => transferring dockerfile: 102B                                                     0.0s
 => [internal] load metadata for quay.io/fedora/s2i-base:35                              0.0s
 => [1/6] FROM quay.io/fedora/s2i-base:35                                                0.0s
 => [internal] load build context                                                        0.0s
 => => transferring context: 1.96kB                                                      0.0s
 => CACHED [2/6] RUN INSTALL_PKGS="python3 python3-devel python3-setuptools python3-pip  0.0s
 => CACHED [3/6] COPY 3.10/s2i/bin/ /usr/libexec/s2i                                     0.0s
 => CACHED [4/6] COPY 3.10/root/ /                                                       0.0s
 => CACHED [5/6] COPY 3.10/root/opt/wheels /opt/wheels                                   0.0s
 => CACHED [6/6] RUN python3.10 -m venv /opt/app-root && /opt/app-root/bin/pip install   0.0s
 => exporting to image                                                                   0.0s
 => => exporting layers                                                                  0.0s
 => => writing image sha256:e6a5aa60ae93aaf587a6c210e09d01cc903e4e4f2a6b5d587da8ce1f4b8  0.0s

I'm afraid we'll need to refactor the parse_output logic. We can either try to detect the buildkit backed and use a proper regex or we can use the old one and if the variable is empty, use the new one but I'm afraid that detecting buildkit is not easy and the current capabilities of parse_output don't allow us to parse it twice.

Or we can come up with a single regex to rule them (the possible outputs) all 🤔

[frenzymadness]

There are two more problems, buildkit backed prints its outputs to the stderr instead of stdout, which makes it even harder because parse_output cannot process them both at the same time, and the image sha hash is not on the last line:

$ docker build -f 3.10/Dockerfile.fedora . 2>&1 | tail -n 2
#11 writing image sha256:3ebc8aa33dba18b62e6a4674eb488338fd907cb798c8b0c798d0319016c6614f done
#11 DONE 0.0s

The awk with the last part covering the new possible output can look like this:

tail -n 2 | awk -F'[: ]' '/Successfully built|(^--> )?(Using cache )?[a-fA-F0-9]+$/{print \$NF} /writing image sha256:/{print \$(NF - 1)}'

But it requires:

• the tail with two last lines instead of just one
• using both space and colon as delimiters for awk
• processing stderr and stdout at the same time

Doesn't an easier way for getting the image hash exist?