Skip to content

[WIP] UPSTREAM: <carry>: automate hermetic build requirements generation #79

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mytreya-rh wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[WIP] UPSTREAM: <carry>: automate hermetic build requirements generation #79

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
107 changes: 19 additions & 88 deletions openshift/Dockerfile.requirements
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ RUN set -e && dnf clean all && rm -rf /var/cache/dnf/* \
&& python3 -m ensurepip --upgrade

COPY ./images/ansible-operator/Pipfile* ./
COPY ./openshift/hack/generate_requirements.py ./

# The build dependencies are required by cachito. Following script
# does exactly the same. More info at: https://github.com/containerbuildsystem/cachito/blob/master/docs/pip.md#build-dependencies
Expand All @@ -15,95 +16,25 @@ RUN curl -LO https://raw.githubusercontent.com/containerbuildsystem/cachito/mast

RUN python3 -m pip install --upgrade pip

# Create requirements.in file from the pipenv created using the
# same Pipfile and Pipfile.lock used for upstream image. Then
# use pip-compile to generate the requirements.txt file.
# Install tooling, then run the requirements generator.
#
# generate_requirements.py dynamically:
# 1. Resolves all runtime packages via pipenv + pip freeze.
# 2. Detects packages that make pip-compile fail due to conflicting dependency
# metadata (e.g. incompatible setuptools/setuptools-scm version ranges) and
# excludes them from compilation, appending them manually to requirements.txt.
# 3. Collects per-package build-system requirements via pip_find_builddeps.py.
# 4. Detects version conflicts across all build deps using pip-compile + error
# parsing and splits conflicting packages into ordered installation phases.
# 5. Produces requirements-pre-build.txt, requirements-build1.txt,
# requirements-build.txt, and requirements.txt with RPM-installed packages
# (cryptography, cffi, pycparser, maturin) commented out in every file.
#
# No package names are hardcoded in the script beyond the RPM exclusion list
# and the cachi2-specific wheel==0.45.1 pin in the pre-build phase.
RUN python3 -m pip install pipenv==2023.11.15 \
&& python3 -m pip install pip-tools \
&& pipenv install --deploy \
&& pipenv check \
&& pipenv run pip freeze --all > ./requirements.in \
# NOTE: Comment out ansible-core, ansible-runner, and requests-unixsocket from
# `requirements.in` to avoid dependency conflicts during pip-compile.
# These are required to properly resolve dependency conflicts:
# 1. Resolve conflict with `setuptools` version:
# - `setuptools>=70.1` is required by ansible-core (via markupsafe).
# - `setuptools<=69.0.2,>=45` is required by ansible-runner.
# - `setuptools>=77.0.3` is required by charset-normalizer (via types-psutil).
# 2. Resolve conflict with `setuptools_scm` version:
# - `setuptools_scm>=8` is required by requests-unixsocket.
# - `setuptools_scm<8` is required by kubernetes.
# Save the original requirements.in before commenting out
&& cp ./requirements.in ./requirements.in.orig \
&& sed -i '/ansible-core==/s/^/#/g' ./requirements.in \
&& sed -i '/ansible-runner==/s/^/#/g' ./requirements.in \
&& sed -i '/requests-unixsocket==/s/^/#/g' ./requirements.in \
&& pip-compile --output-file=./requirements.txt ./requirements.in --strip-extras \
# Pin python-dateutil to 2.9.0 instead of 2.9.0.post0
&& sed -i 's/python-dateutil==2.9.0.post0/python-dateutil==2.9.0/g' ./requirements.txt \
# Add back the commented packages from the original requirements.in to requirements.txt
&& grep "ansible-core==" ./requirements.in.orig >> ./requirements.txt || true \
&& grep "ansible-runner==" ./requirements.in.orig >> ./requirements.txt || true \
&& grep "requests-unixsocket==" ./requirements.in.orig >> ./requirements.txt || true \
# Now comment them out again for pip_find_builddeps.py
&& sed -i '/ansible-core==/s/^/#/g' ./requirements.txt \
&& sed -i '/ansible-runner==/s/^/#/g' ./requirements.txt \
&& sed -i '/requests-unixsocket==/s/^/#/g' ./requirements.txt \
# Also comment out kubernetes to avoid setuptools-scm conflicts in build dependencies
&& sed -i '/^kubernetes==/s/^/#/g' ./requirements.txt \
# Comment out google-auth to prevent pip from pulling in cryptography (and its
# Rust/maturin build chain) as a transitive dependency during pip download
&& sed -i '/^google-auth==/s/^/#/g' ./requirements.txt \
# NOTE: Comment out cryptography and its dependencies from the requirements.txt
# files as these packages can't be installed in the isolated environment of OSBS
# image build. These packages will be installed through rpms.
&& sed -i '/cryptography==/s/^/#/g' ./requirements.txt \
&& sed -i '/cffi==/s/^/#/g' ./requirements.txt \
&& sed -i '/pycparser==/s/^/#/g' ./requirements.txt \
&& sed -i '/maturin==/s/^/#/g' ./requirements.txt \
&& ./pip_find_builddeps.py requirements.txt -o requirements-build.in --append \
# Uncomment ansible-core, ansible-runner, requests-unixsocket, and kubernetes
# so they are present in the final requirements.txt file
&& sed -i '/ansible-core==/s/^#//g' ./requirements.txt \
&& sed -i '/ansible-runner==/s/^#//g' ./requirements.txt \
&& sed -i '/requests-unixsocket==/s/^#//g' ./requirements.txt \
&& sed -i '/^#kubernetes==/s/^#//g' ./requirements.txt \
&& sed -i '/^#google-auth==/s/^#//g' ./requirements.txt \
# Comment out setuptools-scm from requirements-build.in to avoid version conflicts during pip-compile
&& sed -i '/setuptools-scm/s/^/#/g' ./requirements-build.in \
&& sed -i '/setuptools_scm/s/^/#/g' ./requirements-build.in \
&& pip-compile --output-file=./requirements-build.txt ./requirements-build.in --strip-extras --allow-unsafe \
# NOTE: Comment out cryptography and its dependencies from the requirements-build.txt
# files as these packages can't be installed in the isolated environment of OSBS image
# build. These packages will be installed through rpms.
&& sed -i '/cryptography==/s/^/#/g' ./requirements-build.txt \
&& sed -i '/cffi==/s/^/#/g' ./requirements-build.txt \
&& sed -i '/pycparser==/s/^/#/g' ./requirements-build.txt \
&& sed -i '/maturin==/s/^/#/g' ./requirements-build.txt \
# Add ansible-core into a separate requirements-build1-temp.in file to include it
# into the requirements-build1.in file.
&& grep "urllib3==" ./requirements.in >> ./requirements-build1-temp.in || true \
&& ./pip_find_builddeps.py requirements-build1-temp.in -o requirements-build1.in --append \
&& pip-compile --output-file=./requirements-build1.txt ./requirements-build1.in --strip-extras --allow-unsafe \
# NOTE: Comment out cryptography and its dependencies from the requirements-build1.txt
# files as these packages can't be installed in the isolated environment of OSBS
# image build. These packages will be installed through rpms.
&& sed -i '/cryptography==/s/^/#/g' ./requirements-build1.txt \
&& sed -i '/cffi==/s/^/#/g' ./requirements-build1.txt \
&& sed -i '/pycparser==/s/^/#/g' ./requirements-build1.txt \
&& sed -i '/maturin==/s/^/#/g' ./requirements-build1.txt \
# Add ansible-runner into a separate requirements-pre-build-temp.in
# file to include it into the requirements-pre-build.in file.
&& grep "ansible-runner==" ./requirements.txt >> ./requirements-pre-build-temp.in || true \
# Add flit-core to requirements-pre-build.in file as this package is part of the
# build dependencies of some packages in requirements-build.txt file.
&& grep "flit-core==" ./requirements-build.txt >> ./requirements-pre-build.in || true \
&& ./pip_find_builddeps.py requirements-pre-build-temp.in -o requirements-pre-build.in --append \
&& pip-compile --output-file=./requirements-pre-build.txt ./requirements-pre-build.in --strip-extras --allow-unsafe \
# Pin wheel to 0.45.1 in pre-build so cachi2 fetches it for ansible-core's
# build isolation (which requires exactly wheel==0.45.1).
# requirements-build.txt still has wheel==0.46.3 which upgrades it later.
&& sed -i 's/^wheel==.*/wheel==0.45.1/' ./requirements-pre-build.txt
&& python3 generate_requirements.py

VOLUME /tmp/requirements
ENTRYPOINT ["cp", "./requirements.txt", "./requirements-build.txt", "./requirements-build1.txt", "./requirements-pre-build.txt", "/tmp/requirements/"]
ENTRYPOINT ["cp", "./requirements.txt", "./requirements-build.txt", "./requirements-build1.txt", "./requirements-pre-build.txt", "./Pipfile.lock", "/tmp/requirements/"]
Loading