[WIP] UPSTREAM: <carry>: automate hermetic build requirements generation#79
Open
mytreya-rh wants to merge 1 commit intoopenshift:mainfrom
Open
[WIP] UPSTREAM: <carry>: automate hermetic build requirements generation#79mytreya-rh wants to merge 1 commit intoopenshift:mainfrom
mytreya-rh wants to merge 1 commit intoopenshift:mainfrom
Conversation
Replace the manually maintained ~100-line bash pipeline in openshift/Dockerfile.requirements with a Python script that derives all four requirements files entirely from the Pipfile, with no hardcoded package names. openshift/hack/generate_requirements.py implements five stages: Stage 1 – pipenv install + CVE auto-fix via Safety/pipenv update, then pip freeze to capture all pinned runtime packages. Stage 2 – Iterative pip-compile with dynamic conflict exclusion to produce requirements.txt. Packages that make pip-compile fail due to incompatible declared metadata (e.g. conflicting setuptools version ranges) are detected from the error output, excluded from compilation, and appended manually. RPM-installed packages (cryptography, cffi, pycparser, maturin) are commented out in post-processing. Stage 3 – pip_find_builddeps.py is run once per runtime package so that every package's build-system requirements can be associated with it individually. Stage 4 – Conflict detection and phase splitting. Merging all build-dep constraints is attempted with pip-compile; when it fails the conflicting dependency is identified and packages split into an earlier phase (needing the older version) and a later phase (needing the newer version) using three fallback strategies in order: direct upper-bound heuristic, per-package compilation to detect transitive conflicts, and single-package bisection. N discovered phases are mapped to exactly three build files with a greedy merge that verifies compatibility before absorbing each middle phase into the main build group. Build-isolation exact-version pins (e.g. wheel==0.45.1 declared by ansible-core's pyproject.toml) are discovered automatically from pkg_constraints, injected into requirements-pre-build.txt so cachi2 pre-fetches them, and stripped from later phases so those phases resolve newer CVE-fixed versions. No version numbers are hardcoded. Stage 5 – Safety scans each generated build requirements file for CVEs and attempts to fix them by adding minimum-version constraints and re-running pip-compile. Conflicts that prevent the fix are reported with the name of the blocking constraint. openshift/hack/generate_requirements.md documents the full algorithm. openshift/Dockerfile.requirements is reduced to installing the toolchain and invoking the script. The generated requirements files and Pipfile.lock (updated by any CVE auto-fixes) are exported to the mounted volume by the ENTRYPOINT. The regenerated requirements files reflect: - pyasn1 0.6.2 → 0.6.3 (CVE-2026-30922) - requests 2.32.5 → 2.33.1 (CVE-2026-25645) - wheel 0.46.3 → 0.47.0 in requirements-build.txt (CVE-2026-24049; 0.45.1 is retained in requirements-pre-build.txt for ansible-core build isolation as discovered from its build-system metadata) - certifi, charset-normalizer, idna, packaging, pathspec, poetry-core, setuptools, trove-classifiers bumped to latest - kubernetes 33.1.0 build deps skipped (internal setuptools-scm conflict in the package's own build-system declaration) Co-authored-by: Cursor <cursoragent@cursor.com>
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
WalkthroughThe PR introduces a new hermetic requirements generator for OpenShift, replacing a manual multi-step shell pipeline in the Dockerfile with a Python-based tool that automates dependency resolution, build-dependency discovery, phase splitting for conflicting constraints, and CVE remediation across four output requirements files. ChangesRequirements Generation System Overhaul
Sequence DiagramsequenceDiagram
participant Docker as Dockerfile
participant Gen as Generator Script
participant Pipenv as pipenv
participant PipTools as pip-tools
participant BuildDeps as pip_find_builddeps
participant Safety as Safety
participant Output as Artifact Files
Docker->>Gen: invoke generate_requirements.py
Gen->>Pipenv: Stage 1: install --deploy
Pipenv-->>Gen: pinned runtime deps
Gen->>Gen: CVE auto-fix via pipenv check
Gen->>PipTools: Stage 2: iterative pip-compile
PipTools-->>Gen: conflict errors (iteratively)
Gen->>Gen: exclude conflicts, append manually
Gen->>Output: requirements.txt
Gen->>BuildDeps: Stage 3: per-package build deps
BuildDeps-->>Gen: constraint sets
Gen->>Gen: Stage 4: recursive phase-split
Gen->>PipTools: compile each phase
PipTools-->>Gen: success + phase outputs
Gen->>Output: requirements-pre-build.txt<br/>requirements-build1.txt<br/>requirements-build.txt
Gen->>Safety: Stage 5: CVE scan
Safety-->>Gen: vuln reports
Gen->>Gen: infer min safe versions
Gen->>PipTools: recompile with min specs
PipTools-->>Gen: updated pins
Gen->>Output: updated build files
Gen->>Output: export Pipfile.lock
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes 🚥 Pre-merge checks | ✅ 12✅ Passed checks (12 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Comment |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mytreya-rh The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (3)
openshift/hack/generate_requirements.py (3)
454-461: 💤 Low valueConsider using
itertools.pairwise()for successive pairs.Since this script targets Python 3.12,
itertools.pairwise()is a cleaner alternative tozip(unique, unique[1:])for iterating over consecutive elements.♻️ Suggested refactor
+from itertools import pairwise + # ... - for lo, hi in zip(unique, unique[1:]): + for lo, hi in pairwise(unique): gap = (hi.major - lo.major) * 1000 + (hi.minor - lo.minor)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@openshift/hack/generate_requirements.py` around lines 454 - 461, Replace the manual consecutive-pair loop using zip(unique, unique[1:]) with itertools.pairwise for clarity: import pairwise from itertools and iterate "for lo, hi in pairwise(unique):" while preserving the gap calculation and update of split_after and max_gap (the variables split_after and max_gap and the gap computation using hi.major/hi.minor and lo.major/lo.minor should remain identical so behavior is unchanged).
991-991: 💤 Low valueUnused variable
stderr2.The variable
stderr2from the unpacking is never used. Prefix with underscore to indicate intentional discard.♻️ Suggested fix
- ok2, stderr2 = _pip_compile(in_path, txt_path, ["--allow-unsafe"]) + ok2, _stderr2 = _pip_compile(in_path, txt_path, ["--allow-unsafe"])🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@openshift/hack/generate_requirements.py` at line 991, The unpacking of _pip_compile(in_path, txt_path, ["--allow-unsafe"]) assigns stderr2 which is never used; change the unpack to discard the second value by prefixing it with an underscore (e.g., ok2, _stderr2 = _pip_compile(...)) so it's clear the stderr result is intentionally ignored while leaving _pip_compile, ok2, in_path, and txt_path unchanged.
880-884: 💤 Low valueRemove extraneous
fprefix from string without placeholders.Line 882 has an f-string prefix but no placeholders inside the first string portion.
♻️ Suggested fix
if auto_iso_pins: print( - f" Auto-detected build-isolation pins for pre-build: " + " Auto-detected build-isolation pins for pre-build: " + ", ".join(f"{k}=={v}" for k, v in sorted(auto_iso_pins.items())) )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@openshift/hack/generate_requirements.py` around lines 880 - 884, The print call that logs auto-detected pins uses an unnecessary f-string prefix on the first literal; update the print in the auto_iso_pins branch so the first string is a normal string (remove the leading f on " Auto-detected build-isolation pins for pre-build: ") while keeping the concatenation with ", ".join(f"{k}=={v}" for k, v in sorted(auto_iso_pins.items())); locate the print using the auto_iso_pins variable to adjust the literal.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@openshift/hack/generate_requirements.py`:
- Around line 454-461: Replace the manual consecutive-pair loop using
zip(unique, unique[1:]) with itertools.pairwise for clarity: import pairwise
from itertools and iterate "for lo, hi in pairwise(unique):" while preserving
the gap calculation and update of split_after and max_gap (the variables
split_after and max_gap and the gap computation using hi.major/hi.minor and
lo.major/lo.minor should remain identical so behavior is unchanged).
- Line 991: The unpacking of _pip_compile(in_path, txt_path, ["--allow-unsafe"])
assigns stderr2 which is never used; change the unpack to discard the second
value by prefixing it with an underscore (e.g., ok2, _stderr2 =
_pip_compile(...)) so it's clear the stderr result is intentionally ignored
while leaving _pip_compile, ok2, in_path, and txt_path unchanged.
- Around line 880-884: The print call that logs auto-detected pins uses an
unnecessary f-string prefix on the first literal; update the print in the
auto_iso_pins branch so the first string is a normal string (remove the leading
f on " Auto-detected build-isolation pins for pre-build: ") while keeping the
concatenation with ", ".join(f"{k}=={v}" for k, v in
sorted(auto_iso_pins.items())); locate the print using the auto_iso_pins
variable to adjust the literal.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: fe4f21d4-01ca-4770-9bdd-7ecdd39137a3
⛔ Files ignored due to path filters (1)
openshift/Pipfile.lockis excluded by!**/*.lock
📒 Files selected for processing (7)
openshift/Dockerfile.requirementsopenshift/hack/generate_requirements.mdopenshift/hack/generate_requirements.pyopenshift/requirements-build.txtopenshift/requirements-build1.txtopenshift/requirements-pre-build.txtopenshift/requirements.txt
|
@mytreya-rh: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the hand-maintained ~100-line bash pipeline in
openshift/Dockerfile.requirementswithopenshift/hack/generate_requirements.py, a Python script that derives allfour hermetic build requirements files entirely from the Pipfile — no
hardcoded package names, no manual conflict resolution.
pipenv install+ CVE auto-fix via Safety/pipenv update,then
pip freezeto pin all runtime packages.pip-compilewith dynamic conflict exclusion →requirements.txt. Packages causing compilation failures are detected frompip-compile's error output and handled automatically.
pip_find_builddeps.pyrun per-package to map each packageto its build-system requirements individually.
build-dep constraints; on failure identifies the conflicting dependency via
three fallback strategies (direct upper-bound heuristic → per-package
compilation for transitive conflicts → single-package bisection). Generates
all three build requirements files. Build-isolation exact-version pins
(e.g.
wheel==0.45.1from ansible-core'spyproject.toml) areauto-discovered from package metadata, injected into pre-build so cachi2
pre-fetches them, and stripped from later phases so those phases resolve
newer CVE-fixed versions.
minimum-version constraints and re-running pip-compile.
Full algorithm documented in
openshift/hack/generate_requirements.md.Requirements file changes
requirements.txtrequirements-pre-build.txtrequirements-build.txtrequirements-build1.txtTest plan
make -f openshift/Makefile generate-requirementsbuilds and runs cleanlymake -f openshift/Makefile check-requirementspasses (no unexpected diff)Made with Cursor
Summary by CodeRabbit
Release Notes
New Features
Chores