ci: use uv for test dependency installation (fixes Poetry resolve hang) by RonShakutai · Pull Request #2152 · data-privacy-stack/presidio

RonShakutai · 2026-07-09T17:47:10Z

Change Description

CI installs dependencies with poetry install and no committed lock, so Poetry does a universal resolve of the full optional-dependencies graph on every run. That resolve backtracks for many minutes — effectively hanging — driven by the langextract extra's deep, loosely-bounded transitive tree (google-genai, google-cloud-storage, aiohttp…). The same pathology hit the local-build/E2E job, where docker compose build sat on "Resolving dependencies" for ~40 min until the job was cancelled.

Reproduced locally (Python 3.13, warm cache): Poetry >6 min, never completed; uv resolves the same graph in ~2s.

Fix — swap CI dependency installation to uv

Test matrix jobs (.github/workflows/ci.yml):

Install uv with python -m pip install uv==0.11.6 (the astral-sh/setup-uv action is not on the org's allowed-actions list — adding it made the workflow fail at startup, so uv is installed from PyPI via the existing setup-python).
uv sync <extras> --python <matrix> builds the project .venv; the spaCy-download, pytest and diff-cover steps run inside it via uv run --no-sync. pip + the test tooling that previously came from the Poetry dev group are installed explicitly (uv doesn't read that group; pip is needed because spaCy's model download shells out to python -m pip install).
uv's wheel cache is kept on the runner's roomy /mnt disk via UV_CACHE_DIR (mirroring the old POETRY_CACHE_DIR=/mnt/...), with UV_LINK_MODE=copy to avoid cross-filesystem hardlink warnings.
The wheel-build step pre-installs a marker-guarded tomli on Python < 3.11 (Poetry used to pull it onto the runner incidentally; build needs it to read pyproject.toml and it isn't in the hashed requirements file).
Job-level timeout-minutes: 60 (and 30 on the install step) so a bad resolve fails fast instead of hanging.

Service image builds:

The analyzer/anonymizer/image-redactor Dockerfiles install deps with uv pip install --system -r pyproject.toml --extra server (verified valid; the analyzer install layer resolves in ~9s vs Poetry's ~40 min hang).
Their entrypoint.sh scripts run gunicorn directly instead of poetry run gunicorn, since the images no longer install Poetry. Verified locally: containers start healthy and /health returns 200.

pyproject.toml (package metadata) is untouched — this only changes the install/run tooling. The Python 3.14 matrix job is kept as-is.

Verified: all 22 test-matrix jobs (incl. Python 3.14) and the local-build/E2E job pass; the E2E image build dropped from a ~40 min hang to ~2.5 min.

Note

Per @omri374's review: this PR already moves all of CI off Poetry (test jobs + service image builds — ci.yml has no functional Poetry left). A fuller replacement (the poetry-core build backend, the Dockerfile.{stanza,transformers,windows} variants, install_dependencies.sh, and contributor docs) is larger and better handled as a follow-up.

Issue reference

N/A

Checklist

I have reviewed the contribution guidelines
I agree to follow this project's Code of Conduct
I confirm that I have the right to submit this contribution and that it does not knowingly contain proprietary or confidential code.
My code includes unit tests (CI-only change)
All unit tests and lint checks pass locally
My PR contains documentation updates / additions if required (CHANGELOG)

Copilot

Pull request overview

This PR hardens the CI workflow against indefinite hangs during Poetry dependency resolution by adding explicit timeouts and ensuring Poetry runs non-interactively, improving reliability of the test matrix runs across Presidio components.

Changes:

Add a job-level timeout (timeout-minutes: 60) for the matrix test job to prevent unbounded runtime.
Add a step-level timeout (timeout-minutes: 30) specifically for the “Install dependencies” step to cap Poetry resolution time.
Add --no-interaction to poetry install to avoid CI waiting on prompts.

github-actions · 2026-07-09T17:57:19Z

Coverage report (presidio-anonymizer)

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
presidio-anonymizer/presidio_anonymizer
anonymizer_engine.py
presidio-anonymizer/presidio_anonymizer/operators
custom.py
Project Total

_{This report was generated by python-coverage-comment-action}

github-actions · 2026-07-09T17:57:52Z

Coverage report (presidio-image-redactor)

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
presidio-image-redactor/presidio_image_redactor
dicom_image_pii_verify_engine.py
document_intelligence_ocr.py
image_analyzer_engine.py
Project Total

_{This report was generated by python-coverage-comment-action}

github-actions · 2026-07-09T17:58:58Z

Coverage report (presidio-cli)

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
presidio-cli/presidio_cli
cli.py
Project Total

_{This report was generated by python-coverage-comment-action}

github-actions · 2026-07-09T17:59:33Z

Coverage report (presidio-structured)

This PR does not seem to contain any modification to coverable code.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

The analyzer test job runs `poetry install --all-extras` with no committed lock, so Poetry performs a universal resolve of the full optional-dependencies graph on every run. That resolve backtracks for many minutes — effectively hanging — driven by the langextract extra's deep, loosely-bounded transitive tree. Reproduced locally: Poetry >6 min (never completed); uv ~2s for the same 158-package resolution. Switch the test jobs to uv: - Install uv via astral-sh/setup-uv. - Create a per-component venv with `uv venv --seed` (seed keeps pip for the wheel-build step) and expose it via $GITHUB_PATH / $VIRTUAL_ENV. - `uv pip install -e '.<extras>'` plus the test tooling that previously came from the Poetry dev group (uv does not read that group). - Add a job-level `timeout-minutes: 60` (and 30 on the install step) so a bad resolve fails fast instead of hanging. No package metadata changed — pyproject.toml is untouched; this only swaps the CI install tooling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Use "uv run --no-sync" for spaCy model downloads, pytest, and diff-cover instead of exposing the venv on $GITHUB_PATH. "uv sync" builds the project's .venv (resolving the analyzer --all-extras graph in seconds where Poetry's lockless universal resolve hangs), and the subsequent steps execute inside it idiomatically. The --all-extras selection stays in the job matrix, unchanged from main. Wheel-build keeps using the system interpreter, matching the previous Poetry behaviour. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

The astral-sh/setup-uv action is not on the org's allowed-actions list, so adding it made the CI workflow fail at startup (startup_failure) before any job ran. Install a pinned uv from PyPI with the already present setup-python instead; this needs no new third-party action and keeps the resolver benefits that motivated the Poetry -> uv switch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

spaCy's `download` command shells out to `python -m pip install`, but uv-created virtualenvs ship without pip, so runtime/first-use model downloads (e.g. the presidio-cli conftest fetching en_core_web_lg) failed with SystemExit: 1. Install pip alongside the test tooling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Migrate each CI-tested package's development tooling from Poetry's [tool.poetry.group.dev.dependencies] to the standard PEP 735 [dependency-groups] table, and commit a uv.lock per package. This makes pyproject.toml the single source of truth for dev tooling and lets CI and Docker install a pinned, reproducible dependency graph instead of resolving on every run. The [project] metadata and the poetry-core build backend are unchanged, so wheels/sdists are byte-for-byte identical. Covers presidio-analyzer, presidio-anonymizer, presidio-cli, presidio-image-redactor, presidio-structured. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

CI now runs `uv sync --locked --group dev`, which installs exactly the locked graph and fails if pyproject.toml and uv.lock disagree rather than silently resolving new versions. The dev group provides the test tooling (and pip, which spaCy's model download shells out to), so the manual `uv pip install pytest ...` line is gone. The uv cache is persisted across runs via actions/cache keyed on uv.lock and trimmed with `uv cache prune --ci`. The analyzer/anonymizer/image-redactor images consume the same lockfile with `uv sync --locked --no-default-groups --extra server --no-install-project` into UV_PROJECT_ENVIRONMENT=/usr/local, guaranteeing images use the same dependency graph as CI. Splitting the dependency layer from the source COPY keeps it cached until pyproject/uv.lock change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-07-12T15:28:56Z

Dependency Review

The following issues were found:

❌ 1 vulnerable package(s)
❌ 82 package(s) with incompatible licenses
✅ 0 package(s) with invalid SPDX license definitions
⚠️ 152 package(s) with unknown licenses.
⚠️ 5 packages with OpenSSF Scorecard issues.

View full job summary

Copilot

Pull request overview

Copilot reviewed 13 out of 18 changed files in this pull request and generated 1 comment.

The newly committed uv.lock files pin exact versions, so dependency-review now sees two transformers advisories that already exist on main (capped at transformers <5 by spacy-huggingface-pipelines and tracked as Dependabot alerts). They are unrelated to the Poetry->uv migration and will be addressed in separate security PRs, so allow-list the two GHSAs here to keep this PR's dependency review unblocked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

SharonHart

Add uv lock files to be reviewed by CODWOWNERS (like the tomls)
Add instructions for copilot for always checking that uv.lock chnges are always committed when toml dependencies change

@SharonHart

Address review feedback from @SharonHart: - CODEOWNERS: require presidio-administrators approval for **/uv.lock, the same as **/pyproject.toml, so lockfile changes get dependency review. - copilot-instructions: switch the local dev commands to uv and add an explicit rule that editing pyproject.toml dependencies requires regenerating and committing that package's uv.lock in the same change, since CI installs with `uv sync --locked` and fails on drift. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 15 out of 20 changed files in this pull request and generated 5 comments.

Copilot

Pull request overview

Copilot reviewed 15 out of 20 changed files in this pull request and generated 2 comments.

Address review feedback: - ci.yml: use ${{ env.UV_CACHE_DIR }} for the cache action path instead of hardcoding /mnt/uv-cache, so the cache and uv stay in sync if the dir changes. - copilot-instructions: update the "Technology Stack" bullet to name uv as the dependency/install tool (poetry-core retained only as build backend), matching the uv-based dev commands already documented. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 15 out of 20 changed files in this pull request and generated 1 comment.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 15 out of 20 changed files in this pull request and generated 1 comment.

Copilot AI review requested due to automatic review settings July 9, 2026 17:47

Copilot started reviewing on behalf of RonShakutai July 9, 2026 17:55 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 9, 2026 18:22

RonShakutai changed the title ~~fix(ci): add timeout-minutes to prevent Poetry hanging~~ fix(ci): revert analyzer to Python <3.14 to fix Poetry resolve hang Jul 9, 2026

Copilot started reviewing on behalf of RonShakutai July 9, 2026 18:23 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread presidio-analyzer/pyproject.toml Outdated

Copilot AI review requested due to automatic review settings July 9, 2026 20:46

Copilot started reviewing on behalf of RonShakutai July 9, 2026 20:47 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread presidio-analyzer/pyproject.toml Outdated

Copilot AI review requested due to automatic review settings July 12, 2026 08:04

RonShakutai force-pushed the ronshakutai/fix-ci-poetry-timeout branch from a029bf8 to 7340a02 Compare July 12, 2026 08:04

Copilot started reviewing on behalf of RonShakutai July 12, 2026 08:04 View session

RonShakutai changed the title ~~fix(ci): revert analyzer to Python <3.14 to fix Poetry resolve hang~~ ci: use uv for test dependency installation (fixes Poetry resolve hang) Jul 12, 2026