Retrofit repo to use template-python packaging standards by jordanpadams · Pull Request #22 · NASA-PDS/en-ops-utils

jordanpadams · 2026-05-28T16:03:08Z

🗒️ Summary

Begins retrofitting the repo to the template-python packaging standards, scoped to the pds-sync-api script as the first migration. Remaining scripts can be moved one-by-one as they are tested.

Changes:

Add src/pds/en_ops_utils/ package layout (setup.cfg, pyproject.toml, tox.ini, setup.py, MANIFEST.in)
Move pds-sync-api logic into src/pds/en_ops_utils/portal/pds_sync_api.py; reduce scripts/portal/pds-sync-api.py to a thin backward-compatible wrapper
Add .pre-commit-config.yaml (trailing-whitespace, reorder-imports, mypy, flake8, detect-secrets)
Add tests/pds/en_ops_utils/portal/ with 8 unit tests for _get_lidvid, _already_downloaded, _write_harvest_config
Update README.md and CLAUDE.md: correct all bin/ → scripts/ paths, document new setup/dev workflow, describe dual legacy/packaged architecture
Add coverage.xml to .gitignore

Install after merge:

python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pds-sync-api --help

🤖 Generated with Claude Code — ~90% AI-assisted

⚙️ Test Data and/or Report

$ .venv/bin/pytest tests/pds/en_ops_utils/portal/ -v --no-cov
8 passed in 0.91s

$ .venv/bin/pds-sync-api --help
usage: pds-sync-api [-h] [-n NODE_NAME] [-p DOWNLOAD_PATH] [-u URL] [-c CONFIG] [-f] [-v]
...

♻️ Related Issues

Closes #7

🤓 Reviewer Checklist

Code follows PDS coding standards
Tests added/updated
Documentation updated
No sensitive information in PR

- Add pds.en_ops_utils Python package under src/ layout - Move pds-sync-api logic into src/pds/en_ops_utils/portal/pds_sync_api.py - Reduce scripts/portal/pds-sync-api.py to a thin backward-compat wrapper - Add setup.cfg, pyproject.toml, tox.ini, setup.py, MANIFEST.in - Add .pre-commit-config.yaml (trailing-whitespace, mypy, flake8, detect-secrets) - Add tests/pds/en_ops_utils/portal/ with 8 unit tests - Update README and CLAUDE.md with correct scripts/ paths and new setup instructions - Add coverage.xml to .gitignore Closes #7 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Update test/context/ sys.path inserts from bin/context to scripts/context - Add .secrets.baseline for detect-secrets pre-commit hook Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Remove docs env from tox (internal tools repo, READMEs are sufficient) - Add flake8 and detect-secrets to lint env deps for language:system hooks - chmod -x scripts/sitemap/sitemap-partial.xml (no shebang) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- Add dependabot.yml (pip + github-actions, weekly) - Add issue-project-automation.yml (project board + label automation) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

nutjob4life

Picked a few nits here and there! See the comments below.

Also, while pytest worked as described, tox was not as happy:

============================== 8 passed in 1.59s ===============================
.pkg: _exit> python /Users/kelly/Documents/Clients/JPL/PDS/Development/en-ops-utils/.venv/lib/python3.13/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg-cpython312: _exit> python /Users/kelly/Documents/Clients/JPL/PDS/Development/en-ops-utils/.venv/lib/python3.13/site-packages/pyproject_api/_backend.py True setuptools.build_meta
py313: OK ✔ in 2.35 seconds
lint: commands[0]> python -m pre_commit run --color=always --all
Trim Trailing Whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing scripts/ldds/prep_for_ldd_release.sh
Fixing scripts/ldds/ldd-corral.py
Fixing scripts/repos/repo-corral.py
Fixing LICENSE.md
Fixing scripts/ldds/update_ldds_with_template.sh
Fixing scripts/ldds/update-ldd-actions.py

Fix End of Files.........................................................Failed
- hook id: end-of-file-fixer
- exit code: 1
- files were modified by this hook

Fixing scripts/repos/update_action.sh
Fixing scripts/repos/update_templates.sh
Fixing LICENSE.md
Fixing scripts/ldds/update_ldds_with_template.sh
Fixing scripts/solr/deprecated_solr_registry_pds3_export.py

Check that executables have shebangs.....................................Passed
Check for merge conflicts................................................Passed
Debug Statements (Python)................................................Passed
Check Yaml...............................................................Passed
Reorder python imports...................................................Failed
- hook id: reorder-python-imports
- exit code: 1
- files were modified by this hook

Reordering imports in tests/pds/en_ops_utils/portal/test_pds_sync_api.py
Reordering imports in src/pds/en_ops_utils/portal/pds_sync_api.py

flake8...................................................................Passed
detect-secrets...........................................................Failed
- hook id: detect-secrets
- exit code: 1

Executable `/Users/kelly/Documents/Clients/JPL/PDS/Development/en-ops-utils/scripts/detect_secrets_baseline.sh` not found
lint: exit 1 (1.00 seconds) /Users/kelly/Documents/Clients/JPL/PDS/Development/en-ops-utils> python -m pre_commit run --color=always --all pid=49059
  py312: OK (2.37=setup[0.49]+cmd[1.88] seconds)
  py313: OK (2.35=setup[0.51]+cmd[1.84] seconds)
  lint: FAIL code 1 (1.01=setup[0.00]+cmd[1.00] seconds)
  evaluation failed :( (5.80 seconds)

nutjob4life · 2026-05-28T16:57:39Z

-pip3 install --requirement requirements.txt
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -e ".[dev]"          # packaged scripts (pds-sync-api, etc.)


I like --long options in documentation since they're "self-documenting":

pip install --editable ".[dev]" pip install --requirement requirements.txt # legacy scripts not yet packaged

But that's just me! Feel free to ignore this

nutjob4life · 2026-05-28T16:59:42Z

-# Run tests
-pytest test/ -v
+# Run all tests
+pytest tests/ test/ -v


I'd like to see

pytest --verbose tests/ test/

However, more important: it looks like tox.ini is running just one of those, i.e., not both tests/ an test/, sort of silenlty dropping the legacy test suite from Roundup or any other CI.

nutjob4life · 2026-05-28T17:01:07Z

+deps = setuptools<81
+extras = dev
+usedevelop = true
+commands = pytest --cov=src --cov-report=xml {posargs}


This should probably be --cov=pds to harmonize with setup.cfg.

nutjob4life · 2026-05-28T17:01:58Z

+    hooks:
+    -   id: flake8
+        name: flake8
+        entry: flake8 src


Should we add tests, test, and scripts to this?

nutjob4life · 2026-05-28T17:02:37Z

+    hooks:
+    -   id: detect-secrets
+        name: detect-secrets
+        entry: scripts/detect_secrets_baseline.sh


I think detect_secrets_baseline.sh might be missing from the PR 🤔

nutjob4life · 2026-05-28T17:08:57Z

@@ -1,8 +1,10 @@
+# Legacy requirements for un-ported scripts in bin/.


Probably should be scripts/ instead of bin/

nutjob4life · 2026-05-28T17:10:04Z

-pytest>=7.0.0 
-lxml~=6.0.2
+pytest>=7.0.0
+lxml>=6.0.2


Sure about this? Using ~=6.0.2 avoids a supply-chain vulnerability. Is there an upper limit of lxml we can state?

nutjob4life · 2026-05-28T17:11:10Z

+    while True:
+        _logger.debug("Making request to %s with params %r", url, params)
+        r = requests.get(url, params=params)
+        matches = r.json()["data"]


Might want to add r.raise_for_status() right after the requests.get before attempting to use the payload on the next line.

nutjob4life · 2026-05-28T17:12:10Z

+        params["search-after"] = matches[-1]["properties"][_search_key]
+
+
+def _write_harvest_config(node_name: str, download_path: str, config: str) -> None:


It looks like node_name isn't used anymore. Can we remove it from the parameter list? If not, we could add a note to the docstring.

nutjob4life · 2026-05-28T17:14:06Z

@@ -0,0 +1,2 @@
+# -*- coding: utf-8 -*-
+# Namespace package


I think you can remove this entire __init__.py file since namespace packages are implicit with newer Pythons.

- Add branch-cicd, stable-cicd, unstable-cicd, sonarcloud, secrets-detection workflows - Add scripts/detect_secrets_baseline.sh for pre-commit detect-secrets hook - Fix trailing whitespace in scripts/ (pre-commit hook cleanup) - Update .gitignore and LICENSE.md to template standards - Minor linter fixes in setup.cfg, pds_sync_api.py, test_pds_sync_api.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

jordanpadams requested a review from a team as a code owner May 28, 2026 16:03

jordanpadams added the task label May 28, 2026

jordanpadams self-assigned this May 28, 2026

jordanpadams requested a review from nutjob4life May 28, 2026 16:03

jordanpadams and others added 4 commits May 28, 2026 09:11

Fix test sys.path references and add secrets baseline

97778f3

- Update test/context/ sys.path inserts from bin/context to scripts/context - Add .secrets.baseline for detect-secrets pre-commit hook Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix deprecated pre-commit stage name push -> pre-push

bd5c0d7

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add missing GitHub workflows from template-repo-python

86754bd

- Add dependabot.yml (pip + github-actions, weekly) - Add issue-project-automation.yml (project board + label automation) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

nutjob4life requested changes May 28, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Retrofit repo to use template-python packaging standards#22

Retrofit repo to use template-python packaging standards#22
jordanpadams wants to merge 6 commits into
mainfrom
issue7-retrofit-template-python

jordanpadams commented May 28, 2026

Uh oh!

nutjob4life left a comment •

edited

Loading

Uh oh!

nutjob4life May 28, 2026

Uh oh!

nutjob4life May 28, 2026

Uh oh!

nutjob4life May 28, 2026

Uh oh!

nutjob4life May 28, 2026

Uh oh!

nutjob4life May 28, 2026

Uh oh!

nutjob4life May 28, 2026

Uh oh!

nutjob4life May 28, 2026

Uh oh!

nutjob4life May 28, 2026

Uh oh!

nutjob4life May 28, 2026

Uh oh!

nutjob4life May 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -1,8 +1,10 @@
		# Legacy requirements for un-ported scripts in bin/.

		params["search-after"] = matches[-1]["properties"][_search_key]


		def _write_harvest_config(node_name: str, download_path: str, config: str) -> None:

Conversation

jordanpadams commented May 28, 2026

🗒️ Summary

⚙️ Test Data and/or Report

♻️ Related Issues

🤓 Reviewer Checklist

Uh oh!

nutjob4life left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nutjob4life left a comment •

edited

Loading