Skip to content

Add Solr registry check to check_deprecated_lids#23

Open
jordanpadams wants to merge 1 commit into
mainfrom
add-solr-check-to-deprecated-lids
Open

Add Solr registry check to check_deprecated_lids#23
jordanpadams wants to merge 1 commit into
mainfrom
add-solr-check-to-deprecated-lids

Conversation

@jordanpadams
Copy link
Copy Markdown
Member

Summary

  • Extends check_deprecated_lids.py to verify each deprecated LID against both the PDS REST Search API (expects HTTP 404) and the PDS Solr registry (https://pds.nasa.gov/services/search/search?wt=json&q=lid:"<lid>"&rows=0, expects numFound == 0)
  • Failures from each source are collected and reported in separate sections; exit code is 1 if either check has failures
  • Adds query_solr() with specific exception handling for Timeout, ConnectionError, HTTPError, JSONDecodeError, and KeyError

Test plan

  • 21 unit tests pass (pytest test/context/test_check_deprecated_lids.py -v) — 14 new tests covering query_solr error paths and the updated check_deprecated_lids behavior
  • Run manually against a known-deprecated LID to confirm both API and Solr results print correctly

🤖 Generated with Claude Code

Each deprecated LID is now verified against both the PDS REST Search API
(expects 404) and the PDS Solr registry (expects numFound == 0). Failures
from each source are reported separately. Adds 14 new unit tests covering
query_solr error paths and the combined check_deprecated_lids behavior.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@jordanpadams jordanpadams requested a review from a team as a code owner May 29, 2026 22:24
Copy link
Copy Markdown
Member

@nutjob4life nutjob4life left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comments above/below. And feel free to override/countermand them! 😁

Also, good news on the tests:

rootdir: /Users/kelly/Documents/Clients/JPL/PDS/Development/nasa-pds/en-ops-utils
collected 21 items                                                                                                     

test/context/test_check_deprecated_lids.py::test_load_skips_comments_and_header PASSED                           [  4%]
test/context/test_check_deprecated_lids.py::test_load_skips_blank_lines PASSED                                   [  9%]
test/context/test_check_deprecated_lids.py::test_load_empty_file_returns_empty_list PASSED                       [ 14%]
test/context/test_check_deprecated_lids.py::test_query_api_returns_404 PASSED                                    [ 19%]
test/context/test_check_deprecated_lids.py::test_query_api_returns_200 PASSED                                    [ 23%]
test/context/test_check_deprecated_lids.py::test_query_api_timeout PASSED                                        [ 28%]
test/context/test_check_deprecated_lids.py::test_query_api_connection_error PASSED                               [ 33%]
test/context/test_check_deprecated_lids.py::test_query_solr_returns_zero PASSED                                  [ 38%]
test/context/test_check_deprecated_lids.py::test_query_solr_returns_nonzero PASSED                               [ 42%]
test/context/test_check_deprecated_lids.py::test_query_solr_timeout PASSED                                       [ 47%]
test/context/test_check_deprecated_lids.py::test_query_solr_connection_error PASSED                              [ 52%]
test/context/test_check_deprecated_lids.py::test_query_solr_http_error PASSED                                    [ 57%]
test/context/test_check_deprecated_lids.py::test_query_solr_parse_error PASSED                                   [ 61%]
test/context/test_check_deprecated_lids.py::test_query_solr_missing_key PASSED                                   [ 66%]
test/context/test_check_deprecated_lids.py::test_all_clean_no_failures PASSED                                    [ 71%]
test/context/test_check_deprecated_lids.py::test_api_non_404_detected_as_failure PASSED                          [ 76%]
test/context/test_check_deprecated_lids.py::test_solr_nonzero_detected_as_failure PASSED                         [ 80%]
test/context/test_check_deprecated_lids.py::test_api_network_error_recorded_as_failure PASSED                    [ 85%]
test/context/test_check_deprecated_lids.py::test_solr_network_error_recorded_as_failure PASSED                   [ 90%]
test/context/test_check_deprecated_lids.py::test_both_failures_reported_independently PASSED                     [ 95%]
test/context/test_check_deprecated_lids.py::test_mixed_api_results PASSED                                        [100%]

================================================== 21 passed in 0.75s ==================================================

return -1, f"connection error: {e}"
except requests.exceptions.HTTPError as e:
return -1, f"HTTP error: {e}"
except json.JSONDecodeError as e:
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Depending on the underlying JSON API (for example, if simplejson is installed and certain requests versions), the decoding exception may not always be json.JSONDecodeError. You can more safely catch the superclass though:

except ValueError as e:
    return -1, f"Rresponse parse error: {e}"

Tuple of (num_found, error_message).
num_found is -1 and error_message is set on network/parse errors.
"""
params = {"wt": "json", "q": f'lid:"{lid}"', "rows": 0}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works, but you can make the intent ("search everything, filter by exact LID") by using fq:

params = {"wt": "json", "q": "*:*", "fq": f'lid:"{lid}"', "rows": 0}

The difference is subtle, but with just q you get scoring (even if you don't want it) and builds a ful result set. With fq the scoring machinery isn't engaged and results are cached separately.


lids = load_deprecated_lids(csv_path)
print(f"\n✅ All {len(lids)} deprecated LIDs correctly return 404!")
print(f"\n✅ All {len(lids)} deprecated LIDs correctly absent from REST API and Solr registry!")
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Correctly absent" feels just a little awkward; maybe this instead?

All {len(lids)} deprecated LIDs are absent from both the REST API and Solr registry! PARTY TIME! 🥳

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants