Skip to content

Add custom scrapers for Tableau insights#66

Open
rapsoj wants to merge 1 commit into
mainfrom
usda-aphis-tableau
Open

Add custom scrapers for Tableau insights#66
rapsoj wants to merge 1 commit into
mainfrom
usda-aphis-tableau

Conversation

@rapsoj

@rapsoj rapsoj commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

This pull request introduces two new custom scrapers for public health and livestock surveillance data, enhances the chunk-based fact extraction logic to better handle quote routing, and updates the data source configuration. The most significant changes are the addition of robust, analytics-producing scrapers for PAHO Oropouche and USDA APHIS livestock data, and improved logic to attribute quoted facts to the correct document chunk, even when quotes are found in adjacent chunks.

New data scrapers:

  • Added paho_oropouche_portal.py, a custom scraper for PAHO ARBO Oropouche weekly CSV data, which fetches the CSV, computes weekly/country analytics, fits linear and exponential models, and renders a compact HTML summary for downstream processing.
  • Added usda_aphis_livestock.py, a custom scraper for USDA APHIS HPAI livestock detections, which processes the downloadable CSV, computes monthly/daily/state-level statistics, fits models, and renders a structured HTML summary.
  • Registered the new PAHO Oropouche data source in bioscancast/datasets/sources.yaml under specific_pathogen_sources.

Fact extraction improvements:

  • Enhanced the quote attribution logic in extract_facts_from_chunk to check all chunks in a document if a quote is not found in the expected chunk, improving robustness against chunk routing misses. The correct chunk_id is now assigned to the fact source, even if the quote is found in an adjacent chunk. [1] [2]

@rapsoj rapsoj requested a review from smodee July 14, 2026 21:22

@smodee smodee left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not getting the scrapers to work - the URL seems to be a temporary session-scoped export URL, unfortunately not something we can use to reliably scrape from. The unit tests pass only because they mock the fetch, so the dead URL is never exercised. For the scraping, I would suggest going for the curated snapshot suggested in #50.

Nonetheless, I think the other changes in this PR seem solid and worth keeping:

  1. The CSV to HTML analytics renderer - just decouple it from the live fetch
  2. The csv_fetcher injection seam.
  3. The chunk_extractor quote-routing fix (this is independent and could be its own PR)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants