Add custom scrapers for Tableau insights#66
Open
rapsoj wants to merge 1 commit into
Open
Conversation
smodee
requested changes
Jul 14, 2026
smodee
left a comment
Collaborator
There was a problem hiding this comment.
I'm not getting the scrapers to work - the URL seems to be a temporary session-scoped export URL, unfortunately not something we can use to reliably scrape from. The unit tests pass only because they mock the fetch, so the dead URL is never exercised. For the scraping, I would suggest going for the curated snapshot suggested in #50.
Nonetheless, I think the other changes in this PR seem solid and worth keeping:
- The CSV to HTML analytics renderer - just decouple it from the live fetch
- The csv_fetcher injection seam.
- The chunk_extractor quote-routing fix (this is independent and could be its own PR)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces two new custom scrapers for public health and livestock surveillance data, enhances the chunk-based fact extraction logic to better handle quote routing, and updates the data source configuration. The most significant changes are the addition of robust, analytics-producing scrapers for PAHO Oropouche and USDA APHIS livestock data, and improved logic to attribute quoted facts to the correct document chunk, even when quotes are found in adjacent chunks.
New data scrapers:
paho_oropouche_portal.py, a custom scraper for PAHO ARBO Oropouche weekly CSV data, which fetches the CSV, computes weekly/country analytics, fits linear and exponential models, and renders a compact HTML summary for downstream processing.usda_aphis_livestock.py, a custom scraper for USDA APHIS HPAI livestock detections, which processes the downloadable CSV, computes monthly/daily/state-level statistics, fits models, and renders a structured HTML summary.bioscancast/datasets/sources.yamlunderspecific_pathogen_sources.Fact extraction improvements:
extract_facts_from_chunkto check all chunks in a document if a quote is not found in the expected chunk, improving robustness against chunk routing misses. The correctchunk_idis now assigned to the fact source, even if the quote is found in an adjacent chunk. [1] [2]