Data is foundational for data science. Trove lets anyone search, watch, and poll data over time for a wide range of different source data signals.
Every fetch writes a timestamped row to a local SQLite cache.
This project is a hoarding engine for ethemeral time series, and export hands the whole thing
to your other tools as CSV (schema in DATA_DICTIONARY.md).
For personal use only. The bundled sources only hit sanctioned and keyless APIs that the page or source app itself calls.
poll spaces its requests with a real User-Agent and the cached data stays
local. Respects the terms of service and robots.txt of each source.
Every source uses the same eight commands
python trove.py <source> doctor # is the API reachable? (no args)
python trove.py <source> search <query> # find items — the LEFT COLUMN is the ID
python trove.py <source> item <id> # full detail for one item
python trove.py <source> watch add <id> # start tracking it
python trove.py <source> poll # re-check watched items, log history, flag drops/deals
python trove.py <source> deals # watched items that are "good" right now
python trove.py <source> drops # watched items now lower than first seen
python trove.py <source> export # dump the cached time-series to CSVThe only
things that vary per source are the ID format and whether search is free-text or a fixed list:
- Free-text:pass a real query —
steam search "elden ring". - Fixed list (nzski): the source already knows its set; pass
""to list them all, or a word to filter —search "".
python trove.py <source> -h lists its commands and any extra flags
(--cc, --limit, itunes search --entity album, eventcinemas --cc 502).
Every source runs the same commands: doctor search item watch poll deals drops export, plus a few
source-specific search flags (e.g. itunes search --entity album).
Pure stdlib plus requests (pip install -r requirements.txt). No API keys for the bundled sources.
trove.py entrypoint: python trove.py <source> <command> [args]
trove/
session.py retry_session() - shared backoff HTTP session
db.py TrackerDB + Item/Obs - the stateful spine (items, timestamped obs, watch)
tracker.py Source contract + run_cli (the generic command set lives here, once)
sources/
steam.py ~90 lines: endpoints + normalise payload -> Item/Obs + deal semantics
discogs.py
itunes.py
scryfall.py
The core owns everything that is stateful and generic: caching, the timestamped observation log, the
watchlist, drop detection, deal transitions. A source owns only what is unique to it: the endpoints,
how to flatten the payload into Item/Obs, and what "a deal" means.
100 sources in sixteen genres (the same grouping python trove.py prints); several sources hoard
multiple boards (bikeshare 24 GBFS systems, outages 5 utility networks, civic311 3 cities) for 129
boards in total:
| source | join key | timeline value | API |
|---|---|---|---|
| steam | appid | game price + discount % | keyless Storefront API |
| discogs | release id | marketplace lowest price + num for sale | keyless official API |
| itunes | trackId/collectionId | app/album/song price + going free | keyless official Search API |
| scryfall | card id | MTG single price (usd/eur/tix) + foil deal | keyless official API |
| pokemontcg | card id | Pokemon single market price (usd/eur) + under-market deal | keyless official API |
| ygoprodeck | card id (passcode) | Yu-Gi-Oh single price per venue + retailer arbitrage | keyless official API |
| epic | offer id | free-game rotation: RRP -> Free window + which titles given away | keyless store backend |
| steammarket | appid:market_hash_name | Steam Community Market lowest ask + listing depth + 24h volume; deal = below 24h median | keyless market backend |
| source | join key | timeline value | API |
|---|---|---|---|
| spainfuel | province-IDEESS | per-station petrol price (G95E5) + below-area-avg deal | keyless MINETUR open-data REST |
| petrolspy | station id | NZ per-station fuel price (U91) + below-box-avg deal | keyless PetrolSpy map API |
| em6 | grid_zone_id | NZ wholesale electricity spot ($/MWh) + below-NZ-avg deal | keyless em6 public tier |
| octopus | GSP group (A-P) | UK Agile Octopus half-hourly unit rate (p/kWh) + cheap-window/plunge deal | keyless official Octopus API |
| aemo | NEM region (NSW1...) | AU National Electricity Market 5-min spot price ($/MWh) + demand + interconnector flows; deal = below the 5-region avg (or negative) | keyless AEMO visualisations API |
| fuelwatch | suburb:address | WA per-station fuel price (ULP cents/L, legally fixed daily) + below-suburb-avg deal | keyless WA Govt FuelWatch RSS |
| awattar | market (de/at) | Germany/Austria EPEX day-ahead hourly spot (Eur/MWh); one item seeds ~90 days of hourly history; deal = below the window avg (negative = plunge) |
keyless aWATTar marketdata API |
| carbonintensity | GB region (1-17) | Great Britain grid carbon intensity forecast (gCO2/kWh) + generation mix, as-issued (drift); deal = index "very low"/"low" | keyless National Grid ESO Carbon Intensity API |
| nyiso | NY zone (WEST...) | New York ISO real-time zonal electricity price (LBMP $/MWHr), 5-min; deal = at/below the all-zone avg (US twin of em6/aemo) | keyless NYISO realtime_zone CSV |
| francefuel | station id | per-station French forecourt fuel price (--cc = grade: gazole/sp95/sp98/e10/e85/gplc), overwritten in place; deal = below the national sample avg |
keyless Opendatasoft prix-carburants v2 |
| energinet | bidding zone (DK1/DK2/DE/NO2/SE3/SE4) | Nordic/DE day-ahead electricity price (EUR/MWh) per bidding zone; deal = at/below the all-zone avg or negative (EU twin of em6/aemo/nyiso) | keyless Energinet DayAheadPrices |
| aeso | AB (single market) | Alberta wholesale pool price ($/MWh) + AIL demand, live hourly; deal = at/below the 30-hour rolling avg (CA twin of em6/aemo/nyiso) | keyless AESO SMP report CSV |
| elexon | APXMIDP (GB market) | GB half-hourly wholesale market-index price (GBP/MWh) + volume; deal = at/below today's mean so far (GB wholesale twin of octopus retail) | keyless Elexon BMRS Insights |
| ree | series (pvpc/spot) | Spain Iberian electricity price (EUR/MWh) — regulated PVPC + wholesale spot, bound to the current hour; deal = at/below the day mean | keyless Red Electrica apidatos |
| elia | imbalance (BE) | Belgium imbalance price (EUR/MWh), 15-min, can go negative + system imbalance; deal = price at/below zero (being long is paid) | keyless Elia Open Data (ODS ods134) |
| italyfuel | station id (idImpianto) | per-station Italian forecourt fuel price (--cc = grade: benzina/gasolio/gpl/metano), self-service, overwritten in place; deal = below the national sample avg |
keyless MIMIT Osservaprezzi CSV |
| austriafuel | E-Control station id | cheapest live-price diesel near a city (--cc = wien/graz/linz/salzburg/innsbruck/klagenfurt); only the cheapest publish by law; deal = below the nearby-set avg |
keyless E-Control Spritpreisrechner |
| source | join key | timeline value | API |
|---|---|---|---|
| frankfurter | BASE:QUOTE (e.g. NZD:USD) | ECB daily FX fixing — one item call seeds the full daily series since 1999 into the obs log; deal = base at/above the 90th percentile of its trailing year (a strong moment to convert) |
keyless open-source Frankfurter/ECB API |
| paralelobo | usd (BOB pair) | Bolivia parallel-market USD/BOB rate (P2P aggregate) — un-rebuildable black-market series (vs the ~6.96 official peg); deal = >=40% premium over the peg | keyless paralelo.bo /api/rate |
| dolarapi | casa (oficial/blue/...) | Argentina (--cc = ar/ve/uy/cl/br) USD rates per casa incl. the "blue" street rate — un-rebuildable parallel-vs-official gap; deal = casa >=10% over official |
keyless dolarapi.com /v1/dolares |
| source | join key | timeline value | API |
|---|---|---|---|
| grabone | deal URL path | NZ daily-deal price + RRP/discount + live-until-expiry | page-published JSON-LD |
| grabaseat | ORIGIN-DEST route | Air NZ cheapest fare per route + standout-dip deal | keyless grabaseat fare API |
| bookme | activity URL path | NZ activity deal price + spaces-remaining + steep-discount deal | SSR page-parse |
| turners | car detail path | NZ used-car asking price + RRP/discount over a listing's life | page-published microdata |
| eventcinemas | cinemaId:date:sessionId | NZ cinema session seats-remaining ticking down to showtime (scarcity) | keyless GetSessions JSON |
| reverb | listing id | used-gear marketplace ask price + seller markdown over a listing's life (then it sells and vanishes); deal = on-sale | keyless official Reverb API |
| source | join key | timeline value | API |
|---|---|---|---|
| hackernews | story id | HN front-page rank/points trajectory (rank 27 -> 3 -> gone); deal = top-10 | keyless official HN Firebase API |
| appcharts | country:chart:appId | App Store top-chart rank rotation as published (history is paywalled commercially); deal = top-10 | keyless Apple marketing RSS |
| melbped | Melbourne sensor id | City of Melbourne live per-minute pedestrian footfall per street sensor; deal = busier than the current network median | keyless Melbourne Opendatasoft API |
| source | join key | timeline value | API |
|---|---|---|---|
| geonet | publicID | NZ earthquake magnitude + preliminary-to-reviewed quality drift (downgrade signal) | keyless GeoNet GeoJSON API |
| metno | city slug or lat,lon | weather forecast-drift: upcoming-day high + rain, as-issued (un-rebuildable) | keyless MET Norway Locationforecast |
| volcano | volcanoID | NZ volcanic alert level (0-5) + unrest escalation | keyless GeoNet VAL API |
| nzski | resort data-slug | NZ ski-field base depth + lifts/trails open (open = deal) | page-called NZSki feed |
| gwrivers | gauge site name | NZ river flow/level + flood-onset rise (1.5x in 24h) | keyless GW Hilltop XML |
| avalanche | region slug | NZ backcountry avalanche danger rating (1-5) per elevation band, as-issued daily + its revision (drift); deal = Considerable+ (>=3) | keyless page-called avalanche.net.nz /api/forecast |
| mdcrivers | gauge site name | Marlborough (NZ) river flow/level + flood-onset rise (1.5x in 24h) | keyless MDC Hilltop XML |
| horizonsrivers | gauge site name | Manawatu-Whanganui (NZ) river flow/level + flood-onset rise | keyless Horizons Hilltop XML |
| northlandrivers | gauge site name | Northland (NZ) river flow/level + flood-onset rise (subtropical far north) | keyless NRC Hilltop XML |
| westcoastrivers | gauge site name | West Coast (NZ) river flow/level + flood-onset rise (wettest region; flashy alpine rivers) | keyless WCRC Hilltop XML |
| nswrfs | incident id | NSW (AU) bush/grass fire lifecycle: alert level escalating (Advice->Watch and Act->Emergency Warning) + size, then resolution; deal = out-of-control fire at Watch and Act+ | keyless RFS majorIncidents feed |
| vicemergency | event id | Victoria (AU) all-hazards warnings/incidents: alert-level lifecycle across fire/flood/storm; deal = Watch and Act+ | keyless VicEmergency GeoJSON |
| sacfs | incident id | South Australia (AU) CFS incidents: response level + status (GOING->CONTROLLED); deal = still GOING | keyless SA CFS feed |
| beachwatch | site id (uuid) | NSW (AU) beach water-quality star rating + daily pollution forecast; deal = pollution Possible/Likely (swim advisory) | keyless NSW Beachwatch GeoJSON |
| safeswim | beach slug | NZ beach water-quality traffic-light (GREEN/RED/RED+/BLACK), flips with rainfall; deal = a water-quality alert | keyless page-called Safeswim API |
| eafloods | flood-area id | England live flood warnings/alerts: severity lifecycle (Alert->Warning->Severe->stood down), then the area drops off the feed; deal = a live Flood Warning+ (event-driven, often empty in dry spells) | keyless EA flood-monitoring API (OGL) |
| usgs | USGS site number | US river streamflow + gauge height at 5-15min telemetry (--cc = a US state); flood-onset rise (1.5x in 24h) |
keyless USGS Water Services IV |
| wildfire | IRWIN id | US wildland fire incident lifecycle: acreage growth + containment % climb, then it's out and drops off the current layer; deal = active fire >=1000 acres and <50% contained | keyless NIFC/WFIGS ArcGIS Feature Service |
| airquality | sensor id | live citizen PM2.5/PM10 per air-quality sensor (--cc = a curated city); deal = PM2.5 >=25 ug/m3 (unhealthy) |
keyless Sensor.Community API |
| luchtmeetnet | NL station id | official Netherlands PM2.5 per RIVM reference monitor; deal = PM2.5 >=25 ug/m3 (unhealthy); calibrated government twin of airquality | keyless Luchtmeetnet open API |
| usgsquakes | USGS event id | global earthquakes (magnitude + place + depth + felt/MMI/tsunami), as-reported then revised; deal = M>=4.5 or tsunami flag (global geonet twin) | keyless USGS FDSN event API |
| hkweather | forecast date (YYYYMMDD) | Hong Kong 9-day forecast-drift (per-day max/min temp as issued) + live typhoon/rainstorm warnings; deal = max >=33C or a warning in force (metno twin, opens Asia) | keyless HK Observatory opendata |
| ipma | IPMA globalIdLocal | Portugal next-day city forecast (max/min temp as issued + rain probability + weather type); deal = rain probability >=70% (metno twin, EU) | keyless IPMA open-data |
| hbrivers | gauge site name | Hawke's Bay (NZ) river flow/level + flood-onset rise (east-coast Cyclone Gabrielle catchments) | keyless HBRC Hilltop XML |
| imgw | station id | live Polish synoptic-station temp + wind + rain + pressure (whole network, one GET); deal = station recording rain (opens Poland) | keyless IMGW dane publiczne |
| buienradar | station id | live Dutch weather-station temp/wind/pressure (~40 stations); deal = rain in the weather description (opens NL weather) | keyless Buienradar feed |
| smhi | station key | live Swedish air temperature per station (whole network, one GET); deal = temp >=25C (opens Sweden) | keyless SMHI metobs |
| fmi | city slug | live Finnish obs (temp/wind/humidity) per city (--cc); deal = temp >=25C (opens Finland) |
keyless FMI open-data WFS |
| icelandweather | station id | live Icelandic obs (temp/wind/rain) across ~10 stations; deal = station recording precipitation (opens Iceland) | keyless Vedurstofan xmlweather |
| meteireann | station slug | live Irish station obs (temp/wind/rain, ~12 stations); deal = rainfall >0 (opens Ireland) | keyless Met Eireann prodapi |
| jmaweather | office code | Japan next-day city forecast (--cc = tokyo/osaka/…): weather code + max/min temp + rain prob, as issued; deal = rain code or rain prob >=50% (opens Japan) |
keyless JMA open forecast JSON |
| sgrain | station id (S77…) | live Singapore per-gauge rainfall (mm/5min, ~77 gauges) — hyper-local tropical downpours; deal = gauge recording rain (opens SG environment) | keyless data.gov.sg rainfall |
| eqcanada | NRCan EventID | Canadian earthquakes (magnitude + place + depth), as-reported then revised; deal = M>=4.0 or a felt event (opens CA seismic) | keyless NRCan FDSN event API |
| bmkg | event DateTime | Indonesian earthquakes (magnitude + depth + tsunami potential); deal = M>=5.0 or tsunami (opens Indonesia) | keyless BMKG TEWS feed |
| turkeyquake | AFAD eventID | Turkish earthquakes (magnitude + place + province); deal = M>=4.0 (opens Turkey) | keyless AFAD event filter |
| jmaquake | JMA event id (eid) | Japanese earthquakes: magnitude + the JMA seismic-intensity scale (shindo 1-7); deal = M>=4.5 or intensity 5- and above (opens JP seismic) | keyless JMA quake list |
| usgsvolcano | vnum | US volcanoes above background: aviation colour code (GREEN/YELLOW/ORANGE/RED) + alert level, de-escalation drift; deal = ORANGE/RED | keyless USGS HANS |
| gdacs | eventtype:eventid | global multi-hazard alerts (EQ/cyclone/flood/volcano/drought/wildfire): Green/Orange/Red alert lifecycle; deal = ORANGE/RED (global all-hazards board) | keyless GDACS EVENTS4APP |
| source | join key | timeline value | API |
|---|---|---|---|
| spaceweather | UTC forecast date | planetary Kp forecast: per-day peak + storm/aurora drift (Kp>=5 = aurora australis) | keyless NOAA SWPC feed |
| sentry | Sentry designation | asteroid impact-risk drift: Palermo/Torino/impact-probability revisions, then retirement from the risk list | keyless JPL/CNEOS Sentry API |
| spacelaunch | LL2 launch id (uuid) | upcoming rocket launch readiness (Go/TBC/TBD/Hold) + the NET slip as-scheduled (drift), then it flies and leaves upcoming; deal = Go within 24h |
keyless Launch Library 2 (thespacedevs) |
| source | join key | timeline value | API |
|---|---|---|---|
| chcflights | dir:type:flightNo:scheduled | Christchurch Airport flight delay-drift (estimate vs schedule) + gate/status churn; deal = delayed/cancelled | keyless christchurchairport.co.nz /api/flights JSON |
| zqnflights | dir:flightNo:schDate:schTime | Queenstown Airport delay-drift + status churn (NZ's most disruption-prone board); deal = delayed/cancelled | keyless queenstownairport.co.nz /api/flights JSON |
| opensky | icao24 (in a bbox) | live aircraft over a region (--cc picks the box: nz/au/uk/nyc/sf/la): altitude/speed/heading snapshot, un-rebuildable anon; deal = airborne, <3000m and descending (on approach) |
keyless OpenSky Network state vectors (anon) |
| adsblol | icao hex (in range) | live aircraft near a point (--cc region centre+radius: lon/nz/au/syd/nyc/la/sf): altitude (ft)/speed/heading; deal = airborne, <10000ft and sinking (approach); keyless-community opensky twin |
keyless adsb.lol community ADS-B |
| adsbfi | icao hex (in range) | second keyless-community ADS-B network (--cc region as adsblol): live aircraft altitude/speed/heading; deal = airborne, <10000ft and sinking |
keyless adsb.fi community ADS-B |
| source | join key | timeline value | API |
|---|---|---|---|
| nzroads | NZTA event id | national highway disruption lifecycle: impact escalating/easing (Caution/Delays/Road Closed), then resolution (the event vanishes); deal = an unplanned active disruption | keyless page-called journeys.nzta.govt.nz delays.json |
| tfl | line id | London Underground/DLR/Overground/Elizabeth line status ordinal (Good Service -> Minor/Severe Delays -> Part Suspended) drifting through the day; deal = any non-Good-Service | keyless TfL Unified API |
| mbta | alert id | Boston MBTA service alerts: severity (0-10) + effect lifecycle, then the alert clears; deal = a serious effect (suspension/shuttle) or severity>=7 | keyless MBTA V3 JSON:API |
| swisstransport | station|line|to|schedTs | Swiss rail/tram departure delay-drift in the minutes before it leaves, then it's gone; deal = running >=3 min late | keyless transport.opendata.ch stationboard |
| bcferries | DEP-DEST:time | BC Ferries sailing capacity filling (fill % overall + car deck) from on-sale to departure, un-rebuildable; deal = non-cancelled sailing >=80% full (book now) | keyless bcferriesapi.ca |
| source | join key | timeline value | API |
|---|---|---|---|
| bikeshare | system:station_id | dock-based bike-share station availability: bikes/docks free oscillating through the day (the fill/empty cycle), never archived per-station; deal = a renting station running dry (<=2 bikes); --cc picks 1 of 24 systems across US/CA/MX/EU/LatAm (GBFS v2 + v3) |
keyless open GBFS station feed |
| sgtaxi | sg (whole fleet) | Singapore roaming-taxi supply: the island-wide available-taxi count swinging with demand/weather; deal = below 2,000 (tight supply) | keyless data.gov.sg taxi-availability |
| source | join key | timeline value | API |
|---|---|---|---|
| squiggle | year:round:id | AFL (Australian Football League) game score progression: completion % (0-100) + score updates as games play to final; deal = game completed or score changed; --cc = a team name filter |
keyless Squiggle AFL API |
| espnscores | ESPN event id | live sports scores + game status (--cc = league: epl/laliga/nba/nfl/mlb/nhl/…): scoreline + pre->in->post trajectory; deal = game in progress now |
keyless ESPN scoreboard API |
| source | join key | timeline value | API |
|---|---|---|---|
| chcparking | park id | Christchurch (NZ) parking-building free-space count draining/refilling through the day (never archived per-park); deal = <=20 free spaces (nearly full) | keyless page-called CCC SmartView /api/parking |
| sgcarpark | carpark number | Singapore HDB car-park free-space count draining/refilling through the day (never archived per-park); deal = <=10 free car spaces (nearly full) | keyless data.gov.sg carpark-availability |
| source | join key | timeline value | API |
|---|---|---|---|
| outages | network:outage id | live electricity outage lifecycle (customers affected + crew status + ETR drift, restored in stages, then it drops off the feed); deal = unplanned outage affecting >=100 customers; --cc picks the network (powercor = VIC AU, energex = SE QLD AU, westernpower = WA AU, mbhydro = Manitoba Hydro CA, bchydro = British Columbia CA) |
keyless ArcGIS Feature Services (one NETWORKS row per utility) |
| source | join key | timeline value | API |
|---|---|---|---|
| noaatides | station id | US coastal water level (ft above MLLW) rising/falling with the tide + storm surge; deal = rising and near the 24h max (high tide/surge) | keyless NOAA CO-OPS datagetter |
| ndbc | buoy station id | offshore buoy sea state: significant wave height + period + wind + water temp; deal = wave height >=3 m (big swell) | keyless NOAA NDBC latest_obs |
| uktides | measure notation | UK coastal tide level (mAOD, ~214 gauges); the cyclical rise is invisible to drop logic so fetch flags rising; deal = incoming/flooding tide (UK twin of noaatides) | keyless EA flood-monitoring API (OGL) |
| source | join key | timeline value | API |
|---|---|---|---|
| civic311 | city:request id | municipal 311 service-request backlog (--cc = city: nyc/chicago/sf): a request's age-in-queue + Open->Closed lifecycle (oldest-open board = the current backlog); deal "stale" = still open after 7+ days |
keyless city Socrata 311 datasets |
| source | join key | timeline value | API |
|---|---|---|---|
| arbeitnow | job slug | EU/remote tech job-board listing lifecycle: a posting appears, sits on the board, then vanishes when filled/expired (a time-to-fill proxy); deal "fresh" = posted in the last 48h | keyless Arbeitnow job-board-api |
MIT (see LICENSE). The license covers the code.