Skip to content

fix: fetch configured CoinPaprika tickers by id#3555

Open
alceops wants to merge 3 commits into
koala73:mainfrom
alceops:fix/coinpaprika-targeted-ticker-fetch
Open

fix: fetch configured CoinPaprika tickers by id#3555
alceops wants to merge 3 commits into
koala73:mainfrom
alceops:fix/coinpaprika-targeted-ticker-fetch

Conversation

@alceops
Copy link
Copy Markdown

@alceops alceops commented May 1, 2026

Summary

  • Replace full CoinPaprika /v1/tickers?quotes=USD fallback reads with per-ID /v1/tickers/{id}?quotes=USD requests for configured IDs.
  • Share the targeted fetch helper across crypto, stablecoin, and token-panel seeders.
  • Add a node:test regression covering URL selection, duplicate ID de-dupe, and HTTP error context.

Fixes #3528

Verification

  • node --test tests/coinpaprika-targeted-fetch.test.mjs
  • node --check scripts/_seed-utils.mjs
  • node --check scripts/seed-crypto-quotes.mjs
  • node --check scripts/seed-stablecoin-markets.mjs
  • node --check scripts/seed-token-panels.mjs
  • git diff --check

@vercel
Copy link
Copy Markdown

vercel Bot commented May 1, 2026

@alceops is attempting to deploy a commit to the World Monitor Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions github-actions Bot added the trust:caution Brin: contributor trust score caution label May 1, 2026
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 1, 2026

Greptile Summary

This PR replaces the CoinPaprika bulk /v1/tickers?quotes=USD fetch (13k+ records) with per-ID requests to /v1/tickers/{id}?quotes=USD, shared via a new fetchCoinPaprikaTickersById helper in _seed-utils.mjs and adopted by all three crypto/stablecoin/token seeders.

  • The helper uses Promise.all, so a single non-OK response from CoinPaprika (404 for a deprecated ID, 429 rate limit, transient 503) rejects the entire call and drops all ticker data — this is a regression from the old behavior where a missing ID was silently skipped. Promise.allSettled with partial-result filtering would restore that resilience.

Confidence Score: 3/5

Safe to merge after addressing the Promise.all fail-fast issue in the shared helper

One P1 finding: the all-or-nothing error behavior in fetchCoinPaprikaTickersById is a reliability regression vs. the original implementation, and the shared helper is used by all three seeders, so the blast radius is wide. P1 ceiling is 4; the wide blast radius and a secondary rate-limit concern bring it to 3.

scripts/_seed-utils.mjs — the Promise.all fail-fast behavior in fetchCoinPaprikaTickersById affects all three consumer seeders

Important Files Changed

Filename Overview
scripts/_seed-utils.mjs Adds fetchCoinPaprikaTickersById helper; Promise.all fail-fast means any single per-ID error aborts all ticker data
scripts/seed-crypto-quotes.mjs Replaces bulk ticker fetch with targeted helper; logic correct assuming helper is resilient
scripts/seed-stablecoin-markets.mjs Same pattern as seed-crypto-quotes.mjs; same helper dependency
scripts/seed-token-panels.mjs Uses combined ALL_IDS (DeFi + AI tokens), making rate-limit risk highest here; otherwise clean refactor
tests/coinpaprika-targeted-fetch.test.mjs Covers URL selection, dedup, and HTTP error path; no test for partial-failure (mixed ok/error) scenario

Sequence Diagram

sequenceDiagram
    participant S as Seeder<br/>(crypto/stablecoin/token)
    participant U as _seed-utils<br/>fetchCoinPaprikaTickersById
    participant CP as CoinPaprika API<br/>/v1/tickers/{id}

    S->>U: fetchCoinPaprikaTickersById(paprikaIds)
    U->>U: dedupe IDs via Set

    par Promise.all — one request per ID
        U->>CP: GET /v1/tickers/btc-bitcoin?quotes=USD
        CP-->>U: 200 { id, quotes }
    and
        U->>CP: GET /v1/tickers/eth-ethereum?quotes=USD
        CP-->>U: 200 { id, quotes }
    and
        U->>CP: GET /v1/tickers/bad-id?quotes=USD
        CP-->>U: 404 Not Found
    end

    Note over U: Promise.all rejects on first error —<br/>all other ticker data is discarded
    U-->>S: throws Error("CoinPaprika bad-id HTTP 404")
Loading

Reviews (1): Last reviewed commit: "fix: fetch configured CoinPaprika ticker..." | Re-trigger Greptile

Comment thread scripts/_seed-utils.mjs Outdated
Comment on lines +58 to +65
const tickers = await Promise.all(ids.map(async (id) => {
const resp = await fetchFn(`https://api.coinpaprika.com/v1/tickers/${encodeURIComponent(id)}?quotes=USD`, {
headers,
signal: AbortSignal.timeout(timeoutMs),
});
if (!resp.ok) throw new Error(`CoinPaprika ${id} HTTP ${resp.status}`);
return resp.json();
}));
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Promise.all fail-fast drops all data on any single ticker error

Promise.all rejects immediately if any one per-ID request returns a non-OK status (404 for a deprecated ticker, 429 from rate limiting, or a transient 503). The old bulk-fetch path only filtered the result set, so a missing ID was a silent no-op rather than a total failure. Now a single bad ID (or a transient CoinPaprika hiccup on any one request) aborts the entire fetchFromCoinPaprika call, losing all ticker data from the seeder's fallback path.

Consider Promise.allSettled with a filter so partial failures are logged but surviving tickers are still returned:

const results = await Promise.allSettled(ids.map(async (id) => {
  const resp = await fetchFn(`https://api.coinpaprika.com/v1/tickers/${encodeURIComponent(id)}?quotes=USD`, {
    headers,
    signal: AbortSignal.timeout(timeoutMs),
  });
  if (!resp.ok) throw new Error(`CoinPaprika ${id} HTTP ${resp.status}`);
  return resp.json();
}));
const failed = results.filter(r => r.status === 'rejected');
if (failed.length) {
  console.warn(`  [CoinPaprika] ${failed.length} ticker(s) failed: ${failed.map(r => r.reason?.message).join(', ')}`);
}
return results.filter(r => r.status === 'fulfilled').map(r => r.value);

Comment thread scripts/_seed-utils.mjs Outdated
Comment on lines +58 to +65
const tickers = await Promise.all(ids.map(async (id) => {
const resp = await fetchFn(`https://api.coinpaprika.com/v1/tickers/${encodeURIComponent(id)}?quotes=USD`, {
headers,
signal: AbortSignal.timeout(timeoutMs),
});
if (!resp.ok) throw new Error(`CoinPaprika ${id} HTTP ${resp.status}`);
return resp.json();
}));
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Concurrent burst may trigger CoinPaprika rate limits

All N per-ID requests are fired simultaneously via Promise.all. CoinPaprika's free tier enforces a request-per-minute cap. seed-token-panels.mjs combines DeFi and AI token IDs into ALL_IDS, which could be large enough to trigger a 429 on the burst. Because the rate-limit 429 from any single request causes a Promise.all rejection (per the fail-fast concern above), the entire fallback path would fail instead of just the one token.

A small sequential delay between requests, or batching (e.g., groups of 5 with a pause), would reduce the risk without significantly affecting total latency.

@koala73
Copy link
Copy Markdown
Owner

koala73 commented May 2, 2026

Review

🛑 Blocking

1. The issue's primary target file is untouched. Issue #3528 explicitly cites server/worldmonitor/market/v1/_shared.ts:390 as the latent risk — that's the Vercel Edge runtime path where the 13k-ticker payload threatens edge memory limits and adds upstream latency for live user traffic. This PR only fixes the three Railway cron seeders under scripts/ and leaves fetchCoinPaprikaMarkets in _shared.ts reading the full /tickers catalog (still at line 387 on main). Closing #3528 with this PR ships a partial fix under a closed-issue marker. The fix ports cleanly — _shared.ts already has paprikaIds = geckoIds.map(...).filter(Boolean), so a TS-typed version of fetchCoinPaprikaTickersById drops in over lines 384–417.

⚠️ Concerns

2. Promise.all fail-fast amplifies the fallback's failure surface from 1 to N. Previously the catalog endpoint was atomic: one stale/renamed CoinPaprika ID was silently filtered out by paprikaSet.has(t.id). Now a single 404 (stale config, retired coin) or transient 5xx on any per-ID endpoint rejects the whole fallback (scripts/_seed-utils.mjs:64). This is a silent-narrow → loud-fail regression on the path that exists precisely because the primary already failed — same shape as the Decodo .gov* STALE_SEED escalation (a fallback with no recovery path). Suggest Promise.allSettled + skip-failed-IDs (mirrors the old bulk-filter semantics) and a per-ID failure log line.

3. No test for partial-failure behavior. tests/coinpaprika-targeted-fetch.test.mjs covers the all-fail case (one ID, 404 → throws). It doesn't cover the realistic mixed case (1 OK + 1 5xx) — which is the new failure mode this change introduces.

🤏 Nits

4. Header default replaces, doesn't merge (scripts/_seed-utils.mjs:53). options.headers || default means a caller passing { Authorization: 'X' } silently loses Accept and User-Agent. Spread it: { Accept: '...', 'User-Agent': CHROME_UA, ...options.headers }.

✅ Correct

  • De-dup via [...new Set(...)] preserves first-seen order; the test asserts this. ✓
  • N is small (10 crypto, 5 stablecoin, ~9 token) — no concurrency cap needed; CoinPaprika free tier handles 25 req/s. ✓
  • Reverse-map and downstream shape unchanged across all three seeders. ✓
  • Test-injectable fetchFn is a clean DI seam. ✓

Recommendation

Either (a) expand scope to port the helper into server/worldmonitor/market/v1/_shared.ts:384-417 before merge (the issue's stated high-severity path), or (b) narrow the PR's claim — drop "Fixes #3528," reword to "reduce CoinPaprika fallback payload in cron seeders," and open a follow-up for the edge path. The Promise.allPromise.allSettled change should land in this PR regardless.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

trust:caution Brin: contributor trust score caution

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CoinPaprika fetch pulls all 13k+ tickers per request, filters client-side

2 participants