Fall back to text-only posts and collapse Telegram polling noise

[Danswar]

Two independent problems in the social-media notification path, both visible in the deployed mainnet instance's logs.

1. Video-bearing notifications are silently dropped

CONFIG.telegram.imagesDir comes from TELEGRAM_IMAGES_DIR, and the notification assets are not in this repo at all, so every video-bearing message builds a path that is not a readable file — literally undefined/<file>.mp4 when the variable is unset. node-telegram-bot-api reinterprets a path that is not a readable file as a URL, Telegram answers 400 Bad Request: wrong HTTP URL specified, and sendMessage's catch only logs a warning — the notification is dropped, with no text-only fallback.

Affected message types: Trade, MintingUpdate, SavingUpdate, StablecoinBridgeUpdate, FrontendCodeRegistered. Plain-text alerts (minter, leadrate, position) keep working, which is why the bot looks healthy while a whole class of notifications never arrives. The failures go back at least to 2026-06-30 and only fire when there is on-chain activity, which is why they appear in clusters.

Setting the env var alone would not fix it — the media files are not present in the image at all.

This ports the guard d-EURO/api#115 already uses: resolveMediaPath checks the file with existsSync and returns undefined, so the send degrades to text-only instead of sendVideo. TwitterService is the other caller of that shared helper and had the same shape — it logged an upload error per post before falling back — so it is included.

A missing asset is reported once per distinct path per boot, at warn. It is a deployment defect that never heals on its own, so it has to stay visible; with the assets absent this is up to five lines per service per boot, and it goes to zero once the assets ship.

2. Telegram gateway outages produce a burst of raw error lines

Telegram's gateway answers 502/504 during its own restarts. Two library behaviours turn that into a log burst:

• With no polling_error listener attached, node-telegram-bot-api falls back to its own console.error('error: [polling_error] %j', error) (src/telegramPolling.js) — unformatted, at error level, bypassing the application logger entirely.
• The polling loop reschedules on a fixed interval with no backoff regardless of error, and ignores Telegram's retry after. This bot's worst retained window produced 69 error lines over ~18 minutes, one failed poll roughly every 16 seconds, and a short gateway blip produces around three lines per second. Some bursts also carry 429 Too Many Requests: retry after N; in every retained instance the 429 opens the burst, ahead of the 502 run — and because the loop ignores retry after, it answers a retry after 5 with seven 429s in a row, spaced about 324 ms apart, before Telegram switches to 502s.

The listener reports one line per distinct failure, repeats an outage that never clears every five minutes, and closes it with one line. Digit runs are masked out of the failure signature so a counting-down retry after N and a rotating gateway address collapse instead of looking new each time; the number of distinct signatures per outage is capped so a per-attempt token in an upstream error body cannot defeat either the collapsing or the memory bound.

Replaying real failure shapes against the built code:

this bot's worst window, ~18 min ...   69 raw errors ->  6 lines
a 429-led burst, 7x 429 + 12x 502 ..   19 raw errors ->  3 lines
an 8s blip, 27x 502 (sibling shape)    27 raw errors ->  2 lines
single isolated blip ...............    1 raw error  ->  1 line
revoked token, 20 min of polling ... 4000 raw errors ->  4 lines (outage still open)
2000 polls, every message unique ... 2000 raw errors -> 22 lines, signature set held at 20

Sample output for the 429-led burst row above, replayed from that burst's real timestamps:

WARN  Telegram polling failing (attempt 1): ETELEGRAM: 429 Too Many Requests: retry after 5
WARN  Telegram polling failing (attempt 8): ETELEGRAM: 502 Bad Gateway
WARN  Telegram polling errors stopped after 8s and 19 attempts

Below the two limits noted next, every distinct failure is reported on arrival — including a 429 and a 502 within the same burst — and a permanently failing poll keeps reporting. Two mechanisms deliberately bound the worst case rather than reproducing it, and both cost information. The error text is truncated to the first MAX_POLLING_ERROR_LENGTH characters before it is both logged and reduced to a signature, so anything past that never reaches the log, and failures differing only beyond it collapse into one. And once MAX_POLLING_SIGNATURES distinct failures have been reported within a single outage, a further new failure is no longer reported on arrival: from then on the outage emits at most one line per report interval, carrying whichever failure is in flight when that report falls due — so a new failure that has been superseded by then, or an outage that clears first, is never shown. Both log lines — the onset and the closing one — are at warn on purpose: an outage whose end is invisible under a warn-level log configuration is worse than one extra line. The closing line deliberately says errors stopped rather than recovered — the grace period only establishes that no further error arrived, it does not probe the poll.

Note that no inbound updates are lost during these outages — getUpdates does not advance its offset on failure, so Telegram re-delivers on the next successful poll. This change is about how those failures are logged. The retry cadence itself is unchanged — no backoff is added and retry after is still ignored, so the underlying storm, including the 429 bursts described above, still happens; it just stops being reprinted line by line. Message-delivery correctness is unaffected.

The polling listener is proposed for the sibling bot in d-EURO/api#127 with an identical implementation.

Verification

• yarn build clean, eslint and prettier --check clean on all four touched files.
• Collapse behaviour replayed against dist/ for each scenario above.
• No unit tests added: the repo's jest config has rootDir="src" but the sources live at the repo root, so yarn test aborts before test discovery (Validation Error: Directory .../src in the rootDir option was not found) and no test can run today. Reworking the test topology is out of scope, matching the precedent in Handle missing social-media tokens and assets gracefully d-EURO/api#115.

[Danswar]

This went through 12 review passes before it was clean. Recording what changed, since several rounds altered real behaviour:

Code fixes (rounds 1–3)

• The missing-asset notice was at debug, which the deployed logger does not emit. Since the assets are absent the fallback is the permanent state, so the condition would have gone completely silent — it was visible as a warning before this branch. Now warn, once per distinct asset per boot.
• The polling signature compared raw messages against the previous one only. That fails in exactly the two cases that occur: retry after N counts down, and a connect failure carries a rotating gateway address. Digits are now masked out of the signature, and the signatures already reported within the outage are tracked as a set.
• A permanently failing poll — a revoked token answers 401 forever — reported once and then went silent, because the recovery timer only fires once errors stop. It now repeats every five minutes while the outage is open.
• TwitterService, the other caller of the shared helper, was missed and logged an upload error per post.
• The signature set had no upper bound, an isolated blip cost two log lines, and the closing line claimed "recovered" although nothing probes the poll.

Description fixes (rounds 4–12) — all cases where the description claimed more than the code delivers:

• It said the change was about the retry storm. It adds no backoff and retry after is still ignored; only the reprinting stops.
• It said nothing that used to be visible becomes silent. The signature cap and the 200-character truncation both cost information, and both are now stated.
• A 27-line burst was attributed to this bot; it is the sibling bot's. This bot's own worst retained window is 69 lines over ~18 minutes.
• The 429 was described as an escalation on top of the 502. The logs show the opposite in 14 of 14 retained bursts: the 429 opens the burst and the 502 run follows.
• Deployment configuration was asserted as fact in a public repo.

Every replay figure in the description was re-derived against the built code rather than estimated.