fix(blockchain): reliability hardening — issues #917 #933 #934 #937

[knytcomics-ui]

Summary

Resolves four blockchain reliability issues in a single cohesive change. All modifications are in services/api/src/ (Rust/Axum), with supporting changes to config, metrics, alert rules, a new DB migration, and README.

---

Closes

• Closes Watched TX TTL (30 min) is hardcoded — should be configurable solutions-plug/predictIQ#933 — Watched TX TTL is hardcoded
• Closes Watched transactions HashMap has no size limit — unbounded memory growth solutions-plug/predictIQ#934 — Watched transactions HashMap has no size limit
• Closes No deduplication of watched transactions — same TX hash can be registered multiple times solutions-plug/predictIQ#937 — No deduplication of watched transactions
• Closes Stellar network reachability not validated at API startup solutions-plug/predictIQ#917 — Stellar network reachability not validated at API startup

---

Changes

solutions-plug#933 — WATCHED_TX_TTL configurable via WATCHED_TX_TTL_SECS

• Removed hardcoded const WATCHED_TX_TTL: Duration = 30 min
• Added Config::watched_tx_ttl_secs (default 1800) read from WATCHED_TX_TTL_SECS env var
• BlockchainClient now carries a watched_tx_ttl: Duration field populated at construction
• Migration 019_create_watched_transactions.sql adds an expires_at TIMESTAMPTZ column that mirrors the same TTL for any persisted approach

solutions-plug#934 — WATCHED_TX_MAX_SIZE cap + Prometheus gauge + alert rule

• Behaviour change: when the map is at capacity, new registrations are now rejected with 503 Service Unavailable instead of silently evicting the oldest entry. Callers get a clear error; monitoring coverage of existing hashes is never silently degraded.
• Added Config::watched_tx_max_size (default 10_000) read from WATCHED_TX_MAX_SIZE
• Added Metrics::watched_tx_count IntGauge updated on every insert, eviction, and removal
• Added two Prometheus alert rules in performance/config/alerts.yaml:
  • WatchedTxCountHigh: fires when watched_tx_count > 8000 (80% of default cap) for 5 min — severity warning
  • WatchedTxCountCritical: fires when watched_tx_count >= 10000 for 1 min — severity critical

solutions-plug#937 — Deduplication of watched transactions

• watch_transaction now returns Result<(), WatchTxError> instead of ()
• WatchTxError::AlreadyWatched — hash already in map; no second entry inserted
• WatchTxError::CapReached — map full; caller responsible for 503
• blockchain_tx_status handler matches on both variants:
  • AlreadyWatched is idempotent — status is still returned to the caller
  • CapReached returns 503 SERVICE_UNAVAILABLE with a clear message
• Internal sync_once ignores AlreadyWatched (benign) and logs a warning on CapReached
• Migration 019 adds UNIQUE (tx_hash) constraint for DB-layer dedup enforcement

solutions-plug#917 — Stellar RPC reachability probe at startup + /health/ready endpoint

• validate_network_passphrase now differentiates environments via PREDICTIQ_ENV:
  • PREDICTIQ_ENV=production → process::exit(1) on RPC unreachable or passphrase mismatch
  • All other environments → warning logged, process continues (unblocks local dev)
• Added BlockchainClient::probe_stellar_ready() -> bool for on-demand liveness checking
• Added GET /health/ready readiness endpoint:
  • 200 OK + { "ready": true, "stellar_rpc": "ok" } when RPC is reachable and passphrase matches
  • 503 Service Unavailable + { "ready": false, "stellar_rpc": "unreachable" } on failure
  • Registered in main.rs public routes alongside /health

---

Files Changed

File	What changed
src/blockchain.rs	WatchTxError enum; watch_transaction returns Result; configurable TTL/cap; probe_stellar_ready; dev/prod passphrase behaviour; gauge updates
src/config.rs	watched_tx_ttl_secs, watched_tx_max_size, is_production fields; test struct fixes
src/metrics.rs	watched_tx_count IntGauge; set_watched_tx_count method
src/handlers.rs	blockchain_tx_status handles WatchTxError; new health_ready handler
src/main.rs	/health/ready route registered
database/migrations/019_create_watched_transactions.sql	New table with UNIQUE (tx_hash) and expires_at
database/migrations/rollbacks/019_create_watched_transactions_down.sql	Rollback
performance/config/alerts.yaml	WatchedTxCountHigh and WatchedTxCountCritical alert rules
README.md	Docs for all new env vars, /health/ready, and watched_tx_count metric
.env.example	Commented-out examples for new variables

---

Testing

Unit tests updated in blockchain.rs to cover:

• watch_transaction_dedup_returns_already_watched — second registration of same hash returns AlreadyWatched and does not insert a duplicate
• watch_transaction_cap_returns_cap_reached — insertion when map is full returns CapReached and map size stays at cap
• watched_txs_ttl_evicts_stale_entries — entries older than TTL are removed on next write
• watch_tx_error_variants_are_distinct — enum variants are distinguishable

Metrics test updated to assert watched_tx_count 42 appears in rendered output.