feat: latency SLI histograms (delta staleness, tick duration, drain cycle)

[mikhail-dcl]

Summary

Adds latency measurements to Pulse: how long the server takes to fan out state (M1), how long a simulation tick takes (M2), how long the outbound flush takes (M3), and each peer's network RTT split by continent (M4). Exported as native Prometheus histograms and shown in a new Latency group on the console dashboard.

No protocol changes. No new fields on hot-path structs (OutgoingMessage, IncomingEvent, PeerSnapshot) — M1 reuses the ServerTick already stamped on every snapshot, M4 reuses the RTT that ENet already maintains.

SLIs and suggested SLOs

SLI	Prometheus series	What it measures	Suggested SLO*
Delta staleness	dcl_pulse_delta_staleness_ms{tier="0".."2"}	Delay from a player's state being published to it being handed off for an observer (per AoI tier)	tier0: p95 ≤ 40 ms, p99 ≤ 80 ms
Tick duration	dcl_pulse_tick_duration_us	SimulateTick wall time	p99 ≤ 50% of tick budget
Tick overruns	dcl_pulse_tick_overruns_total	Ticks that exceeded BaseTickMs	0
Drain cycle	dcl_pulse_outgoing_drain_cycle_us	Outbound flush duration on the ENet thread (non-empty cycles only)	p99 ≤ 5 ms
Peer RTT	dcl_pulse_peer_rtt_ms{region="af..sa,unknown"}	ENet smoothed round-trip time per peer, sampled every 5 s, split by the peer's continent	observability SLI — no target; feeds region-deployment decisions

*Starting points, not calibrated targets — validate against the first real load test before alerting on them.

How to treat each one

Delta staleness — the player-facing number. Judge the SLO on tier 0 only: tiers 1/2 update at 2×/4× tick cadence, so they sit higher by design. Resync deltas are excluded, so every sample reflects steady-state fan-out.

• p99 rising → check tick duration and queue depths first; they explain almost every staleness regression.

Tick duration — the main explainer for staleness. Creeping toward the budget means CPU saturation or AoI fan-out growth: scale out or reduce CCU per instance.

Tick overruns — treat as an alert, not a trend. Any sustained non-zero rate means the instance is over capacity.

Drain cycle — bounds how long an outgoing packet waits for the ENet thread. Growing together with dcl_pulse_outgoing_queue_depth means the ENet thread is saturated.

Peer RTT — network latency, not server latency: it separates "our server is slow" from "the player is far away". One region's p50 far above the rest (e.g. as at 250 ms against an EU deployment) is the quantified case for a closer instance — and the same series verifies the improvement after deploying one. unknown covers private/unresolvable IPs (test bots land here); if unknown dominates in production, the geo database is missing from the image. Peer continents come from the CC0 geo-whois-asn-country database, fetched once at Docker build (no runtime downloads); ENet seeds RTT at 500 ms until the first ACK, so freshly connected peers add brief upward noise.

Grafana

Data source: the Prometheus instance scraping /metrics.

1. Staleness percentiles — Time series panel, unit ms, threshold line at 80:

histogram_quantile(0.99, sum by (le) (rate(dcl_pulse_delta_staleness_ms_bucket{tier="0"}[5m])))

Duplicate the query with 0.95 and 0.50. sum by (le) aggregates correctly across instances — add label filters to scope to one.

2. Tick duration p99 — Time series panel, unit µs:

histogram_quantile(0.99, sum by (le) (rate(dcl_pulse_tick_duration_us_bucket[5m])))

3. Tick overruns — Stat panel, thresholds: green = 0, red > 0:

sum(rate(dcl_pulse_tick_overruns_total[5m]))

4. Drain cycle p99 — same shape as (2) with dcl_pulse_outgoing_drain_cycle_us_bucket.

5. Peer RTT by region — Time series panel, unit ms, one query per region or a single query legended by region:

histogram_quantile(0.5, sum by (le, region) (rate(dcl_pulse_peer_rtt_ms_bucket[5m])))

6. Staleness heatmap (optional) — Heatmap panel, query format Heatmap:

sum by (le) (rate(dcl_pulse_delta_staleness_ms_bucket{tier="0"}[1m]))

Mean value when needed: rate(<series>_sum[5m]) / rate(<series>_count[5m]).

Testing

• TDD throughout; full suite 403/403 green.
• Docker: all three images build and verified locally — geo CSVs present in each image (8.2 MB IPv4 + 17.5 MB IPv6), and the prod image loads 550,668 ranges at startup (smoke-booted). A .gitattributes *.sh eol=lf rule was added so Windows checkouts can build the Linux images.
• Operator docs: docs/metrics.md → "Latency metrics".

🤖 Generated with Claude Code

[decentraland-bot]

PR Review — feat: latency SLI histograms

STEP 2 — Root-cause check

PASS. This PR adds observability instrumentation — five new SLIs (delta staleness, tick duration, tick overruns, drain cycle, peer RTT by continent). It solves the problem of having no quantitative latency visibility. The metrics measure real, well-defined quantities at the correct measurement points. No symptom-masking.

STEP 3 — Design & integration

PASS. All new units are correctly placed:

BucketHistogram / HistogramSnapshot / HistogramTracker — extend the existing metrics pipeline pattern (Counter → RateTracker → RateStatsView → ConsoleDashboard) with a histogram variant. BucketHistogram is the accumulator (parallel to the Interlocked counters already in MeterListenerMetricsCollector), HistogramTracker is the dashboard adapter (parallel to RateTracker). No lifecycle duplication — these are stateless value containers within the existing MeterListenerMetricsCollector → MetricsSnapshot → ConsoleDashboard/PrometheusFormatter pipeline.

ContinentResolver — a pure lookup table loaded once at startup from static files, registered as a DI singleton. Consumed by ENetHostedService at peer connect time to resolve continent, stored as part of the new ConnectedPeer record struct alongside the ENet Peer handle. The continent is resolved at the connect moment (not polled per-tick), which is correct. ConnectedPeer entries are created at connect (HandleEvent, line 289) and removed at disconnect (TeardownPeerSlot, line 155 — connectedPeers.Remove(peerIndex)). Lifecycle is correctly tied to the existing ENet peer lifecycle — no parallel lifecycle introduced.

RTT sampling — the 5-second sweep (SamplePeerRtt) runs on the ENet thread as a timestamp check inside the existing event loop, not a separate timer. No additional teardown needed.

TEARDOWN / CONSUMPTION TRACE:

• BucketHistogram fields → owned by MeterListenerMetricsCollector (hosted service) → disposed at shutdown
• ConnectedPeer entries → removed in TeardownPeerSlot on disconnect (line 155)
• RTT sampling timestamp → local variable on ENet thread, no cleanup needed
• All histogram snapshots consumed by both PrometheusFormatter.Write (scrape) and ConsoleDashboard.TryConsumeSnapshot (500ms poll) — no dead infrastructure

STEP 4 — Member audit

Member	Consumers	Verdict
BucketHistogram.Record(long)	MeterListenerMetricsCollector.OnLongMeasurement (7 histogram cases)	Correctly routes per instrument name
BucketHistogram.Snapshot()	MeterListenerMetricsCollector.TakeSnapshot (6 calls + SnapshotPeerRtt)	1:1 with fields
HistogramSnapshot.Percentile(double)	HistogramTracker.ToPercentileStats (3 percentile values)	Correctly computes p50/p95/p99
HistogramSnapshots.Merge(…)	ConsoleDashboard.TryConsumeSnapshot (1 call, merging 7 RTT histograms)	Single consumer — could be inlined, but small and tested (P2)
ContinentResolver.Resolve(string)	ENetHostedService.HandleEvent (peer connect, line 289)	Single point of use, correct placement
ContinentResolver.LoadFromDirectory(…)	Program.cs DI factory (line 70)	Single consumer, correct
PeersManager.RecordTickDuration(long, uint)	RunSimulationTick (line 297) + tests	Internal static for testability — matches RecordDrainCycle pattern
ENetHostedService.RecordDrainCycle(long, int)	FlushOutgoing (line 223) + tests	Internal static for testability
ENetHostedService.RecordPeerRtt(Continent, uint)	SamplePeerRtt (line 246) + tests	Internal static for testability

All members are consumed. No single-use members that should be inlined (except Merge, noted as P2).

STEP 5 — Line-level review

P2 — LoadFromDirectory file-open not wrapped in try/catch (ContinentResolver.cs:96-98)
Missing-file cases degrade gracefully to Empty with a warning, but once File.Exists passes, the new StreamReader(…) and subsequent Load() can throw on permission errors, encoding issues, or a TOCTOU race. This would crash the DI factory at host startup — inconsistent with the class's own stated philosophy ("a corrupt field must degrade the load, not crash startup"). Wrapping lines 96-103 in a try/catch that returns Empty with a warning would make the degradation complete. Low practical risk in Docker (files baked in at build), but the defensive gap is real.

P2 — Debug.Assert in HistogramSnapshots.Merge compiled out in Release (HistogramSnapshot.cs:76-77)
The bounds-mismatch guard (ReferenceEquals(bounds, part.UpperBounds)) is a Debug.Assert, which is a no-op in Release builds. If this method is ever reused with histograms of different bucket layouts, it would silently sum incompatible buckets and produce wrong percentiles. Safe today (all callers share RTT_BUCKETS_MS), but the assert gives false confidence about runtime safety. Consider if (!ReferenceEquals(…)) throw new ArgumentException(…) or continue with a log, if reuse is ever intended.

P2 — Unpinned, unverified external downloads in Dockerfiles (Dockerfile:35-38, Dockerfile.dev-debug:42-45, Dockerfile.debug:10-13, DCLPulse.csproj:57-69)
The geo CSVs are ADD'd from cdn.jsdelivr.net and download.geonames.org with no version pin and no checksum verification. The blast radius is low (the resolver only emits a Continent enum value and skips unparseable rows — no code execution path), but pinning to a specific npm release and/or verifying a SHA-256 after download would harden the supply chain. Not blocking given the data-only nature and graceful degradation.

P2 — HistogramSnapshots.Merge has a single consumer (HistogramSnapshot.cs:54-92)
Only called from ConsoleDashboard.cs:363. The method is small and well-tested, but it's a public static utility for a single use case. Minor abstraction overhead — acceptable as-is.

No P0 or P1 issues found. No bugs, no security vulnerabilities, no missing error handling on hot paths, no resource leaks, no correctness issues with:

• BucketHistogram.Record bucket routing (linear scan, inclusive upper bounds, +Inf overflow) ✓
• HistogramSnapshot.Percentile interpolation (rank calculation, division-by-zero impossible because Counts[i] > 0 when reached, overflow bucket returns last finite bound) ✓
• ContinentResolver.Lookup binary search (correct for inclusive start/end ranges) ✓
• IPv4/IPv6 → UInt128 normalization (V4_MAPPED_PREFIX = 0xFFFF << 32, v4-mapped IPv6 correctly unmapped first) ✓
• RecordDeltaStaleness uint subtraction (wrap-safe unsigned arithmetic, test covers rollover) ✓
• Prometheus exposition (cumulative buckets, correct _bucket/_sum/_count ordering, null/default snapshot tolerance) ✓
• Instrument-name ↔ array-index alignment (7 RTT instruments match Continent enum order, locked by Peer_rtt_instrument_at_each_index_carries_the_matching_region_label test) ✓

STEP 6 — Complexity

COMPLEX. Touches metrics collection, Prometheus export, console dashboard, ENet transport, Docker build configuration, and DI registration — multiple risk subsystems per review protocol.

STEP 7 — QA assessment

QA_REQUIRED: NO. No changes to what a connected client sees or experiences. No wire format changes. No auth/handshake changes. No runtime behavior changes beyond recording and exposing metrics. The 403/403 green test suite covers the new code thoroughly.

STEP 8 — Non-blocking warnings

⚠️ Deployment configuration modified. Three Dockerfiles changed (identical ADD block for geo CSVs). Verify the change works in both local debug and production Docker contexts, and that new build-pipeline files are added to the prod/dev-debug Dockerfiles' selective COPY lines (CLAUDE.md, Build instructions). The PR description confirms all three images build and were verified locally with 550,668 ranges loaded.

STEP 9 — Verdict

REVIEW_RESULT: PASS ✅
COMPLEXITY: COMPLEX
COMPLEXITY_REASON: Touches metrics collection pipeline, Prometheus exposition, console dashboard, ENet transport (peer RTT sampling + continent resolution), Docker build configuration, DI registration, and introduces new measurement types (histogram).
QA_REQUIRED: NO

---

Reviewed by Jarvis 🤖 · Requested by Mikhail Agapov (<@U04A7TYN13L>) via Slack

[mikhail-dcl]

Review P2s addressed in 970b16d:

Finding	Resolution
LoadFromDirectory open/load can crash startup after File.Exists	Wrapped in try/catch (IOException/UnauthorizedAccessException) → warning + Empty resolver; locked by a test using an exclusive file lock
Debug.Assert bounds guard in HistogramSnapshots.Merge compiled out in Release	Now an always-on ArgumentException on bucket-layout mismatch; test added
Unpinned/unverified geo downloads	Deliberate trade-off, now documented in docs/metrics.md: builds must ship current data; contained by data-only files, skip-and-count parsing, and region="unknown" degradation on an unusable dataset
Merge single consumer	No action — acceptable as-is per the review

Full suite 407/407 green.

🤖 Generated with Claude Code