feat(viewer): Tokio Stats "Worker activity" rollup + focus deep-link - #670
Conversation
aaabfa5 to
069b739
Compare
Port IRIS's `workers` rollup into the Tokio Stats aggregate page with busyness metric and multi-runtime awareness. Server: `worker_activity: Vec<WorkerStats>` keyed by (host, worker_id) so multi-host scopes don't conflate workers from different runtimes. Each worker tracks busy_ns (sum of ALL poll durations), span_ns (max end_ns − min start_ns = the worker's observed lifetime), and busy_pct (busy_ns / span_ns × 100). Using end_ns for the span upper bound ensures busy_pct ≤ 100%. Old part-files without worker_id silently skipped. The Phase enum boxes TokioStatsAccum to satisfy clippy's large_enum_variant lint. UI: one row per host with aggregate stats (total busyness, polls, share, worst poll), expandable to show individual workers. Column headers are clickable to sort. Worker rows show a verification tooltip on the Busy cell (busy_ns / span_ns = busy_pct). busynessHeat helper colors utilization (≥80% red, ≥50% amber, else green). Tests: demo-trace assertions (shares sum to 100%, busyness > 0, ranked desc, single-host grouping); JS busynessHeat threshold tests.
069b739 to
ea06b3a
Compare
Per rcoh's review on dial9-rs#670, all in the "Worker activity" card: - Host "Busy" was Σ busy_ns / max(span_ns): unbounded and misleading (a 64-worker runtime 1/64-saturated per worker read as "100% saturated"). Aggregate as the mean of workers' per-worker busy_pct instead — bounded, and equal to the average of the per-worker rows shown on expand. Extracted as pure helper hostBusyPct(). - Host label was interpolated into onclick="toggleWorkerHost('...')" and broke on a host containing a quote. Switch to the data-host + this.dataset.host pattern (as openExemplar). Round-trip reproduces the raw host, so the expand/collapse toggle still keys correctly. - Remove dead worstWorker (and now-unused busyNs / maxSpanNs) from the per-host aggregate. Add hostBusyPct unit tests (incl. the 64-worker case and the <=100% bound) to test_tokio_stats_api.js.
|
this works decently well. It would be nice to follow up by using the segment metadata about workers to resolve the total worker count. Right now, its using the number of workers that are observed but that is an undercount of how many workers are actually available |
|
What does the |
Not sure why its not letting me reply normally, but Share and Notable are both for polls; poll share and notable polls (>1ms) |
Hmm, didn't think about this. I can look into this and add a follow up pr for this, unless you'd want to include it in this one |
|
The busy percentage isn't really accurate because it reports a node that happens to only be running fewer works as more busy...its actually basically backwards and very misleading |
|
ideally we would show something like Workers (Active / Total) I still don't know what "Poll share" means |
Oh thats very weird. I didn't notice this, but I was also testing on a smaller/more basic application. Looking into it now |
…e/total workers Follow-up review on the "Worker activity" rollup: - Busyness was backwards: a host sampled more sparsely read as busier than one doing more work. Each worker's busy_pct used max(end) − min(start) across ALL segments as the denominator, which counts the idle gaps between sampled segments. Fix: denominator is now observed active time — the sum of per-segment windows (one polls part-file = one segment) — so it stays on the same time base as busy_ns and is bounded ≤100% (a worker's polls are sequential). Host busyness is the pooled ratio Σ busy_ns / Σ span_ns, weighting workers by observed time rather than a mean-of-ratios (which had the same sparse-window bias). - Remove the "Share" column (poll_share_pct + workerShareHeat): the metric was unclear to reviewers. - Add a "Workers (active / total)" column: active = observed workers, total = max worker_id + 1 (Tokio numbers workers 0..N-1), surfacing idle-but-available capacity the observed count alone hides. Not a trace-format change (only API aggregation of existing columns). Tests: rewrote hostBusyPct tests for the pooled ratio, added hostWorkerCounts tests, added busy_pct ≤ 100% invariant to the demo test. cargo test --lib + node ui tests green; fmt + clippy clean.
|
nice! works pretty well. I think we can continue to enhance it as well |
2 participants
Added
workersrollup (fallback path: poll counts + share%) into the Tokio Stats aggregate page. Shows per-worker poll distribution to surface work imbalance across the runtime's thread pool.Server:
worker_activity: Vec<WorkerStats>on TokioStatsResponse, accumulated viaby_worker: HashMap<u32, WorkerAccum>— counts ALL polls (before the 100µs floor) for accurate share%, then tracks notable polls + worst exemplar for above-floor rows. Old part-files without worker_id column silently skipped (same guard as long-polls).UI:
renderWorkerActivitycard with worker/total/share/bar/notable/ worst-poll columns; rows deep-link to the worst poll's trace segment.workerShareHeat(sharePct, numWorkers)helper colors imbalance (>2× ideal red, >1.5× amber, else green). Each worker can be clicked upon as well, taking the user to the instance of the longest poll on that worker.