Skip to content

fix(miner): resolve prediction correct/incorrect in metrics CLI#8510

Merged
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
andriypolanski:fix/8315-miner-metrics-outcome-join
Jul 24, 2026
Merged

fix(miner): resolve prediction correct/incorrect in metrics CLI#8510
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
andriypolanski:fix/8315-miner-metrics-outcome-join

Conversation

@andriypolanski

Copy link
Copy Markdown
Contributor

Summary

  • Closes fix(miner): metrics-cli's collectPredictionMetricRows never resolves correct/incorrect despite calibration-cli.ts already having the outcome join #8315
  • Exports buildOutcomeDecisionMap and resolvePredictionCorrectness from calibration.ts (same join rules as buildCalibrationReport; no second implementation).
  • collectPredictionMetricRows now pairs each prediction-ledger row with realized pr_outcome events via toPredictionRecords / toOutcomeRecords and sets correct: true | false | undefined.
  • runMetrics opens the event ledger (read-only, closed in finally) the same way bare calibration does, then passes resolved rows to renderMinerPredictionMetrics.
  • hold and pending predictions leave correct unset; only directional merge/close predictions with a realized merge/close outcome are scored.

Scope

  • The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
  • This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
  • This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
  • I linked a currently open issue this PR resolves (e.g. Closes #123) — a linked open issue is required for every contributor PR.

Validation

  • git diff --check
  • npm run actionlint
  • npm run typecheck
  • npm run build:miner
  • npx vitest run test/unit/miner-metrics-cli.test.ts test/unit/miner-calibration.test.ts — 20 passed
  • CI-scoped Codecov simulation: npm run build:miner && npx vitest run --changed=origin/main --coverage.all=false --coverage100% changed lines/branches on calibration.ts + metrics-cli.ts
  • npm run test:coverage (full unsharded gate)
  • npm run test:ci
  • npm run test:workers
  • npm run build:mcp
  • npm run test:mcp-pack
  • npm run ui:openapi:check
  • npm run ui:lint
  • npm run ui:typecheck
  • npm run ui:build
  • npm audit --audit-level=moderate
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

If any required check was skipped, explain why:

  • Miner-only change (packages/loopover-miner/lib/**). Compiled lib/*.js is gitignored and rebuilt by CI via build:miner. Tests import .ts sources via variable specifiers (same pattern as miner-replay-snapshot.test.ts) so codecov/patch grades the .ts paths under --coverage.all=false.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
  • API/OpenAPI/MCP behavior is updated and tested where needed.
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
  • Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

@codecov

codecov Bot commented Jul 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.61%. Comparing base (f50d010) to head (876e403).
⚠️ Report is 24 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8510      +/-   ##
==========================================
+ Coverage   81.11%   89.61%   +8.50%     
==========================================
  Files         791       99     -692     
  Lines       79348    22777   -56571     
  Branches    23970     3899   -20071     
==========================================
- Hits        64364    20412   -43952     
+ Misses      12171     2187    -9984     
+ Partials     2813      178    -2635     
Flag Coverage Δ
shard-1 36.61% <53.84%> (-21.09%) ⬇️
shard-2 63.38% <46.15%> (?)
shard-3 ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
packages/loopover-miner/lib/calibration.ts 100.00% <100.00%> (ø)
packages/loopover-miner/lib/metrics-cli.ts 100.00% <100.00%> (ø)

... and 692 files with indirect coverage changes

@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 24, 2026
@loopover-orb

loopover-orb Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Tip

✅ LoopOver review result - approve/merge recommended

Review updated: 2026-07-24 17:27:25 UTC

4 files · 1 AI reviewer · no blockers · readiness 98/100 · CI green · clean

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
This PR wires a real outcome join into the miner metrics CLI: it extracts `buildOutcomeDecisionMap`/`resolvePredictionCorrectness` from `calibration.ts` (no duplicate logic), reuses them in `collectPredictionMetricRows`, and has `runMetrics` open/close a second (event) ledger with correct ownership semantics mirroring `withPredictionLedger`. The join, undefined-vs-false `correct` semantics, and ledger lifecycle (only closing ledgers it opened itself) all trace correctly through the diff and the full file content, and are backed by tests exercising confirmed/false/hold/pending/cross-repo cases plus close-ownership. It closes the linked #8315 and is scoped narrowly to that fix.

Nits — 4 non-blocking
  • packages/loopover-miner/lib/metrics-cli.ts:63-70 — if `initEventLedger` throws inside the callback passed to `withPredictionLedger`, the prediction ledger still closes correctly, but consider a comment noting the event ledger is guaranteed unassigned in that case (it's correct as written, just non-obvious on first read).
  • test/unit/miner-metrics-cli.test.ts and miner-calibration.test.ts import the `.ts` source via a variable specifier to dodge a coverage gap on compiled `.js` — worth a one-line pointer to the tracking issue for removing this workaround once the coverage tool is fixed upstream, per the comment's own mention of `miner-replay-snapshot.test.ts` doing the same.
  • Consider tightening `buildOutcomeDecisionMap`'s and `resolvePredictionCorrectness`'s return type alias into a shared named type (e.g. `NormalizedDecision`) instead of repeating the `"merge" | "close" | "hold" | ""` union in both signatures, for readability.
  • Nothing else material — the join, ledger lifecycle, and undefined-vs-false semantics all check out against the full file content.

Decision drivers

  • ✅ Code review — No blockers (1 reviewer)
  • ✅ Gate result — Passing (No configured blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ✅ Linked #8315
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 220 registered-repo PR(s), 135 merged, 22 issue(s).
Contributor context ✅ Confirmed Gittensor contributor andriypolanski; Gittensor profile; 220 PR(s), 22 issue(s).
Improvement ✅ Minor risk: clean · value: minor · LLM: moderate
Linked issue satisfaction

Addressed
The PR reuses calibration.ts's join logic (extracted into buildOutcomeDecisionMap/resolvePredictionCorrectness) inside collectPredictionMetricRows, wires runMetrics to open/close the event ledger read-only, updates the stale header comment, and adds unit tests covering merge/close match, mismatch, hold, pending, cross-repo, and malformed-outcome cases as required.

Review context
  • Author: andriypolanski
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: Python, TypeScript, Rust, Cuda, JavaScript, Kotlin, MDX, Scala
  • Official Gittensor activity: 220 PR(s), 22 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Keep the PR focused and include validation evidence before maintainer review.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@loopover-orb loopover-orb Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LoopOver approves — the gate is satisfied and CI is green.

@loopover-orb
loopover-orb Bot merged commit 048aeb7 into JSONbored:main Jul 24, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(miner): metrics-cli's collectPredictionMetricRows never resolves correct/incorrect despite calibration-cli.ts already having the outcome join

2 participants