fix(queue): isolate per-repo failures in backfill-registered-repos fan-out (#8355)

[philluiz2323]

Summary

• processJob's "backfill-registered-repos" case fans out one per-repo job via a bare Promise.all when the cron-scheduled full-fleet sweep runs with no repoFullName. Promise.all rejects as soon as any single env.JOBS.send call rejects, throwing out of the whole processJob invocation.
• The queue's own retry mechanism then re-invokes the entire handler from scratch, re-running the full repositories.map fan-out again — including for every repo whose env.JOBS.send had already succeeded, causing duplicate per-repo backfill dispatch across the whole fleet from a single transient send failure.
• Switches to Promise.allSettled so every repo's send is attempted exactly once regardless of an earlier one's outcome. Each individual failure is logged (repoFullName + reason) matching this file's existing structured-logging convention (the "sync-brokered-installed-repos" case a few lines above). A genuine failure still surfaces via a thrown error after the settle, so the cron invocation itself is marked failed for observability — but only once every repo's send has been attempted.

Closes #8355

Scope

• The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
• This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
• This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
• I linked an issue, or this is small enough that the summary explains why an issue is not needed.

Validation

• git diff --check
• npm run actionlint
• npm run typecheck
• npm run test:coverage locally; codecov/patch requires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.
• npm run test:workers
• npm run build:mcp
• npm run test:mcp-pack
• npm run ui:openapi:check
• npm run ui:lint
• npm run ui:typecheck
• npm run ui:build
• npm audit --audit-level=moderate
• New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

If any required check was skipped, explain why:

• This change touches only src/queue/job-dispatch.ts (pure backend logic, no UI/MCP/workers/OpenAPI surface) plus its test file. Verified via npx vitest run test/unit/job-dispatch.test.ts: all 4 tests pass, including the 2 new ones — one asserting every OTHER repo's send is still attempted when one repo's send rejects (and that the job throws with the exact failed repo named, after every send was attempted), and one asserting no throw/log when every send succeeds. Coverage on the changed lines (verified directly against coverage/lcov.info): the rejection-detection branch inside the settled.forEach shows both arms hit (BRDA:135,10,0,1 / BRDA:135,10,1,4), and the final failedRepoFullNames.length > 0 throw-condition shows both arms hit (BRDA:141,11,0,1 / BRDA:141,11,1,1).

Safety

• No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
• Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
• Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. (N/A — no auth/session/CORS changes.)
• API/OpenAPI/MCP behavior is updated and tested where needed. (N/A — no API/OpenAPI/MCP surface changed.)
• UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks. (N/A — no UI changes.)
• Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots. (N/A — no visible UI changes.)
• Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs. (N/A.)

Notes

• Used Promise.allSettled directly at the call site rather than mapWithConcurrency (also acceptable per the issue): mapWithConcurrency's own worker pool is itself built on Promise.all internally, so it would not provide per-item isolation without the mapper also catching its own errors — Promise.allSettled gives that isolation directly with no extra wrapping.

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.65%. Comparing base (7f5550b) to head (42c03d4).
⚠️ Report is 17 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8473      +/-   ##
==========================================
+ Coverage   80.86%   89.65%   +8.78%     
==========================================
  Files         791       98     -693     
  Lines       79320    22864   -56456     
  Branches    23954     3918   -20036     
==========================================
- Hits        64142    20498   -43644     
+ Misses      12156     2187    -9969     
+ Partials     3022      179    -2843     

Flag	Coverage Δ
shard-1	79.11% <33.33%> (+20.74%)	⬆️
shard-2	15.82% <0.00%> (-35.29%)	⬇️
shard-3	23.41% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/queue/job-dispatch.ts	99.36% <100.00%> (+44.03%)	⬆️

... and 693 files with indirect coverage changes

[loopover-orb]

Tip

✅ LoopOver review result - approve/merge recommended

_{Review updated: 2026-07-24 14:59:41 UTC}

2 files · 1 AI reviewer · no blockers · readiness 93/100 · CI green · clean

✅ Suggested Action - Approve/Merge

• safe to merge

Review summary
Replaces a bare Promise.all with Promise.allSettled in the backfill-registered-repos cron fan-out so one repo's env.JOBS.send rejection no longer aborts sibling sends or causes the queue's retry to re-dispatch the entire fleet; individual failures are logged with repoFullName/reason, and a summary error is still thrown after every send has been attempted so the invocation is marked failed for observability. This mirrors the identical fix already applied to fanOutBacklogConvergenceSweepJobs (src/queue/processors.ts, #audit-sweep-fanout-isolation) and is well-tested: both the partial-failure and all-success paths are covered with real env.JOBS.send mocks and real DB rows via upsertInstallation/upsertRepositoryFromGitHub rather than fabricated payloads.

Nits — 5 non-blocking

• The final `throw new Error(...)` after collecting all failures means the queue's retry mechanism will still re-run the whole `repositories.map` fan-out on that thrown error, re-sending to every repo including the ones that already succeeded on the first pass — the PR description implies duplication is fully solved, but it's only solved for the failures *within* a single invocation, not across a retry of the summary throw itself; worth clarifying in the description or wrapping the summary in a non-retryable signal if the queue distinguishes those.
• The inline `console.error(JSON.stringify({...}))` call on job-dispatch.ts:138 is a long single line mixing the failure-loop logic with logging; consider extracting a small helper to match the more structured style used elsewhere in the file (e.g. `errorMessage(error)` usage in processors.ts) for consistency.
• The large explanatory comment block above the `Promise.allSettled` call is quite verbose relative to the size of the code change; a shorter pointer to the referenced issue/PR would suffice given the file's existing convention of terser comments elsewhere.
• Consider whether the summary throw should distinguish 'some repos failed' from 'all repos failed' so operators/alerts can tell partial vs total outage severity apart at a glance.
• If the queue's retry-on-throw behavior re-runs the entire handler (as implied by the PR's own problem description), consider dispatching a separate non-repo-scoped observability signal (e.g. `incr()` counter, matching the `sync-brokered-installed-repos` case's `incr("loopover_orb_installed_repos_sync_failures_total")` pattern) instead of relying solely on the throw for visibility, to avoid duplicate dispatch to the already-succeeded repos on retry.

Decision drivers

• ✅ Code review — No blockers (1 reviewer)
• ✅ Gate result — Passing (No configured blocker found.)

Context & advisory signals — never blocks the verdict

Signal	Result	Evidence
Linked issue	✅ Linked	#8355
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 1040 registered-repo PR(s), 615 merged, 126 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	philluiz2323; Gittensor profile; 1040 PR(s), 126 issue(s).
Improvement	✅ Minor	risk: clean · value: minor · LLM: moderate

Linked issue satisfaction

Addressed
The diff replaces the bare Promise.all with Promise.allSettled, logs each individual repo send failure via console.error with repoFullName and reason in the file's JSON-structured logging style, and throws only after every send has been attempted, matching all stated requirements. New tests verify that a mid-fan-out failure for one repo does not block sibling sends and that the failure/no-failure

Review context

• Author: philluiz2323
• Role context: outside_contributor
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: not available
• Official Gittensor activity: 1040 PR(s), 126 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Start here: Triage stale or unlinked PRs.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
• @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

• Re-run LoopOver review

[loopover-orb]

LoopOver approves — the gate is satisfied and CI is green.