Skip to content

fix(queue): backfill-registered-repos cron fan-out has no per-repo isolation, risking duplicate dispatch on partial failure #8355

Description

@JSONbored

Context

src/queue/job-dispatch.ts, processJob's "backfill-registered-repos" case (lines ~97-129). When the cron-scheduled sweep enqueues this job with no repoFullName (the periodic full-fleet path), it fetches every installed repo and fans out one per-repo backfill-registered-repos job:

await Promise.all(
  repositories.map((repo, index) => {
    const repoMessage: JobMessage = { ... };
    const delaySeconds = Math.min(index * delayStepSeconds, 900);
    return delaySeconds > 0
      ? env.JOBS.send(repoMessage, { delaySeconds })
      : env.JOBS.send(repoMessage);
  }),
);

Promise.all rejects as soon as any single env.JOBS.send call rejects (a transient queue-send failure), which throws out of the whole processJob invocation for this job. The queue's own retry mechanism then re-invokes the entire handler from scratch, re-running the full repositories.map fan-out again — including for every repo whose env.JOBS.send already succeeded before the failure. Since each successfully-sent per-repo job independently triggers backfillRegisteredRepositories/enqueueRepositoryOpenDataBackfill downstream, a single mid-loop send failure can cause every already-dispatched repo to receive a second, duplicate per-repo backfill job on retry — extra GitHub API load and duplicate work across the whole fleet, from one transient failure partway through the fan-out.

Requirements

  • Change the Promise.all(repositories.map(...)) fan-out so that one repo's env.JOBS.send failure does not cause already-successfully-dispatched repos to be redispatched on retry. Use Promise.allSettled (or an equivalent per-repo try/catch) so every repo's send is attempted independently regardless of an earlier one's outcome.
  • Log (via console.error, matching this file's existing JSON-structured logging convention — see the "sync-brokered-installed-repos" case a few lines above, ~line 93) any individual repo's send failure, including the repoFullName, so an operator can see exactly which repos didn't get dispatched this cycle.
  • After the settle, the job should still surface a genuine failure signal (e.g. throw if any repo's send failed, so the cron invocation itself is marked failed for observability) — but that throw must happen after every repo's send has been attempted, never abort the loop partway through.
  • Either Promise.allSettled or the already-imported mapWithConcurrency (src/queue/map-with-concurrency.ts) is acceptable — the actual requirement is per-repo isolation (a failed send never blocks or duplicates a sibling repo's send), not the specific primitive used.

Deliverables

  • The backfill-registered-repos fan-out in src/queue/job-dispatch.ts no longer uses a bare Promise.all where one repo's rejection can cause retry-driven duplicate dispatch for other repos.
  • A per-repo send failure is logged with enough detail (repoFullName, error) to diagnose which repos were skipped this cycle.
  • New test(s) in test/unit/job-dispatch.test.ts: given a mocked env.JOBS.send that fails for one specific repo out of several, assert every OTHER repo's send is still attempted exactly once, and document/assert the exact retry-dedup behavior chosen.

Test Coverage Requirements

99%+ Codecov patch coverage, branch-counted, on every changed line/branch in src/queue/job-dispatch.ts, per codecov.yml, including the new per-repo failure-logging branch.

Expected Outcome

A single transient env.JOBS.send failure partway through the fleet-wide backfill fan-out no longer risks duplicate backfill dispatch across every already-succeeded repo in the fleet on retry, and per-repo send failures are individually observable instead of silently aborting the remainder of the fan-out.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions