Context
src/queue/job-dispatch.ts, processJob's "backfill-registered-repos" case (lines ~97-129). When the cron-scheduled sweep enqueues this job with no repoFullName (the periodic full-fleet path), it fetches every installed repo and fans out one per-repo backfill-registered-repos job:
await Promise.all(
repositories.map((repo, index) => {
const repoMessage: JobMessage = { ... };
const delaySeconds = Math.min(index * delayStepSeconds, 900);
return delaySeconds > 0
? env.JOBS.send(repoMessage, { delaySeconds })
: env.JOBS.send(repoMessage);
}),
);
Promise.all rejects as soon as any single env.JOBS.send call rejects (a transient queue-send failure), which throws out of the whole processJob invocation for this job. The queue's own retry mechanism then re-invokes the entire handler from scratch, re-running the full repositories.map fan-out again — including for every repo whose env.JOBS.send already succeeded before the failure. Since each successfully-sent per-repo job independently triggers backfillRegisteredRepositories/enqueueRepositoryOpenDataBackfill downstream, a single mid-loop send failure can cause every already-dispatched repo to receive a second, duplicate per-repo backfill job on retry — extra GitHub API load and duplicate work across the whole fleet, from one transient failure partway through the fan-out.
Requirements
- Change the
Promise.all(repositories.map(...)) fan-out so that one repo's env.JOBS.send failure does not cause already-successfully-dispatched repos to be redispatched on retry. Use Promise.allSettled (or an equivalent per-repo try/catch) so every repo's send is attempted independently regardless of an earlier one's outcome.
- Log (via
console.error, matching this file's existing JSON-structured logging convention — see the "sync-brokered-installed-repos" case a few lines above, ~line 93) any individual repo's send failure, including the repoFullName, so an operator can see exactly which repos didn't get dispatched this cycle.
- After the settle, the job should still surface a genuine failure signal (e.g. throw if any repo's send failed, so the cron invocation itself is marked failed for observability) — but that throw must happen after every repo's send has been attempted, never abort the loop partway through.
- Either
Promise.allSettled or the already-imported mapWithConcurrency (src/queue/map-with-concurrency.ts) is acceptable — the actual requirement is per-repo isolation (a failed send never blocks or duplicates a sibling repo's send), not the specific primitive used.
Deliverables
Test Coverage Requirements
99%+ Codecov patch coverage, branch-counted, on every changed line/branch in src/queue/job-dispatch.ts, per codecov.yml, including the new per-repo failure-logging branch.
Expected Outcome
A single transient env.JOBS.send failure partway through the fleet-wide backfill fan-out no longer risks duplicate backfill dispatch across every already-succeeded repo in the fleet on retry, and per-repo send failures are individually observable instead of silently aborting the remainder of the fan-out.
Links & Resources
Context
src/queue/job-dispatch.ts,processJob's"backfill-registered-repos"case (lines ~97-129). When the cron-scheduled sweep enqueues this job with norepoFullName(the periodic full-fleet path), it fetches every installed repo and fans out one per-repobackfill-registered-reposjob:Promise.allrejects as soon as any singleenv.JOBS.sendcall rejects (a transient queue-send failure), which throws out of the wholeprocessJobinvocation for this job. The queue's own retry mechanism then re-invokes the entire handler from scratch, re-running the fullrepositories.mapfan-out again — including for every repo whoseenv.JOBS.sendalready succeeded before the failure. Since each successfully-sent per-repo job independently triggersbackfillRegisteredRepositories/enqueueRepositoryOpenDataBackfilldownstream, a single mid-loop send failure can cause every already-dispatched repo to receive a second, duplicate per-repo backfill job on retry — extra GitHub API load and duplicate work across the whole fleet, from one transient failure partway through the fan-out.Requirements
Promise.all(repositories.map(...))fan-out so that one repo'senv.JOBS.sendfailure does not cause already-successfully-dispatched repos to be redispatched on retry. UsePromise.allSettled(or an equivalent per-repo try/catch) so every repo's send is attempted independently regardless of an earlier one's outcome.console.error, matching this file's existing JSON-structured logging convention — see the"sync-brokered-installed-repos"case a few lines above, ~line 93) any individual repo's send failure, including therepoFullName, so an operator can see exactly which repos didn't get dispatched this cycle.Promise.allSettledor the already-importedmapWithConcurrency(src/queue/map-with-concurrency.ts) is acceptable — the actual requirement is per-repo isolation (a failed send never blocks or duplicates a sibling repo's send), not the specific primitive used.Deliverables
backfill-registered-reposfan-out insrc/queue/job-dispatch.tsno longer uses a barePromise.allwhere one repo's rejection can cause retry-driven duplicate dispatch for other repos.test/unit/job-dispatch.test.ts: given a mockedenv.JOBS.sendthat fails for one specific repo out of several, assert every OTHER repo's send is still attempted exactly once, and document/assert the exact retry-dedup behavior chosen.Test Coverage Requirements
99%+ Codecov patch coverage, branch-counted, on every changed line/branch in
src/queue/job-dispatch.ts, percodecov.yml, including the new per-repo failure-logging branch.Expected Outcome
A single transient
env.JOBS.sendfailure partway through the fleet-wide backfill fan-out no longer risks duplicate backfill dispatch across every already-succeeded repo in the fleet on retry, and per-repo send failures are individually observable instead of silently aborting the remainder of the fan-out.Links & Resources
src/queue/job-dispatch.ts(processJob,"backfill-registered-repos"case)src/queue/map-with-concurrency.ts