feat: implement rubygems registry [CM-1241]#4322
Merged
Merged
Conversation
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
|
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a new RubyGems registry ingestion capability to the packages_worker service, following the established multi-entry-point worker pattern (most closely mirroring the existing NuGet worker). It introduces an API client for rubygems.org, normalization utilities, two Temporal batch loops (a "core" metadata/downloads pass and a "critical" versions/maintainers pass), their activities/workflows/schedules, a dev/prod Docker Compose file, and supporting data-access-layer helpers.
Changes:
- New
rubygems/worker module:client.ts,normalize.ts,types.ts,runRubyGemsCoreLoop.ts,runRubyGemsCriticalLoop.ts,activities.ts,workflows.ts,schedule.ts, andbin/rubygems-worker.ts, wired intoactivities.ts,workflows/index.ts, andgetRubyGemsConfig. - New DAL helpers:
osspckgs/rubygems.ts(package selection queries) and a genericosspckgs/downloads.ts(recordDownloadSnapshot), plus new root exports. - Ops wiring:
scripts/services/rubygems-worker.yamlandpackage.jsonstart/dev scripts (inspect port9244).
Reviewed changes
Copilot reviewed 16 out of 17 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
services/libs/data-access-layer/src/osspckgs/rubygems.ts |
Keyset/priority selection queries for core and critical RubyGems packages. |
services/libs/data-access-layer/src/osspckgs/downloads.ts |
Generic download-snapshot recorder (duplicate of the NuGet variant). |
services/libs/data-access-layer/src/index.ts |
Exports new osspckgs/types, downloads, rubygems modules. |
services/apps/packages_worker/src/workflows/index.ts |
Exports RubyGems workflows. |
services/apps/packages_worker/src/rubygems/workflows.ts |
Core/critical continue-as-new workflows. |
services/apps/packages_worker/src/rubygems/types.ts |
Fetch/normalized RubyGems type definitions. |
services/apps/packages_worker/src/rubygems/schedule.ts |
Daily cron schedules for both workflows. |
services/apps/packages_worker/src/rubygems/runRubyGemsCriticalLoop.ts |
Versions/maintainers batch loop (non-draining on skip/error). |
services/apps/packages_worker/src/rubygems/runRubyGemsCoreLoop.ts |
Metadata/downloads batch loop with error marking. |
services/apps/packages_worker/src/rubygems/normalize.ts |
Pure normalization helpers (untested). |
services/apps/packages_worker/src/rubygems/client.ts |
Axios client with 404/429 error classification. |
services/apps/packages_worker/src/rubygems/activities.ts |
Activity wrappers over the two loops. |
services/apps/packages_worker/src/config.ts |
Adds getRubyGemsConfig (used only by the core loop). |
services/apps/packages_worker/src/bin/rubygems-worker.ts |
Worker entrypoint registering both schedules. |
services/apps/packages_worker/src/activities.ts |
Registers the new RubyGems activities. |
services/apps/packages_worker/package.json |
start/dev scripts for the RubyGems worker. |
scripts/services/rubygems-worker.yaml |
Compose service (omits CROWD_TEMPORAL_NAMESPACE). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
…l loop Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Comment on lines
+59
to
+74
| if (isRubyGemsFetchError(versionsResult)) { | ||
| if (versionsResult.kind === 'NOT_FOUND') { | ||
| log.warn({ purl: pkg.purl }, 'Package not found on RubyGems versions endpoint — skipping') | ||
| return 'skipped' | ||
| } | ||
| if (versionsResult.kind === 'RATE_LIMIT') { | ||
| log.warn({ purl: pkg.purl }, 'Rate limited fetching RubyGems versions — will retry next pass') | ||
| return 'error' | ||
| } | ||
| throw new Error(`Transient error fetching versions for ${pkg.purl}: ${versionsResult.message}`) | ||
| } | ||
|
|
||
| const normalizedVersions = normalizeRubyGemsVersions(versionsResult) | ||
| const latest = pickLatestRubyGemsVersion(normalizedVersions) | ||
| const ownersFetchFailed = isRubyGemsFetchError(ownersResult) | ||
| const owners = ownersFetchFailed ? [] : normalizeRubyGemsOwners(ownersResult) |
Comment on lines
+5
to
+13
| export async function recordDownloadSnapshot( | ||
| qx: QueryExecutor, | ||
| params: { | ||
| packageId: number | ||
| purl: string | ||
| totalDownloads: number | ||
| today: string // YYYY-MM-DD | ||
| }, | ||
| ): Promise<string[]> { |
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
| qx.tx(async (t) => { | ||
| const changed = new Set<string>() | ||
|
|
||
| const { id: packageDbId, changedFields: pkgChanged } = await upsertPackage(t, { |
|
|
||
| export async function ingestRubyGemsCriticalDetails(): Promise<void> { | ||
| const result = await acts.processRubyGemsCriticalBatch() | ||
| if (result.processed + result.skipped + result.error + result.unchanged === 0) { |
| pkgChanged.forEach((f) => changed.add(f)) | ||
|
|
||
| if (normalizedVersions.length > 0) { | ||
| const versionRows: IDbVersionUpsert[] = normalizedVersions.map((v) => ({ |
Comment on lines
+107
to
+108
| batchSize: parseInt(process.env.RUBYGEMS_DEPENDENTS_BATCH_SIZE ?? '50000', 10), | ||
| concurrency: parseInt(process.env.RUBYGEMS_DEPENDENTS_CONCURRENCY ?? '50', 10), |
| ): Promise<void> { | ||
| await qx.result( | ||
| `UPDATE packages | ||
| SET dependent_count = $(dependentCount) |
| import { upsertLast30dDownload } from '../packages/downloadsLast30d' | ||
| import { QueryExecutor } from '../queryExecutor' | ||
|
|
||
| export async function recordDownloadSnapshot( |
| SERVICE: rubygems-worker | ||
| SHELL: /bin/sh | ||
| SUPPRESS_NO_CONFIG_WARNING: 'true' | ||
| CROWD_TEMPORAL_TASKQUEUE: rubygems-worker |
Comment on lines
+46
to
+48
| export function pickLatestRubyGemsVersion( | ||
| versions: NormalizedRubyGemsVersion[], | ||
| ): NormalizedRubyGemsVersion | null { |
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
| import { QueryExecutor } from '../queryExecutor' | ||
|
|
||
| export type RubyGemsPackageToSync = { | ||
| id: number |
Comment on lines
+88
to
+91
| if (gemResult.kind === 'RATE_LIMIT') { | ||
| log.warn({ purl: pkg.purl }, 'Rate limited by RubyGems registry — will retry next pass') | ||
| return 'error' | ||
| } |
Comment on lines
+63
to
+66
| for (let batchStart = 0; batchStart < packages.length; batchStart += config.concurrency) { | ||
| const group = packages.slice(batchStart, batchStart + config.concurrency) | ||
|
|
||
| await Promise.all( |
Comment on lines
+88
to
+91
| if (gemResult.kind === 'RATE_LIMIT') { | ||
| log.warn({ purl: pkg.purl }, 'Rate limited by RubyGems registry — will retry next pass') | ||
| return 'error' | ||
| } |
| AND ( | ||
| p.ingestion_source IS NULL | ||
| OR p.ingestion_source <> ALL($(workerOutcomes)::text[]) | ||
| OR p.last_synced_at < date_trunc('day', NOW()) |
| SERVICE: rubygems-worker | ||
| SHELL: /bin/sh | ||
| SUPPRESS_NO_CONFIG_WARNING: 'true' | ||
| CROWD_TEMPORAL_TASKQUEUE: rubygems-worker |
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Comment on lines
+5
to
+7
| const acts = proxyActivities<typeof activities>({ | ||
| startToCloseTimeout: '30 minutes', | ||
| retry: { initialInterval: '30 seconds', backoffCoefficient: 2, maximumAttempts: 5 }, |
Comment on lines
+3
to
+5
| export type RubyGemsPackageToSync = { | ||
| id: number | ||
| purl: string |
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f87aa20. Configure here.
| ): NormalizedRubyGemsVersion | null { | ||
| if (versions.length === 0) return null | ||
| const stable = versions.find((v) => !v.isPrerelease) | ||
| return stable ?? versions[0] |
There was a problem hiding this comment.
Critical latest version mismatch
High Severity
pickLatestRubyGemsVersion picks the first non-prerelease entry in the versions list, while the core ingest path sets latestVersion from the gem JSON version field. When the registry’s current release is a prerelease, critical sync can persist an older stable as latest_version and mark the wrong row is_latest, diverging from core ingest and from RubyGems.org.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit f87aa20. Configure here.
| } | ||
|
|
||
| counts.lastId = packages[packages.length - 1].id | ||
| return counts |
There was a problem hiding this comment.
Pagination skips failed packages
Medium Severity
Dependents and critical workflows always set lastId to the final row in the batch, including packages that returned error without updating the database. continueAsNew then resumes after that id, so those failures are not retried until the next scheduled run (weekly for dependents).
Additional Locations (1)
Reviewed by Cursor Bugbot for commit f87aa20. Configure here.
Comment on lines
+50
to
+53
| export function fetchOwners(name: string): Promise<RubyGemsFetchResult<RubyGemsOwner[]>> { | ||
| return rubyGemsGet<RubyGemsOwner[]>( | ||
| `https://rubygems.org/api/v1/gems/${encodeURIComponent(name)}/owners.json`, | ||
| ) |
| SERVICE: rubygems-worker | ||
| SHELL: /bin/sh | ||
| SUPPRESS_NO_CONFIG_WARNING: 'true' | ||
| CROWD_TEMPORAL_TASKQUEUE: rubygems-worker |
| import { upsertLast30dDownload } from '../packages/downloadsLast30d' | ||
| import { QueryExecutor } from '../queryExecutor' | ||
|
|
||
| export async function recordDownloadSnapshot( |
| number: v.number, | ||
| isLatest: v.number === latest?.number, | ||
| isPrerelease: v.isPrerelease, | ||
| license: v.licenses && v.licenses.length > 0 ? v.licenses[0] : null, |
Comment on lines
+35
to
+38
| export function normalizeRubyGemsVersions( | ||
| items: RubyGemsVersionItem[], | ||
| ): NormalizedRubyGemsVersion[] { | ||
| return items.map((item) => ({ |
Comment on lines
+1
to
+2
| const MAX_RPS = Math.max(1, parseInt(process.env.RUBYGEMS_MAX_RPS ?? '10', 10)) | ||
| const INTERVAL_MS = 1000 / MAX_RPS |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new RubyGems worker to the
packages_workerservice, adding support for ingesting and processing RubyGems package data. The main changes include Docker Compose and script updates for the new worker, as well as the implementation of RubyGems batch processing logic for both core and critical packages. The code adds API clients, data normalization, and database integration for RubyGems packages, versions, and maintainers.RubyGems Worker Integration:
rubygems-worker.yaml) to run and develop the new RubyGems worker service with appropriate environment variables, volumes, and network settings.package.jsonscripts to support starting and developing the RubyGems worker locally and in Docker.RubyGems Batch Processing Implementation:
processRubyGemsCoreBatchandprocessRubyGemsCriticalBatchactivities, which handle batch processing for core and critical RubyGems packages, respectively. [1] [2]runRubyGemsCoreLoop.tsandrunRubyGemsCriticalLoop.tsfor fetching, normalizing, and upserting package, version, and maintainer data into the database, including error handling and retry logic. [1] [2]RubyGems Registry Integration:
Configuration:
getRubyGemsConfigto manage batch size, concurrency, and delay settings for RubyGems batch processing, with environment variable support.Note
Medium Risk
Touches shared packages DB write paths (versions upsert, downloads, maintainers) and runs high-concurrency external API ingestion; misconfiguration could stress rubygems.org or write incorrect package metadata at scale.
Overview
Adds a dedicated
rubygems-workerTemporal service (Docker Compose, package scripts, build list) that registers three scheduled ingestion pipelines on therubygems-workertask queue.Core sync (daily) pulls gem metadata from rubygems.org, upserts package fields and download snapshots (daily deltas + rolling last-30d), with rate limiting, 404/
rubygems_not_foundhandling, and a fail-the-batch backoff when every row is rate-limited.Critical sync (daily) enriches
is_criticalgems with full version history (includingpublished_at), owners/maintainers, and audit logging.Dependents sync (weekly) walks all rubygems packages by id and stores reverse-dependency counts used to prioritize core sync ordering.
Shared data-access additions include RubyGems-specific list/update queries, a reusable
recordDownloadSnapshothelper, and extendingupsertVersionsBatch/IDbVersionUpsertto persist version publish times.Reviewed by Cursor Bugbot for commit f87aa20. Bugbot is set up for automated code reviews on this repo. Configure here.