feat(traces): add bulk trace seeding script with configurable options

[adambenhassen]

Description:

Add a script for seeding massive trace datasets into ClickHouse with configurable parameters and chunked processing to avoid memory pressure issues.

Features:

• Readable count format (1k, 500k, 1m)
• Configurable time range (--days flag, defaults to 1 day)
• Custom ClickHouse connection string support
• Chunked duplication (5k batches) to prevent memory exhaustion
• Uses OTEL load-test trace samples with realistic timings (9-300ms jitter)
• Server-side duplication via ClickHouse queries for efficiency

Performance:

• 60k traces: ~5 seconds
• 500k traces: ~29 seconds
• 1M traces: ~53 seconds

Usage:

• pnpm seed:traces [--count=N] [--days=N] [--clickhouse=URL]

Examples:

• pnpm seed:traces the-guild/test/prod --count=1k
• pnpm seed:traces the-guild/test/prod --count=500k --days=30
• pnpm seed:traces the-guild/test/prod --count=1m --clickhouse=http://user:pass@host:8123

Execution Example:

> tsx scripts/seed-traces/seed-traces-bulk.ts the-guild/test/prod --count=100k --days=7

  ClickHouse:         http://localhost:8123
  Target slug:        the-guild/test/prod
  Requested traces:   100,000

  Estimated output:
  - Traces:           ~100,002
  - Spans:            ~1,000,020
  - Time range:       Last 7 days

Loading trace samples...
Loaded 6 trace samples
Generating traces from all samples...
Generated 56 spans across 6 unique traces
Inserting unique traces into ClickHouse...
Inserted 56/56 spans (100%)
- Unique traces inserted successfully!

Duplicating to reach 100,002 total traces:
Processing chunk 1/4: duplicating 5,000 times...
Processing chunk 2/4: duplicating 5,000 times...
Processing chunk 3/4: duplicating 5,000 times...
Processing chunk 4/4: duplicating 1,666 times...

- Successfully created 100,002 traces (933,352 spans)
- Data distributed across last 7 days

- Trace seeding completed!

[gemini-code-assist]

Summary of Changes

Hello @adambenhassen, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a powerful new tool for developers and testers to generate and inject large-scale OpenTelemetry trace data into ClickHouse, facilitating robust testing and performance analysis of tracing functionalities. Concurrently, it implements a global feature flag to easily enable or disable OTEL tracing across the entire application environment, streamlining development workflows and feature rollout.

Highlights

• Bulk Trace Seeding Script: A new script has been added to efficiently seed large volumes of OpenTelemetry (OTEL) traces into ClickHouse. This script supports configurable options for trace count, time range, and ClickHouse connection details.
• Configurable Seeding Options: The trace seeding script allows specifying the total number of traces (e.g., 1k, 500k, 1m), the time range in days to scatter traces across, and a custom ClickHouse connection string. It uses OTEL load-test trace samples with realistic timings.
• Efficient Server-Side Duplication: To handle massive datasets and prevent memory exhaustion, the script performs chunked duplication (5k batches) and leverages server-side duplication via ClickHouse queries, significantly improving performance for large trace counts.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[github-actions]

📚 Storybook Deployment

The latest changes are available as preview in: https://pr-7319.hive-storybook.pages.dev

[gemini-code-assist]

Code Review

This pull request introduces a powerful trace seeding script and wires up the necessary feature flags to enable OTEL tracing. The script is well-designed, using server-side duplication in ClickHouse for efficiency. I've found a critical issue in the script that would prevent traces from being correctly associated with their target, and a couple of medium-severity issues related to argument parsing and random ID generation. The rest of the changes for enabling the feature flag look solid.

[github-actions]

💻 Website Preview

The latest changes are available as preview in: https://pr-7319.hive-landing-page.pages.dev

[github-actions]

🐋 This PR was built and pushed to the following Docker images:

Targets: build

Platforms: linux/amd64

Image Tag: 1ab0492234b39b30185318065af3c40a8b2ca526

[n1ru4l]

nice 😻