Create FTS test datasets generator - #534
Draft
m-szymon wants to merge 2 commits into
Draft
Conversation
Add a Latte-based performance testing suite for ScyllaDB Full-Text Search (BM25), covering the full lifecycle from data loading through index build to search with IR accuracy metrics. Workload (fts.rn): - Schema phase: idempotent keyspace/table creation with optional index drop/creation - Load phase: bulk document ingestion from TSV files, with shard support for large datasets - Build index phase: drops and recreates the FTS index, then probes with BM25 queries until serving; reports index_ready_seconds and indexing_throughput_docs_per_sec - Search phase: runs queries with configurable concurrency and computes recall@k, precision@k, MRR, and nDCG@k against qrels Metrics (metrics.rn): - recall@k, precision@k, reciprocal rank, nDCG@k using graded relevance and precomputed log2 discount factors Orchestrator (run_benchmark.py): - Drives all phases via latte CLI calls with external wall-clock timing for indexing throughput - Parses latte JSON reports for latency histograms and custom metrics - All phases individually skippable (--skip-schema, --skip-load, --skip-build-index, --skip-search) - Automatic cleanup (index drop) after search; --no-cleanup to skip - Sharded dataset loading for large corpora Includes testdata/ smoke-test fixture (10 docs, 8 queries, qrels) and README with phase-by-phase usage instructions.
Add standalone FTS dataset preparation tooling Introduce prepare_dataset.py to build benchmark datasets from BEIR, custom TSV input, or previously prepared datasets. Support replicate and synthetic scaling, synthetic query generation, sharding, and manifest/statistics output. Also document the dataset format and preparation workflow in the full-text-search docs, and update the benchmark README to point to the new preparation guide. Since we are in read-only plan mode, the next step is to inspect the affected file and surrounding context so I can turn those findings into a concrete fix plan with minimal-risk changes.
Collaborator
|
@swasik please take a look |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes https://scylladb.atlassian.net/browse/VECTOR-765