From 3ccd944752a42a4a146885100f69dcb45840e407 Mon Sep 17 00:00:00 2001 From: beinan Date: Wed, 15 Oct 2025 19:31:43 +0000 Subject: [PATCH] docs(readme): add Benchmarks section with run steps, report paths, and sample results MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add instructions to run cypher_execution/basic_node_filter/100 time: [548.60 µs 554.26 µs 559.53 µs] thrpt: [178.72 Kelem/s 180.42 Kelem/s 182.28 Kelem/s] change: time: [-26.371% -21.140% -17.228%] (p = 0.00 < 0.05) thrpt: [+20.814% +26.807% +35.817%] Performance has improved. Found 15 outliers among 100 measurements (15.00%) 4 (4.00%) high mild 11 (11.00%) high severe cypher_execution/basic_node_filter/10000 time: [603.65 µs 610.29 µs 617.29 µs] thrpt: [16.200 Melem/s 16.386 Melem/s 16.566 Melem/s] change: time: [-19.356% -15.932% -11.751%] (p = 0.00 < 0.05) thrpt: [+13.316% +18.951% +24.001%] Performance has improved. Found 2 outliers among 100 measurements (2.00%) 2 (2.00%) high mild cypher_execution/basic_node_filter/1000000 time: [584.30 µs 589.22 µs 595.80 µs] thrpt: [1.6784 Gelem/s 1.6972 Gelem/s 1.7115 Gelem/s] change: time: [-22.167% -15.311% -8.9133%] (p = 0.00 < 0.05) thrpt: [+9.7855% +18.080% +28.481%] Performance has improved. Found 8 outliers among 100 measurements (8.00%) 2 (2.00%) high mild 6 (6.00%) high severe cypher_execution/single_hop_expand/100 time: [1.7184 ms 1.7704 ms 1.8189 ms] thrpt: [54.978 Kelem/s 56.484 Kelem/s 58.192 Kelem/s] change: time: [-47.029% -41.384% -36.569%] (p = 0.00 < 0.05) thrpt: [+57.651% +70.603% +88.783%] Performance has improved. cypher_execution/single_hop_expand/10000 time: [2.2412 ms 2.2792 ms 2.3206 ms] thrpt: [4.3092 Melem/s 4.3875 Melem/s 4.4619 Melem/s] change: time: [-46.576% -41.221% -35.844%] (p = 0.00 < 0.05) thrpt: [+55.871% +70.130% +87.181%] Performance has improved. Found 8 outliers among 100 measurements (8.00%) 2 (2.00%) low mild 3 (3.00%) high mild 3 (3.00%) high severe cypher_execution/single_hop_expand/1000000 time: [2.2271 ms 2.2880 ms 2.3526 ms] thrpt: [425.07 Melem/s 437.06 Melem/s 449.01 Melem/s] change: time: [-42.334% -36.263% -30.127%] (p = 0.00 < 0.05) thrpt: [+43.118% +56.896% +73.412%] Performance has improved. Found 3 outliers among 100 measurements (3.00%) 2 (2.00%) high mild 1 (1.00%) high severe cypher_execution/two_hop_expand/100 time: [3.0845 ms 3.1279 ms 3.1774 ms] thrpt: [31.472 Kelem/s 31.970 Kelem/s 32.420 Kelem/s] change: time: [-36.108% -31.979% -28.217%] (p = 0.00 < 0.05) thrpt: [+39.309% +47.013% +56.513%] Performance has improved. Found 4 outliers among 100 measurements (4.00%) 2 (2.00%) high mild 2 (2.00%) high severe cypher_execution/two_hop_expand/10000 time: [4.0882 ms 4.1576 ms 4.2319 ms] thrpt: [2.3630 Melem/s 2.4053 Melem/s 2.4460 Melem/s] change: time: [-36.871% -35.211% -33.554%] (p = 0.00 < 0.05) thrpt: [+50.497% +54.348% +58.406%] Performance has improved. Found 8 outliers among 100 measurements (8.00%) 2 (2.00%) low severe 2 (2.00%) low mild 2 (2.00%) high mild 2 (2.00%) high severe cypher_execution/two_hop_expand/1000000 time: [3.9845 ms 4.0563 ms 4.1399 ms] thrpt: [241.55 Melem/s 246.53 Melem/s 250.97 Melem/s] change: time: [-37.383% -35.323% -33.035%] (p = 0.00 < 0.05) thrpt: [+49.332% +54.614% +59.702%] Performance has improved. Found 8 outliers among 100 measurements (8.00%) 2 (2.00%) low severe 1 (1.00%) low mild 1 (1.00%) high mild 4 (4.00%) high severe - Include quick-run flags () - Document requirement for Usage: protoc [OPTION] PROTO_FILES Parse PROTO_FILES and generate output based on the options given: -IPATH, --proto_path=PATH Specify the directory in which to search for imports. May be specified multiple times; directories will be searched in order. If not given, the current working directory is used. If not found in any of the these directories, the --descriptor_set_in descriptors will be checked for required proto file. --version Show version info and exit. -h, --help Show this text and exit. --encode=MESSAGE_TYPE Read a text-format message of the given type from standard input and write it in binary to standard output. The message type must be defined in PROTO_FILES or their imports. --deterministic_output When using --encode, ensure map fields are deterministically ordered. Note that this order is not canonical, and changes across builds or releases of protoc. --decode=MESSAGE_TYPE Read a binary message of the given type from standard input and write it in text format to standard output. The message type must be defined in PROTO_FILES or their imports. --decode_raw Read an arbitrary protocol message from standard input and write the raw tag/value pairs in text format to standard output. No PROTO_FILES should be given when using this flag. --descriptor_set_in=FILES Specifies a delimited list of FILES each containing a FileDescriptorSet (a protocol buffer defined in descriptor.proto). The FileDescriptor for each of the PROTO_FILES provided will be loaded from these FileDescriptorSets. If a FileDescriptor appears multiple times, the first occurrence will be used. -oFILE, Writes a FileDescriptorSet (a protocol buffer, --descriptor_set_out=FILE defined in descriptor.proto) containing all of the input files to FILE. --include_imports When using --descriptor_set_out, also include all dependencies of the input files in the set, so that the set is self-contained. --include_source_info When using --descriptor_set_out, do not strip SourceCodeInfo from the FileDescriptorProto. This results in vastly larger descriptors that include information about the original location of each decl in the source file as well as surrounding comments. --dependency_out=FILE Write a dependency output file in the format expected by make. This writes the transitive set of input file paths to FILE --error_format=FORMAT Set the format in which to print errors. FORMAT may be 'gcc' (the default) or 'msvs' (Microsoft Visual Studio format). --fatal_warnings Make warnings be fatal (similar to -Werr in gcc). This flag will make protoc return with a non-zero exit code if any warnings are generated. --print_free_field_numbers Print the free field numbers of the messages defined in the given proto files. Groups share the same field number space with the parent message. Extension ranges are counted as occupied fields numbers. --plugin=EXECUTABLE Specifies a plugin executable to use. Normally, protoc searches the PATH for plugins, but you may specify additional executables not in the path using this flag. Additionally, EXECUTABLE may be of the form NAME=PATH, in which case the given plugin name is mapped to the given executable even if the executable's own name differs. --cpp_out=OUT_DIR Generate C++ header and source. --csharp_out=OUT_DIR Generate C# source file. --java_out=OUT_DIR Generate Java source file. --kotlin_out=OUT_DIR Generate Kotlin file. --objc_out=OUT_DIR Generate Objective-C header and source. --php_out=OUT_DIR Generate PHP source file. --pyi_out=OUT_DIR Generate python pyi stub. --python_out=OUT_DIR Generate Python source file. --ruby_out=OUT_DIR Generate Ruby source file. @ Read options and filenames from file. If a relative file path is specified, the file will be searched in the working directory. The --proto_path option will not affect how this argument file is searched. Content of the file will be expanded in the position of @ as in the argument list. Note that shell expansion is not applied to the content of the file (i.e., you cannot use quotes, wildcards, escapes, commands, etc.). Each line corresponds to a single argument, even if it contains spaces. () and optional - Link Criterion HTML reports: - rust/lance-graph/target/criterion/report/index.html - rust/lance-graph/target/criterion/cypher_execution/report/index.html - Add a small table of typical results (median times and throughput) from a recent run covering , , and at sizes 100, 10,000, and 1,000,000 - Note that numbers vary by hardware and environment Updates: rust/lance-graph/README.md --- rust/lance-graph/README.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/rust/lance-graph/README.md b/rust/lance-graph/README.md index 299f6ce4..137a5311 100644 --- a/rust/lance-graph/README.md +++ b/rust/lance-graph/README.md @@ -125,6 +125,41 @@ Most APIs return `Result`. Errors include parsing failures, missi cargo test -p lance-graph ``` +## Benchmarks + +- **Requirements**: + - **protoc**: install `protobuf-compiler` (Debian/Ubuntu: `sudo apt-get install -y protobuf-compiler`). + - Optional: **gnuplot** for Criterion's gnuplot backend; otherwise the plotters backend is used. + +- **Run** (from `rust/lance-graph`): + +```bash +cargo bench --bench graph_execution + +# Quicker local run (shorter warm-up/measurement): +cargo bench --bench graph_execution -- --warm-up-time 1 --measurement-time 2 --sample-size 10 +``` + +- **Reports**: + - Global index: `rust/lance-graph/target/criterion/report/index.html` + - Group index: `rust/lance-graph/target/criterion/cypher_execution/report/index.html` + +- **Typical results** (x86_64, quick run: warm-up 1s, measurement 2s, sample size 10): + +| Benchmark | Size | Median time | Approx. throughput | +|-------------------------------------|-----------|-------------|--------------------| +| `basic_node_filter` | 100 | ~680 µs | ~147 Kelem/s | +| `basic_node_filter` | 10,000 | ~715 µs | ~13.98 Melem/s | +| `basic_node_filter` | 1,000,000 | ~743 µs | ~1.35 Gelem/s | +| `single_hop_expand` | 100 | ~2.79 ms | ~35.9 Kelem/s | +| `single_hop_expand` | 10,000 | ~3.77 ms | ~2.65 Melem/s | +| `single_hop_expand` | 1,000,000 | ~3.70 ms | ~270 Melem/s | +| `two_hop_expand` | 100 | ~4.52 ms | ~22.1 Kelem/s | +| `two_hop_expand` | 10,000 | ~6.41 ms | ~1.56 Melem/s | +| `two_hop_expand` | 1,000,000 | ~6.16 ms | ~162 Melem/s | + +Numbers are illustrative; your hardware, compiler, and runtime load will affect results. + ## Python Bindings Python bindings for this crate live under `python/src/graph.rs` and expose the same configuration and query APIs via PyO3.