perf: implement bitpack encoding for LID and MID blocks by cheb0 · Pull Request #328 · ozontech/seq-db

cheb0 · 2026-01-26T06:17:58Z

Description

Fixes #312

I have read and followed all requirements in CONTRIBUTING.md;
I used LLM/AI assistance to make this pull request;

If you have used LLM/AI assistance please provide model name and full prompt:

Model: {{model-name}}
Prompt: {{prompt}}

cheb0 · 2026-02-06T14:06:49Z

@seqbenchbot up main search-logbench

seqbenchbot · 2026-02-06T14:06:52Z

Nice, @cheb0 <(-^,^-)=b!

Your request was successfully served.
Identificator for your ongoing benchmark - acee5e23.

Here is a list of helpful links:

Take a look at [Grafana](https://comparison.seqbench.ozon.dev/grafana/d/1f4558f2-1bd6-4034-b6db-6607726e78df?var-scenario=search-logbench&var-container=seqbench-seqdb-acee5e23-baseline&var-container=seqbench-seqdb-acee5e23-comparison&from=seqbench-seqdb-acee5e23-comparison%!(EXTRA string=2026-02-06T14:06:53.520Z)) dashboard;
Live-tailing logs are also available;

Have a great time!

codecov-commenter · 2026-03-10T12:02:45Z

Codecov Report

❌ Patch coverage is 78.21229% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.35%. Comparing base (03387c3) to head (4f795f8).

Files with missing lines	Patch %	Lines
packer/delta_bitpacker.go	79.59%	10 Missing and 10 partials ⚠️
frac/sealed/lids/block.go	80.00%	4 Missing and 2 partials ⚠️
frac/sealed/seqids/blocks.go	58.33%	4 Missing and 1 partial ⚠️
packer/bytes_unpacker.go	0.00%	5 Missing ⚠️
cmd/index_analyzer/main.go	0.00%	2 Missing ⚠️
frac/sealed/seqids/loader.go	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #328      +/-   ##
==========================================
- Coverage   71.40%   71.35%   -0.05%     
==========================================
  Files         206      207       +1     
  Lines       15057    15203     +146     
==========================================
+ Hits        10751    10848      +97     
- Misses       3523     3559      +36     
- Partials      783      796      +13

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-03-10T12:07:11Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`And/size=1000-4`	`03387c`	`228a3d`
	`4.29 ns/op`	`5.21 ns/op`	`1.21`	`🔴`
`Block_Pack-4`	`------`	`228a3d`
	`NaN B/op`	`1144.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`8.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`1478.00 ns/op`	`NaN`	`🔴`
`Block_Unpack-4`	`------`	`228a3d`
	`NaN B/op`	`96260.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`2.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`73415.00 ns/op`	`NaN`	`🔴`
`FindSequence_Random/medium-4`	`03387c`	`228a3d`
	`11010.51 MB/s`	`9478.32 MB/s`	`0.86`	`🔴`
	`93.34 ns/op`	`108.00 ns/op`	`1.16`	`🔴`
`FindSequence_Random/small-4`	`03387c`	`228a3d`
	`6427.67 MB/s`	`5110.10 MB/s`	`0.80`	`🔴`
	`40.85 ns/op`	`50.10 ns/op`	`1.23`	`🔴`
`ProcessDocuments-4`	`03387c`	`228a3d`
	`29061.00 B/op`	`32645.00 B/op`	`1.12`	`🔴`
	`101.50 MB/s`	`91.92 MB/s`	`0.91`	`🔴`
`Sealing_NoSort-4`	`03387c`	`228a3d`
	`24735232.00 B/op`	`70504464.00 B/op`	`2.85`	`🔴`
	`5899.00 allocs/op`	`7128.00 allocs/op`	`1.21`	`🔴`
`Sealing_WithSort-4`	`03387c`	`228a3d`
	`41258232.00 B/op`	`86940672.00 B/op`	`2.11`	`🔴`
	`5980.00 allocs/op`	`7211.00 allocs/op`	`1.21`	`🔴`

github-actions · 2026-03-10T12:45:39Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`AggDeep/size=10000-4`	`03387c`	`f02de4`
	`45145.00 ns/op`	`56064.00 ns/op`	`1.24`	`🔴`
`AggDeep/size=1000000-4`	`03387c`	`f02de4`
	`4554859.00 ns/op`	`5743173.00 ns/op`	`1.26`	`🔴`
`And/size=1000-4`	`03387c`	`f02de4`
	`4.29 ns/op`	`4.99 ns/op`	`1.16`	`🔴`
`Block_Pack-4`	`------`	`f02de4`
	`NaN B/op`	`1144.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`8.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`1468.00 ns/op`	`NaN`	`🔴`
`Block_Unpack-4`	`------`	`f02de4`
	`NaN B/op`	`96260.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`2.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`84777.00 ns/op`	`NaN`	`🔴`
`FindSequence_Random/small-4`	`03387c`	`f02de4`
	`6427.67 MB/s`	`5639.08 MB/s`	`0.88`	`🔴`
	`40.85 ns/op`	`45.40 ns/op`	`1.11`	`🔴`
`Sealing_NoSort-4`	`03387c`	`f02de4`
	`24735232.00 B/op`	`70506000.00 B/op`	`2.85`	`🔴`
	`5899.00 allocs/op`	`7136.00 allocs/op`	`1.21`	`🔴`
`Sealing_WithSort-4`	`03387c`	`f02de4`
	`41258232.00 B/op`	`86941072.00 B/op`	`2.11`	`🔴`
	`5980.00 allocs/op`	`7214.00 allocs/op`	`1.21`	`🔴`

cheb0 · 2026-03-11T10:29:14Z

@seqbenchbot up main bulk

seqbenchbot · 2026-03-11T10:29:17Z

Nice, @cheb0 <(-^,^-)=b!

Your request was successfully served.
Identificator for your ongoing benchmark - 7252a670.

Here is a list of helpful links:

Take a look at Grafana dashboard;
Live-tailing logs are also available;

Have a great time!

github-actions · 2026-03-11T10:35:46Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`AggDeep/size=10000-4`	`03387c`	`89fc6b`
	`45145.00 ns/op`	`55175.00 ns/op`	`1.22`	`🔴`
`AggDeep/size=1000000-4`	`03387c`	`89fc6b`
	`4554859.00 ns/op`	`5600751.00 ns/op`	`1.23`	`🔴`
`Block_Pack-4`	`------`	`89fc6b`
	`NaN B/op`	`113402.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`19.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`106478.00 ns/op`	`NaN`	`🔴`
`Block_Unpack-4`	`------`	`89fc6b`
	`NaN B/op`	`262190.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`2.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`92405.00 ns/op`	`NaN`	`🔴`
`ProcessDocuments-4`	`03387c`	`89fc6b`
	`29061.00 B/op`	`32523.00 B/op`	`1.12`	`🔴`
`Sealing_NoSort-4`	`03387c`	`89fc6b`
	`24735232.00 B/op`	`46557248.00 B/op`	`1.88`	`🔴`
`Sealing_WithSort-4`	`03387c`	`89fc6b`
	`41258232.00 B/op`	`62993128.00 B/op`	`1.53`	`🔴`

cheb0 · 2026-03-11T10:50:37Z

@seqbenchbot down 7252a670

seqbenchbot · 2026-03-11T10:50:40Z

Nice, @cheb0 <(-^,^-)=b!

The benchmark with identificator 7252a670 was finished.
I've prepared a summary for you. Click on Show summary button to see it:

Show summary

Query	Type	`mean (ms)`			`stddev (ms)`			`p(50) (ms)`			`p(95) (ms)`			`p(99) (ms)`			`iterations`
Query	Type	base	comp	diff	base	comp	diff	base	comp	diff	base	comp	diff	base	comp	diff	base	comp	diff
`bulk`	warm	`61.80`	`62.50`	`+1.13%`	`24.56`	`24.98`	`+1.71%`	`54.00`	`55.00`	`+1.85%`	`112.00`	`114.00`	`+1.79%`	`153.00`	`156.00`	`+1.96%`	`26256.00`	`26184.00`	`-0.27%`

Have a great time!

github-actions · 2026-03-11T13:24:12Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`And/size=1000-4`	`03387c`	`9e24e6`
	`4.29 ns/op`	`4.87 ns/op`	`1.14`	`🔴`
`Block_Pack-4`	`------`	`9e24e6`
	`NaN B/op`	`113402.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`19.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`104289.00 ns/op`	`NaN`	`🔴`
`Block_Unpack-4`	`------`	`9e24e6`
	`NaN B/op`	`262190.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`2.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`92226.00 ns/op`	`NaN`	`🔴`
`FindSequence_Random/small-4`	`03387c`	`9e24e6`
	`6427.67 MB/s`	`5582.19 MB/s`	`0.87`	`🔴`
	`40.85 ns/op`	`45.86 ns/op`	`1.12`	`🔴`

github-actions · 2026-03-11T14:04:18Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`AggDeep/size=10000-4`	`03387c`	`683ccd`
	`45145.00 ns/op`	`52262.00 ns/op`	`1.16`	`🔴`
`And/size=1000-4`	`03387c`	`683ccd`
	`4.29 ns/op`	`4.87 ns/op`	`1.14`	`🔴`
`AndTree/size=1000-4`	`03387c`	`683ccd`
	`4.26 ns/op`	`4.88 ns/op`	`1.15`	`🔴`
`Block_Pack-4`	`------`	`683ccd`
	`NaN B/op`	`113402.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`19.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`102899.00 ns/op`	`NaN`	`🔴`
`Block_Unpack-4`	`------`	`683ccd`
	`NaN B/op`	`262189.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`2.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`90524.00 ns/op`	`NaN`	`🔴`
`FindSequence_Random/small-4`	`03387c`	`683ccd`
	`6427.67 MB/s`	`5784.02 MB/s`	`0.90`	`🔴`

github-actions · 2026-03-11T14:11:53Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Block_Pack-4`	`------`	`a5dd12`
	`NaN B/op`	`113402.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`19.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`102100.00 ns/op`	`NaN`	`🔴`
`Block_Unpack-4`	`------`	`a5dd12`
	`NaN B/op`	`262189.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`2.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`89196.00 ns/op`	`NaN`	`🔴`
`FindSequence_Random/medium-4`	`03387c`	`a5dd12`
	`11010.51 MB/s`	`9307.48 MB/s`	`0.85`	`🔴`
	`93.34 ns/op`	`110.00 ns/op`	`1.18`	`🔴`
`FindSequence_Random/small-4`	`03387c`	`a5dd12`
	`6427.67 MB/s`	`5334.37 MB/s`	`0.83`	`🔴`
	`40.85 ns/op`	`47.99 ns/op`	`1.17`	`🔴`

github-actions · 2026-03-11T14:20:10Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Block_Pack-4`	`------`	`e65e13`
	`NaN B/op`	`113402.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`19.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`103719.00 ns/op`	`NaN`	`🔴`
`Block_Unpack-4`	`------`	`e65e13`
	`NaN B/op`	`262191.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`2.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`99584.00 ns/op`	`NaN`	`🔴`
`FindSequence_Random/large-4`	`03387c`	`e65e13`
	`14232.04 MB/s`	`12829.64 MB/s`	`0.90`	`🔴`
`FindSequence_Random/small-4`	`03387c`	`e65e13`
	`6427.67 MB/s`	`5215.72 MB/s`	`0.81`	`🔴`
	`40.85 ns/op`	`49.08 ns/op`	`1.20`	`🔴`

bitpack encoding for LID and MID blocks

d73e08a

ozontech deleted a comment from seqbenchbot Feb 6, 2026

ozontech deleted a comment from github-actions bot Feb 6, 2026

ozontech deleted a comment from seqbenchbot Feb 6, 2026

dkharms added the performance Features or improvements that positively affect seq-db performance label Feb 10, 2026

ozontech deleted a comment from seqbenchbot Feb 13, 2026

cheb0 changed the title ~~perf: bitpack encoding for LID and MID blocks (draft, work in progress)~~ perf: implement bitpack encoding for LID and MID blocks (draft, work in progress) Mar 10, 2026

cheb0 added 2 commits March 10, 2026 14:57

unify bitpack files

fcc9d26

Merge branch 'main' into 312-bitpack-encoding

5e22f88

simplify intcomp usage, refine tests

53ef5eb

cheb0 force-pushed the 312-bitpack-encoding branch from 3accfd6 to 53ef5eb Compare March 11, 2026 10:27

cheb0 changed the title ~~perf: implement bitpack encoding for LID and MID blocks (draft, work in progress)~~ perf: implement bitpack encoding for LID and MID blocks Mar 11, 2026

cheb0 added 3 commits March 11, 2026 15:47

fixes

f2f5e39

naming

ae2bd9f

larger preallocated buf

6d734b5

cheb0 added 3 commits March 11, 2026 16:56

linter fixes, minor fixes

29153ed

tidy mods

0ce64a0

tidy mods

4f795f8

Conversation

cheb0 commented Jan 26, 2026

Description

Uh oh!

cheb0 commented Feb 6, 2026

Uh oh!

seqbenchbot commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Mar 10, 2026

🔴 Performance Degradation

Uh oh!

github-actions bot commented Mar 10, 2026

🔴 Performance Degradation

Uh oh!

cheb0 commented Mar 11, 2026

Uh oh!

seqbenchbot commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 11, 2026

🔴 Performance Degradation

Uh oh!

cheb0 commented Mar 11, 2026

Uh oh!

seqbenchbot commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 11, 2026

🔴 Performance Degradation

Uh oh!

github-actions bot commented Mar 11, 2026

🔴 Performance Degradation

Uh oh!

github-actions bot commented Mar 11, 2026

🔴 Performance Degradation

Uh oh!

github-actions bot commented Mar 11, 2026

🔴 Performance Degradation

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

seqbenchbot commented Feb 6, 2026 •

edited

Loading

codecov-commenter commented Mar 10, 2026 •

edited

Loading

seqbenchbot commented Mar 11, 2026 •

edited

Loading

seqbenchbot commented Mar 11, 2026 •

edited

Loading