feat(table): optional prefix bloom filters for prefix-scan pruning#2305
Open
shaunpatterson wants to merge 1 commit into
Open
feat(table): optional prefix bloom filters for prefix-scan pruning#2305shaunpatterson wants to merge 1 commit into
shaunpatterson wants to merge 1 commit into
Conversation
Add WithBloomPrefixLength(n) option. When n>0, the table builder hashes the n-byte prefix of each user key into a SEPARATE bloom filter, stored in two new FlatBuffers TableIndex fields (bloom_prefix_filter, bloom_prefix_length) appended at the end of the schema. Appending is forward/backward compatible: old readers ignore the new fields and old tables decode them as nil/0. Prefix-bounded iterators (opt.Prefix set, not single-key) consult the prefix bloom via Table.DoesNotHavePrefix in pickTable/pickTables to skip whole SSTables that provably contain no key with the scan prefix. Tables without a prefix bloom always return false, preserving existing behavior. Default (n=0) is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NtGkC4K2J2XYwcAKwjhHbM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The per-table bloom filter is built over the full user key and is only consulted for single-key iteration (
prefixIsKey). A real prefix scan over many distinct keys sharing a prefix gets no bloom benefit — only key-range pruning. Prefix scans are a common access pattern (all keys under a predicate/namespace).What changed
New opt-in
WithBloomPrefixLength(n)(default 0 = disabled). Whenn > 0, each SSTable additionally stores a bloom filter built over the firstnbytes of every user key. A prefix-bounded scan then consults it (Table.DoesNotHavePrefix) inpickTable/pickTablesto skip whole SSTables that provably contain no key with the scan prefix — pruning tables whose key range straddles the prefix where range-pruning alone can't help.Stored as two appended
TableIndexflatbuffer fields (bloom_prefix_filter,bloom_prefix_length).Compatibility & correctness
DoesNotHavePrefixis conservative: it returns "do not skip" unless the table has a prefix bloom and the scan prefix is at leastprefixBloomLengthbytes and the bloom proves absence. Keys shorter thannare hashed whole (they can't match a longer scan prefix anyway). Tables written without a prefix bloom (old format, orn == 0) are never skipped. Default (BloomPrefixLength == 0) behavior is byte-for-byte unchanged.Testing
go build,go vet,go test . ./table/...green. Tests cover: present prefix returns all matches; absent prefix pruned (incl. a table whose key range straddles the absent prefix); short scan prefix conservatively not pruned; old-format table opens and is never skipped; default-off unchanged.🤖 Generated with Claude Code
https://claude.ai/code/session_01NtGkC4K2J2XYwcAKwjhHbM