[Enhance]: Export DB

[croqaz]

Affected Component

Python API

Current Behavior

Hi. I don't know for sure if this is an enhancement, or feature request. I searched the documentation over and over and I still don't know how to export a whole DB into something like JSON lines.

Why? Well, I am running different processes on different machines to calculate statistics and embeddings and when they are finished, I want to join all the Zvec DBs into one. So maybe that's the main feature, joining multiple DBs?

But exporting as JSON will allow better compatibility with any other tool from any other programming language.

The error when I'm trying to export a whole DB:

  File ".venv/lib/python3.14/site-packages/zvec/executor/query_executor.py", line 236, in execute
    docs = self._do_execute(query_vectors, collection)
  File ".venv/lib/python3.14/site-packages/zvec/executor/query_executor.py", line 189, in _do_execute
    docs = collection.Query(query)
ValueError: query validate failed: topk[10000000] is too large, max is 1024

I don't know what entries I Have in there, so I can either fetch them one by one. The DBs are big, tens of millions of entries.

I know you are using RocksDB somewhere in there, so it should be easy to expose some lazy iter() function for keys + values, that should solve the problem.

Desired Improvement

Export feature/ Merge DBs/ Lazy iter

Impact

[zhourrr]

Hi there, thank you for using zvec and for this detailed feature request.

We currently don't have a native export function. Regarding the topk error, we are relaxing the limit to 100,000 in the next release. This should serve as a workaround for smaller databases. As for joining databases, that sounds like a distinct feature, so please feel free to open a separate issue for that if you need it.

We are leaning towards exporting in parquet format rather than JSON. Since vector indexes can be massive, storing numbers in ASCII text (like JSON) would result in extremely large files.

To help us design a solution that fits your needs, could you share a bit more about your workflow? For instance, what is the scale of your data (number of vectors or index size), and how do you plan to consume the exported data?

[croqaz]

Hi! Thank you for the response. I think you shouldn't design features for my specific needs, since I'm only one user and you have a project with almost 10k stars, so you want to think about everybody using this project.

MY scale is roughly 50 million records.

But my reasoning is: since this is a RocksDB underneath, I want to profit from the massive scale that this DB can store and retrieve. (RocksDB and LevelDB are used to store all transactions from Ethereum and Bitcoin, so they can scale)
So, the more I think about different usecases, the best solution is to allow iterating the DB over keys and values( eg: db.iterator(..), look at the API of this project for inspiration: https://github.com/Level/abstract-level#iterator--dbiteratoroptions )

Such lazy iteration can be used for extracting stats on the whole DB with little memory usage, and if you iterate a few open DBs at the same time, you can also merge them.

The way I understand it, right now I can only store up to 1024 records inside Zvec, because I have no way of iterating the whole DB to extract more than that, is that true ?
And in the future, I will only be able to query as much as 100k records ?

How can I access the rest of the data ?

Collection.query is limited to 1000/ 100k, right? And Collection.fetch requires a list of IDs, right? So how can I get the list of IDs from? The DB is a black box with no way of knowing what's inside.

Natively exporting as Parquet is surely nice, but the iterator is way more powerful.

[croqaz]

This issue: #335 also mentions pagination for query, which I think it's a great idea, something like: query 1000 documents, offset by 3000.

[zhourrr]

Thanks for the detailed follow-up — and yes, absolutely, we should design this as a general capability that benefits a broad set of users, not just one workflow. Your use case is still very helpful though, because it gives us a concrete starting point for thinking through the requirements.

A few clarifications that may help:

• Although we do use RocksDB internally, it is not the primary storage abstraction for the whole system. It serves as the backend for some internal components.
• That said, exposing some form of iterator over primary keys / values is feasible. It would require some engineering work, but it is a reasonable direction.
• The current topk cap is a hard limit for vector similarity search and filtered search. In the next release, that limit will be increased to 100K.
• The difficulty with adding something like offset pagination to query() is that vector indexes do not have a natural deterministic order like a traditional SQL table. To return “results 3001–4000,” we would typically still need to retrieve the top 4000 first, then discard the first 3000. So offset-based pagination is not as straightforward in ANN/vector search as it is in relational databases.
• That is also part of why we are increasing the topk limit to 100K — our hope is that this will cover many practical similarity-search and filtered-search use cases.

That said, we do agree with the core point: users need a programmatic way to get their data out of the database, independent of similarity search limits. An iterator-like API is a strong candidate for that. Exporting to a file is also a valid option here. We’ll take some time to think through the right API/design for it. In the meantime, ideas and suggestions on the interface or workflow are very welcome.

[croqaz]

Thank you for that! In the meantime, I indexed a small part of my dataset and while I was writing this issue, I realized I have no way of getting it back. I built a small brute force script to extract all the entries from the black box, but it's impossible. Eg:

import zvec

coll = zvec.open('./my_index')
total = coll.stats.doc_count
print(f'Total docs: {total}')

all_ids = set()
TOPK = 1024

# Scan every integer length value
for length_val in range(11, 10_000_001):
    batch = coll.query(
        filter=f'length = {length_val}',
        topk=TOPK,
        output_fields=[],
    )
    if batch:
        for doc in batch:
            all_ids.add(doc.id)
        if len(batch) == TOPK:
            print(f'  WARNING: length={length_val} saturated at {TOPK}')
    if len(all_ids) >= total:
        break
    if length_val % 10_000 == 0:
        print(f'  scanned up to length={length_val}, found {len(all_ids):,}')

print(f'Collected {len(all_ids):,} / {total:,}')

Output for a small dataset:

Total docs: 154,702
  scanned up to length=10000, found 154,587
  scanned up to length=20000, found 154,598
  scanned up to length=30000, found 154,614
  scanned up to length=40000, found 154,634
  scanned up to length=50000, found 154,662
  scanned up to length=60000, found 154,673
  scanned up to length=70000, found 154,688
  scanned up to length=80000, found 154,695
  scanned up to length=90000, found 154,699
  scanned up to length=100000, found 154,700
Collected 154,702 / 154,702

Output for a medium dataset:

Total docs: 2,695,830
  WARNING: length=11 saturated at 1024
  WARNING: length=12 saturated at 1024
  WARNING: length=13 saturated at 1024
  WARNING: length=14 saturated at 1024
  WARNING: length=15 saturated at 1024
  WARNING: length=16 saturated at 1024
  WARNING: length=17 saturated at 1024
  WARNING: length=18 saturated at 1024
  WARNING: length=19 saturated at 1024
  WARNING: length=20 saturated at 1024
...
...
  scanned up to length=10000, found 2,051,123
  scanned up to length=20000, found 2,318,430
  scanned up to length=30000, found 2,397,458
  scanned up to length=40000, found 2,428,196
  scanned up to length=50000, found 2,442,324
  scanned up to length=60000, found 2,449,377
  scanned up to length=70000, found 2,452,905
  scanned up to length=80000, found 2,454,801
  scanned up to length=90000, found 2,455,784
  scanned up to length=100000, found 2,456,324
...

Notice it's impossible to retrieve all 2,695,830 documents, even with brute force.

Even with a larger topk=100k it's still impossible to extract all the documents using brute force if the DB is large enough.

EDIT: In reality in my case, the IDs are hashes for text, so I can hash the text again and retrieve the actual documents from Zvec, but imagine a SUPER valid usecase when the IDs are UUIDs (eg: 3fa88562-7c79-43f0-9dc9-904c9688925e) so they are all random. In that case it's impossible to get back your data from the black box, if the DB is large enough.

[croqaz]

In the meantime, ideas and suggestions on the interface or workflow are very welcome.

I suggest this tried and tested API, used for these Node.js collection of libraries:

• https://github.com/Level/abstract-level
• https://github.com/Level/level
• https://github.com/Level/classic-level
• https://github.com/Level/rocksdb
• https://github.com/Level/level-rocksdb
• etc, etc, many more

This is the gist of their iteration API:

// Iterate entries with keys that are greater than 'a' and smaller than 'm'
for await (const [key, value] of db.iterator({ gt: 'a', lt: 'm', limit: 10 })) {
  console.log(value)
}

All params are optional. You basically have a window of key names to iterate, the computation is minimal. Iteration is lazy, you get one item at a time. I am sure that some underlying storage DB you are using has a way of opening the DB in a streaming way, so it doesn't blow up the memory OR CPU.

In my usecase, I have xxhash.xxh64_hexdigest as the main key (so they are UINT64 numbers as string), so I can easily iterate that.