btree distributed build cause fragment not ordered in manifest

[jackye1995]

User reported bug, creating this for tracking purpose:

On a smaller table I was able to create a btree index, but on a larger table I was running into an error. I was wondering if anyone had any ideas before I dug deeper. The error was assertion failed: the_id as i64 > last_id

Looks like somehow the fragments are not ordered in the manifest. Very likely that in create_index(fragment_ids=<...>) it's not sorted. We probably can look at how ray is doing the create_scalar_index, maybe it's somehow passing arbitrary order.

cc @chenghao-guo @xloya

[jtuglu1]

I looked into that fragment ordering thing and I wonder if this is where issue arises. Fragments are sorted correctly in manifest (I checked the files directly), but to evenly distribute these fragments we do a .sort() on size to do best-effort bin-packing into workers. Can either do something here like:

non_empty_batches = [sorted(batch) for batch in worker_batches if batch] # otherwise unsorted batches can be passed to the ray job

here. Or, just sort the fragment_ids list in lance dataset.rs in the create_scalar_index hook. IMO the latter option is better (changing in Rust source, so all stubbed clients can benefit from it).

Monkey-patching the function with the fix:

_original_distribute = lr.index._distribute_fragments_balanced
def stub(fragments, num_workers, logger):
    batches = _original_distribute(fragments=fragments, num_workers=num_workers, logger=logger)
    return [sorted(batch) for batch in batches]

lr.index._distribute_fragments_balanced = stub

works.