fix(vectors): harden large-repo merge_insert (mem pool + PK scalar index)

[HumanBean17]

Context

Follow-up to the init-performance investigation (lever #3). After #392 (concurrent file drain) shipped, I characterized how merge_insert behaves at scale. Two independent issues surfaced; this PR fixes both. They are separate commits, independently revertable.

1. --full-reprocess can crash the hash-join memory pool on large repos

cocoindex is a bare pass-through to lancedb — it never sets a Session/memory_limit, with zero references to memory_pool/LANCE_MEM_POOL_SIZE across its Python package and its 56 MB Rust engine (source-confirmed). So it inherits lance's ~100 MiB FairSpillPool default, tuned for query workloads — too small for the single big merge_insert a flow component emits at commit.

On --full-reprocess, every row matches the existing table (bulk-update path), so the hash-join build side is large. Around 75k–100k chunks it can exhaust the pool:

RuntimeError: lance error: Resources exhausted: Failed to allocate ~200 MB
for HashJoinInput ... 100.0 MB remain available for the total pool

Honest caveat: the crash is allocation-granularity dependent and non-deterministic — observed once under sustained bench load (a real CLI run does lots of embed/graph memory churn before the final merge_insert, so it's closer to the crash-prone case than an isolated probe); it did not reproduce on clean isolated reruns up to 200k. The mechanism and the LANCE_MEM_POOL_SIZE knob are source-confirmed (error string matches DataFusion 52.4.0 embedded in lancedb._lancedb.abi3.so; var read once at session init).

Fix: raise the default to 1 GiB in cocoindex_subprocess_env_defaults() (applied via setdefault, so an operator's own LANCE_MEM_POOL_SIZE still wins). The pool is a reservation ceiling, not a pre-allocation — 1 GiB costs nothing upfront and is safe on constrained hosts. Covers ~500k-chunk build sides; larger repos can raise it further.

Scope: first init (insert into fresh table) and increment (small batch) are unaffected — only reprocess walks into this.

2. Every merge_insert does a full PK scan (no scalar index)

cocoindex's merge_insert defaults to use_index=True, but cocoindex never creates a scalar index on the primary key (declaring primary_key in the schema does NOT auto-build a lance index). So every merge_insert — increment included — is a forced full scan of the id column, O(existing rows).

Measured ~flat through 100k (<5 ms) because the constant is tiny (columnar PK-only scan), but it is O(N) and would dominate increment wall-clock on a multi-million-chunk repo. With the index present, the join does lookups (~O(batch·log N)).

Fix: build a BTREE scalar index on id in optimize_lance_tables(), right after table.optimize() succeeds — mirroring the existing FTS block (local lancedb.index.BTree import, replace=True for idempotency, non-fatal broad except that never flips the "ok" status). Runs after every init/increment/reprocess, so the index exists before the next merge_insert; table.optimize() maintains it afterward.

Validation

• 1126 tests pass (full suite), +1 new test.
• End-to-end (real lancedb): optimize_lance_tables() on a temp index yields [Index(BTree, id), Index(FTS, text)] on all three tables; empty tables (sql/yaml at count=0) create the index without error.
• cocoindex_subprocess_env_defaults() now returns both COCOINDEX_MAX_INFLIGHT_COMPONENTS=256 and LANCE_MEM_POOL_SIZE=1073741824.
• New BTree test is mutation-checked — fails without the lance_optimize change (the pre-existing FTS test's fake lancedb.index module defined only FTS, so the BTree import silently no-op'd under the broad except; fixed by also defining BTree and asserting by column rather than call count).

Bench: ~/jrag-bench/bench_merge_insert.py.

🤖 Generated with Claude Code