perf(vectors): bump cocoindex 1.0.7→1.0.16; fix mount_table_target API

[HumanBean17]

Summary

Bump the vector trio (cocoindex 1.0.7 → 1.0.16, cascading to lancedb 0.30 → 0.34, pyarrow 23 → 25, rich 14 → 15) and keep the single-component loop. This is the "cocoindex-bump-only" follow-up that came out of investigating #380 (which was closed as wontfix — mount_each was benchmarked and rejected).

Why

• Install drift (the main reason). Our pins were cocoindex>=1.0.7,<2 + lancedb>=0.25.3,<0.31. Because cocoindex 1.0.15+ requires lancedb>=0.34, a fresh pip install today resolves to cocoindex 1.0.11, not 1.0.7 — so dev machines (1.0.7), CI, and new users could all sit on different versions. Pinning explicitly forward removes that ambiguity.
• Stay current. 9 minor versions behind an active upstream; picks up lancedb connector fixes (#2235/#2236 nullable schema-only rewrites, #2245 table maintenance, #2239 reject null-PK rows — our rows always carry a UUID id, so low-risk, but it's a correctness guard).
• ~15% faster bulk vectors (measured, see below).

The required API fix

cocoindex 1.0.16 removed lancedb.mount_table_target(num_transactions_before_optimize=…). Optimize is now inline + stats-driven (_RowHandler._maybe_optimize / _evaluate_optimize — compacts only when small fragments accumulate, inside the merge_insert commit path). That replaces the old 1.0.7 background asyncio optimize that raced concurrent Deletes (lancedb#1504), which we previously disabled via num_transactions_before_optimize=10**12.

So this PR:

• Drops the num_transactions_before_optimize kwarg from the 3 mount_table_target calls and removes the now-unused _NUM_TXN_BEFORE_OPTIMIZE constant + rewrites its comment.
• Updates the lance_optimize.py docstring (the race it described is gone; the module still runs a final serialized compaction + scalar/FTS index build — now a harmless double-pass with cocoindex's inline optimize).
• Bumps the four pins.
• Does NOT revert to mount_each (the loop is retained — see perf(vectors): adopt cocoindex native cross-component batching (#2219) when available; revert single-component loop #380).

Benchmark (Shopizer, 1210 Java files / 3475 chunks)

metric	before (loop + cocoindex 1.0.7)	after (loop + cocoindex 1.0.16)
bulk reprocess (vectors)	27.5s	23.5s
no-change increment	3.0s	3.1s
1-file increment	10.4s	10.4s
Lance commits / build	~3	~2

(Methodology and the full 3-config attribution — including the rejected mount_each variant — are in the #380 thread.)

Test notes

Full local run on cocoindex 1.0.16 (Shopizer-scale fixtures):

• Full non-heavy suite: 1316 passed, 14 skipped, 0 failures (exit 0). This includes the 65 test_java_codebase_rag_cli.py tests that exercise the CLI/lifecycle paths.
• Heavy e2e (test_lancedb_e2e.py): 2/3 pass. The 1 failure (test_lancedb_ignore_file_reduces_indexed_java_files) is pre-existing — it fails identically on cocoindex 1.0.7 (the .java-codebase-rag/ignore exclusion isn't applied at the walk matcher; orthogonal to the vector trio).
• A segfault (exit 139) occurs in test_mcp_hints.py but only with JAVA_CODEBASE_RAG_RUN_HEAVY=1, and it is pre-existing — the identical crash reproduces on master + cocoindex 1.0.7 (same test, same Fatal Python error: Segmentation fault). It's a test-env issue (C-extension interaction during the heavy e2e run), not introduced by this bump. Worth filing separately.
• ruff check clean; uv pip check clean (90 packages, no conflicts).

🤖 Generated with Claude Code