fix(reprocess): drop Lance tables before full-reprocess (99% hang)#394
Merged
Merged
Conversation
…row-delete hang reprocess (cocoindex ``update --full-reprocess`` against an EXISTING table) took the in-place bulk-update merge_insert path, which cocoindex implements as ~one deletion-vector + version commit PER matched row — O(rows) of tiny file IO. Measured on Shopizer (3475 chunks): 3481 versions / 3474 deletion files / 83s sys time, degrading to a multi-minute 99%-CPU hang (native ``_lancedb.abi3.so``, main thread parked in ``kevent``) that reproduces every run and worsens with repo size — a larger repo hung 15+ min at "99%". init (insert into a fresh table) is unaffected: 3 versions / 0 deletions / 4.5s sys. drop+recreate is semantically identical to a full rebuild (all rows are recomputed either way), so make reprocess DROP the Lance target tables first and let cocoindex recreate them via the fast INSERT path. Applied at both ``--full-reprocess`` sites: - ``pipeline.run_cocoindex_update`` → covers ``reprocess --vectors-only`` - ``server.run_refresh_pipeline`` → covers the default ``reprocess`` (vectors+graph) Drop failure is non-fatal: a non-preflight failure logs and falls back to the slow in-place path; a 127/126 preflight stub (graph-only installs) stays silent. Validated on Shopizer: reprocess 105s→hang → 48s (≈ init), sys 83s→4.9s, deletions 3474→0, versions 3481→3. Independent of #392 (concurrency) and #393 (mem-pool/PK-index): ``LANCE_MEM_POOL_SIZE=4GiB`` did not help — this is file-IO amplification, not memory. Tests: ``tests/test_pipeline.py`` — full_reprocess drops exactly once; increment does not drop (would lose the table); drop failure falls back to in-place; a preflight drop stub is silent. Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
reprocesshangs at 99% for many minutes (15+ on a large repo), every run, even after a clean reinit.erase → initworks fine.Root cause
reprocess=cocoindex update --full-reprocessagainst an existing table → the in-place bulk-updatemerge_insertpath. cocoindex implements that as ~one deletion-vector + version commit per matched row — O(rows) of tiny file IO, not a batched operation. It spins at 99% CPU in native_lancedb.abi3.so(main thread parked inkevent), and the deletion files accumulate without cleanup, so it worsens each run.Measured on Shopizer (3475 chunks), with a process sample during the hang confirming native lancedb + heavy
libsystem_malloc/libsystem_kernel:_versions_deletionssystimeinit(insert, fresh table)reprocess(bulk-update)drop+reprocess(insert)LANCE_MEM_POOL_SIZE=4GiBdoes not help (still 105s) — this is file-IO amplification, not memory. So independent of #392 (concurrency) and #393 (mem-pool/PK-index); neither causes nor addresses it.Fix
A full reprocess rebuilds every row, so drop the Lance target tables first and let cocoindex recreate them via the fast INSERT path. Drop+recreate is semantically identical to a full rebuild — all rows are recomputed either way; only the write path differs. Drop only touches the 3 vector tables, not the graph, so
reprocess's graph phase is unaffected.Applied at both
--full-reprocesssites:pipeline.run_cocoindex_update→reprocess --vectors-onlyserver.run_refresh_pipeline→ defaultreprocess(vectors+graph) — the path users hitDrop failure is non-fatal: a non-preflight failure logs and falls back to the slow in-place path; a 127/126 preflight stub (graph-only installs) stays silent.
Validation
End-to-end on Shopizer via the real CLI (default
reprocess):systime_deletions_versionsThe child log confirms the drop cleared the tables (
"No existing dataset ... it will be created"×3) and vectors ticked 25→1150 with no stall. 1130 tests pass (+4 new intests/test_pipeline.py: full_reprocess drops once; increment does not drop; drop failure falls back; preflight stub silent).Workaround until merged
erase → initinstead ofreprocess(same fast insert path, identical rebuild).🤖 Generated with Claude Code