Skip to content

Add LanceDB compaction to reclaim disk space#61

Merged
rendro merged 1 commit intomainfrom
add-lancedb-compaction
Mar 30, 2026
Merged

Add LanceDB compaction to reclaim disk space#61
rendro merged 1 commit intomainfrom
add-lancedb-compaction

Conversation

@rendro
Copy link
Copy Markdown
Owner

@rendro rendro commented Mar 30, 2026

Summary

  • Add sediment compact CLI command that runs LanceDB's three-phase optimization (compact fragments, prune old versions, rebuild indices) and reports disk space reclaimed
  • Add automatic background compaction every 50 store/forget operations with conservative settings (single thread, prune versions >1 day old)
  • Add Database::optimize_tables() core method with configurable CompactConfig

Fixes unbounded storage growth caused by LanceDB's append-only architecture — every store, delete, consolidation merge, and FTS rebuild creates new data files without reclaiming space.

Usage

# Manual compaction (reclaim all space)
sediment compact

# Also delete unverified files (<7 days old)
sediment compact --force

# Skip version pruning
sediment compact --no-prune

Test plan

  • cargo fmt — no formatting issues
  • cargo build --release — compiles cleanly
  • cargo test — all 80 tests pass (29 ignored as before, require model download)
  • sediment compact --help — shows correct usage
  • Manual: du -sh ~/.sediment/data/ before and after sediment compact on a production database

@rendro rendro force-pushed the add-lancedb-compaction branch from e6b4dc4 to fdff1cf Compare March 30, 2026 20:47
LanceDB is append-only: every store, delete, and consolidation merge
creates new data files without reclaiming space, causing unbounded
storage growth. This adds three-phase optimization (compact fragments,
prune old versions, rebuild indices) via LanceDB's Table::optimize API.

- `sediment compact` CLI command for manual optimization
- Automatic background compaction every 50 write operations
- Configurable prune duration, thread count, and force mode

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@rendro rendro force-pushed the add-lancedb-compaction branch from fdff1cf to 1848a16 Compare March 30, 2026 20:53
@rendro rendro merged commit 438a947 into main Mar 30, 2026
2 checks passed
@rendro rendro deleted the add-lancedb-compaction branch March 30, 2026 21:09
@rendro rendro mentioned this pull request Mar 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant