Add LanceDB vector database backend

[Copilot]

Change Description

Implements LanceDB as a third vector database backend option alongside ChromaDB and Qdrant.

Solution Description

Added LanceDB class implementing the VectorDB interface with these characteristics:

• Connection: Uses lancedb.connect(path) for database access and open_table() for existing collections
• Storage: Single table per database (vectors.lance), automatic schema creation on first insert
• Duplicate handling: Filters existing IDs before insertion by querying current table state
• ID normalization: Converts all IDs to strings internally while preserving original type in return values
• Search: Native LanceDB vector search with configurable k-nearest neighbors

Implementation files:

• src/hyrax/vector_dbs/lancedb_impl.py - Core implementation
• src/hyrax/vector_dbs/vector_db_factory.py - Factory registration
• src/hyrax/hyrax_default_config.toml - Configuration section
• tests/hyrax/test_lancedb_impl.py - Test coverage

Usage:

[vector_db]
name = "lancedb"  # Options: "chromadb", "qdrant", "lancedb"

Dependency:

dependencies = [
    "lancedb",  # Added to pyproject.toml
]

The implementation follows the same pattern as existing vector DB backends, requiring no changes to consuming code.

Code Quality

• I have read the Contribution Guide and agree to the Code of Conduct
• My code follows the code style of this project
• My code builds (or compiles) cleanly without any errors or warnings
• My code contains relevant comments and necessary documentation

Original prompt

Commit Changes and Delegate: "Delegate to cloud agent"

[Chronological Review: The conversation began with the user asking about the differences between `lance.dataset` and the combination of `lance.connect` and `.open_table`. This was followed by inquiries about the support of `table.take_offsets` for various index types and the capabilities of `HyraxDataset`. The user expressed concerns about returning `Torch.tensor` and its promises by `HyraxDataset`. The conversation then shifted to discussing a design document and the need for a tractable implementation plan. Finally, the user requested to start the implementation and then to commit changes and delegate the task to a cloud agent.]

[Intent Mapping:

1. "Is there a difference between using lance.dataset and the combination of lance.connect and .open_table? Is one preferred?" - User seeks clarification on usage preferences.
2. "Does table.take_offsets support all of these index types?" - User wants to confirm compatibility with HyraxDataset.
3. "What about returning Torch.tensor? Is that promised by HyraxDataset?" - User is questioning the return types and their guarantees.
4. "Take a look at this design document and develop a plan for implementing it..." - User requests a structured implementation plan.
5. "Start implementation" - User initiates the implementation phase.
6. "Commit Changes and Delegate: 'Delegate to cloud agent'" - User requests to commit changes and delegate the task.]

[Technical Inventory:

• lance.dataset: A method for accessing datasets.
• lance.connect: A method for establishing connections to data sources.
• table.take_offsets: A method for retrieving offsets from tables.
• HyraxDataset: A dataset class that may have specific return type guarantees.
• Torch.tensor: A tensor type from the PyTorch library, which the user is concerned about being returned by HyraxDataset.]

[Code Archaeology:

• The user is working on the file /Users/derekjones/code/work/hyrax/specs/lance_db_spec.md, which likely contains specifications or tests related to the Lance database and Hyrax integration.
• No specific code changes were detailed in the conversation, but the user is focused on ensuring compatibility and functionality of the dataset handling.]

[Progress Assessment:

• Completed Tasks: User has clarified several technical aspects regarding dataset handling and has initiated an implementation plan.
• Partially Complete Work: The implementation plan is in the process of being developed, and the user has started the implementation phase.
• Validated Outcomes: No specific features or code have been confirmed working yet, as the implementation is just beginning.]

[Context Validation: All critical information for continuation is captured, including user requests for clarification, implementation planning, and delegation of tasks.]

[Recent Commands Analysis:

• Last Agent Commands: "Start implementation" and "Commit Changes and Delegate: 'Delegate to cloud agent'".
• Tool Results Summary: The user has not provided specific results from tool executions, but the commands indicate a transition from planning to execution and delegation.
• Pre-Summary State: The agent was actively working on implementing the design document and preparing to delegate the task to a cloud agent when the token budget was exceeded.
• Operation Context: These commands were executed to move forward with the implementation of the design document while ensuring that the work is manageable and testable.]

1. Conversation Overview: - Primary Objectives: - "Is there a difference between using `lance.dataset` and the combination of `lance.connect` and `.open_table`? Is one preferred?" - "Does `table.take_offsets` support all of these index types?" - "What about returning Torch.tensor? Is that promised by HyraxDataset?" - "Take a look at this design document and develop a plan for implementing it..." - "Start implementation" - "Commit Changes and Delegate: 'Delegate to cloud agent'" - Session Context: The conversation flowed from technical inquiries about dataset handling to planning and initiating implementation. - User Intent Evolution: The user transitioned from seeking clarifications to actively engaging in implementation and delegation.

2. Technical Foundation:

• lance.dataset: Used for accessing datasets.
• lance.connect: Establishes connections to data sources.
• table.take_offsets: Retrieves offsets from tables.
• HyraxDataset: Dataset class with specific return type guarantees.
• Torch.tensor: Tensor type from PyTorch, questioned for compatibility.

3. Codebase Status:

• File Name: /Users/derekjones/code/work/hyrax/specs/lance_db_spec.md
• Purpose: Contains specifications or tests related to Lance database and Hyrax integration.
• Current State: User has initiated implementation and is preparing to delegate tasks.
• Key Code Segments: Not specified in the conversation.
• Dependencies: Rel...

Created from VS Code.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.