feat(vector): Vector Storage with LanceDB and Transformers.js

[prosdev]

Summary

Implements semantic search capabilities using LanceDB (embedded vector database) and Transformers.js (local ML embeddings).

Features

✅ Local-first architecture - No API keys, no external dependencies
✅ Semantic search - Find code by meaning, not just keywords
✅ Automatic embedding generation - Using all-MiniLM-L6-v2 model
✅ Efficient storage - LanceDB columnar format
✅ Batch processing - Configurable batch size (default 32)
✅ Rich metadata - Store context alongside vectors
✅ Document retrieval - Get docs by ID

Testing

• ✅ 40 comprehensive tests, all passing
• ✅ 85.6% statement coverage (100% function coverage)
• ✅ 64.1% branch coverage
• ✅ Tested: initialization, embedding, search, batch operations, edge cases
• ⚠️ Uncovered: Error paths requiring complex mocking infrastructure (14.4%)

Performance

• Query latency: ~10-20ms (embedding + vector search)
• Batch processing: 50 documents in <30 seconds
• Cold start: 1-2 seconds (model loading)
• Storage: ~1.5KB per document (384 float32 + metadata)

Architecture

VectorStorage (high-level API)
    ├── TransformersEmbedder (all-MiniLM-L6-v2, 384 dims)
    └── LanceDBVectorStore (persistent vector storage)

Documentation

• ✅ Comprehensive README with usage examples
• ✅ Real-world repository indexing example
• ✅ API reference and best practices
• ✅ Comparison to alternatives (hash-based vs semantic)
• ✅ Troubleshooting guide

Example Usage

import { VectorStorage } from '@lytics/dev-agent-core';

// Initialize
const storage = new VectorStorage({
  storePath: './vector-data/my-project.lance',
});
await storage.initialize();

// Add documents
await storage.addDocuments([
  {
    id: 'auth-middleware',
    text: 'Authentication middleware with JWT validation',
    metadata: { file: 'src/auth.ts', type: 'function' },
  },
]);

// Semantic search
const results = await storage.search('How do I authenticate users?', {
  limit: 5,
  scoreThreshold: 0.7,
});

Coverage Report

-------------|---------|----------|---------|---------|
File         | % Stmts | % Branch | % Funcs | % Lines |
-------------|---------|----------|---------|---------|
All files    |   85.6  |    64.1  |   100   |   85.03 |
 embedder.ts |   89.18 |      65  |   100   |   88.57 |
 index.ts    |   89.47 |    81.25 |   100   |   89.18 |
 store.ts    |    80.7 |    57.14 |   100   |      80 |
-------------|---------|----------|---------|---------|

Known Limitations

• ⚠️ Delete operation not yet implemented (LanceDB API limitation)
• ⚠️ Model download (~50MB) required on first run (cached thereafter)

Closes

Closes #4

---

Ready for review! This provides the foundation for semantic code intelligence in dev-agent.