diff --git a/.gitignore b/.gitignore index 141701d..ddd09b3 100644 --- a/.gitignore +++ b/.gitignore @@ -13,3 +13,9 @@ ts/node_modules/ .env infra/node_modules/ infra/cdk.out/ + +# Aperant data directory +.auto-claude/ + +# Git worktrees +.worktrees/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 8659cd0..7b72ad7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,23 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). +## [1.1.0] — 2026-03-21 — "Enterprise Platform" + +### Added + +- **F1: Guided Bootstrap** (`lore bootstrap`): Single command validates Python version, Postgres, pgvector, Docker, runs migrations, and verifies server health. `--fix` flag auto-remediates missing dependencies. +- **F2: Enhanced Setup Wizard**: Config validation (`--validate`), server connectivity test (`--test-connection`), dry-run mode (`--dry-run`), timestamped config backups with rollback instructions. New `POST /v1/setup/validate` endpoint. +- **F3: SLO Dashboard + Alerting**: Define SLO targets for p50/p95/p99 latency and hit rate. Background checker evaluates every 60s. Webhook and email alert channels. Time-series API for charts. Full CRUD via `lore slo` CLI and REST API. +- **F4: Adaptive Retrieval Profiles**: Named profiles stored in Postgres with per-request `?profile=` param. Three built-in presets (coding, incident-response, research). Profiles control semantic weight, graph weight, recency bias, tier filters, and min score. 60s in-memory cache. +- **F5: Graph Approval Inbox with Risk Scoring**: Risk score computed via SQL CTE (weight + conflict history + age). Sortable review queue, batch approve/reject, reviewer notes, `review_decisions` audit table. `GET /v1/review/history` for audit trail. +- **F6: Policy-Based Retention**: Declarative lifecycle policies with per-tier retention windows, cron-based snapshot schedules, max snapshot limits. Restore drills with timing metrics. Cross-policy compliance dashboard. +- **F7: Multi-Tenant Workspaces**: Workspace isolation within orgs. Scoped API keys, member management (writer/admin roles), full audit log of every action. `lore workspace create/switch/members`, `lore audit` CLI commands. +- **F8: Plugin SDK**: `LorePlugin` ABC with 5 lifecycle hooks (`on_remember`, `on_recall`, `on_enrich`, `on_extract`, `on_score`). Discovery via Python entry_points. Hot-reload, enable/disable, scaffold CLI (`lore plugin create`), test harness. +- **F9: Proactive Recommendations**: Multi-signal scoring engine (context similarity, entity overlap, temporal patterns, access patterns). Human-readable explanations. Feedback loop with per-user weight adjustment. New MCP tool: `suggest`. Configurable aggressiveness and cooldown. +- 6 new database migrations (012–017) +- 118 new tests (2081 total) +- Background tasks: SLO checker + retention policy scheduler in FastAPI lifespan + ## [1.0.0] — 2026-03-14 — "Total Recall" ### Added diff --git a/CLAUDE.md b/CLAUDE.md index 5a3b5f5..9ea0c2b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -31,9 +31,11 @@ save_snapshot(content="Current state: ...", title="mid-refactor checkpoint") | `remember(content, type)` | Save a memory (types: note, lesson, fact, preference, pattern, convention) | | `recent_activity(hours)` | Load recent session context | | `save_snapshot(content)` | Manually checkpoint current state | +| `suggest(context)` | Get proactive memory recommendations based on session context | | `topics()` | Browse auto-generated topic summaries | | `graph_query(query)` | Explore knowledge graph connections | | `entity_map(name)` | Find everything related to an entity | +| `review_digest()` | Review pending graph connections (with risk scores) | | `on_this_day()` | Memories from this date in prior years | | `export(format)` | Export all data (json/markdown) | diff --git a/README.md b/README.md index d09b084..166dce6 100644 --- a/README.md +++ b/README.md @@ -61,15 +61,50 @@ Auto-generated concept hubs that cluster related memories, entities, and facts a Full data export in JSON and Markdown formats. Obsidian-compatible output for browsing your knowledge graph in a PKM tool. Snapshots for backup and migration. -### Approval UX -`review_digest` · `review_connection` +### Approval UX with Risk Scoring +`review_digest` · `review_connection` · `lore review list --sort risk` -Review discovered knowledge graph connections before they become permanent. Approve, reject, or skip — keep your graph clean. +Review discovered knowledge graph connections with computed risk scores. Batch approve/reject with notes, full audit trail of decisions. Sort by risk, confidence, or age. + +### Guided Bootstrap +`lore bootstrap` + +Single command that validates Python version, Postgres, pgvector, Docker, runs migrations, and verifies server health. Use `--fix` to auto-remediate missing dependencies. ### Multi-Agent Setup `lore setup claude-code` · `lore setup openclaw` · `lore setup cursor` · `lore setup codex` -One-command hook installation for all major AI coding agents. Auto-retrieval injected into every prompt — no code changes needed. +One-command hook installation for all major AI coding agents. Auto-retrieval injected into every prompt — no code changes needed. Includes `--validate`, `--test-connection`, and `--dry-run` flags. + +### SLO Dashboard + Alerting +`lore slo create` · `lore slo status` · `GET /v1/slo/status` + +Define SLO targets for retrieval latency (p50/p95/p99) and hit rate. Background checker evaluates every 60s and fires webhook or email alerts on breach. Time-series API for charting. + +### Adaptive Retrieval Profiles +`lore profiles list` · `GET /v1/profiles` · `?profile=coding` + +Named retrieval profiles stored in Postgres. Presets for coding (recency-biased), incident response (graph-heavy), and research (long-term). Select per-request or set as API key default. + +### Policy-Based Retention +`lore policy create` · `lore restore-drill` · `GET /v1/policies/compliance` + +Declarative lifecycle policies with per-tier retention windows, cron-based snapshot schedules, and restore drills with timing metrics. Compliance dashboard across all policies. + +### Multi-Tenant Workspaces +`lore workspace create` · `lore workspace switch` · `lore audit` + +Workspace isolation within orgs. Scoped API keys, member management with RBAC roles, and a full audit log of every action (memory.create, key.revoke, etc.). + +### Plugin SDK +`lore plugin create` · `lore plugin list` · `lore plugin reload` + +Extend Lore with plugins discovered via Python entry_points. Five lifecycle hooks (`on_remember`, `on_recall`, `on_enrich`, `on_extract`, `on_score`), hot-reload, scaffold CLI, and test harness. + +### Proactive Recommendations +`suggest` · `lore suggest --context "..."` · `GET /v1/recommendations` + +Surface relevant memories before explicit queries. Multi-signal scoring (context similarity, entity overlap, temporal patterns, access patterns) with human-readable explanations and a feedback loop. ### Retrieval Analytics `GET /v1/analytics/retrieval` · Prometheus metrics @@ -230,6 +265,7 @@ curl -s "http://localhost:8765/v1/retrieve?query=your+prompt&limit=5&min_score=0 | `review_digest` | Get pending connections for review | | `review_connection` | Approve/reject a pending connection | | `on_this_day` | Memories from same date across years | +| `suggest` | Proactive memory recommendations based on session context | ## CLI Reference @@ -257,18 +293,49 @@ lore import backup.json lore snapshot-save --title "before refactor" # Server & setup +lore bootstrap # validate prerequisites lore serve # start HTTP server lore mcp # start MCP server lore ui # start web UI lore setup claude-code # install hooks -lore setup openclaw -lore setup cursor -lore setup codex +lore setup claude-code --validate --test-connection + +# SLO management +lore slo create --name "P99 < 50ms" --metric p99_latency --threshold 50 --operator lt +lore slo status +lore slo alerts + +# Retrieval profiles +lore profiles list +lore profiles create --name fast-coding --semantic-weight 1.0 --recency-bias 7 + +# Retention policies +lore policy create --name prod --snapshot-schedule "0 2 * * *" --max-snapshots 30 +lore policy compliance +lore restore-drill --latest + +# Workspaces +lore workspace create dev-team +lore workspace switch dev-team +lore audit --since 24h + +# Plugins +lore plugin create my-tagger +lore plugin list +lore plugin reload my-tagger + +# Recommendations +lore suggest --context "setting up docker" + +# Review (with risk scoring) +lore review list --sort risk +lore review approve --note "Verified" +lore review batch approve --ids id1,id2 # API keys -lore keys-create --name "my-agent" -lore keys-list -lore keys-revoke +lore keys create --name "my-agent" +lore keys list +lore keys revoke ``` ## API Reference @@ -276,6 +343,7 @@ lore keys-revoke ### Key endpoints ``` +# Memory CRUD GET /v1/retrieve # Auto-retrieval (for hooks) POST /v1/memories # Create memory POST /v1/memories/search # Semantic search @@ -284,26 +352,71 @@ GET /v1/memories/{id} # Get memory PATCH /v1/memories/{id} # Update memory DELETE /v1/memories/{id} # Delete memory +# Knowledge graph GET /v1/graph # Knowledge graph GET /v1/graph/topics # Topic list GET /v1/graph/topics/{name} # Topic detail GET /v1/graph/entity/{id} # Entity detail +# Ingestion POST /v1/conversations # Extract memories from conversation POST /v1/ingest # Ingest external content -POST /v1/ingest/batch # Batch ingest -GET /v1/review # Pending connection reviews -POST /v1/review/{id} # Approve/reject connection +# Review + risk scoring +GET /v1/review # Pending reviews (sortable by risk) +POST /v1/review/{id} # Approve/reject with notes +POST /v1/review/bulk # Batch approve/reject +GET /v1/review/history # Decision audit trail +# Export & snapshots POST /v1/export # Export all data POST /v1/import # Import data POST /v1/export/snapshots # Create snapshot GET /v1/export/snapshots # List snapshots +# SLO dashboard +GET /v1/slo # List SLO definitions +POST /v1/slo # Create SLO +GET /v1/slo/status # Current pass/fail per SLO +GET /v1/slo/alerts # Alert history +GET /v1/slo/timeseries # Time-series for charts + +# Retrieval profiles +GET /v1/profiles # List profiles +POST /v1/profiles # Create profile +GET /v1/retrieve?profile=coding # Retrieve with profile + +# Retention policies +GET /v1/policies # List policies +POST /v1/policies # Create policy +GET /v1/policies/compliance # Compliance summary +POST /v1/policies/{id}/drill # Execute restore drill + +# Workspaces + RBAC +POST /v1/workspaces # Create workspace +GET /v1/workspaces # List workspaces +POST /v1/workspaces/{id}/members # Add member +GET /v1/audit # Query audit log + +# Plugins +GET /v1/plugins # List plugins +POST /v1/plugins/{name}/enable # Enable plugin +POST /v1/plugins/{name}/reload # Hot-reload plugin + +# Recommendations +POST /v1/recommendations # Get proactive suggestions +POST /v1/recommendations/{id}/feedback # Thumbs up/down +PATCH /v1/recommendations/config # Adjust aggressiveness + +# Setup validation +POST /v1/setup/validate # Test connectivity + +# Analytics & monitoring GET /v1/recent # Recent activity -GET /v1/analytics/retrieval # Retrieval analytics + Prometheus +GET /v1/analytics/retrieval # Retrieval analytics +GET /metrics # Prometheus metrics +# API keys POST /v1/keys # Create API key GET /v1/keys # List API keys DELETE /v1/keys/{id} # Revoke API key @@ -329,6 +442,14 @@ DELETE /v1/keys/{id} # Revoke API key | `LORE_GRAPH_CONFIDENCE_THRESHOLD` | `0.5` | Entity confidence threshold | | `LORE_HTTP_TIMEOUT` | `30` | HTTP timeout (seconds) | | `OPENAI_API_KEY` | — | Auto-enables enrichment when set | +| `SLO_CHECK_INTERVAL` | `60` | SLO evaluation interval (seconds) | +| `ALERT_WEBHOOK_URL` | — | Default webhook URL for SLO alerts | +| `SMTP_HOST` | — | SMTP server for email alerts | +| `SMTP_PORT` | `587` | SMTP port | +| `SMTP_USER` | — | SMTP username | +| `SMTP_FROM` | — | Email sender address | +| `AUTH_MODE` | `api-key-only` | Auth mode: `api-key-only`, `dual`, `oidc-required` | +| `LORE_WORKSPACE` | — | Default workspace slug | ## Architecture @@ -342,14 +463,19 @@ DELETE /v1/keys/{id} # Revoke API key ┌──────────────────────────────────────────────────────────────┐ │ Lore Server (:8765) │ │ │ -│ REST API · MCP Server · Web UI (/ui/) │ +│ REST API · MCP Server · Web UI (/ui/) · Plugin SDK │ │ │ │ ┌─────────────┐ ┌──────────────┐ ┌─────────────────────┐ │ │ │ Embedder │ │ Knowledge │ │ LLM Pipeline │ │ │ │ (ONNX) │ │ Graph │ │ (optional) │ │ -│ │ │ │ Engine │ │ classify · enrich │ │ -│ │ pgvector │ │ entities │ │ extract · consolidate│ -│ │ search │ │ relations │ │ │ │ +│ │ pgvector │ │ + Review │ │ classify · enrich │ │ +│ │ + Profiles │ │ + Risk │ │ extract · recommend │ │ +│ └─────────────┘ └──────────────┘ └─────────────────────┘ │ +│ │ +│ ┌─────────────┐ ┌──────────────┐ ┌─────────────────────┐ │ +│ │ SLO │ │ Retention │ │ Workspaces │ │ +│ │ Checker │ │ Scheduler │ │ + RBAC │ │ +│ │ + Alerting │ │ + Drills │ │ + Audit Log │ │ │ └─────────────┘ └──────────────┘ └─────────────────────┘ │ └──────────────────────────┬───────────────────────────────────┘ │ @@ -357,6 +483,8 @@ DELETE /v1/keys/{id} # Revoke API key │ PostgreSQL + pgvector │ │ memories · entities │ │ relationships · facts │ + │ slo · profiles · audit │ + │ workspaces · policies │ └─────────────────────────┘ ``` diff --git a/docs/self-hosted.md b/docs/self-hosted.md index f47bfd7..8becacb 100644 --- a/docs/self-hosted.md +++ b/docs/self-hosted.md @@ -10,7 +10,7 @@ Run Lore Cloud on your own infrastructure with Docker Compose. ## Quick Start ```bash -git clone https://github.com/amitpaz1/lore.git +git clone https://github.com/agentkitai/lore.git cd lore # Set a real password for production @@ -21,6 +21,15 @@ docker compose -f docker-compose.prod.yml up -d Server is now running at `http://localhost:8765`. Migrations run automatically on startup. +### Guided Bootstrap + +For non-Docker installs, use the bootstrap command to validate prerequisites: + +```bash +lore bootstrap --verbose +lore bootstrap --fix # auto-remediate missing deps +``` + ## Initialize Your Organization ```bash @@ -61,6 +70,13 @@ lessons = lore.query("my problem") |----------|---------|-------------| | `DATABASE_URL` | (set in compose) | PostgreSQL connection string | | `POSTGRES_PASSWORD` | `lore` | DB password (set in `.env` or export) | +| `AUTH_MODE` | `api-key-only` | Auth mode: `api-key-only`, `dual`, `oidc-required` | +| `SLO_CHECK_INTERVAL` | `60` | SLO evaluation interval (seconds) | +| `ALERT_WEBHOOK_URL` | — | Webhook URL for SLO alerts | +| `SMTP_HOST` | — | SMTP server for email alerts | +| `SMTP_PORT` | `587` | SMTP port | +| `SMTP_USER` | — | SMTP username | +| `SMTP_FROM` | — | Sender address for alert emails | ## Persistence