Skip to content

PROGRESS.md

Latest commit

 

History

History
192 lines (151 loc) · 18.1 KB

PROGRESS.md

File metadata and controls

192 lines (151 loc) · 18.1 KB
Raw

Progress

Tracks build progress against the phases in LEGATE_AGENT_BUILD_PLAN.md (Section 12).

Status summary

Phase Title Status
0 Foundation and scaffolding ✅ Complete
1 Auth, workspaces, RBAC ✅ Complete
2 LLM layer and Agent Core ✅ Complete
3 RAG knowledge layer ✅ Complete
4 Connector framework and credential vault ✅ Complete
5 Policy Engine and approvals ✅ Complete
6 Workflow engine and triggers ✅ Complete
7 Dashboard UX ✅ Complete
8 Enterprise hardening ✅ Complete

Phase 0 — Foundation and scaffolding ✅

Done

  • Monorepo layout per Section 10 (apps/api, apps/web, packages/shared-types, docs/).
  • docker-compose.yml with postgres, redis, qdrant, api, worker, web (+ prod overlay with Caddy TLS).
  • FastAPI app skeleton with GET /health returning 200.
  • Pydantic-settings config loader reading every variable in Section 11.
  • Tooling: ruff, black, mypy (backend); eslint/prettier config (frontend); pytest.
  • GitHub Actions CI: lint + format check + types + tests + CodeQL.
  • .env.example, README.md, PROGRESS.md, MIT LICENSE, Makefile.

Acceptance: docker compose up brings up all services; GET /health returns 200; CI runs lint + tests.

Phase 1 — Auth, workspaces, RBAC ✅

Done

  • Models + Alembic migration: User, Workspace, Membership, ApiKey, AuditLog (UUID PKs, timestamps, tenant workspace_id).
  • Register / login / refresh with JWT access + refresh tokens (short access TTL, rotation on refresh).
  • Argon2 password hashing.
  • Workspace create + list + switch (active workspace carried in the access token).
  • RBAC: Owner / Admin / Builder / Operator / Viewer with the exact permission matrix from Section 7.3, enforced by a require(permission) dependency.
  • API keys: create (plaintext shown once), list, revoke; stored as SHA-256 hashes with scopes; X-API-Key auth with scope checks.
  • Hash-chained, append-only audit log; every privileged action writes an entry.
  • Tenant filtering at the repository layer — a principal can only read/write within its active workspace.

Acceptance: a user can register, create a workspace, invite a member with a role, and role checks block unauthorized calls; API key auth works; every privileged action writes an audit entry; tests cover the RBAC matrix, tenant isolation, and the audit hash chain.

Decisions / TODOs

  • # TODO(decision): invited users who do not yet exist are created as inactive, password-less accounts; the accept-invite + set-password flow is deferred to the dashboard phase.
  • # TODO(decision): API-key management (create/list/revoke) is gated under the members:manage permission (Owner/Admin) until a dedicated admin capability is introduced.
  • Audit seq is assigned per-workspace in the application layer; a DB-level monotonic guard is a follow-up for high-concurrency writes.

Phase 2 — LLM layer and Agent Core ✅

Done

  • Provider-agnostic async LLM client (legate/llm): OpenRouter + Ollama providers, a config-driven factory, and a scripted FakeLLMClient for tests.
  • Agent, Run, Step models + Alembic migration (RunStatus / StepType enums).
  • Agent CRUD API with tool-allow-list validation, plus GET /tools discovery.
  • Plan-act-observe loop (legate/agent/runner.py): structured {thought, action, args} output, JSON-schema arg validation, Policy Engine gate, tool execution, and observations fed back as untrusted data.
  • Guardrails/budgets: max steps, max tokens, and max wall-clock — per-agent overrides falling back to workspace settings; malformed output is retried (bounded) then fails safe.
  • Short-term memory: transcript token budgeting/trimming (system messages always retained).
  • Two built-in tools: kb.search (stub until Phase 3) and read-only http.request.
  • Live run streaming over WebSocket (/runs/{id}/stream) via an in-process event bus, with DB replay for late subscribers; background vs inline execution.
  • Minimal Policy Engine (legate/policy/engine.py) with the final allow / deny / require_approval vocabulary — default-deny + sensitive-tool suspend now, full rules in Phase 5.

Acceptance: create an agent, run it, and watch steps stream in over WebSocket; the agent calls a tool, gets an observation, and returns a final answer; step/token/wall-clock budgets are enforced; malformed LLM output is retried then fails safe. Tests mock the LLM and assert the loop, budgets, retries, RBAC, tenant isolation, and streaming.

Decisions / TODOs

  • # TODO(decision): sensitive tools currently suspend the run as waiting_approval with no Approval record; the inbox/resume flow lands in Phase 5.
  • # TODO(decision): kb.search returns an empty result set until the RAG layer (Phase 3) backs it with Qdrant.
  • Background runs use asyncio.create_task (in-process); moving execution onto Celery workers is a later hardening step.

Phase 3 — RAG knowledge layer ✅

Done

  • KnowledgeBase, Document, Chunk models (+ DocumentStatus enum) and Alembic migration.
  • Swappable embeddings (legate/rag/embeddings): OpenAIEmbedder + an offline, deterministic HashingEmbedder, behind a factory.
  • Swappable vector store (legate/rag/vectorstore): QdrantVectorStore + in-process InMemoryVectorStore; lazy collection creation, per-workspace+KB isolation.
  • Ingestion pipeline (legate/rag/ingestion.py): parse (TXT/MD native, PDF via pypdf, DOCX via python-docx) → overlapping chunking → embed → upsert, with pending → processing → ready | failed status tracking; idempotent, retryable re-ingestion.
  • Local object storage for uploads (legate/rag/storage.py); Celery ingestion task + inline mode (KB_INGEST_INLINE).
  • Retrieval with citations (legate/rag/retrieval.py); the kb.search tool now performs real per-workspace retrieval (degrading to empty when unconfigured).
  • Knowledge API: KB CRUD, document upload/list/get/delete/reingest, and POST /knowledge-bases/{id}/search.

Acceptance: upload a document, watch it ingest, and have an agent answer from it with citations; deleting a document removes its vectors; ingestion failures are visible (status=failed, error) and retryable via /documents/{id}/reingest. Verified by unit tests, API tests, an end-to-end agent-RAG test, and a 21/21 live-server smoke run.

Decisions / TODOs

  • # TODO(decision): kb.search searches all of a workspace's knowledge bases; a per-agent KB scoping option can come with the agent builder UI.
  • S3 storage backend and embedding via Ollama are deferred (local storage + OpenAI/hashing now).
  • Document→Chunk cleanup relies on Postgres ON DELETE CASCADE in production; the API also clears chunks/vectors explicitly on delete.

Phase 4 — Connector framework and credential vault ✅

Done

  • Connector interface + registry (legate/connectors): Connector protocol with validate_config, list_tools, test_connection, execute; a self-registering catalog so adding a connector needs no core changes.
  • First real connectors: HTTP (http.call, authenticated/any-method, sensitive), Slack (slack.post_message / list_channels / reply_thread), Email (email.send (sensitive) / email.draft / email.search).
  • Connector + Credential models and Alembic migration.
  • Envelope-encrypted credential vault (legate/vault): per-record DEK wrapped by a versioned master key (LEGATE_MASTER_KEY); rewrap supports key rotation. Plaintext only in memory; never logged or returned.
  • Connector API: install / list / get / update / delete, POST /connectors/{id}/test, GET /connectors/{id}/tools, and GET /connector-types. Secrets are write-only.
  • Tool resolution (legate/connectors/resolution.py): each run's tool registry = built-ins + the workspace's installed-connector tools; a connector tool decrypts its credential only when invoked.
  • Connector tools flow through the existing policy gate and JSON-schema arg validation; httpx/SMTP/IMAP behind injection seams for offline tests.

Acceptance: install a Slack connector with credentials, test the connection, and have an agent post a message; secrets are encrypted at rest and never returned by the API; adding a new connector requires no core changes. Verified by vault tests, connector unit tests, connector API tests (incl. a secret-leakage check), an end-to-end agent-posts-to-Slack test, and a 24/24 live-server smoke run.

Decisions / TODOs

  • # TODO(decision): slack.post_message is non-sensitive (so the agent can post in Phase 4); http.call and email.send are sensitive and will route through the approval inbox once Phase 5 lands.
  • One credential per connector (1:1); multi-account per type and per-connector tool disambiguation are future work.
  • Connector catalog since expanded well beyond the first three — now ten built-in types: GitHub (github.*), Notion (notion.*), an outgoing Webhook (webhook.send, HMAC-signed), PostgreSQL and MySQL (shared BaseSqlConnector: *.query read-only-guarded / *.execute sensitive), Stripe (stripe.get_customer / list_charges / create_refund sensitive), and Google Workspace (gsheets.* + gdrive.* over one OAuth2 credential) are all implemented and tested. Multi-account-per-type and Drive binary upload remain future work.
  • MCP client support (legate/mcp): Legate connects to remote MCP servers over Streamable HTTP as an mcp connector type. Tools are discovered on install/test and cached on the connector, then registered per-run as mcp.<id>.<tool> (sensitive unless the server marks a tool read-only). Surfaced automatically in the agent allow-list and workflow tool picker. Verified end-to-end against a live MCP server. stdio transport and exposing Legate as an MCP server are future work.

Phase 5 — Policy Engine and approvals ✅

Done

  • Full Policy Engine (legate/policy/engine.py): allow / deny / require_approval, with per-workspace approval rules (auto_approve / deny lists) read from workspace.settings_json.
  • Approval model (+ ApprovalStatus enum) and Alembic migration.
  • Suspend/resume in the runner: on require_approval it snapshots the transcript into an Approval, sets the run to waiting_approval, and stops; AgentRunner.resume rebuilds the transcript and continues — executing the action on approve, or feeding a rejection observation back to replan on reject.
  • Approvals inbox API: GET /approvals, GET /approvals/{id}, POST /approvals/{id}/approve|reject. Deciding requires approvals:decide; resume dispatches inline (tests) or in the background (prod).
  • Expiry (APPROVAL_TTL_HOURS) with auto-expire on read/decide; optional per-approval role gate; 409 on expired or already-decided approvals.
  • Every policy decision audited (policy.deny / policy.require_approval / policy.auto_approve) plus approval.approve / approval.reject.

Acceptance: an agent attempting a sensitive action is paused; an authorized approver approves it from the inbox and the action then executes; rejection resumes the run so the agent replans; an unauthorized user cannot approve; all decisions are audited. Verified by policy-engine unit tests and a full approval-flow suite (suspend / approve+execute / reject+replan / RBAC / expiry / double-decide / auto-approve / tenant isolation / audit), plus a 28/28 live-server smoke run.

Decisions / TODOs

  • Reject resumes the run so the agent can replan (rather than hard-stopping); a "reject-and-cancel" option can be added later.
  • Approval expiry is evaluated lazily (on read/decide); a Celery-beat sweep for proactive expiry is a later addition.
  • required_role is stored and enforced when set; the UI to set it arrives with the dashboard.

Phase 6 — Workflow engine and triggers ✅

Done

  • Workflow model (+ runs.workflow_id / runs.state_json) and Alembic migration.
  • Graph executor (legate/workflow/executor.py): topological execution with edge/branch following, one Step per node, so runs are fully reconstructable.
  • Node executors: trigger, agent (nested agent run), tool (through the policy engine), condition, transform, approval, delay (subworkflow reserved).
  • Sandboxed {{ }} templating (legate/workflow/templating.py) with a hand-written AST evaluator for conditions — no code execution.
  • Triggers: manual (POST /workflows/{id}/run), webhook (HMAC-signed POST /hooks/{token}, invalid → 401), schedule (cron via croniter + a Celery-beat task), and internal events (emit_event).
  • Pause/resume: approval nodes and sensitive tool nodes suspend the run and resume through the existing approvals inbox (state in run.state_json).
  • Per-node retry with backoff.
  • Workflow API: CRUD, run, and the signed webhook hook; managing requires build:write, running requires runs:execute.

Acceptance: build a workflow (webhook → agent → condition → tool) and fire the webhook to execute it end to end including an approval gate; a scheduled workflow runs on cron; invalid webhook signatures are rejected; runs are fully reconstructable from Steps. Verified by templating unit tests and a workflow suite (manual/condition/webhook/agent-node/approval-gate/reject/slack-tool/retry/schedule/event/RBAC/tenancy), plus a 34/34 live-server smoke run.

Decisions / TODOs

  • # TODO(decision): trigger config lives on the Workflow (trigger_type + trigger_config + webhook_*) rather than a separate Trigger table.
  • delay nodes continue immediately in inline execution; a scheduled delayed-resume is a later addition. subworkflow nodes are deferred.
  • Node join semantics are OR (a node runs if any incoming edge is taken); richer AND/wait-all joins can come later.

Phase 7 — Dashboard UX ✅

Done

  • Premium Next.js 14 dashboard (apps/web) with an original "Ink & Amber control-deck" design system — distinctive fonts (Bricolage Grotesque / Hanken Grotesk / JetBrains Mono), a single authoritative amber accent, monospace precision for IDs/hashes, and a subtle layered atmosphere.
  • Auth screens (login/register) with a two-column brand aside; client auth context with JWT storage + silent refresh + workspace switching.
  • App shell: sidebar nav (Overview/Build/Operate/Govern), workspace switcher, role-aware controls, mobile drawer.
  • Agent builder (prompt, model, tool allow-list with sensitive badges, guardrail budgets) + list + run.
  • Workflow builder (JSON graph editor + trigger picker; webhook credentials shown once) + list + run.
  • Run viewer with live WebSocket streaming — a step-by-step "mission log" timeline with per-step input/output.
  • Approval inbox (approve/reject with reason, tabs, live refresh).
  • Connector setup wizard (schema-driven install for any connector type) + test connection.
  • Knowledge base manager (create, upload, per-KB document status, search with citations).
  • Audit explorer (hash-chained, tamper-evident) and Admin (members + roles, API keys shown once).
  • Backend: CORS middleware for the dashboard origin (CORS_ORIGINS).
  • Playwright e2e covering the full acceptance path (apps/web/tests/e2e/core-path.spec.ts).

Acceptance: a non-technical user can register, create an agent, connect Slack, build and run a workflow, approve an action, and read the audit trail — all from the UI, no API calls. Verified by a green Playwright e2e run against the live stack, plus manual screenshots of every core screen.

Decisions / TODOs

  • The workflow builder ships as a JSON/form editor (per the plan's de-risking note); a visual graph editor is a follow-up.
  • Data fetching uses TanStack Query; the run viewer streams over the existing WebSocket.

Phase 8 — Enterprise hardening ✅

Done

  • Billing & plans (legate/billing): free/pro/enterprise plans with limits + features; enforcement on agents/workflows/members/run-quota (402 over limit); usage metering (UsageRecord + migration); GET /billing, GET /usage; Stripe checkout + webhook that updates a workspace's plan. Gated by BILLING_ENABLED.
  • SSO via OIDC (legate/security/oidc.py, legate/api/sso.py): authorization-code login + callback with just-in-time user provisioning; swappable client so tests run against a simulated IdP. SAML is a documented extension point.
  • Rate limiting (legate/security/ratelimit.py): per-workspace run limiter (429), gated by RATE_LIMIT_ENABLED; plan run quotas layer on top.
  • Observability: Prometheus /metrics + request middleware + run counters; OpenTelemetry tracing (config-gated, optional [tracing] extra); structured JSON logging in production.
  • CORS for the dashboard origin.
  • Infra: TLS deploy/Caddyfile; a full kustomize k8s manifest set (deploy/k8s/* — configmap/secret template, Postgres/Redis/Qdrant, api/worker/beat/web, ingress, migrate init-container); hardened docker-compose.prod.yml overlay.
  • Docs: docs/backup.md (backup + restore, incl. the master-key caveat), docs/operations.md.
  • CI security pass: CodeQL (existing) + pip-audit + gitleaks secret scan.

Acceptance: plans gate features and meter usage (402 + /usage); SSO login works against a (test) IdP; metrics and traces are exportable (/metrics, OTLP); a documented restore procedure brings the system back (incl. verifying the master key still unwraps credentials); CI runs CodeQL + dependency + secret scans. Verified by billing/SSO/rate-limit/metrics tests.

Decisions / TODOs

  • OIDC is implemented; SAML is documented as an extension behind the same provisioning seam.
  • The rate limiter is in-process (fine for one replica); a Redis-backed limiter drops in behind enforce_run_rate for multi-replica.
  • k8s manifests ship as kustomize; a packaged Helm chart is a straightforward follow-up.

🎉 All phases complete

Phases 0–8 are done: the agent runtime, RAG, connectors + vault, policy + approvals, workflows, the dashboard, and enterprise hardening. See each phase section above for details.