diff --git a/echo/AGENTS.md b/echo/AGENTS.md index 9c66edc0..e60e43b6 100644 --- a/echo/AGENTS.md +++ b/echo/AGENTS.md @@ -2,17 +2,17 @@ Context for AI coding assistants on the ECHO codebase. Only patterns and rules that aren't obvious from one read of the relevant file live here. -ECHO is an event-driven platform for collective sense-making — workshops, consultations, civic forums collect and analyze conversations. +ECHO is an event-driven platform for collective sense-making: workshops, consultations, civic forums collect and analyze conversations. ## Maintenance Protocol - Read this file before making changes. Fix stale links/paths immediately when you spot them -- Rely on `git log` / `git blame` for timing — no manual timestamps in this file +- Rely on `git log` / `git blame` for timing. No manual timestamps in this file - Auto-correct typos and formatting without asking; escalate only on new patterns or contradictions - Keep instructions aligned with repo reality. If something drifts, repair it - Skip documenting secrets, temporary hacks, or anything that would rot within a sprint -When to propose an addition — primary signal is **what the user just told you**: +When to propose an addition. The primary signal is **what the user just told you**: - User taught a convention ("we use X here", "never do Y", "the reason for Z is…") → "Add this to AGENTS.md?" - User corrected your approach with a rule that would help another teammate → "Capture this?" @@ -24,38 +24,52 @@ Secondary signals (look for these on top of user input, not instead of it): - Same pattern recurring across files you already had to read for the task - A bug fix where the root cause would surprise a reader -What **not** to add: anything a smart model can derive in ≤2 turns from `ls`, `cat package.json`, `git log`, or a single file read. Repo structure, file inventories, TODO lists, build commands, dep versions, change hotspots — leave those to the tooling. +What **not** to add: anything a smart model can derive in ≤2 turns from `ls`, `cat package.json`, `git log`, or a single file read. Repo structure, file inventories, TODO lists, build commands, dep versions, change hotspots: leave those to the tooling. ## Stakeholder Q&A docs Docs like `*-QUESTIONS-FOR-.md` follow a tag-in-place convention so pending vs answered stays scannable: -- `🔴 blocking` — blocks other work -- `🟡 non-blocking` — can proceed without -- `✅ answered ` — resolved +- `🔴 blocking`: blocks other work +- `🟡 non-blocking`: can proceed without +- `✅ answered `: resolved -New questions go to the top. Answered questions **stay in place** — don't move them to an "Answered" section at the bottom. Update the heading tag to `✅ answered ` and add an `**Answer:**` line near the top of the block. +New questions go to the top. Answered questions **stay in place**. Don't move them to an "Answered" section at the bottom. Update the heading tag to `✅ answered ` and add an `**Answer:**` line near the top of the block. ## Brand & UI Copy Follow `brand/STYLE_GUIDE.md` for all user-facing text. - Shortest possible, highest clarity. No jargon. -- Never say "AI" — use "language model" or just describe the action ("Generating your report…" not "Generating with AI…") -- Never say "successfully" — state what happened ("Saved", not "Successfully saved") +- **Never use em dashes (—)** in user-facing copy OR in these agent docs. Use periods, commas, colons, or "and". Agents mimic the style of the doc they're reading, so this rule applies here too +- Never say "AI". Use "language model" or just describe the action ("Generating your report…" not "Generating with AI…") +- Never say "successfully". State what happened ("Saved", not "Successfully saved") - "dembrane" is always lowercase, even at sentence start -- Never use bold for emphasis — use Royal Blue (`#4169e1`) or italics +- Never use bold for emphasis. Use Royal Blue (`#4169e1`) or italics - Say "participants/hosts", not "users" - Dutch translations use informal "je/jij"; keep English terms when they sound better (Dashboard, Upload, Chat) -- Italian translations use informal "tu", target A2 reading level, sentence case for titles, active voice over passive — see `brand/STYLE_GUIDE.md` for the glossary +- Italian translations use informal "tu", target A2 reading level, sentence case for titles, active voice over passive. See `brand/STYLE_GUIDE.md` for the glossary ## UI Rules -- Never stack multiple `Alert` components — pick one +### Buttons and colors + +The Mantine theme already sets `` and the brand styling applies. + +- **Never** pass `variant="default"` on `Button` or `ActionIcon`. The "default" gray Mantine look is off-brand +- **Never** pass `color="blue"`. Use `color="primary"` (or omit; primary is the theme default). Royal Blue is already `primary` in the theme +- Allowed `variant` values: omit (filled), `"outline"`, `"subtle"`. Use `"light"` only when nothing else fits, never as a stylistic default +- For destructive actions: `color="red"` is correct (`ConfirmModal` already handles this) +- Don't hardcode hex colors in components. Use Mantine color tokens (`primary`, `parchment`, `graphite`, etc. from `src/colors.ts`) or Tailwind classes from the theme +- Chat mode accents in `ChatModeSelector` are an intentional exception (theme-independent identity); see `frontend/AGENTS.md` + +### Components + +- Never stack multiple `Alert` components, pick one - Don't use `@mantine/charts` - Loading spinners: always pass `alwaysDembrane` on `DembraneLoadingSpinner` for whitelabel safety; never `animate-spin` on custom logos - Show emails only on hover, never in list rows by default -- Conversations come from QR codes or audio uploads — never add "new conversation" buttons +- Conversations come from QR codes or audio uploads, never add "new conversation" buttons - Prefer text buttons over icon-only buttons for important actions - Destructive actions: `ConfirmModal` (`confirmColor="red"`), never `window.confirm` - Single-field prompts: `InputModal`, never `window.prompt` @@ -63,7 +77,7 @@ Follow `brand/STYLE_GUIDE.md` for all user-facing text. ## Translations -- Lingui — `` component or `` t` `` template literal +- Lingui: `` component or `` t` `` template literal - Supported: en-US, nl-NL, de-DE, fr-FR, es-ES, it-IT - Workflow: `pnpm messages:extract` → edit `.po` files → `pnpm messages:compile` @@ -81,7 +95,7 @@ See [docs/branching_and_releases.md](docs/branching_and_releases.md) for the ful - **Feature flow**: branch off `main` → (optional) `testing` → PR to `main` → auto-deploys to Echo Next - **Releases**: tagged from `main` every ~2 weeks → auto-deploys to production - **Hotfixes**: branch off release tag → fix → new release → cherry-pick back into `main` -- Always check for Directus data migrations before deploying — see [docs/database_migrations.md](docs/database_migrations.md) +- Always check for Directus data migrations before deploying. See [docs/database_migrations.md](docs/database_migrations.md) ## Architecture @@ -96,13 +110,13 @@ Frontend (React/Vite/Mantine) → Backend API (FastAPI) → Directus (headle Agent Service (LangGraph, port 8001) ``` -- **Directus** is the data layer — all collections (projects, conversations, reports) live there +- **Directus** is the data layer; all collections (projects, conversations, reports) live there - **LiteLLM** routes all LLM calls with automatic failover between deployments - **Agent service** runs separately on port 8001; agentic chat streams via `POST /api/agentic/runs/{run_id}/stream` (no Dramatiq dispatch). The runtime is reconnect-driven and lease-based in Redis ### BFF Pattern -Backend-for-frontend routes under `/bff/` aggregate data the frontend needs into one call — prefer this over having the frontend make multiple Directus SDK calls. Example: `/bff/projects/home` bundles pinned projects, paginated list, search, and admin info. +Backend-for-frontend routes under `/bff/` aggregate data the frontend needs into one call; prefer this over having the frontend make multiple Directus SDK calls. Example: `/bff/projects/home` bundles pinned projects, paginated list, search, and admin info. ### URL-Driven State @@ -114,31 +128,31 @@ Long-running progress streams via Server-Sent Events backed by Redis pub/sub (re ### Dramatiq & Async Rules -- **No `asyncio` in Dramatiq actors** — recurring event-loop corruption bugs led to this. Use `gevent` pools + `dramatiq.group()` instead. Report generation is fully synchronous +- **No `asyncio` in Dramatiq actors**. Recurring event-loop corruption bugs led to this. Use `gevent` pools + `dramatiq.group()` instead. Report generation is fully synchronous - `gevent.pool.Pool` is only safe on the `network` queue (uses `dramatiq-gevent`); the CPU queue runs standard dramatiq - Use `gevent.sleep()` (not `time.sleep()`) in network-queue actors -- Restart workers after changing actor signatures — positional args are serialized +- Restart workers after changing actor signatures; positional args are serialized - `SkipRetryOnUnrecoverableError` middleware skips retries for `TypeError`, `SyntaxError`, `AttributeError`, `ImportError`, `NotImplementedError` -- To invoke async code from a Dramatiq actor: `run_async_in_new_loop` from `dembrane.async_helpers` — never `asyncio.run` (clashes with nested loops) +- To invoke async code from a Dramatiq actor: `run_async_in_new_loop` from `dembrane.async_helpers`. Never `asyncio.run` (clashes with nested loops) - Wrap blocking I/O in async endpoints with `run_in_thread_pool` from `dembrane.async_helpers` (Directus, service-layer, S3, token counting). Don't wrap already-async calls (e.g. `rag.aquery`) ### LLM Model Groups -Which group powers which feature is non-obvious — don't downgrade silently. +Which group powers which feature is non-obvious, so don't downgrade silently. -- `MULTI_MODAL_PRO` (Gemini 2.5 Pro) — chat, report generation, transcript correction. **Do not downgrade chat to Flash** -- `MULTI_MODAL_FAST` (Gemini 2.5 Flash) — suggestions, verification, stateless endpoints -- `TEXT_FAST` (Azure GPT-4.1) — being deprecated, migrating to Gemini +- `MULTI_MODAL_PRO` (Gemini 2.5 Pro): chat, report generation, transcript correction. **Do not downgrade chat to Flash** +- `MULTI_MODAL_FAST` (Gemini 2.5 Flash): suggestions, verification, stateless endpoints +- `TEXT_FAST` (Azure GPT-4.1): being deprecated, migrating to Gemini - Report prompt templates are written **in the target language**, not English with a "write in X" instruction - LLM router supports failover: define primary as `LLM____*` and fallbacks as `LLM___1__*`, `_2__*`, etc. ### Transcription - AssemblyAI `universal-3-pro` supports en, es, pt, fr, de, it -- Dutch (`nl`) **requires** `universal-2` fallback — `universal-3-pro` does not support it +- Dutch (`nl`) **requires** `universal-2` fallback; `universal-3-pro` does not support it - Production uses webhook mode (`ASSEMBLYAI_WEBHOOK_URL`); polling is only a fallback - After raw transcription, a Gemini pass corrects, normalizes hotwords, redacts PII, and adds recording feedback -- Load S3 audio via the shared file service (`_get_audio_file_object`) — signed URLs may expire mid-request +- Load S3 audio via the shared file service (`_get_audio_file_object`); signed URLs may expire mid-request ## Directus Rules (Critical) @@ -147,18 +161,18 @@ Which group powers which feature is non-obvious — don't downgrade silently. 1. Write an idempotent Python script that uses the Directus REST API (`POST /collections`, `POST /fields`, `POST /relations`) with the admin token. Check `collection_exists()` / `field_exists()` before creating 2. Run it step-by-step to verify each change against a local Directus 3. Pull the schema: `cd directus && bash sync.sh -u http://directus:8055 -e admin@dembrane.com -p admin pull` -4. Commit the snapshot JSON under `directus/sync/snapshot/` — that is the source of truth; the one-shot migration script does not need to be committed +4. Commit the snapshot JSON under `directus/sync/snapshot/`. That is the source of truth; the one-shot migration script does not need to be committed ### Python DirectusClient -- `create_item` / `update_item` return `{"data": {...}}` — **MUST** unwrap with `["data"]` +- `create_item` / `update_item` return `{"data": {...}}`. **MUST** unwrap with `["data"]` - `get_items` / `get_item` return data directly (no wrapper) - `get_items` requires `{"query": {filter, fields, sort, ...}}` wrapper -- `search()` silently returns `{"error": "..."}` on failure — always validate the return is a list before iterating +- `search()` silently returns `{"error": "..."}` on failure; always validate the return is a list before iterating ### TypeScript Directus SDK -- Auto-unwraps everything — no `["data"]` needed +- Auto-unwraps everything, no `["data"]` needed - Type error on `.count`? Add the type to `typesDirectus.d.ts` and use `count("")` in fields ### File Cleanup @@ -173,6 +187,6 @@ See `server/dembrane/api/user_settings.py` (`remove_avatar`, `remove_whitelabel_ ## Project Management -- Linear for issue tracking — tickets are `ECHO-xxx` +- Linear for issue tracking; tickets are `ECHO-xxx` - Two-week cycles - GitOps repo: `dembrane/echo-gitops` (separate repo, vendored under `echo-gitops/`) diff --git a/echo/frontend/AGENTS.md b/echo/frontend/AGENTS.md index 06accd95..e271053f 100644 --- a/echo/frontend/AGENTS.md +++ b/echo/frontend/AGENTS.md @@ -1,16 +1,43 @@ -# AGENTS — frontend +# AGENTS: frontend Cross-cutting rules (brand, UI, Directus, BFF, architecture, translations) live in @../AGENTS.md, which also defines the maintenance protocol for these files. This file only adds frontend-specific patterns and non-obvious gotchas. ## Patterns - **React Query hook hubs**: each feature owns a `hooks/index.ts` exposing `useQuery`/`useMutation` wrappers with shared `useQueryClient` invalidation. See `src/components/{conversation,project,chat,participant,...}/hooks/index.ts` -- **Lingui macros**: routed screens import `t` from `@lingui/core/macro` and `Trans` from `@lingui/react/macro` — not the runtime imports +- **Lingui macros**: routed screens import `t` from `@lingui/core/macro` and `Trans` from `@lingui/react/macro`, not the runtime imports - **Mantine + Tailwind blend**: compose with Mantine primitives (`Stack`, `Group`, `ActionIcon`) and layer Tailwind utility classes via `className` on the same element - **Custom Directus POSTs** (e.g. 2FA) use `directus.request` with a function signature, not `restRequest`. Reuse `postDirectus` from `src/components/settings/hooks/index.ts` - **Auth session state** lives under the `['auth','session']` React Query key. Invalidate it on login/logout before fetching `['users','me']` -- **2FA flow**: Directus surfaces it by returning `INVALID_OTP` — toggle a Mantine `PinInput` field and retry the same mutation. See `src/routes/auth/Login.tsx` -- **Transitions**: login/logout flows call `useTransitionCurtain().runTransition()` before navigation — animations expect the Directus mutation promise to be awaited +- **2FA flow**: Directus surfaces it by returning `INVALID_OTP`. Toggle a Mantine `PinInput` field and retry the same mutation. See `src/routes/auth/Login.tsx` +- **Transitions**: login/logout flows call `useTransitionCurtain().runTransition()` before navigation; animations expect the Directus mutation promise to be awaited + +## Buttons and brand colors + +The Mantine theme (`src/theme.tsx`) already sets ` + +// Correct: explicit outline / subtle when needed + + + +// Wrong: variant="default" is the off-brand gray Mantine default + + +// Wrong: color="blue" is raw Mantine blue, not brand Royal Blue + +... +``` + +Rules: + +- Allowed `Button` / `ActionIcon` variants: omit (filled), `"outline"`, `"subtle"`. `"light"` only when nothing else fits +- Allowed colors: `"primary"` (or omit), `"red"` for destructive, brand accent keys from `src/colors.ts`. Never `"blue"` +- Don't hardcode hex colors in components. Use Mantine color tokens or Tailwind classes from the theme +- The Royal Blue brand color **is** `color="primary"`. There is no reason to ever pass `color="blue"` ## Sidebar Navigation @@ -22,15 +49,15 @@ Cross-cutting rules (brand, UI, Directus, BFF, architecture, translations) live ## Analytics (PostHog) - `posthog-js` + `@posthog/react` are initialized in `src/main.tsx`; the app is wrapped in `PostHogProvider` -- Call `posthog.identify(email)` on login and registration, `posthog.reset()` on logout — never identify by Directus user id +- Call `posthog.identify(email)` on login and registration, `posthog.reset()` on logout. Never identify by Directus user id - Event naming: `snake_case` past-tense verb (`user_logged_in`, `project_created`, `chat_message_sent`) -- Current tracked events — grep for `posthog.capture(` to verify the live set: +- Current tracked events (grep for `posthog.capture(` to verify the live set): - `user_logged_in`, `user_login_failed`, `user_registered`, `user_logged_out` - `project_created` - `chat_mode_selected`, `chat_message_sent` - `report_generated` - `conversation_upload_started` -- Dashboard + insights live in the PostHog EU project (id 160282). Don't add new dashboards from code — wire the event and let analytics own the visualization +- Dashboard + insights live in the PostHog EU project (id 160282). Don't add new dashboards from code; wire the event and let analytics own the visualization ## Modal conventions @@ -43,10 +70,10 @@ Cross-cutting rules (brand, UI, Directus, BFF, architecture, translations) live Theme is driven by CSS variables, not Tailwind tokens, so `dark:` classes don't propagate. Use the variables when colors need to follow the active theme. - Variables defined in `src/index.css`, updated at runtime by `src/hooks/useAppPreferences.tsx`: - - `--app-background` — page/component background - - `--app-text` — default text color - - `--app-font-family` — font family -- Font preference is **linked** to a color scheme — switching font also switches the palette: + - `--app-background`: page/component background + - `--app-text`: default text color + - `--app-font-family`: font family +- Font preference is **linked** to a color scheme; switching font also switches the palette: - DM Sans → Parchment `#F6F4F1` background + Graphite `#2D2D2C` text - Space Grotesk → White background + Black text - Mantine theme (`src/theme.tsx`) overrides `white` and `black` and pins Modal/Drawer/Popover backgrounds to `var(--app-background)` diff --git a/echo/server/AGENTS.md b/echo/server/AGENTS.md index f1745c7b..14c4f9a5 100644 --- a/echo/server/AGENTS.md +++ b/echo/server/AGENTS.md @@ -1,29 +1,29 @@ -# AGENTS — server +# AGENTS: server Cross-cutting rules (Directus, BFF, LLM model groups, Dramatiq/gevent, transcription, brand) live in @../AGENTS.md, which also defines the maintenance protocol for these files. This file only adds server-specific patterns and non-obvious gotchas. ## Patterns - Local entry points always go through `uv run` so env + deps stay consistent (uvicorn, scheduler, dramatiq runners) -- For API handlers reading project/conversation data, prefer Directus queries over raw SQLAlchemy sessions — keeps behavior aligned with the admin console -- Config lives in `dembrane/settings.py`. Add new env vars as fields on `AppSettings`; group accessors (e.g. `feature_flags`, `directus`) when multiple modules read them. Fetch at runtime with `settings = get_settings()` — **never import env vars directly** +- For API handlers reading project/conversation data, prefer Directus queries over raw SQLAlchemy sessions; keeps behavior aligned with the admin console +- Config lives in `dembrane/settings.py`. Add new env vars as fields on `AppSettings`; group accessors (e.g. `feature_flags`, `directus`) when multiple modules read them. Fetch at runtime with `settings = get_settings()`. **Never import env vars directly** - Embeddings: populate `EMBEDDING_*` env vars (model, key, base URL, version) before calling `dembrane.embedding.embed_text`. The placeholder in `dembrane/embedding.py` is not yet the production implementation -- Production API uses a **custom asyncio uvicorn worker** (`dembrane.asyncio_uvicorn_worker.AsyncioUvicornWorker`) — avoid `uvloop` for `nest_asyncio` compatibility +- Production API uses a **custom asyncio uvicorn worker** (`dembrane.asyncio_uvicorn_worker.AsyncioUvicornWorker`); avoid `uvloop` for `nest_asyncio` compatibility ## Background task design When fixing or extending Dramatiq flows: -1. **Fix root causes, not symptoms** — if a flag isn't being set, fix the flag-setting logic rather than adding catch-up task workarounds +1. **Fix root causes, not symptoms**. If a flag isn't being set, fix the flag-setting logic rather than adding catch-up task workarounds 2. **Single source of truth per flag**: - - `is_finished` — user/system marked the conversation done - - `is_all_chunks_transcribed` — ready for summarization (audio **and** text conversations) - - `summary != null` — summarization complete -3. **Layered reconciliation, simple catch-ups** — each layer has a normal flow (event-triggered) and a catch-up flow (scheduler finds stuck items). Catch-up tasks should check exactly **one** flag, not compound conditions: + - `is_finished`: user/system marked the conversation done + - `is_all_chunks_transcribed`: ready for summarization (audio **and** text conversations) + - `summary != null`: summarization complete +3. **Layered reconciliation, simple catch-ups**. Each layer has a normal flow (event-triggered) and a catch-up flow (scheduler finds stuck items). Catch-up tasks should check exactly **one** flag, not compound conditions: - L1 `task_collect_and_finish_unfinished_conversations` (~2 min) → sets `is_finished=True` for abandoned conversations - L2 `task_reconcile_transcribed_flag` (~3 min) → sets `is_all_chunks_transcribed=True` for finished conversations with no pending chunks - L3 `task_catch_up_unsummarized_conversations` (~5 min) → `is_all_chunks_transcribed=True AND summary=null` → summarize -4. **TEXT and AUDIO conversations share the same state machine** — both must converge to the same flags +4. **TEXT and AUDIO conversations share the same state machine**; both must converge to the same flags ## Worker tuning @@ -32,7 +32,7 @@ When fixing or extending Dramatiq flows: ## Verification topic reconciliation -Verification topics reconcile at startup (`reconcile_default_verification_topics`) and use a Redis lock `dembrane:verification_topics:reconcile_lock` (5m TTL). If logs say another worker holds the lock, wait for it to release — or manually delete the key if a crash left it behind. +Verification topics reconcile at startup (`reconcile_default_verification_topics`) and use a Redis lock `dembrane:verification_topics:reconcile_lock` (5m TTL). If logs say another worker holds the lock, wait for it to release, or manually delete the key if a crash left it behind. ## Agentic runtime